734
Pl
pi-linux-use / files
pi-linux-use / pi-linux-use.md
9.1 KB
1# Pi Linux Use 2 3Idea: build a Pi extension that gives the agent computer-use tools for Linux desktop environments. The extension would expose tool calls for observing the desktop/window state and performing GUI actions such as clicking, typing, scrolling, waiting, and selecting target windows. 4 5Known facts: 6 7- Pi supports TypeScript extensions that can register custom LLM tools through `pi.registerTool()`. 8- Pi tool results can include images (`{ type: "image", data, mimeType }`), so an observe tool can return a screenshot directly to vision-capable models. 9- Linux desktop access depends on the session/compositor. Wayland and X11 require different approaches. 10- [dotool](https://git.sr.ht/~geb/dotool) gives a full input control api. 11 12Open questions: 13 14- desktop-agnostic `xdg-desktop-portal` APIs: ScreenCast/PipeWire for observation? How do we let the agent "see" 15- does the mvp include a discoverability api? To list open windows, and other data driven operations, this would be an optimization to the simple capture + input base. 16 17## Observability layer research 18 19Goal: provide the agent with enough visual and structural context to decide where to click/type. Treat observability as layered, not a single API: 20 211. **Raster view**: screenshot/frame image + coordinate system. 222. **Display topology**: monitors, scale, global/stream coordinates, cursor position. 233. **Window metadata**: active window, titles, bounds, app IDs, z-order where available. 244. **Semantic UI**: accessibility tree, OCR, and/or CV-derived elements. 255. **Wait/change detection**: wait for screen/window/accessibility changes after actions. 26 27### Capture backends 28 29#### XDG desktop portals 30 31Best cross-desktop Wayland-safe direction, but designed around explicit user consent. 32 33- `org.freedesktop.portal.Screenshot` 34 - `Screenshot(parent_window, options) -> uri`. 35 - Options include `interactive` and `modal`. 36 - Good fallback/manual capture path. 37 - Weakness: likely prompts per capture and has no restore token/session stream, so it is not ideal for frequent agent observe loops. 38 39- `org.freedesktop.portal.ScreenCast` + PipeWire 40 - Flow: `CreateSession -> SelectSources -> Start -> OpenPipeWireRemote`. 41 - Source types: `MONITOR=1`, `WINDOW=2`, `VIRTUAL=4`. 42 - Cursor modes: `Hidden=1`, `Embedded=2`, `Metadata=4`. 43 - `Start` returns PipeWire stream node IDs plus metadata such as `position`, `size`, `source_type`, `id`, `mapping_id`. 44 - Supports `persist_mode` and one-use `restore_token`, so this is the likely “real” desktop-agnostic observer for a long-running session. 45 - Implementation note: `OpenPipeWireRemote` returns a Unix FD. A plain `gdbus` shell call is not enough; use a helper/library that can receive the FD and pass it to PipeWire/GStreamer. `gst pipewiresrc` supports `fd` and `path`, so a small Python/Rust helper could expose “latest frame as PNG/JPEG” to the TypeScript extension. 46 47- `org.freedesktop.portal.RemoteDesktop` 48 - Same consent/session model, but includes input devices (`KEYBOARD=1`, `POINTER=2`, `TOUCHSCREEN=4`) and can include ScreenCast sources. 49 - Potential future replacement for `dotool` on Wayland if we want portal-sanctioned input + observation with matching coordinate spaces. 50 51#### Native screenshot helpers 52 53Useful for MVP because they are simple and often work without repeated prompts, but they are not universal. 54 55- KDE/Plasma Wayland: `spectacle -b -n -f -o <file>` works for full-desktop screenshots; KWin also exposes `org.kde.KWin.ScreenShot2` D-Bus methods. 56- wlroots compositors: `grim`/`slurp` for screenshots/regions, compositor IPC (`swaymsg`, `hyprctl`) for metadata. 57- GNOME Wayland: `gnome-screenshot` or GNOME Shell screenshot D-Bus where available; portal ScreenCast is the more portable path. 58- X11: `xwd`, ImageMagick `import`, `maim`/`scrot` are straightforward; window metadata via EWMH tools. 59 60### Discoverability / metadata 61 62- **Session detection**: inspect `XDG_SESSION_TYPE`, `WAYLAND_DISPLAY`, `DISPLAY`, `XDG_CURRENT_DESKTOP`, and available commands/services. 63- **Monitor geometry**: 64 - X11: `xrandr`. 65 - KDE: `kscreen-doctor -o`. 66 - GNOME/wlroots: compositor-specific APIs where available. 67 - Portal ScreenCast stream metadata gives selected-stream position/size, but note logical compositor coordinates may differ from pixel coordinates under scaling. 68- **Window list**: 69 - X11: `wmctrl`, `xdotool`, `xprop`, `xwininfo` via EWMH. 70 - Wayland: intentionally not globally exposed. Need compositor-specific adapters: 71 - KDE: KWin scripting/D-Bus can expose windows. 72 - wlroots: foreign-toplevel protocols via compositor-specific tools. 73 - GNOME: likely requires Shell extensions/private APIs; do not rely on generic enumeration. 74 - Therefore window listing should be best-effort and optional, not a prerequisite for MVP. 75- **Accessibility tree**: 76 - AT-SPI (`org.a11y.Bus`) can provide roles, names, bounds, actions, focus, and events for apps that support accessibility. 77 - Cross-toolkit but uneven; may require enabling accessibility. Great semantic layer after screenshot MVP. 78- **OCR/CV fallback**: 79 - OCR can turn screenshots into clickable text targets when accessibility/window APIs are unavailable. 80 - CV can identify buttons, text fields, cursor, and image diffs, but should be additive because raw screenshots are already useful to vision models. 81 82### Proposed tool surface 83 84MVP tools: 85 86- `desktop_observe({ target?: "desktop" | "active_window" | "monitor" | "region", region?, maxWidth?, includeCursor? })` 87 - Captures one screenshot/frame. 88 - Returns a text summary plus an image attachment for vision models. 89 - `details` should include backend, timestamp, image dimensions, monitor layout, coordinate mapping, and temp file path. 90- `desktop_wait({ condition: "screen_change" | "stable" | "duration", timeoutMs, region? })` 91 - Lets the agent wait after actions without blind sleeps. 92 93Next tools: 94 95- `desktop_state({ includeWindows?: boolean, includeA11y?: boolean })` 96 - Best-effort monitors/windows/focus/accessibility metadata. 97- `desktop_stream({ action: "start" | "stop" | "status", sourceType?: "monitor" | "window", persist?: boolean })` 98 - Starts/stops a portal ScreenCast session and stores restore token/session state. 99- `desktop_zoom({ region, maxWidth? })` 100 - Crops a region from the latest frame for detailed inspection. 101 102### Backend selection sketch 103 1041. Detect environment and model image support. 1052. If Wayland: 106 - KDE: use `spectacle` for MVP screenshots; later use portal ScreenCast or KWin ScreenShot2 helper. 107 - wlroots: use `grim`; later portal ScreenCast if available. 108 - GNOME/unknown: try desktop screenshot helper, else portal Screenshot; for repeated observe, portal ScreenCast helper. 1093. If X11: 110 - screenshot via `xwd`/`import`/`maim`; 111 - metadata via `xrandr` + EWMH tools. 1124. If text-only model: 113 - save screenshot path and return OCR/window metadata; optionally ask user/model switch for vision. 114 115### Local environment notes from this machine 116 117- Current session: `XDG_SESSION_TYPE=wayland`, `XDG_CURRENT_DESKTOP=KDE`, `WAYLAND_DISPLAY=wayland-0`, `DISPLAY=:0`. 118- Portal services are active: `xdg-desktop-portal`, `plasma-xdg-desktop-portal-kde`, PipeWire, WirePlumber. 119- Portal exposes: 120 - `ScreenCast` version 5, `AvailableSourceTypes=7`, `AvailableCursorModes=7`. 121 - `RemoteDesktop` version 2, `AvailableDeviceTypes=7`. 122 - `Screenshot` version 2. 123- `spectacle -b -n -f -o /tmp/pi-observe-test.png` successfully captured a `5120x1440` PNG. 124- `xprop`/`xdotool` only see an XWayland dummy active window here, confirming that X11 metadata tools are not enough on KDE Wayland. 125- Python `gi` + GStreamer are available locally, and `gst-inspect-1.0 pipewiresrc` shows `fd` and `path` properties. This makes a portal ScreenCast helper feasible. 126 127## Recommended path 128 1291. **Build `desktop_observe` first** using pragmatic screenshot helpers and return image tool results. This proves the agent can see and coordinate clicks. 1302. Add **region crop/zoom and wait-for-change** so the agent can operate iteratively without huge images every turn. 1313. Add **best-effort metadata** (`desktop_state`) for monitors and windows where available. 1324. Add a **portal ScreenCast helper** for robust Wayland repeated observation with one user grant. 1335. Add **AT-SPI/OCR** for semantic targeting once the basic screenshot/action loop works. 134 135## Minimal POC 136 137A project-local Pi extension lives at `.pi/extensions/linux-use.ts`. 138 139A project-local Agent Skill lives at `.pi/skills/linux-desktop-use/SKILL.md` with usage guidance for the screenshot/input loop. 140 141Tools: 142 143- `desktop_observe` — calls `spectacle -b -n -f -o <tmp.png>`, returns the screenshot as an image attachment when the active model supports image input, and records the image dimensions for coordinate mapping. 144- `desktop_input` — calls `dotool` with generated stdin actions for click/move/type/key/scroll/raw script. Click/move x/y coordinates are pixels in the latest `desktop_observe` screenshot. 145 146Test flow after installing/configuring dotool: 147 1481. `/reload` in Pi, or start Pi from this repo so `.pi/extensions/linux-use.ts` is auto-discovered. 1492. Ask the agent to call `desktop_observe`. 1503. Ask it to click/type using `desktop_input` with x/y coordinates from the screenshot.