xonecas.io
734
Pl

pi-linux-use / files

pi-linux-use / pi-linux-use.md

9.1 KB
1# Pi Linux Use
2
3Idea: build a Pi extension that gives the agent computer-use tools for Linux desktop environments. The extension would expose tool calls for observing the desktop/window state and performing GUI actions such as clicking, typing, scrolling, waiting, and selecting target windows.
4
5Known facts:
6
7- Pi supports TypeScript extensions that can register custom LLM tools through `pi.registerTool()`.
8- Pi tool results can include images (`{ type: "image", data, mimeType }`), so an observe tool can return a screenshot directly to vision-capable models.
9- Linux desktop access depends on the session/compositor. Wayland and X11 require different approaches.
10- [dotool](https://git.sr.ht/~geb/dotool) gives a full input control api.
11
12Open questions:
13
14- desktop-agnostic `xdg-desktop-portal` APIs: ScreenCast/PipeWire for observation? How do we let the agent "see"
15- does the mvp include a discoverability api? To list open windows, and other data driven operations, this would be an optimization to the simple capture + input base.
16
17## Observability layer research
18
19Goal: provide the agent with enough visual and structural context to decide where to click/type. Treat observability as layered, not a single API:
20
211. **Raster view**: screenshot/frame image + coordinate system.
222. **Display topology**: monitors, scale, global/stream coordinates, cursor position.
233. **Window metadata**: active window, titles, bounds, app IDs, z-order where available.
244. **Semantic UI**: accessibility tree, OCR, and/or CV-derived elements.
255. **Wait/change detection**: wait for screen/window/accessibility changes after actions.
26
27### Capture backends
28
29#### XDG desktop portals
30
31Best cross-desktop Wayland-safe direction, but designed around explicit user consent.
32
33- `org.freedesktop.portal.Screenshot`
34  - `Screenshot(parent_window, options) -> uri`.
35  - Options include `interactive` and `modal`.
36  - Good fallback/manual capture path.
37  - Weakness: likely prompts per capture and has no restore token/session stream, so it is not ideal for frequent agent observe loops.
38
39- `org.freedesktop.portal.ScreenCast` + PipeWire
40  - Flow: `CreateSession -> SelectSources -> Start -> OpenPipeWireRemote`.
41  - Source types: `MONITOR=1`, `WINDOW=2`, `VIRTUAL=4`.
42  - Cursor modes: `Hidden=1`, `Embedded=2`, `Metadata=4`.
43  - `Start` returns PipeWire stream node IDs plus metadata such as `position`, `size`, `source_type`, `id`, `mapping_id`.
44  - Supports `persist_mode` and one-use `restore_token`, so this is the likely “real” desktop-agnostic observer for a long-running session.
45  - Implementation note: `OpenPipeWireRemote` returns a Unix FD. A plain `gdbus` shell call is not enough; use a helper/library that can receive the FD and pass it to PipeWire/GStreamer. `gst pipewiresrc` supports `fd` and `path`, so a small Python/Rust helper could expose “latest frame as PNG/JPEG” to the TypeScript extension.
46
47- `org.freedesktop.portal.RemoteDesktop`
48  - Same consent/session model, but includes input devices (`KEYBOARD=1`, `POINTER=2`, `TOUCHSCREEN=4`) and can include ScreenCast sources.
49  - Potential future replacement for `dotool` on Wayland if we want portal-sanctioned input + observation with matching coordinate spaces.
50
51#### Native screenshot helpers
52
53Useful for MVP because they are simple and often work without repeated prompts, but they are not universal.
54
55- KDE/Plasma Wayland: `spectacle -b -n -f -o <file>` works for full-desktop screenshots; KWin also exposes `org.kde.KWin.ScreenShot2` D-Bus methods.
56- wlroots compositors: `grim`/`slurp` for screenshots/regions, compositor IPC (`swaymsg`, `hyprctl`) for metadata.
57- GNOME Wayland: `gnome-screenshot` or GNOME Shell screenshot D-Bus where available; portal ScreenCast is the more portable path.
58- X11: `xwd`, ImageMagick `import`, `maim`/`scrot` are straightforward; window metadata via EWMH tools.
59
60### Discoverability / metadata
61
62- **Session detection**: inspect `XDG_SESSION_TYPE`, `WAYLAND_DISPLAY`, `DISPLAY`, `XDG_CURRENT_DESKTOP`, and available commands/services.
63- **Monitor geometry**:
64  - X11: `xrandr`.
65  - KDE: `kscreen-doctor -o`.
66  - GNOME/wlroots: compositor-specific APIs where available.
67  - Portal ScreenCast stream metadata gives selected-stream position/size, but note logical compositor coordinates may differ from pixel coordinates under scaling.
68- **Window list**:
69  - X11: `wmctrl`, `xdotool`, `xprop`, `xwininfo` via EWMH.
70  - Wayland: intentionally not globally exposed. Need compositor-specific adapters:
71    - KDE: KWin scripting/D-Bus can expose windows.
72    - wlroots: foreign-toplevel protocols via compositor-specific tools.
73    - GNOME: likely requires Shell extensions/private APIs; do not rely on generic enumeration.
74  - Therefore window listing should be best-effort and optional, not a prerequisite for MVP.
75- **Accessibility tree**:
76  - AT-SPI (`org.a11y.Bus`) can provide roles, names, bounds, actions, focus, and events for apps that support accessibility.
77  - Cross-toolkit but uneven; may require enabling accessibility. Great semantic layer after screenshot MVP.
78- **OCR/CV fallback**:
79  - OCR can turn screenshots into clickable text targets when accessibility/window APIs are unavailable.
80  - CV can identify buttons, text fields, cursor, and image diffs, but should be additive because raw screenshots are already useful to vision models.
81
82### Proposed tool surface
83
84MVP tools:
85
86- `desktop_observe({ target?: "desktop" | "active_window" | "monitor" | "region", region?, maxWidth?, includeCursor? })`
87  - Captures one screenshot/frame.
88  - Returns a text summary plus an image attachment for vision models.
89  - `details` should include backend, timestamp, image dimensions, monitor layout, coordinate mapping, and temp file path.
90- `desktop_wait({ condition: "screen_change" | "stable" | "duration", timeoutMs, region? })`
91  - Lets the agent wait after actions without blind sleeps.
92
93Next tools:
94
95- `desktop_state({ includeWindows?: boolean, includeA11y?: boolean })`
96  - Best-effort monitors/windows/focus/accessibility metadata.
97- `desktop_stream({ action: "start" | "stop" | "status", sourceType?: "monitor" | "window", persist?: boolean })`
98  - Starts/stops a portal ScreenCast session and stores restore token/session state.
99- `desktop_zoom({ region, maxWidth? })`
100  - Crops a region from the latest frame for detailed inspection.
101
102### Backend selection sketch
103
1041. Detect environment and model image support.
1052. If Wayland:
106   - KDE: use `spectacle` for MVP screenshots; later use portal ScreenCast or KWin ScreenShot2 helper.
107   - wlroots: use `grim`; later portal ScreenCast if available.
108   - GNOME/unknown: try desktop screenshot helper, else portal Screenshot; for repeated observe, portal ScreenCast helper.
1093. If X11:
110   - screenshot via `xwd`/`import`/`maim`;
111   - metadata via `xrandr` + EWMH tools.
1124. If text-only model:
113   - save screenshot path and return OCR/window metadata; optionally ask user/model switch for vision.
114
115### Local environment notes from this machine
116
117- Current session: `XDG_SESSION_TYPE=wayland`, `XDG_CURRENT_DESKTOP=KDE`, `WAYLAND_DISPLAY=wayland-0`, `DISPLAY=:0`.
118- Portal services are active: `xdg-desktop-portal`, `plasma-xdg-desktop-portal-kde`, PipeWire, WirePlumber.
119- Portal exposes:
120  - `ScreenCast` version 5, `AvailableSourceTypes=7`, `AvailableCursorModes=7`.
121  - `RemoteDesktop` version 2, `AvailableDeviceTypes=7`.
122  - `Screenshot` version 2.
123- `spectacle -b -n -f -o /tmp/pi-observe-test.png` successfully captured a `5120x1440` PNG.
124- `xprop`/`xdotool` only see an XWayland dummy active window here, confirming that X11 metadata tools are not enough on KDE Wayland.
125- Python `gi` + GStreamer are available locally, and `gst-inspect-1.0 pipewiresrc` shows `fd` and `path` properties. This makes a portal ScreenCast helper feasible.
126
127## Recommended path
128
1291. **Build `desktop_observe` first** using pragmatic screenshot helpers and return image tool results. This proves the agent can see and coordinate clicks.
1302. Add **region crop/zoom and wait-for-change** so the agent can operate iteratively without huge images every turn.
1313. Add **best-effort metadata** (`desktop_state`) for monitors and windows where available.
1324. Add a **portal ScreenCast helper** for robust Wayland repeated observation with one user grant.
1335. Add **AT-SPI/OCR** for semantic targeting once the basic screenshot/action loop works.
134
135## Minimal POC
136
137A project-local Pi extension lives at `.pi/extensions/linux-use.ts`.
138
139A project-local Agent Skill lives at `.pi/skills/linux-desktop-use/SKILL.md` with usage guidance for the screenshot/input loop.
140
141Tools:
142
143- `desktop_observe` — calls `spectacle -b -n -f -o <tmp.png>`, returns the screenshot as an image attachment when the active model supports image input, and records the image dimensions for coordinate mapping.
144- `desktop_input` — calls `dotool` with generated stdin actions for click/move/type/key/scroll/raw script. Click/move x/y coordinates are pixels in the latest `desktop_observe` screenshot.
145
146Test flow after installing/configuring dotool:
147
1481. `/reload` in Pi, or start Pi from this repo so `.pi/extensions/linux-use.ts` is auto-discovered.
1492. Ask the agent to call `desktop_observe`.
1503. Ask it to click/type using `desktop_input` with x/y coordinates from the screenshot.