734
Pl
pi-linux-use / files
pi-linux-use / README.md
1.9 KB
1# Pi Linux Use 2 3Minimal proof-of-concept Pi extension for Linux desktop computer-use. 4 5It gives Pi two tools: 6 7- `desktop_observe` — captures the visible desktop with KDE Spectacle and returns a screenshot to the model. 8- `desktop_input` — sends mouse/keyboard input through `dotool`. 9 10A companion skill, `linux-desktop-use`, teaches the agent the observe → act → observe workflow. 11 12## Requirements 13 14Current POC targets KDE/Plasma Wayland first. 15 16Install: 17 18- `spectacle` for screenshots 19- `dotool` for input 20 21Verify dotool works: 22 23```bash 24echo type hello | dotool 25``` 26 27If dotool cannot access `/dev/uinput`, follow dotool's setup instructions, usually adding your user to the `input` group and reloading udev/rebooting. 28 29## Quick start 30 31From this repo: 32 33```bash 34pi 35``` 36 37Then reload project extensions/skills if Pi is already running: 38 39```text 40/reload 41``` 42 43Ask something like: 44 45```text 46Look at my desktop and tell me what is open. 47``` 48 49or: 50 51```text 52Browse the repo I have open in the browser. 53``` 54 55The agent should use: 56 571. `desktop_observe` to capture the screen. 582. `desktop_input` to click, type, scroll, or press keys. 593. `desktop_observe` again to verify what changed. 60 61## Tool notes 62 63Coordinates for `desktop_input` are pixels in the latest `desktop_observe` screenshot. On multi-monitor setups, the screenshot spans the whole desktop. 64 65Example input actions: 66 67```json 68{ "action": "click", "x": 3335, "y": 548, "button": "left" } 69{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 } 70{ "action": "key", "key": "alt+left" } 71{ "action": "type", "text": "hello" } 72``` 73 74## Using it elsewhere 75 76Copy these into another project: 77 78```text 79.pi/extensions/linux-use.ts 80.pi/skills/linux-desktop-use/SKILL.md 81``` 82 83Run `/reload` in Pi. 84 85## Status 86 87This is intentionally tiny: Spectacle screenshot + dotool input. Future work can add portal ScreenCast/PipeWire, better window metadata, accessibility, OCR, crop/zoom, and wait-for-change helpers.