xonecas.io
734
Pl

pi-linux-use / files

pi-linux-use / README.md

1.9 KB
1# Pi Linux Use
2
3Minimal proof-of-concept Pi extension for Linux desktop computer-use.
4
5It gives Pi two tools:
6
7- `desktop_observe` — captures the visible desktop with KDE Spectacle and returns a screenshot to the model.
8- `desktop_input` — sends mouse/keyboard input through `dotool`.
9
10A companion skill, `linux-desktop-use`, teaches the agent the observe → act → observe workflow.
11
12## Requirements
13
14Current POC targets KDE/Plasma Wayland first.
15
16Install:
17
18- `spectacle` for screenshots
19- `dotool` for input
20
21Verify dotool works:
22
23```bash
24echo type hello | dotool
25```
26
27If dotool cannot access `/dev/uinput`, follow dotool's setup instructions, usually adding your user to the `input` group and reloading udev/rebooting.
28
29## Quick start
30
31From this repo:
32
33```bash
34pi
35```
36
37Then reload project extensions/skills if Pi is already running:
38
39```text
40/reload
41```
42
43Ask something like:
44
45```text
46Look at my desktop and tell me what is open.
47```
48
49or:
50
51```text
52Browse the repo I have open in the browser.
53```
54
55The agent should use:
56
571. `desktop_observe` to capture the screen.
582. `desktop_input` to click, type, scroll, or press keys.
593. `desktop_observe` again to verify what changed.
60
61## Tool notes
62
63Coordinates for `desktop_input` are pixels in the latest `desktop_observe` screenshot. On multi-monitor setups, the screenshot spans the whole desktop.
64
65Example input actions:
66
67```json
68{ "action": "click", "x": 3335, "y": 548, "button": "left" }
69{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 }
70{ "action": "key", "key": "alt+left" }
71{ "action": "type", "text": "hello" }
72```
73
74## Using it elsewhere
75
76Copy these into another project:
77
78```text
79.pi/extensions/linux-use.ts
80.pi/skills/linux-desktop-use/SKILL.md
81```
82
83Run `/reload` in Pi.
84
85## Status
86
87This is intentionally tiny: Spectacle screenshot + dotool input. Future work can add portal ScreenCast/PipeWire, better window metadata, accessibility, OCR, crop/zoom, and wait-for-change helpers.