README.md
Pi Linux Use
Minimal proof-of-concept Pi extension for Linux desktop computer-use.
It gives Pi two tools:
desktop_observe— captures the visible desktop with KDE Spectacle and returns a screenshot to the model.desktop_input— sends mouse/keyboard input throughdotool.
A companion skill, linux-desktop-use, teaches the agent the observe → act → observe workflow.
Requirements
Current POC targets KDE/Plasma Wayland first.
Install:
spectaclefor screenshotsdotoolfor input
Verify dotool works:
echo type hello | dotool
If dotool cannot access /dev/uinput, follow dotool's setup instructions, usually adding your user to the input group and reloading udev/rebooting.
Quick start
From this repo:
pi
Then reload project extensions/skills if Pi is already running:
/reload
Ask something like:
Look at my desktop and tell me what is open.
or:
Browse the repo I have open in the browser.
The agent should use:
desktop_observeto capture the screen.desktop_inputto click, type, scroll, or press keys.desktop_observeagain to verify what changed.
Tool notes
Coordinates for desktop_input are pixels in the latest desktop_observe screenshot. On multi-monitor setups, the screenshot spans the whole desktop.
Example input actions:
{ "action": "click", "x": 3335, "y": 548, "button": "left" }
{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 }
{ "action": "key", "key": "alt+left" }
{ "action": "type", "text": "hello" }
Using it elsewhere
Copy these into another project:
.pi/extensions/linux-use.ts
.pi/skills/linux-desktop-use/SKILL.md
Run /reload in Pi.
Status
This is intentionally tiny: Spectacle screenshot + dotool input. Future work can add portal ScreenCast/PipeWire, better window metadata, accessibility, OCR, crop/zoom, and wait-for-change helpers.