xonecas.io

pi-linux-use

Looking into allowing the Pi coding agent to use a linux desktop.

post-AI small skills

4 commits · 2026-05-13 to 2026-05-13 · 6 files

xonecasfeat: working poc8e7de552026-05-13
▸ .pi/ 2026-05-13
· .gitignore 2026-05-13
· LICENSE 2026-05-13
· pi-linux-use.md 2026-05-13
· README.md 2026-05-13

README.md

Pi Linux Use

Minimal proof-of-concept Pi extension for Linux desktop computer-use.

It gives Pi two tools:

  • desktop_observe — captures the visible desktop with KDE Spectacle and returns a screenshot to the model.
  • desktop_input — sends mouse/keyboard input through dotool.

A companion skill, linux-desktop-use, teaches the agent the observe → act → observe workflow.

Requirements

Current POC targets KDE/Plasma Wayland first.

Install:

  • spectacle for screenshots
  • dotool for input

Verify dotool works:

echo type hello | dotool

If dotool cannot access /dev/uinput, follow dotool's setup instructions, usually adding your user to the input group and reloading udev/rebooting.

Quick start

From this repo:

pi

Then reload project extensions/skills if Pi is already running:

/reload

Ask something like:

Look at my desktop and tell me what is open.

or:

Browse the repo I have open in the browser.

The agent should use:

  1. desktop_observe to capture the screen.
  2. desktop_input to click, type, scroll, or press keys.
  3. desktop_observe again to verify what changed.

Tool notes

Coordinates for desktop_input are pixels in the latest desktop_observe screenshot. On multi-monitor setups, the screenshot spans the whole desktop.

Example input actions:

{ "action": "click", "x": 3335, "y": 548, "button": "left" }
{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 }
{ "action": "key", "key": "alt+left" }
{ "action": "type", "text": "hello" }

Using it elsewhere

Copy these into another project:

.pi/extensions/linux-use.ts
.pi/skills/linux-desktop-use/SKILL.md

Run /reload in Pi.

Status

This is intentionally tiny: Spectacle screenshot + dotool input. Future work can add portal ScreenCast/PipeWire, better window metadata, accessibility, OCR, crop/zoom, and wait-for-change helpers.