xonecas.io
734
Pl

pi-linux-use / files

pi-linux-use / .pi / skills / linux-desktop-use / SKILL.md

3.5 KB
1---
2name: linux-desktop-use
3description: Use with the pi-linux-use extension to inspect and operate a visible Linux desktop using desktop_observe screenshots and desktop_input dotool actions. Load when the user asks to click, type, scroll, browse a visible website/repo, test GUI behavior, or otherwise interact with the desktop environment.
4compatibility: Requires the pi-linux-use extension tools desktop_observe and desktop_input, plus KDE Spectacle for screenshots and dotool for input.
5---
6
7# Linux Desktop Use
8
9Use this skill when controlling the user's visible Linux desktop through the `desktop_observe` and `desktop_input` tools.
10
11## Core loop
12
131. Call `desktop_observe` before any coordinate-based action.
142. Inspect the screenshot and choose pixel coordinates in the **full screenshot coordinate space**, not window-relative coordinates.
153. Call `desktop_input` for one small action: click, key, type, scroll, move, or script.
164. Call `desktop_observe` again to verify the result before continuing.
17
18Prefer short observe/action/observe loops over large batches. The screen can move, scroll, or refocus after every action, making old coordinates stale.
19
20## Coordinate rules
21
22- `desktop_input` x/y coordinates are pixels from the latest `desktop_observe` screenshot.
23- Coordinates are global desktop coordinates across all monitors.
24- Re-observe after scrolling, navigation, window focus changes, modal dialogs, or any action that changes layout.
25- If a target is partly covered by another window, do not guess. Focus/move to a visible target or ask the user.
26
27## Safe interaction practices
28
29- Click inside the target window before typing, pressing keys, or scrolling.
30- For scrolling, place the cursor over the scrollable region first. Negative wheel amounts scroll down; positive amounts scroll up.
31- Use small reversible actions first. Avoid destructive clicks unless the user explicitly asked for them and the target is clearly visible.
32- Be careful not to type into Pi's own prompt/editor by accident. If focus is uncertain, observe and click the intended application first.
33- If an action appears to do nothing, assume one of: stale screenshot, wrong focus, cursor over wrong pane, page not loaded, or target obscured. Re-observe and correct.
34
35## Browser/repo browsing pattern
36
37When the user asks to browse a repo or website already open in the desktop browser, use the visible browser window unless they explicitly ask for separate browser automation.
38
39Useful actions:
40
41```json
42{ "action": "click", "x": 3335, "y": 548, "button": "left" }
43{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 }
44{ "action": "key", "key": "alt+left" }
45{ "action": "key", "key": "ctrl+l" }
46{ "action": "type", "text": "https://example.com" }
47{ "action": "key", "key": "enter" }
48```
49
50For a visible repository page, first observe the high-level state: URL, repo name, active tab/section, visible files, branches, commits, and any selected file content. Then navigate with clicks or keyboard shortcuts and re-observe.
51
52## Key names
53
54Use dotool-style key/chord names in `desktop_input`:
55
56- `enter`, `tab`, `esc`, `backspace`, `delete`
57- `ctrl+l`, `ctrl+r`, `ctrl+f`, `ctrl+shift+t`
58- `alt+left`, `alt+right`, `alt+f4`
59- `super`
60
61## Raw dotool scripts
62
63Use `action: "script"` only for simple, deliberate batches where intermediate visual feedback is not needed. Example:
64
65```json
66{
67  "action": "script",
68  "script": "key ctrl+l\ntype https://codeberg.org/xonecas/pi-linux-use\nkey enter"
69}
70```
71
72Do not batch risky clicks or text entry into unknown fields. Observe first, then act.