734
Pl
pi-linux-use / files
pi-linux-use / .pi / skills / linux-desktop-use / SKILL.md
3.5 KB
1--- 2name: linux-desktop-use 3description: Use with the pi-linux-use extension to inspect and operate a visible Linux desktop using desktop_observe screenshots and desktop_input dotool actions. Load when the user asks to click, type, scroll, browse a visible website/repo, test GUI behavior, or otherwise interact with the desktop environment. 4compatibility: Requires the pi-linux-use extension tools desktop_observe and desktop_input, plus KDE Spectacle for screenshots and dotool for input. 5--- 6 7# Linux Desktop Use 8 9Use this skill when controlling the user's visible Linux desktop through the `desktop_observe` and `desktop_input` tools. 10 11## Core loop 12 131. Call `desktop_observe` before any coordinate-based action. 142. Inspect the screenshot and choose pixel coordinates in the **full screenshot coordinate space**, not window-relative coordinates. 153. Call `desktop_input` for one small action: click, key, type, scroll, move, or script. 164. Call `desktop_observe` again to verify the result before continuing. 17 18Prefer short observe/action/observe loops over large batches. The screen can move, scroll, or refocus after every action, making old coordinates stale. 19 20## Coordinate rules 21 22- `desktop_input` x/y coordinates are pixels from the latest `desktop_observe` screenshot. 23- Coordinates are global desktop coordinates across all monitors. 24- Re-observe after scrolling, navigation, window focus changes, modal dialogs, or any action that changes layout. 25- If a target is partly covered by another window, do not guess. Focus/move to a visible target or ask the user. 26 27## Safe interaction practices 28 29- Click inside the target window before typing, pressing keys, or scrolling. 30- For scrolling, place the cursor over the scrollable region first. Negative wheel amounts scroll down; positive amounts scroll up. 31- Use small reversible actions first. Avoid destructive clicks unless the user explicitly asked for them and the target is clearly visible. 32- Be careful not to type into Pi's own prompt/editor by accident. If focus is uncertain, observe and click the intended application first. 33- If an action appears to do nothing, assume one of: stale screenshot, wrong focus, cursor over wrong pane, page not loaded, or target obscured. Re-observe and correct. 34 35## Browser/repo browsing pattern 36 37When the user asks to browse a repo or website already open in the desktop browser, use the visible browser window unless they explicitly ask for separate browser automation. 38 39Useful actions: 40 41```json 42{ "action": "click", "x": 3335, "y": 548, "button": "left" } 43{ "action": "scroll", "x": 4550, "y": 940, "amount": -5 } 44{ "action": "key", "key": "alt+left" } 45{ "action": "key", "key": "ctrl+l" } 46{ "action": "type", "text": "https://example.com" } 47{ "action": "key", "key": "enter" } 48``` 49 50For a visible repository page, first observe the high-level state: URL, repo name, active tab/section, visible files, branches, commits, and any selected file content. Then navigate with clicks or keyboard shortcuts and re-observe. 51 52## Key names 53 54Use dotool-style key/chord names in `desktop_input`: 55 56- `enter`, `tab`, `esc`, `backspace`, `delete` 57- `ctrl+l`, `ctrl+r`, `ctrl+f`, `ctrl+shift+t` 58- `alt+left`, `alt+right`, `alt+f4` 59- `super` 60 61## Raw dotool scripts 62 63Use `action: "script"` only for simple, deliberate batches where intermediate visual feedback is not needed. Example: 64 65```json 66{ 67 "action": "script", 68 "script": "key ctrl+l\ntype https://codeberg.org/xonecas/pi-linux-use\nkey enter" 69} 70``` 71 72Do not batch risky clicks or text entry into unknown fields. Observe first, then act.