xonecas.io
56101
Df

dotfiles / commit 3715e6e

skill: computer use

sacenox2026-11-07 3715e6e4487d05b1f4a872d0f826f900329c5d80 parent f0366d0

2 files changed

.claude/skills/kde-computer-use/SKILL.md+56−0
@@ -0,0 +1,56 @@
1+---
2+name: kde-computer-use
3+description: Apply when you need to interact with a KDE Plasma desktop on Wayland as a user would. Use it to test and verify GUIs there. It finds and focuses windows through KWin, presses keys through uinput, and takes screenshots with Spectacle.
4+---
5+
6+# Control a KDE Plasma desktop on Wayland
7+
8+`xdotool`, `wmctrl` and `maim` cannot reach native Wayland windows. Use KWin's D-Bus, a uinput keyboard and Spectacle instead. The user shares this desktop, so warn them first, and touch only the windows you opened.
9+
10+## Find a window
11+
12+Search by title. Each row starts with the window's match id, `0_{uuid}`:
13+
14+```bash
15+busctl --user call org.kde.KWin /WindowsRunner org.kde.krunner1 Match s rats
16+```
17+
18+Read a window's size, position, `pid` and `desktopFile`, which is its Wayland app id:
19+
20+```bash
21+qdbus6 org.kde.KWin /KWin org.kde.KWin.getWindowInfo '{uuid}'
22+```
23+
24+Two windows can share a title. Tell yours from the user's by `pid`.
25+
26+## Press keys
27+
28+`scripts/key.py` focuses the window, then presses keys and clicks. Pass the uuid with its braces. Coordinates are global, so add the window's `x` and `y`:
29+
30+```bash
31+python3 ~/.claude/skills/kde-computer-use/scripts/key.py '{uuid}' click:1180,422 alt+o radio enter
32+```
33+
34+It stops if the window loses the focus before a step. It needs write access to `/dev/uinput` and `python-evdev`. It moves the user's real pointer, and each run takes the focus from them.
35+
36+Count volume or list steps from a known state, and check the result in a screenshot. A miscount looks like a bug.
37+
38+## Take a screenshot
39+
40+The user keeps working, so the active window may be theirs by the time you capture. Focus yours in the same command, with no wait between. Read the image, and delete it at once if it shows anything but your window:
41+
42+```bash
43+python3 ~/.claude/skills/kde-computer-use/scripts/key.py '{uuid}' && spectacle --background --nonotify --activewindow --output /path/to/shot.png
44+```
45+
46+Spectacle takes a moment after it returns: wait for the file, then read it.
47+
48+## Hear what a window plays
49+
50+You cannot hear it, so measure it. Record the sink's monitor, which is what actually reaches the sink, before the sink's own volume. It holds every app's sound, so first check that only yours is playing:
51+
52+```bash
53+timeout 4 pw-record -P '{ stream.capture.sink=true }' --target <sink-node-name> --rate 48000 --channels 2 --format s16 /path/to/out.wav
54+```
55+
56+Prove the monitor reads with a quiet control tone from `pw-play`. Recording a stream's node directly misleads: it read sound from a stream the speakers never got. A stream in `running` state and moving bars do not prove sound.
.claude/skills/kde-computer-use/scripts/key.py+105−0
@@ -0,0 +1,105 @@
1+#!/usr/bin/env python3
2+"""Focus one KWin window, then press keys and click in it through a short-lived uinput device.
3+
4+Usage: key.py <window-uuid> <step> [...]
5+A step is a combo or key name (alt+o, ctrl+v, enter, space, down), click:X,Y or move:X,Y in
6+global coordinates, or text to type, in lowercase letters, digits and spaces.
7+It stops if the window loses the focus, so nothing lands in another window.
8+"""
9+import subprocess
10+import sys
11+import tempfile
12+import time
13+
14+from evdev import UInput, ecodes as e
15+
16+NAMES = {"alt": "LEFTALT", "ctrl": "LEFTCTRL", "shift": "LEFTSHIFT", "super": "LEFTMETA", "esc": "ESC"}
17+KEYS = {"enter", "space", "tab", "esc", "up", "down", "left", "right", "backspace"}
18+SCRIPT = 'print("%s " + workspace.cursorPos.x + " " + workspace.cursorPos.y + " " + (workspace.activeWindow ? workspace.activeWindow.internalId : ""));'
19+
20+
21+def qdbus(path, method, *args):
22+ return subprocess.run(["qdbus6", "org.kde.KWin", path, method, *args], check=True, capture_output=True, text=True).stdout.strip()
23+
24+
25+def state():
26+ """The pointer's position and the active window, from a KWin script that prints them."""
27+ token = f"KDECU{time.time_ns()}"
28+ since = int(time.time()) - 1
29+ with tempfile.NamedTemporaryFile("w", suffix=".js") as script:
30+ script.write(SCRIPT % token)
31+ script.flush()
32+ number = qdbus("/Scripting", "org.kde.kwin.Scripting.loadScript", script.name, "kdecu")
33+ qdbus(f"/Scripting/Script{number}", "org.kde.kwin.Script.run")
34+ qdbus("/Scripting", "org.kde.kwin.Scripting.unloadScript", "kdecu")
35+ for _ in range(40):
36+ log = subprocess.run(["journalctl", "--user", "--since", f"@{since}", "-o", "cat", "--grep", token], capture_output=True, text=True).stdout
37+ if token in log:
38+ _, x, y, *window = log.strip().splitlines()[-1].split()
39+ return int(x), int(y), window[0] if window else ""
40+ time.sleep(0.05)
41+ raise SystemExit("key.py: KWin did not answer")
42+
43+
44+def code(name):
45+ return getattr(e, "KEY_" + NAMES.get(name, name).upper())
46+
47+
48+def press(device, codes):
49+ for key in codes:
50+ device.write(e.EV_KEY, key, 1)
51+ device.syn()
52+ time.sleep(0.03)
53+ for key in reversed(codes):
54+ device.write(e.EV_KEY, key, 0)
55+ device.syn()
56+ time.sleep(0.03)
57+
58+
59+def travel(device, axis, distance):
60+ for _ in range(abs(distance) // 2):
61+ device.write(e.EV_REL, axis, 2 if distance > 0 else -2)
62+ device.syn()
63+ time.sleep(0.008) # At this pace one unit is one pixel. Faster, the acceleration overshoots.
64+
65+
66+def move(device, x, y):
67+ """The pointer is relative, so it moves, looks where it landed and corrects."""
68+ for _ in range(6):
69+ here_x, here_y, _window = state()
70+ if abs(here_x - x) <= 3 and abs(here_y - y) <= 3:
71+ return
72+ travel(device, e.REL_X, x - here_x)
73+ travel(device, e.REL_Y, y - here_y)
74+ time.sleep(0.05)
75+ raise SystemExit(f"key.py: the pointer stopped at {state()[:2]}, not {x},{y}")
76+
77+
78+def main():
79+ window, steps = sys.argv[1], sys.argv[2:]
80+ capabilities = [v for k, v in vars(e).items() if k.startswith("KEY_") and isinstance(v, int) and v < 249]
81+ events = {e.EV_KEY: capabilities + [e.BTN_LEFT], e.EV_REL: [e.REL_X, e.REL_Y]}
82+ with UInput(events, name="kde-computer-use") as device:
83+ time.sleep(0.6) # The compositor has to see the new device first.
84+ subprocess.run(
85+ ["busctl", "--user", "call", "org.kde.KWin", "/WindowsRunner", "org.kde.krunner1", "Run", "ss", f"0_{window}", ""],
86+ check=True,
87+ )
88+ time.sleep(0.2)
89+ for step in steps:
90+ if state()[2] != window:
91+ raise SystemExit(f"key.py: {window} lost the focus, stopped before {step}")
92+ if step.startswith(("move:", "click:")):
93+ action, place = step.split(":")
94+ move(device, *map(int, place.split(",")))
95+ if action == "click":
96+ press(device, [e.BTN_LEFT])
97+ elif "+" in step or step in KEYS:
98+ press(device, [code(part) for part in step.split("+")])
99+ else:
100+ for char in step:
101+ press(device, [e.KEY_SPACE if char == " " else code(char)])
102+ time.sleep(0.05)
103+
104+
105+main()