xonecas.io
56101
Df

dotfiles / files

dotfiles / .claude / skills / xfce-computer-use / SKILL.md

4.9 KB
1---
2name: xfce-computer-use
3description: Apply when you need to interact with the desktop as a user would in a xfce desktop environment. Use it to test and verify GUIs or TUIs inside the Kitty terminal.
4---
5
6# Control the XFCE desktop on X11
7
8Use `wmctrl`, `xdotool`, `xwininfo`, `maim`, and `kitten` to inspect and control the XFCE desktop on this machine.
9
10## Prepare the environment
11
12Non-GUI shells start with `XDG_SESSION_TYPE=tty` and an empty `DISPLAY` variable. Export `DISPLAY=:0` before you run any X11 command, and store screenshots or sockets in `/tmp/computer-use-shots`.
13
14```bash
15	export DISPLAY=:0
16```
17
18The screen resolution is `1280x800`. The top `xfce4-panel` occupies `y=0` to `y=27`, and the bottom panel occupies `x=487` to `x=793` from `y=751` to `y=800`. Place windows between `y=27` and `y=750` so that the panels do not cover them.
19
20## Find, focus, move, and close windows
21
22Use `wmctrl` and `xdotool` to manage windows. Both tools, along with `xwininfo -id` and `maim -i`, accept hexadecimal or decimal window IDs.
23
241. List all open windows with their window ID, workspace, PID, geometry, window class, and title:
25
26```bash
27	DISPLAY=:0 wmctrl -lpGx
28```
29
302. Launch a GUI application in the background and poll `xdotool` until its window maps:
31
32```bash
33	export DISPLAY=:0
34	nohup kitty >/dev/null 2>&1 &
35	APP_PID=$!
36	for i in {1..30}; do
37		WIN_ID=$(xdotool search --onlyvisible --pid "$APP_PID" 2>/dev/null | head -n1)
38		[ -n "$WIN_ID" ] && break
39		sleep 0.1
40	done
41```
42
433. Focus a window and wait until it becomes active:
44
45```bash
46	DISPLAY=:0 xdotool windowactivate --sync "$WIN_ID"
47```
48
494. Move and resize a window so that it fits inside the `1280x800` screen:
50
51```bash
52	DISPLAY=:0 xdotool windowmove --sync "$WIN_ID" 80 60
53	DISPLAY=:0 xdotool windowsize --sync "$WIN_ID" 800 500
54```
55
56You can also move and resize a window in one command with `wmctrl`:
57
58```bash
59	DISPLAY=:0 wmctrl -ir "$WIN_ID" -e 0,80,60,800,500
60```
61
625. Check the exact coordinates, size, and map state of a window:
63
64```bash
65	DISPLAY=:0 xwininfo -id "$WIN_ID"
66```
67
686. Close a window cleanly through the window manager:
69
70```bash
71	DISPLAY=:0 wmctrl -ic "$WIN_ID"
72```
73
74## Control the mouse and keyboard
75
76Use `xdotool` to move the pointer, click, drag, and send keystrokes.
77
781. Read the current pointer coordinates and the window under the pointer:
79
80```bash
81	DISPLAY=:0 xdotool getmouselocation --shell
82```
83
842. Move the pointer and click a mouse button. Button `1` is left click, `2` is middle click, `3` is right click, `4` is scroll up, and `5` is scroll down:
85
86```bash
87	DISPLAY=:0 xdotool mousemove --sync 300 250 click 1
88```
89
903. Drag the pointer from one point to another:
91
92```bash
93	DISPLAY=:0 xdotool mousemove --sync 300 250 mousedown 1 mousemove --sync 500 250 mouseup 1
94```
95
964. Type text or press key combinations in the focused window:
97
98```bash
99	DISPLAY=:0 xdotool type --delay 20 "echo 'hello world'"
100	DISPLAY=:0 xdotool key Return
101	DISPLAY=:0 xdotool key ctrl+shift+enter
102```
103
104## Capture screenshots with the mouse cursor visible
105
106`maim` draws the XFixes mouse cursor in screenshots by default. Do not pass `-u` when you need to verify the pointer location. After you save a screenshot, inspect it with the `read` tool.
107
1081. Capture the entire desktop with the mouse cursor visible:
109
110```bash
111	DISPLAY=:0 maim /tmp/opencode/desktop.png
112```
113
1142. Capture a specific window by its window ID with the mouse cursor visible:
115
116```bash
117	DISPLAY=:0 maim -i "$WIN_ID" /tmp/opencode/window.png
118```
119
1203. Capture a rectangular region of the screen:
121
122```bash
123	DISPLAY=:0 maim -g 800x500+80+60 /tmp/opencode/region.png
124```
125
126## Automate the Kitty terminal with Kitten
127
128The user config at `~/.config/kitty/kitty.conf` sets two options that affect automation:
129
130- `initial_window_width 120c` and `initial_window_height 40c` open `kitty` at `1227x1067`, which extends past the bottom of the `1280x800` screen. Resize `kitty` after launch so the whole terminal is visible.
131- `hide_window_decorations yes` removes the titlebar. Move `kitty` with `xdotool windowmove` or `wmctrl -ir` instead of dragging a titlebar.
132
133Remote control is off by default in `~/.config/kitty/kitty.conf`. To drive `kitty` with `kitten @`, enable remote control and bind a UNIX socket when you start `kitty`:
134
135```bash
136	export DISPLAY=:0
137	rm -f /tmp/opencode/kitty.sock
138	nohup kitty -o allow_remote_control=yes --listen-on unix:/tmp/opencode/kitty.sock >/dev/null 2>&1 &
139	KITTY_PID=$!
140	for i in {1..30}; do
141		WIN_ID=$(xdotool search --onlyvisible --pid "$KITTY_PID" 2>/dev/null | head -n1)
142		[ -n "$WIN_ID" ] && [ -S /tmp/opencode/kitty.sock ] && break
143		sleep 0.1
144	done
145	wmctrl -ir "$WIN_ID" -e 0,240,90,850,520
146	kitten @ --to unix:/tmp/opencode/kitty.sock set-tab-title "Work"
147	kitten @ --to unix:/tmp/opencode/kitty.sock set-colors background=#1a1b26 foreground=#c0caf5
148	kitten @ --to unix:/tmp/opencode/kitty.sock launch --location=vsplit --cwd=current
149	kitten @ --to unix:/tmp/opencode/kitty.sock send-text "ls -la\r"
150```