@@ -0,0 +1,150 @@
1+---
2+name: xfce-computer-use
3+description: Apply when you need to interact with the desktop as a user would in a xfce desktop environment. Use it to test and verify GUIs or TUIs inside the Kitty terminal.
4+---
5+
6+# Control the XFCE desktop on X11
7+
8+Use `wmctrl`, `xdotool`, `xwininfo`, `maim`, and `kitten` to inspect and control the XFCE desktop on this machine.
9+
10+## Prepare the environment
11+
12+Non-GUI shells start with `XDG_SESSION_TYPE=tty` and an empty `DISPLAY` variable. Export `DISPLAY=:0` before you run any X11 command, and store screenshots or sockets in `/tmp/computer-use-shots`.
13+
14+```bash
15+ export DISPLAY=:0
16+```
17+
18+The screen resolution is `1280x800`. The top `xfce4-panel` occupies `y=0` to `y=27`, and the bottom panel occupies `x=487` to `x=793` from `y=751` to `y=800`. Place windows between `y=27` and `y=750` so that the panels do not cover them.
19+
20+## Find, focus, move, and close windows
21+
22+Use `wmctrl` and `xdotool` to manage windows. Both tools, along with `xwininfo -id` and `maim -i`, accept hexadecimal or decimal window IDs.
23+
24+1. List all open windows with their window ID, workspace, PID, geometry, window class, and title:
25+
26+```bash
27+ DISPLAY=:0 wmctrl -lpGx
28+```
29+
30+2. Launch a GUI application in the background and poll `xdotool` until its window maps:
31+
32+```bash
33+ export DISPLAY=:0
34+ nohup kitty >/dev/null 2>&1 &
35+ APP_PID=$!
36+ for i in {1..30}; do
37+ WIN_ID=$(xdotool search --onlyvisible --pid "$APP_PID" 2>/dev/null | head -n1)
38+ [ -n "$WIN_ID" ] && break
39+ sleep 0.1
40+ done
41+```
42+
43+3. Focus a window and wait until it becomes active:
44+
45+```bash
46+ DISPLAY=:0 xdotool windowactivate --sync "$WIN_ID"
47+```
48+
49+4. Move and resize a window so that it fits inside the `1280x800` screen:
50+
51+```bash
52+ DISPLAY=:0 xdotool windowmove --sync "$WIN_ID" 80 60
53+ DISPLAY=:0 xdotool windowsize --sync "$WIN_ID" 800 500
54+```
55+
56+You can also move and resize a window in one command with `wmctrl`:
57+
58+```bash
59+ DISPLAY=:0 wmctrl -ir "$WIN_ID" -e 0,80,60,800,500
60+```
61+
62+5. Check the exact coordinates, size, and map state of a window:
63+
64+```bash
65+ DISPLAY=:0 xwininfo -id "$WIN_ID"
66+```
67+
68+6. Close a window cleanly through the window manager:
69+
70+```bash
71+ DISPLAY=:0 wmctrl -ic "$WIN_ID"
72+```
73+
74+## Control the mouse and keyboard
75+
76+Use `xdotool` to move the pointer, click, drag, and send keystrokes.
77+
78+1. Read the current pointer coordinates and the window under the pointer:
79+
80+```bash
81+ DISPLAY=:0 xdotool getmouselocation --shell
82+```
83+
84+2. Move the pointer and click a mouse button. Button `1` is left click, `2` is middle click, `3` is right click, `4` is scroll up, and `5` is scroll down:
85+
86+```bash
87+ DISPLAY=:0 xdotool mousemove --sync 300 250 click 1
88+```
89+
90+3. Drag the pointer from one point to another:
91+
92+```bash
93+ DISPLAY=:0 xdotool mousemove --sync 300 250 mousedown 1 mousemove --sync 500 250 mouseup 1
94+```
95+
96+4. Type text or press key combinations in the focused window:
97+
98+```bash
99+ DISPLAY=:0 xdotool type --delay 20 "echo 'hello world'"
100+ DISPLAY=:0 xdotool key Return
101+ DISPLAY=:0 xdotool key ctrl+shift+enter
102+```
103+
104+## Capture screenshots with the mouse cursor visible
105+
106+`maim` draws the XFixes mouse cursor in screenshots by default. Do not pass `-u` when you need to verify the pointer location. After you save a screenshot, inspect it with the `read` tool.
107+
108+1. Capture the entire desktop with the mouse cursor visible:
109+
110+```bash
111+ DISPLAY=:0 maim /tmp/opencode/desktop.png
112+```
113+
114+2. Capture a specific window by its window ID with the mouse cursor visible:
115+
116+```bash
117+ DISPLAY=:0 maim -i "$WIN_ID" /tmp/opencode/window.png
118+```
119+
120+3. Capture a rectangular region of the screen:
121+
122+```bash
123+ DISPLAY=:0 maim -g 800x500+80+60 /tmp/opencode/region.png
124+```
125+
126+## Automate the Kitty terminal with Kitten
127+
128+The user config at `~/.config/kitty/kitty.conf` sets two options that affect automation:
129+
130+- `initial_window_width 120c` and `initial_window_height 40c` open `kitty` at `1227x1067`, which extends past the bottom of the `1280x800` screen. Resize `kitty` after launch so the whole terminal is visible.
131+- `hide_window_decorations yes` removes the titlebar. Move `kitty` with `xdotool windowmove` or `wmctrl -ir` instead of dragging a titlebar.
132+
133+Remote control is off by default in `~/.config/kitty/kitty.conf`. To drive `kitty` with `kitten @`, enable remote control and bind a UNIX socket when you start `kitty`:
134+
135+```bash
136+ export DISPLAY=:0
137+ rm -f /tmp/opencode/kitty.sock
138+ nohup kitty -o allow_remote_control=yes --listen-on unix:/tmp/opencode/kitty.sock >/dev/null 2>&1 &
139+ KITTY_PID=$!
140+ for i in {1..30}; do
141+ WIN_ID=$(xdotool search --onlyvisible --pid "$KITTY_PID" 2>/dev/null | head -n1)
142+ [ -n "$WIN_ID" ] && [ -S /tmp/opencode/kitty.sock ] && break
143+ sleep 0.1
144+ done
145+ wmctrl -ir "$WIN_ID" -e 0,240,90,850,520
146+ kitten @ --to unix:/tmp/opencode/kitty.sock set-tab-title "Work"
147+ kitten @ --to unix:/tmp/opencode/kitty.sock set-colors background=#1a1b26 foreground=#c0caf5
148+ kitten @ --to unix:/tmp/opencode/kitty.sock launch --location=vsplit --cwd=current
149+ kitten @ --to unix:/tmp/opencode/kitty.sock send-text "ls -la\r"
150+```