xonecas.io
8447
Co

conveyor / commit 217a0f9

Spike 7: audio, video and a text editor, one layout per project

Three first-try builds: a music player (ffplay), a cinema (ffmpeg into
rune::Pixels at 30 fps) and a writing room (highlighting editor, live
preview). Placement no longer mounts a kind a widget renders by name,
or one with no size (a layout). Toast titles truncate.

Co-Authored-By: Claude Opus 5.5

sacenox2026-11-09 217a0f94d4a05eaeaf1ce0111b5255f4a221d3eb parent 65016c7

6 files changed

DESIGN.md+1−0
@@ -330,6 +330,7 @@ Each spike is a throwaway script that answers one question with numbers. They ra
330330 | 4 | Spike 2 for opencode | done | Same behaviour with DeepSeek v4.1 flash, 45 s |
331331 | 5 | What API does a self-modifying client need? | done | 4/4 builds first try, 3 to 4 min each. Gaps ranked in [The widget builder](#the-widget-builder) |
332332 | 6 | Where is the builder's ceiling? | done | 3/3 hard builds first try (blur, vector diagram, paste → ticket). Limits are in the harness. See `spikes/6-ceiling/NOTES.md` |
333+| 7 | Audio, video, text editing, a layout per project | done | 3/3 first try, 7 to 11 min each. ffmpeg into `rune::Pixels` plays 30 fps video. The limits are app restarts and instances that only exist once rendered. See `spikes/7-media/NOTES.md` |
333334
334335 ### What the spikes taught us
335336
crates/conveyor-app/src/status.rs+1−1
@@ -26,7 +26,7 @@ fn toast(b: &Value) -> El {
2626 col().key(format!("toast-{id}")).w(380.0).radius(10.0).fill(t.card).padding(12.0).gap(4.0).role("button").label("Dismiss")
2727 .border(1.0, color).hover(t.hover)
2828 .on_press_tx(move |s: &mut State, _| { s.seen.insert(id); save_ids(s.live.base(), "seen", &s.seen); })
29- .child(text(format!("#{id} {state} · {}", b["title"].as_str().unwrap_or(""))).font(tok::type_::LABEL).color(color))
29+ .child(text(format!("#{id} {state} · {}", b["title"].as_str().unwrap_or(""))).font(tok::type_::LABEL).color(color).w_full().truncate())
3030 .child(text(said).color(t.text).w_full().wrap())
3131 .child(text(if state == "done" { "Click to dismiss" } else { "Ctrl+click the widget to follow up · click to dismiss" }).font(tok::type_::LABEL).color(t.text_secondary))
3232 }
crates/conveyor/src/library.rs+9−0
@@ -119,11 +119,20 @@ pub fn install(p: &Project, id: u32, branch: &str) -> Result<(), String> {
119119 return Err(format!("the app doesn't build with this branch, so it was not merged:\n{e}"));
120120 }
121121 let mut b = build(id).unwrap_or_default();
122+ let created = created.into_iter().filter(|k| !rendered(&p.path, k)).collect();
122123 (b.created, b.changed) = (created, kinds(changed));
123124 save_build(id, &b);
124125 Ok(())
125126 }
126127
128+/// Some widget renders this kind itself (`child("kind"`), so the app must not place it too,
129+/// even while that parent keeps it off screen.
130+fn rendered(library: &std::path::Path, kind: &str) -> bool {
131+ let call = format!("child(\"{kind}\"");
132+ std::fs::read_dir(library.join("widgets")).into_iter().flatten().flatten()
133+ .any(|f| std::fs::read_to_string(f.path()).is_ok_and(|src| src.contains(&call)))
134+}
135+
127136 /// The app restarts itself when its binary changes.
128137 fn rebuild_app() -> Result<(), String> {
129138 let _one = BUILD.lock().unwrap();
spikes/7-media/NOTES.md+82−0
@@ -0,0 +1,82 @@
1+# Spike 7: audio, video, text editing, a layout per project
2+
3+Goal: probe the three media the ceiling table left open, with each project laid out its own
4+way. Three test projects in `/tmp`, made with ffmpeg and 0 slots: `music` (ogg, flac, mp3),
5+`film` (two mp4 with sound, one silent webm), `notes` (Markdown and Rust).
6+
7+## Builds
8+
9+Each was one Ctrl+click on the board, with the project's tab selected, and one prompt. All
10+three were done first try.
11+
12+| # | Project | Time | Lines | What the agent built |
13+|---|---|---|---|---|
14+| 5 | music | 7.3 min | 432 | `layout_<slug>` per project in `canvas`. `music_player`: playlist, waveform from ffmpeg (2 kHz mono, cached), seek by click or drag, prev/next, volume, auto-advance. Playback is a detached `ffplay` with a heartbeat watchdog |
15+| 6 | film | 8.7 min | 348 | `cinema`: ffmpeg decodes to raw RGBA on a pipe, into `rune::Pixels` at 30 fps (`-re` paced). The same ffmpeg sends the sound to PulseAudio. Live 12 fps thumbnails, controls that fade, status pane over the video |
16+| 7 | notes | 11.3 min | 644 | `file_tree`, `note_editor` (line numbers, Markdown and Rust colours in the field itself, Tab, Ctrl+S, unsaved dot, buffers kept across restarts), `note_preview` (live Markdown), board in a drawer |
17+
18+The library is now 2097 lines over 17 widgets.
19+
20+```
21+ canvas ── selected project? ── layout_<slug> exists? ──yes──▶ that layout
22+ └──no───▶ board + status pane
23+ music : music_player over board strip
24+ film : cinema under status pane
25+ notes : file_tree | note_editor | note_preview, board drawer
26+ others : board under status pane
27+```
28+
29+## Checked on the desktop
30+
31+| Check | Result |
32+|---|---|
33+| Music reaches the speakers | sink monitor RMS 5304 on drift. Seek to 75% ran `ffplay -ss 29.81` |
34+| Auto-advance, music on other tabs | drift → pulse while the film tab was up |
35+| Video with sound | 30.0 fps (frame counter 566 at 18.867 s). Sink RMS 2886 from ffmpeg's own Pulse stream |
36+| Cost | the app at 3.5% CPU while playing. ffmpeg at 189% decoding and scaling 1080p on the CPU |
37+| Controls fade, status pane blurs the video | yes |
38+| Off-screen thumbnails | their decoders stop when another tab is up |
39+| Editor | typing, live preview, unsaved dot, Ctrl+S wrote the file. Ctrl+S didn't open the inspector |
40+| Media across an app restart | music kept its ffplay (re-adopted by pid), video restarted at its spot after ~2 s. See the restart limit below |
41+
42+## What the agent couldn't do
43+
44+- **Hear.** The preview has no sound, so it reported "I couldn't hear the sound or check that it
45+ stays in sync with the picture."
46+- **See the project's files.** `/tmp/notes` and the media folders don't exist in the VM. It
47+ made its own sample files with ffmpeg in the guest.
48+- **Know the host's programs.** It assumed `ffplay` and `ffmpeg` are on the host. They are,
49+ but nothing told it so.
50+- **Publish from `update`.** The editor shares its buffer and unsaved set through the
51+ `on_layout` marker hack again, as in spike 6.
52+
53+## Core fixes in this spike
54+
55+- **Placement**, which misplaced widgets twice more. xonecas's `ticket_editor` and this
56+ spike's `layout_film` each landed as a stray card on the canvas, the second one inside the
57+ *music* project, because their parents only draw them sometimes. Two rules now:
58+ - conveyord leaves a kind out of `created` when a widget names it in `child("kind"`;
59+ - `Live::mount` never places a kind with a zero size (a layout).
60+
61+ This is still guessing. Explicit placement stays on the list.
62+- Toast titles truncate instead of running off the toast.
63+
64+## Where the ceiling is now
65+
66+The media ceiling is the host's programs, not Rune or the API: anything ffmpeg can decode, a
67+widget can show and play. The limits:
68+
69+| Limit | Seen as | API to aim for |
70+|---|---|---|
71+| Instances exist only once rendered | after a restart, music playing on a hidden project dies within 10 s: nothing runs its `update` | restore every instance at start, or a `background` flag on `Widget` |
72+| Every library change restarts the app | video stops for ~2 s; players re-adopt detached processes by pid; each media widget reinvents a watchdog | a host-owned media service that outlives restarts (`cx.play(path)` returns a handle), or hot reload |
73+| Rune's `Player` is macOS-only | widgets shell out to ffplay/ffmpeg; seek and volume restart the process | a Linux `Player` backend upstream (GStreamer or ffmpeg), and the API above built on it |
74+| The preview is silent | the agent can't check sound or sync | `preview --audio out.wav` and `--frames DIR`, so the agent can measure what it can't hear |
75+| The agent can't see the project's files | it tested on samples it made itself | a build ticket carries a listing (and small samples) of the projects the widget reads |
76+| The host's programs are unknown | the agent assumed ffplay | the ticket lists the host's relevant programs, or the media service above makes it moot |
77+| Actions only from handlers | the marker hack, a second time | `Cx` carries actions (`cx.share`, `cx.new_ticket`), as spike 6 found |
78+| Ctrl is the builder's key | Ctrl+S worked, but by design holding Ctrl (word moves) shows the inspector, and it takes a Ctrl+click in an editor | a chord or a long-hold to inspect, so editors keep Ctrl |
79+| No global now-playing | music keeps playing on other tabs, and only the music tab can stop it | product question: a media slot in the bottom bar |
80+
81+Still open as probes: a VM's live desktop in a widget, large files in the editor, and widgets
82+across several windows.
templates/library/CLAUDE.md+1−1
@@ -77,7 +77,7 @@ When the API covers most of it, build that part and name the rest with `conveyor
7777 }
7878 ```
7979
80- A text field binds with `.on_edit(edit(cx.id, |w: &mut MyWidget| &mut w.text))`. `size` is what the canvas gives it when it's placed there.
80+ A text field binds with `.on_edit(edit(cx.id, |w: &mut MyWidget| &mut w.text))`. `size` is what the canvas gives it when it's placed there. A widget that only fills what its parent gives it (a layout) has size `(0.0, 0.0)`, and is never placed.
8181 3. Use Rune's dark theme (`tok::theme(true)`) and fill the box the parent gives you.
8282
8383 ## Check it
templates/library/api.rs+4−1
@@ -284,8 +284,11 @@ impl Live {
284284 }
285285
286286 /// Offers a new widget to each instance in turn, at its own point: the one clicked first,
287- /// then its parents. The id of the one that kept it.
287+ /// then its parents. The id of the one that kept it. A kind with no size of its own (a
288+ /// layout, which only fills its parent) is never placed.
288289 pub fn mount(&mut self, targets: &[(String, (f32, f32))], kind: &str) -> Option<String> {
290+ let (w, h) = super::ALL.iter().find(|(n, _)| *n == kind)?.1.size;
291+ if w == 0.0 || h == 0.0 { return None; }
289292 targets.iter().find(|(id, at)| self.with(id, |w, cx| w.mount(kind, *at, cx)) == Some(true)).map(|(id, _)| id.clone())
290293 }