xonecas.io
7517
Cd

clanker-and-the-dev / files

clanker-and-the-dev / comic / sdxl-generation-plan.md

7.4 KB
1# SDXL Comic Generation Refactor Plan
2
3Goal: replace the current deterministic SVG comic renderer with a non-deterministic SDXL image pipeline driven by the `add-new-comic` agent skill.
4
5## Decisions
6
7- The agent skill is the entrypoint for the full flow.
8- The deterministic SVG renderer is not kept as a fallback.
9- Existing generated posts should be regenerated with the new image-based pipeline.
10- The Python generation code belongs with the agent skill and must respect the Agent Skills spec from agentskills.io.
11- CUDA is expected to be available; target hardware is an RTX 3080 Ti 12 GB.
12- Default model backend is SDXL.
13- Default model is `stabilityai/stable-diffusion-xl-base-1.0`.
14- Use generated image panels plus deterministic text/comic compositing, not model-rendered text.
15
16## Recommended pipeline
17
18```txt
19user joke / premise
20  -> add-new-comic skill
21  -> turn premise into panel script + visual directions
22  -> generate rough panel composition guides
23  -> generate panel art with SDXL conditioned by references
24  -> composite speech bubbles, captions, terminal text, borders with Python/Pillow
25  -> write PNG panels/assets
26  -> write Jekyll markdown post
27```
28
29## Why not plain text-to-image only
30
31Plain SDXL text-to-image does not use files in `art-reference/` just because they exist in the repo. If the pipeline only sends a prompt, the model never sees the reference sheets.
32
33To make the model actually follow the Clanker/dev visual identity, the new path should use image conditioning.
34
35## Image conditioning strategy
36
37### First implementation: SDXL + IP-Adapter
38
39Use IP-Adapter to condition SDXL on project reference images.
40
41Chosen default checkpoint:
42
43```txt
44repo: h94/IP-Adapter
45subfolder: sdxl_models
46weight: ip-adapter_sdxl.bin
47```
48
49Reference images now default to:
50
51- `art-reference/training-reference.png`
52- `art-reference/ip-character-reference.png`
53
54The generator extracts large per-panel crops from the clean training sheet, extracts character-only crops from the two-character IP reference, and averages IP-Adapter image embeddings. This gives the model character/style identity without asking it to copy a full grid/reference sheet or character-card layout.
55
56Expected benefits:
57
58- Better style matching than prompt-only SDXL.
59- No model training required.
60- Works with existing reference sheets.
61
62Known limitations:
63
64- Character consistency may still drift.
65- Multi-character panels can confuse identity.
66- Reference strength will need tuning.
67
68### Composition control: ControlNet
69
70ControlNet is now enabled by default because SDXL + IP-Adapter alone did not match the references consistently enough.
71
72The generator creates black/white composition guides from `art-reference/` character sprites plus generated props/effects for each panel and uses ControlNet to preserve structure:
73
74- Dev position
75- Clanker position
76- terminal/monitor position
77- important props
78- reserved speech bubble areas
79- panel framing
80
81ControlNet helps composition, but does not replace IP-Adapter for style/identity. The generated guides are written as `assets/comics/<slug>/control-panel-XX.png` for review/debugging, and the compositor clips raw line art to the guide before merging it back into the final panel to repair drift and remove scanline/blob artifacts.
82
83### Future improvement: SDXL LoRA
84
85If IP-Adapter is not consistent enough, train a small SDXL LoRA for the comic style and characters.
86
87This should be treated as a later phase because it requires a dataset and training workflow.
88
89## Text and lettering
90
91Do not rely on Stable Diffusion for readable text.
92
93The Python compositor should add:
94
95- speech bubbles
96- captions
97- terminal text
98- panel borders
99- title/credit if needed
100
101This keeps the comic readable while allowing the art to be non-deterministic.
102
103## Proposed file layout
104
105```txt
106.agents/skills/add-new-comic/SKILL.md
107.agents/skills/add-new-comic/pyproject.toml
108.agents/skills/add-new-comic/requirements.txt
109.agents/skills/add-new-comic/scripts/models.py
110.agents/skills/add-new-comic/scripts/generate_comic.py
111.agents/skills/add-new-comic/scripts/generate_panel.py
112.agents/skills/add-new-comic/scripts/composite_panel.py
113
114assets/comics/<slug>/control-panel-01.png
115assets/comics/<slug>/panel-01.png
116assets/comics/<slug>/panel-02.png
117assets/comics/<slug>/panel-03.png
118assets/comics/<slug>/panel-04.png
119assets/comics/<slug>/strip.png
120_posts/YYYY-MM-DD-<slug>.md
121```
122
123Notes:
124
125- Skill-local Python project metadata lives in `.agents/skills/add-new-comic/pyproject.toml` and is executed with `uv run --project .agents/skills/add-new-comic ...`.
126- Generated comic assets should live under `assets/comics/<slug>/` so Jekyll can serve them directly.
127
128## Python dependencies
129
130Use Python 3.10+ and `uv` for Python management.
131
132Core dependencies:
133
134```txt
135torch
136diffusers
137transformers
138accelerate
139safetensors
140Pillow
141pydantic
142opencv-python
143peft
144```
145
146The current ControlNet path uses Diffusers' SDXL ControlNet pipeline plus Pillow/OpenCV preprocessing; no separate renderer is used.
147
148## Generation config
149
150Create a config model for repeatable runs and easier tuning:
151
152- model id
153- refiner enabled/disabled
154- IP-Adapter model/weights
155- reference image paths
156- image size
157- steps
158- guidance scale
159- negative prompt
160- seed policy
161- output paths
162- CUDA/device settings
163- memory optimization toggles
164
165Seed policy should allow both:
166
167- non-deterministic generation by default
168- optional explicit seed for debugging or reruns
169
170## Post format
171
172Image-based posts should use normal Jekyll markdown and embed the final panels-only strip image. Titles, credits, surrounding borders, and page chrome are provided by Jekyll HTML/CSS instead of being baked into the PNG.
173
174Panels are generated separately for easier review and regeneration, then glued into one final strip image with the Pillow compositor.
175
176Example:
177
178```md
179---
180layout: post
181title: The Refactor
182---
183
184<div class="comic-strip image-comic">
185  <img src="/assets/comics/the-refactor/strip.png" alt="The Refactor comic strip">
186</div>
187```
188
189## Refactor phases
190
191### Phase 1: Document and prepare
192
193- Confirm Agent Skills spec requirements.
194- Update `add-new-comic` skill workflow.
195- Add Python dependency documentation.
196- Decide exact generated asset/post structure.
197
198### Phase 2: Build image pipeline
199
200- Add SDXL panel generator.
201- Add IP-Adapter reference conditioning.
202- Add prompt/config models.
203- Add image compositor for bubbles, captions, terminal text, and panel borders.
204- Add Jekyll post writer.
205
206### Phase 3: Replace old flow
207
208- Remove `comic/render-comic.js` and `comic/chat-to-spec.js` from the generation path.
209- Remove `job.sh`; the agent skill is the only formal generation path.
210- Remove deterministic-renderer assumptions from docs and skill text.
211- Update README.
212
213### Phase 4: Regenerate existing posts
214
215Regenerate existing generated comics as image posts:
216
217- `the-cycle-of-ai-coding-burnout`
218- `the-dumb-zone`
219- `the-refactor`
220
221Review generated assets and posts before committing.
222
223### Phase 5: Tune quality
224
225- Tune reference image selection.
226- Tune IP-Adapter scale.
227- Tune negative prompts.
228- Tune ControlNet layout guide strength and end step.
229- Evaluate whether a future LoRA is needed.
230
231## Open implementation questions
232
233- Which IP-Adapter checkpoint should be used for SDXL?
234- ControlNet is now part of the default path; tune guide generation and strength as needed.
235- Generate separate panel images, then compose a final panels-only `strip.png`.
236- Replace old JSON specs with a new image-generation spec format.
237- Do not keep `job.sh`; the agent skill is the formal and only generation path.