7517
Cd
clanker-and-the-dev / files
clanker-and-the-dev / comic / sdxl-generation-plan.md
7.4 KB
1# SDXL Comic Generation Refactor Plan 2 3Goal: replace the current deterministic SVG comic renderer with a non-deterministic SDXL image pipeline driven by the `add-new-comic` agent skill. 4 5## Decisions 6 7- The agent skill is the entrypoint for the full flow. 8- The deterministic SVG renderer is not kept as a fallback. 9- Existing generated posts should be regenerated with the new image-based pipeline. 10- The Python generation code belongs with the agent skill and must respect the Agent Skills spec from agentskills.io. 11- CUDA is expected to be available; target hardware is an RTX 3080 Ti 12 GB. 12- Default model backend is SDXL. 13- Default model is `stabilityai/stable-diffusion-xl-base-1.0`. 14- Use generated image panels plus deterministic text/comic compositing, not model-rendered text. 15 16## Recommended pipeline 17 18```txt 19user joke / premise 20 -> add-new-comic skill 21 -> turn premise into panel script + visual directions 22 -> generate rough panel composition guides 23 -> generate panel art with SDXL conditioned by references 24 -> composite speech bubbles, captions, terminal text, borders with Python/Pillow 25 -> write PNG panels/assets 26 -> write Jekyll markdown post 27``` 28 29## Why not plain text-to-image only 30 31Plain SDXL text-to-image does not use files in `art-reference/` just because they exist in the repo. If the pipeline only sends a prompt, the model never sees the reference sheets. 32 33To make the model actually follow the Clanker/dev visual identity, the new path should use image conditioning. 34 35## Image conditioning strategy 36 37### First implementation: SDXL + IP-Adapter 38 39Use IP-Adapter to condition SDXL on project reference images. 40 41Chosen default checkpoint: 42 43```txt 44repo: h94/IP-Adapter 45subfolder: sdxl_models 46weight: ip-adapter_sdxl.bin 47``` 48 49Reference images now default to: 50 51- `art-reference/training-reference.png` 52- `art-reference/ip-character-reference.png` 53 54The generator extracts large per-panel crops from the clean training sheet, extracts character-only crops from the two-character IP reference, and averages IP-Adapter image embeddings. This gives the model character/style identity without asking it to copy a full grid/reference sheet or character-card layout. 55 56Expected benefits: 57 58- Better style matching than prompt-only SDXL. 59- No model training required. 60- Works with existing reference sheets. 61 62Known limitations: 63 64- Character consistency may still drift. 65- Multi-character panels can confuse identity. 66- Reference strength will need tuning. 67 68### Composition control: ControlNet 69 70ControlNet is now enabled by default because SDXL + IP-Adapter alone did not match the references consistently enough. 71 72The generator creates black/white composition guides from `art-reference/` character sprites plus generated props/effects for each panel and uses ControlNet to preserve structure: 73 74- Dev position 75- Clanker position 76- terminal/monitor position 77- important props 78- reserved speech bubble areas 79- panel framing 80 81ControlNet helps composition, but does not replace IP-Adapter for style/identity. The generated guides are written as `assets/comics/<slug>/control-panel-XX.png` for review/debugging, and the compositor clips raw line art to the guide before merging it back into the final panel to repair drift and remove scanline/blob artifacts. 82 83### Future improvement: SDXL LoRA 84 85If IP-Adapter is not consistent enough, train a small SDXL LoRA for the comic style and characters. 86 87This should be treated as a later phase because it requires a dataset and training workflow. 88 89## Text and lettering 90 91Do not rely on Stable Diffusion for readable text. 92 93The Python compositor should add: 94 95- speech bubbles 96- captions 97- terminal text 98- panel borders 99- title/credit if needed 100 101This keeps the comic readable while allowing the art to be non-deterministic. 102 103## Proposed file layout 104 105```txt 106.agents/skills/add-new-comic/SKILL.md 107.agents/skills/add-new-comic/pyproject.toml 108.agents/skills/add-new-comic/requirements.txt 109.agents/skills/add-new-comic/scripts/models.py 110.agents/skills/add-new-comic/scripts/generate_comic.py 111.agents/skills/add-new-comic/scripts/generate_panel.py 112.agents/skills/add-new-comic/scripts/composite_panel.py 113 114assets/comics/<slug>/control-panel-01.png 115assets/comics/<slug>/panel-01.png 116assets/comics/<slug>/panel-02.png 117assets/comics/<slug>/panel-03.png 118assets/comics/<slug>/panel-04.png 119assets/comics/<slug>/strip.png 120_posts/YYYY-MM-DD-<slug>.md 121``` 122 123Notes: 124 125- Skill-local Python project metadata lives in `.agents/skills/add-new-comic/pyproject.toml` and is executed with `uv run --project .agents/skills/add-new-comic ...`. 126- Generated comic assets should live under `assets/comics/<slug>/` so Jekyll can serve them directly. 127 128## Python dependencies 129 130Use Python 3.10+ and `uv` for Python management. 131 132Core dependencies: 133 134```txt 135torch 136diffusers 137transformers 138accelerate 139safetensors 140Pillow 141pydantic 142opencv-python 143peft 144``` 145 146The current ControlNet path uses Diffusers' SDXL ControlNet pipeline plus Pillow/OpenCV preprocessing; no separate renderer is used. 147 148## Generation config 149 150Create a config model for repeatable runs and easier tuning: 151 152- model id 153- refiner enabled/disabled 154- IP-Adapter model/weights 155- reference image paths 156- image size 157- steps 158- guidance scale 159- negative prompt 160- seed policy 161- output paths 162- CUDA/device settings 163- memory optimization toggles 164 165Seed policy should allow both: 166 167- non-deterministic generation by default 168- optional explicit seed for debugging or reruns 169 170## Post format 171 172Image-based posts should use normal Jekyll markdown and embed the final panels-only strip image. Titles, credits, surrounding borders, and page chrome are provided by Jekyll HTML/CSS instead of being baked into the PNG. 173 174Panels are generated separately for easier review and regeneration, then glued into one final strip image with the Pillow compositor. 175 176Example: 177 178```md 179--- 180layout: post 181title: The Refactor 182--- 183 184<div class="comic-strip image-comic"> 185 <img src="/assets/comics/the-refactor/strip.png" alt="The Refactor comic strip"> 186</div> 187``` 188 189## Refactor phases 190 191### Phase 1: Document and prepare 192 193- Confirm Agent Skills spec requirements. 194- Update `add-new-comic` skill workflow. 195- Add Python dependency documentation. 196- Decide exact generated asset/post structure. 197 198### Phase 2: Build image pipeline 199 200- Add SDXL panel generator. 201- Add IP-Adapter reference conditioning. 202- Add prompt/config models. 203- Add image compositor for bubbles, captions, terminal text, and panel borders. 204- Add Jekyll post writer. 205 206### Phase 3: Replace old flow 207 208- Remove `comic/render-comic.js` and `comic/chat-to-spec.js` from the generation path. 209- Remove `job.sh`; the agent skill is the only formal generation path. 210- Remove deterministic-renderer assumptions from docs and skill text. 211- Update README. 212 213### Phase 4: Regenerate existing posts 214 215Regenerate existing generated comics as image posts: 216 217- `the-cycle-of-ai-coding-burnout` 218- `the-dumb-zone` 219- `the-refactor` 220 221Review generated assets and posts before committing. 222 223### Phase 5: Tune quality 224 225- Tune reference image selection. 226- Tune IP-Adapter scale. 227- Tune negative prompts. 228- Tune ControlNet layout guide strength and end step. 229- Evaluate whether a future LoRA is needed. 230 231## Open implementation questions 232 233- Which IP-Adapter checkpoint should be used for SDXL? 234- ControlNet is now part of the default path; tune guide generation and strength as needed. 235- Generate separate panel images, then compose a final panels-only `strip.png`. 236- Replace old JSON specs with a new image-generation spec format. 237- Do not keep `job.sh`; the agent skill is the formal and only generation path.