Story Mode is an experimental open-source production engine designed to transform stories into consistent animated videos. Instead of treating generative AI as an unconstrained chat loop, Story Mode separates creative reasoning from deterministic control—providing persistent state, durable job scheduling, hardware safety envelopes, character identity preservation, and swappable AI backends.

Generative models produce impressive single images and short video clips, but turning an entire story into a coherent animated video fails today:

• No State or Memory: Models forget character traits, costumes, and palettes across shots.
• Unreliable Agent Loops: Raw LLM loops hallucinate parameters, violate schemas, and crash workstations with VRAM overflows.
• Missing Production Pipeline: Animation requires scene breakdowns, shot lists, asset lineage, quality control, and deterministic composition.

Story Mode provides the control plane for AI video production.

A director (human or autonomous agent) supplies a story premise. Story Mode compiles it into a production DAG, generates keyframes with character and style consistency, animates them via local or cloud video runtimes, synthesizes synchronized voice/audio, runs quality control, and renders a finished 9:16 vertical video.

Story Idea ──► Scene Planning ──► Keyframes ──► Video Animation ──► Audio/TTS ──► Vision QC ──► Final Video
   (LLM)       (Story Mode DAG)   (Diffusion)       (SVD-XT)         (Future M9)   (Future M11)   (FFmpeg 9:16)

Story Mode separates reasoning, engine orchestration, generative backends, and deterministic rendering:

• Director / Agent: Reasoning layer interacting strictly through the typed Agent Tool Protocol.
• Story Mode Engine: Deterministic control plane owning SQLite state (WAL), scheduling, leases, character identity versioning, style bibles, and artifact management.
• Providers: Replaceable AI execution backends (ComfyUI Image, ComfyUI Video, Local LLM).
• Runtime: Physical local or remote inference (ComfyUI daemon, CUDA, VRAM resource limits).
• Renderer: Deterministic composition and media stream validation (FFmpeg, 9:16 vertical delivery).

Detailed architecture documentation: docs/architecture.md.

Story Mode strictly distinguishes between production-ready infrastructure, experimental features, and unsolved research challenges:

• Durable Project State & SQLite Persistence: Full relational persistence in SQLite with WAL mode, foreign keys, and ordered schema migrations (001 through 011).
• Agent Tool Protocol & Typed Runtime Tools: Strict JSON Schema tools (project.*, story.*, character.*, style.*, scene.*, shot.*, image.*, video.*, worker.run_once, artifact.*).
• Durable Scheduler, Workers & Leases: Time-bounded worker leases, lease heartbeats, and automatic crash recovery.
• Resource Locking & Safety Envelopes: Mutual exclusion GPU locks (RESOURCE_GPU_LOCK) and pre-queue VRAM rejection (RESOURCE_REJECTED at 12GB ceiling).
• Project Isolation & Artifact Management: Canonical path traversal protection, project-scoped artifact deduplication, SHA-256 content addressing, and creative provenance tracking.
• Character Identity & Style Consistency: Immutable vs mutable trait versioning, canonical reference threading (PRIMARY_IDENTITY), trait conflict detection, and versioned style bibles.
• Provider Boundaries: Replaceable adapter architecture for LLMs, image generation, and video generation.
• Local Image Runtime Integration: ComfyUI local image adapter with standard 7-node KSampler DAG compilation for SD 1.5 and SDXL Turbo.
• Local Video Runtime (SVD-XT 1.1): Native CLIP Vision ComfyUI graph compilation, container header validation, and ffprobe stream inspection.
• Physical Verification & 9:16 Composition: Live verified on local RTX 2060 12GB hardware; FFmpeg blurred-background vertical delivery.
• Automated Tests: 129/129 passing unit and integration tests.

• Local Video Generation Quality: SVD-XT produces smooth short clips but motion is constrained to camera drift and ambient water/wind motion.
• Short-Shot Animation: Coherence degrades past 25 frames (~3.125s); multi-second shots require chaining.
• SVD-XT as an Animation Backend: Inability to handle complex locomotion (running, walking), gestures, or facial articulation.
• Local Model Integration: Tightly bounded by consumer GPU VRAM limits (12GB envelope).

• High-Quality Cinematic Video Generation: Production-scale motion dynamics and cinematic camera work.
• Wan 2.2 Production Integration: Exploration target only; 14B models exceed consumer host RAM (>32GB required for CPU offload).
• Audio / Speech / TTS Pipeline: Video outputs are silent; voice synthesis, audio stems, and sync are unbuilt (planned for M9).
• Semantic Vision-Based QC: Automated multimodal validation of anatomy, extra limbs, and visual continuity (planned for M11).
• Complete Autonomous Story → Final Reel Loop: Single-prompt autonomous end-to-end film production without supervisor intervention.
• Robust Long-Form Temporal Consistency & Multi-Shot Continuity: Seamless character and wardrobe persistence across multiple sequential video shots.

Generated through the Story Mode pipeline on local hardware (NVIDIA RTX 2060 12GB) using the authoritative Character Identity and Style Bible for Benny the Bear:

Scene 1: Forest Bridge	Scene 2: Ancient Oak Tree	Scene 3: Talking with Bumblebee

Honest Note on Consistency: Character facial identity, fur texture, and storybook watercolor aesthetic remained consistent (9.2/10). However, clothing and accessory consistency remains an open limitation—Benny wears no neck accessory at the bridge, gains a blue bandana under the tree, and wears a green scarf while running. Resolving costume drift requires dedicated reference conditioning (e.g. IP-Adapter).

SVD-XT Local Animation (Preview)	9:16 Vertical Delivery (1080×1920)
Subtle camera drift & ambient water motion	Centered uncropped 16:9 over blurred background

For technical analysis of unsolved challenges and open engineering blockers, see docs/current-limitations.md.

The repository is at Milestone M8.1 (Production Hardening & Local T2I Recovery). The core engine, scheduler, domain state, character/style consistency systems, ComfyUI image/video providers, and 9:16 composition pipeline are fully functional, physically verified, and validated by 129 passing automated tests. We are opening Story Mode to the community to solve the remaining generative and cinematic challenges together.

• Node.js: v20.0.0 or higher
• npm: v9.0.0 or higher
• Optional: Local ComfyUI (:8188) and FFmpeg for local inference.

# Clone the repository and navigate into directory
cd story-mode

# Install dependencies and verify
npm install
npm test
npm run cli -- --help

# Check runtime status
npm run cli -- image runtime
npm run cli -- video runtime

# Run worker loop once
npm run cli -- worker run-once

# Agent Tool Protocol
npm run agent -- tools
npm run agent -- call project.list "{}"

Developer guide: docs/development.md.

Model weights are intentionally NOT included in this repository. Checkpoints range from 2GB to 20GB+ and carry differing licenses. Story Mode is model-agnostic.

For local execution via ComfyUI:

• Text-to-Image: Stable Diffusion 1.5 EMA Pruned FP16 (v1-5-pruned-emaonly-fp16.safetensors, 2.13GB) or SDXL Turbo.
• Image-to-Video: Stable Video Diffusion XT 1.1 (svd_xt_1_1.safetensors, 4.78GB).

Setup instructions & download links: docs/local-models.md.

129 automated unit and integration tests pass via Node's native test runner (npm test).

Physical benchmarks on NVIDIA GeForce RTX 2060 12GB (16GB RAM, Windows 11):

• Image Generation (SD 1.5): 512×512 @ 20 steps: ~1.8s, 3,840 MB peak VRAM.
• Video Generation (SVD-XT 1.1): 576×384 Draft: 91.6s; 768×432 Standard: 134.6s; 1024×576 Hero: 256.5s. Peak VRAM under 8.3 GB.
• 9:16 Composition: 1080×1920 delivery: ~3.2s via FFmpeg.

Full benchmark reports: docs/benchmarks.md.

• M1–M3: Engine Core, Scheduler & Worker — SQLite WAL, DAG compiler, leases, GPU locks.
• M4–M5: Identity & Style Consistency — Character versioning, trait conflict detection, style bibles.
• M6–M7: Provider Architecture & Image Runtime — ComfyUI integration, SD 1.5 local generation.
• M8: Video Runtime (SVD-XT) — Local video generation, VRAM envelopes, 9:16 vertical composition.
• M8.1: Production Hardening — Capability validation, project-scoped dedup, schema ergonomics.
• M9: Audio, Speech & TTS — Voice actor consistency, speech synthesis, sound effects.
• M10: Multi-Shot Timeline & Editing — Timeline compilation, shot transitions, audio-video sync.
• M11: Vision Quality Control — Automated multimodal continuity and anatomy validation.

We welcome community contributions, especially in our key open engineering challenges:

1. Next-Gen Video Models: Integrating Wan 2.2, HunyuanVideo, or LTX-Video with quantized consumer inference.
2. Character & Costume Conditioning: Implementing IP-Adapter or ControlNet reference preprocessors to prevent wardrobe drift across shots.
3. Autonomous Director Loop: Building agent director loops with automated error recovery and adaptive shot re-planning.
4. Audio & Voice Synthesis (M9): Designing the local TTS provider boundary with consistent voice cloning and timing synchronization.
5. Multimodal Vision QC (M11): Building vision evaluators to catch character drift, extra limbs, and rendering defects.

Please read CONTRIBUTING.md for architectural rules, provider guidelines, and pull request submission requirements.

This project was developed using AI-assisted programming and vibe-coding workflows. The project direction, system architecture, requirements, testing strategy, validation criteria, and engineering decisions were actively designed, reviewed, and physically verified on local hardware by the project author.

This project is licensed under the Apache License, Version 2.0.