led2/.planning/ROADMAP.md

Roadmap: LightSync

Overview

LightSync is built bottom-up: data model first, audio engine second, UDP protocol third, timeline editor fourth, live execution fifth, AI enhancements sixth, and microcontroller firmware last. Each phase delivers a coherent, independently testable capability. The ordering is dictated by dependencies — the show file schema underpins everything, audio position drives all timing, the UDP protocol must be frozen before firmware is written, and the firmware can only be built once the full stack is validated against a simulator. The result is a complete music-to-light show system where you can load a song, build a synchronized show, and play it back live to real LED hardware.

Phases

Phase Numbering:

• Integer phases (1, 2, 3): Planned milestone work
• Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)

Decimal phases appear between their surrounding integers in numeric order.

• Phase 1: Foundation - FastAPI skeleton, show file schema (Pydantic), device registry, terminal-aesthetic frontend shell (gap closure in progress) (completed 2026-04-05)
• Phase 2: Audio Engine - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix (completed 2026-04-06)
• Phase 3: Communication Protocol - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator (completed 2026-04-06)
• Phase 4: Timeline Editor - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat (completed 2026-04-06)
• Phase 5: Live Show Execution - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview (completed 2026-04-07)
• Phase 6: AI Sync - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation (completed 2026-04-07)
• Phase 7: Microcontroller Firmware - MicroPython firmware for ESP32/Pico/RPi, animation command + frame receivers for SK6812 and WS2801

Phase Details

Phase 1: Foundation

Goal: The project skeleton exists — data models, API shell, device registry, and a working terminal-aesthetic UI shell the rest of the system will fill in Depends on: Nothing (first phase) Requirements: INF-01, INF-02, DEV-01, DEV-02, DEV-03, SHW-01, SHW-02, UI-01, UI-02 Success Criteria (what must be TRUE):

1. A device can be registered with name, strip type, LED count, IP, and port — and survives an app restart
2. A show file can be saved and reloaded as JSON with schema_version, audio reference, device snapshot, and empty track list
3. The web app loads in a browser with terminal/hacker aesthetic — dark background, monospace fonts, no generic SaaS look
4. FastAPI serves the frontend and exposes REST stubs for shows and devices
5. Adding a new strip type requires only a new device class — no changes to core engine code Plans: 4 plans

Plans:

• 01-01: Backend scaffold — FastAPI app, Pydantic show/device models, show store (JSON load/save), device registry
• 01-02: Frontend shell — HTML/CSS terminal aesthetic, WebSocket client stub, device panel, show panel
• 01-03: REST API — CRUD endpoints for devices and shows, static file serving, integration smoke test
• 01-04: Gap closure — device CRUD UI (add form, remove button), CSS variable naming fix, panel IDs UI hint: yes

Phase 2: Audio Engine

Goal: Audio plays back reliably with sub-20ms position accuracy — the master clock the entire show system depends on Depends on: Phase 1 Requirements: AUD-01, AUD-02, AUD-03, AUD-04, AUD-05, INF-03 Success Criteria (what must be TRUE):

1. A local audio file (MP3, WAV, FLAC, OGG) loads and plays via MPV with play/pause/seek controls in the UI
2. Audio position is reported to the browser at 10Hz or better with no observable drift during a 5-minute playback
3. The waveform is rendered in the timeline UI from the loaded audio file
4. Windows 11 timer resolution is set to 1ms at startup (timeBeginPeriod(1)) — confirmed via timing harness showing <2ms tick variance
5. MPV IPC runs in a dedicated thread — pausing or seeking does not block WebSocket message processing Plans: 3 plans

Plans:

• 02-01: MPV IPC bridge — python-mpv subprocess, named pipe reader thread, shared position variable, Windows timer fix
• 02-02: Show engine skeleton + WebSocket hub — playback state machine, position broadcast at 10Hz, transport command dispatcher
• 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples UI hint: yes

Phase 3: Communication Protocol

Goal: The UDP protocol is fully specified, encoded, and validated against a software simulator — the contract microcontrollers will implement Depends on: Phase 2 Requirements: UDP-01, UDP-02, UDP-03, UDP-04, ANI-01, ANI-02, ANI-03, ANI-04 Success Criteria (what must be TRUE):

1. An animation command packet (device ID, animation type, params) can be sent to a device and decoded correctly by the simulator
2. A raw DRGB/DRGBW pixel frame for a 300-LED strip is split across multiple packets when it exceeds MTU — simulator reassembles correctly
3. The animation library contains all 7 built-in types (chase, pulse, rainbow, strobe, color wipe, fire, solid color) with configurable params
4. SK6812 (RGBW, 4-byte) and WS2801 (RGB, 3-byte) frames are encoded distinctly — a simulator can distinguish them by packet header
5. The software simulator logs all received packets — no hardware required to validate protocol behavior Plans: 3 plans

Plans:

• 03-01-PLAN.md — Protocol encoders (DRGB/DRGBW/DNRGB + animation command), UDP sender, device stub completion
• 03-02-PLAN.md — Animation library (7 built-in types with AnimationBase ABC, registry, factory)
• 03-03-PLAN.md — UDP simulator, FastAPI integration, end-to-end protocol validation

Phase 4: Timeline Editor

Goal: The timeline editor is fully functional — users can place, move, resize, and delete animation blocks on per-device tracks with beat-snapping and undo/redo Depends on: Phase 3 Requirements: TL-01, TL-02, TL-03, TL-04, TL-05, TL-06, TL-07, TL-08, SYNC-01, SYNC-02 Success Criteria (what must be TRUE):

1. Per-device horizontal tracks display on a time axis; animation blocks can be dragged from the animation library and placed on a track
2. Blocks can be moved, resized, and deleted; at least 20 undo/redo steps work correctly for all three operations
3. Beat detection runs on the loaded audio file and overlays beat marks on the timeline; blocks snap to beat positions when placed or resized
4. Each block has a color picker and editable parameters (speed, colors, direction, length); changes persist in the show file
5. The playback cursor moves in real time across the timeline as audio plays, reflecting audio position accurately
6. A beat calibration offset control shifts all beat marks backward by a configurable amount to compensate for librosa's systematic latency bias Plans: 4 plans

Plans:

• 04-01: Timeline canvas — per-device tracks, time axis, block rendering, playback cursor
• 04-02: Block interactions — drag-and-place, move, resize, delete; command pattern undo/redo (min 20 steps)
• 04-03: Beat detection + snap — librosa beat_track + plp + onset_detect in ProcessPoolExecutor, beat mark overlay, snap-to-beat, calibration offset UI
• 04-04: Block editor — animation library panel, color picker, parameter controls per block, changes reflected in show file UI hint: yes

Phase 5: Live Show Execution

Goal: A complete show runs end-to-end — audio plays, cues fire at correct timestamps via UDP, live preview updates in the browser, and the show persists across sessions Depends on: Phase 4 Requirements: SHW-03, UI-03, UI-04 Success Criteria (what must be TRUE):

1. Pressing Play fires animation commands to devices (or simulator) at their correct timestamps — cue timing is within 20ms of audio position
2. Seeking to any position in the show reschedules pending cues correctly — no missed or double-fired cues
3. The live preview panel shows a simulated LED strip visualization in the browser canvas, updated in sync with playback
4. All primary operations (play, pause, seek, undo, save) are accessible via keyboard shortcuts without touching the mouse
5. A show saved during one session loads completely — all blocks, devices, and analysis data — in a subsequent session Plans: 4 plans

Plans:

• 05-01-PLAN.md — Cue scheduler: server-side position-driven scheduling, 60ms look-ahead, seek-safe fired_ids reset
• 05-02-PLAN.md — Live preview panel + show selector: DOM device strips, preview_update handler, show load into timeline
• 05-03-PLAN.md — Keyboard shortcuts: Space, arrows, Ctrl+Z/Y/S for mouse-free operation
• 05-04-PLAN.md — Gap closure: wire WebSocket play/pause messages from mouse transport button UI hint: yes

Phase 6: AI Sync

Goal: AI-assisted show generation fills the timeline automatically from beat and segment analysis; YouTube loading works; structural segmentation is visible Depends on: Phase 5 Requirements: SYNC-03, AUD-02 Success Criteria (what must be TRUE):

1. A YouTube URL can be loaded into the app — audio plays via MPV with no manual download step required
2. Triggering AI sync fills the timeline with animation blocks matched to beats, drops, and song sections — producing a playable starting point for editing
3. Structural segmentation (verse/chorus/bridge via librosa) appears as colored region overlays on the waveform Plans: 3 plans

Plans:

• 06-01: YouTube loading — yt-dlp download to /app/shows/, SSE progress streaming, transport bar URL input
• 06-02: Feature extraction + segmentation — onset/chroma/RMS extraction, structural segmentation API, colored section band overlay on waveform
• 06-03: AI sync engine — heuristic block mapper + optional LLM generation, AI Sync panel UI, per-section fill

Phase 7: Microcontroller Firmware

Goal: Physical LED hardware responds to UDP packets from the show system — both animation command mode and raw frame mode work on real strips Depends on: Phase 6 Requirements: (FW-01, FW-02 from v2 deferred list — treated as v1 completion) Success Criteria (what must be TRUE):

1. A microcontroller receives animation command packets and renders the correct animation on a physical LED strip
2. A microcontroller receives DRGB/DRGBW raw frame packets and outputs correct RGB/RGBW pixel values to the strip
3. SK6812 (RGBW) and WS2801 (RGB) are both verified on physical hardware against the protocol spec from Phase 3
4. Firmware runs on at least one of: ESP32, Pico, or Raspberry Pi 4B Plans: 3 plans

Plans:

• 07-01: Firmware scaffold — MicroPython UDP socket listener, packet type discriminator, device config (LED count, strip type)
• 07-02: Animation command renderer — parse animation+params packets, run animations on LED strip
• 07-03: Frame mode renderer — parse DRGB/DRGBW frames, handle multi-packet reassembly, output to physical strip

Progress

Execution Order: Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7

Phase	Plans Complete	Status	Completed
1. Foundation	4/4	Complete	2026-04-05
2. Audio Engine	3/3	Complete	2026-04-06
3. Communication Protocol	3/3	Complete	2026-04-06
4. Timeline Editor	4/4	Complete	2026-04-06
5. Live Show Execution	4/4	Complete	2026-04-07
6. AI Sync	3/3	Complete	2026-04-07
7. Microcontroller Firmware	1/3	In Progress