From 8b7c0b87adeefeeda2b0f7b43b502ae751f53db2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 5 Apr 2026 18:03:01 +0000 Subject: [PATCH] docs: add requirements, roadmap, and initial state Co-Authored-By: Claude Sonnet 4.6 --- .planning/REQUIREMENTS.md | 134 +++++++++++++++++++++++++++++++ .planning/ROADMAP.md | 160 ++++++++++++++++++++++++++++++++++++++ .planning/STATE.md | 66 ++++++++++++++++ 3 files changed, 360 insertions(+) create mode 100644 .planning/REQUIREMENTS.md create mode 100644 .planning/ROADMAP.md create mode 100644 .planning/STATE.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md new file mode 100644 index 0000000..332fb04 --- /dev/null +++ b/.planning/REQUIREMENTS.md @@ -0,0 +1,134 @@ +# Requirements: LightSync + +**Defined:** 2026-04-05 +**Core Value:** Load a song, build a synchronized light show, play it back live — with precise control over every effect on every device. + +--- + +## v1 Requirements + +### Audio Engine + +- [ ] **AUD-01**: Load audio from a local file (MP3, WAV, FLAC, OGG) +- [ ] **AUD-02**: Load audio from a YouTube URL via yt-dlp + MPV +- [ ] **AUD-03**: Playback controls: play, pause, seek to position +- [ ] **AUD-04**: Realtime position reporting from MPV to UI (via named pipe IPC, ≤10ms poll interval) +- [ ] **AUD-05**: Waveform display rendered from audio samples in the timeline UI + +### Device Registry + +- [ ] **DEV-01**: Register a device with: name, strip type (SK6812 / WS2801 / generic), LED count, target IP, target port +- [ ] **DEV-02**: Devices persist across sessions (stored in config file) +- [ ] **DEV-03**: Device list shown in UI — enable/disable per show + +### Animation Library + +- [ ] **ANI-01**: Built-in animation types: chase, pulse, rainbow, strobe, color wipe, fire, solid color +- [ ] **ANI-02**: Per-animation parameters: speed, primary color(s), secondary color(s), direction, length/density +- [ ] **ANI-03**: RGBW-aware color handling for SK6812 (separate W channel) +- [ ] **ANI-04**: RGB color handling for WS2801 + +### Timeline Editor + +- [ ] **TL-01**: Per-device horizontal tracks on a time axis +- [ ] **TL-02**: Drag-and-place animation blocks onto a track +- [ ] **TL-03**: Move, resize, and delete animation blocks +- [ ] **TL-04**: Color picker per animation block +- [ ] **TL-05**: Playback cursor (vertical line) tracking audio position in realtime +- [ ] **TL-06**: Undo/redo for all timeline mutations (min 20 steps, command pattern) +- [ ] **TL-07**: Beat detection marks overlaid on timeline (librosa beat_track) +- [ ] **TL-08**: Snap-to-beat when placing/resizing blocks + +### Show Engine + +- [ ] **SHW-01**: Show files saved and loaded as JSON (schema_version field required in v1) +- [ ] **SHW-02**: Show file includes: audio path/URL, device registry snapshot, all animation blocks with timestamps, beat analysis data +- [ ] **SHW-03**: Live show execution — follow MPV position, dispatch animation commands to devices via UDP at correct timestamps + +### UDP Communication + +- [ ] **UDP-01**: Animation command packet: device ID, animation type, params — sent when block starts +- [ ] **UDP-02**: Raw pixel frame packet: WLED DRGB/DNRGB/DRGBW protocol — used for AI-generated sequences +- [ ] **UDP-03**: Multi-packet framing for frames >MTU (SK6812 300 LEDs RGBW = 1200 bytes, exceeds WiFi MTU) +- [ ] **UDP-04**: Software UDP receiver/simulator for testing without physical hardware + +### Automatic Sync + +- [ ] **SYNC-01**: Beat detection via librosa — produces beat timestamps and onset marks from loaded audio +- [ ] **SYNC-02**: Beat calibration offset (UI control) — compensate for librosa's systematic 20-60ms latency bias +- [ ] **SYNC-03**: AI-assisted show generation — given beat/segment analysis, auto-fill timeline with matched animations (basic heuristic mapping in v1; LLM enhancement optional) + +### UI / UX + +- [ ] **UI-01**: Web app served by Python backend — no external hosting required +- [ ] **UI-02**: Terminal/hacker aesthetic — dark theme, monospace fonts, CRT/scanline vibe; NOT generic SaaS look +- [ ] **UI-03**: All primary operations accessible without mouse (keyboard shortcuts for play/pause/seek/undo) +- [ ] **UI-04**: Live preview panel — simulated LED strip visualization in browser canvas, synced to playback + +### Infrastructure + +- [ ] **INF-01**: Python 3.11 backend (FastAPI + uvicorn) +- [ ] **INF-02**: Modular device abstraction — adding a new strip type requires only a new device class, no changes to core engine +- [ ] **INF-03**: Windows 11 timer resolution fix applied at startup (`timeBeginPeriod(1)` via ctypes) + +--- + +## v2 Requirements + +### Enhancements (deferred) + +- **V2-01**: Structural segmentation overlay — librosa section detection (verse/chorus/bridge) shown as colored regions on waveform +- **V2-02**: Effect grouping / scenes — group blocks across devices into named scenes, drag-place whole scenes +- **V2-03**: Frequency band mapping — bass/mid/treble energy mapped to device params +- **V2-04**: Parameter animation curves — envelope curves on per-block params (speed, brightness) over time +- **V2-05**: Show export for standalone/offline playback (self-contained show package) + +### Firmware (deferred — last phase) + +- **FW-01**: MicroPython firmware for ESP32/Pico/RPi 4B — receives UDP animation command packets, renders animations locally +- **FW-02**: MicroPython firmware — receives DRGB/DNRGB/DRGBW raw frame packets, outputs to LED strip + +--- + +## Out of Scope + +| Feature | Reason | +|---------|--------| +| Real-time microphone input | Conflicts with file-based deterministic sync model; adds latency pipeline | +| Multi-user / collaborative editing | Zero personal-use value; conflicts complicate state management | +| Cloud sync / remote access | Local network only; Authelia provides LAN access | +| MIDI controller support | Beat detection covers the use case; adds OS driver complexity | +| DMX / ArtNet / E1.31 | Direct UDP is simpler and lower latency at this scale | +| Plugin / extension system | Add effects directly in Python; modular device abstraction covers extensibility | +| 3D fixture visualizer | 2D canvas preview is sufficient for LED strips | +| Scheduling / playlist automation | Interactive personal tool; manual launch from UI is correct | + +--- + +## Traceability + +| Requirement | Phase | Status | +|-------------|-------|--------| +| INF-01, INF-02 | Phase 1 | Pending | +| SHW-01, SHW-02 | Phase 1 | Pending | +| DEV-01, DEV-02, DEV-03 | Phase 1 | Pending | +| AUD-01, AUD-02, AUD-03, AUD-04 | Phase 2 | Pending | +| AUD-05 | Phase 2 | Pending | +| INF-03 | Phase 2 | Pending | +| UDP-01, UDP-02, UDP-03, UDP-04 | Phase 3 | Pending | +| ANI-01, ANI-02, ANI-03, ANI-04 | Phase 3 | Pending | +| TL-01, TL-02, TL-03, TL-04, TL-05, TL-06 | Phase 4 | Pending | +| TL-07, TL-08, SYNC-01, SYNC-02 | Phase 4 | Pending | +| SHW-03, UI-03 | Phase 5 | Pending | +| UI-01, UI-02, UI-04 | Phases 1-5 (progressive) | Pending | +| SYNC-03 | Phase 6 | Pending | +| FW-01, FW-02 | Phase 7 | Pending | + +**Coverage:** +- v1 requirements: 31 total +- Mapped to phases: 31 +- Unmapped: 0 ✓ + +--- +*Requirements defined: 2026-04-05* +*Last updated: 2026-04-05 after initialization* diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md new file mode 100644 index 0000000..8f49cd0 --- /dev/null +++ b/.planning/ROADMAP.md @@ -0,0 +1,160 @@ +# Roadmap: LightSync + +## Overview + +LightSync is built bottom-up: data model first, audio engine second, UDP protocol third, timeline editor fourth, live execution fifth, AI enhancements sixth, and microcontroller firmware last. Each phase delivers a coherent, independently testable capability. The ordering is dictated by dependencies — the show file schema underpins everything, audio position drives all timing, the UDP protocol must be frozen before firmware is written, and the firmware can only be built once the full stack is validated against a simulator. The result is a complete music-to-light show system where you can load a song, build a synchronized show, and play it back live to real LED hardware. + +## Phases + +**Phase Numbering:** +- Integer phases (1, 2, 3): Planned milestone work +- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED) + +Decimal phases appear between their surrounding integers in numeric order. + +- [ ] **Phase 1: Foundation** - FastAPI skeleton, show file schema (Pydantic), device registry, terminal-aesthetic frontend shell +- [ ] **Phase 2: Audio Engine** - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix +- [ ] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator +- [ ] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat +- [ ] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview +- [ ] **Phase 6: AI Sync** - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation +- [ ] **Phase 7: Microcontroller Firmware** - MicroPython firmware for ESP32/Pico/RPi, animation command + frame receivers for SK6812 and WS2801 + +## Phase Details + +### Phase 1: Foundation +**Goal**: The project skeleton exists — data models, API shell, device registry, and a working terminal-aesthetic UI shell the rest of the system will fill in +**Depends on**: Nothing (first phase) +**Requirements**: INF-01, INF-02, DEV-01, DEV-02, DEV-03, SHW-01, SHW-02, UI-01, UI-02 +**Success Criteria** (what must be TRUE): + 1. A device can be registered with name, strip type, LED count, IP, and port — and survives an app restart + 2. A show file can be saved and reloaded as JSON with schema_version, audio reference, device snapshot, and empty track list + 3. The web app loads in a browser with terminal/hacker aesthetic — dark background, monospace fonts, no generic SaaS look + 4. FastAPI serves the frontend and exposes REST stubs for shows and devices + 5. Adding a new strip type requires only a new device class — no changes to core engine code +**Plans**: TBD + +Plans: +- [ ] 01-01: Backend scaffold — FastAPI app, Pydantic show/device models, show store (JSON load/save), device registry +- [ ] 01-02: Frontend shell — HTML/CSS terminal aesthetic, WebSocket client stub, device panel, show panel +- [ ] 01-03: REST API — CRUD endpoints for devices and shows, static file serving, integration smoke test +**UI hint**: yes + +### Phase 2: Audio Engine +**Goal**: Audio plays back reliably with sub-20ms position accuracy — the master clock the entire show system depends on +**Depends on**: Phase 1 +**Requirements**: AUD-01, AUD-02, AUD-03, AUD-04, AUD-05, INF-03 +**Success Criteria** (what must be TRUE): + 1. A local audio file (MP3, WAV, FLAC, OGG) loads and plays via MPV with play/pause/seek controls in the UI + 2. Audio position is reported to the browser at 10Hz or better with no observable drift during a 5-minute playback + 3. The waveform is rendered in the timeline UI from the loaded audio file + 4. Windows 11 timer resolution is set to 1ms at startup (timeBeginPeriod(1)) — confirmed via timing harness showing <2ms tick variance + 5. MPV IPC runs in a dedicated thread — pausing or seeking does not block WebSocket message processing +**Plans**: TBD + +Plans: +- [ ] 02-01: MPV IPC bridge — python-mpv subprocess, named pipe reader thread, shared position variable, Windows timer fix +- [ ] 02-02: Show engine skeleton + WebSocket hub — playback state machine, position broadcast at 10Hz, transport command dispatcher +- [ ] 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples +**UI hint**: yes + +### Phase 3: Communication Protocol +**Goal**: The UDP protocol is fully specified, encoded, and validated against a software simulator — the contract microcontrollers will implement +**Depends on**: Phase 2 +**Requirements**: UDP-01, UDP-02, UDP-03, UDP-04, ANI-01, ANI-02, ANI-03, ANI-04 +**Success Criteria** (what must be TRUE): + 1. An animation command packet (device ID, animation type, params) can be sent to a device and decoded correctly by the simulator + 2. A raw DRGB/DRGBW pixel frame for a 300-LED strip is split across multiple packets when it exceeds MTU — simulator reassembles correctly + 3. The animation library contains all 7 built-in types (chase, pulse, rainbow, strobe, color wipe, fire, solid color) with configurable params + 4. SK6812 (RGBW, 4-byte) and WS2801 (RGB, 3-byte) frames are encoded distinctly — a simulator can distinguish them by packet header + 5. The software simulator logs all received packets — no hardware required to validate protocol behavior +**Plans**: TBD + +Plans: +- [ ] 03-01: Device abstraction + UDP sender — BaseDevice ABC, SK6812 and WS2801 implementations, async UDP socket pool +- [ ] 03-02: Packet protocol — DRGB/DNRGB/DRGBW encoders, multi-packet framing for >MTU frames, animation command packet format +- [ ] 03-03: Animation library + simulator — 7 built-in animation types with params, software UDP receiver/simulator, end-to-end protocol validation + +### Phase 4: Timeline Editor +**Goal**: The timeline editor is fully functional — users can place, move, resize, and delete animation blocks on per-device tracks with beat-snapping and undo/redo +**Depends on**: Phase 3 +**Requirements**: TL-01, TL-02, TL-03, TL-04, TL-05, TL-06, TL-07, TL-08, SYNC-01, SYNC-02 +**Success Criteria** (what must be TRUE): + 1. Per-device horizontal tracks display on a time axis; animation blocks can be dragged from the animation library and placed on a track + 2. Blocks can be moved, resized, and deleted; at least 20 undo/redo steps work correctly for all three operations + 3. Beat detection runs on the loaded audio file and overlays beat marks on the timeline; blocks snap to beat positions when placed or resized + 4. Each block has a color picker and editable parameters (speed, colors, direction, length); changes persist in the show file + 5. The playback cursor moves in real time across the timeline as audio plays, reflecting audio position accurately + 6. A beat calibration offset control shifts all beat marks backward by a configurable amount to compensate for librosa's systematic latency bias +**Plans**: TBD + +Plans: +- [ ] 04-01: Timeline canvas — per-device tracks, time axis, block rendering, playback cursor +- [ ] 04-02: Block interactions — drag-and-place, move, resize, delete; command pattern undo/redo (min 20 steps) +- [ ] 04-03: Beat detection + snap — librosa beat_track + plp + onset_detect in ProcessPoolExecutor, beat mark overlay, snap-to-beat, calibration offset UI +- [ ] 04-04: Block editor — animation library panel, color picker, parameter controls per block, changes reflected in show file +**UI hint**: yes + +### Phase 5: Live Show Execution +**Goal**: A complete show runs end-to-end — audio plays, cues fire at correct timestamps via UDP, live preview updates in the browser, and the show persists across sessions +**Depends on**: Phase 4 +**Requirements**: SHW-03, UI-03, UI-04 +**Success Criteria** (what must be TRUE): + 1. Pressing Play fires animation commands to devices (or simulator) at their correct timestamps — cue timing is within 20ms of audio position + 2. Seeking to any position in the show reschedules pending cues correctly — no missed or double-fired cues + 3. The live preview panel shows a simulated LED strip visualization in the browser canvas, updated in sync with playback + 4. All primary operations (play, pause, seek, undo, save) are accessible via keyboard shortcuts without touching the mouse + 5. A show saved during one session loads completely — all blocks, devices, and analysis data — in a subsequent session +**Plans**: TBD + +Plans: +- [ ] 05-01: Cue scheduler — position-driven scheduling loop, pre-fire offset, seek-safe cue pointer reset +- [ ] 05-02: Live preview panel — canvas LED strip visualization synced to playback via WebSocket position ticks +- [ ] 05-03: Show persistence + keyboard shortcuts — complete save/load round-trip, keyboard bindings for all primary operations +**UI hint**: yes + +### Phase 6: AI Sync +**Goal**: AI-assisted show generation fills the timeline automatically from beat and segment analysis; YouTube loading works; structural segmentation is visible +**Depends on**: Phase 5 +**Requirements**: SYNC-03, AUD-02 +**Success Criteria** (what must be TRUE): + 1. A YouTube URL can be loaded into the app — audio plays via MPV with no manual download step required + 2. Triggering AI sync fills the timeline with animation blocks matched to beats, drops, and song sections — producing a playable starting point for editing + 3. Structural segmentation (verse/chorus/bridge via librosa) appears as colored region overlays on the waveform +**Plans**: TBD + +Plans: +- [ ] 06-01: YouTube loading — yt-dlp passthrough via MPV --ytdl flag, URL input field, loading state feedback +- [ ] 06-02: AI sync engine — heuristic rule mapper (beat/onset/chroma → animation selection + placement), auto-fill timeline action +- [ ] 06-03: Structural segmentation overlay — librosa section detection, colored region rendering on waveform + +### Phase 7: Microcontroller Firmware +**Goal**: Physical LED hardware responds to UDP packets from the show system — both animation command mode and raw frame mode work on real strips +**Depends on**: Phase 6 +**Requirements**: (FW-01, FW-02 from v2 deferred list — treated as v1 completion) +**Success Criteria** (what must be TRUE): + 1. A microcontroller receives animation command packets and renders the correct animation on a physical LED strip + 2. A microcontroller receives DRGB/DRGBW raw frame packets and outputs correct RGB/RGBW pixel values to the strip + 3. SK6812 (RGBW) and WS2801 (RGB) are both verified on physical hardware against the protocol spec from Phase 3 + 4. Firmware runs on at least one of: ESP32, Pico, or Raspberry Pi 4B +**Plans**: TBD + +Plans: +- [ ] 07-01: Firmware scaffold — MicroPython UDP socket listener, packet type discriminator, device config (LED count, strip type) +- [ ] 07-02: Animation command renderer — parse animation+params packets, run animations on LED strip +- [ ] 07-03: Frame mode renderer — parse DRGB/DRGBW frames, handle multi-packet reassembly, output to physical strip + +## Progress + +**Execution Order:** +Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 + +| Phase | Plans Complete | Status | Completed | +|-------|----------------|--------|-----------| +| 1. Foundation | 0/3 | Not started | - | +| 2. Audio Engine | 0/3 | Not started | - | +| 3. Communication Protocol | 0/3 | Not started | - | +| 4. Timeline Editor | 0/4 | Not started | - | +| 5. Live Show Execution | 0/3 | Not started | - | +| 6. AI Sync | 0/3 | Not started | - | +| 7. Microcontroller Firmware | 0/3 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md new file mode 100644 index 0000000..3baedc9 --- /dev/null +++ b/.planning/STATE.md @@ -0,0 +1,66 @@ +# Project State + +## Project Reference + +See: .planning/PROJECT.md (updated 2026-04-05) + +**Core value:** Load a song, build a synchronized light show, play it back live — with precise control over every effect on every device. +**Current focus:** Phase 1 — Foundation + +## Current Position + +Phase: 1 of 7 (Foundation) +Plan: 0 of 3 in current phase +Status: Ready to plan +Last activity: 2026-04-05 — Roadmap and state initialized + +Progress: [░░░░░░░░░░] 0% + +## Performance Metrics + +**Velocity:** +- Total plans completed: 0 +- Average duration: — +- Total execution time: 0 hours + +**By Phase:** + +| Phase | Plans | Total | Avg/Plan | +|-------|-------|-------|----------| +| - | - | - | - | + +**Recent Trend:** +- Last 5 plans: — +- Trend: — + +*Updated after each plan completion* + +## Accumulated Context + +### Decisions + +Decisions are logged in PROJECT.md Key Decisions table. +Recent decisions affecting current work: + +- Init: Web UI over TUI — richer DAW interactions while keeping terminal aesthetic via CSS +- Init: python-mpv-jsonipc preferred over ctypes binding — no libmpv.dll dependency on Windows +- Init: Firmware deferred to Phase 7 — protocol must be validated against simulator first +- Init: undo/redo built in Phase 4 — retrofitting command pattern is expensive + +### Pending Todos + +None yet. + +### Blockers/Concerns + +- Phase 2: python-mpv-jsonipc latency on Windows 11 under load is unbenchmarked — validate with timing harness in 02-01 +- Phase 4: librosa MP3 loading path on Windows needs explicit validation (soundfile falls back to audioread which requires FFmpeg on PATH) +- Phase 6: AI sync approach (LLM-assisted) has no reference implementation — treat as spike, not commitment + +## Session Continuity + +Last session: 2026-04-05 +Stopped at: Roadmap created, STATE.md initialized +Resume file: None + +Next action: `/gsd:plan-phase 1`