From 6e05abe7a82d8a1fa8cd799c1a0bf637da3147d8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 5 Apr 2026 17:59:51 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20add=20domain=20research=20=E2=80=94=20s?= =?UTF-8?q?tack,=20features,=20architecture,=20pitfalls?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .planning/research/ARCHITECTURE.md | 554 +++++++++++++++++++++++++++++ .planning/research/FEATURES.md | 231 ++++++++++++ .planning/research/PITFALLS.md | 377 ++++++++++++++++++++ .planning/research/STACK.md | 214 +++++++++++ .planning/research/SUMMARY.md | 216 +++++++++++ 5 files changed, 1592 insertions(+) create mode 100644 .planning/research/ARCHITECTURE.md create mode 100644 .planning/research/FEATURES.md create mode 100644 .planning/research/PITFALLS.md create mode 100644 .planning/research/STACK.md create mode 100644 .planning/research/SUMMARY.md diff --git a/.planning/research/ARCHITECTURE.md b/.planning/research/ARCHITECTURE.md new file mode 100644 index 0000000..9e7bfe9 --- /dev/null +++ b/.planning/research/ARCHITECTURE.md @@ -0,0 +1,554 @@ +# Architecture Research + +**Domain:** Music-to-light synchronization system (show controller) +**Researched:** 2026-04-05 +**Confidence:** HIGH (core patterns are well-established; AI sync specifics are MEDIUM) + +## Standard Architecture + +### System Overview + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Browser Frontend │ +│ ┌──────────────┐ ┌────────────────┐ ┌──────────────────────┐ │ +│ │ Timeline UI │ │ Transport Bar │ │ Device + Show Panel │ │ +│ │ (DAW editor) │ │ (play/pause/ │ │ (registry + library)│ │ +│ │ │ │ seek/pos) │ │ │ │ +│ └──────┬───────┘ └───────┬────────┘ └──────────┬───────────┘ │ +│ │ │ │ │ +│ └──────────────────┴───────────────────────┘ │ +│ │ WebSocket (full-duplex) │ +└────────────────────────────┼────────────────────────────────────────┘ + │ +┌────────────────────────────┼────────────────────────────────────────┐ +│ FastAPI Backend │ +│ ┌─────────────────────────┴──────────────────────────────────┐ │ +│ │ WebSocket Hub │ │ +│ │ (broadcasts: playback state, position ticks, show events) │ │ +│ └───────┬─────────────────┬───────────────────────────────────┘ │ +│ │ │ │ +│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │ +│ │ Show Engine │ │ Audio Manager │ │ Show Store │ │ +│ │ (playback │ │ (MPV subprocess │ │ (JSON load/save, │ │ +│ │ scheduler, │ │ + IPC bridge) │ │ device registry) │ │ +│ │ cue runner) │ │ │ │ │ │ +│ └───────┬──────┘ └───────┬──────────┘ └────────────────────┘ │ +│ │ │ │ +│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │ +│ │ UDP Sender │ │ Analysis Engine │ │ Device Registry │ │ +│ │ (per-device │ │ (librosa beat │ │ (SK6812, WS2801, │ │ +│ │ dispatcher) │ │ detection + │ │ future: lasers, │ │ +│ │ │ │ AI sync) │ │ fog, heads) │ │ +│ └───────┬──────┘ └──────────────────┘ └────────────────────┘ │ +│ │ │ +└──────────┼────────────────────────────────────────────────────────--┘ + │ UDP (fire-and-forget, per device) + │ +┌──────────┴────────────────────────────────────────────────────────┐ +│ Physical Devices │ +│ ┌────────────────┐ ┌────────────────┐ ┌──────────────────┐ │ +│ │ SK6812 strip │ │ WS2801 strip │ │ Future: lasers │ │ +│ │ (300 LEDs, │ │ (160 LEDs, │ │ fog, heads... │ │ +│ │ wall) │ │ shelf U) │ │ │ │ +│ │ ESP/Pico/RPi │ │ ESP/Pico/RPi │ │ │ │ +│ └────────────────┘ └────────────────┘ └──────────────────┘ │ +└───────────────────────────────────────────────────────────────────-┘ +``` + +### Component Responsibilities + +| Component | Responsibility | Implementation | +|-----------|----------------|----------------| +| Timeline UI | DAW-style editor: tracks per device, drag-place animation blocks, beat markers | Vanilla JS or lightweight framework, canvas or DOM | +| Transport Bar | Play/pause/seek controls, position display, MPV state mirror | Reads WebSocket position ticks, sends commands | +| WebSocket Hub | Single persistent connection between browser and backend; broadcasts state changes and position ticks | FastAPI WebSocket endpoint + connection manager | +| Show Engine | Central coordinator: owns playback state machine, fires cues at the right timestamps, calls UDP Sender | asyncio background task, wall-clock + audio-position based scheduling | +| Audio Manager | Launches MPV as subprocess, opens IPC pipe, polls `time-pos`, translates seek/play/pause commands | python-mpv-jsonipc or direct named-pipe JSON IPC | +| Show Store | Load/save show JSON files, device registry CRUD, animation library catalog | In-process with JSON files on disk (no database needed) | +| Analysis Engine | Pre-process audio file: beat detection, onset detection, tempo map, optional AI sync | librosa offline analysis, results stored back into show JSON | +| UDP Sender | Per-device packet dispatcher: either animation+params commands OR raw DNRGB/DRGB pixel frames | asyncio UDP socket, one socket per target device | +| Device Registry | Catalog of connected devices with type, LED count, IP, port, protocol | Config in show JSON or separate devices.json | + +## Recommended Project Structure + +``` +lightsync/ +├── backend/ +│ ├── main.py # FastAPI app, route registration, lifespan +│ ├── ws/ +│ │ └── hub.py # WebSocket connection manager, broadcast +│ ├── engine/ +│ │ ├── show_engine.py # Playback state machine, cue scheduler +│ │ ├── audio_manager.py # MPV subprocess + IPC bridge +│ │ └── scheduler.py # asyncio timing loop, position tracking +│ ├── devices/ +│ │ ├── base.py # BaseDevice abstract class +│ │ ├── sk6812.py # SK6812 RGBW device driver +│ │ ├── ws2801.py # WS2801 RGB device driver +│ │ └── registry.py # Device registry + factory +│ ├── udp/ +│ │ ├── sender.py # UDP socket pool, async send +│ │ └── protocols.py # DRGB, DNRGB, DRGBW packet builders +│ ├── analysis/ +│ │ ├── beat_detector.py # librosa beat/onset analysis +│ │ └── ai_sync.py # AI sync step (optional, later phase) +│ ├── store/ +│ │ ├── show_store.py # Load/save show JSON +│ │ └── models.py # Pydantic models for show file schema +│ └── api/ +│ ├── shows.py # REST: CRUD shows, list shows +│ ├── devices.py # REST: CRUD devices +│ └── analysis.py # REST: trigger analysis run +├── frontend/ +│ ├── index.html +│ ├── style.css # Terminal aesthetic: dark, monospace +│ ├── app.js # App bootstrap, WebSocket client +│ ├── timeline.js # Timeline editor component +│ ├── transport.js # Playback controls +│ └── devices.js # Device panel +├── shows/ # Saved show JSON files (persistent volume) +├── devices.json # Device registry (persistent) +└── docker-compose.yml +``` + +### Structure Rationale + +- **engine/:** Owns all stateful runtime behavior. Nothing outside engine/ should hold playback state. +- **devices/:** Plugin point for new hardware types — add a file, register in registry.py. +- **udp/:** Separated from devices/ because packet encoding is protocol-level, not device-level. SK6812 and WS2801 both use DRGB variants; a future DMX device would add a new protocol here. +- **store/models.py:** Pydantic models serve as the schema contract for show files — validation on load, serialization on save. +- **frontend/:** Static files served by FastAPI or nginx. No build step required for v1 — keeps it simple. + +## Architectural Patterns + +### Pattern 1: Position-Driven Cue Scheduling + +**What:** The Show Engine does not use `asyncio.sleep()` for absolute timing. Instead, it maintains a high-resolution loop that polls MPV's `time-pos` and compares it against the sorted cue list. When `current_pos >= cue.timestamp`, the cue fires. + +**When to use:** Always — this is the core sync model for this system. + +**Trade-offs:** Polling MPV IPC adds a small latency (typically 10-50ms depending on poll interval). Pre-fire cues 1-2 frames early to compensate. Simpler and more reliable than trying to align asyncio timers with real audio position, which drifts on any seek or pause. + +**Example:** +```python +async def playback_loop(self): + pending_cues = sorted(self.show.cues, key=lambda c: c.timestamp) + next_cue_idx = 0 + while self.state == PlaybackState.PLAYING: + pos = await self.audio_manager.get_position() # polls MPV time-pos + while next_cue_idx < len(pending_cues): + cue = pending_cues[next_cue_idx] + if pos >= cue.timestamp - PRE_FIRE_OFFSET: + await self.udp_sender.send_cue(cue) + next_cue_idx += 1 + else: + break + await asyncio.sleep(0.005) # 5ms poll — tight enough, light enough +``` + +### Pattern 2: Device Abstraction via Abstract Base Class + +**What:** All device types inherit from `BaseDevice`, which defines the interface. The Show Engine and UDP Sender only speak to `BaseDevice` — they never check device types. + +**When to use:** Immediately — even for v1 with only two device types. The pattern costs almost nothing now and prevents messy `if device.type == "sk6812"` branches later. + +**Trade-offs:** Slight indirection overhead (negligible). Requires discipline in keeping the interface stable. + +**Example:** +```python +class BaseDevice: + def __init__(self, name: str, led_count: int, ip: str, port: int): + ... + + def encode_frame(self, pixels: list[tuple]) -> bytes: + """Encode a full pixel frame to UDP payload bytes.""" + raise NotImplementedError + + def encode_animation_cmd(self, animation: str, params: dict) -> bytes: + """Encode an animation+params command packet.""" + raise NotImplementedError + + @property + def max_leds_per_packet(self) -> int: + """Protocol limit on LEDs per UDP packet.""" + raise NotImplementedError + +class SK6812Device(BaseDevice): + """RGBW — uses DRGBW protocol (4 bytes per LED).""" + def encode_frame(self, pixels): + payload = bytearray([3, 2]) # DRGBW, 2s timeout + for r, g, b, w in pixels: + payload.extend([r, g, b, w]) + return bytes(payload) +``` + +### Pattern 3: Dual UDP Mode (Animation Command vs. Raw Frame) + +**What:** Two fundamentally different UDP payload strategies: +- **Animation mode** (manual show playback): send a short command packet — animation name ID + parameters. Microcontroller interprets and renders locally. Low bandwidth (~20 bytes/packet). +- **Frame mode** (AI-generated sequences): send the full DRGB/DNRGB pixel buffer for every frame. High bandwidth (300 LEDs * 3 bytes = 900 bytes/frame at 30fps = 27KB/s per device). + +**When to use:** Animation mode for manually-authored shows. Frame mode for AI sequences where the backend computed exact pixel values. Devices must support both — the firmware handles both packet types. + +**Trade-offs:** Frame mode requires reliable network on local LAN (generally fine). At 60fps frame mode is 54KB/s per device — still well within LAN capacity. The firmware protocol must include a type discriminator so the microcontroller knows which mode is active. + +### Pattern 4: Offline Analysis Pipeline + +**What:** Beat detection and AI sync are NOT done in real-time during playback. They are offline processing steps triggered before show editing starts. Results are stored back into the show JSON as a `analysis` section. + +**When to use:** Always — librosa analysis of a 3-minute song takes 2-10 seconds. Doing it at load time and caching the results avoids blocking playback. + +**Trade-offs:** Show file grows by ~50-200KB per analysed song (beat timestamps are compact). Analysis must be re-triggered if source audio changes. Well worth it for the performance guarantee. + +**Example flow:** +``` +User loads song + → Analysis Engine runs librosa offline + → Extracts: beat_times[], onset_times[], tempo_bpm, sections[] + → Stores in show.analysis JSON block + → Timeline UI renders beat markers from this data + → Auto-sync snaps cues to show.analysis.beat_times +``` + +## Data Flow + +### "Play This Show" — Browser Click to LED Pixel + +``` +User clicks Play in Transport Bar + │ + ▼ WebSocket message: { type: "transport", cmd: "play" } + │ + ▼ WebSocket Hub dispatches to Show Engine + │ + ▼ Show Engine sets state = PLAYING, resets cue pointer + │ + ▼ Show Engine calls Audio Manager: mpv.play() + │ + ▼ Audio Manager sends JSON IPC command to MPV process + │ + ▼ MPV starts audio output (platform audio device) + │ + ═══════════════════ Playback Loop (5ms tick) ══════════════════ + │ + ▼ Show Engine polls Audio Manager for time-pos + │ + ▼ Audio Manager reads MPV IPC: { "data": 4.237 } ← seconds + │ + ▼ Show Engine compares pos against pending cues + │ + ├─ No cue yet → continue loop + │ + └─ Cue fires: { device: "wall-strip", animation: "pulse", params: {...} } + │ + ▼ UDP Sender looks up device in Device Registry + │ + ▼ Calls device.encode_animation_cmd(animation, params) + │ + ▼ Sends UDP packet to device IP:port + │ + ▼ Microcontroller receives, renders animation on LEDs + │ + ▼ Show Engine broadcasts position tick via WebSocket Hub + │ + ▼ Browser Transport Bar updates position display (no round-trip needed) +``` + +### Seek Flow + +``` +User drags playhead to 00:42.500 + │ + ▼ WebSocket: { type: "transport", cmd: "seek", position: 42.5 } + │ + ▼ Show Engine pauses cue loop + │ + ▼ Audio Manager: mpv.seek(42.5, "absolute") + │ + ▼ Show Engine resets cue pointer to first cue >= 42.5s + │ + ▼ Show Engine broadcasts: { type: "position", pos: 42.5 } + │ + ▼ Resumes cue loop +``` + +### Analysis Flow + +``` +User triggers "Analyse Song" via UI + │ + ▼ REST POST /api/shows/{id}/analyse + │ + ▼ Analysis Engine loads audio file + │ + ▼ librosa.load() → y, sr + ▼ librosa.beat.beat_track() → tempo, beat_frames + ▼ librosa.frames_to_time(beat_frames) → beat_times[] + ▼ librosa.onset.onset_detect() → onset_times[] + │ + ▼ Results stored in show.analysis JSON block (Show Store.save()) + │ + ▼ WebSocket broadcast: { type: "analysis_complete", showId: "..." } + │ + ▼ Timeline UI redraws beat markers +``` + +### WebSocket vs. Polling + +**Use WebSocket for:** +- Position ticks during playback (backend pushes ~10Hz, browser just renders) +- State changes: play/pause/stop/seek events +- Analysis completion notifications +- Cue fire events (for visual feedback in timeline during playback) + +**Use REST for:** +- Show CRUD (load/save/list) +- Device registry CRUD +- Trigger analysis run +- Export show file + +**Rationale:** WebSocket is the right choice here — playback position ticks (10-30 per second) make polling impractical. A 100ms poll interval is too coarse for smooth position display; 10ms polling hammers the backend with HTTP overhead. One persistent WebSocket eliminates all of that. REST handles all the "slow path" operations (file I/O, config changes) where request-response semantics are appropriate. + +## Show File JSON Schema + +### Design Principles + +- Human-readable and hand-editable +- Self-contained: a show file is everything needed to replay a show +- Stable: adding new animation types or device types doesn't require schema migration +- Analysis data is embedded, not a sidecar file + +### Schema + +```json +{ + "version": 1, + "id": "uuid-v4", + "name": "Party Night Vol. 1", + "created_at": "2026-04-05T20:00:00Z", + "updated_at": "2026-04-05T22:30:00Z", + + "audio": { + "source_type": "file", + "path": "/shows/audio/party_night.mp3", + "duration_seconds": 214.3, + "yt_url": null + }, + + "devices": [ + { + "id": "wall-strip", + "name": "Wall LED Strip", + "type": "sk6812", + "led_count": 300, + "ip": "192.168.1.101", + "port": 21324 + }, + { + "id": "shelf-strip", + "name": "Shelf U-Turn", + "type": "ws2801", + "led_count": 160, + "ip": "192.168.1.102", + "port": 21324 + } + ], + + "analysis": { + "analysed_at": "2026-04-05T20:15:00Z", + "tempo_bpm": 128.0, + "beat_times": [0.47, 0.94, 1.41, 1.88], + "onset_times": [0.12, 0.47, 0.83, 1.19], + "sections": [ + { "start": 0.0, "end": 32.0, "label": "intro" }, + { "start": 32.0, "end": 96.0, "label": "verse" } + ] + }, + + "tracks": [ + { + "device_id": "wall-strip", + "cues": [ + { + "id": "cue-001", + "timestamp": 0.47, + "mode": "animation", + "animation": "pulse", + "params": { + "color": [255, 80, 0], + "speed": 1.5, + "length": 2.0 + } + }, + { + "id": "cue-002", + "timestamp": 32.0, + "mode": "animation", + "animation": "chase", + "params": { + "color_a": [255, 0, 100], + "color_b": [0, 0, 255], + "speed": 2.0 + } + } + ] + }, + { + "device_id": "shelf-strip", + "cues": [ + { + "id": "cue-003", + "timestamp": 0.0, + "mode": "animation", + "animation": "static", + "params": { "color": [30, 0, 60] } + } + ] + } + ], + + "ai_sequences": [] +} +``` + +**Key design decisions:** +- `mode: "animation"` vs `mode: "frame_sequence"` — the discriminator for UDP strategy +- `ai_sequences` is a top-level array separate from tracks — AI-generated pixel sequences are stored as frame data referenced here, not inline in cues (they'd be too large) +- `analysis` block is optional — show works without it (no beat markers, no auto-snap) +- `devices` embedded in show file — a show is portable without a separate device config + +## Anti-Patterns + +### Anti-Pattern 1: asyncio.sleep() for Cue Timing + +**What people do:** Schedule cues with `asyncio.create_task(send_cue_after(cue, delay_seconds))`. + +**Why it's wrong:** asyncio event loop can be delayed by other tasks. A 200ms blocking call elsewhere pushes every `sleep` timer by 200ms. More critically: if the user seeks or pauses, all pre-scheduled timers are now wrong and must be cancelled and rescheduled. + +**Do this instead:** Position-driven scheduling — poll audio position, fire cues when position crosses their timestamp. Handles seeks and pauses automatically with zero extra logic. + +### Anti-Pattern 2: Blocking MPV IPC Calls in the Async Loop + +**What people do:** Use `socket.recv()` (blocking) to read MPV IPC responses inside an asyncio coroutine. + +**Why it's wrong:** Blocks the entire event loop. WebSocket messages stop processing, UDP sends are delayed, position ticks stall. + +**Do this instead:** Wrap all MPV IPC communication in `asyncio.to_thread()` or use the non-blocking version of the IPC library. Keep the event loop non-blocking throughout. + +### Anti-Pattern 3: Per-Frame WebSocket Position Updates + +**What people do:** Broadcast audio position on every 5ms tick to the browser. + +**Why it's wrong:** At 200 position updates/second, WebSocket bandwidth is dominated by tiny position messages. Browser render loop can't use 200Hz position data anyway — it renders at 60fps. + +**Do this instead:** Send position ticks at 10-15Hz. Browser extrapolates between ticks using `performance.now()` for smooth display. Backend position is authoritative; browser position is display-only interpolation. + +### Anti-Pattern 4: One Giant Show Engine Class + +**What people do:** Put all playback logic — MPV control, UDP sending, analysis, WebSocket broadcasting — into a single class. + +**Why it's wrong:** Untestable. Audio Manager changes break cue scheduling tests. UDP protocol changes ripple everywhere. + +**Do this instead:** Each subsystem is its own class with a clean interface. Show Engine orchestrates them but doesn't own their internals. Test each in isolation. + +### Anti-Pattern 5: Storing Pixel Frame Data Inline in Show JSON + +**What people do:** Store AI-generated pixel sequences as base64 blobs inside the show JSON. + +**Why it's wrong:** A 3-minute song at 30fps * 300 LEDs * 3 bytes = 48MB of frame data — inline in a JSON file meant to be edited. + +**Do this instead:** Store a reference path in `ai_sequences`. Frame data lives in a separate binary file (e.g., `.fseq` compatible or custom binary). The show JSON stays human-readable and under 100KB. + +## Integration Points + +### External Services + +| Service | Integration Pattern | Notes | +|---------|---------------------|-------| +| MPV | Subprocess + JSON IPC over named pipe (Windows) or Unix socket | python-mpv-jsonipc handles the pipe; key properties: `time-pos`, `duration`, `pause` | +| yt-dlp | MPV handles this natively via `--ytdl` flag — no direct Python integration needed | MPV passes YouTube URLs to yt-dlp internally | +| librosa | In-process Python call, runs in thread pool to avoid blocking event loop | `asyncio.to_thread(run_analysis, audio_path)` | + +### Internal Boundaries + +| Boundary | Communication | Notes | +|----------|---------------|-------| +| Show Engine ↔ Audio Manager | Async method calls (get_position, play, pause, seek) | Audio Manager is a dependency-injected service | +| Show Engine ↔ UDP Sender | Async method calls (send_cue, send_frame) | UDP Sender takes a BaseDevice + payload | +| Show Engine ↔ WebSocket Hub | Hub is passed to engine; engine calls hub.broadcast() | One-directional push; hub doesn't call engine | +| WebSocket Hub ↔ Browser | Full-duplex WebSocket; browser sends commands, backend broadcasts state | All browser commands go through a message dispatcher | +| Analysis Engine ↔ Show Store | Analysis returns a dict; Show Store merges into show JSON and saves | Loose coupling — analysis doesn't know about the store | + +## Suggested Build Order + +Dependencies flow bottom-up. Each layer can be built and tested before the next. + +``` +Phase 1 — Foundation (no hardware needed) +├── Show Store: Pydantic models, load/save, device registry +├── FastAPI skeleton: app setup, static file serving, REST stubs +└── Frontend skeleton: HTML/CSS terminal aesthetic, WebSocket client stub + +Phase 2 — Audio + Transport +├── Audio Manager: MPV subprocess, IPC bridge, position polling +├── Show Engine: state machine, position-driven playback loop +├── WebSocket Hub: connection manager, broadcast +└── Transport Bar: play/pause/seek UI connected to WebSocket + +Phase 3 — Device + UDP +├── BaseDevice + SK6812 + WS2801 implementations +├── UDP packet builders (DRGB, DRGBW, DNRGB) +├── UDP Sender: async socket pool +└── First end-to-end test: show fires cues → UDP packets → (simulator or device) + +Phase 4 — Timeline Editor +├── Timeline UI: tracks, cue blocks, drag-place, beat markers +├── Analysis Engine: librosa beat detection, results in show JSON +└── Auto-snap cues to beats + +Phase 5 — AI Sync +├── AI analysis step (on top of beat detection) +├── Frame sequence storage + playback +└── Frame-mode UDP sending + +Phase 6 — Microcontroller Firmware +├── Protocol is stable by now — implement against spec +├── MicroPython on ESP/Pico or Python on RPi +└── Both DRGB/DRGBW (frame mode) and animation command mode +``` + +**Why this order:** + +1. Show Store first — everything else depends on the data model. Defining the schema early prevents model changes from cascading. +2. Audio + Transport second — this is the critical path. Without reliable audio position tracking, nothing else can be timed. +3. Device + UDP third — the output layer. Can be tested with a simulator (a Python script that prints received packets) before firmware exists. +4. Timeline fourth — the editing UX. Depends on analysis data and the show file schema being stable. +5. AI sync fifth — optional enhancement layer, builds on beat detection. +6. Firmware last — the protocol is fully validated against a software simulator before burning anything to hardware. + +## Scaling Considerations + +This is a single-operator local system. Scaling is not a concern. The relevant "scale" questions are latency and reliability, not throughput. + +| Concern | Target | Approach | +|---------|--------|----------| +| Cue timing precision | ±20ms | Position polling at 5-10ms intervals, pre-fire offset | +| UDP latency | <5ms on LAN | Fire-and-forget UDP, no ACK waiting | +| WebSocket position tick | 10-15Hz | Sufficient for smooth UI; browser interpolates | +| Analysis time | <10s for 5min song | Offline, runs in thread pool, results cached in show JSON | +| Frame-mode bandwidth | ~27KB/s per device at 30fps (300 LEDs) | Well within LAN; 5 devices = 135KB/s, trivial | + +## Sources + +- [WLED UDP Realtime Protocol](https://kno.wled.ge/interfaces/udp-realtime/) — DRGB/DNRGB/DRGBW packet format (HIGH confidence — official documentation) +- [python-mpv-jsonipc](https://github.com/iwalton3/python-mpv-jsonipc) — IPC bridge library for MPV subprocess (HIGH confidence — active project, used by Jellyfin) +- [MPV JSON IPC Protocol](https://mpv.io/manual/master/) — `time-pos`, `playback-time`, seek commands (HIGH confidence — official docs) +- [librosa beat_track](https://librosa.org/doc/main/generated/librosa.beat.beat_track.html) — returns beat frame indices, convertible to timestamps via `frames_to_time` (HIGH confidence — official docs) +- [FastAPI WebSocket background tasks](https://hexshift.medium.com/implementing-background-tasks-with-websockets-in-fastapi-034cdf803430) — asyncio.create_task pattern alongside websocket receive loop (MEDIUM confidence — community article) +- [audio-reactive-led-strip](https://github.com/scottlawsonbc/audio-reactive-led-strip) — reference Python + UDP + ESP8266 architecture (MEDIUM confidence — older project but patterns are standard) +- [xLights FSEQ format](https://github.com/Cryptkeeper/fseq-file-format) — reference for binary frame sequence format (LOW confidence — using as inspiration only, not adopting format) + +--- +*Architecture research for: LightSync — music-to-light synchronization system* +*Researched: 2026-04-05* diff --git a/.planning/research/FEATURES.md b/.planning/research/FEATURES.md new file mode 100644 index 0000000..028d88a --- /dev/null +++ b/.planning/research/FEATURES.md @@ -0,0 +1,231 @@ +# Feature Research + +**Domain:** Music-to-light synchronization system (personal LED show controller) +**Researched:** 2026-04-05 +**Confidence:** HIGH (cross-referenced across QLC+, xLights, SoundSwitch, Lightjams, WLED, Resolume, and Python/librosa ecosystem) + +--- + +## Feature Landscape + +### Table Stakes (Users Expect These) + +Features that any music-synchronized light controller must have to feel complete. Missing +these causes the whole system to feel broken or amateur. + +| Feature | Why Expected | Complexity | Notes | +|---------|--------------|------------|-------| +| Audio file loading + playback | You cannot sync to music without playing it | LOW | MPV subprocess handles this; need play/pause/seek/position | +| Waveform display | Every sequencer since the 90s shows the waveform; without it you're editing blind | MEDIUM | Render audio waveform as visual reference in timeline; requires audio decoding to samples | +| Timeline with per-device tracks | The core editing metaphor — every show editor (xLights, VenueMagic, QLC+ Show Manager) uses it | HIGH | Each device gets a horizontal track; effects are blocks placed on the track | +| Audio playback position cursor | Moving vertical line on timeline showing current playback position | LOW | Follows MPV position; drives what's rendered live | +| Effect blocks on timeline | Drag-and-place blocks representing an animation + its params | MEDIUM | Block = animation type + duration + params; resize/move/delete | +| Animation library | Preset animations to apply (chase, pulse, rainbow, strobe, etc.) | MEDIUM | Users pick from a list, configure per-block params (speed, color, direction) | +| Beat detection / timing marks | All major tools (xLights via VAMP, SoundSwitch, Lightjams) treat this as core | MEDIUM | librosa beat_track + onset_detect; results shown as marks on timeline | +| Snap-to-beat | Effects placed/resized should snap to detected beat marks | LOW | Once timing marks exist, snapping is a positioning assist; low complexity | +| Play a show live | The whole point — execute the timeline and send frames/commands to devices | MEDIUM | Follows playback position, dispatches UDP packets at correct timestamps | +| Device registry | Name, type, LED count, IP, port — without this, you can't address anything | LOW | Static config; a JSON file or DB table | +| Show save / load | Persist your work as JSON so you don't lose it between sessions | LOW | Timeline state → JSON; all sequencers support this | +| Color picker per effect block | Choose colors for an animation block directly in the UI | LOW | Standard HTML color input; feeds into animation params | +| Basic undo/redo | Lightkey markets this as a selling point because so many tools lack it | MEDIUM | Command pattern on timeline mutations; minimal viable: last 20 actions | + +### Differentiators (Competitive Advantage) + +Features that elevate the tool above the baseline. These align with the LightSync core +value: precise control + automated intelligence. + +| Feature | Value Proposition | Complexity | Notes | +|---------|-------------------|------------|-------| +| YouTube URL loading via yt-dlp | No other personal tool does this cleanly; commercial tools don't touch it | LOW | Subprocess yt-dlp + MPV pipe; already decided in PROJECT.md | +| Structural segmentation overlay | librosa can detect verse/chorus/bridge boundaries — display as colored regions on waveform | HIGH | librosa.segment + recurrence matrix; surfaces song structure for manual cue placement | +| AI-assisted show generation | Given beat/segment analysis, auto-fill a timeline with matching animations | HIGH | Post-processing step on top of beat detection; requires designing a mapping heuristic or LLM prompt | +| Dual UDP protocol (command vs frames) | Low-bandwidth animation commands for simple effects; raw RGBW frames for complex AI sequences | MEDIUM | Two encoders: one sends (animation_id, params), other sends per-frame pixel arrays | +| Live preview panel | Show a simulated LED strip rendering in the browser as playback runs | MEDIUM | Canvas/WebGL; render animation frames based on current timeline state without needing hardware | +| Parameter animation curves | Let effect params (speed, brightness) vary over the block's duration using envelope curves | HIGH | Think DAW automation lanes; very powerful but complex to implement and edit in UI | +| Frequency band mapping | Map bass / mid / treble energy to different devices or effect params | HIGH | librosa spectral analysis per frame; enables bass-driven floor lights, treble-driven ceiling flicker | +| Effect grouping / scenes | Group multiple device blocks into a named scene, re-use it across the timeline | MEDIUM | Reduces repetition when building multi-device shows | +| Show export for standalone playback | Export a show as a self-contained playback file + firmware-flashable sequence | HIGH | Firmware deferred; but JSON export format should support offline playback eventually | +| Terminal/hacker aesthetic UI | Differentiates visually from every off-the-shelf DMX tool | LOW | CSS/font work; monospace, dark theme, CRT vibes — already decided in PROJECT.md | + +### Anti-Features (Commonly Requested, Often Problematic) + +Explicitly what NOT to build for a personal single-operator tool. + +| Feature | Why Requested | Why Problematic | Alternative | +|---------|---------------|-----------------|-------------| +| Real-time microphone input | Feels cool, "reactive" | Adds latency-sensitive audio pipeline, conflicts with file-based sync model, requires AGC tuning | Stay file/URL based; deterministic playback always wins for show editing | +| Multi-user collaborative editing | "What if a friend wants to help?" | Requires conflict resolution, WebSocket state sync, presence — enormous complexity for zero personal-use value | Single-operator; share the JSON file if collaboration is needed | +| Cloud sync / remote access | "Access from phone" | Adds auth, hosted backend, security surface — out of scope per PROJECT.md | Local network only; Traefik + Authelia provides access from the local network | +| MIDI controller hardware support | DJs use it; feels pro | Hardware dependency, OS driver layer, latency calibration — high complexity for a personal show builder | Beat detection handles what MIDI would be used for in this context | +| Plugin / extension system | "Make it extensible" | Proper plugin APIs are complex to design well; versioning, sandboxing, discovery | Codebase is open; add effects directly in Python. Modular device abstraction covers extensibility | +| 3D fixture positioning / visualizer | Professional VJs use Resolume Arena / L8 for this | Massive scope; requires a 3D renderer, fixture database, spatial math | 2D live preview panel is sufficient for LED strips | +| DMX universe / ArtNet / E1.31 | Professional standard; QLC+ and xLights support it | Adds protocol translation layer; LightSync uses direct UDP to microcontrollers — simpler and lower latency | Custom UDP protocol is more direct and lower overhead for this scale | +| Scheduling / playlist / show runner | Light-O-Rama and VenueMagic have this | Automated playlists make sense for installation art / Christmas shows, not interactive personal shows | Manual launch from the web UI is the right model here | +| MIDI clock sync with DAW | Ableton Link, Lightkey advertise this | No DAW in this workflow; adds dependency on external BPM source | librosa beat detection is fully offline and self-contained | + +--- + +## Feature Dependencies + +``` +Audio Playback (MPV subprocess) + └──enables──> Waveform Display + └──enables──> Beat Detection / Timing Marks + └──enables──> Snap-to-Beat + └──enables──> AI-Assisted Show Generation + └──enables──> Structural Segmentation Overlay + +Device Registry + └──required by──> Live Show Execution (UDP dispatch) + └──required by──> Live Preview Panel + +Timeline Editor (tracks + effect blocks) + └──requires──> Animation Library (something to place) + └──requires──> Device Registry (what to assign blocks to) + └──enables──> Show Save/Load + └──enables──> AI-Assisted Show Generation (fills the timeline) + +Animation Library + └──enables──> Effect Blocks on Timeline + └──enhanced by──> Color Picker per Block + └──enhanced by──> Parameter Animation Curves + +Live Show Execution + └──requires──> Timeline (what to execute) + └──requires──> Device Registry (where to send) + └──requires──> Audio Playback (position reference) + └──enhanced by──> Dual UDP Protocol (command vs frames) + +Live Preview Panel + └──requires──> Animation Library (rendering logic) + └──requires──> Device Registry (LED counts / layout) + └──enhanced by──> Live Show Execution (synced preview) + +Beat Detection + └──enhances──> Timeline Editor (snap grid) + └──enhances──> AI Show Generation (cue placement) + +Frequency Band Mapping + └──requires──> Beat Detection (same audio analysis pass) + └──conflicts with──> Simple show model (adds per-frame complexity) +``` + +### Dependency Notes + +- **Beat detection requires audio playback to be loaded first** — analysis runs on the audio file at load or on-demand; cannot run without the file. +- **AI show generation requires beat detection** — it uses timing marks as anchor points; generating without them produces incoherent results. +- **Live show execution requires device registry** — without knowing IP/port/LED count, UDP packets can't be formed. +- **Live preview panel requires animation rendering logic** — the same code that renders frames for UDP output should power the preview; they must share a renderer. +- **Dual UDP protocol requires clear protocol spec first** — command mode and frame mode must be designed together before implementing either. +- **Undo/redo must be designed early** — retrofitting undo into a stateful timeline editor is painful; build the command pattern from the first timeline operation. + +--- + +## MVP Definition + +### Launch With (v1) + +Minimum viable for the system to be useful at all. + +- [ ] Audio file loading + MPV playback (play/pause/seek) — without this nothing works +- [ ] Waveform display — required for editing without guessing +- [ ] Beat detection via librosa — auto-generates timing marks on the timeline +- [ ] Device registry (static JSON config) — at least 2 devices (SK6812 + WS2801) +- [ ] Animation library with 8-12 core effects (rainbow, chase, pulse, strobe, color fill, sparkle, fade in/out, wipe) — minimum useful palette +- [ ] Timeline editor: per-device tracks, drag-and-place effect blocks, snap-to-beat +- [ ] Color picker + basic params (speed, brightness, direction) per effect block +- [ ] Show save / load as JSON +- [ ] UDP command sender — dispatch animation + params at the right timestamp +- [ ] Undo/redo for timeline mutations — build from day one or regret it + +### Add After Validation (v1.x) + +Add once the core editing loop works and you've built a real show with it. + +- [ ] YouTube URL loading via yt-dlp — once local-file workflow is proven +- [ ] Live preview panel — valuable once you have a library of shows to tweak +- [ ] Structural segmentation overlay — once beat marks aren't enough +- [ ] UDP frame sender (raw pixel data) — needed for AI-generated sequences +- [ ] AI-assisted show generation — on top of the raw frame pipeline +- [ ] Effect grouping / scenes — once shows are complex enough to need it + +### Future Consideration (v2+) + +Defer until v1 is regularly used and the pain points are clear. + +- [ ] Parameter animation curves — powerful but high editor complexity; wait until basic params feel limiting +- [ ] Frequency band mapping — adds real-time analysis path; revisit when advanced reactive effects are wanted +- [ ] Moving heads / laser support — requires extending device abstraction; build only when hardware exists +- [ ] Fog machine control — simple on/off trigger; easy to add when hardware is present + +--- + +## Feature Prioritization Matrix + +| Feature | User Value | Implementation Cost | Priority | +|---------|------------|---------------------|----------| +| Audio playback (MPV) | HIGH | LOW | P1 | +| Waveform display | HIGH | MEDIUM | P1 | +| Beat detection (librosa) | HIGH | MEDIUM | P1 | +| Timeline editor | HIGH | HIGH | P1 | +| Animation library (core 8-12) | HIGH | MEDIUM | P1 | +| Device registry | HIGH | LOW | P1 | +| Show save/load | HIGH | LOW | P1 | +| UDP command sender | HIGH | MEDIUM | P1 | +| Undo/redo | MEDIUM | MEDIUM | P1 | +| Snap-to-beat | HIGH | LOW | P1 | +| Color picker per block | MEDIUM | LOW | P1 | +| YouTube loading (yt-dlp) | HIGH | LOW | P2 | +| Live preview panel | HIGH | MEDIUM | P2 | +| Structural segmentation overlay | MEDIUM | HIGH | P2 | +| UDP frame sender | MEDIUM | MEDIUM | P2 | +| AI show generation | HIGH | HIGH | P2 | +| Effect grouping / scenes | MEDIUM | MEDIUM | P2 | +| Parameter animation curves | MEDIUM | HIGH | P3 | +| Frequency band mapping | MEDIUM | HIGH | P3 | +| Moving head / laser support | LOW | MEDIUM | P3 | + +**Priority key:** +- P1: Must have for launch +- P2: Should have, add after core is validated +- P3: Nice to have, future consideration + +--- + +## Competitor Feature Analysis + +| Feature | xLights | QLC+ | SoundSwitch | LightSync Approach | +|---------|---------|------|-------------|-------------------| +| Timeline editor | Yes — DAW-style, per-model tracks | Yes — Show Manager | No (real-time only) | Yes — per-device tracks, effect blocks | +| Beat detection | Yes — VAMP plugins | No built-in | Yes — beatgrid sync | Yes — librosa (offline, no plugin install) | +| Audio waveform | Yes — sonographic display | No | N/A | Yes — rendered from audio file | +| Snap to beat | Yes — snaps to timing marks | N/A | N/A | Yes — on placement + resize | +| Animation library | Yes — 100+ effects | Yes — scenes/functions | Yes — autoloops (32 built-in) | Yes — curated set (~12 core, extensible) | +| AI show generation | No | No | Phrase detection (limited) | Yes — planned differentiator | +| Structural segmentation | No | No | No | Yes — librosa segment analysis | +| Live preview | Yes — 2D/3D layout | Yes — fixture map | No | Yes — canvas strip simulation | +| Custom protocol | E1.31 / ArtNet / DDP | DMX / ArtNet | MIDI + DMX | Custom UDP (command + frame modes) | +| YouTube support | No | No | No | Yes — yt-dlp passthrough | +| Undo/redo | Partial (some operations) | Limited | N/A | Yes — command pattern from day 1 | +| Personal tool scale | Overkill (100+ universes) | Overkill (full DMX) | Wrong use case (DJ) | Right scale — 2 strips + future devices | + +--- + +## Sources + +- [QLC+ Features page](https://www.qlcplus.org/discover/features) — DMX feature set, Show Manager, RGB Matrix +- [xLights Manual — Timeline and Waveform](https://manual.xlights.org/xlights/chapters/chapter-four-sequencer/timeline-and-waveform) — zoom, playback, waveform display +- [xLights Manual — Timing Tracks](https://manual.xlights.org/xlights/chapters/chapter-four-sequencer/timing-tracks) — VAMP plugins, beat detection, snap behavior +- [SoundSwitch — What's New in 2.9](https://support.soundswitch.com/en/support/solutions/articles/69000858481-soundswitch-what-s-new-in-soundswitch-2-9-) — BPM detection, phrase detection, autoloops +- [WLED Audio Reactive documentation](https://kno.wled.ge/advanced/audio-reactive/) — 30+ sound-reactive effects, audio input options +- [Lightjams — Sync Music and Lights](https://www.lightjams.com/musicToDMX.html) — 120 frequency bands, beat/mid/treble energy +- [Resolume Avenue/Arena](https://resolume.com/software/avenue-arena) — FFT audio analysis, BPM sync, LED mapping +- [librosa documentation](https://librosa.org/doc/0.11.0/tutorial.html) — beat tracking, onset detection, structural segmentation +- [WLED UDP Realtime Control](https://github.com/Aircoookie/WLED/wiki/UDP-Realtime-Control) — protocol patterns, FPS guidance (15-30 recommended) +- [VenueMagic Show Control](https://www.venuemagic.com/show-control-software/) — drag-drop timeline, beat tapper +- [DMXDesktop Effect Types](https://www.dmxdesktop.com/knowledgebase/effect-types) — chase, sparkle, pulse, strobe definitions + +--- + +*Feature research for: music-to-light synchronization system (LightSync)* +*Researched: 2026-04-05* diff --git a/.planning/research/PITFALLS.md b/.planning/research/PITFALLS.md new file mode 100644 index 0000000..6632d73 --- /dev/null +++ b/.planning/research/PITFALLS.md @@ -0,0 +1,377 @@ +# Pitfalls Research + +**Domain:** Music-to-light synchronization system (Python backend, UDP LED control, web UI, MPV audio) +**Researched:** 2026-04-05 +**Confidence:** MEDIUM — Core timing and protocol pitfalls are HIGH confidence from official docs and community reports; genre-specific beat detection accuracy is MEDIUM from GitHub issues; some Windows-specific timing claims are LOW confidence from limited benchmarks. + +--- + +## Critical Pitfalls + +### Pitfall 1: Audio Clock vs Wall Clock Drift + +**What goes wrong:** +The backend polls MPV's playback position over IPC (named pipe on Windows) and uses that value to schedule UDP packets. The polling loop runs on Python's event loop, which has its own cadence. Over a 4-minute song, even a 10ms per-poll drift compounds into visible desync — lights ahead or behind the music by hundreds of milliseconds. + +**Why it happens:** +Two clocks that are never formally synchronized: MPV's internal audio clock (tied to the audio hardware buffer) and Python's event loop clock (tied to OS scheduling). `asyncio.sleep()` on Windows is notoriously imprecise — the Windows OS timer resolution defaults to 15.6ms unless explicitly set to 1ms via `timeBeginPeriod`. Polling at 50Hz with 15ms jitter per tick means position reads that are consistently stale or erratic. + +**How to avoid:** +- Use MPV's `playback-time` property via IPC as the authoritative clock — never derive position from wall clock elapsed time. +- Read MPV position on every scheduling tick, not just once at playback start. +- Add a configurable global offset (in ms) to compensate for known LED pipeline latency — expose this in the UI as a sync calibration control. +- Do not use `asyncio.sleep()` for the tight scheduling loop — use a dedicated `threading.Thread` with `time.perf_counter()` for sub-millisecond resolution. +- On Windows, call `timeBeginPeriod(1)` at startup (via `ctypes`) to raise the OS timer resolution from 15.6ms to 1ms. + +**Warning signs:** +- Lights consistently lag or lead the beat by a fixed offset (fixed offset = pipeline latency, fixable with calibration offset). +- Offset grows over song duration rather than staying constant (growing drift = clock mismatch, not just latency). +- Position reads from MPV IPC have jitter larger than ~5ms on a lightly loaded system. + +**Phase to address:** +Phase where MPV playback loop is first implemented. The calibration offset UI should be part of the same phase — not deferred. + +--- + +### Pitfall 2: MPV IPC Named Pipe Blocking on Windows + +**What goes wrong:** +On Windows, MPV IPC uses a named pipe (not a Unix socket). Named pipe reads are blocking by default. If the scheduling loop queries MPV position synchronously on the same thread that sends UDP packets, one slow IPC response stalls the entire send loop. At 30fps, you have 33ms per frame — a single blocked read can consume the entire budget. + +**Why it happens:** +Linux MPV IPC uses Unix domain sockets, which are well-supported in Python's `asyncio`. Windows named pipes require either overlapped I/O or a separate thread. The `python-mpv-jsonipc` library handles this but its response latency on Windows is not documented and varies with system load. Developers port code from Linux and assume the same behavior. + +**How to avoid:** +- Run the MPV IPC reader in a dedicated daemon thread, storing the last known position in a shared variable protected by a `threading.Lock` (or use `threading.Event`). +- The UDP sender thread reads from this shared variable — it never blocks on IPC. +- Use `python-mpv-jsonipc` rather than raw pipe I/O to avoid Windows-specific overlapped I/O complexity. +- If using `python-mpv` (C binding via libmpv), prefer the event-based callback API over property polling. + +**Warning signs:** +- Intermittent UDP packet bursts (multiple frames sent in rapid succession, then silence) — sign that the send loop is getting stalled and catching up. +- IPC response times that vary from 1ms to 50ms+ under load. + +**Phase to address:** +Phase where MPV integration is built. Test on Windows explicitly — do not assume Linux behavior transfers. + +--- + +### Pitfall 3: librosa Beat Detection Returns Wrong Timestamps + +**What goes wrong:** +`librosa.beat.beat_track()` returns beat timestamps that are consistently 20–60ms late relative to the actual musical onset. Additionally, it estimates a single global tempo and may place beats at rhythmically correct but perceptually wrong positions, especially near the start and end of a song. Show files built on these timestamps produce lights that trigger visibly after the beat. + +**Why it happens:** +The beat tracker reports beats at onset peaks (the spectral flux maximum), not at the perceptual beat moment (the onset itself). This is a documented design behavior, not a bug. The function also smooths aggressively over the first and last few beats to avoid tempo estimation errors — these boundary beats are often the most visible ones in a show. + +**How to avoid:** +- After running `beat_track()`, apply a fixed backward shift of 20–40ms to all returned timestamps (tune by ear against the actual track). +- Cross-validate with `librosa.onset.onset_detect()` for the first 4 beats — if they disagree significantly, trust onset detection for downbeat placement. +- Expose beat offsets as editable in the timeline — treat librosa output as a starting suggestion, not ground truth. +- For tempo-varying music (see Pitfall 4), use `librosa.beat.plp()` (predominant local pulse) which handles tempo drift better than `beat_track`. + +**Warning signs:** +- Lights fire consistently after the kick drum in EDM tracks. +- Beat positions reported by librosa do not align with any visible transient in a waveform view. +- First and last beat in the array seem off while middle beats seem accurate. + +**Phase to address:** +Beat detection analysis phase. The backward-shift calibration should be validated with at least 3 different genres before shipping. + +--- + +### Pitfall 4: Beat Tracking Fails on Jazz, Classical, and Free-Tempo Music + +**What goes wrong:** +`librosa.beat.beat_track()` assumes approximately constant tempo. On jazz recordings with swing feel, accelerating classical passages, or tracks with breaks and tempo changes, the tracker either locks to the wrong subdivision (halftime/doubletime error) or produces rhythmically incorrect beats that cannot be fixed by a simple offset. + +**Why it happens:** +The algorithm uses dynamic programming that optimizes for tempo consistency over the whole track. A section that runs 5 BPM slower forces the entire track's grid to compromise. The default prior also strongly favors 120 BPM, biasing detection on sparse or ambiguous tracks. + +**How to avoid:** +- For known tempo-variable music, use `librosa.beat.beat_track(units='time', start_bpm=, tightness=50)` — lower tightness allows more tempo variation. +- Use `librosa.beat.plp()` to get a per-frame pulse probability, then find peaks — this handles tempo variation far better than `beat_track`. +- Offer a "manual beat tap" mode in the UI where the user can define beats at playback time and override the detected grid. +- For classical and jazz content, onset detection (`librosa.onset.onset_detect`) with hand-picked onsets is more useful than beat tracking. + +**Warning signs:** +- Detected tempo is exactly half or double what the music sounds like (halftime/doubletime error). +- Beat grid looks correct for the first 30 seconds, then drifts noticeably. +- High-energy tracks with clear drums produce good results, but slow tracks with soft attacks do not. + +**Phase to address:** +Beat detection phase. Implement `plp()` alongside `beat_track()` from the start — not as a later upgrade. + +--- + +### Pitfall 5: UDP Packet Loss Causes Show Corruption, Not Graceful Degradation + +**What goes wrong:** +UDP is fire-and-forget. If a frame packet is dropped (WiFi interference, microcontroller buffer overflow, OS socket buffer overflow), the LED strip holds the previous frame indefinitely. In a fast animation, a dropped frame is invisible. In a slow fade or color hold, a dropped frame means the strip stays on the wrong color for the entire hold duration — visually obvious corruption. + +**Why it happens:** +Developers test over localhost (no drops) or on idle WiFi (no congestion). Production WiFi with other devices, microwave interference, or microcontroller processing lag causes sporadic drops. The microcontroller does not request retransmission — it simply never receives the packet. + +**How to avoid:** +- Implement sequence numbers in the packet header. The microcontroller firmware checks for gaps and logs them (visible in the UI during testing). +- For slow animations (fades, color holds), re-send the same frame 2–3 times with identical sequence numbers. Microcontroller deduplicates by sequence number — no visual effect, but drop resilience increases significantly. +- Add a "heartbeat" packet: if no packet is received within 500ms, microcontroller fades to black (fail-safe) rather than holding the last state. +- Test with intentional packet loss (Linux `tc netem`, or simply run over a congested WiFi network) before marking the communication phase complete. + +**Warning signs:** +- Show looks perfect on localhost simulator but has occasional glitches on real hardware. +- LED strip "freezes" on a color mid-animation. +- Packet loss only appears during high-bandwidth phases (many LEDs, high frame rate). + +**Phase to address:** +Communication protocol phase. The sequence number and retransmit strategy must be in the protocol spec before firmware is written — retrofitting it later requires firmware updates to all devices. + +--- + +### Pitfall 6: MTU Fragmentation for Large LED Strips + +**What goes wrong:** +The SK6812 strip has 300 LEDs at 4 bytes/pixel (RGBW) = 1200 bytes of pixel data per frame. Combined with packet header overhead, this exceeds the WiFi MTU of ~1470 bytes if any framing is added. IP fragmentation silently occurs — the OS splits the packet, both fragments must arrive, and reassembly latency adds jitter. Fragment drops (which are more common than whole-packet drops) cause total frame loss. + +**Why it happens:** +Developers calculate 300 × 4 = 1200 bytes and conclude "that fits." They don't account for packet headers (UDP: 8 bytes, IP: 20 bytes, Ethernet: ~18 bytes) plus any custom protocol header. They also don't test the specific WiFi link's effective MTU, which may be lower than 1500 due to WPA encryption overhead (~8 bytes) or VPN tunnels. + +**How to avoid:** +- For 300 RGBW LEDs: use two packets with a start-index header (similar to the WLED DNRGB protocol approach). Packet 1 covers LEDs 0–169, packet 2 covers LEDs 170–299. +- Set packet boundaries to 480 RGB or 360 RGBW pixels maximum to stay comfortably under 1470 bytes. +- In the device registry, store `leds_per_packet` per device — the sender handles multi-packet frames transparently. +- Test with `ping -f -l 1472 ` (Windows) to verify the actual path MTU to each microcontroller. + +**Warning signs:** +- Full-strip animations work for half the strip only. +- Frame corruption that appears as a sharp vertical line (packets for different frame segments arriving from different frames). +- Wireshark shows IP fragmentation on the UDP stream. + +**Phase to address:** +Protocol design phase. The multi-packet frame format must be in the spec before implementing the sender. + +--- + +### Pitfall 7: Windows Firewall Blocks UDP to Microcontrollers + +**What goes wrong:** +The Python backend sends UDP to microcontroller IPs on the local network. Windows Defender Firewall prompts on first run; if the user clicks "Private network only" or if the laptop switches to a "Public" network profile (common at different locations), outbound UDP is silently blocked. The app appears to work (no errors), LEDs simply do not respond. + +**Why it happens:** +Windows treats outbound UDP differently from inbound — by default outbound is allowed, but "Public" network profiles block more aggressively. Some corporate or home routers apply "AP isolation" (clients cannot talk to each other on WiFi), which looks identical to a firewall block from the app's perspective. + +**How to avoid:** +- On startup, the backend performs a UDP reachability test to each registered device and reports status in the UI (green/red per device). +- Document the firewall rule requirement in the app's setup page: allow Python on private networks. +- Include a diagnostic command in the UI that sends a test packet and shows whether a response was received (requires firmware cooperation — the microcontroller sends a UDP ACK back to the sender on a diagnostic port). +- Warn explicitly if the Windows network profile is "Public" (detectable via `subprocess` + `netsh interface show interface`). + +**Warning signs:** +- LEDs work when the laptop is on one WiFi network but not another. +- No Python socket errors are thrown — the send succeeds silently. +- `ping` to the microcontroller works but UDP does not (ICMP and UDP have different firewall handling). + +**Phase to address:** +Communication layer phase. The startup device health check should be part of the initial integration, not an afterthought. + +--- + +### Pitfall 8: Show File Format Locked to Absolute Timestamps + +**What goes wrong:** +Show files stored with absolute second-timestamps become tightly coupled to a specific audio file and its exact tempo map. If the audio file is ever replaced (different encode, trimmed intro, different yt-dlp download) the entire show is invalidated. There is no way to "shift" a show by 2 seconds without editing every single timestamp. + +**Why it happens:** +Absolute timestamps are the obvious starting format — they map directly to MPV's playback position. Beat-relative storage (e.g., "beat 32, subdivision 2") requires a tempo map at edit time and at playback time, which feels like extra complexity. Developers defer the tempo-map integration and end up with a format they can't change without a migration. + +**How to avoid:** +- Store events in the show file with both absolute time (seconds) and beat position (beat index + subdivision) if a beat grid is available. +- Include a `source_fingerprint` field (SHA256 of first 64KB of audio) and a `generated_at_tempo` field so mismatches can be detected at load time. +- Support a "shift all events by N seconds" operation in the UI — this requires only a single loop over the event list. +- Design the JSON schema before writing any playback code. Include a `schema_version` field from day one — even if v1 and v2 are identical, the field must exist to allow future migration. + +**Warning signs:** +- Show works perfectly for one audio file but drifts immediately on a different download of the same song. +- Users cannot easily create variations of an existing show for a remixed track. +- Any tempo change requires rebuilding the show from scratch. + +**Phase to address:** +Show file design phase (before any show playback is implemented). Once shows are saved in a format, migration is painful. + +--- + +### Pitfall 9: Python GIL Starves the UDP Send Loop + +**What goes wrong:** +The beat detection analysis, JSON processing, or WebSocket broadcast runs CPU-bound Python code on the same interpreter as the UDP send loop. The GIL allows only one thread to hold the interpreter at a time. Under load, the send loop is starved — it wakes up late, misses its frame window, and sends a burst of stale packets. + +**Why it happens:** +Python threads feel like concurrency but CPU-bound code (numpy operations, beat analysis) holds the GIL for entire computation chunks. `asyncio` does not help here — it is single-threaded and cooperative. Beat analysis that takes 2 seconds in a background thread still causes GIL contention with anything else running Python bytecode. + +**How to avoid:** +- Run beat analysis in a `ProcessPoolExecutor` (separate process, own GIL), not a `ThreadPoolExecutor`. +- Keep the UDP send loop and MPV position polling in a dedicated `threading.Thread` — this is I/O-bound and releases the GIL appropriately. +- Never run numpy-heavy processing in the same process as the real-time send loop (or accept that analysis is only triggered while playback is paused). +- FastAPI's async endpoints are fine for HTTP/WebSocket — but any CPU-bound analysis must be offloaded with `asyncio.run_in_executor(pool, ...)` using a `ProcessPoolExecutor`. + +**Warning signs:** +- Send loop timing is smooth when the app is idle but irregular when the frontend is actively communicating. +- Beat analysis triggered during playback causes visible LED stutters. +- `time.perf_counter()` measurements in the send loop show consistent jitter during JSON processing. + +**Phase to address:** +Architecture phase. The process boundary between analysis and real-time playback must be established before either is implemented. + +--- + +### Pitfall 10: Frontend Audio Controls Mirror Wrong Clock + +**What goes wrong:** +The web UI displays playback position, seek controls, and a timeline cursor. If the frontend derives position from its own JavaScript clock (Date.now() since playback started), it drifts from MPV's actual position. After a seek, the frontend and backend disagree on position until the next WebSocket update. The user drags the timeline cursor to a timestamp, but the show plays from the wrong position. + +**Why it happens:** +`Date.now()` interpolation between WebSocket position updates is a natural optimization (avoids relying on slow poll updates for smooth cursor movement). But every seek, pause/resume, or buffer event invalidates the interpolated position. The backend MPV clock and the frontend JavaScript clock are never formally synchronized. + +**How to avoid:** +- The backend broadcasts MPV position updates at 10Hz via WebSocket — include a monotonic sequence number and the backend's `time.perf_counter()` timestamp alongside the MPV position. +- The frontend uses MPV position as ground truth on every update, but only interpolates between updates using the known update interval (100ms) — not wall clock elapsed time. +- After every seek command, the frontend enters a "pending" state (cursor frozen, grayed out) until the next position update confirms the seek landed. +- Never let the frontend calculate what time it "should" be — only what time it "was told" it is. + +**Warning signs:** +- Seek operations that visually snap to position but playback starts from a different point. +- Timeline cursor running ahead of or behind the waveform after a fast seek. +- Position display that looks smooth but is inconsistent with audible playback position. + +**Phase to address:** +Playback UI phase. The state contract between backend clock authority and frontend display must be explicit in the design. + +--- + +## Technical Debt Patterns + +| Shortcut | Immediate Benefit | Long-term Cost | When Acceptable | +|----------|-------------------|----------------|-----------------| +| Poll MPV position on every frame instead of event-driven | Simple to implement | IPC overhead at 30fps, latency coupling between IPC and send loop | Never — use a background reader thread from day one | +| Absolute-only timestamps in show files | Simple storage format | Shows break on audio file replacement; no "shift" operation | Only if shows are never expected to be portable or edited | +| Single asyncio event loop for everything (IPC + UDP + WebSocket) | Less code | GIL starvation, IPC blocks stall UDP sends | Never for real-time send path | +| Hard-coded device IPs in code | Faster first demo | Every test environment requires code changes | Development only — device registry must exist before first firmware test | +| Run librosa analysis synchronously in HTTP handler | Simple request/response | Blocks other requests for 2–10 seconds on a 4-minute song | Never — always offload to ProcessPoolExecutor | +| Use WebSocket polling as the "heartbeat" for LED state | One transport for everything | WebSocket reconnects create brief LED state gaps | Only if willing to implement reconnect recovery explicitly | + +--- + +## Integration Gotchas + +| Integration | Common Mistake | Correct Approach | +|-------------|----------------|------------------| +| MPV IPC on Windows | Assume Unix socket behavior; use blocking reads on named pipe | Dedicated reader thread; `python-mpv-jsonipc` for abstraction | +| MPV IPC on Windows | Query `time-pos` (may return null when paused) | Query `playback-time` — it returns the last valid position even when paused | +| librosa.load() | Load at native sample rate (`sr=None`) for consistency | Always resample to 22050 Hz (`sr=22050`) — beat tracking is tuned to this rate; native SR breaks tempo estimation | +| librosa.load() with MP3 | Pass a file object | Pass a file path string — soundfile cannot decode MP3, only audioread (file path) can | +| UDP send on Windows | Assume loopback always works | Loopback UDP works but "Public" network profile may block non-loopback targets — test against a real device IP early | +| yt-dlp + MPV | Buffer yt-dlp output into MPV stdin | Use `--input` with a yt-dlp URL passthrough — MPV handles yt-dlp integration natively via `--script-opts` or URL schemes | +| FastAPI WebSocket | Use `asyncio.sleep()` in background task | Background task must check disconnect events; use `asyncio.wait_for()` with timeout | + +--- + +## Performance Traps + +| Trap | Symptoms | Prevention | When It Breaks | +|------|----------|------------|----------------| +| Sending full-strip RGBW frames at 60fps over WiFi | WiFi congestion, irregular frame delivery, ESP32 TX buffer overflow | Cap at 30fps for WiFi; use 40fps only with Ethernet | With 300 LEDs RGBW at 60fps = ~288 KB/s continuous — exceeds reliable WiFi throughput under contention | +| Loading full audio into memory for librosa analysis | 4-minute WAV at 44100Hz = ~40MB; 20+ songs in a session = memory pressure | Stream analysis in chunks, or enforce that analysis is single-file and results are cached to disk as JSON | At ~10 simultaneous cached analyses or >5 songs loaded at once | +| WebSocket broadcasting position updates at 60Hz | CPU overhead on backend; browser rendering stalls from processing too many messages | 10Hz is sufficient for UI display; 30Hz max if sub-frame cursor interpolation is required | At 30+ connected clients (not relevant for single-operator app, but good discipline) | +| Synchronous JSON serialization of large show files | Blocking the event loop during save | Use `asyncio.run_in_executor` for file I/O; or write to a temp file and rename atomically | Show files with >10,000 events and high-resolution per-LED animation data | + +--- + +## Security Mistakes + +| Mistake | Risk | Prevention | +|---------|------|------------| +| Accepting yt-dlp URLs without validation | An attacker with UI access could trigger yt-dlp against arbitrary URLs on the internal network (SSRF) | Behind Authelia SSO — single admin user — risk is low; still, validate URL scheme (`https://` only) and domain against an allowlist | +| Storing device IPs in show files | Hardcoded IPs in exported shows break when device moves; no security risk but operational problem | Store device by name in show files; resolve to IP at playback time from the device registry | +| No rate limiting on analysis endpoint | User can queue 20 CPU-intensive analyses simultaneously | Single analysis queue with one concurrent worker; reject if queue is non-empty | + +--- + +## UX Pitfalls + +| Pitfall | User Impact | Better Approach | +|---------|-------------|-----------------| +| Beat detection runs synchronously on "Sync" button click | UI freezes for 2–10 seconds with no feedback | Show a progress indicator; run analysis async and push result via WebSocket when complete | +| No visual diff between "detected beats" and "manually placed events" | User cannot tell which events are reliable vs hand-placed | Color-code events by source (auto-detected vs manual vs AI) in the timeline | +| Seeking during playback updates position but not device state | Devices hold last-sent color — after seek, devices show wrong color until next animation event | On seek, send an "all clear" or "fade to black" packet immediately to all registered devices | +| Animation preview only visible during live playback | User must play the whole show to verify new event placement | Provide a scrub-preview: single-frame UDP send on timeline cursor position change | +| No feedback when UDP send fails silently | Lights don't respond; user doesn't know if it's app, network, or device | Per-device health indicator in the UI; UDP send loop logs drop count accessible in real time | + +--- + +## "Looks Done But Isn't" Checklist + +- [ ] **MPV position polling:** Often missing thread safety — verify that position reads and writes use a lock or atomic structure. +- [ ] **Beat detection:** Often returns beats without a backward-shift calibration — verify by overlaying beat grid on a waveform at 10x zoom and checking against kick drum transients. +- [ ] **Multi-packet frames:** Often implemented for the send side only — verify the microcontroller simulator reassembles multi-packet frames correctly before writing firmware. +- [ ] **Show file format:** Often missing `schema_version` field — verify it is in the schema before any shows are saved that would need migration. +- [ ] **Windows timer resolution:** Often not set — verify by measuring `time.perf_counter()` jitter in the send loop; should be <2ms on Windows with `timeBeginPeriod(1)`. +- [ ] **WebSocket disconnect handling:** Often the background send task keeps running after the browser tab closes — verify the task is cancelled on disconnect. +- [ ] **UDP device health check:** Often bypassed during development — verify it actually detects a powered-off device before marking communication phase complete. +- [ ] **librosa MP3 loading:** Works with WAV, often breaks silently with MP3 if soundfile is the only backend — verify with an MP3 file explicitly. + +--- + +## Recovery Strategies + +| Pitfall | Recovery Cost | Recovery Steps | +|---------|---------------|----------------| +| Show files in wrong format (no schema_version) | MEDIUM | Write a one-time migration script; add schema_version = 1 to all existing files; add version check on load | +| Audio-LED clock drift discovered after shows are built | LOW | Add calibration offset field to show file; user adjusts offset per-show via UI without rebuilding | +| MTU issue discovered after firmware is written | HIGH | Requires firmware update for all devices to support multi-packet protocol; test MTU before finalizing protocol spec | +| GIL starvation causing send loop jitter | MEDIUM | Refactor analysis to run in ProcessPoolExecutor; no show file changes required | +| Librosa beat timestamps systematically wrong | LOW | Add a per-show or per-analysis backward-shift field; re-analyze existing shows with corrected offset | +| Windows firewall blocking UDP post-deployment | LOW | Document firewall rule; add startup diagnostic; add to setup guide | + +--- + +## Pitfall-to-Phase Mapping + +| Pitfall | Prevention Phase | Verification | +|---------|------------------|--------------| +| Audio clock vs wall clock drift | MPV integration phase | Measure drift over a 4-minute song; delta between MPV position and expected position must be <50ms | +| MPV IPC named pipe blocking | MPV integration phase | Run IPC read and UDP send in parallel for 10 minutes; measure max send loop jitter | +| librosa beat timestamps late | Beat detection phase | Visual overlay of beat grid vs waveform transients on 3 genre samples | +| Beat tracking on tempo-variable music | Beat detection phase | Test against a jazz track and a classical track; verify `plp()` fallback exists | +| UDP packet loss show corruption | Protocol design phase | Intentional 5% packet loss test; verify strip recovers gracefully | +| MTU fragmentation | Protocol design phase | Test with 300-LED full-frame packet; verify no IP fragmentation in Wireshark capture | +| Windows firewall blocking UDP | Communication layer phase | Test from a "Public" network profile; startup diagnostic must detect and report blocked state | +| Show file absolute timestamp lock-in | Show file design phase | Verify schema_version, source_fingerprint, and shift-all-events operation exist before first show is saved | +| Python GIL starving send loop | Architecture phase | Beat analysis triggered during playback must not cause >5ms send loop jitter | +| Frontend clock drift | Playback UI phase | Seek 10 times rapidly; verify frontend position matches backend position within 100ms after each seek | + +--- + +## Sources + +- librosa beat tracking documentation and known timing bias: https://librosa.org/doc/main/generated/librosa.beat.beat_track.html +- librosa beat timing issue (20–60ms late): https://github.com/librosa/librosa/issues/1052 +- librosa default resampling rationale: https://librosa.org/blog/2019/07/17/resample-on-load/ +- librosa onset vs beat tracking: https://deepwiki.com/librosa/librosa/5.1-beat-tracking +- MPV IPC protocol documentation: https://github.com/mpv-player/mpv/blob/master/DOCS/man/ipc.rst +- MPV Windows named pipe IPC challenges: https://github.com/mpv-player/mpv/issues/10116 +- python-mpv-jsonipc library: https://github.com/iwalton3/python-mpv-jsonipc +- ESP32 UDP packet loss and buffer overflow: https://www.esp32.com/viewtopic.php?t=5411 +- ESP32 UDP performance issues: https://github.com/espressif/arduino-esp32/issues/1317 +- WLED UDP realtime protocol and MTU: https://kno.wled.ge/interfaces/udp-realtime/ +- WLED DNRGB multi-packet protocol: https://github.com/Aircoookie/WLED/wiki/UDP-Realtime-Control +- UDP pixel streaming protocol (start index header): https://github.com/EnviralDesign/NodeMCU-Arduino-Pixel-Driver/wiki/UDP-Pixel-Streaming-protocol +- Windows asyncio timing limitations: https://runebook.dev/en/docs/python/library/asyncio-platforms/windows +- Python GIL and real-time threading: https://realpython.com/python-gil/ +- FastAPI WebSocket common mistakes: https://hexshift.medium.com/managing-per-user-websocket-state-in-fastapi-9ceaa2b312ac +- Audio sync in browser vs backend: https://medium.com/fender-engineering/near-realtime-animations-with-synchronized-audio-in-javascript-6d845afcf1c5 +- Windows firewall UDP blocking: https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/windows-fireware-rule-block-udp-communication +- Timeline absolute vs BPM-relative anchoring: https://resolume.com/forum/viewtopic.php?t=13032 + +--- +*Pitfalls research for: music-to-light synchronization (LightSync)* +*Researched: 2026-04-05* diff --git a/.planning/research/STACK.md b/.planning/research/STACK.md new file mode 100644 index 0000000..18a6852 --- /dev/null +++ b/.planning/research/STACK.md @@ -0,0 +1,214 @@ +# Stack Research + +**Domain:** Music-to-light synchronization system (Python backend, browser frontend, UDP control) +**Researched:** 2026-04-05 +**Confidence:** MEDIUM-HIGH (core stack HIGH, audio analysis MEDIUM, AI sync LOW) + +--- + +## Recommended Stack + +### Core Technologies + +| Technology | Version | Purpose | Why Recommended | +|------------|---------|---------|-----------------| +| Python | 3.11 | Runtime | 3.11 hits the sweet spot: modern async, good Windows support, librosa 0.11 compatible. Avoid 3.12+ until madmom resolves its incompatibility. Avoid 3.9 as it's EOL in Oct 2025. | +| FastAPI | 0.135.x | Web framework + REST API + WebSocket server | Native async/await, built-in WebSocket support, Pydantic validation, serves static files — one process covers API + UI + realtime push. Flask is sync-first and requires extensions for every feature this app needs. | +| uvicorn | 0.34.x | ASGI server | Standard FastAPI runner, works on Windows without Gunicorn. Use `uvicorn.run()` programmatically so the app can self-start without a separate process manager. | +| librosa | 0.11.0 | Beat detection, onset analysis, tempo estimation | Released March 2025, supports NumPy 2.0, actively maintained. Correct choice for offline audio analysis from files. scipy FFT backend is more accurate than numpy. | +| python-mpv | 1.0.8 | MPV process control + playback position polling | Released April 2025. ctypes-based — gives property observers and event hooks inside Python. Better than raw subprocess for position tracking. Windows uses named pipe IPC automatically. | +| Pydantic | 2.x (bundled with FastAPI) | Show file validation, device registry schemas, config | Already a FastAPI dependency. Use BaseModel for all JSON show file structures — free validation and serialization. | + +### Supporting Libraries + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| numpy | 2.0+ | Array math for pixel frame buffers, beat arrays | Always — librosa depends on it, and frame-based UDP payloads are numpy arrays before packing | +| scipy | 1.14+ | FFT, signal processing (librosa backend) | Always — librosa uses it as FFT backend since 0.11.0 | +| soundfile | 0.12+ | Audio file loading (WAV, FLAC, OGG) for librosa | When loading local audio files for analysis pre-playback | +| aiofiles | 24.x | Async file I/O for show file read/write | When persisting show files from async FastAPI handlers | +| SQLite (stdlib) | built-in | Persistent storage for devices, shows, project state | Use Python's built-in `sqlite3` or `aiosqlite` for async; no ORM needed for this scale | +| aiosqlite | 0.20+ | Async SQLite wrapper | For non-blocking database access from FastAPI async handlers | +| structlog | 24.x | Structured logging | When you want JSON-formatted logs for the backend; easier to parse than stdlib logging | +| python-dotenv | 1.x | Environment config (host, port, UDP targets) | For externalizing runtime config without hardcoding | +| madmom | 0.16 | High-accuracy beat tracking (DBN model) | ONLY if using Python 3.9 in a separate subprocess or venv — madmom is incompatible with Python 3.10+. Use as an optional enhancement path, not core dependency. | + +### Frontend Stack + +| Technology | Version | Purpose | Why Recommended | +|------------|---------|---------|-----------------| +| Vanilla JS (ES2022+) | native | Timeline editor, WebSocket client, UI interactions | This app has a custom DAW-like UI — no component framework matches the drag-drop timeline without fighting abstractions. React adds 200KB+ and solves problems this app doesn't have. | +| Native WebSocket API | native | Real-time playback position updates from backend | Browser-native, zero dependencies, sufficient for position polling and event pushes | +| CSS Custom Properties + Grid/Flexbox | native | Terminal aesthetic: dark theme, monospace, track layout | No CSS framework needed — a bespoke terminal aesthetic is easier to build raw than to override Bootstrap or Tailwind | +| Web Audio API | native | Waveform visualization (optional) | If displaying waveform in timeline, decode via AudioContext.decodeAudioData() — no library needed | + +### Development Tools + +| Tool | Purpose | Notes | +|------|---------|-------| +| uv | Python package manager + venv | Faster than pip, reproducible installs, handles Windows path quirks well | +| pyinstaller (optional) | Bundle to Windows .exe | Only if distributing; not needed for personal dev machine use | + +--- + +## Installation + +```bash +# Core backend +uv pip install fastapi==0.135.3 uvicorn==0.34.1 python-mpv==1.0.8 pydantic==2.10.x + +# Audio analysis +uv pip install librosa==0.11.0 soundfile==0.12.1 + +# Persistence + async +uv pip install aiosqlite==0.20.0 aiofiles==24.1.0 + +# Config + utilities +uv pip install python-dotenv==1.0.1 structlog==24.4.0 +``` + +Note: `numpy` and `scipy` install automatically as librosa dependencies. Do not pin them separately unless you hit a conflict. + +--- + +## Alternatives Considered + +| Recommended | Alternative | When to Use Alternative | +|-------------|-------------|-------------------------| +| FastAPI | Flask | If you already have a Flask app or need Jinja templating; Flask-SocketIO is a viable WebSocket path but adds complexity | +| FastAPI | Django | Never for this project — Django's ORM and admin are overhead for a single-operator music sync tool | +| Vanilla JS | React | If the team knows React and the timeline editor becomes a complex stateful SPA (drag-drop state machines, undo/redo) that benefits from component isolation | +| Vanilla JS | HTMX | HTMX is wrong for this use case — the timeline editor requires fine-grained JS state, not server-rendered HTML swaps | +| python-mpv (ctypes) | python-mpv-jsonipc | Use jsonipc variant if libmpv.dll is unavailable on the target Windows system (jsonipc controls an external mpv.exe via named pipe, no DLL required) | +| python-mpv | subprocess + JSON IPC manual | More control, no DLL dependency, but requires implementing the IPC protocol yourself | +| librosa (offline) | madmom (offline) | madmom has better beat accuracy for complex rhythms, BUT it is incompatible with Python 3.10+. Use only in an isolated subprocess with Python 3.9 if high-accuracy tracking is critical. | +| aiosqlite | SQLAlchemy + SQLModel | Use SQLAlchemy if the schema grows complex (multiple joined tables, migrations). For show files + device registry, raw aiosqlite is sufficient. | +| SQLite | JSON flat files | JSON files are fine for show data but lose query capability for device registry lookups and show search. SQLite has zero deployment overhead on Windows. | + +--- + +## What NOT to Use + +| Avoid | Why | Use Instead | +|-------|-----|-------------| +| aubio | Last release: February 2019 (0.4.9). Unmaintained, no Python 3.10+ wheels, Windows install is fragile. Described as "Beta" still after 6 years. | librosa for offline analysis | +| madmom as core dependency | Incompatible with Python 3.10+ due to `collections.MutableSequence` removal. Requires Python ≤ 3.9. Outstanding GitHub issues with no resolution timeline. | librosa as primary; madmom only in isolated venv if needed | +| essentia | Excellent accuracy, but C++ binding installation on Windows is non-trivial. Pre-built wheels are inconsistent. Adds significant complexity for marginal beat detection gain. | librosa covers the use case well on Windows | +| BeatNet | Requires madmom as a dependency, inheriting the Python 3.10+ incompatibility problem. Also requires PyTorch, which is 200–800MB. Overkill for a personal project. | librosa; revisit if AI sync accuracy becomes a priority | +| gevent / eventlet | Old async approaches, sometimes required with Flask-SocketIO. Not needed with FastAPI's native async. | FastAPI + uvicorn native async | +| Flask-SocketIO | Adds an abstraction layer over Socket.IO protocol instead of raw WebSocket. Overkill, and Socket.IO client required on frontend. | FastAPI native WebSocket | +| PyAudio / sounddevice for playback | This project uses MPV for playback. Mixing MPV subprocess with Python audio I/O for the same audio creates timing conflicts. | python-mpv for all playback | +| Art-Net / sACN / E1.31 protocol | These are DMX-over-UDP protocols designed for theatrical equipment. The project uses custom UDP packets to custom firmware — don't add DMX protocol overhead. | Raw UDP asyncio datagrams | + +--- + +## Stack Patterns by Variant + +**For beat detection on local audio files:** +- Load with `soundfile.read()` → pass to `librosa.beat.beat_track(y, sr)` → returns tempo + beat frame indices +- Convert frames to timestamps: `librosa.frames_to_time(beats, sr=sr)` +- Onset detection for drop/hit detection: `librosa.onset.onset_detect(y, sr, units='time')` + +**For UDP frame sending (AI-generated pixel sequences):** +- Pack RGB values into bytes: `struct.pack('B' * (led_count * 3), *flat_rgb_array)` +- Send via asyncio datagram endpoint: `transport.sendto(payload, (target_ip, target_port))` +- Fire-and-forget: no acknowledgement, send at frame rate (e.g., 30fps = 33ms interval) + +**For UDP animation+params sending (manual mode):** +- Small JSON payload or fixed binary struct: `{"anim": "pulse", "color": [255, 0, 128], "speed": 1.5}` +- Encode as JSON bytes or define a compact binary protocol with `struct.pack` +- JSON is easier to debug during early dev; migrate to binary if bandwidth becomes an issue + +**For playback position sync (frontend timeline scrubbing):** +- Use `python-mpv` property observer: `player.observe_property('time-pos', callback)` +- In callback, push position via WebSocket to all connected browser clients +- Frontend timeline animates playhead based on received position events + +**For AI sync (optional enhancement — LOW confidence):** +- No mature turnkey Python library for music-to-light AI in 2025 that runs on Windows without GPU +- Practical path: run librosa full analysis (beat + onset + chroma + spectral contrast), feed features into a rule-based mapper (loud = bright, beat = flash, drop = color change) +- True AI path: fine-tune a small model offline using MIDI-to-light training data — deferred to later phases +- OpenAI/Claude audio analysis APIs could be used to get high-level song structure (verse/chorus/bridge) from a transcript or description, but audio file analysis APIs do not exist at useful granularity for per-beat sync + +**For Windows 11 development:** +- Use `python-mpv` with `libmpv-2.dll` placed next to the app (ship with the project) +- Alternatively, use `python-mpv-jsonipc` which controls `mpv.exe` via named pipe — no DLL needed +- Named pipe path: `\\.\pipe\lightsync_mpv` +- UDP `asyncio` works natively on Windows with `ProactorEventLoop` (Python 3.8+ default on Windows) + +--- + +## JSON Show File Format + +No industry-standard JSON schema exists for this exact use case. Custom format recommended. + +```json +{ + "version": "1.0", + "meta": { + "title": "My Show", + "audio_source": "file:///path/to/song.mp3", + "duration_seconds": 213.4, + "bpm": 128.0, + "created_at": "2026-04-05T12:00:00Z" + }, + "beats": [0.47, 0.94, 1.41], + "devices": [ + { + "id": "wall_strip", + "name": "Wall Strip", + "type": "SK6812", + "led_count": 300, + "target_ip": "192.168.1.50", + "port": 4210 + } + ], + "timeline": { + "wall_strip": [ + { + "start": 0.47, + "end": 1.41, + "animation": "pulse", + "params": {"color": [255, 0, 128], "speed": 1.0} + } + ] + } +} +``` + +Key design decisions for the format: +- Timestamps in seconds (float) — not frames or ticks. Decoupled from sample rate. +- Beat array stored in show file — precomputed at import time, not recalculated on playback. +- Device definitions embedded in show file — show is self-contained, no external registry lookup required during playback. +- Animation params as open object — allows per-animation parameter schemas without breaking the format. + +--- + +## Version Compatibility + +| Package | Compatible With | Notes | +|---------|-----------------|-------| +| librosa 0.11.0 | numpy >= 1.17, scipy >= 1.0, Python 3.8+ | Requires soundfile for local file loading | +| python-mpv 1.0.8 | Python >= 3.9, libmpv 0.35+ | libmpv-2.dll required on Windows; ship with app | +| FastAPI 0.135.x | pydantic v2, Python 3.8+, uvicorn 0.34+ | Pydantic v1 compatibility mode dropped | +| madmom 0.16 | Python < 3.10 ONLY | Do not install in same venv as main app if using Python 3.11+ | +| numpy 2.0+ | librosa 0.11.0, scipy 1.14+, soundfile 0.12+ | librosa 0.11.0 explicitly added NumPy 2.0 support | + +--- + +## Sources + +- [FastAPI release notes](https://fastapi.tiangolo.com/release-notes/) — version 0.135.3 confirmed current (HIGH confidence) +- [librosa 0.11.0 changelog](https://librosa.org/doc/main/changelog.html) — released March 2025, NumPy 2.0 support confirmed (HIGH confidence) +- [python-mpv PyPI](https://pypi.org/project/python-mpv/) — version 1.0.8 released April 2025, Windows support confirmed (HIGH confidence) +- [aubio PyPI](https://pypi.org/project/aubio/) — last release 2019, confirmed stale (HIGH confidence) +- [madmom Python 3.10+ incompatibility — GitHub issue](https://github.com/CPJKU/beat_this/issues/9) — confirmed broken on Python >= 3.10 (HIGH confidence) +- [BIFF.ai beat detection rundown](https://biff.ai/a-rundown-of-open-source-beat-detection-models/) — madmom vs BeatNet comparison (MEDIUM confidence, single source) +- [MPV IPC documentation](https://github.com/mpv-player/mpv/blob/master/DOCS/man/ipc.rst) — named pipe on Windows confirmed (HIGH confidence) +- [uvicorn release notes](https://uvicorn.dev/release-notes/) — 0.34.1 current stable, Windows ProactorEventLoop (HIGH confidence) +- WebSearch: FastAPI WebSocket patterns, async UDP asyncio, JSON show file formats — (MEDIUM confidence, multiple sources) + +--- + +*Stack research for: LightSync — music-to-light synchronization system* +*Researched: 2026-04-05* diff --git a/.planning/research/SUMMARY.md b/.planning/research/SUMMARY.md new file mode 100644 index 0000000..740249f --- /dev/null +++ b/.planning/research/SUMMARY.md @@ -0,0 +1,216 @@ +# Project Research Summary + +**Project:** LightSync — music-to-light synchronization system +**Domain:** Show controller (Python backend, DAW-style web UI, UDP LED control) +**Researched:** 2026-04-05 +**Confidence:** MEDIUM-HIGH + +## Executive Summary + +LightSync is a personal show controller in a well-understood category: professional tools like xLights, QLC+, and SoundSwitch solve the same core problem but are overkill for a 2-strip home setup and don't support the custom UDP protocol this project requires. The right approach is a DAW-style timeline editor backed by librosa for offline beat detection, MPV for audio playback, FastAPI for the backend, and vanilla JS for the frontend. Nothing exotic is needed — the stack is entirely standard Python web application territory, with the non-standard element being the real-time scheduling loop that must be isolated from the async event loop to achieve reliable timing. + +The two highest-risk areas are timing and protocol design. The scheduling loop that drives UDP output must poll MPV's audio clock (not wall clock), run in a dedicated thread with 1ms OS timer resolution on Windows, and pre-fire cues by a configurable offset to compensate for LED pipeline latency. The UDP protocol — specifically the packet format for 300-LED RGBW strips — must be settled before firmware is written, because MTU fragmentation and packet loss recovery behavior are hard to retrofit into deployed microcontrollers. Both of these architectural constraints shape the build order more than any feature decision. + +The recommended build strategy is bottom-up: data model first, audio engine second, UDP output third, timeline editor fourth. This order ensures each layer is tested in isolation before adding the next. The microcontroller firmware goes last — the protocol should be fully validated against a software simulator before any hardware is flashed. Beat detection and AI sync are enhancers on top of a working show engine, not prerequisites. + +## Key Findings + +### Recommended Stack + +The stack is intentionally narrow. FastAPI (0.135.x) on uvicorn handles the REST API, WebSocket hub, and static file serving in a single process — no need for a separate web server or Socket.IO abstraction. Audio is handled entirely by MPV via `python-mpv-jsonipc`, which controls an external `mpv.exe` via named pipe and avoids the DLL dependency of the ctypes binding. Beat detection uses librosa 0.11.0 with scipy as the FFT backend — the March 2025 release adds NumPy 2.0 support and is the current stable choice. The frontend is vanilla JS with no framework; a DAW-style timeline editor fights React's abstractions and doesn't need component isolation. + +**Core technologies:** +- **Python 3.11**: Sweet spot — modern async, librosa 0.11.0 compatible, avoids madmom's Python 3.10+ incompatibility +- **FastAPI 0.135.x + uvicorn 0.34.x**: Native async/await, WebSocket, REST, and static file serving in one process +- **python-mpv-jsonipc 1.0.8**: MPV subprocess control via named pipe — no libmpv.dll required on Windows +- **librosa 0.11.0 + soundfile 0.12**: Offline beat detection, onset analysis, structural segmentation +- **aiosqlite 0.20**: Async SQLite for device registry and show persistence — no ORM needed at this scale +- **Vanilla JS (ES2022)**: Timeline editor, WebSocket client — zero framework overhead + +**What to avoid:** aubio (last release 2019), madmom (Python 3.10+ incompatible), essentia (Windows install fragile), BeatNet (requires PyTorch, inherits madmom incompatibility). Do not use asyncio.sleep() for real-time cue scheduling — use a dedicated thread with `time.perf_counter()`. + +### Expected Features + +Every professional show editor has waveform display, per-device timeline tracks, beat detection, snap-to-beat, and show save/load. These are non-negotiable — missing any of them makes the editor feel broken. Undo/redo is frequently absent in competitors (xLights, QLC+) but users notice and regret the omission — build it from day one using the command pattern or retrofitting is painful. + +**Must have (table stakes):** +- Audio file loading + MPV playback (play/pause/seek/position display) +- Waveform display — editing without it is guessing +- Beat detection via librosa — timing marks drive everything downstream +- Timeline editor with per-device tracks and drag-and-place effect blocks +- Animation library with 8-12 core effects (rainbow, chase, pulse, strobe, sparkle, fade, wipe, static) +- Snap-to-beat on placement and resize +- Color picker + basic params (speed, brightness, direction) per effect block +- Show save/load as JSON +- UDP command sender (animation + params at correct timestamp) +- Undo/redo for timeline mutations — build first, not later + +**Should have (differentiators):** +- YouTube URL loading via yt-dlp — no other personal tool does this cleanly +- Live preview panel (canvas LED strip simulation without hardware) +- Structural segmentation overlay (verse/chorus/bridge from librosa.segment) +- UDP frame sender (raw RGBW pixel data for AI-generated sequences) +- AI-assisted show generation (rule-based mapping on top of beat analysis) +- Effect grouping / scenes + +**Defer to v2+:** +- Parameter animation curves (DAW automation lanes — powerful, high UI complexity) +- Frequency band mapping (per-frame bass/mid/treble routing) +- Moving heads, lasers, fog machine support (extend device abstraction when hardware exists) + +### Architecture Approach + +The system is a single Python process with four distinct subsystems that must not bleed into each other: Show Engine (playback state machine and cue scheduler), Audio Manager (MPV IPC bridge), UDP Sender (per-device packet dispatcher), and WebSocket Hub (browser push channel). The Show Engine orchestrates the others but never owns their internals. The critical isolation is the scheduling loop — it runs in a dedicated `threading.Thread` reading MPV position into a shared variable, never blocking the asyncio event loop. Analysis (librosa) runs in a `ProcessPoolExecutor` to avoid GIL starvation of the real-time loop. + +**Major components:** +1. **Show Engine** — playback state machine, position-driven cue scheduler, orchestrates other subsystems +2. **Audio Manager** — MPV subprocess + named pipe IPC bridge, position polling into shared variable +3. **UDP Sender** — async socket pool, per-device packet dispatch (animation command mode and frame mode) +4. **WebSocket Hub** — connection manager, broadcasts position ticks at 10Hz and state change events +5. **Analysis Engine** — librosa beat/onset/segment processing in ProcessPoolExecutor, results stored in show JSON +6. **Show Store** — Pydantic-validated JSON load/save, device registry, animation library catalog +7. **Timeline UI** — vanilla JS DAW editor, effect blocks, beat markers, transport controls + +**Key patterns:** +- Position-driven cue scheduling (poll MPV, fire when `pos >= cue.timestamp`) — handles seek/pause correctly; asyncio.sleep-based scheduling does not +- Device abstraction via `BaseDevice` ABC — SK6812 and WS2801 implement it; future devices add a file +- Dual UDP mode — short animation+params commands for manual shows; full RGBW pixel frames for AI sequences +- Offline analysis pipeline — beat detection runs once at load, results cached in show JSON; never during playback + +### Critical Pitfalls + +1. **Audio clock vs wall clock drift** — Use MPV's `playback-time` property (not `time-pos`, which returns null when paused) as the authoritative clock on every scheduling tick. On Windows, call `timeBeginPeriod(1)` via ctypes at startup to raise OS timer resolution from 15.6ms to 1ms. Expose a per-show sync calibration offset in the UI — do not defer this. + +2. **MPV named pipe blocking on Windows** — Named pipe reads block. Run the IPC reader in a dedicated daemon thread storing position in a shared variable. The send loop reads from that variable — it never touches IPC. Use `python-mpv-jsonipc` to avoid raw overlapped I/O complexity. + +3. **MTU fragmentation for large strips** — 300 RGBW LEDs = 1200 bytes of pixel data, exceeding WiFi MTU when headers are added. Split into two packets with a start-index header (DNRGB protocol pattern). Store `leds_per_packet` in the device registry. This must be in the protocol spec before firmware is written — retrofitting costs firmware updates on all devices. + +4. **Show file format locked to absolute timestamps** — Include `schema_version`, `source_fingerprint` (SHA256 of first 64KB of audio), and a "shift all events by N seconds" operation from day one. Also store beat position (beat index + subdivision) alongside absolute seconds so shows can be adapted to re-encoded audio. + +5. **librosa beat timestamps are 20-60ms late** — This is documented behavior, not a bug. Apply a configurable backward shift after `beat_track()`. Cross-validate with `librosa.onset.onset_detect()` for downbeat placement. Also implement `librosa.beat.plp()` from the start for tempo-variable music (jazz, classical) — `beat_track()` alone fails on free-tempo content. + +6. **Python GIL starves the UDP send loop** — librosa analysis is numpy-heavy and holds the GIL. Run it in `ProcessPoolExecutor` (separate process, own GIL), not `ThreadPoolExecutor`. Never trigger analysis during playback without this boundary. + +## Implications for Roadmap + +Based on research, the build order is dictated by dependencies, not by feature value. Each phase must be validated before the next is started — particularly the protocol spec, which gates firmware development. + +### Phase 1: Foundation — Data Model and Project Skeleton + +**Rationale:** The show file schema is the contract every other component builds against. Defining it last forces expensive migrations. The Pydantic models, device registry, and FastAPI skeleton cost almost nothing to build first and prevent cascading changes later. +**Delivers:** Validated show file schema, device registry, FastAPI app skeleton with static file serving, terminal-aesthetic HTML/CSS shell +**Addresses:** Show save/load, device registry (table stakes); `schema_version` and `source_fingerprint` (PITFALL 4) +**Avoids:** Show file format lock-in (Pitfall 8) — schema_version, source_fingerprint, and shift-all-events operation are required outputs of this phase + +### Phase 2: Audio Engine and Transport + +**Rationale:** Audio position is the master clock. Everything else (cues, timeline cursor, UDP timing) is derived from it. This is the highest-risk phase — Windows named pipe behavior and timer precision must be validated here, not discovered later. +**Delivers:** MPV subprocess control, IPC bridge with shared position variable, playback state machine, WebSocket hub, transport controls in UI +**Uses:** python-mpv-jsonipc, FastAPI WebSocket, threading.Thread for IPC reader +**Implements:** Audio Manager + Show Engine skeleton + WebSocket Hub +**Avoids:** Audio clock drift (Pitfall 1) — `timeBeginPeriod(1)` and calibration offset UI are required; MPV named pipe blocking (Pitfall 2) — dedicated reader thread is required; Frontend clock drift (Pitfall 10) — backend position is authoritative, frontend interpolates only + +### Phase 3: Communication Protocol and UDP Output + +**Rationale:** The UDP protocol must be fully specced and tested against a software simulator before firmware exists. MTU fragmentation and packet loss recovery are hard to retrofit post-firmware. This phase defines the contract the microcontrollers will implement. +**Delivers:** BaseDevice abstraction, SK6812 and WS2801 packet encoders, async UDP sender, protocol spec document, software device simulator, per-device health check on startup +**Uses:** asyncio UDP datagrams, DRGB/DNRGB/DRGBW packet formats, struct.pack +**Implements:** Device Registry + UDP Sender + BaseDevice hierarchy +**Avoids:** MTU fragmentation (Pitfall 6) — multi-packet frame support required; UDP packet loss corruption (Pitfall 5) — sequence numbers and 2-3x retransmit for slow animations; Windows firewall blocking (Pitfall 7) — startup diagnostic required + +### Phase 4: Timeline Editor + +**Rationale:** The timeline editor is the core UX and the feature that makes or breaks the app. It depends on analysis data (beat marks to snap to) and a stable show file schema. Building it after both are solid prevents churn. Undo/redo must be built here — retrofitting the command pattern into an existing stateful editor is expensive. +**Delivers:** Per-device track layout, drag-and-place effect blocks, beat marker overlay, snap-to-beat, color picker, effect params, waveform display, undo/redo (command pattern) +**Uses:** Vanilla JS, Web Audio API for waveform rendering, librosa beat_track + plp + onset_detect +**Implements:** Analysis Engine + Timeline UI +**Avoids:** librosa timestamp lateness (Pitfall 3 and 4) — backward-shift calibration and plp() fallback required; GIL starvation (Pitfall 9) — analysis runs in ProcessPoolExecutor + +### Phase 5: Live Show Execution — End-to-End Integration + +**Rationale:** First phase where all subsystems run together. Audio position drives cue firing which drives UDP output. Integration tests here validate the timing guarantees established in Phase 2 and Phase 3 under real-world conditions. +**Delivers:** Full show playback (load show, press play, lights execute), seek-safe cue rescheduling, position-driven scheduler loop, integration test against device simulator +**Uses:** Show Engine cue scheduler, UDP Sender, Audio Manager, WebSocket position ticks +**Implements:** Complete Show Engine with sub-20ms cue timing target + +### Phase 6: Enhancements — YouTube, Preview, AI Sync + +**Rationale:** These are differentiators that build on a working foundation. YouTube loading is low-cost (yt-dlp + MPV natively). Live preview and AI sync are medium-to-high complexity and should only be built once the core editing loop has been validated with real shows. +**Delivers:** YouTube URL loading via yt-dlp, live canvas preview panel, structural segmentation overlay, AI-assisted show generation, raw frame-mode UDP sender +**Avoids:** AI sync requires beat detection to be solid (Phase 4) and frame-mode UDP to be specced (Phase 3 protocol doc) + +### Phase 7: Microcontroller Firmware + +**Rationale:** Firmware is last by design. The protocol spec from Phase 3, validated through Phases 5 and 6, is the implementation target. Flashing firmware against a moving protocol spec means reflashing every iteration. +**Delivers:** MicroPython (or C/Arduino) firmware for ESP/Pico/RPi4B supporting DRGB/DRGBW frame mode and animation command mode, tested against both SK6812 and WS2801 strips +**Implements:** Hardware integration for the two v1 devices + +### Phase Ordering Rationale + +- Show file schema before everything — the Pydantic models in Phase 1 are the contract that lets every subsequent phase proceed without model churn +- Audio engine before UDP — you cannot validate timing without knowing what time MPV says it is +- Protocol spec before firmware — MTU fragmentation and packet loss recovery are architectural, not implementation details; they must be frozen before microcontrollers are programmed +- Timeline editor before live execution — the cue data structure must be stable before the show engine fires cues +- Enhancements after validation — AI sync and live preview should be built on a foundation that has been used for at least one real show +- Firmware last — no exceptions; protocol stability is the prerequisite + +### Research Flags + +Phases likely needing deeper research during planning: +- **Phase 2 (Audio Engine):** Windows named pipe behavior under load is under-documented. Validate `python-mpv-jsonipc` latency characteristics on Windows 11 before committing to the threading model. The `timeBeginPeriod(1)` call requires explicit testing to confirm it achieves <2ms timer resolution. +- **Phase 4 (Timeline Editor):** Canvas-based DAW timeline with drag-drop, resize, and snap is complex UI territory. Research interaction patterns from open-source sequencers (e.g., LMMS web, Audion) before writing the first JS line. Undo/redo command pattern design should be explicitly sketched before implementation. +- **Phase 6 (AI Sync):** No mature Python library for music-to-light AI sync exists in 2025. The practical path (librosa features → rule-based mapper) is clear, but the LLM-assisted path (song structure analysis → cue placement) has no established pattern. Needs a prototype spike before committing to an approach. + +Phases with well-documented patterns (skip deeper research): +- **Phase 1 (Foundation):** Pydantic v2 schemas and FastAPI skeleton are standard; no research needed +- **Phase 3 (UDP Protocol):** WLED's DRGB/DNRGB/DRGBW packet format is fully documented and battle-tested; adopt it directly +- **Phase 5 (Live Execution):** The position-driven scheduler pattern from ARCHITECTURE.md is fully specified; implement it directly +- **Phase 7 (Firmware):** MicroPython UDP socket handling for ESP32/Pico is well-documented; firmware complexity is protocol implementation, not discovery + +## Confidence Assessment + +| Area | Confidence | Notes | +|------|------------|-------| +| Stack | HIGH | All core library versions verified against official changelogs and PyPI; madmom/aubio/essentia incompatibilities confirmed from GitHub issues and official sources | +| Features | HIGH | Cross-referenced against 6 commercial/open-source tools (xLights, QLC+, SoundSwitch, Lightjams, WLED, Resolume); feature table is comprehensive | +| Architecture | HIGH | Core patterns (position-driven scheduling, BaseDevice abstraction, dual UDP mode, offline analysis) are established in reference implementations; AI sync specifics are MEDIUM | +| Pitfalls | MEDIUM-HIGH | Timing and protocol pitfalls are HIGH confidence from official docs and community reports; Windows asyncio timer precision is MEDIUM from limited benchmarks; beat detection accuracy claims are backed by librosa GitHub issues | + +**Overall confidence:** MEDIUM-HIGH + +### Gaps to Address + +- **libmpv.dll vs jsonipc on Windows 11:** Research recommends `python-mpv-jsonipc` to avoid the DLL dependency, but its response latency under load on Windows 11 is not benchmarked. Validate in Phase 2 with a timing harness before building the full audio engine around it. +- **librosa MP3 loading path:** `soundfile` cannot decode MP3 — it falls back to `audioread`. This is documented but the fallback behavior in librosa 0.11.0 on Windows needs explicit validation (audioread depends on FFmpeg being on PATH or bundled). Verify in Phase 4 before analysis is used on real songs. +- **AI sync approach:** The rule-based mapper (librosa features → animation selection) is the practical v1 path. The LLM-assisted approach (song structure description → cue placement) is speculative — no existing implementation to reference. Treat as a Phase 6 spike, not a commitment. +- **Microcontroller choice:** PROJECT.md lists ESP/Pico/RPi4B as TBD. The protocol spec (Phase 3) works for all three, but MicroPython async UDP socket handling differs between platforms. Choose before Phase 7 begins. + +## Sources + +### Primary (HIGH confidence) +- [librosa 0.11.0 changelog](https://librosa.org/doc/main/changelog.html) — version and NumPy 2.0 support +- [FastAPI release notes](https://fastapi.tiangolo.com/release-notes/) — version 0.135.3 current +- [python-mpv PyPI](https://pypi.org/project/python-mpv/) — version 1.0.8, Windows support +- [MPV IPC documentation](https://mpv.io/manual/master/) — named pipe, time-pos, playback-time +- [WLED UDP Realtime Protocol](https://kno.wled.ge/interfaces/udp-realtime/) — DRGB/DNRGB/DRGBW packet format +- [madmom Python 3.10+ incompatibility](https://github.com/CPJKU/beat_this/issues/9) — confirmed broken +- [aubio PyPI](https://pypi.org/project/aubio/) — last release 2019, confirmed stale +- [librosa beat timing issue](https://github.com/librosa/librosa/issues/1052) — 20-60ms lateness documented +- [python-mpv-jsonipc](https://github.com/iwalton3/python-mpv-jsonipc) — Windows named pipe abstraction + +### Secondary (MEDIUM confidence) +- [xLights Manual — Timeline and Waveform](https://manual.xlights.org/xlights/chapters/chapter-four-sequencer/timeline-and-waveform) — feature baseline +- [SoundSwitch What's New in 2.9](https://support.soundswitch.com/en/support/solutions/articles/69000858481) — BPM detection, phrase detection +- [WLED Audio Reactive documentation](https://kno.wled.ge/advanced/audio-reactive/) — 30+ effects reference +- [Lightjams — Sync Music and Lights](https://www.lightjams.com/musicToDMX.html) — frequency band mapping patterns +- [FastAPI WebSocket background tasks](https://hexshift.medium.com/implementing-background-tasks-with-websockets-in-fastapi-034cdf803430) — async patterns +- [audio-reactive-led-strip](https://github.com/scottlawsonbc/audio-reactive-led-strip) — reference Python + UDP + ESP8266 architecture +- [Windows asyncio timing limitations](https://runebook.dev/en/docs/python/library/asyncio-platforms/windows) — 15.6ms default resolution + +### Tertiary (LOW confidence) +- AI sync approach (rule-based mapper + LLM) — inferred from librosa capabilities; no existing reference implementation +- Windows firewall UDP blocking behavior — [Microsoft docs](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/windows-fireware-rule-block-udp-communication) + community reports; actual blocking conditions vary by system configuration + +--- +*Research completed: 2026-04-05* +*Ready for roadmap: yes*