# Architecture Research **Domain:** Music-to-light synchronization system (show controller) **Researched:** 2026-04-05 **Confidence:** HIGH (core patterns are well-established; AI sync specifics are MEDIUM) ## Standard Architecture ### System Overview ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Browser Frontend │ │ ┌──────────────┐ ┌────────────────┐ ┌──────────────────────┐ │ │ │ Timeline UI │ │ Transport Bar │ │ Device + Show Panel │ │ │ │ (DAW editor) │ │ (play/pause/ │ │ (registry + library)│ │ │ │ │ │ seek/pos) │ │ │ │ │ └──────┬───────┘ └───────┬────────┘ └──────────┬───────────┘ │ │ │ │ │ │ │ └──────────────────┴───────────────────────┘ │ │ │ WebSocket (full-duplex) │ └────────────────────────────┼────────────────────────────────────────┘ │ ┌────────────────────────────┼────────────────────────────────────────┐ │ FastAPI Backend │ │ ┌─────────────────────────┴──────────────────────────────────┐ │ │ │ WebSocket Hub │ │ │ │ (broadcasts: playback state, position ticks, show events) │ │ │ └───────┬─────────────────┬───────────────────────────────────┘ │ │ │ │ │ │ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │ │ │ Show Engine │ │ Audio Manager │ │ Show Store │ │ │ │ (playback │ │ (MPV subprocess │ │ (JSON load/save, │ │ │ │ scheduler, │ │ + IPC bridge) │ │ device registry) │ │ │ │ cue runner) │ │ │ │ │ │ │ └───────┬──────┘ └───────┬──────────┘ └────────────────────┘ │ │ │ │ │ │ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │ │ │ UDP Sender │ │ Analysis Engine │ │ Device Registry │ │ │ │ (per-device │ │ (librosa beat │ │ (SK6812, WS2801, │ │ │ │ dispatcher) │ │ detection + │ │ future: lasers, │ │ │ │ │ │ AI sync) │ │ fog, heads) │ │ │ └───────┬──────┘ └──────────────────┘ └────────────────────┘ │ │ │ │ └──────────┼────────────────────────────────────────────────────────--┘ │ UDP (fire-and-forget, per device) │ ┌──────────┴────────────────────────────────────────────────────────┐ │ Physical Devices │ │ ┌────────────────┐ ┌────────────────┐ ┌──────────────────┐ │ │ │ SK6812 strip │ │ WS2801 strip │ │ Future: lasers │ │ │ │ (300 LEDs, │ │ (160 LEDs, │ │ fog, heads... │ │ │ │ wall) │ │ shelf U) │ │ │ │ │ │ ESP/Pico/RPi │ │ ESP/Pico/RPi │ │ │ │ │ └────────────────┘ └────────────────┘ └──────────────────┘ │ └───────────────────────────────────────────────────────────────────-┘ ``` ### Component Responsibilities | Component | Responsibility | Implementation | |-----------|----------------|----------------| | Timeline UI | DAW-style editor: tracks per device, drag-place animation blocks, beat markers | Vanilla JS or lightweight framework, canvas or DOM | | Transport Bar | Play/pause/seek controls, position display, MPV state mirror | Reads WebSocket position ticks, sends commands | | WebSocket Hub | Single persistent connection between browser and backend; broadcasts state changes and position ticks | FastAPI WebSocket endpoint + connection manager | | Show Engine | Central coordinator: owns playback state machine, fires cues at the right timestamps, calls UDP Sender | asyncio background task, wall-clock + audio-position based scheduling | | Audio Manager | Launches MPV as subprocess, opens IPC pipe, polls `time-pos`, translates seek/play/pause commands | python-mpv-jsonipc or direct named-pipe JSON IPC | | Show Store | Load/save show JSON files, device registry CRUD, animation library catalog | In-process with JSON files on disk (no database needed) | | Analysis Engine | Pre-process audio file: beat detection, onset detection, tempo map, optional AI sync | librosa offline analysis, results stored back into show JSON | | UDP Sender | Per-device packet dispatcher: either animation+params commands OR raw DNRGB/DRGB pixel frames | asyncio UDP socket, one socket per target device | | Device Registry | Catalog of connected devices with type, LED count, IP, port, protocol | Config in show JSON or separate devices.json | ## Recommended Project Structure ``` lightsync/ ├── backend/ │ ├── main.py # FastAPI app, route registration, lifespan │ ├── ws/ │ │ └── hub.py # WebSocket connection manager, broadcast │ ├── engine/ │ │ ├── show_engine.py # Playback state machine, cue scheduler │ │ ├── audio_manager.py # MPV subprocess + IPC bridge │ │ └── scheduler.py # asyncio timing loop, position tracking │ ├── devices/ │ │ ├── base.py # BaseDevice abstract class │ │ ├── sk6812.py # SK6812 RGBW device driver │ │ ├── ws2801.py # WS2801 RGB device driver │ │ └── registry.py # Device registry + factory │ ├── udp/ │ │ ├── sender.py # UDP socket pool, async send │ │ └── protocols.py # DRGB, DNRGB, DRGBW packet builders │ ├── analysis/ │ │ ├── beat_detector.py # librosa beat/onset analysis │ │ └── ai_sync.py # AI sync step (optional, later phase) │ ├── store/ │ │ ├── show_store.py # Load/save show JSON │ │ └── models.py # Pydantic models for show file schema │ └── api/ │ ├── shows.py # REST: CRUD shows, list shows │ ├── devices.py # REST: CRUD devices │ └── analysis.py # REST: trigger analysis run ├── frontend/ │ ├── index.html │ ├── style.css # Terminal aesthetic: dark, monospace │ ├── app.js # App bootstrap, WebSocket client │ ├── timeline.js # Timeline editor component │ ├── transport.js # Playback controls │ └── devices.js # Device panel ├── shows/ # Saved show JSON files (persistent volume) ├── devices.json # Device registry (persistent) └── docker-compose.yml ``` ### Structure Rationale - **engine/:** Owns all stateful runtime behavior. Nothing outside engine/ should hold playback state. - **devices/:** Plugin point for new hardware types — add a file, register in registry.py. - **udp/:** Separated from devices/ because packet encoding is protocol-level, not device-level. SK6812 and WS2801 both use DRGB variants; a future DMX device would add a new protocol here. - **store/models.py:** Pydantic models serve as the schema contract for show files — validation on load, serialization on save. - **frontend/:** Static files served by FastAPI or nginx. No build step required for v1 — keeps it simple. ## Architectural Patterns ### Pattern 1: Position-Driven Cue Scheduling **What:** The Show Engine does not use `asyncio.sleep()` for absolute timing. Instead, it maintains a high-resolution loop that polls MPV's `time-pos` and compares it against the sorted cue list. When `current_pos >= cue.timestamp`, the cue fires. **When to use:** Always — this is the core sync model for this system. **Trade-offs:** Polling MPV IPC adds a small latency (typically 10-50ms depending on poll interval). Pre-fire cues 1-2 frames early to compensate. Simpler and more reliable than trying to align asyncio timers with real audio position, which drifts on any seek or pause. **Example:** ```python async def playback_loop(self): pending_cues = sorted(self.show.cues, key=lambda c: c.timestamp) next_cue_idx = 0 while self.state == PlaybackState.PLAYING: pos = await self.audio_manager.get_position() # polls MPV time-pos while next_cue_idx < len(pending_cues): cue = pending_cues[next_cue_idx] if pos >= cue.timestamp - PRE_FIRE_OFFSET: await self.udp_sender.send_cue(cue) next_cue_idx += 1 else: break await asyncio.sleep(0.005) # 5ms poll — tight enough, light enough ``` ### Pattern 2: Device Abstraction via Abstract Base Class **What:** All device types inherit from `BaseDevice`, which defines the interface. The Show Engine and UDP Sender only speak to `BaseDevice` — they never check device types. **When to use:** Immediately — even for v1 with only two device types. The pattern costs almost nothing now and prevents messy `if device.type == "sk6812"` branches later. **Trade-offs:** Slight indirection overhead (negligible). Requires discipline in keeping the interface stable. **Example:** ```python class BaseDevice: def __init__(self, name: str, led_count: int, ip: str, port: int): ... def encode_frame(self, pixels: list[tuple]) -> bytes: """Encode a full pixel frame to UDP payload bytes.""" raise NotImplementedError def encode_animation_cmd(self, animation: str, params: dict) -> bytes: """Encode an animation+params command packet.""" raise NotImplementedError @property def max_leds_per_packet(self) -> int: """Protocol limit on LEDs per UDP packet.""" raise NotImplementedError class SK6812Device(BaseDevice): """RGBW — uses DRGBW protocol (4 bytes per LED).""" def encode_frame(self, pixels): payload = bytearray([3, 2]) # DRGBW, 2s timeout for r, g, b, w in pixels: payload.extend([r, g, b, w]) return bytes(payload) ``` ### Pattern 3: Dual UDP Mode (Animation Command vs. Raw Frame) **What:** Two fundamentally different UDP payload strategies: - **Animation mode** (manual show playback): send a short command packet — animation name ID + parameters. Microcontroller interprets and renders locally. Low bandwidth (~20 bytes/packet). - **Frame mode** (AI-generated sequences): send the full DRGB/DNRGB pixel buffer for every frame. High bandwidth (300 LEDs * 3 bytes = 900 bytes/frame at 30fps = 27KB/s per device). **When to use:** Animation mode for manually-authored shows. Frame mode for AI sequences where the backend computed exact pixel values. Devices must support both — the firmware handles both packet types. **Trade-offs:** Frame mode requires reliable network on local LAN (generally fine). At 60fps frame mode is 54KB/s per device — still well within LAN capacity. The firmware protocol must include a type discriminator so the microcontroller knows which mode is active. ### Pattern 4: Offline Analysis Pipeline **What:** Beat detection and AI sync are NOT done in real-time during playback. They are offline processing steps triggered before show editing starts. Results are stored back into the show JSON as a `analysis` section. **When to use:** Always — librosa analysis of a 3-minute song takes 2-10 seconds. Doing it at load time and caching the results avoids blocking playback. **Trade-offs:** Show file grows by ~50-200KB per analysed song (beat timestamps are compact). Analysis must be re-triggered if source audio changes. Well worth it for the performance guarantee. **Example flow:** ``` User loads song → Analysis Engine runs librosa offline → Extracts: beat_times[], onset_times[], tempo_bpm, sections[] → Stores in show.analysis JSON block → Timeline UI renders beat markers from this data → Auto-sync snaps cues to show.analysis.beat_times ``` ## Data Flow ### "Play This Show" — Browser Click to LED Pixel ``` User clicks Play in Transport Bar │ ▼ WebSocket message: { type: "transport", cmd: "play" } │ ▼ WebSocket Hub dispatches to Show Engine │ ▼ Show Engine sets state = PLAYING, resets cue pointer │ ▼ Show Engine calls Audio Manager: mpv.play() │ ▼ Audio Manager sends JSON IPC command to MPV process │ ▼ MPV starts audio output (platform audio device) │ ═══════════════════ Playback Loop (5ms tick) ══════════════════ │ ▼ Show Engine polls Audio Manager for time-pos │ ▼ Audio Manager reads MPV IPC: { "data": 4.237 } ← seconds │ ▼ Show Engine compares pos against pending cues │ ├─ No cue yet → continue loop │ └─ Cue fires: { device: "wall-strip", animation: "pulse", params: {...} } │ ▼ UDP Sender looks up device in Device Registry │ ▼ Calls device.encode_animation_cmd(animation, params) │ ▼ Sends UDP packet to device IP:port │ ▼ Microcontroller receives, renders animation on LEDs │ ▼ Show Engine broadcasts position tick via WebSocket Hub │ ▼ Browser Transport Bar updates position display (no round-trip needed) ``` ### Seek Flow ``` User drags playhead to 00:42.500 │ ▼ WebSocket: { type: "transport", cmd: "seek", position: 42.5 } │ ▼ Show Engine pauses cue loop │ ▼ Audio Manager: mpv.seek(42.5, "absolute") │ ▼ Show Engine resets cue pointer to first cue >= 42.5s │ ▼ Show Engine broadcasts: { type: "position", pos: 42.5 } │ ▼ Resumes cue loop ``` ### Analysis Flow ``` User triggers "Analyse Song" via UI │ ▼ REST POST /api/shows/{id}/analyse │ ▼ Analysis Engine loads audio file │ ▼ librosa.load() → y, sr ▼ librosa.beat.beat_track() → tempo, beat_frames ▼ librosa.frames_to_time(beat_frames) → beat_times[] ▼ librosa.onset.onset_detect() → onset_times[] │ ▼ Results stored in show.analysis JSON block (Show Store.save()) │ ▼ WebSocket broadcast: { type: "analysis_complete", showId: "..." } │ ▼ Timeline UI redraws beat markers ``` ### WebSocket vs. Polling **Use WebSocket for:** - Position ticks during playback (backend pushes ~10Hz, browser just renders) - State changes: play/pause/stop/seek events - Analysis completion notifications - Cue fire events (for visual feedback in timeline during playback) **Use REST for:** - Show CRUD (load/save/list) - Device registry CRUD - Trigger analysis run - Export show file **Rationale:** WebSocket is the right choice here — playback position ticks (10-30 per second) make polling impractical. A 100ms poll interval is too coarse for smooth position display; 10ms polling hammers the backend with HTTP overhead. One persistent WebSocket eliminates all of that. REST handles all the "slow path" operations (file I/O, config changes) where request-response semantics are appropriate. ## Show File JSON Schema ### Design Principles - Human-readable and hand-editable - Self-contained: a show file is everything needed to replay a show - Stable: adding new animation types or device types doesn't require schema migration - Analysis data is embedded, not a sidecar file ### Schema ```json { "version": 1, "id": "uuid-v4", "name": "Party Night Vol. 1", "created_at": "2026-04-05T20:00:00Z", "updated_at": "2026-04-05T22:30:00Z", "audio": { "source_type": "file", "path": "/shows/audio/party_night.mp3", "duration_seconds": 214.3, "yt_url": null }, "devices": [ { "id": "wall-strip", "name": "Wall LED Strip", "type": "sk6812", "led_count": 300, "ip": "192.168.1.101", "port": 21324 }, { "id": "shelf-strip", "name": "Shelf U-Turn", "type": "ws2801", "led_count": 160, "ip": "192.168.1.102", "port": 21324 } ], "analysis": { "analysed_at": "2026-04-05T20:15:00Z", "tempo_bpm": 128.0, "beat_times": [0.47, 0.94, 1.41, 1.88], "onset_times": [0.12, 0.47, 0.83, 1.19], "sections": [ { "start": 0.0, "end": 32.0, "label": "intro" }, { "start": 32.0, "end": 96.0, "label": "verse" } ] }, "tracks": [ { "device_id": "wall-strip", "cues": [ { "id": "cue-001", "timestamp": 0.47, "mode": "animation", "animation": "pulse", "params": { "color": [255, 80, 0], "speed": 1.5, "length": 2.0 } }, { "id": "cue-002", "timestamp": 32.0, "mode": "animation", "animation": "chase", "params": { "color_a": [255, 0, 100], "color_b": [0, 0, 255], "speed": 2.0 } } ] }, { "device_id": "shelf-strip", "cues": [ { "id": "cue-003", "timestamp": 0.0, "mode": "animation", "animation": "static", "params": { "color": [30, 0, 60] } } ] } ], "ai_sequences": [] } ``` **Key design decisions:** - `mode: "animation"` vs `mode: "frame_sequence"` — the discriminator for UDP strategy - `ai_sequences` is a top-level array separate from tracks — AI-generated pixel sequences are stored as frame data referenced here, not inline in cues (they'd be too large) - `analysis` block is optional — show works without it (no beat markers, no auto-snap) - `devices` embedded in show file — a show is portable without a separate device config ## Anti-Patterns ### Anti-Pattern 1: asyncio.sleep() for Cue Timing **What people do:** Schedule cues with `asyncio.create_task(send_cue_after(cue, delay_seconds))`. **Why it's wrong:** asyncio event loop can be delayed by other tasks. A 200ms blocking call elsewhere pushes every `sleep` timer by 200ms. More critically: if the user seeks or pauses, all pre-scheduled timers are now wrong and must be cancelled and rescheduled. **Do this instead:** Position-driven scheduling — poll audio position, fire cues when position crosses their timestamp. Handles seeks and pauses automatically with zero extra logic. ### Anti-Pattern 2: Blocking MPV IPC Calls in the Async Loop **What people do:** Use `socket.recv()` (blocking) to read MPV IPC responses inside an asyncio coroutine. **Why it's wrong:** Blocks the entire event loop. WebSocket messages stop processing, UDP sends are delayed, position ticks stall. **Do this instead:** Wrap all MPV IPC communication in `asyncio.to_thread()` or use the non-blocking version of the IPC library. Keep the event loop non-blocking throughout. ### Anti-Pattern 3: Per-Frame WebSocket Position Updates **What people do:** Broadcast audio position on every 5ms tick to the browser. **Why it's wrong:** At 200 position updates/second, WebSocket bandwidth is dominated by tiny position messages. Browser render loop can't use 200Hz position data anyway — it renders at 60fps. **Do this instead:** Send position ticks at 10-15Hz. Browser extrapolates between ticks using `performance.now()` for smooth display. Backend position is authoritative; browser position is display-only interpolation. ### Anti-Pattern 4: One Giant Show Engine Class **What people do:** Put all playback logic — MPV control, UDP sending, analysis, WebSocket broadcasting — into a single class. **Why it's wrong:** Untestable. Audio Manager changes break cue scheduling tests. UDP protocol changes ripple everywhere. **Do this instead:** Each subsystem is its own class with a clean interface. Show Engine orchestrates them but doesn't own their internals. Test each in isolation. ### Anti-Pattern 5: Storing Pixel Frame Data Inline in Show JSON **What people do:** Store AI-generated pixel sequences as base64 blobs inside the show JSON. **Why it's wrong:** A 3-minute song at 30fps * 300 LEDs * 3 bytes = 48MB of frame data — inline in a JSON file meant to be edited. **Do this instead:** Store a reference path in `ai_sequences`. Frame data lives in a separate binary file (e.g., `.fseq` compatible or custom binary). The show JSON stays human-readable and under 100KB. ## Integration Points ### External Services | Service | Integration Pattern | Notes | |---------|---------------------|-------| | MPV | Subprocess + JSON IPC over named pipe (Windows) or Unix socket | python-mpv-jsonipc handles the pipe; key properties: `time-pos`, `duration`, `pause` | | yt-dlp | MPV handles this natively via `--ytdl` flag — no direct Python integration needed | MPV passes YouTube URLs to yt-dlp internally | | librosa | In-process Python call, runs in thread pool to avoid blocking event loop | `asyncio.to_thread(run_analysis, audio_path)` | ### Internal Boundaries | Boundary | Communication | Notes | |----------|---------------|-------| | Show Engine ↔ Audio Manager | Async method calls (get_position, play, pause, seek) | Audio Manager is a dependency-injected service | | Show Engine ↔ UDP Sender | Async method calls (send_cue, send_frame) | UDP Sender takes a BaseDevice + payload | | Show Engine ↔ WebSocket Hub | Hub is passed to engine; engine calls hub.broadcast() | One-directional push; hub doesn't call engine | | WebSocket Hub ↔ Browser | Full-duplex WebSocket; browser sends commands, backend broadcasts state | All browser commands go through a message dispatcher | | Analysis Engine ↔ Show Store | Analysis returns a dict; Show Store merges into show JSON and saves | Loose coupling — analysis doesn't know about the store | ## Suggested Build Order Dependencies flow bottom-up. Each layer can be built and tested before the next. ``` Phase 1 — Foundation (no hardware needed) ├── Show Store: Pydantic models, load/save, device registry ├── FastAPI skeleton: app setup, static file serving, REST stubs └── Frontend skeleton: HTML/CSS terminal aesthetic, WebSocket client stub Phase 2 — Audio + Transport ├── Audio Manager: MPV subprocess, IPC bridge, position polling ├── Show Engine: state machine, position-driven playback loop ├── WebSocket Hub: connection manager, broadcast └── Transport Bar: play/pause/seek UI connected to WebSocket Phase 3 — Device + UDP ├── BaseDevice + SK6812 + WS2801 implementations ├── UDP packet builders (DRGB, DRGBW, DNRGB) ├── UDP Sender: async socket pool └── First end-to-end test: show fires cues → UDP packets → (simulator or device) Phase 4 — Timeline Editor ├── Timeline UI: tracks, cue blocks, drag-place, beat markers ├── Analysis Engine: librosa beat detection, results in show JSON └── Auto-snap cues to beats Phase 5 — AI Sync ├── AI analysis step (on top of beat detection) ├── Frame sequence storage + playback └── Frame-mode UDP sending Phase 6 — Microcontroller Firmware ├── Protocol is stable by now — implement against spec ├── MicroPython on ESP/Pico or Python on RPi └── Both DRGB/DRGBW (frame mode) and animation command mode ``` **Why this order:** 1. Show Store first — everything else depends on the data model. Defining the schema early prevents model changes from cascading. 2. Audio + Transport second — this is the critical path. Without reliable audio position tracking, nothing else can be timed. 3. Device + UDP third — the output layer. Can be tested with a simulator (a Python script that prints received packets) before firmware exists. 4. Timeline fourth — the editing UX. Depends on analysis data and the show file schema being stable. 5. AI sync fifth — optional enhancement layer, builds on beat detection. 6. Firmware last — the protocol is fully validated against a software simulator before burning anything to hardware. ## Scaling Considerations This is a single-operator local system. Scaling is not a concern. The relevant "scale" questions are latency and reliability, not throughput. | Concern | Target | Approach | |---------|--------|----------| | Cue timing precision | ±20ms | Position polling at 5-10ms intervals, pre-fire offset | | UDP latency | <5ms on LAN | Fire-and-forget UDP, no ACK waiting | | WebSocket position tick | 10-15Hz | Sufficient for smooth UI; browser interpolates | | Analysis time | <10s for 5min song | Offline, runs in thread pool, results cached in show JSON | | Frame-mode bandwidth | ~27KB/s per device at 30fps (300 LEDs) | Well within LAN; 5 devices = 135KB/s, trivial | ## Sources - [WLED UDP Realtime Protocol](https://kno.wled.ge/interfaces/udp-realtime/) — DRGB/DNRGB/DRGBW packet format (HIGH confidence — official documentation) - [python-mpv-jsonipc](https://github.com/iwalton3/python-mpv-jsonipc) — IPC bridge library for MPV subprocess (HIGH confidence — active project, used by Jellyfin) - [MPV JSON IPC Protocol](https://mpv.io/manual/master/) — `time-pos`, `playback-time`, seek commands (HIGH confidence — official docs) - [librosa beat_track](https://librosa.org/doc/main/generated/librosa.beat.beat_track.html) — returns beat frame indices, convertible to timestamps via `frames_to_time` (HIGH confidence — official docs) - [FastAPI WebSocket background tasks](https://hexshift.medium.com/implementing-background-tasks-with-websockets-in-fastapi-034cdf803430) — asyncio.create_task pattern alongside websocket receive loop (MEDIUM confidence — community article) - [audio-reactive-led-strip](https://github.com/scottlawsonbc/audio-reactive-led-strip) — reference Python + UDP + ESP8266 architecture (MEDIUM confidence — older project but patterns are standard) - [xLights FSEQ format](https://github.com/Cryptkeeper/fseq-file-format) — reference for binary frame sequence format (LOW confidence — using as inspiration only, not adopting format) --- *Architecture research for: LightSync — music-to-light synchronization system* *Researched: 2026-04-05*