28 KiB
Architecture Research
Domain: Music-to-light synchronization system (show controller) Researched: 2026-04-05 Confidence: HIGH (core patterns are well-established; AI sync specifics are MEDIUM)
Standard Architecture
System Overview
┌─────────────────────────────────────────────────────────────────────┐
│ Browser Frontend │
│ ┌──────────────┐ ┌────────────────┐ ┌──────────────────────┐ │
│ │ Timeline UI │ │ Transport Bar │ │ Device + Show Panel │ │
│ │ (DAW editor) │ │ (play/pause/ │ │ (registry + library)│ │
│ │ │ │ seek/pos) │ │ │ │
│ └──────┬───────┘ └───────┬────────┘ └──────────┬───────────┘ │
│ │ │ │ │
│ └──────────────────┴───────────────────────┘ │
│ │ WebSocket (full-duplex) │
└────────────────────────────┼────────────────────────────────────────┘
│
┌────────────────────────────┼────────────────────────────────────────┐
│ FastAPI Backend │
│ ┌─────────────────────────┴──────────────────────────────────┐ │
│ │ WebSocket Hub │ │
│ │ (broadcasts: playback state, position ticks, show events) │ │
│ └───────┬─────────────────┬───────────────────────────────────┘ │
│ │ │ │
│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │
│ │ Show Engine │ │ Audio Manager │ │ Show Store │ │
│ │ (playback │ │ (MPV subprocess │ │ (JSON load/save, │ │
│ │ scheduler, │ │ + IPC bridge) │ │ device registry) │ │
│ │ cue runner) │ │ │ │ │ │
│ └───────┬──────┘ └───────┬──────────┘ └────────────────────┘ │
│ │ │ │
│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │
│ │ UDP Sender │ │ Analysis Engine │ │ Device Registry │ │
│ │ (per-device │ │ (librosa beat │ │ (SK6812, WS2801, │ │
│ │ dispatcher) │ │ detection + │ │ future: lasers, │ │
│ │ │ │ AI sync) │ │ fog, heads) │ │
│ └───────┬──────┘ └──────────────────┘ └────────────────────┘ │
│ │ │
└──────────┼────────────────────────────────────────────────────────--┘
│ UDP (fire-and-forget, per device)
│
┌──────────┴────────────────────────────────────────────────────────┐
│ Physical Devices │
│ ┌────────────────┐ ┌────────────────┐ ┌──────────────────┐ │
│ │ SK6812 strip │ │ WS2801 strip │ │ Future: lasers │ │
│ │ (300 LEDs, │ │ (160 LEDs, │ │ fog, heads... │ │
│ │ wall) │ │ shelf U) │ │ │ │
│ │ ESP/Pico/RPi │ │ ESP/Pico/RPi │ │ │ │
│ └────────────────┘ └────────────────┘ └──────────────────┘ │
└───────────────────────────────────────────────────────────────────-┘
Component Responsibilities
| Component | Responsibility | Implementation |
|---|---|---|
| Timeline UI | DAW-style editor: tracks per device, drag-place animation blocks, beat markers | Vanilla JS or lightweight framework, canvas or DOM |
| Transport Bar | Play/pause/seek controls, position display, MPV state mirror | Reads WebSocket position ticks, sends commands |
| WebSocket Hub | Single persistent connection between browser and backend; broadcasts state changes and position ticks | FastAPI WebSocket endpoint + connection manager |
| Show Engine | Central coordinator: owns playback state machine, fires cues at the right timestamps, calls UDP Sender | asyncio background task, wall-clock + audio-position based scheduling |
| Audio Manager | Launches MPV as subprocess, opens IPC pipe, polls time-pos, translates seek/play/pause commands |
python-mpv-jsonipc or direct named-pipe JSON IPC |
| Show Store | Load/save show JSON files, device registry CRUD, animation library catalog | In-process with JSON files on disk (no database needed) |
| Analysis Engine | Pre-process audio file: beat detection, onset detection, tempo map, optional AI sync | librosa offline analysis, results stored back into show JSON |
| UDP Sender | Per-device packet dispatcher: either animation+params commands OR raw DNRGB/DRGB pixel frames | asyncio UDP socket, one socket per target device |
| Device Registry | Catalog of connected devices with type, LED count, IP, port, protocol | Config in show JSON or separate devices.json |
Recommended Project Structure
lightsync/
├── backend/
│ ├── main.py # FastAPI app, route registration, lifespan
│ ├── ws/
│ │ └── hub.py # WebSocket connection manager, broadcast
│ ├── engine/
│ │ ├── show_engine.py # Playback state machine, cue scheduler
│ │ ├── audio_manager.py # MPV subprocess + IPC bridge
│ │ └── scheduler.py # asyncio timing loop, position tracking
│ ├── devices/
│ │ ├── base.py # BaseDevice abstract class
│ │ ├── sk6812.py # SK6812 RGBW device driver
│ │ ├── ws2801.py # WS2801 RGB device driver
│ │ └── registry.py # Device registry + factory
│ ├── udp/
│ │ ├── sender.py # UDP socket pool, async send
│ │ └── protocols.py # DRGB, DNRGB, DRGBW packet builders
│ ├── analysis/
│ │ ├── beat_detector.py # librosa beat/onset analysis
│ │ └── ai_sync.py # AI sync step (optional, later phase)
│ ├── store/
│ │ ├── show_store.py # Load/save show JSON
│ │ └── models.py # Pydantic models for show file schema
│ └── api/
│ ├── shows.py # REST: CRUD shows, list shows
│ ├── devices.py # REST: CRUD devices
│ └── analysis.py # REST: trigger analysis run
├── frontend/
│ ├── index.html
│ ├── style.css # Terminal aesthetic: dark, monospace
│ ├── app.js # App bootstrap, WebSocket client
│ ├── timeline.js # Timeline editor component
│ ├── transport.js # Playback controls
│ └── devices.js # Device panel
├── shows/ # Saved show JSON files (persistent volume)
├── devices.json # Device registry (persistent)
└── docker-compose.yml
Structure Rationale
- engine/: Owns all stateful runtime behavior. Nothing outside engine/ should hold playback state.
- devices/: Plugin point for new hardware types — add a file, register in registry.py.
- udp/: Separated from devices/ because packet encoding is protocol-level, not device-level. SK6812 and WS2801 both use DRGB variants; a future DMX device would add a new protocol here.
- store/models.py: Pydantic models serve as the schema contract for show files — validation on load, serialization on save.
- frontend/: Static files served by FastAPI or nginx. No build step required for v1 — keeps it simple.
Architectural Patterns
Pattern 1: Position-Driven Cue Scheduling
What: The Show Engine does not use asyncio.sleep() for absolute timing. Instead, it maintains a high-resolution loop that polls MPV's time-pos and compares it against the sorted cue list. When current_pos >= cue.timestamp, the cue fires.
When to use: Always — this is the core sync model for this system.
Trade-offs: Polling MPV IPC adds a small latency (typically 10-50ms depending on poll interval). Pre-fire cues 1-2 frames early to compensate. Simpler and more reliable than trying to align asyncio timers with real audio position, which drifts on any seek or pause.
Example:
async def playback_loop(self):
pending_cues = sorted(self.show.cues, key=lambda c: c.timestamp)
next_cue_idx = 0
while self.state == PlaybackState.PLAYING:
pos = await self.audio_manager.get_position() # polls MPV time-pos
while next_cue_idx < len(pending_cues):
cue = pending_cues[next_cue_idx]
if pos >= cue.timestamp - PRE_FIRE_OFFSET:
await self.udp_sender.send_cue(cue)
next_cue_idx += 1
else:
break
await asyncio.sleep(0.005) # 5ms poll — tight enough, light enough
Pattern 2: Device Abstraction via Abstract Base Class
What: All device types inherit from BaseDevice, which defines the interface. The Show Engine and UDP Sender only speak to BaseDevice — they never check device types.
When to use: Immediately — even for v1 with only two device types. The pattern costs almost nothing now and prevents messy if device.type == "sk6812" branches later.
Trade-offs: Slight indirection overhead (negligible). Requires discipline in keeping the interface stable.
Example:
class BaseDevice:
def __init__(self, name: str, led_count: int, ip: str, port: int):
...
def encode_frame(self, pixels: list[tuple]) -> bytes:
"""Encode a full pixel frame to UDP payload bytes."""
raise NotImplementedError
def encode_animation_cmd(self, animation: str, params: dict) -> bytes:
"""Encode an animation+params command packet."""
raise NotImplementedError
@property
def max_leds_per_packet(self) -> int:
"""Protocol limit on LEDs per UDP packet."""
raise NotImplementedError
class SK6812Device(BaseDevice):
"""RGBW — uses DRGBW protocol (4 bytes per LED)."""
def encode_frame(self, pixels):
payload = bytearray([3, 2]) # DRGBW, 2s timeout
for r, g, b, w in pixels:
payload.extend([r, g, b, w])
return bytes(payload)
Pattern 3: Dual UDP Mode (Animation Command vs. Raw Frame)
What: Two fundamentally different UDP payload strategies:
- Animation mode (manual show playback): send a short command packet — animation name ID + parameters. Microcontroller interprets and renders locally. Low bandwidth (~20 bytes/packet).
- Frame mode (AI-generated sequences): send the full DRGB/DNRGB pixel buffer for every frame. High bandwidth (300 LEDs * 3 bytes = 900 bytes/frame at 30fps = 27KB/s per device).
When to use: Animation mode for manually-authored shows. Frame mode for AI sequences where the backend computed exact pixel values. Devices must support both — the firmware handles both packet types.
Trade-offs: Frame mode requires reliable network on local LAN (generally fine). At 60fps frame mode is 54KB/s per device — still well within LAN capacity. The firmware protocol must include a type discriminator so the microcontroller knows which mode is active.
Pattern 4: Offline Analysis Pipeline
What: Beat detection and AI sync are NOT done in real-time during playback. They are offline processing steps triggered before show editing starts. Results are stored back into the show JSON as a analysis section.
When to use: Always — librosa analysis of a 3-minute song takes 2-10 seconds. Doing it at load time and caching the results avoids blocking playback.
Trade-offs: Show file grows by ~50-200KB per analysed song (beat timestamps are compact). Analysis must be re-triggered if source audio changes. Well worth it for the performance guarantee.
Example flow:
User loads song
→ Analysis Engine runs librosa offline
→ Extracts: beat_times[], onset_times[], tempo_bpm, sections[]
→ Stores in show.analysis JSON block
→ Timeline UI renders beat markers from this data
→ Auto-sync snaps cues to show.analysis.beat_times
Data Flow
"Play This Show" — Browser Click to LED Pixel
User clicks Play in Transport Bar
│
▼ WebSocket message: { type: "transport", cmd: "play" }
│
▼ WebSocket Hub dispatches to Show Engine
│
▼ Show Engine sets state = PLAYING, resets cue pointer
│
▼ Show Engine calls Audio Manager: mpv.play()
│
▼ Audio Manager sends JSON IPC command to MPV process
│
▼ MPV starts audio output (platform audio device)
│
═══════════════════ Playback Loop (5ms tick) ══════════════════
│
▼ Show Engine polls Audio Manager for time-pos
│
▼ Audio Manager reads MPV IPC: { "data": 4.237 } ← seconds
│
▼ Show Engine compares pos against pending cues
│
├─ No cue yet → continue loop
│
└─ Cue fires: { device: "wall-strip", animation: "pulse", params: {...} }
│
▼ UDP Sender looks up device in Device Registry
│
▼ Calls device.encode_animation_cmd(animation, params)
│
▼ Sends UDP packet to device IP:port
│
▼ Microcontroller receives, renders animation on LEDs
│
▼ Show Engine broadcasts position tick via WebSocket Hub
│
▼ Browser Transport Bar updates position display (no round-trip needed)
Seek Flow
User drags playhead to 00:42.500
│
▼ WebSocket: { type: "transport", cmd: "seek", position: 42.5 }
│
▼ Show Engine pauses cue loop
│
▼ Audio Manager: mpv.seek(42.5, "absolute")
│
▼ Show Engine resets cue pointer to first cue >= 42.5s
│
▼ Show Engine broadcasts: { type: "position", pos: 42.5 }
│
▼ Resumes cue loop
Analysis Flow
User triggers "Analyse Song" via UI
│
▼ REST POST /api/shows/{id}/analyse
│
▼ Analysis Engine loads audio file
│
▼ librosa.load() → y, sr
▼ librosa.beat.beat_track() → tempo, beat_frames
▼ librosa.frames_to_time(beat_frames) → beat_times[]
▼ librosa.onset.onset_detect() → onset_times[]
│
▼ Results stored in show.analysis JSON block (Show Store.save())
│
▼ WebSocket broadcast: { type: "analysis_complete", showId: "..." }
│
▼ Timeline UI redraws beat markers
WebSocket vs. Polling
Use WebSocket for:
- Position ticks during playback (backend pushes ~10Hz, browser just renders)
- State changes: play/pause/stop/seek events
- Analysis completion notifications
- Cue fire events (for visual feedback in timeline during playback)
Use REST for:
- Show CRUD (load/save/list)
- Device registry CRUD
- Trigger analysis run
- Export show file
Rationale: WebSocket is the right choice here — playback position ticks (10-30 per second) make polling impractical. A 100ms poll interval is too coarse for smooth position display; 10ms polling hammers the backend with HTTP overhead. One persistent WebSocket eliminates all of that. REST handles all the "slow path" operations (file I/O, config changes) where request-response semantics are appropriate.
Show File JSON Schema
Design Principles
- Human-readable and hand-editable
- Self-contained: a show file is everything needed to replay a show
- Stable: adding new animation types or device types doesn't require schema migration
- Analysis data is embedded, not a sidecar file
Schema
{
"version": 1,
"id": "uuid-v4",
"name": "Party Night Vol. 1",
"created_at": "2026-04-05T20:00:00Z",
"updated_at": "2026-04-05T22:30:00Z",
"audio": {
"source_type": "file",
"path": "/shows/audio/party_night.mp3",
"duration_seconds": 214.3,
"yt_url": null
},
"devices": [
{
"id": "wall-strip",
"name": "Wall LED Strip",
"type": "sk6812",
"led_count": 300,
"ip": "192.168.1.101",
"port": 21324
},
{
"id": "shelf-strip",
"name": "Shelf U-Turn",
"type": "ws2801",
"led_count": 160,
"ip": "192.168.1.102",
"port": 21324
}
],
"analysis": {
"analysed_at": "2026-04-05T20:15:00Z",
"tempo_bpm": 128.0,
"beat_times": [0.47, 0.94, 1.41, 1.88],
"onset_times": [0.12, 0.47, 0.83, 1.19],
"sections": [
{ "start": 0.0, "end": 32.0, "label": "intro" },
{ "start": 32.0, "end": 96.0, "label": "verse" }
]
},
"tracks": [
{
"device_id": "wall-strip",
"cues": [
{
"id": "cue-001",
"timestamp": 0.47,
"mode": "animation",
"animation": "pulse",
"params": {
"color": [255, 80, 0],
"speed": 1.5,
"length": 2.0
}
},
{
"id": "cue-002",
"timestamp": 32.0,
"mode": "animation",
"animation": "chase",
"params": {
"color_a": [255, 0, 100],
"color_b": [0, 0, 255],
"speed": 2.0
}
}
]
},
{
"device_id": "shelf-strip",
"cues": [
{
"id": "cue-003",
"timestamp": 0.0,
"mode": "animation",
"animation": "static",
"params": { "color": [30, 0, 60] }
}
]
}
],
"ai_sequences": []
}
Key design decisions:
mode: "animation"vsmode: "frame_sequence"— the discriminator for UDP strategyai_sequencesis a top-level array separate from tracks — AI-generated pixel sequences are stored as frame data referenced here, not inline in cues (they'd be too large)analysisblock is optional — show works without it (no beat markers, no auto-snap)devicesembedded in show file — a show is portable without a separate device config
Anti-Patterns
Anti-Pattern 1: asyncio.sleep() for Cue Timing
What people do: Schedule cues with asyncio.create_task(send_cue_after(cue, delay_seconds)).
Why it's wrong: asyncio event loop can be delayed by other tasks. A 200ms blocking call elsewhere pushes every sleep timer by 200ms. More critically: if the user seeks or pauses, all pre-scheduled timers are now wrong and must be cancelled and rescheduled.
Do this instead: Position-driven scheduling — poll audio position, fire cues when position crosses their timestamp. Handles seeks and pauses automatically with zero extra logic.
Anti-Pattern 2: Blocking MPV IPC Calls in the Async Loop
What people do: Use socket.recv() (blocking) to read MPV IPC responses inside an asyncio coroutine.
Why it's wrong: Blocks the entire event loop. WebSocket messages stop processing, UDP sends are delayed, position ticks stall.
Do this instead: Wrap all MPV IPC communication in asyncio.to_thread() or use the non-blocking version of the IPC library. Keep the event loop non-blocking throughout.
Anti-Pattern 3: Per-Frame WebSocket Position Updates
What people do: Broadcast audio position on every 5ms tick to the browser.
Why it's wrong: At 200 position updates/second, WebSocket bandwidth is dominated by tiny position messages. Browser render loop can't use 200Hz position data anyway — it renders at 60fps.
Do this instead: Send position ticks at 10-15Hz. Browser extrapolates between ticks using performance.now() for smooth display. Backend position is authoritative; browser position is display-only interpolation.
Anti-Pattern 4: One Giant Show Engine Class
What people do: Put all playback logic — MPV control, UDP sending, analysis, WebSocket broadcasting — into a single class.
Why it's wrong: Untestable. Audio Manager changes break cue scheduling tests. UDP protocol changes ripple everywhere.
Do this instead: Each subsystem is its own class with a clean interface. Show Engine orchestrates them but doesn't own their internals. Test each in isolation.
Anti-Pattern 5: Storing Pixel Frame Data Inline in Show JSON
What people do: Store AI-generated pixel sequences as base64 blobs inside the show JSON.
Why it's wrong: A 3-minute song at 30fps * 300 LEDs * 3 bytes = 48MB of frame data — inline in a JSON file meant to be edited.
Do this instead: Store a reference path in ai_sequences. Frame data lives in a separate binary file (e.g., .fseq compatible or custom binary). The show JSON stays human-readable and under 100KB.
Integration Points
External Services
| Service | Integration Pattern | Notes |
|---|---|---|
| MPV | Subprocess + JSON IPC over named pipe (Windows) or Unix socket | python-mpv-jsonipc handles the pipe; key properties: time-pos, duration, pause |
| yt-dlp | MPV handles this natively via --ytdl flag — no direct Python integration needed |
MPV passes YouTube URLs to yt-dlp internally |
| librosa | In-process Python call, runs in thread pool to avoid blocking event loop | asyncio.to_thread(run_analysis, audio_path) |
Internal Boundaries
| Boundary | Communication | Notes |
|---|---|---|
| Show Engine ↔ Audio Manager | Async method calls (get_position, play, pause, seek) | Audio Manager is a dependency-injected service |
| Show Engine ↔ UDP Sender | Async method calls (send_cue, send_frame) | UDP Sender takes a BaseDevice + payload |
| Show Engine ↔ WebSocket Hub | Hub is passed to engine; engine calls hub.broadcast() | One-directional push; hub doesn't call engine |
| WebSocket Hub ↔ Browser | Full-duplex WebSocket; browser sends commands, backend broadcasts state | All browser commands go through a message dispatcher |
| Analysis Engine ↔ Show Store | Analysis returns a dict; Show Store merges into show JSON and saves | Loose coupling — analysis doesn't know about the store |
Suggested Build Order
Dependencies flow bottom-up. Each layer can be built and tested before the next.
Phase 1 — Foundation (no hardware needed)
├── Show Store: Pydantic models, load/save, device registry
├── FastAPI skeleton: app setup, static file serving, REST stubs
└── Frontend skeleton: HTML/CSS terminal aesthetic, WebSocket client stub
Phase 2 — Audio + Transport
├── Audio Manager: MPV subprocess, IPC bridge, position polling
├── Show Engine: state machine, position-driven playback loop
├── WebSocket Hub: connection manager, broadcast
└── Transport Bar: play/pause/seek UI connected to WebSocket
Phase 3 — Device + UDP
├── BaseDevice + SK6812 + WS2801 implementations
├── UDP packet builders (DRGB, DRGBW, DNRGB)
├── UDP Sender: async socket pool
└── First end-to-end test: show fires cues → UDP packets → (simulator or device)
Phase 4 — Timeline Editor
├── Timeline UI: tracks, cue blocks, drag-place, beat markers
├── Analysis Engine: librosa beat detection, results in show JSON
└── Auto-snap cues to beats
Phase 5 — AI Sync
├── AI analysis step (on top of beat detection)
├── Frame sequence storage + playback
└── Frame-mode UDP sending
Phase 6 — Microcontroller Firmware
├── Protocol is stable by now — implement against spec
├── MicroPython on ESP/Pico or Python on RPi
└── Both DRGB/DRGBW (frame mode) and animation command mode
Why this order:
- Show Store first — everything else depends on the data model. Defining the schema early prevents model changes from cascading.
- Audio + Transport second — this is the critical path. Without reliable audio position tracking, nothing else can be timed.
- Device + UDP third — the output layer. Can be tested with a simulator (a Python script that prints received packets) before firmware exists.
- Timeline fourth — the editing UX. Depends on analysis data and the show file schema being stable.
- AI sync fifth — optional enhancement layer, builds on beat detection.
- Firmware last — the protocol is fully validated against a software simulator before burning anything to hardware.
Scaling Considerations
This is a single-operator local system. Scaling is not a concern. The relevant "scale" questions are latency and reliability, not throughput.
| Concern | Target | Approach |
|---|---|---|
| Cue timing precision | ±20ms | Position polling at 5-10ms intervals, pre-fire offset |
| UDP latency | <5ms on LAN | Fire-and-forget UDP, no ACK waiting |
| WebSocket position tick | 10-15Hz | Sufficient for smooth UI; browser interpolates |
| Analysis time | <10s for 5min song | Offline, runs in thread pool, results cached in show JSON |
| Frame-mode bandwidth | ~27KB/s per device at 30fps (300 LEDs) | Well within LAN; 5 devices = 135KB/s, trivial |
Sources
- WLED UDP Realtime Protocol — DRGB/DNRGB/DRGBW packet format (HIGH confidence — official documentation)
- python-mpv-jsonipc — IPC bridge library for MPV subprocess (HIGH confidence — active project, used by Jellyfin)
- MPV JSON IPC Protocol —
time-pos,playback-time, seek commands (HIGH confidence — official docs) - librosa beat_track — returns beat frame indices, convertible to timestamps via
frames_to_time(HIGH confidence — official docs) - FastAPI WebSocket background tasks — asyncio.create_task pattern alongside websocket receive loop (MEDIUM confidence — community article)
- audio-reactive-led-strip — reference Python + UDP + ESP8266 architecture (MEDIUM confidence — older project but patterns are standard)
- xLights FSEQ format — reference for binary frame sequence format (LOW confidence — using as inspiration only, not adopting format)
Architecture research for: LightSync — music-to-light synchronization system Researched: 2026-04-05