docs: add domain research — stack, features, architecture, pitfalls
This commit is contained in:
Claude
554
.planning/research/ARCHITECTURE.md
Normal file
554
.planning/research/ARCHITECTURE.md
Normal file
@@ -0,0 +1,554 @@
|
||||
# Architecture Research
|
||||
|
||||
**Domain:** Music-to-light synchronization system (show controller)
|
||||
**Researched:** 2026-04-05
|
||||
**Confidence:** HIGH (core patterns are well-established; AI sync specifics are MEDIUM)
|
||||
|
||||
## Standard Architecture
|
||||
|
||||
### System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Browser Frontend │
|
||||
│ ┌──────────────┐ ┌────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ Timeline UI │ │ Transport Bar │ │ Device + Show Panel │ │
|
||||
│ │ (DAW editor) │ │ (play/pause/ │ │ (registry + library)│ │
|
||||
│ │ │ │ seek/pos) │ │ │ │
|
||||
│ └──────┬───────┘ └───────┬────────┘ └──────────┬───────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └──────────────────┴───────────────────────┘ │
|
||||
│ │ WebSocket (full-duplex) │
|
||||
└────────────────────────────┼────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────────┼────────────────────────────────────────┐
|
||||
│ FastAPI Backend │
|
||||
│ ┌─────────────────────────┴──────────────────────────────────┐ │
|
||||
│ │ WebSocket Hub │ │
|
||||
│ │ (broadcasts: playback state, position ticks, show events) │ │
|
||||
│ └───────┬─────────────────┬───────────────────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │
|
||||
│ │ Show Engine │ │ Audio Manager │ │ Show Store │ │
|
||||
│ │ (playback │ │ (MPV subprocess │ │ (JSON load/save, │ │
|
||||
│ │ scheduler, │ │ + IPC bridge) │ │ device registry) │ │
|
||||
│ │ cue runner) │ │ │ │ │ │
|
||||
│ └───────┬──────┘ └───────┬──────────┘ └────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌───────┴──────┐ ┌───────┴──────────┐ ┌────────────────────┐ │
|
||||
│ │ UDP Sender │ │ Analysis Engine │ │ Device Registry │ │
|
||||
│ │ (per-device │ │ (librosa beat │ │ (SK6812, WS2801, │ │
|
||||
│ │ dispatcher) │ │ detection + │ │ future: lasers, │ │
|
||||
│ │ │ │ AI sync) │ │ fog, heads) │ │
|
||||
│ └───────┬──────┘ └──────────────────┘ └────────────────────┘ │
|
||||
│ │ │
|
||||
└──────────┼────────────────────────────────────────────────────────--┘
|
||||
│ UDP (fire-and-forget, per device)
|
||||
│
|
||||
┌──────────┴────────────────────────────────────────────────────────┐
|
||||
│ Physical Devices │
|
||||
│ ┌────────────────┐ ┌────────────────┐ ┌──────────────────┐ │
|
||||
│ │ SK6812 strip │ │ WS2801 strip │ │ Future: lasers │ │
|
||||
│ │ (300 LEDs, │ │ (160 LEDs, │ │ fog, heads... │ │
|
||||
│ │ wall) │ │ shelf U) │ │ │ │
|
||||
│ │ ESP/Pico/RPi │ │ ESP/Pico/RPi │ │ │ │
|
||||
│ └────────────────┘ └────────────────┘ └──────────────────┘ │
|
||||
└───────────────────────────────────────────────────────────────────-┘
|
||||
```
|
||||
|
||||
### Component Responsibilities
|
||||
|
||||
| Component | Responsibility | Implementation |
|
||||
|-----------|----------------|----------------|
|
||||
| Timeline UI | DAW-style editor: tracks per device, drag-place animation blocks, beat markers | Vanilla JS or lightweight framework, canvas or DOM |
|
||||
| Transport Bar | Play/pause/seek controls, position display, MPV state mirror | Reads WebSocket position ticks, sends commands |
|
||||
| WebSocket Hub | Single persistent connection between browser and backend; broadcasts state changes and position ticks | FastAPI WebSocket endpoint + connection manager |
|
||||
| Show Engine | Central coordinator: owns playback state machine, fires cues at the right timestamps, calls UDP Sender | asyncio background task, wall-clock + audio-position based scheduling |
|
||||
| Audio Manager | Launches MPV as subprocess, opens IPC pipe, polls `time-pos`, translates seek/play/pause commands | python-mpv-jsonipc or direct named-pipe JSON IPC |
|
||||
| Show Store | Load/save show JSON files, device registry CRUD, animation library catalog | In-process with JSON files on disk (no database needed) |
|
||||
| Analysis Engine | Pre-process audio file: beat detection, onset detection, tempo map, optional AI sync | librosa offline analysis, results stored back into show JSON |
|
||||
| UDP Sender | Per-device packet dispatcher: either animation+params commands OR raw DNRGB/DRGB pixel frames | asyncio UDP socket, one socket per target device |
|
||||
| Device Registry | Catalog of connected devices with type, LED count, IP, port, protocol | Config in show JSON or separate devices.json |
|
||||
|
||||
## Recommended Project Structure
|
||||
|
||||
```
|
||||
lightsync/
|
||||
├── backend/
|
||||
│ ├── main.py # FastAPI app, route registration, lifespan
|
||||
│ ├── ws/
|
||||
│ │ └── hub.py # WebSocket connection manager, broadcast
|
||||
│ ├── engine/
|
||||
│ │ ├── show_engine.py # Playback state machine, cue scheduler
|
||||
│ │ ├── audio_manager.py # MPV subprocess + IPC bridge
|
||||
│ │ └── scheduler.py # asyncio timing loop, position tracking
|
||||
│ ├── devices/
|
||||
│ │ ├── base.py # BaseDevice abstract class
|
||||
│ │ ├── sk6812.py # SK6812 RGBW device driver
|
||||
│ │ ├── ws2801.py # WS2801 RGB device driver
|
||||
│ │ └── registry.py # Device registry + factory
|
||||
│ ├── udp/
|
||||
│ │ ├── sender.py # UDP socket pool, async send
|
||||
│ │ └── protocols.py # DRGB, DNRGB, DRGBW packet builders
|
||||
│ ├── analysis/
|
||||
│ │ ├── beat_detector.py # librosa beat/onset analysis
|
||||
│ │ └── ai_sync.py # AI sync step (optional, later phase)
|
||||
│ ├── store/
|
||||
│ │ ├── show_store.py # Load/save show JSON
|
||||
│ │ └── models.py # Pydantic models for show file schema
|
||||
│ └── api/
|
||||
│ ├── shows.py # REST: CRUD shows, list shows
|
||||
│ ├── devices.py # REST: CRUD devices
|
||||
│ └── analysis.py # REST: trigger analysis run
|
||||
├── frontend/
|
||||
│ ├── index.html
|
||||
│ ├── style.css # Terminal aesthetic: dark, monospace
|
||||
│ ├── app.js # App bootstrap, WebSocket client
|
||||
│ ├── timeline.js # Timeline editor component
|
||||
│ ├── transport.js # Playback controls
|
||||
│ └── devices.js # Device panel
|
||||
├── shows/ # Saved show JSON files (persistent volume)
|
||||
├── devices.json # Device registry (persistent)
|
||||
└── docker-compose.yml
|
||||
```
|
||||
|
||||
### Structure Rationale
|
||||
|
||||
- **engine/:** Owns all stateful runtime behavior. Nothing outside engine/ should hold playback state.
|
||||
- **devices/:** Plugin point for new hardware types — add a file, register in registry.py.
|
||||
- **udp/:** Separated from devices/ because packet encoding is protocol-level, not device-level. SK6812 and WS2801 both use DRGB variants; a future DMX device would add a new protocol here.
|
||||
- **store/models.py:** Pydantic models serve as the schema contract for show files — validation on load, serialization on save.
|
||||
- **frontend/:** Static files served by FastAPI or nginx. No build step required for v1 — keeps it simple.
|
||||
|
||||
## Architectural Patterns
|
||||
|
||||
### Pattern 1: Position-Driven Cue Scheduling
|
||||
|
||||
**What:** The Show Engine does not use `asyncio.sleep()` for absolute timing. Instead, it maintains a high-resolution loop that polls MPV's `time-pos` and compares it against the sorted cue list. When `current_pos >= cue.timestamp`, the cue fires.
|
||||
|
||||
**When to use:** Always — this is the core sync model for this system.
|
||||
|
||||
**Trade-offs:** Polling MPV IPC adds a small latency (typically 10-50ms depending on poll interval). Pre-fire cues 1-2 frames early to compensate. Simpler and more reliable than trying to align asyncio timers with real audio position, which drifts on any seek or pause.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
async def playback_loop(self):
|
||||
pending_cues = sorted(self.show.cues, key=lambda c: c.timestamp)
|
||||
next_cue_idx = 0
|
||||
while self.state == PlaybackState.PLAYING:
|
||||
pos = await self.audio_manager.get_position() # polls MPV time-pos
|
||||
while next_cue_idx < len(pending_cues):
|
||||
cue = pending_cues[next_cue_idx]
|
||||
if pos >= cue.timestamp - PRE_FIRE_OFFSET:
|
||||
await self.udp_sender.send_cue(cue)
|
||||
next_cue_idx += 1
|
||||
else:
|
||||
break
|
||||
await asyncio.sleep(0.005) # 5ms poll — tight enough, light enough
|
||||
```
|
||||
|
||||
### Pattern 2: Device Abstraction via Abstract Base Class
|
||||
|
||||
**What:** All device types inherit from `BaseDevice`, which defines the interface. The Show Engine and UDP Sender only speak to `BaseDevice` — they never check device types.
|
||||
|
||||
**When to use:** Immediately — even for v1 with only two device types. The pattern costs almost nothing now and prevents messy `if device.type == "sk6812"` branches later.
|
||||
|
||||
**Trade-offs:** Slight indirection overhead (negligible). Requires discipline in keeping the interface stable.
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
class BaseDevice:
|
||||
def __init__(self, name: str, led_count: int, ip: str, port: int):
|
||||
...
|
||||
|
||||
def encode_frame(self, pixels: list[tuple]) -> bytes:
|
||||
"""Encode a full pixel frame to UDP payload bytes."""
|
||||
raise NotImplementedError
|
||||
|
||||
def encode_animation_cmd(self, animation: str, params: dict) -> bytes:
|
||||
"""Encode an animation+params command packet."""
|
||||
raise NotImplementedError
|
||||
|
||||
@property
|
||||
def max_leds_per_packet(self) -> int:
|
||||
"""Protocol limit on LEDs per UDP packet."""
|
||||
raise NotImplementedError
|
||||
|
||||
class SK6812Device(BaseDevice):
|
||||
"""RGBW — uses DRGBW protocol (4 bytes per LED)."""
|
||||
def encode_frame(self, pixels):
|
||||
payload = bytearray([3, 2]) # DRGBW, 2s timeout
|
||||
for r, g, b, w in pixels:
|
||||
payload.extend([r, g, b, w])
|
||||
return bytes(payload)
|
||||
```
|
||||
|
||||
### Pattern 3: Dual UDP Mode (Animation Command vs. Raw Frame)
|
||||
|
||||
**What:** Two fundamentally different UDP payload strategies:
|
||||
- **Animation mode** (manual show playback): send a short command packet — animation name ID + parameters. Microcontroller interprets and renders locally. Low bandwidth (~20 bytes/packet).
|
||||
- **Frame mode** (AI-generated sequences): send the full DRGB/DNRGB pixel buffer for every frame. High bandwidth (300 LEDs * 3 bytes = 900 bytes/frame at 30fps = 27KB/s per device).
|
||||
|
||||
**When to use:** Animation mode for manually-authored shows. Frame mode for AI sequences where the backend computed exact pixel values. Devices must support both — the firmware handles both packet types.
|
||||
|
||||
**Trade-offs:** Frame mode requires reliable network on local LAN (generally fine). At 60fps frame mode is 54KB/s per device — still well within LAN capacity. The firmware protocol must include a type discriminator so the microcontroller knows which mode is active.
|
||||
|
||||
### Pattern 4: Offline Analysis Pipeline
|
||||
|
||||
**What:** Beat detection and AI sync are NOT done in real-time during playback. They are offline processing steps triggered before show editing starts. Results are stored back into the show JSON as a `analysis` section.
|
||||
|
||||
**When to use:** Always — librosa analysis of a 3-minute song takes 2-10 seconds. Doing it at load time and caching the results avoids blocking playback.
|
||||
|
||||
**Trade-offs:** Show file grows by ~50-200KB per analysed song (beat timestamps are compact). Analysis must be re-triggered if source audio changes. Well worth it for the performance guarantee.
|
||||
|
||||
**Example flow:**
|
||||
```
|
||||
User loads song
|
||||
→ Analysis Engine runs librosa offline
|
||||
→ Extracts: beat_times[], onset_times[], tempo_bpm, sections[]
|
||||
→ Stores in show.analysis JSON block
|
||||
→ Timeline UI renders beat markers from this data
|
||||
→ Auto-sync snaps cues to show.analysis.beat_times
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### "Play This Show" — Browser Click to LED Pixel
|
||||
|
||||
```
|
||||
User clicks Play in Transport Bar
|
||||
│
|
||||
▼ WebSocket message: { type: "transport", cmd: "play" }
|
||||
│
|
||||
▼ WebSocket Hub dispatches to Show Engine
|
||||
│
|
||||
▼ Show Engine sets state = PLAYING, resets cue pointer
|
||||
│
|
||||
▼ Show Engine calls Audio Manager: mpv.play()
|
||||
│
|
||||
▼ Audio Manager sends JSON IPC command to MPV process
|
||||
│
|
||||
▼ MPV starts audio output (platform audio device)
|
||||
│
|
||||
═══════════════════ Playback Loop (5ms tick) ══════════════════
|
||||
│
|
||||
▼ Show Engine polls Audio Manager for time-pos
|
||||
│
|
||||
▼ Audio Manager reads MPV IPC: { "data": 4.237 } ← seconds
|
||||
│
|
||||
▼ Show Engine compares pos against pending cues
|
||||
│
|
||||
├─ No cue yet → continue loop
|
||||
│
|
||||
└─ Cue fires: { device: "wall-strip", animation: "pulse", params: {...} }
|
||||
│
|
||||
▼ UDP Sender looks up device in Device Registry
|
||||
│
|
||||
▼ Calls device.encode_animation_cmd(animation, params)
|
||||
│
|
||||
▼ Sends UDP packet to device IP:port
|
||||
│
|
||||
▼ Microcontroller receives, renders animation on LEDs
|
||||
│
|
||||
▼ Show Engine broadcasts position tick via WebSocket Hub
|
||||
│
|
||||
▼ Browser Transport Bar updates position display (no round-trip needed)
|
||||
```
|
||||
|
||||
### Seek Flow
|
||||
|
||||
```
|
||||
User drags playhead to 00:42.500
|
||||
│
|
||||
▼ WebSocket: { type: "transport", cmd: "seek", position: 42.5 }
|
||||
│
|
||||
▼ Show Engine pauses cue loop
|
||||
│
|
||||
▼ Audio Manager: mpv.seek(42.5, "absolute")
|
||||
│
|
||||
▼ Show Engine resets cue pointer to first cue >= 42.5s
|
||||
│
|
||||
▼ Show Engine broadcasts: { type: "position", pos: 42.5 }
|
||||
│
|
||||
▼ Resumes cue loop
|
||||
```
|
||||
|
||||
### Analysis Flow
|
||||
|
||||
```
|
||||
User triggers "Analyse Song" via UI
|
||||
│
|
||||
▼ REST POST /api/shows/{id}/analyse
|
||||
│
|
||||
▼ Analysis Engine loads audio file
|
||||
│
|
||||
▼ librosa.load() → y, sr
|
||||
▼ librosa.beat.beat_track() → tempo, beat_frames
|
||||
▼ librosa.frames_to_time(beat_frames) → beat_times[]
|
||||
▼ librosa.onset.onset_detect() → onset_times[]
|
||||
│
|
||||
▼ Results stored in show.analysis JSON block (Show Store.save())
|
||||
│
|
||||
▼ WebSocket broadcast: { type: "analysis_complete", showId: "..." }
|
||||
│
|
||||
▼ Timeline UI redraws beat markers
|
||||
```
|
||||
|
||||
### WebSocket vs. Polling
|
||||
|
||||
**Use WebSocket for:**
|
||||
- Position ticks during playback (backend pushes ~10Hz, browser just renders)
|
||||
- State changes: play/pause/stop/seek events
|
||||
- Analysis completion notifications
|
||||
- Cue fire events (for visual feedback in timeline during playback)
|
||||
|
||||
**Use REST for:**
|
||||
- Show CRUD (load/save/list)
|
||||
- Device registry CRUD
|
||||
- Trigger analysis run
|
||||
- Export show file
|
||||
|
||||
**Rationale:** WebSocket is the right choice here — playback position ticks (10-30 per second) make polling impractical. A 100ms poll interval is too coarse for smooth position display; 10ms polling hammers the backend with HTTP overhead. One persistent WebSocket eliminates all of that. REST handles all the "slow path" operations (file I/O, config changes) where request-response semantics are appropriate.
|
||||
|
||||
## Show File JSON Schema
|
||||
|
||||
### Design Principles
|
||||
|
||||
- Human-readable and hand-editable
|
||||
- Self-contained: a show file is everything needed to replay a show
|
||||
- Stable: adding new animation types or device types doesn't require schema migration
|
||||
- Analysis data is embedded, not a sidecar file
|
||||
|
||||
### Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"id": "uuid-v4",
|
||||
"name": "Party Night Vol. 1",
|
||||
"created_at": "2026-04-05T20:00:00Z",
|
||||
"updated_at": "2026-04-05T22:30:00Z",
|
||||
|
||||
"audio": {
|
||||
"source_type": "file",
|
||||
"path": "/shows/audio/party_night.mp3",
|
||||
"duration_seconds": 214.3,
|
||||
"yt_url": null
|
||||
},
|
||||
|
||||
"devices": [
|
||||
{
|
||||
"id": "wall-strip",
|
||||
"name": "Wall LED Strip",
|
||||
"type": "sk6812",
|
||||
"led_count": 300,
|
||||
"ip": "192.168.1.101",
|
||||
"port": 21324
|
||||
},
|
||||
{
|
||||
"id": "shelf-strip",
|
||||
"name": "Shelf U-Turn",
|
||||
"type": "ws2801",
|
||||
"led_count": 160,
|
||||
"ip": "192.168.1.102",
|
||||
"port": 21324
|
||||
}
|
||||
],
|
||||
|
||||
"analysis": {
|
||||
"analysed_at": "2026-04-05T20:15:00Z",
|
||||
"tempo_bpm": 128.0,
|
||||
"beat_times": [0.47, 0.94, 1.41, 1.88],
|
||||
"onset_times": [0.12, 0.47, 0.83, 1.19],
|
||||
"sections": [
|
||||
{ "start": 0.0, "end": 32.0, "label": "intro" },
|
||||
{ "start": 32.0, "end": 96.0, "label": "verse" }
|
||||
]
|
||||
},
|
||||
|
||||
"tracks": [
|
||||
{
|
||||
"device_id": "wall-strip",
|
||||
"cues": [
|
||||
{
|
||||
"id": "cue-001",
|
||||
"timestamp": 0.47,
|
||||
"mode": "animation",
|
||||
"animation": "pulse",
|
||||
"params": {
|
||||
"color": [255, 80, 0],
|
||||
"speed": 1.5,
|
||||
"length": 2.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cue-002",
|
||||
"timestamp": 32.0,
|
||||
"mode": "animation",
|
||||
"animation": "chase",
|
||||
"params": {
|
||||
"color_a": [255, 0, 100],
|
||||
"color_b": [0, 0, 255],
|
||||
"speed": 2.0
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"device_id": "shelf-strip",
|
||||
"cues": [
|
||||
{
|
||||
"id": "cue-003",
|
||||
"timestamp": 0.0,
|
||||
"mode": "animation",
|
||||
"animation": "static",
|
||||
"params": { "color": [30, 0, 60] }
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
"ai_sequences": []
|
||||
}
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
- `mode: "animation"` vs `mode: "frame_sequence"` — the discriminator for UDP strategy
|
||||
- `ai_sequences` is a top-level array separate from tracks — AI-generated pixel sequences are stored as frame data referenced here, not inline in cues (they'd be too large)
|
||||
- `analysis` block is optional — show works without it (no beat markers, no auto-snap)
|
||||
- `devices` embedded in show file — a show is portable without a separate device config
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### Anti-Pattern 1: asyncio.sleep() for Cue Timing
|
||||
|
||||
**What people do:** Schedule cues with `asyncio.create_task(send_cue_after(cue, delay_seconds))`.
|
||||
|
||||
**Why it's wrong:** asyncio event loop can be delayed by other tasks. A 200ms blocking call elsewhere pushes every `sleep` timer by 200ms. More critically: if the user seeks or pauses, all pre-scheduled timers are now wrong and must be cancelled and rescheduled.
|
||||
|
||||
**Do this instead:** Position-driven scheduling — poll audio position, fire cues when position crosses their timestamp. Handles seeks and pauses automatically with zero extra logic.
|
||||
|
||||
### Anti-Pattern 2: Blocking MPV IPC Calls in the Async Loop
|
||||
|
||||
**What people do:** Use `socket.recv()` (blocking) to read MPV IPC responses inside an asyncio coroutine.
|
||||
|
||||
**Why it's wrong:** Blocks the entire event loop. WebSocket messages stop processing, UDP sends are delayed, position ticks stall.
|
||||
|
||||
**Do this instead:** Wrap all MPV IPC communication in `asyncio.to_thread()` or use the non-blocking version of the IPC library. Keep the event loop non-blocking throughout.
|
||||
|
||||
### Anti-Pattern 3: Per-Frame WebSocket Position Updates
|
||||
|
||||
**What people do:** Broadcast audio position on every 5ms tick to the browser.
|
||||
|
||||
**Why it's wrong:** At 200 position updates/second, WebSocket bandwidth is dominated by tiny position messages. Browser render loop can't use 200Hz position data anyway — it renders at 60fps.
|
||||
|
||||
**Do this instead:** Send position ticks at 10-15Hz. Browser extrapolates between ticks using `performance.now()` for smooth display. Backend position is authoritative; browser position is display-only interpolation.
|
||||
|
||||
### Anti-Pattern 4: One Giant Show Engine Class
|
||||
|
||||
**What people do:** Put all playback logic — MPV control, UDP sending, analysis, WebSocket broadcasting — into a single class.
|
||||
|
||||
**Why it's wrong:** Untestable. Audio Manager changes break cue scheduling tests. UDP protocol changes ripple everywhere.
|
||||
|
||||
**Do this instead:** Each subsystem is its own class with a clean interface. Show Engine orchestrates them but doesn't own their internals. Test each in isolation.
|
||||
|
||||
### Anti-Pattern 5: Storing Pixel Frame Data Inline in Show JSON
|
||||
|
||||
**What people do:** Store AI-generated pixel sequences as base64 blobs inside the show JSON.
|
||||
|
||||
**Why it's wrong:** A 3-minute song at 30fps * 300 LEDs * 3 bytes = 48MB of frame data — inline in a JSON file meant to be edited.
|
||||
|
||||
**Do this instead:** Store a reference path in `ai_sequences`. Frame data lives in a separate binary file (e.g., `.fseq` compatible or custom binary). The show JSON stays human-readable and under 100KB.
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External Services
|
||||
|
||||
| Service | Integration Pattern | Notes |
|
||||
|---------|---------------------|-------|
|
||||
| MPV | Subprocess + JSON IPC over named pipe (Windows) or Unix socket | python-mpv-jsonipc handles the pipe; key properties: `time-pos`, `duration`, `pause` |
|
||||
| yt-dlp | MPV handles this natively via `--ytdl` flag — no direct Python integration needed | MPV passes YouTube URLs to yt-dlp internally |
|
||||
| librosa | In-process Python call, runs in thread pool to avoid blocking event loop | `asyncio.to_thread(run_analysis, audio_path)` |
|
||||
|
||||
### Internal Boundaries
|
||||
|
||||
| Boundary | Communication | Notes |
|
||||
|----------|---------------|-------|
|
||||
| Show Engine ↔ Audio Manager | Async method calls (get_position, play, pause, seek) | Audio Manager is a dependency-injected service |
|
||||
| Show Engine ↔ UDP Sender | Async method calls (send_cue, send_frame) | UDP Sender takes a BaseDevice + payload |
|
||||
| Show Engine ↔ WebSocket Hub | Hub is passed to engine; engine calls hub.broadcast() | One-directional push; hub doesn't call engine |
|
||||
| WebSocket Hub ↔ Browser | Full-duplex WebSocket; browser sends commands, backend broadcasts state | All browser commands go through a message dispatcher |
|
||||
| Analysis Engine ↔ Show Store | Analysis returns a dict; Show Store merges into show JSON and saves | Loose coupling — analysis doesn't know about the store |
|
||||
|
||||
## Suggested Build Order
|
||||
|
||||
Dependencies flow bottom-up. Each layer can be built and tested before the next.
|
||||
|
||||
```
|
||||
Phase 1 — Foundation (no hardware needed)
|
||||
├── Show Store: Pydantic models, load/save, device registry
|
||||
├── FastAPI skeleton: app setup, static file serving, REST stubs
|
||||
└── Frontend skeleton: HTML/CSS terminal aesthetic, WebSocket client stub
|
||||
|
||||
Phase 2 — Audio + Transport
|
||||
├── Audio Manager: MPV subprocess, IPC bridge, position polling
|
||||
├── Show Engine: state machine, position-driven playback loop
|
||||
├── WebSocket Hub: connection manager, broadcast
|
||||
└── Transport Bar: play/pause/seek UI connected to WebSocket
|
||||
|
||||
Phase 3 — Device + UDP
|
||||
├── BaseDevice + SK6812 + WS2801 implementations
|
||||
├── UDP packet builders (DRGB, DRGBW, DNRGB)
|
||||
├── UDP Sender: async socket pool
|
||||
└── First end-to-end test: show fires cues → UDP packets → (simulator or device)
|
||||
|
||||
Phase 4 — Timeline Editor
|
||||
├── Timeline UI: tracks, cue blocks, drag-place, beat markers
|
||||
├── Analysis Engine: librosa beat detection, results in show JSON
|
||||
└── Auto-snap cues to beats
|
||||
|
||||
Phase 5 — AI Sync
|
||||
├── AI analysis step (on top of beat detection)
|
||||
├── Frame sequence storage + playback
|
||||
└── Frame-mode UDP sending
|
||||
|
||||
Phase 6 — Microcontroller Firmware
|
||||
├── Protocol is stable by now — implement against spec
|
||||
├── MicroPython on ESP/Pico or Python on RPi
|
||||
└── Both DRGB/DRGBW (frame mode) and animation command mode
|
||||
```
|
||||
|
||||
**Why this order:**
|
||||
|
||||
1. Show Store first — everything else depends on the data model. Defining the schema early prevents model changes from cascading.
|
||||
2. Audio + Transport second — this is the critical path. Without reliable audio position tracking, nothing else can be timed.
|
||||
3. Device + UDP third — the output layer. Can be tested with a simulator (a Python script that prints received packets) before firmware exists.
|
||||
4. Timeline fourth — the editing UX. Depends on analysis data and the show file schema being stable.
|
||||
5. AI sync fifth — optional enhancement layer, builds on beat detection.
|
||||
6. Firmware last — the protocol is fully validated against a software simulator before burning anything to hardware.
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
This is a single-operator local system. Scaling is not a concern. The relevant "scale" questions are latency and reliability, not throughput.
|
||||
|
||||
| Concern | Target | Approach |
|
||||
|---------|--------|----------|
|
||||
| Cue timing precision | ±20ms | Position polling at 5-10ms intervals, pre-fire offset |
|
||||
| UDP latency | <5ms on LAN | Fire-and-forget UDP, no ACK waiting |
|
||||
| WebSocket position tick | 10-15Hz | Sufficient for smooth UI; browser interpolates |
|
||||
| Analysis time | <10s for 5min song | Offline, runs in thread pool, results cached in show JSON |
|
||||
| Frame-mode bandwidth | ~27KB/s per device at 30fps (300 LEDs) | Well within LAN; 5 devices = 135KB/s, trivial |
|
||||
|
||||
## Sources
|
||||
|
||||
- [WLED UDP Realtime Protocol](https://kno.wled.ge/interfaces/udp-realtime/) — DRGB/DNRGB/DRGBW packet format (HIGH confidence — official documentation)
|
||||
- [python-mpv-jsonipc](https://github.com/iwalton3/python-mpv-jsonipc) — IPC bridge library for MPV subprocess (HIGH confidence — active project, used by Jellyfin)
|
||||
- [MPV JSON IPC Protocol](https://mpv.io/manual/master/) — `time-pos`, `playback-time`, seek commands (HIGH confidence — official docs)
|
||||
- [librosa beat_track](https://librosa.org/doc/main/generated/librosa.beat.beat_track.html) — returns beat frame indices, convertible to timestamps via `frames_to_time` (HIGH confidence — official docs)
|
||||
- [FastAPI WebSocket background tasks](https://hexshift.medium.com/implementing-background-tasks-with-websockets-in-fastapi-034cdf803430) — asyncio.create_task pattern alongside websocket receive loop (MEDIUM confidence — community article)
|
||||
- [audio-reactive-led-strip](https://github.com/scottlawsonbc/audio-reactive-led-strip) — reference Python + UDP + ESP8266 architecture (MEDIUM confidence — older project but patterns are standard)
|
||||
- [xLights FSEQ format](https://github.com/Cryptkeeper/fseq-file-format) — reference for binary frame sequence format (LOW confidence — using as inspiration only, not adopting format)
|
||||
|
||||
---
|
||||
*Architecture research for: LightSync — music-to-light synchronization system*
|
||||
*Researched: 2026-04-05*
|
||||
Reference in New Issue
Block a user