# Project Research Summary

**Project:** LightSync — music-to-light synchronization system
**Domain:** Show controller (Python backend, DAW-style web UI, UDP LED control)
**Researched:** 2026-04-05
**Confidence:** MEDIUM-HIGH

## Executive Summary

LightSync is a personal show controller in a well-understood category: professional tools like xLights, QLC+, and SoundSwitch solve the same core problem but are overkill for a 2-strip home setup and don't support the custom UDP protocol this project requires. The right approach is a DAW-style timeline editor backed by librosa for offline beat detection, MPV for audio playback, FastAPI for the backend, and vanilla JS for the frontend. Nothing exotic is needed — the stack is entirely standard Python web application territory, with the non-standard element being the real-time scheduling loop that must be isolated from the async event loop to achieve reliable timing.

The two highest-risk areas are timing and protocol design. The scheduling loop that drives UDP output must poll MPV's audio clock (not wall clock), run in a dedicated thread with 1ms OS timer resolution on Windows, and pre-fire cues by a configurable offset to compensate for LED pipeline latency. The UDP protocol — specifically the packet format for 300-LED RGBW strips — must be settled before firmware is written, because MTU fragmentation and packet loss recovery behavior are hard to retrofit into deployed microcontrollers. Both of these architectural constraints shape the build order more than any feature decision.

The recommended build strategy is bottom-up: data model first, audio engine second, UDP output third, timeline editor fourth. This order ensures each layer is tested in isolation before adding the next. The microcontroller firmware goes last — the protocol should be fully validated against a software simulator before any hardware is flashed. Beat detection and AI sync are enhancers on top of a working show engine, not prerequisites.

## Key Findings

### Recommended Stack

The stack is intentionally narrow. FastAPI (0.135.x) on uvicorn handles the REST API, WebSocket hub, and static file serving in a single process — no need for a separate web server or Socket.IO abstraction. Audio is handled entirely by MPV via `python-mpv-jsonipc`, which controls an external `mpv.exe` via named pipe and avoids the DLL dependency of the ctypes binding. Beat detection uses librosa 0.11.0 with scipy as the FFT backend — the March 2025 release adds NumPy 2.0 support and is the current stable choice. The frontend is vanilla JS with no framework; a DAW-style timeline editor fights React's abstractions and doesn't need component isolation.

**Core technologies:**
- **Python 3.11**: Sweet spot — modern async, librosa 0.11.0 compatible, avoids madmom's Python 3.10+ incompatibility
- **FastAPI 0.135.x + uvicorn 0.34.x**: Native async/await, WebSocket, REST, and static file serving in one process
- **python-mpv-jsonipc 1.0.8**: MPV subprocess control via named pipe — no libmpv.dll required on Windows
- **librosa 0.11.0 + soundfile 0.12**: Offline beat detection, onset analysis, structural segmentation
- **aiosqlite 0.20**: Async SQLite for device registry and show persistence — no ORM needed at this scale
- **Vanilla JS (ES2022)**: Timeline editor, WebSocket client — zero framework overhead

**What to avoid:** aubio (last release 2019), madmom (Python 3.10+ incompatible), essentia (Windows install fragile), BeatNet (requires PyTorch, inherits madmom incompatibility). Do not use asyncio.sleep() for real-time cue scheduling — use a dedicated thread with `time.perf_counter()`.

### Expected Features

Every professional show editor has waveform display, per-device timeline tracks, beat detection, snap-to-beat, and show save/load. These are non-negotiable — missing any of them makes the editor feel broken. Undo/redo is frequently absent in competitors (xLights, QLC+) but users notice and regret the omission — build it from day one using the command pattern or retrofitting is painful.

**Must have (table stakes):**
- Audio file loading + MPV playback (play/pause/seek/position display)
- Waveform display — editing without it is guessing
- Beat detection via librosa — timing marks drive everything downstream
- Timeline editor with per-device tracks and drag-and-place effect blocks
- Animation library with 8-12 core effects (rainbow, chase, pulse, strobe, sparkle, fade, wipe, static)
- Snap-to-beat on placement and resize
- Color picker + basic params (speed, brightness, direction) per effect block
- Show save/load as JSON
- UDP command sender (animation + params at correct timestamp)
- Undo/redo for timeline mutations — build first, not later

**Should have (differentiators):**
- YouTube URL loading via yt-dlp — no other personal tool does this cleanly
- Live preview panel (canvas LED strip simulation without hardware)
- Structural segmentation overlay (verse/chorus/bridge from librosa.segment)
- UDP frame sender (raw RGBW pixel data for AI-generated sequences)
- AI-assisted show generation (rule-based mapping on top of beat analysis)
- Effect grouping / scenes

**Defer to v2+:**
- Parameter animation curves (DAW automation lanes — powerful, high UI complexity)
- Frequency band mapping (per-frame bass/mid/treble routing)
- Moving heads, lasers, fog machine support (extend device abstraction when hardware exists)

### Architecture Approach

The system is a single Python process with four distinct subsystems that must not bleed into each other: Show Engine (playback state machine and cue scheduler), Audio Manager (MPV IPC bridge), UDP Sender (per-device packet dispatcher), and WebSocket Hub (browser push channel). The Show Engine orchestrates the others but never owns their internals. The critical isolation is the scheduling loop — it runs in a dedicated `threading.Thread` reading MPV position into a shared variable, never blocking the asyncio event loop. Analysis (librosa) runs in a `ProcessPoolExecutor` to avoid GIL starvation of the real-time loop.

**Major components:**
1. **Show Engine** — playback state machine, position-driven cue scheduler, orchestrates other subsystems
2. **Audio Manager** — MPV subprocess + named pipe IPC bridge, position polling into shared variable
3. **UDP Sender** — async socket pool, per-device packet dispatch (animation command mode and frame mode)
4. **WebSocket Hub** — connection manager, broadcasts position ticks at 10Hz and state change events
5. **Analysis Engine** — librosa beat/onset/segment processing in ProcessPoolExecutor, results stored in show JSON
6. **Show Store** — Pydantic-validated JSON load/save, device registry, animation library catalog
7. **Timeline UI** — vanilla JS DAW editor, effect blocks, beat markers, transport controls

**Key patterns:**
- Position-driven cue scheduling (poll MPV, fire when `pos >= cue.timestamp`) — handles seek/pause correctly; asyncio.sleep-based scheduling does not
- Device abstraction via `BaseDevice` ABC — SK6812 and WS2801 implement it; future devices add a file
- Dual UDP mode — short animation+params commands for manual shows; full RGBW pixel frames for AI sequences
- Offline analysis pipeline — beat detection runs once at load, results cached in show JSON; never during playback

### Critical Pitfalls

1. **Audio clock vs wall clock drift** — Use MPV's `playback-time` property (not `time-pos`, which returns null when paused) as the authoritative clock on every scheduling tick. On Windows, call `timeBeginPeriod(1)` via ctypes at startup to raise OS timer resolution from 15.6ms to 1ms. Expose a per-show sync calibration offset in the UI — do not defer this.

2. **MPV named pipe blocking on Windows** — Named pipe reads block. Run the IPC reader in a dedicated daemon thread storing position in a shared variable. The send loop reads from that variable — it never touches IPC. Use `python-mpv-jsonipc` to avoid raw overlapped I/O complexity.

3. **MTU fragmentation for large strips** — 300 RGBW LEDs = 1200 bytes of pixel data, exceeding WiFi MTU when headers are added. Split into two packets with a start-index header (DNRGB protocol pattern). Store `leds_per_packet` in the device registry. This must be in the protocol spec before firmware is written — retrofitting costs firmware updates on all devices.

4. **Show file format locked to absolute timestamps** — Include `schema_version`, `source_fingerprint` (SHA256 of first 64KB of audio), and a "shift all events by N seconds" operation from day one. Also store beat position (beat index + subdivision) alongside absolute seconds so shows can be adapted to re-encoded audio.

5. **librosa beat timestamps are 20-60ms late** — This is documented behavior, not a bug. Apply a configurable backward shift after `beat_track()`. Cross-validate with `librosa.onset.onset_detect()` for downbeat placement. Also implement `librosa.beat.plp()` from the start for tempo-variable music (jazz, classical) — `beat_track()` alone fails on free-tempo content.

6. **Python GIL starves the UDP send loop** — librosa analysis is numpy-heavy and holds the GIL. Run it in `ProcessPoolExecutor` (separate process, own GIL), not `ThreadPoolExecutor`. Never trigger analysis during playback without this boundary.

## Implications for Roadmap

Based on research, the build order is dictated by dependencies, not by feature value. Each phase must be validated before the next is started — particularly the protocol spec, which gates firmware development.

### Phase 1: Foundation — Data Model and Project Skeleton

**Rationale:** The show file schema is the contract every other component builds against. Defining it last forces expensive migrations. The Pydantic models, device registry, and FastAPI skeleton cost almost nothing to build first and prevent cascading changes later.
**Delivers:** Validated show file schema, device registry, FastAPI app skeleton with static file serving, terminal-aesthetic HTML/CSS shell
**Addresses:** Show save/load, device registry (table stakes); `schema_version` and `source_fingerprint` (PITFALL 4)
**Avoids:** Show file format lock-in (Pitfall 8) — schema_version, source_fingerprint, and shift-all-events operation are required outputs of this phase

### Phase 2: Audio Engine and Transport

**Rationale:** Audio position is the master clock. Everything else (cues, timeline cursor, UDP timing) is derived from it. This is the highest-risk phase — Windows named pipe behavior and timer precision must be validated here, not discovered later.
**Delivers:** MPV subprocess control, IPC bridge with shared position variable, playback state machine, WebSocket hub, transport controls in UI
**Uses:** python-mpv-jsonipc, FastAPI WebSocket, threading.Thread for IPC reader
**Implements:** Audio Manager + Show Engine skeleton + WebSocket Hub
**Avoids:** Audio clock drift (Pitfall 1) — `timeBeginPeriod(1)` and calibration offset UI are required; MPV named pipe blocking (Pitfall 2) — dedicated reader thread is required; Frontend clock drift (Pitfall 10) — backend position is authoritative, frontend interpolates only

### Phase 3: Communication Protocol and UDP Output

**Rationale:** The UDP protocol must be fully specced and tested against a software simulator before firmware exists. MTU fragmentation and packet loss recovery are hard to retrofit post-firmware. This phase defines the contract the microcontrollers will implement.
**Delivers:** BaseDevice abstraction, SK6812 and WS2801 packet encoders, async UDP sender, protocol spec document, software device simulator, per-device health check on startup
**Uses:** asyncio UDP datagrams, DRGB/DNRGB/DRGBW packet formats, struct.pack
**Implements:** Device Registry + UDP Sender + BaseDevice hierarchy
**Avoids:** MTU fragmentation (Pitfall 6) — multi-packet frame support required; UDP packet loss corruption (Pitfall 5) — sequence numbers and 2-3x retransmit for slow animations; Windows firewall blocking (Pitfall 7) — startup diagnostic required

### Phase 4: Timeline Editor

**Rationale:** The timeline editor is the core UX and the feature that makes or breaks the app. It depends on analysis data (beat marks to snap to) and a stable show file schema. Building it after both are solid prevents churn. Undo/redo must be built here — retrofitting the command pattern into an existing stateful editor is expensive.
**Delivers:** Per-device track layout, drag-and-place effect blocks, beat marker overlay, snap-to-beat, color picker, effect params, waveform display, undo/redo (command pattern)
**Uses:** Vanilla JS, Web Audio API for waveform rendering, librosa beat_track + plp + onset_detect
**Implements:** Analysis Engine + Timeline UI
**Avoids:** librosa timestamp lateness (Pitfall 3 and 4) — backward-shift calibration and plp() fallback required; GIL starvation (Pitfall 9) — analysis runs in ProcessPoolExecutor

### Phase 5: Live Show Execution — End-to-End Integration

**Rationale:** First phase where all subsystems run together. Audio position drives cue firing which drives UDP output. Integration tests here validate the timing guarantees established in Phase 2 and Phase 3 under real-world conditions.
**Delivers:** Full show playback (load show, press play, lights execute), seek-safe cue rescheduling, position-driven scheduler loop, integration test against device simulator
**Uses:** Show Engine cue scheduler, UDP Sender, Audio Manager, WebSocket position ticks
**Implements:** Complete Show Engine with sub-20ms cue timing target

### Phase 6: Enhancements — YouTube, Preview, AI Sync

**Rationale:** These are differentiators that build on a working foundation. YouTube loading is low-cost (yt-dlp + MPV natively). Live preview and AI sync are medium-to-high complexity and should only be built once the core editing loop has been validated with real shows.
**Delivers:** YouTube URL loading via yt-dlp, live canvas preview panel, structural segmentation overlay, AI-assisted show generation, raw frame-mode UDP sender
**Avoids:** AI sync requires beat detection to be solid (Phase 4) and frame-mode UDP to be specced (Phase 3 protocol doc)

### Phase 7: Microcontroller Firmware

**Rationale:** Firmware is last by design. The protocol spec from Phase 3, validated through Phases 5 and 6, is the implementation target. Flashing firmware against a moving protocol spec means reflashing every iteration.
**Delivers:** MicroPython (or C/Arduino) firmware for ESP/Pico/RPi4B supporting DRGB/DRGBW frame mode and animation command mode, tested against both SK6812 and WS2801 strips
**Implements:** Hardware integration for the two v1 devices

### Phase Ordering Rationale

- Show file schema before everything — the Pydantic models in Phase 1 are the contract that lets every subsequent phase proceed without model churn
- Audio engine before UDP — you cannot validate timing without knowing what time MPV says it is
- Protocol spec before firmware — MTU fragmentation and packet loss recovery are architectural, not implementation details; they must be frozen before microcontrollers are programmed
- Timeline editor before live execution — the cue data structure must be stable before the show engine fires cues
- Enhancements after validation — AI sync and live preview should be built on a foundation that has been used for at least one real show
- Firmware last — no exceptions; protocol stability is the prerequisite

### Research Flags

Phases likely needing deeper research during planning:
- **Phase 2 (Audio Engine):** Windows named pipe behavior under load is under-documented. Validate `python-mpv-jsonipc` latency characteristics on Windows 11 before committing to the threading model. The `timeBeginPeriod(1)` call requires explicit testing to confirm it achieves <2ms timer resolution.
- **Phase 4 (Timeline Editor):** Canvas-based DAW timeline with drag-drop, resize, and snap is complex UI territory. Research interaction patterns from open-source sequencers (e.g., LMMS web, Audion) before writing the first JS line. Undo/redo command pattern design should be explicitly sketched before implementation.
- **Phase 6 (AI Sync):** No mature Python library for music-to-light AI sync exists in 2025. The practical path (librosa features → rule-based mapper) is clear, but the LLM-assisted path (song structure analysis → cue placement) has no established pattern. Needs a prototype spike before committing to an approach.

Phases with well-documented patterns (skip deeper research):
- **Phase 1 (Foundation):** Pydantic v2 schemas and FastAPI skeleton are standard; no research needed
- **Phase 3 (UDP Protocol):** WLED's DRGB/DNRGB/DRGBW packet format is fully documented and battle-tested; adopt it directly
- **Phase 5 (Live Execution):** The position-driven scheduler pattern from ARCHITECTURE.md is fully specified; implement it directly
- **Phase 7 (Firmware):** MicroPython UDP socket handling for ESP32/Pico is well-documented; firmware complexity is protocol implementation, not discovery

## Confidence Assessment

| Area | Confidence | Notes |
|------|------------|-------|
| Stack | HIGH | All core library versions verified against official changelogs and PyPI; madmom/aubio/essentia incompatibilities confirmed from GitHub issues and official sources |
| Features | HIGH | Cross-referenced against 6 commercial/open-source tools (xLights, QLC+, SoundSwitch, Lightjams, WLED, Resolume); feature table is comprehensive |
| Architecture | HIGH | Core patterns (position-driven scheduling, BaseDevice abstraction, dual UDP mode, offline analysis) are established in reference implementations; AI sync specifics are MEDIUM |
| Pitfalls | MEDIUM-HIGH | Timing and protocol pitfalls are HIGH confidence from official docs and community reports; Windows asyncio timer precision is MEDIUM from limited benchmarks; beat detection accuracy claims are backed by librosa GitHub issues |

**Overall confidence:** MEDIUM-HIGH

### Gaps to Address

- **libmpv.dll vs jsonipc on Windows 11:** Research recommends `python-mpv-jsonipc` to avoid the DLL dependency, but its response latency under load on Windows 11 is not benchmarked. Validate in Phase 2 with a timing harness before building the full audio engine around it.
- **librosa MP3 loading path:** `soundfile` cannot decode MP3 — it falls back to `audioread`. This is documented but the fallback behavior in librosa 0.11.0 on Windows needs explicit validation (audioread depends on FFmpeg being on PATH or bundled). Verify in Phase 4 before analysis is used on real songs.
- **AI sync approach:** The rule-based mapper (librosa features → animation selection) is the practical v1 path. The LLM-assisted approach (song structure description → cue placement) is speculative — no existing implementation to reference. Treat as a Phase 6 spike, not a commitment.
- **Microcontroller choice:** PROJECT.md lists ESP/Pico/RPi4B as TBD. The protocol spec (Phase 3) works for all three, but MicroPython async UDP socket handling differs between platforms. Choose before Phase 7 begins.

## Sources

### Primary (HIGH confidence)
- [librosa 0.11.0 changelog](https://librosa.org/doc/main/changelog.html) — version and NumPy 2.0 support
- [FastAPI release notes](https://fastapi.tiangolo.com/release-notes/) — version 0.135.3 current
- [python-mpv PyPI](https://pypi.org/project/python-mpv/) — version 1.0.8, Windows support
- [MPV IPC documentation](https://mpv.io/manual/master/) — named pipe, time-pos, playback-time
- [WLED UDP Realtime Protocol](https://kno.wled.ge/interfaces/udp-realtime/) — DRGB/DNRGB/DRGBW packet format
- [madmom Python 3.10+ incompatibility](https://github.com/CPJKU/beat_this/issues/9) — confirmed broken
- [aubio PyPI](https://pypi.org/project/aubio/) — last release 2019, confirmed stale
- [librosa beat timing issue](https://github.com/librosa/librosa/issues/1052) — 20-60ms lateness documented
- [python-mpv-jsonipc](https://github.com/iwalton3/python-mpv-jsonipc) — Windows named pipe abstraction

### Secondary (MEDIUM confidence)
- [xLights Manual — Timeline and Waveform](https://manual.xlights.org/xlights/chapters/chapter-four-sequencer/timeline-and-waveform) — feature baseline
- [SoundSwitch What's New in 2.9](https://support.soundswitch.com/en/support/solutions/articles/69000858481) — BPM detection, phrase detection
- [WLED Audio Reactive documentation](https://kno.wled.ge/advanced/audio-reactive/) — 30+ effects reference
- [Lightjams — Sync Music and Lights](https://www.lightjams.com/musicToDMX.html) — frequency band mapping patterns
- [FastAPI WebSocket background tasks](https://hexshift.medium.com/implementing-background-tasks-with-websockets-in-fastapi-034cdf803430) — async patterns
- [audio-reactive-led-strip](https://github.com/scottlawsonbc/audio-reactive-led-strip) — reference Python + UDP + ESP8266 architecture
- [Windows asyncio timing limitations](https://runebook.dev/en/docs/python/library/asyncio-platforms/windows) — 15.6ms default resolution

### Tertiary (LOW confidence)
- AI sync approach (rule-based mapper + LLM) — inferred from librosa capabilities; no existing reference implementation
- Windows firewall UDP blocking behavior — [Microsoft docs](https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/windows-fireware-rule-block-udp-communication) + community reports; actual blocking conditions vary by system configuration

---
*Research completed: 2026-04-05*
*Ready for roadmap: yes*