14 KiB
14 KiB
Stack Research
Domain: Music-to-light synchronization system (Python backend, browser frontend, UDP control) Researched: 2026-04-05 Confidence: MEDIUM-HIGH (core stack HIGH, audio analysis MEDIUM, AI sync LOW)
Recommended Stack
Core Technologies
| Technology | Version | Purpose | Why Recommended |
|---|---|---|---|
| Python | 3.11 | Runtime | 3.11 hits the sweet spot: modern async, good Windows support, librosa 0.11 compatible. Avoid 3.12+ until madmom resolves its incompatibility. Avoid 3.9 as it's EOL in Oct 2025. |
| FastAPI | 0.135.x | Web framework + REST API + WebSocket server | Native async/await, built-in WebSocket support, Pydantic validation, serves static files — one process covers API + UI + realtime push. Flask is sync-first and requires extensions for every feature this app needs. |
| uvicorn | 0.34.x | ASGI server | Standard FastAPI runner, works on Windows without Gunicorn. Use uvicorn.run() programmatically so the app can self-start without a separate process manager. |
| librosa | 0.11.0 | Beat detection, onset analysis, tempo estimation | Released March 2025, supports NumPy 2.0, actively maintained. Correct choice for offline audio analysis from files. scipy FFT backend is more accurate than numpy. |
| python-mpv | 1.0.8 | MPV process control + playback position polling | Released April 2025. ctypes-based — gives property observers and event hooks inside Python. Better than raw subprocess for position tracking. Windows uses named pipe IPC automatically. |
| Pydantic | 2.x (bundled with FastAPI) | Show file validation, device registry schemas, config | Already a FastAPI dependency. Use BaseModel for all JSON show file structures — free validation and serialization. |
Supporting Libraries
| Library | Version | Purpose | When to Use |
|---|---|---|---|
| numpy | 2.0+ | Array math for pixel frame buffers, beat arrays | Always — librosa depends on it, and frame-based UDP payloads are numpy arrays before packing |
| scipy | 1.14+ | FFT, signal processing (librosa backend) | Always — librosa uses it as FFT backend since 0.11.0 |
| soundfile | 0.12+ | Audio file loading (WAV, FLAC, OGG) for librosa | When loading local audio files for analysis pre-playback |
| aiofiles | 24.x | Async file I/O for show file read/write | When persisting show files from async FastAPI handlers |
| SQLite (stdlib) | built-in | Persistent storage for devices, shows, project state | Use Python's built-in sqlite3 or aiosqlite for async; no ORM needed for this scale |
| aiosqlite | 0.20+ | Async SQLite wrapper | For non-blocking database access from FastAPI async handlers |
| structlog | 24.x | Structured logging | When you want JSON-formatted logs for the backend; easier to parse than stdlib logging |
| python-dotenv | 1.x | Environment config (host, port, UDP targets) | For externalizing runtime config without hardcoding |
| madmom | 0.16 | High-accuracy beat tracking (DBN model) | ONLY if using Python 3.9 in a separate subprocess or venv — madmom is incompatible with Python 3.10+. Use as an optional enhancement path, not core dependency. |
Frontend Stack
| Technology | Version | Purpose | Why Recommended |
|---|---|---|---|
| Vanilla JS (ES2022+) | native | Timeline editor, WebSocket client, UI interactions | This app has a custom DAW-like UI — no component framework matches the drag-drop timeline without fighting abstractions. React adds 200KB+ and solves problems this app doesn't have. |
| Native WebSocket API | native | Real-time playback position updates from backend | Browser-native, zero dependencies, sufficient for position polling and event pushes |
| CSS Custom Properties + Grid/Flexbox | native | Terminal aesthetic: dark theme, monospace, track layout | No CSS framework needed — a bespoke terminal aesthetic is easier to build raw than to override Bootstrap or Tailwind |
| Web Audio API | native | Waveform visualization (optional) | If displaying waveform in timeline, decode via AudioContext.decodeAudioData() — no library needed |
Development Tools
| Tool | Purpose | Notes |
|---|---|---|
| uv | Python package manager + venv | Faster than pip, reproducible installs, handles Windows path quirks well |
| pyinstaller (optional) | Bundle to Windows .exe | Only if distributing; not needed for personal dev machine use |
Installation
# Core backend
uv pip install fastapi==0.135.3 uvicorn==0.34.1 python-mpv==1.0.8 pydantic==2.10.x
# Audio analysis
uv pip install librosa==0.11.0 soundfile==0.12.1
# Persistence + async
uv pip install aiosqlite==0.20.0 aiofiles==24.1.0
# Config + utilities
uv pip install python-dotenv==1.0.1 structlog==24.4.0
Note: numpy and scipy install automatically as librosa dependencies. Do not pin them separately unless you hit a conflict.
Alternatives Considered
| Recommended | Alternative | When to Use Alternative |
|---|---|---|
| FastAPI | Flask | If you already have a Flask app or need Jinja templating; Flask-SocketIO is a viable WebSocket path but adds complexity |
| FastAPI | Django | Never for this project — Django's ORM and admin are overhead for a single-operator music sync tool |
| Vanilla JS | React | If the team knows React and the timeline editor becomes a complex stateful SPA (drag-drop state machines, undo/redo) that benefits from component isolation |
| Vanilla JS | HTMX | HTMX is wrong for this use case — the timeline editor requires fine-grained JS state, not server-rendered HTML swaps |
| python-mpv (ctypes) | python-mpv-jsonipc | Use jsonipc variant if libmpv.dll is unavailable on the target Windows system (jsonipc controls an external mpv.exe via named pipe, no DLL required) |
| python-mpv | subprocess + JSON IPC manual | More control, no DLL dependency, but requires implementing the IPC protocol yourself |
| librosa (offline) | madmom (offline) | madmom has better beat accuracy for complex rhythms, BUT it is incompatible with Python 3.10+. Use only in an isolated subprocess with Python 3.9 if high-accuracy tracking is critical. |
| aiosqlite | SQLAlchemy + SQLModel | Use SQLAlchemy if the schema grows complex (multiple joined tables, migrations). For show files + device registry, raw aiosqlite is sufficient. |
| SQLite | JSON flat files | JSON files are fine for show data but lose query capability for device registry lookups and show search. SQLite has zero deployment overhead on Windows. |
What NOT to Use
| Avoid | Why | Use Instead |
|---|---|---|
| aubio | Last release: February 2019 (0.4.9). Unmaintained, no Python 3.10+ wheels, Windows install is fragile. Described as "Beta" still after 6 years. | librosa for offline analysis |
| madmom as core dependency | Incompatible with Python 3.10+ due to collections.MutableSequence removal. Requires Python ≤ 3.9. Outstanding GitHub issues with no resolution timeline. |
librosa as primary; madmom only in isolated venv if needed |
| essentia | Excellent accuracy, but C++ binding installation on Windows is non-trivial. Pre-built wheels are inconsistent. Adds significant complexity for marginal beat detection gain. | librosa covers the use case well on Windows |
| BeatNet | Requires madmom as a dependency, inheriting the Python 3.10+ incompatibility problem. Also requires PyTorch, which is 200–800MB. Overkill for a personal project. | librosa; revisit if AI sync accuracy becomes a priority |
| gevent / eventlet | Old async approaches, sometimes required with Flask-SocketIO. Not needed with FastAPI's native async. | FastAPI + uvicorn native async |
| Flask-SocketIO | Adds an abstraction layer over Socket.IO protocol instead of raw WebSocket. Overkill, and Socket.IO client required on frontend. | FastAPI native WebSocket |
| PyAudio / sounddevice for playback | This project uses MPV for playback. Mixing MPV subprocess with Python audio I/O for the same audio creates timing conflicts. | python-mpv for all playback |
| Art-Net / sACN / E1.31 protocol | These are DMX-over-UDP protocols designed for theatrical equipment. The project uses custom UDP packets to custom firmware — don't add DMX protocol overhead. | Raw UDP asyncio datagrams |
Stack Patterns by Variant
For beat detection on local audio files:
- Load with
soundfile.read()→ pass tolibrosa.beat.beat_track(y, sr)→ returns tempo + beat frame indices - Convert frames to timestamps:
librosa.frames_to_time(beats, sr=sr) - Onset detection for drop/hit detection:
librosa.onset.onset_detect(y, sr, units='time')
For UDP frame sending (AI-generated pixel sequences):
- Pack RGB values into bytes:
struct.pack('B' * (led_count * 3), *flat_rgb_array) - Send via asyncio datagram endpoint:
transport.sendto(payload, (target_ip, target_port)) - Fire-and-forget: no acknowledgement, send at frame rate (e.g., 30fps = 33ms interval)
For UDP animation+params sending (manual mode):
- Small JSON payload or fixed binary struct:
{"anim": "pulse", "color": [255, 0, 128], "speed": 1.5} - Encode as JSON bytes or define a compact binary protocol with
struct.pack - JSON is easier to debug during early dev; migrate to binary if bandwidth becomes an issue
For playback position sync (frontend timeline scrubbing):
- Use
python-mpvproperty observer:player.observe_property('time-pos', callback) - In callback, push position via WebSocket to all connected browser clients
- Frontend timeline animates playhead based on received position events
For AI sync (optional enhancement — LOW confidence):
- No mature turnkey Python library for music-to-light AI in 2025 that runs on Windows without GPU
- Practical path: run librosa full analysis (beat + onset + chroma + spectral contrast), feed features into a rule-based mapper (loud = bright, beat = flash, drop = color change)
- True AI path: fine-tune a small model offline using MIDI-to-light training data — deferred to later phases
- OpenAI/Claude audio analysis APIs could be used to get high-level song structure (verse/chorus/bridge) from a transcript or description, but audio file analysis APIs do not exist at useful granularity for per-beat sync
For Windows 11 development:
- Use
python-mpvwithlibmpv-2.dllplaced next to the app (ship with the project) - Alternatively, use
python-mpv-jsonipcwhich controlsmpv.exevia named pipe — no DLL needed - Named pipe path:
\\.\pipe\lightsync_mpv - UDP
asyncioworks natively on Windows withProactorEventLoop(Python 3.8+ default on Windows)
JSON Show File Format
No industry-standard JSON schema exists for this exact use case. Custom format recommended.
{
"version": "1.0",
"meta": {
"title": "My Show",
"audio_source": "file:///path/to/song.mp3",
"duration_seconds": 213.4,
"bpm": 128.0,
"created_at": "2026-04-05T12:00:00Z"
},
"beats": [0.47, 0.94, 1.41],
"devices": [
{
"id": "wall_strip",
"name": "Wall Strip",
"type": "SK6812",
"led_count": 300,
"target_ip": "192.168.1.50",
"port": 4210
}
],
"timeline": {
"wall_strip": [
{
"start": 0.47,
"end": 1.41,
"animation": "pulse",
"params": {"color": [255, 0, 128], "speed": 1.0}
}
]
}
}
Key design decisions for the format:
- Timestamps in seconds (float) — not frames or ticks. Decoupled from sample rate.
- Beat array stored in show file — precomputed at import time, not recalculated on playback.
- Device definitions embedded in show file — show is self-contained, no external registry lookup required during playback.
- Animation params as open object — allows per-animation parameter schemas without breaking the format.
Version Compatibility
| Package | Compatible With | Notes |
|---|---|---|
| librosa 0.11.0 | numpy >= 1.17, scipy >= 1.0, Python 3.8+ | Requires soundfile for local file loading |
| python-mpv 1.0.8 | Python >= 3.9, libmpv 0.35+ | libmpv-2.dll required on Windows; ship with app |
| FastAPI 0.135.x | pydantic v2, Python 3.8+, uvicorn 0.34+ | Pydantic v1 compatibility mode dropped |
| madmom 0.16 | Python < 3.10 ONLY | Do not install in same venv as main app if using Python 3.11+ |
| numpy 2.0+ | librosa 0.11.0, scipy 1.14+, soundfile 0.12+ | librosa 0.11.0 explicitly added NumPy 2.0 support |
Sources
- FastAPI release notes — version 0.135.3 confirmed current (HIGH confidence)
- librosa 0.11.0 changelog — released March 2025, NumPy 2.0 support confirmed (HIGH confidence)
- python-mpv PyPI — version 1.0.8 released April 2025, Windows support confirmed (HIGH confidence)
- aubio PyPI — last release 2019, confirmed stale (HIGH confidence)
- madmom Python 3.10+ incompatibility — GitHub issue — confirmed broken on Python >= 3.10 (HIGH confidence)
- BIFF.ai beat detection rundown — madmom vs BeatNet comparison (MEDIUM confidence, single source)
- MPV IPC documentation — named pipe on Windows confirmed (HIGH confidence)
- uvicorn release notes — 0.34.1 current stable, Windows ProactorEventLoop (HIGH confidence)
- WebSearch: FastAPI WebSocket patterns, async UDP asyncio, JSON show file formats — (MEDIUM confidence, multiple sources)
Stack research for: LightSync — music-to-light synchronization system Researched: 2026-04-05