215 lines
14 KiB
Markdown
215 lines
14 KiB
Markdown
# Stack Research
|
||
|
||
**Domain:** Music-to-light synchronization system (Python backend, browser frontend, UDP control)
|
||
**Researched:** 2026-04-05
|
||
**Confidence:** MEDIUM-HIGH (core stack HIGH, audio analysis MEDIUM, AI sync LOW)
|
||
|
||
---
|
||
|
||
## Recommended Stack
|
||
|
||
### Core Technologies
|
||
|
||
| Technology | Version | Purpose | Why Recommended |
|
||
|------------|---------|---------|-----------------|
|
||
| Python | 3.11 | Runtime | 3.11 hits the sweet spot: modern async, good Windows support, librosa 0.11 compatible. Avoid 3.12+ until madmom resolves its incompatibility. Avoid 3.9 as it's EOL in Oct 2025. |
|
||
| FastAPI | 0.135.x | Web framework + REST API + WebSocket server | Native async/await, built-in WebSocket support, Pydantic validation, serves static files — one process covers API + UI + realtime push. Flask is sync-first and requires extensions for every feature this app needs. |
|
||
| uvicorn | 0.34.x | ASGI server | Standard FastAPI runner, works on Windows without Gunicorn. Use `uvicorn.run()` programmatically so the app can self-start without a separate process manager. |
|
||
| librosa | 0.11.0 | Beat detection, onset analysis, tempo estimation | Released March 2025, supports NumPy 2.0, actively maintained. Correct choice for offline audio analysis from files. scipy FFT backend is more accurate than numpy. |
|
||
| python-mpv | 1.0.8 | MPV process control + playback position polling | Released April 2025. ctypes-based — gives property observers and event hooks inside Python. Better than raw subprocess for position tracking. Windows uses named pipe IPC automatically. |
|
||
| Pydantic | 2.x (bundled with FastAPI) | Show file validation, device registry schemas, config | Already a FastAPI dependency. Use BaseModel for all JSON show file structures — free validation and serialization. |
|
||
|
||
### Supporting Libraries
|
||
|
||
| Library | Version | Purpose | When to Use |
|
||
|---------|---------|---------|-------------|
|
||
| numpy | 2.0+ | Array math for pixel frame buffers, beat arrays | Always — librosa depends on it, and frame-based UDP payloads are numpy arrays before packing |
|
||
| scipy | 1.14+ | FFT, signal processing (librosa backend) | Always — librosa uses it as FFT backend since 0.11.0 |
|
||
| soundfile | 0.12+ | Audio file loading (WAV, FLAC, OGG) for librosa | When loading local audio files for analysis pre-playback |
|
||
| aiofiles | 24.x | Async file I/O for show file read/write | When persisting show files from async FastAPI handlers |
|
||
| SQLite (stdlib) | built-in | Persistent storage for devices, shows, project state | Use Python's built-in `sqlite3` or `aiosqlite` for async; no ORM needed for this scale |
|
||
| aiosqlite | 0.20+ | Async SQLite wrapper | For non-blocking database access from FastAPI async handlers |
|
||
| structlog | 24.x | Structured logging | When you want JSON-formatted logs for the backend; easier to parse than stdlib logging |
|
||
| python-dotenv | 1.x | Environment config (host, port, UDP targets) | For externalizing runtime config without hardcoding |
|
||
| madmom | 0.16 | High-accuracy beat tracking (DBN model) | ONLY if using Python 3.9 in a separate subprocess or venv — madmom is incompatible with Python 3.10+. Use as an optional enhancement path, not core dependency. |
|
||
|
||
### Frontend Stack
|
||
|
||
| Technology | Version | Purpose | Why Recommended |
|
||
|------------|---------|---------|-----------------|
|
||
| Vanilla JS (ES2022+) | native | Timeline editor, WebSocket client, UI interactions | This app has a custom DAW-like UI — no component framework matches the drag-drop timeline without fighting abstractions. React adds 200KB+ and solves problems this app doesn't have. |
|
||
| Native WebSocket API | native | Real-time playback position updates from backend | Browser-native, zero dependencies, sufficient for position polling and event pushes |
|
||
| CSS Custom Properties + Grid/Flexbox | native | Terminal aesthetic: dark theme, monospace, track layout | No CSS framework needed — a bespoke terminal aesthetic is easier to build raw than to override Bootstrap or Tailwind |
|
||
| Web Audio API | native | Waveform visualization (optional) | If displaying waveform in timeline, decode via AudioContext.decodeAudioData() — no library needed |
|
||
|
||
### Development Tools
|
||
|
||
| Tool | Purpose | Notes |
|
||
|------|---------|-------|
|
||
| uv | Python package manager + venv | Faster than pip, reproducible installs, handles Windows path quirks well |
|
||
| pyinstaller (optional) | Bundle to Windows .exe | Only if distributing; not needed for personal dev machine use |
|
||
|
||
---
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
# Core backend
|
||
uv pip install fastapi==0.135.3 uvicorn==0.34.1 python-mpv==1.0.8 pydantic==2.10.x
|
||
|
||
# Audio analysis
|
||
uv pip install librosa==0.11.0 soundfile==0.12.1
|
||
|
||
# Persistence + async
|
||
uv pip install aiosqlite==0.20.0 aiofiles==24.1.0
|
||
|
||
# Config + utilities
|
||
uv pip install python-dotenv==1.0.1 structlog==24.4.0
|
||
```
|
||
|
||
Note: `numpy` and `scipy` install automatically as librosa dependencies. Do not pin them separately unless you hit a conflict.
|
||
|
||
---
|
||
|
||
## Alternatives Considered
|
||
|
||
| Recommended | Alternative | When to Use Alternative |
|
||
|-------------|-------------|-------------------------|
|
||
| FastAPI | Flask | If you already have a Flask app or need Jinja templating; Flask-SocketIO is a viable WebSocket path but adds complexity |
|
||
| FastAPI | Django | Never for this project — Django's ORM and admin are overhead for a single-operator music sync tool |
|
||
| Vanilla JS | React | If the team knows React and the timeline editor becomes a complex stateful SPA (drag-drop state machines, undo/redo) that benefits from component isolation |
|
||
| Vanilla JS | HTMX | HTMX is wrong for this use case — the timeline editor requires fine-grained JS state, not server-rendered HTML swaps |
|
||
| python-mpv (ctypes) | python-mpv-jsonipc | Use jsonipc variant if libmpv.dll is unavailable on the target Windows system (jsonipc controls an external mpv.exe via named pipe, no DLL required) |
|
||
| python-mpv | subprocess + JSON IPC manual | More control, no DLL dependency, but requires implementing the IPC protocol yourself |
|
||
| librosa (offline) | madmom (offline) | madmom has better beat accuracy for complex rhythms, BUT it is incompatible with Python 3.10+. Use only in an isolated subprocess with Python 3.9 if high-accuracy tracking is critical. |
|
||
| aiosqlite | SQLAlchemy + SQLModel | Use SQLAlchemy if the schema grows complex (multiple joined tables, migrations). For show files + device registry, raw aiosqlite is sufficient. |
|
||
| SQLite | JSON flat files | JSON files are fine for show data but lose query capability for device registry lookups and show search. SQLite has zero deployment overhead on Windows. |
|
||
|
||
---
|
||
|
||
## What NOT to Use
|
||
|
||
| Avoid | Why | Use Instead |
|
||
|-------|-----|-------------|
|
||
| aubio | Last release: February 2019 (0.4.9). Unmaintained, no Python 3.10+ wheels, Windows install is fragile. Described as "Beta" still after 6 years. | librosa for offline analysis |
|
||
| madmom as core dependency | Incompatible with Python 3.10+ due to `collections.MutableSequence` removal. Requires Python ≤ 3.9. Outstanding GitHub issues with no resolution timeline. | librosa as primary; madmom only in isolated venv if needed |
|
||
| essentia | Excellent accuracy, but C++ binding installation on Windows is non-trivial. Pre-built wheels are inconsistent. Adds significant complexity for marginal beat detection gain. | librosa covers the use case well on Windows |
|
||
| BeatNet | Requires madmom as a dependency, inheriting the Python 3.10+ incompatibility problem. Also requires PyTorch, which is 200–800MB. Overkill for a personal project. | librosa; revisit if AI sync accuracy becomes a priority |
|
||
| gevent / eventlet | Old async approaches, sometimes required with Flask-SocketIO. Not needed with FastAPI's native async. | FastAPI + uvicorn native async |
|
||
| Flask-SocketIO | Adds an abstraction layer over Socket.IO protocol instead of raw WebSocket. Overkill, and Socket.IO client required on frontend. | FastAPI native WebSocket |
|
||
| PyAudio / sounddevice for playback | This project uses MPV for playback. Mixing MPV subprocess with Python audio I/O for the same audio creates timing conflicts. | python-mpv for all playback |
|
||
| Art-Net / sACN / E1.31 protocol | These are DMX-over-UDP protocols designed for theatrical equipment. The project uses custom UDP packets to custom firmware — don't add DMX protocol overhead. | Raw UDP asyncio datagrams |
|
||
|
||
---
|
||
|
||
## Stack Patterns by Variant
|
||
|
||
**For beat detection on local audio files:**
|
||
- Load with `soundfile.read()` → pass to `librosa.beat.beat_track(y, sr)` → returns tempo + beat frame indices
|
||
- Convert frames to timestamps: `librosa.frames_to_time(beats, sr=sr)`
|
||
- Onset detection for drop/hit detection: `librosa.onset.onset_detect(y, sr, units='time')`
|
||
|
||
**For UDP frame sending (AI-generated pixel sequences):**
|
||
- Pack RGB values into bytes: `struct.pack('B' * (led_count * 3), *flat_rgb_array)`
|
||
- Send via asyncio datagram endpoint: `transport.sendto(payload, (target_ip, target_port))`
|
||
- Fire-and-forget: no acknowledgement, send at frame rate (e.g., 30fps = 33ms interval)
|
||
|
||
**For UDP animation+params sending (manual mode):**
|
||
- Small JSON payload or fixed binary struct: `{"anim": "pulse", "color": [255, 0, 128], "speed": 1.5}`
|
||
- Encode as JSON bytes or define a compact binary protocol with `struct.pack`
|
||
- JSON is easier to debug during early dev; migrate to binary if bandwidth becomes an issue
|
||
|
||
**For playback position sync (frontend timeline scrubbing):**
|
||
- Use `python-mpv` property observer: `player.observe_property('time-pos', callback)`
|
||
- In callback, push position via WebSocket to all connected browser clients
|
||
- Frontend timeline animates playhead based on received position events
|
||
|
||
**For AI sync (optional enhancement — LOW confidence):**
|
||
- No mature turnkey Python library for music-to-light AI in 2025 that runs on Windows without GPU
|
||
- Practical path: run librosa full analysis (beat + onset + chroma + spectral contrast), feed features into a rule-based mapper (loud = bright, beat = flash, drop = color change)
|
||
- True AI path: fine-tune a small model offline using MIDI-to-light training data — deferred to later phases
|
||
- OpenAI/Claude audio analysis APIs could be used to get high-level song structure (verse/chorus/bridge) from a transcript or description, but audio file analysis APIs do not exist at useful granularity for per-beat sync
|
||
|
||
**For Windows 11 development:**
|
||
- Use `python-mpv` with `libmpv-2.dll` placed next to the app (ship with the project)
|
||
- Alternatively, use `python-mpv-jsonipc` which controls `mpv.exe` via named pipe — no DLL needed
|
||
- Named pipe path: `\\.\pipe\lightsync_mpv`
|
||
- UDP `asyncio` works natively on Windows with `ProactorEventLoop` (Python 3.8+ default on Windows)
|
||
|
||
---
|
||
|
||
## JSON Show File Format
|
||
|
||
No industry-standard JSON schema exists for this exact use case. Custom format recommended.
|
||
|
||
```json
|
||
{
|
||
"version": "1.0",
|
||
"meta": {
|
||
"title": "My Show",
|
||
"audio_source": "file:///path/to/song.mp3",
|
||
"duration_seconds": 213.4,
|
||
"bpm": 128.0,
|
||
"created_at": "2026-04-05T12:00:00Z"
|
||
},
|
||
"beats": [0.47, 0.94, 1.41],
|
||
"devices": [
|
||
{
|
||
"id": "wall_strip",
|
||
"name": "Wall Strip",
|
||
"type": "SK6812",
|
||
"led_count": 300,
|
||
"target_ip": "192.168.1.50",
|
||
"port": 4210
|
||
}
|
||
],
|
||
"timeline": {
|
||
"wall_strip": [
|
||
{
|
||
"start": 0.47,
|
||
"end": 1.41,
|
||
"animation": "pulse",
|
||
"params": {"color": [255, 0, 128], "speed": 1.0}
|
||
}
|
||
]
|
||
}
|
||
}
|
||
```
|
||
|
||
Key design decisions for the format:
|
||
- Timestamps in seconds (float) — not frames or ticks. Decoupled from sample rate.
|
||
- Beat array stored in show file — precomputed at import time, not recalculated on playback.
|
||
- Device definitions embedded in show file — show is self-contained, no external registry lookup required during playback.
|
||
- Animation params as open object — allows per-animation parameter schemas without breaking the format.
|
||
|
||
---
|
||
|
||
## Version Compatibility
|
||
|
||
| Package | Compatible With | Notes |
|
||
|---------|-----------------|-------|
|
||
| librosa 0.11.0 | numpy >= 1.17, scipy >= 1.0, Python 3.8+ | Requires soundfile for local file loading |
|
||
| python-mpv 1.0.8 | Python >= 3.9, libmpv 0.35+ | libmpv-2.dll required on Windows; ship with app |
|
||
| FastAPI 0.135.x | pydantic v2, Python 3.8+, uvicorn 0.34+ | Pydantic v1 compatibility mode dropped |
|
||
| madmom 0.16 | Python < 3.10 ONLY | Do not install in same venv as main app if using Python 3.11+ |
|
||
| numpy 2.0+ | librosa 0.11.0, scipy 1.14+, soundfile 0.12+ | librosa 0.11.0 explicitly added NumPy 2.0 support |
|
||
|
||
---
|
||
|
||
## Sources
|
||
|
||
- [FastAPI release notes](https://fastapi.tiangolo.com/release-notes/) — version 0.135.3 confirmed current (HIGH confidence)
|
||
- [librosa 0.11.0 changelog](https://librosa.org/doc/main/changelog.html) — released March 2025, NumPy 2.0 support confirmed (HIGH confidence)
|
||
- [python-mpv PyPI](https://pypi.org/project/python-mpv/) — version 1.0.8 released April 2025, Windows support confirmed (HIGH confidence)
|
||
- [aubio PyPI](https://pypi.org/project/aubio/) — last release 2019, confirmed stale (HIGH confidence)
|
||
- [madmom Python 3.10+ incompatibility — GitHub issue](https://github.com/CPJKU/beat_this/issues/9) — confirmed broken on Python >= 3.10 (HIGH confidence)
|
||
- [BIFF.ai beat detection rundown](https://biff.ai/a-rundown-of-open-source-beat-detection-models/) — madmom vs BeatNet comparison (MEDIUM confidence, single source)
|
||
- [MPV IPC documentation](https://github.com/mpv-player/mpv/blob/master/DOCS/man/ipc.rst) — named pipe on Windows confirmed (HIGH confidence)
|
||
- [uvicorn release notes](https://uvicorn.dev/release-notes/) — 0.34.1 current stable, Windows ProactorEventLoop (HIGH confidence)
|
||
- WebSearch: FastAPI WebSocket patterns, async UDP asyncio, JSON show file formats — (MEDIUM confidence, multiple sources)
|
||
|
||
---
|
||
|
||
*Stack research for: LightSync — music-to-light synchronization system*
|
||
*Researched: 2026-04-05*
|