Gitea-Admin/led2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add domain research — stack, features, architecture, pitfalls

Browse Source
This commit is contained in:
Claude
2026-04-05 17:59:51 +00:00
parent 3d756129fa
commit 6e05abe7a8
5 changed files with 1592 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

214
.planning/research/STACK.md Normal file
View File

@@ -0,0 +1,214 @@
# Stack Research
**Domain:** Music-to-light synchronization system (Python backend, browser frontend, UDP control)
**Researched:** 2026-04-05
**Confidence:** MEDIUM-HIGH (core stack HIGH, audio analysis MEDIUM, AI sync LOW)
---
## Recommended Stack
### Core Technologies
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| Python | 3.11 | Runtime | 3.11 hits the sweet spot: modern async, good Windows support, librosa 0.11 compatible. Avoid 3.12+ until madmom resolves its incompatibility. Avoid 3.9 as it's EOL in Oct 2025. |
| FastAPI | 0.135.x | Web framework + REST API + WebSocket server | Native async/await, built-in WebSocket support, Pydantic validation, serves static files — one process covers API + UI + realtime push. Flask is sync-first and requires extensions for every feature this app needs. |
| uvicorn | 0.34.x | ASGI server | Standard FastAPI runner, works on Windows without Gunicorn. Use `uvicorn.run()` programmatically so the app can self-start without a separate process manager. |
| librosa | 0.11.0 | Beat detection, onset analysis, tempo estimation | Released March 2025, supports NumPy 2.0, actively maintained. Correct choice for offline audio analysis from files. scipy FFT backend is more accurate than numpy. |
| python-mpv | 1.0.8 | MPV process control + playback position polling | Released April 2025. ctypes-based — gives property observers and event hooks inside Python. Better than raw subprocess for position tracking. Windows uses named pipe IPC automatically. |
| Pydantic | 2.x (bundled with FastAPI) | Show file validation, device registry schemas, config | Already a FastAPI dependency. Use BaseModel for all JSON show file structures — free validation and serialization. |
### Supporting Libraries
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| numpy | 2.0+ | Array math for pixel frame buffers, beat arrays | Always — librosa depends on it, and frame-based UDP payloads are numpy arrays before packing |
| scipy | 1.14+ | FFT, signal processing (librosa backend) | Always — librosa uses it as FFT backend since 0.11.0 |
| soundfile | 0.12+ | Audio file loading (WAV, FLAC, OGG) for librosa | When loading local audio files for analysis pre-playback |
| aiofiles | 24.x | Async file I/O for show file read/write | When persisting show files from async FastAPI handlers |
| SQLite (stdlib) | built-in | Persistent storage for devices, shows, project state | Use Python's built-in `sqlite3` or `aiosqlite` for async; no ORM needed for this scale |
| aiosqlite | 0.20+ | Async SQLite wrapper | For non-blocking database access from FastAPI async handlers |
| structlog | 24.x | Structured logging | When you want JSON-formatted logs for the backend; easier to parse than stdlib logging |
| python-dotenv | 1.x | Environment config (host, port, UDP targets) | For externalizing runtime config without hardcoding |
| madmom | 0.16 | High-accuracy beat tracking (DBN model) | ONLY if using Python 3.9 in a separate subprocess or venv — madmom is incompatible with Python 3.10+. Use as an optional enhancement path, not core dependency. |
### Frontend Stack
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| Vanilla JS (ES2022+) | native | Timeline editor, WebSocket client, UI interactions | This app has a custom DAW-like UI — no component framework matches the drag-drop timeline without fighting abstractions. React adds 200KB+ and solves problems this app doesn't have. |
| Native WebSocket API | native | Real-time playback position updates from backend | Browser-native, zero dependencies, sufficient for position polling and event pushes |
| CSS Custom Properties + Grid/Flexbox | native | Terminal aesthetic: dark theme, monospace, track layout | No CSS framework needed — a bespoke terminal aesthetic is easier to build raw than to override Bootstrap or Tailwind |
| Web Audio API | native | Waveform visualization (optional) | If displaying waveform in timeline, decode via AudioContext.decodeAudioData() — no library needed |
### Development Tools
| Tool | Purpose | Notes |
|------|---------|-------|
| uv | Python package manager + venv | Faster than pip, reproducible installs, handles Windows path quirks well |
| pyinstaller (optional) | Bundle to Windows .exe | Only if distributing; not needed for personal dev machine use |
---
## Installation
```bash
# Core backend
uv pip install fastapi==0.135.3 uvicorn==0.34.1 python-mpv==1.0.8 pydantic==2.10.x
# Audio analysis
uv pip install librosa==0.11.0 soundfile==0.12.1
# Persistence + async
uv pip install aiosqlite==0.20.0 aiofiles==24.1.0
# Config + utilities
uv pip install python-dotenv==1.0.1 structlog==24.4.0
```
Note: `numpy` and `scipy` install automatically as librosa dependencies. Do not pin them separately unless you hit a conflict.
---
## Alternatives Considered
| Recommended | Alternative | When to Use Alternative |
|-------------|-------------|-------------------------|
| FastAPI | Flask | If you already have a Flask app or need Jinja templating; Flask-SocketIO is a viable WebSocket path but adds complexity |
| FastAPI | Django | Never for this project — Django's ORM and admin are overhead for a single-operator music sync tool |
| Vanilla JS | React | If the team knows React and the timeline editor becomes a complex stateful SPA (drag-drop state machines, undo/redo) that benefits from component isolation |
| Vanilla JS | HTMX | HTMX is wrong for this use case — the timeline editor requires fine-grained JS state, not server-rendered HTML swaps |
| python-mpv (ctypes) | python-mpv-jsonipc | Use jsonipc variant if libmpv.dll is unavailable on the target Windows system (jsonipc controls an external mpv.exe via named pipe, no DLL required) |
| python-mpv | subprocess + JSON IPC manual | More control, no DLL dependency, but requires implementing the IPC protocol yourself |
| librosa (offline) | madmom (offline) | madmom has better beat accuracy for complex rhythms, BUT it is incompatible with Python 3.10+. Use only in an isolated subprocess with Python 3.9 if high-accuracy tracking is critical. |
| aiosqlite | SQLAlchemy + SQLModel | Use SQLAlchemy if the schema grows complex (multiple joined tables, migrations). For show files + device registry, raw aiosqlite is sufficient. |
| SQLite | JSON flat files | JSON files are fine for show data but lose query capability for device registry lookups and show search. SQLite has zero deployment overhead on Windows. |
---
## What NOT to Use
| Avoid | Why | Use Instead |
|-------|-----|-------------|
| aubio | Last release: February 2019 (0.4.9). Unmaintained, no Python 3.10+ wheels, Windows install is fragile. Described as "Beta" still after 6 years. | librosa for offline analysis |
| madmom as core dependency | Incompatible with Python 3.10+ due to `collections.MutableSequence` removal. Requires Python ≤ 3.9. Outstanding GitHub issues with no resolution timeline. | librosa as primary; madmom only in isolated venv if needed |
| essentia | Excellent accuracy, but C++ binding installation on Windows is non-trivial. Pre-built wheels are inconsistent. Adds significant complexity for marginal beat detection gain. | librosa covers the use case well on Windows |
| BeatNet | Requires madmom as a dependency, inheriting the Python 3.10+ incompatibility problem. Also requires PyTorch, which is 200–800MB. Overkill for a personal project. | librosa; revisit if AI sync accuracy becomes a priority |
| gevent / eventlet | Old async approaches, sometimes required with Flask-SocketIO. Not needed with FastAPI's native async. | FastAPI + uvicorn native async |
| Flask-SocketIO | Adds an abstraction layer over Socket.IO protocol instead of raw WebSocket. Overkill, and Socket.IO client required on frontend. | FastAPI native WebSocket |
| PyAudio / sounddevice for playback | This project uses MPV for playback. Mixing MPV subprocess with Python audio I/O for the same audio creates timing conflicts. | python-mpv for all playback |
| Art-Net / sACN / E1.31 protocol | These are DMX-over-UDP protocols designed for theatrical equipment. The project uses custom UDP packets to custom firmware — don't add DMX protocol overhead. | Raw UDP asyncio datagrams |
---
## Stack Patterns by Variant
**For beat detection on local audio files:**
- Load with `soundfile.read()` → pass to `librosa.beat.beat_track(y, sr)` → returns tempo + beat frame indices
- Convert frames to timestamps: `librosa.frames_to_time(beats, sr=sr)`
- Onset detection for drop/hit detection: `librosa.onset.onset_detect(y, sr, units='time')`
**For UDP frame sending (AI-generated pixel sequences):**
- Pack RGB values into bytes: `struct.pack('B' * (led_count * 3), *flat_rgb_array)`
- Send via asyncio datagram endpoint: `transport.sendto(payload, (target_ip, target_port))`
- Fire-and-forget: no acknowledgement, send at frame rate (e.g., 30fps = 33ms interval)
**For UDP animation+params sending (manual mode):**
- Small JSON payload or fixed binary struct: `{"anim": "pulse", "color": [255, 0, 128], "speed": 1.5}`
- Encode as JSON bytes or define a compact binary protocol with `struct.pack`
- JSON is easier to debug during early dev; migrate to binary if bandwidth becomes an issue
**For playback position sync (frontend timeline scrubbing):**
- Use `python-mpv` property observer: `player.observe_property('time-pos', callback)`
- In callback, push position via WebSocket to all connected browser clients
- Frontend timeline animates playhead based on received position events
**For AI sync (optional enhancement — LOW confidence):**
- No mature turnkey Python library for music-to-light AI in 2025 that runs on Windows without GPU
- Practical path: run librosa full analysis (beat + onset + chroma + spectral contrast), feed features into a rule-based mapper (loud = bright, beat = flash, drop = color change)
- True AI path: fine-tune a small model offline using MIDI-to-light training data — deferred to later phases
- OpenAI/Claude audio analysis APIs could be used to get high-level song structure (verse/chorus/bridge) from a transcript or description, but audio file analysis APIs do not exist at useful granularity for per-beat sync
**For Windows 11 development:**
- Use `python-mpv` with `libmpv-2.dll` placed next to the app (ship with the project)
- Alternatively, use `python-mpv-jsonipc` which controls `mpv.exe` via named pipe — no DLL needed
- Named pipe path: `\\.\pipe\lightsync_mpv`
- UDP `asyncio` works natively on Windows with `ProactorEventLoop` (Python 3.8+ default on Windows)
---
## JSON Show File Format
No industry-standard JSON schema exists for this exact use case. Custom format recommended.
```json
{
"version": "1.0",
"meta": {
"title": "My Show",
"audio_source": "file:///path/to/song.mp3",
"duration_seconds": 213.4,
"bpm": 128.0,
"created_at": "2026-04-05T12:00:00Z"
},
"beats": [0.47, 0.94, 1.41],
"devices": [
{
"id": "wall_strip",
"name": "Wall Strip",
"type": "SK6812",
"led_count": 300,
"target_ip": "192.168.1.50",
"port": 4210
}
],
"timeline": {
"wall_strip": [
{
"start": 0.47,
"end": 1.41,
"animation": "pulse",
"params": {"color": [255, 0, 128], "speed": 1.0}
}
]
}
}
```
Key design decisions for the format:
- Timestamps in seconds (float) — not frames or ticks. Decoupled from sample rate.
- Beat array stored in show file — precomputed at import time, not recalculated on playback.
- Device definitions embedded in show file — show is self-contained, no external registry lookup required during playback.
- Animation params as open object — allows per-animation parameter schemas without breaking the format.
---
## Version Compatibility
| Package | Compatible With | Notes |
|---------|-----------------|-------|
| librosa 0.11.0 | numpy >= 1.17, scipy >= 1.0, Python 3.8+ | Requires soundfile for local file loading |
| python-mpv 1.0.8 | Python >= 3.9, libmpv 0.35+ | libmpv-2.dll required on Windows; ship with app |
| FastAPI 0.135.x | pydantic v2, Python 3.8+, uvicorn 0.34+ | Pydantic v1 compatibility mode dropped |
| madmom 0.16 | Python < 3.10 ONLY | Do not install in same venv as main app if using Python 3.11+ |
| numpy 2.0+ | librosa 0.11.0, scipy 1.14+, soundfile 0.12+ | librosa 0.11.0 explicitly added NumPy 2.0 support |
---
## Sources
- [FastAPI release notes](https://fastapi.tiangolo.com/release-notes/) — version 0.135.3 confirmed current (HIGH confidence)
- [librosa 0.11.0 changelog](https://librosa.org/doc/main/changelog.html) — released March 2025, NumPy 2.0 support confirmed (HIGH confidence)
- [python-mpv PyPI](https://pypi.org/project/python-mpv/) — version 1.0.8 released April 2025, Windows support confirmed (HIGH confidence)
- [aubio PyPI](https://pypi.org/project/aubio/) — last release 2019, confirmed stale (HIGH confidence)
- [madmom Python 3.10+ incompatibility — GitHub issue](https://github.com/CPJKU/beat_this/issues/9) — confirmed broken on Python >= 3.10 (HIGH confidence)
- [BIFF.ai beat detection rundown](https://biff.ai/a-rundown-of-open-source-beat-detection-models/) — madmom vs BeatNet comparison (MEDIUM confidence, single source)
- [MPV IPC documentation](https://github.com/mpv-player/mpv/blob/master/DOCS/man/ipc.rst) — named pipe on Windows confirmed (HIGH confidence)
- [uvicorn release notes](https://uvicorn.dev/release-notes/) — 0.34.1 current stable, Windows ProactorEventLoop (HIGH confidence)
- WebSearch: FastAPI WebSocket patterns, async UDP asyncio, JSON show file formats — (MEDIUM confidence, multiple sources)
---
*Stack research for: LightSync — music-to-light synchronization system*
*Researched: 2026-04-05*