9c1ca03957
- 02-03-SUMMARY.md: full redesign summary, decisions, deviations documented - STATE.md: decisions updated, session recorded, progress at 100% (7/7 plans) - ROADMAP.md: Phase 2 confirmed Complete (3/3 plans)
172 lines
8.4 KiB
Markdown
172 lines
8.4 KiB
Markdown
---
|
|
phase: 02-audio-engine
|
|
plan: "03"
|
|
subsystem: audio
|
|
tags: [librosa, beat-detection, browser-audio, websocket, waveform, canvas, animation-selector]
|
|
|
|
requires:
|
|
- phase: 02-audio-engine/02-01
|
|
provides: MPVEngine (now deleted — replaced by browser audio)
|
|
- phase: 02-audio-engine/02-02
|
|
provides: WebSocket hub, REST audio endpoints
|
|
|
|
provides:
|
|
- Browser <audio> element handles playback (HTML5, user's speakers)
|
|
- librosa beat detection via lightsync/audio/beats.py — analyze(path) -> {tempo, beats}
|
|
- Beat data cached in app.state.beats (filepath -> beat data dict)
|
|
- GET /api/audio/beats?path=... — returns cached or on-demand beat analysis
|
|
- POST /api/audio/upload — saves file, runs beat analysis, caches result
|
|
- GET /api/audio/waveform?path=... — waveform peaks (now takes path query param)
|
|
- WebSocket tick/beat protocol — browser sends position ticks, server sends beat events
|
|
- /shows/{filename} static file serving — browser <audio src=...> loads uploaded files
|
|
- Beat marker overlay on waveform canvas (yellow tick lines at beat timestamps)
|
|
- Beat flash indicator in header (yellow dot animation on beat events)
|
|
- Animation selector panel with 4 placeholder options
|
|
|
|
affects: [03-communication-protocol, 04-timeline-editor]
|
|
|
|
tech-stack:
|
|
added:
|
|
- librosa>=0.10.0 (beat detection, replaces python-mpv for audio analysis)
|
|
removed:
|
|
- python-mpv>=1.0.5 (server-side playback replaced by browser audio)
|
|
- mpv-libs Alpine package
|
|
- libmpv.so symlink in Dockerfile
|
|
patterns:
|
|
- "Browser sends {type: tick, position: N} at 10Hz; server finds nearest beat within 50ms"
|
|
- "Beat detection: librosa.beat.beat_track + frames_to_time, cached in app.state.beats"
|
|
- "Static /shows mount: FastAPI StaticFiles serves uploaded audio to browser"
|
|
- "bisect.bisect_left for O(log n) nearest-beat lookup per tick"
|
|
- "per-connection last_beat_reported prevents re-firing same beat"
|
|
- "Upload returns beat data inline — frontend caches without extra fetch"
|
|
|
|
key-files:
|
|
created:
|
|
- lightsync/audio/beats.py
|
|
deleted:
|
|
- lightsync/audio/engine.py
|
|
modified:
|
|
- pyproject.toml
|
|
- Dockerfile
|
|
- lightsync/main.py
|
|
- lightsync/api/audio.py
|
|
- lightsync/api/ws.py
|
|
- lightsync/frontend/index.html
|
|
- lightsync/frontend/app.js
|
|
- lightsync/frontend/style.css
|
|
|
|
key-decisions:
|
|
- "Browser plays audio — eliminates python-mpv, Docker libmpv dependency, and server-side position polling"
|
|
- "librosa.beat.beat_track over onset_detect — gives both tempo and beat grid with one call"
|
|
- "Beat cache keyed by filepath in app.state.beats — survives WebSocket reconnects"
|
|
- "50ms BEAT_WINDOW_SEC tolerance — matches human perception of beat timing"
|
|
- "Upload response includes beat data inline — avoids second /api/audio/beats fetch after upload"
|
|
- "setInterval 10Hz + timeupdate fallback — belt-and-suspenders for position tick delivery"
|
|
- "last_beat_reported per-connection — prevents duplicate beat events during slow ticks"
|
|
|
|
duration: ~4 min
|
|
completed: 2026-04-06
|
|
---
|
|
|
|
# Phase 02 Plan 03: Browser Audio + Librosa Beat Detection (Architecture Redesign)
|
|
|
|
**Browser plays audio via HTML5 <audio> element; server does beat analysis via librosa; WebSocket carries position ticks (browser) and beat notifications (server)**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** ~4 min
|
|
- **Started:** 2026-04-06T15:35:26Z
|
|
- **Completed:** 2026-04-06T15:39:29Z
|
|
- **Tasks:** 9 auto commits (architecture redesign, no checkpoints)
|
|
- **Files modified:** 8 (1 created, 1 deleted)
|
|
|
|
## Accomplishments
|
|
|
|
- Removed python-mpv from pyproject.toml and mpv-libs from Dockerfile entirely
|
|
- Created `lightsync/audio/beats.py` — `analyze(path)` using librosa.beat.beat_track, returns `{tempo, beats}` as timestamps in seconds
|
|
- Redesigned `lightsync/api/audio.py` — POST /upload runs beat analysis and caches; GET /beats returns cached or on-demand; GET /waveform now takes path query param; removed engine-dependent endpoints
|
|
- Redesigned `lightsync/api/ws.py` — removed server broadcast_loop; browser sends tick messages; server does 50ms beat proximity check and fires beat events back; bisect-based O(log n) lookup
|
|
- Updated `lightsync/main.py` — removed MPVEngine lifespan; added `app.state.beats = {}` cache; added `/shows` StaticFiles mount for browser audio loading
|
|
- Deleted `lightsync/audio/engine.py` — fully replaced by browser audio + librosa
|
|
- Frontend `index.html` — added hidden `<audio id="player">`, beat indicator dot in header, animation selector panel
|
|
- Frontend `app.js` — wires audio element events, 10Hz setInterval tick loop, beat flash on WS beat events, beat marker lines drawn on waveform canvas, loadSelectedFile sets audio.src
|
|
- Frontend `style.css` — beat-indicator with beat-flash animation, animation-select-input styles
|
|
|
|
## Task Commits
|
|
|
|
1. `5d93294` — chore: replace python-mpv with librosa in pyproject.toml
|
|
2. `cce76a2` — chore: remove mpv-libs from Dockerfile
|
|
3. `92cc09e` — feat: add beats.py librosa beat detection module
|
|
4. `636160e` — feat: redesign audio.py — browser-audio architecture, beat endpoint
|
|
5. `2c9ca0d` — feat: redesign WebSocket protocol — browser tick-driven beat detection
|
|
6. `6366d84` — feat: remove MPVEngine from main.py, add beats cache and shows route
|
|
7. `1f1ff92` — chore: delete engine.py
|
|
8. `f1941a9` — feat: redesign frontend — browser audio element, beat markers, animation selector
|
|
9. `0963ad4` — fix: upload file selection path consistency
|
|
|
|
## Files Created/Modified
|
|
|
|
- `lightsync/audio/beats.py` — analyze(), analyze_async(), librosa beat_track wrapper
|
|
- `lightsync/audio/engine.py` — DELETED (MPV engine removed)
|
|
- `lightsync/api/audio.py` — upload+beats cache, GET /beats endpoint, waveform takes path param
|
|
- `lightsync/api/ws.py` — tick/beat protocol, bisect beat lookup, per-connection state
|
|
- `lightsync/main.py` — no engine, app.state.beats, /shows StaticFiles
|
|
- `lightsync/frontend/index.html` — audio element, beat indicator, animation selector
|
|
- `lightsync/frontend/app.js` — audio wiring, 10Hz ticks, beat flash, beat markers
|
|
- `lightsync/frontend/style.css` — beat-indicator, animation-select-input
|
|
- `pyproject.toml` — python-mpv removed, librosa added
|
|
- `Dockerfile` — mpv-libs removed
|
|
|
|
## Decisions Made
|
|
|
|
- Browser handles audio playback — no server-side audio process or Docker audio dependency
|
|
- librosa.beat.beat_track provides both tempo and beat grid in one call, sufficient for Phase 2
|
|
- Beat cache in app.state.beats survives WebSocket reconnects and multiple client connections
|
|
- 50ms window balances precision vs. tick rate jitter (100ms tick interval)
|
|
- Static /shows mount allows browser to load audio files with a simple URL path
|
|
|
|
## Deviations from Plan
|
|
|
|
This plan was a complete architecture redesign replacing the original 02-03 implementation. The original 02-03 built on MPV; this revision removes MPV entirely.
|
|
|
|
### Changes vs Original 02-03-PLAN.md
|
|
|
|
**Removed from plan:**
|
|
- MPV waveform-based transport UI (seek bar, time display driven by WS tick from server)
|
|
- Server-side broadcast_loop polling MPV position
|
|
|
|
**Added vs original plan (per user redesign spec):**
|
|
- librosa beat detection module
|
|
- Browser audio element as playback source
|
|
- 10Hz client tick loop
|
|
- Beat proximity detection in WebSocket server
|
|
- Beat flash visual indicator
|
|
- Beat marker lines on waveform canvas
|
|
- Animation selector panel
|
|
- /shows StaticFiles mount
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**[Rule 1 - Bug] Fixed upload path pre-selection inconsistency**
|
|
- **Found during:** Upload handler review
|
|
- **Issue:** refreshFileList options used `/app/shows/filename` values but upload called it with `/shows/filename`, preventing auto-selection of uploaded file
|
|
- **Fix:** Pass full abs path from upload response directly to refreshFileList
|
|
- **Files modified:** lightsync/frontend/app.js
|
|
- **Commit:** 0963ad4
|
|
|
|
## Known Stubs
|
|
|
|
- Animation selector (`<select id="animation-select">`) has placeholder options — no animation logic connected yet (Phase 3+ will wire UDP commands)
|
|
- WebSocket play/pause/seek messages from browser are logged only — no UDP trigger yet (deferred to Phase 3+ cue scheduler)
|
|
|
|
## Next Phase Readiness
|
|
|
|
- Beat timestamps available via app.state.beats for any Phase 3+ cue scheduler
|
|
- /api/audio/beats endpoint provides analysis data for Phase 4 timeline editor beat-snap
|
|
- WebSocket protocol extended cleanly — Phase 3 can add beat → UDP animation dispatch
|
|
- Browser audio element is the master clock — Phase 5 live execution reads audio.currentTime
|
|
|
|
---
|
|
*Phase: 02-audio-engine*
|
|
*Completed: 2026-04-06*
|