led2/.planning/phases/02-audio-engine/02-03-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, duration, completed

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	duration	completed
02-audio-engine	03	audio	librosa beat-detection browser-audio websocket waveform canvas animation-selector	phase provides 02-audio-engine/02-01 MPVEngine (now deleted — replaced by browser audio) phase provides 02-audio-engine/02-02 WebSocket hub, REST audio endpoints	Browser <audio> element handles playback (HTML5, user's speakers) librosa beat detection via lightsync/audio/beats.py — analyze(path) -> {tempo, beats} Beat data cached in app.state.beats (filepath -> beat data dict) GET /api/audio/beats?path=... — returns cached or on-demand beat analysis POST /api/audio/upload — saves file, runs beat analysis, caches result GET /api/audio/waveform?path=... — waveform peaks (now takes path query param) WebSocket tick/beat protocol — browser sends position ticks, server sends beat events /shows/{filename} static file serving — browser <audio src=...> loads uploaded files Beat marker overlay on waveform canvas (yellow tick lines at beat timestamps) Beat flash indicator in header (yellow dot animation on beat events) Animation selector panel with 4 placeholder options	03-communication-protocol 04-timeline-editor	added removed patterns librosa>=0.10.0 (beat detection, replaces python-mpv for audio analysis) python-mpv>=1.0.5 (server-side playback replaced by browser audio) mpv-libs Alpine package libmpv.so symlink in Dockerfile Browser sends {type: tick, position: N} at 10Hz; server finds nearest beat within 50ms Beat detection: librosa.beat.beat_track + frames_to_time, cached in app.state.beats Static /shows mount: FastAPI StaticFiles serves uploaded audio to browser bisect.bisect_left for O(log n) nearest-beat lookup per tick per-connection last_beat_reported prevents re-firing same beat Upload returns beat data inline — frontend caches without extra fetch	created deleted modified lightsync/audio/beats.py lightsync/audio/engine.py pyproject.toml Dockerfile lightsync/main.py lightsync/api/audio.py lightsync/api/ws.py lightsync/frontend/index.html lightsync/frontend/app.js lightsync/frontend/style.css	Browser plays audio — eliminates python-mpv, Docker libmpv dependency, and server-side position polling librosa.beat.beat_track over onset_detect — gives both tempo and beat grid with one call Beat cache keyed by filepath in app.state.beats — survives WebSocket reconnects 50ms BEAT_WINDOW_SEC tolerance — matches human perception of beat timing Upload response includes beat data inline — avoids second /api/audio/beats fetch after upload setInterval 10Hz + timeupdate fallback — belt-and-suspenders for position tick delivery last_beat_reported per-connection — prevents duplicate beat events during slow ticks	~4 min	2026-04-06

Phase 02 Plan 03: Browser Audio + Librosa Beat Detection (Architecture Redesign)

Browser plays audio via HTML5 element; server does beat analysis via librosa; WebSocket carries position ticks (browser) and beat notifications (server)

Performance

• Duration: ~4 min
• Started: 2026-04-06T15:35:26Z
• Completed: 2026-04-06T15:39:29Z
• Tasks: 9 auto commits (architecture redesign, no checkpoints)
• Files modified: 8 (1 created, 1 deleted)

Accomplishments

• Removed python-mpv from pyproject.toml and mpv-libs from Dockerfile entirely
• Created lightsync/audio/beats.py — analyze(path) using librosa.beat.beat_track, returns {tempo, beats} as timestamps in seconds
• Redesigned lightsync/api/audio.py — POST /upload runs beat analysis and caches; GET /beats returns cached or on-demand; GET /waveform now takes path query param; removed engine-dependent endpoints
• Redesigned lightsync/api/ws.py — removed server broadcast_loop; browser sends tick messages; server does 50ms beat proximity check and fires beat events back; bisect-based O(log n) lookup
• Updated lightsync/main.py — removed MPVEngine lifespan; added app.state.beats = {} cache; added /shows StaticFiles mount for browser audio loading
• Deleted lightsync/audio/engine.py — fully replaced by browser audio + librosa
• Frontend index.html — added hidden <audio id="player">, beat indicator dot in header, animation selector panel
• Frontend app.js — wires audio element events, 10Hz setInterval tick loop, beat flash on WS beat events, beat marker lines drawn on waveform canvas, loadSelectedFile sets audio.src
• Frontend style.css — beat-indicator with beat-flash animation, animation-select-input styles

Task Commits

1. 5d93294 — chore: replace python-mpv with librosa in pyproject.toml
2. cce76a2 — chore: remove mpv-libs from Dockerfile
3. 92cc09e — feat: add beats.py librosa beat detection module
4. 636160e — feat: redesign audio.py — browser-audio architecture, beat endpoint
5. 2c9ca0d — feat: redesign WebSocket protocol — browser tick-driven beat detection
6. 6366d84 — feat: remove MPVEngine from main.py, add beats cache and shows route
7. 1f1ff92 — chore: delete engine.py
8. f1941a9 — feat: redesign frontend — browser audio element, beat markers, animation selector
9. 0963ad4 — fix: upload file selection path consistency

Files Created/Modified

• lightsync/audio/beats.py — analyze(), analyze_async(), librosa beat_track wrapper
• lightsync/audio/engine.py — DELETED (MPV engine removed)
• lightsync/api/audio.py — upload+beats cache, GET /beats endpoint, waveform takes path param
• lightsync/api/ws.py — tick/beat protocol, bisect beat lookup, per-connection state
• lightsync/main.py — no engine, app.state.beats, /shows StaticFiles
• lightsync/frontend/index.html — audio element, beat indicator, animation selector
• lightsync/frontend/app.js — audio wiring, 10Hz ticks, beat flash, beat markers
• lightsync/frontend/style.css — beat-indicator, animation-select-input
• pyproject.toml — python-mpv removed, librosa added
• Dockerfile — mpv-libs removed

Decisions Made

• Browser handles audio playback — no server-side audio process or Docker audio dependency
• librosa.beat.beat_track provides both tempo and beat grid in one call, sufficient for Phase 2
• Beat cache in app.state.beats survives WebSocket reconnects and multiple client connections
• 50ms window balances precision vs. tick rate jitter (100ms tick interval)
• Static /shows mount allows browser to load audio files with a simple URL path

Deviations from Plan

This plan was a complete architecture redesign replacing the original 02-03 implementation. The original 02-03 built on MPV; this revision removes MPV entirely.

Changes vs Original 02-03-PLAN.md

Removed from plan:

• MPV waveform-based transport UI (seek bar, time display driven by WS tick from server)
• Server-side broadcast_loop polling MPV position

Added vs original plan (per user redesign spec):

• librosa beat detection module
• Browser audio element as playback source
• 10Hz client tick loop
• Beat proximity detection in WebSocket server
• Beat flash visual indicator
• Beat marker lines on waveform canvas
• Animation selector panel
• /shows StaticFiles mount

Auto-fixed Issues

[Rule 1 - Bug] Fixed upload path pre-selection inconsistency

• Found during: Upload handler review
• Issue: refreshFileList options used /app/shows/filename values but upload called it with /shows/filename, preventing auto-selection of uploaded file
• Fix: Pass full abs path from upload response directly to refreshFileList
• Files modified: lightsync/frontend/app.js
• Commit: 0963ad4

Known Stubs

• Animation selector (<select id="animation-select">) has placeholder options — no animation logic connected yet (Phase 3+ will wire UDP commands)
• WebSocket play/pause/seek messages from browser are logged only — no UDP trigger yet (deferred to Phase 3+ cue scheduler)

Next Phase Readiness

• Beat timestamps available via app.state.beats for any Phase 3+ cue scheduler
• /api/audio/beats endpoint provides analysis data for Phase 4 timeline editor beat-snap
• WebSocket protocol extended cleanly — Phase 3 can add beat → UDP animation dispatch
• Browser audio element is the master clock — Phase 5 live execution reads audio.currentTime

---

Phase: 02-audio-engine Completed: 2026-04-06