---
phase: 02-audio-engine
plan: "03"
subsystem: audio
tags: [librosa, beat-detection, browser-audio, websocket, waveform, canvas, animation-selector]

requires:
  - phase: 02-audio-engine/02-01
    provides: MPVEngine (now deleted — replaced by browser audio)
  - phase: 02-audio-engine/02-02
    provides: WebSocket hub, REST audio endpoints

provides:
  - Browser <audio> element handles playback (HTML5, user's speakers)
  - librosa beat detection via lightsync/audio/beats.py — analyze(path) -> {tempo, beats}
  - Beat data cached in app.state.beats (filepath -> beat data dict)
  - GET /api/audio/beats?path=... — returns cached or on-demand beat analysis
  - POST /api/audio/upload — saves file, runs beat analysis, caches result
  - GET /api/audio/waveform?path=... — waveform peaks (now takes path query param)
  - WebSocket tick/beat protocol — browser sends position ticks, server sends beat events
  - /shows/{filename} static file serving — browser <audio src=...> loads uploaded files
  - Beat marker overlay on waveform canvas (yellow tick lines at beat timestamps)
  - Beat flash indicator in header (yellow dot animation on beat events)
  - Animation selector panel with 4 placeholder options

affects: [03-communication-protocol, 04-timeline-editor]

tech-stack:
  added:
    - librosa>=0.10.0 (beat detection, replaces python-mpv for audio analysis)
  removed:
    - python-mpv>=1.0.5 (server-side playback replaced by browser audio)
    - mpv-libs Alpine package
    - libmpv.so symlink in Dockerfile
  patterns:
    - "Browser sends {type: tick, position: N} at 10Hz; server finds nearest beat within 50ms"
    - "Beat detection: librosa.beat.beat_track + frames_to_time, cached in app.state.beats"
    - "Static /shows mount: FastAPI StaticFiles serves uploaded audio to browser"
    - "bisect.bisect_left for O(log n) nearest-beat lookup per tick"
    - "per-connection last_beat_reported prevents re-firing same beat"
    - "Upload returns beat data inline — frontend caches without extra fetch"

key-files:
  created:
    - lightsync/audio/beats.py
  deleted:
    - lightsync/audio/engine.py
  modified:
    - pyproject.toml
    - Dockerfile
    - lightsync/main.py
    - lightsync/api/audio.py
    - lightsync/api/ws.py
    - lightsync/frontend/index.html
    - lightsync/frontend/app.js
    - lightsync/frontend/style.css

key-decisions:
  - "Browser plays audio — eliminates python-mpv, Docker libmpv dependency, and server-side position polling"
  - "librosa.beat.beat_track over onset_detect — gives both tempo and beat grid with one call"
  - "Beat cache keyed by filepath in app.state.beats — survives WebSocket reconnects"
  - "50ms BEAT_WINDOW_SEC tolerance — matches human perception of beat timing"
  - "Upload response includes beat data inline — avoids second /api/audio/beats fetch after upload"
  - "setInterval 10Hz + timeupdate fallback — belt-and-suspenders for position tick delivery"
  - "last_beat_reported per-connection — prevents duplicate beat events during slow ticks"

duration: ~4 min
completed: 2026-04-06
---

# Phase 02 Plan 03: Browser Audio + Librosa Beat Detection (Architecture Redesign)

**Browser plays audio via HTML5 <audio> element; server does beat analysis via librosa; WebSocket carries position ticks (browser) and beat notifications (server)**

## Performance

- **Duration:** ~4 min
- **Started:** 2026-04-06T15:35:26Z
- **Completed:** 2026-04-06T15:39:29Z
- **Tasks:** 9 auto commits (architecture redesign, no checkpoints)
- **Files modified:** 8 (1 created, 1 deleted)

## Accomplishments

- Removed python-mpv from pyproject.toml and mpv-libs from Dockerfile entirely
- Created `lightsync/audio/beats.py` — `analyze(path)` using librosa.beat.beat_track, returns `{tempo, beats}` as timestamps in seconds
- Redesigned `lightsync/api/audio.py` — POST /upload runs beat analysis and caches; GET /beats returns cached or on-demand; GET /waveform now takes path query param; removed engine-dependent endpoints
- Redesigned `lightsync/api/ws.py` — removed server broadcast_loop; browser sends tick messages; server does 50ms beat proximity check and fires beat events back; bisect-based O(log n) lookup
- Updated `lightsync/main.py` — removed MPVEngine lifespan; added `app.state.beats = {}` cache; added `/shows` StaticFiles mount for browser audio loading
- Deleted `lightsync/audio/engine.py` — fully replaced by browser audio + librosa
- Frontend `index.html` — added hidden `<audio id="player">`, beat indicator dot in header, animation selector panel
- Frontend `app.js` — wires audio element events, 10Hz setInterval tick loop, beat flash on WS beat events, beat marker lines drawn on waveform canvas, loadSelectedFile sets audio.src
- Frontend `style.css` — beat-indicator with beat-flash animation, animation-select-input styles

## Task Commits

1. `5d93294` — chore: replace python-mpv with librosa in pyproject.toml
2. `cce76a2` — chore: remove mpv-libs from Dockerfile
3. `92cc09e` — feat: add beats.py librosa beat detection module
4. `636160e` — feat: redesign audio.py — browser-audio architecture, beat endpoint
5. `2c9ca0d` — feat: redesign WebSocket protocol — browser tick-driven beat detection
6. `6366d84` — feat: remove MPVEngine from main.py, add beats cache and shows route
7. `1f1ff92` — chore: delete engine.py
8. `f1941a9` — feat: redesign frontend — browser audio element, beat markers, animation selector
9. `0963ad4` — fix: upload file selection path consistency

## Files Created/Modified

- `lightsync/audio/beats.py` — analyze(), analyze_async(), librosa beat_track wrapper
- `lightsync/audio/engine.py` — DELETED (MPV engine removed)
- `lightsync/api/audio.py` — upload+beats cache, GET /beats endpoint, waveform takes path param
- `lightsync/api/ws.py` — tick/beat protocol, bisect beat lookup, per-connection state
- `lightsync/main.py` — no engine, app.state.beats, /shows StaticFiles
- `lightsync/frontend/index.html` — audio element, beat indicator, animation selector
- `lightsync/frontend/app.js` — audio wiring, 10Hz ticks, beat flash, beat markers
- `lightsync/frontend/style.css` — beat-indicator, animation-select-input
- `pyproject.toml` — python-mpv removed, librosa added
- `Dockerfile` — mpv-libs removed

## Decisions Made

- Browser handles audio playback — no server-side audio process or Docker audio dependency
- librosa.beat.beat_track provides both tempo and beat grid in one call, sufficient for Phase 2
- Beat cache in app.state.beats survives WebSocket reconnects and multiple client connections
- 50ms window balances precision vs. tick rate jitter (100ms tick interval)
- Static /shows mount allows browser to load audio files with a simple URL path

## Deviations from Plan

This plan was a complete architecture redesign replacing the original 02-03 implementation. The original 02-03 built on MPV; this revision removes MPV entirely.

### Changes vs Original 02-03-PLAN.md

**Removed from plan:**
- MPV waveform-based transport UI (seek bar, time display driven by WS tick from server)
- Server-side broadcast_loop polling MPV position

**Added vs original plan (per user redesign spec):**
- librosa beat detection module
- Browser audio element as playback source
- 10Hz client tick loop
- Beat proximity detection in WebSocket server
- Beat flash visual indicator
- Beat marker lines on waveform canvas
- Animation selector panel
- /shows StaticFiles mount

### Auto-fixed Issues

**[Rule 1 - Bug] Fixed upload path pre-selection inconsistency**
- **Found during:** Upload handler review
- **Issue:** refreshFileList options used `/app/shows/filename` values but upload called it with `/shows/filename`, preventing auto-selection of uploaded file
- **Fix:** Pass full abs path from upload response directly to refreshFileList
- **Files modified:** lightsync/frontend/app.js
- **Commit:** 0963ad4

## Known Stubs

- Animation selector (`<select id="animation-select">`) has placeholder options — no animation logic connected yet (Phase 3+ will wire UDP commands)
- WebSocket play/pause/seek messages from browser are logged only — no UDP trigger yet (deferred to Phase 3+ cue scheduler)

## Next Phase Readiness

- Beat timestamps available via app.state.beats for any Phase 3+ cue scheduler
- /api/audio/beats endpoint provides analysis data for Phase 4 timeline editor beat-snap
- WebSocket protocol extended cleanly — Phase 3 can add beat → UDP animation dispatch
- Browser audio element is the master clock — Phase 5 live execution reads audio.currentTime

---
*Phase: 02-audio-engine*
*Completed: 2026-04-06*