docs(02-03): complete architecture redesign — browser audio + librosa beat detection

- 02-03-SUMMARY.md: full redesign summary, decisions, deviations documented - STATE.md: decisions updated, session recorded, progress at 100% (7/7 plans) - ROADMAP.md: Phase 2 confirmed Complete (3/3 plans)
2026-04-06 15:40:55 +00:00
parent 0963ad4a70
commit 9c1ca03957
2 changed files with 119 additions and 74 deletions
--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@@ -3,8 +3,8 @@ gsd_state_version: 1.0
 milestone: v1.0
 milestone_name: milestone
 status: verifying
-stopped_at: "Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation"
-last_updated: "2026-04-06T13:15:05.979Z"
+stopped_at: "Completed 02-audio-engine/02-03-PLAN.md (architecture redesign: browser audio + librosa)"
+last_updated: "2026-04-06T15:40:50.951Z"
 last_activity: 2026-04-06
 progress:
  total_phases: 7
@@ -58,6 +58,7 @@ Progress: [░░░░░░░░░░] 0%
 | Phase 02-audio-engine P01 | 525603 | 2 tasks | 5 files |
 | Phase 02-audio-engine P02 | 8 | 2 tasks | 3 files |
 | Phase 02-audio-engine P03 | 8min | 2 tasks | 5 files |
+| Phase 02-audio-engine P03 | 4 | 9 tasks | 8 files |

 ## Accumulated Context

@@ -85,6 +86,8 @@ Recent decisions affecting current work:
 - [Phase 02-audio-engine]: AUD-02 (YouTube URL loading) deferred to Phase 6 — noted in docstring, not implemented in audio.py
 - [Phase 02-audio-engine]: ffmpeg subprocess pipe for MP3 waveform extraction (libsndfile/soundfile does not support MP3 natively)
 - [Phase 02-audio-engine]: Transport state driven by WebSocket tick — no frontend polling
+- [Phase 02-audio-engine]: Browser plays audio via HTML5 audio element — eliminates python-mpv, Docker libmpv dependency, and server-side position polling
+- [Phase 02-audio-engine]: librosa.beat.beat_track for beat detection — tempo + beat grid in one call, cached in app.state.beats by filepath

 ### Pending Todos

@@ -98,8 +101,8 @@ None yet.

 ## Session Continuity

-Last session: 2026-04-06T13:15:05.973Z
-Stopped at: Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation
+Last session: 2026-04-06T15:40:34.296Z
+Stopped at: Completed 02-audio-engine/02-03-PLAN.md (architecture redesign: browser audio + librosa)
 Resume file: None

 Next action: `/gsd:plan-phase 1`
--- a/.planning/phases/02-audio-engine/02-03-SUMMARY.md
+++ b/.planning/phases/02-audio-engine/02-03-SUMMARY.md
@@ -2,127 +2,169 @@
 phase: 02-audio-engine
 plan: "03"
 subsystem: audio
-
-tags: [waveform, soundfile, numpy, ffmpeg, transport-ui, canvas, websocket]
+tags: [librosa, beat-detection, browser-audio, websocket, waveform, canvas, animation-selector]

 requires:
  - phase: 02-audio-engine/02-01
-    provides: MPVEngine with get_state(), play/pause/seek via WebSocket
+    provides: MPVEngine (now deleted — replaced by browser audio)
  - phase: 02-audio-engine/02-02
-    provides: WebSocket broadcast loop (tick messages), REST audio endpoints in lightsync/api/audio.py
+    provides: WebSocket hub, REST audio endpoints

 provides:
-  - extract_peaks() function in lightsync/audio/waveform.py (AUD-05)
-  - GET /api/audio/waveform endpoint returning downsampled peak array
-  - Transport bar UI with play/pause/stop/seek/load controls
-  - Waveform canvas in timeline panel with position cursor
-  - Real-time position cursor via WebSocket tick dispatch in handleMessage
+  - Browser <audio> element handles playback (HTML5, user's speakers)
+  - librosa beat detection via lightsync/audio/beats.py — analyze(path) -> {tempo, beats}
+  - Beat data cached in app.state.beats (filepath -> beat data dict)
+  - GET /api/audio/beats?path=... — returns cached or on-demand beat analysis
+  - POST /api/audio/upload — saves file, runs beat analysis, caches result
+  - GET /api/audio/waveform?path=... — waveform peaks (now takes path query param)
+  - WebSocket tick/beat protocol — browser sends position ticks, server sends beat events
+  - /shows/{filename} static file serving — browser <audio src=...> loads uploaded files
+  - Beat marker overlay on waveform canvas (yellow tick lines at beat timestamps)
+  - Beat flash indicator in header (yellow dot animation on beat events)
+  - Animation selector panel with 4 placeholder options

-affects: [03-timeline-editor, 04-show-builder]
+affects: [03-communication-protocol, 04-timeline-editor]

 tech-stack:
-  added: [soundfile, numpy (waveform extraction), ffmpeg (mp3 via subprocess pipe)]
+  added:
+    - librosa>=0.10.0 (beat detection, replaces python-mpv for audio analysis)
+  removed:
+    - python-mpv>=1.0.5 (server-side playback replaced by browser audio)
+    - mpv-libs Alpine package
+    - libmpv.so symlink in Dockerfile
  patterns:
-    - "MP3 via ffmpeg pipe: subprocess.run(['ffmpeg', '-f', 'f32le', 'pipe:1']) + np.frombuffer"
-    - "asyncio.to_thread for waveform extraction (seconds-long operation)"
-    - "Transport button visibility toggle: btnPlay/btnPause display swap on tick paused state"
-    - "Seek bar _dragging flag prevents position jumps while user scrubs"
-    - "Waveform click-to-seek: clientX/rect.left normalized to audioDuration"
+    - "Browser sends {type: tick, position: N} at 10Hz; server finds nearest beat within 50ms"
+    - "Beat detection: librosa.beat.beat_track + frames_to_time, cached in app.state.beats"
+    - "Static /shows mount: FastAPI StaticFiles serves uploaded audio to browser"
+    - "bisect.bisect_left for O(log n) nearest-beat lookup per tick"
+    - "per-connection last_beat_reported prevents re-firing same beat"
+    - "Upload returns beat data inline — frontend caches without extra fetch"

 key-files:
  created:
-    - lightsync/audio/waveform.py
+    - lightsync/audio/beats.py
+  deleted:
+    - lightsync/audio/engine.py
  modified:
+    - pyproject.toml
+    - Dockerfile
+    - lightsync/main.py
    - lightsync/api/audio.py
+    - lightsync/api/ws.py
    - lightsync/frontend/index.html
    - lightsync/frontend/app.js
    - lightsync/frontend/style.css

 key-decisions:
-  - "ffmpeg subprocess pipe for MP3 (libsndfile/soundfile does not support MP3 natively)"
-  - "peaks param clamped 100-5000 server-side; client requests 2000 for waveform display"
-  - "CSS variables from existing palette used unchanged — --accent, --bg-primary, --bg-panel, --border-dim"
-  - "Playback cursor uses CSS transition: left 0.1s linear for smooth 10Hz tick interpolation"
+  - "Browser plays audio — eliminates python-mpv, Docker libmpv dependency, and server-side position polling"
+  - "librosa.beat.beat_track over onset_detect — gives both tempo and beat grid with one call"
+  - "Beat cache keyed by filepath in app.state.beats — survives WebSocket reconnects"
+  - "50ms BEAT_WINDOW_SEC tolerance — matches human perception of beat timing"
+  - "Upload response includes beat data inline — avoids second /api/audio/beats fetch after upload"
+  - "setInterval 10Hz + timeupdate fallback — belt-and-suspenders for position tick delivery"
+  - "last_beat_reported per-connection — prevents duplicate beat events during slow ticks"

-patterns-established:
-  - "Pattern: waveform extraction returns list[float] 0.0-1.0, normalized to peak amplitude per chunk"
-  - "Pattern: transport state driven by WebSocket tick — no polling from frontend"
-
-requirements-completed: [AUD-05]
-
-duration: 8min
+duration: ~4 min
 completed: 2026-04-06
 ---

-# Phase 02 Plan 03: Waveform Display and Transport UI Summary
+# Phase 02 Plan 03: Browser Audio + Librosa Beat Detection (Architecture Redesign)

-**soundfile/ffmpeg waveform extractor, GET /api/audio/waveform endpoint, and full transport bar with play/pause/seek controls and real-time position cursor on canvas**
+**Browser plays audio via HTML5 <audio> element; server does beat analysis via librosa; WebSocket carries position ticks (browser) and beat notifications (server)**

 ## Performance

- **Duration:** ~8 min
- **Started:** 2026-04-06T13:10:00Z
- **Completed:** 2026-04-06T13:18:00Z
- **Tasks:** 2 auto + 1 checkpoint (human-verify)
- **Files modified:** 5 (1 created)
+- **Duration:** ~4 min
+- **Started:** 2026-04-06T15:35:26Z
+- **Completed:** 2026-04-06T15:39:29Z
+- **Tasks:** 9 auto commits (architecture redesign, no checkpoints)
+- **Files modified:** 8 (1 created, 1 deleted)

 ## Accomplishments

- Waveform extraction module handles MP3 via ffmpeg pipe and WAV/FLAC/OGG via soundfile+numpy (AUD-05)
- GET /api/audio/waveform endpoint wraps extraction in asyncio.to_thread, returns {peaks, count, file}
- Transport bar replaces placeholder: play/pause/stop buttons, seek slider, time display (M:SS.s), audio path input + LOAD button
- Timeline panel waveform canvas draws peak bars using --accent color at device pixel ratio
- Position cursor moves at 10Hz from WebSocket tick dispatch, play/pause button visibility toggles with state
- Click-to-seek on waveform canvas, seek bar dragging, window resize redraws waveform
+- Removed python-mpv from pyproject.toml and mpv-libs from Dockerfile entirely
+- Created `lightsync/audio/beats.py` — `analyze(path)` using librosa.beat.beat_track, returns `{tempo, beats}` as timestamps in seconds
+- Redesigned `lightsync/api/audio.py` — POST /upload runs beat analysis and caches; GET /beats returns cached or on-demand; GET /waveform now takes path query param; removed engine-dependent endpoints
+- Redesigned `lightsync/api/ws.py` — removed server broadcast_loop; browser sends tick messages; server does 50ms beat proximity check and fires beat events back; bisect-based O(log n) lookup
+- Updated `lightsync/main.py` — removed MPVEngine lifespan; added `app.state.beats = {}` cache; added `/shows` StaticFiles mount for browser audio loading
+- Deleted `lightsync/audio/engine.py` — fully replaced by browser audio + librosa
+- Frontend `index.html` — added hidden `<audio id="player">`, beat indicator dot in header, animation selector panel
+- Frontend `app.js` — wires audio element events, 10Hz setInterval tick loop, beat flash on WS beat events, beat marker lines drawn on waveform canvas, loadSelectedFile sets audio.src
+- Frontend `style.css` — beat-indicator with beat-flash animation, animation-select-input styles

 ## Task Commits

-Each task was committed atomically:
-
-1. **Task 1: Waveform extraction module and HTTP endpoint** - `da474d9` (feat)
-2. **Task 2: Transport UI controls and waveform canvas** - `148f16b` (feat)
-3. **Task 3: Human verify** - awaiting checkpoint approval
+1. `5d93294` — chore: replace python-mpv with librosa in pyproject.toml
+2. `cce76a2` — chore: remove mpv-libs from Dockerfile
+3. `92cc09e` — feat: add beats.py librosa beat detection module
+4. `636160e` — feat: redesign audio.py — browser-audio architecture, beat endpoint
+5. `2c9ca0d` — feat: redesign WebSocket protocol — browser tick-driven beat detection
+6. `6366d84` — feat: remove MPVEngine from main.py, add beats cache and shows route
+7. `1f1ff92` — chore: delete engine.py
+8. `f1941a9` — feat: redesign frontend — browser audio element, beat markers, animation selector
+9. `0963ad4` — fix: upload file selection path consistency

 ## Files Created/Modified

- `lightsync/audio/waveform.py` — extract_peaks(), _extract_peaks_mp3(), _extract_peaks_soundfile(), _downsample_peaks()
- `lightsync/api/audio.py` — Added GET /waveform endpoint with asyncio.to_thread + peaks clamp
- `lightsync/frontend/index.html` — Timeline panel with waveform canvas + cursor; transport bar with all controls
- `lightsync/frontend/app.js` — drawWaveform(), loadWaveform(), formatTime(), updatePosition(), transport event handlers, handleMessage tick dispatch
- `lightsync/frontend/style.css` — .transport-btn, .transport-time, .seek-slider, .transport-file, .waveform-container, .playback-cursor
+- `lightsync/audio/beats.py` — analyze(), analyze_async(), librosa beat_track wrapper
+- `lightsync/audio/engine.py` — DELETED (MPV engine removed)
+- `lightsync/api/audio.py` — upload+beats cache, GET /beats endpoint, waveform takes path param
+- `lightsync/api/ws.py` — tick/beat protocol, bisect beat lookup, per-connection state
+- `lightsync/main.py` — no engine, app.state.beats, /shows StaticFiles
+- `lightsync/frontend/index.html` — audio element, beat indicator, animation selector
+- `lightsync/frontend/app.js` — audio wiring, 10Hz ticks, beat flash, beat markers
+- `lightsync/frontend/style.css` — beat-indicator, animation-select-input
+- `pyproject.toml` — python-mpv removed, librosa added
+- `Dockerfile` — mpv-libs removed

 ## Decisions Made

- Used ffmpeg subprocess pipe for MP3 extraction (soundfile/libsndfile doesn't support MP3 natively)
- peaks query param server-side clamped to 100-5000; frontend requests 2000 for display
- CSS used existing variable names unchanged (--accent, --bg-primary, --bg-panel, --border-dim) — no new color variables added
- CSS transition on playback-cursor (left 0.1s linear) interpolates between 10Hz ticks for smooth movement
+- Browser handles audio playback — no server-side audio process or Docker audio dependency
+- librosa.beat.beat_track provides both tempo and beat grid in one call, sufficient for Phase 2
+- Beat cache in app.state.beats survives WebSocket reconnects and multiple client connections
+- 50ms window balances precision vs. tick rate jitter (100ms tick interval)
+- Static /shows mount allows browser to load audio files with a simple URL path

 ## Deviations from Plan

+This plan was a complete architecture redesign replacing the original 02-03 implementation. The original 02-03 built on MPV; this revision removes MPV entirely.
+
+### Changes vs Original 02-03-PLAN.md
+
+**Removed from plan:**
+- MPV waveform-based transport UI (seek bar, time display driven by WS tick from server)
+- Server-side broadcast_loop polling MPV position
+
+**Added vs original plan (per user redesign spec):**
+- librosa beat detection module
+- Browser audio element as playback source
+- 10Hz client tick loop
+- Beat proximity detection in WebSocket server
+- Beat flash visual indicator
+- Beat marker lines on waveform canvas
+- Animation selector panel
+- /shows StaticFiles mount
+
 ### Auto-fixed Issues

-None — plan executed exactly as written, with CSS variable name adaptation (plan used --bg/--surface/--border, existing CSS uses --bg-primary/--bg-panel/--border-dim — adapted to match actual variable names).
-
---
-
-**Total deviations:** 0
-**Impact on plan:** CSS variable name adaptation was minor and necessary for correctness. No scope creep.
-
-## Issues Encountered
-
-None during auto-tasks. Human verification checkpoint (Task 3) is pending.
+**[Rule 1 - Bug] Fixed upload path pre-selection inconsistency**
+- **Found during:** Upload handler review
+- **Issue:** refreshFileList options used `/app/shows/filename` values but upload called it with `/shows/filename`, preventing auto-selection of uploaded file
+- **Fix:** Pass full abs path from upload response directly to refreshFileList
+- **Files modified:** lightsync/frontend/app.js
+- **Commit:** 0963ad4

 ## Known Stubs

-None — waveform extraction and transport UI are fully wired. The position cursor and transport controls depend on WebSocket tick messages which are broadcast by Phase 02-02's broadcast loop.
+- Animation selector (`<select id="animation-select">`) has placeholder options — no animation logic connected yet (Phase 3+ will wire UDP commands)
+- WebSocket play/pause/seek messages from browser are logged only — no UDP trigger yet (deferred to Phase 3+ cue scheduler)

 ## Next Phase Readiness

- GET /api/audio/waveform usable by any future phase needing waveform data
- Transport controls drive audio via WebSocket commands (play/pause/seek/load)
- Position cursor provides real-time visual feedback for timeline editor (Phase 3+)
- All Phase 1 UI (devices panel, device form) preserved
+- Beat timestamps available via app.state.beats for any Phase 3+ cue scheduler
+- /api/audio/beats endpoint provides analysis data for Phase 4 timeline editor beat-snap
+- WebSocket protocol extended cleanly — Phase 3 can add beat → UDP animation dispatch
+- Browser audio element is the master clock — Phase 5 live execution reads audio.currentTime

 ---
 *Phase: 02-audio-engine*