docs(02-03): complete architecture redesign — browser audio + librosa beat detection
- 02-03-SUMMARY.md: full redesign summary, decisions, deviations documented - STATE.md: decisions updated, session recorded, progress at 100% (7/7 plans) - ROADMAP.md: Phase 2 confirmed Complete (3/3 plans)
This commit is contained in:
Claude
@@ -3,8 +3,8 @@ gsd_state_version: 1.0
|
|||||||
milestone: v1.0
|
milestone: v1.0
|
||||||
milestone_name: milestone
|
milestone_name: milestone
|
||||||
status: verifying
|
status: verifying
|
||||||
stopped_at: "Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation"
|
stopped_at: "Completed 02-audio-engine/02-03-PLAN.md (architecture redesign: browser audio + librosa)"
|
||||||
last_updated: "2026-04-06T13:15:05.979Z"
|
last_updated: "2026-04-06T15:40:50.951Z"
|
||||||
last_activity: 2026-04-06
|
last_activity: 2026-04-06
|
||||||
progress:
|
progress:
|
||||||
total_phases: 7
|
total_phases: 7
|
||||||
@@ -58,6 +58,7 @@ Progress: [░░░░░░░░░░] 0%
|
|||||||
| Phase 02-audio-engine P01 | 525603 | 2 tasks | 5 files |
|
| Phase 02-audio-engine P01 | 525603 | 2 tasks | 5 files |
|
||||||
| Phase 02-audio-engine P02 | 8 | 2 tasks | 3 files |
|
| Phase 02-audio-engine P02 | 8 | 2 tasks | 3 files |
|
||||||
| Phase 02-audio-engine P03 | 8min | 2 tasks | 5 files |
|
| Phase 02-audio-engine P03 | 8min | 2 tasks | 5 files |
|
||||||
|
| Phase 02-audio-engine P03 | 4 | 9 tasks | 8 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -85,6 +86,8 @@ Recent decisions affecting current work:
|
|||||||
- [Phase 02-audio-engine]: AUD-02 (YouTube URL loading) deferred to Phase 6 — noted in docstring, not implemented in audio.py
|
- [Phase 02-audio-engine]: AUD-02 (YouTube URL loading) deferred to Phase 6 — noted in docstring, not implemented in audio.py
|
||||||
- [Phase 02-audio-engine]: ffmpeg subprocess pipe for MP3 waveform extraction (libsndfile/soundfile does not support MP3 natively)
|
- [Phase 02-audio-engine]: ffmpeg subprocess pipe for MP3 waveform extraction (libsndfile/soundfile does not support MP3 natively)
|
||||||
- [Phase 02-audio-engine]: Transport state driven by WebSocket tick — no frontend polling
|
- [Phase 02-audio-engine]: Transport state driven by WebSocket tick — no frontend polling
|
||||||
|
- [Phase 02-audio-engine]: Browser plays audio via HTML5 audio element — eliminates python-mpv, Docker libmpv dependency, and server-side position polling
|
||||||
|
- [Phase 02-audio-engine]: librosa.beat.beat_track for beat detection — tempo + beat grid in one call, cached in app.state.beats by filepath
|
||||||
|
|
||||||
### Pending Todos
|
### Pending Todos
|
||||||
|
|
||||||
@@ -98,8 +101,8 @@ None yet.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-04-06T13:15:05.973Z
|
Last session: 2026-04-06T15:40:34.296Z
|
||||||
Stopped at: Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation
|
Stopped at: Completed 02-audio-engine/02-03-PLAN.md (architecture redesign: browser audio + librosa)
|
||||||
Resume file: None
|
Resume file: None
|
||||||
|
|
||||||
Next action: `/gsd:plan-phase 1`
|
Next action: `/gsd:plan-phase 1`
|
||||||
|
|||||||
@@ -2,127 +2,169 @@
|
|||||||
phase: 02-audio-engine
|
phase: 02-audio-engine
|
||||||
plan: "03"
|
plan: "03"
|
||||||
subsystem: audio
|
subsystem: audio
|
||||||
|
tags: [librosa, beat-detection, browser-audio, websocket, waveform, canvas, animation-selector]
|
||||||
tags: [waveform, soundfile, numpy, ffmpeg, transport-ui, canvas, websocket]
|
|
||||||
|
|
||||||
requires:
|
requires:
|
||||||
- phase: 02-audio-engine/02-01
|
- phase: 02-audio-engine/02-01
|
||||||
provides: MPVEngine with get_state(), play/pause/seek via WebSocket
|
provides: MPVEngine (now deleted — replaced by browser audio)
|
||||||
- phase: 02-audio-engine/02-02
|
- phase: 02-audio-engine/02-02
|
||||||
provides: WebSocket broadcast loop (tick messages), REST audio endpoints in lightsync/api/audio.py
|
provides: WebSocket hub, REST audio endpoints
|
||||||
|
|
||||||
provides:
|
provides:
|
||||||
- extract_peaks() function in lightsync/audio/waveform.py (AUD-05)
|
- Browser <audio> element handles playback (HTML5, user's speakers)
|
||||||
- GET /api/audio/waveform endpoint returning downsampled peak array
|
- librosa beat detection via lightsync/audio/beats.py — analyze(path) -> {tempo, beats}
|
||||||
- Transport bar UI with play/pause/stop/seek/load controls
|
- Beat data cached in app.state.beats (filepath -> beat data dict)
|
||||||
- Waveform canvas in timeline panel with position cursor
|
- GET /api/audio/beats?path=... — returns cached or on-demand beat analysis
|
||||||
- Real-time position cursor via WebSocket tick dispatch in handleMessage
|
- POST /api/audio/upload — saves file, runs beat analysis, caches result
|
||||||
|
- GET /api/audio/waveform?path=... — waveform peaks (now takes path query param)
|
||||||
|
- WebSocket tick/beat protocol — browser sends position ticks, server sends beat events
|
||||||
|
- /shows/{filename} static file serving — browser <audio src=...> loads uploaded files
|
||||||
|
- Beat marker overlay on waveform canvas (yellow tick lines at beat timestamps)
|
||||||
|
- Beat flash indicator in header (yellow dot animation on beat events)
|
||||||
|
- Animation selector panel with 4 placeholder options
|
||||||
|
|
||||||
affects: [03-timeline-editor, 04-show-builder]
|
affects: [03-communication-protocol, 04-timeline-editor]
|
||||||
|
|
||||||
tech-stack:
|
tech-stack:
|
||||||
added: [soundfile, numpy (waveform extraction), ffmpeg (mp3 via subprocess pipe)]
|
added:
|
||||||
|
- librosa>=0.10.0 (beat detection, replaces python-mpv for audio analysis)
|
||||||
|
removed:
|
||||||
|
- python-mpv>=1.0.5 (server-side playback replaced by browser audio)
|
||||||
|
- mpv-libs Alpine package
|
||||||
|
- libmpv.so symlink in Dockerfile
|
||||||
patterns:
|
patterns:
|
||||||
- "MP3 via ffmpeg pipe: subprocess.run(['ffmpeg', '-f', 'f32le', 'pipe:1']) + np.frombuffer"
|
- "Browser sends {type: tick, position: N} at 10Hz; server finds nearest beat within 50ms"
|
||||||
- "asyncio.to_thread for waveform extraction (seconds-long operation)"
|
- "Beat detection: librosa.beat.beat_track + frames_to_time, cached in app.state.beats"
|
||||||
- "Transport button visibility toggle: btnPlay/btnPause display swap on tick paused state"
|
- "Static /shows mount: FastAPI StaticFiles serves uploaded audio to browser"
|
||||||
- "Seek bar _dragging flag prevents position jumps while user scrubs"
|
- "bisect.bisect_left for O(log n) nearest-beat lookup per tick"
|
||||||
- "Waveform click-to-seek: clientX/rect.left normalized to audioDuration"
|
- "per-connection last_beat_reported prevents re-firing same beat"
|
||||||
|
- "Upload returns beat data inline — frontend caches without extra fetch"
|
||||||
|
|
||||||
key-files:
|
key-files:
|
||||||
created:
|
created:
|
||||||
- lightsync/audio/waveform.py
|
- lightsync/audio/beats.py
|
||||||
|
deleted:
|
||||||
|
- lightsync/audio/engine.py
|
||||||
modified:
|
modified:
|
||||||
|
- pyproject.toml
|
||||||
|
- Dockerfile
|
||||||
|
- lightsync/main.py
|
||||||
- lightsync/api/audio.py
|
- lightsync/api/audio.py
|
||||||
|
- lightsync/api/ws.py
|
||||||
- lightsync/frontend/index.html
|
- lightsync/frontend/index.html
|
||||||
- lightsync/frontend/app.js
|
- lightsync/frontend/app.js
|
||||||
- lightsync/frontend/style.css
|
- lightsync/frontend/style.css
|
||||||
|
|
||||||
key-decisions:
|
key-decisions:
|
||||||
- "ffmpeg subprocess pipe for MP3 (libsndfile/soundfile does not support MP3 natively)"
|
- "Browser plays audio — eliminates python-mpv, Docker libmpv dependency, and server-side position polling"
|
||||||
- "peaks param clamped 100-5000 server-side; client requests 2000 for waveform display"
|
- "librosa.beat.beat_track over onset_detect — gives both tempo and beat grid with one call"
|
||||||
- "CSS variables from existing palette used unchanged — --accent, --bg-primary, --bg-panel, --border-dim"
|
- "Beat cache keyed by filepath in app.state.beats — survives WebSocket reconnects"
|
||||||
- "Playback cursor uses CSS transition: left 0.1s linear for smooth 10Hz tick interpolation"
|
- "50ms BEAT_WINDOW_SEC tolerance — matches human perception of beat timing"
|
||||||
|
- "Upload response includes beat data inline — avoids second /api/audio/beats fetch after upload"
|
||||||
|
- "setInterval 10Hz + timeupdate fallback — belt-and-suspenders for position tick delivery"
|
||||||
|
- "last_beat_reported per-connection — prevents duplicate beat events during slow ticks"
|
||||||
|
|
||||||
patterns-established:
|
duration: ~4 min
|
||||||
- "Pattern: waveform extraction returns list[float] 0.0-1.0, normalized to peak amplitude per chunk"
|
|
||||||
- "Pattern: transport state driven by WebSocket tick — no polling from frontend"
|
|
||||||
|
|
||||||
requirements-completed: [AUD-05]
|
|
||||||
|
|
||||||
duration: 8min
|
|
||||||
completed: 2026-04-06
|
completed: 2026-04-06
|
||||||
---
|
---
|
||||||
|
|
||||||
# Phase 02 Plan 03: Waveform Display and Transport UI Summary
|
# Phase 02 Plan 03: Browser Audio + Librosa Beat Detection (Architecture Redesign)
|
||||||
|
|
||||||
**soundfile/ffmpeg waveform extractor, GET /api/audio/waveform endpoint, and full transport bar with play/pause/seek controls and real-time position cursor on canvas**
|
**Browser plays audio via HTML5 <audio> element; server does beat analysis via librosa; WebSocket carries position ticks (browser) and beat notifications (server)**
|
||||||
|
|
||||||
## Performance
|
## Performance
|
||||||
|
|
||||||
- **Duration:** ~8 min
|
- **Duration:** ~4 min
|
||||||
- **Started:** 2026-04-06T13:10:00Z
|
- **Started:** 2026-04-06T15:35:26Z
|
||||||
- **Completed:** 2026-04-06T13:18:00Z
|
- **Completed:** 2026-04-06T15:39:29Z
|
||||||
- **Tasks:** 2 auto + 1 checkpoint (human-verify)
|
- **Tasks:** 9 auto commits (architecture redesign, no checkpoints)
|
||||||
- **Files modified:** 5 (1 created)
|
- **Files modified:** 8 (1 created, 1 deleted)
|
||||||
|
|
||||||
## Accomplishments
|
## Accomplishments
|
||||||
|
|
||||||
- Waveform extraction module handles MP3 via ffmpeg pipe and WAV/FLAC/OGG via soundfile+numpy (AUD-05)
|
- Removed python-mpv from pyproject.toml and mpv-libs from Dockerfile entirely
|
||||||
- GET /api/audio/waveform endpoint wraps extraction in asyncio.to_thread, returns {peaks, count, file}
|
- Created `lightsync/audio/beats.py` — `analyze(path)` using librosa.beat.beat_track, returns `{tempo, beats}` as timestamps in seconds
|
||||||
- Transport bar replaces placeholder: play/pause/stop buttons, seek slider, time display (M:SS.s), audio path input + LOAD button
|
- Redesigned `lightsync/api/audio.py` — POST /upload runs beat analysis and caches; GET /beats returns cached or on-demand; GET /waveform now takes path query param; removed engine-dependent endpoints
|
||||||
- Timeline panel waveform canvas draws peak bars using --accent color at device pixel ratio
|
- Redesigned `lightsync/api/ws.py` — removed server broadcast_loop; browser sends tick messages; server does 50ms beat proximity check and fires beat events back; bisect-based O(log n) lookup
|
||||||
- Position cursor moves at 10Hz from WebSocket tick dispatch, play/pause button visibility toggles with state
|
- Updated `lightsync/main.py` — removed MPVEngine lifespan; added `app.state.beats = {}` cache; added `/shows` StaticFiles mount for browser audio loading
|
||||||
- Click-to-seek on waveform canvas, seek bar dragging, window resize redraws waveform
|
- Deleted `lightsync/audio/engine.py` — fully replaced by browser audio + librosa
|
||||||
|
- Frontend `index.html` — added hidden `<audio id="player">`, beat indicator dot in header, animation selector panel
|
||||||
|
- Frontend `app.js` — wires audio element events, 10Hz setInterval tick loop, beat flash on WS beat events, beat marker lines drawn on waveform canvas, loadSelectedFile sets audio.src
|
||||||
|
- Frontend `style.css` — beat-indicator with beat-flash animation, animation-select-input styles
|
||||||
|
|
||||||
## Task Commits
|
## Task Commits
|
||||||
|
|
||||||
Each task was committed atomically:
|
1. `5d93294` — chore: replace python-mpv with librosa in pyproject.toml
|
||||||
|
2. `cce76a2` — chore: remove mpv-libs from Dockerfile
|
||||||
1. **Task 1: Waveform extraction module and HTTP endpoint** - `da474d9` (feat)
|
3. `92cc09e` — feat: add beats.py librosa beat detection module
|
||||||
2. **Task 2: Transport UI controls and waveform canvas** - `148f16b` (feat)
|
4. `636160e` — feat: redesign audio.py — browser-audio architecture, beat endpoint
|
||||||
3. **Task 3: Human verify** - awaiting checkpoint approval
|
5. `2c9ca0d` — feat: redesign WebSocket protocol — browser tick-driven beat detection
|
||||||
|
6. `6366d84` — feat: remove MPVEngine from main.py, add beats cache and shows route
|
||||||
|
7. `1f1ff92` — chore: delete engine.py
|
||||||
|
8. `f1941a9` — feat: redesign frontend — browser audio element, beat markers, animation selector
|
||||||
|
9. `0963ad4` — fix: upload file selection path consistency
|
||||||
|
|
||||||
## Files Created/Modified
|
## Files Created/Modified
|
||||||
|
|
||||||
- `lightsync/audio/waveform.py` — extract_peaks(), _extract_peaks_mp3(), _extract_peaks_soundfile(), _downsample_peaks()
|
- `lightsync/audio/beats.py` — analyze(), analyze_async(), librosa beat_track wrapper
|
||||||
- `lightsync/api/audio.py` — Added GET /waveform endpoint with asyncio.to_thread + peaks clamp
|
- `lightsync/audio/engine.py` — DELETED (MPV engine removed)
|
||||||
- `lightsync/frontend/index.html` — Timeline panel with waveform canvas + cursor; transport bar with all controls
|
- `lightsync/api/audio.py` — upload+beats cache, GET /beats endpoint, waveform takes path param
|
||||||
- `lightsync/frontend/app.js` — drawWaveform(), loadWaveform(), formatTime(), updatePosition(), transport event handlers, handleMessage tick dispatch
|
- `lightsync/api/ws.py` — tick/beat protocol, bisect beat lookup, per-connection state
|
||||||
- `lightsync/frontend/style.css` — .transport-btn, .transport-time, .seek-slider, .transport-file, .waveform-container, .playback-cursor
|
- `lightsync/main.py` — no engine, app.state.beats, /shows StaticFiles
|
||||||
|
- `lightsync/frontend/index.html` — audio element, beat indicator, animation selector
|
||||||
|
- `lightsync/frontend/app.js` — audio wiring, 10Hz ticks, beat flash, beat markers
|
||||||
|
- `lightsync/frontend/style.css` — beat-indicator, animation-select-input
|
||||||
|
- `pyproject.toml` — python-mpv removed, librosa added
|
||||||
|
- `Dockerfile` — mpv-libs removed
|
||||||
|
|
||||||
## Decisions Made
|
## Decisions Made
|
||||||
|
|
||||||
- Used ffmpeg subprocess pipe for MP3 extraction (soundfile/libsndfile doesn't support MP3 natively)
|
- Browser handles audio playback — no server-side audio process or Docker audio dependency
|
||||||
- peaks query param server-side clamped to 100-5000; frontend requests 2000 for display
|
- librosa.beat.beat_track provides both tempo and beat grid in one call, sufficient for Phase 2
|
||||||
- CSS used existing variable names unchanged (--accent, --bg-primary, --bg-panel, --border-dim) — no new color variables added
|
- Beat cache in app.state.beats survives WebSocket reconnects and multiple client connections
|
||||||
- CSS transition on playback-cursor (left 0.1s linear) interpolates between 10Hz ticks for smooth movement
|
- 50ms window balances precision vs. tick rate jitter (100ms tick interval)
|
||||||
|
- Static /shows mount allows browser to load audio files with a simple URL path
|
||||||
|
|
||||||
## Deviations from Plan
|
## Deviations from Plan
|
||||||
|
|
||||||
|
This plan was a complete architecture redesign replacing the original 02-03 implementation. The original 02-03 built on MPV; this revision removes MPV entirely.
|
||||||
|
|
||||||
|
### Changes vs Original 02-03-PLAN.md
|
||||||
|
|
||||||
|
**Removed from plan:**
|
||||||
|
- MPV waveform-based transport UI (seek bar, time display driven by WS tick from server)
|
||||||
|
- Server-side broadcast_loop polling MPV position
|
||||||
|
|
||||||
|
**Added vs original plan (per user redesign spec):**
|
||||||
|
- librosa beat detection module
|
||||||
|
- Browser audio element as playback source
|
||||||
|
- 10Hz client tick loop
|
||||||
|
- Beat proximity detection in WebSocket server
|
||||||
|
- Beat flash visual indicator
|
||||||
|
- Beat marker lines on waveform canvas
|
||||||
|
- Animation selector panel
|
||||||
|
- /shows StaticFiles mount
|
||||||
|
|
||||||
### Auto-fixed Issues
|
### Auto-fixed Issues
|
||||||
|
|
||||||
None — plan executed exactly as written, with CSS variable name adaptation (plan used --bg/--surface/--border, existing CSS uses --bg-primary/--bg-panel/--border-dim — adapted to match actual variable names).
|
**[Rule 1 - Bug] Fixed upload path pre-selection inconsistency**
|
||||||
|
- **Found during:** Upload handler review
|
||||||
---
|
- **Issue:** refreshFileList options used `/app/shows/filename` values but upload called it with `/shows/filename`, preventing auto-selection of uploaded file
|
||||||
|
- **Fix:** Pass full abs path from upload response directly to refreshFileList
|
||||||
**Total deviations:** 0
|
- **Files modified:** lightsync/frontend/app.js
|
||||||
**Impact on plan:** CSS variable name adaptation was minor and necessary for correctness. No scope creep.
|
- **Commit:** 0963ad4
|
||||||
|
|
||||||
## Issues Encountered
|
|
||||||
|
|
||||||
None during auto-tasks. Human verification checkpoint (Task 3) is pending.
|
|
||||||
|
|
||||||
## Known Stubs
|
## Known Stubs
|
||||||
|
|
||||||
None — waveform extraction and transport UI are fully wired. The position cursor and transport controls depend on WebSocket tick messages which are broadcast by Phase 02-02's broadcast loop.
|
- Animation selector (`<select id="animation-select">`) has placeholder options — no animation logic connected yet (Phase 3+ will wire UDP commands)
|
||||||
|
- WebSocket play/pause/seek messages from browser are logged only — no UDP trigger yet (deferred to Phase 3+ cue scheduler)
|
||||||
|
|
||||||
## Next Phase Readiness
|
## Next Phase Readiness
|
||||||
|
|
||||||
- GET /api/audio/waveform usable by any future phase needing waveform data
|
- Beat timestamps available via app.state.beats for any Phase 3+ cue scheduler
|
||||||
- Transport controls drive audio via WebSocket commands (play/pause/seek/load)
|
- /api/audio/beats endpoint provides analysis data for Phase 4 timeline editor beat-snap
|
||||||
- Position cursor provides real-time visual feedback for timeline editor (Phase 3+)
|
- WebSocket protocol extended cleanly — Phase 3 can add beat → UDP animation dispatch
|
||||||
- All Phase 1 UI (devices panel, device form) preserved
|
- Browser audio element is the master clock — Phase 5 live execution reads audio.currentTime
|
||||||
|
|
||||||
---
|
---
|
||||||
*Phase: 02-audio-engine*
|
*Phase: 02-audio-engine*
|
||||||
|
|||||||
Reference in New Issue
Block a user