docs(02-03): complete waveform display and transport UI plan — AUD-05 fulfilled

- SUMMARY.md for 02-03 (waveform extractor + transport UI) - STATE.md: checkpoint at human-verify task 3, decisions recorded - ROADMAP.md: phase 2 progress updated (3/3 plans) - REQUIREMENTS.md: AUD-05 marked complete
2026-04-06 13:15:17 +00:00
parent 148f16ba5e
commit 7f9efee47a
4 changed files with 145 additions and 13 deletions
--- a/.planning/REQUIREMENTS.md
+++ b/.planning/REQUIREMENTS.md
@@ -13,7 +13,7 @@
 - [x] **AUD-02**: Load audio from a YouTube URL via yt-dlp + MPV
 - [x] **AUD-03**: Playback controls: play, pause, seek to position
 - [x] **AUD-04**: Realtime position reporting from MPV to UI (via named pipe IPC, ≤10ms poll interval)
- [ ] **AUD-05**: Waveform display rendered from audio samples in the timeline UI
+- [x] **AUD-05**: Waveform display rendered from audio samples in the timeline UI

 ### Device Registry

@@ -113,7 +113,7 @@
 | SHW-01, SHW-02 | Phase 1 | Pending |
 | DEV-01, DEV-02, DEV-03 | Phase 1 | Pending |
 | AUD-01, AUD-02, AUD-03, AUD-04 | Phase 2 | Pending |
-| AUD-05 | Phase 2 | Pending |
+| AUD-05 | Phase 2 | Complete |
 | INF-03 | Phase 2 | Complete |
 | UDP-01, UDP-02, UDP-03, UDP-04 | Phase 3 | Pending |
 | ANI-01, ANI-02, ANI-03, ANI-04 | Phase 3 | Pending |
--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@@ -13,7 +13,7 @@ LightSync is built bottom-up: data model first, audio engine second, UDP protoco
 Decimal phases appear between their surrounding integers in numeric order.

 - [x] **Phase 1: Foundation** - FastAPI skeleton, show file schema (Pydantic), device registry, terminal-aesthetic frontend shell (gap closure in progress) (completed 2026-04-05)
- [ ] **Phase 2: Audio Engine** - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix
+- [x] **Phase 2: Audio Engine** - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix (completed 2026-04-06)
 - [ ] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator
 - [ ] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat
 - [ ] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview
@@ -56,7 +56,7 @@ Plans:
 Plans:
 - [x] 02-01: MPV IPC bridge — python-mpv subprocess, named pipe reader thread, shared position variable, Windows timer fix
 - [x] 02-02: Show engine skeleton + WebSocket hub — playback state machine, position broadcast at 10Hz, transport command dispatcher
- [ ] 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples
+- [x] 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples
 **UI hint**: yes

 ### Phase 3: Communication Protocol
@@ -153,7 +153,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7
 | Phase | Plans Complete | Status | Completed |
 |-------|----------------|--------|-----------|
 | 1. Foundation | 4/4 | Complete   | 2026-04-05 |
-| 2. Audio Engine | 2/3 | In Progress|  |
+| 2. Audio Engine | 3/3 | Complete   | 2026-04-06 |
 | 3. Communication Protocol | 0/3 | Not started | - |
 | 4. Timeline Editor | 0/4 | Not started | - |
 | 5. Live Show Execution | 0/3 | Not started | - |
--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@@ -2,15 +2,15 @@
 gsd_state_version: 1.0
 milestone: v1.0
 milestone_name: milestone
-status: executing
-stopped_at: Completed 02-audio-engine/02-02-PLAN.md
-last_updated: "2026-04-06T13:08:39.827Z"
+status: verifying
+stopped_at: "Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation"
+last_updated: "2026-04-06T13:15:05.979Z"
 last_activity: 2026-04-06
 progress:
  total_phases: 7
-  completed_phases: 1
+  completed_phases: 2
  total_plans: 7
-  completed_plans: 6
+  completed_plans: 7
  percent: 0
 ---

@@ -27,7 +27,7 @@ See: .planning/PROJECT.md (updated 2026-04-05)

 Phase: 02 (audio-engine) — EXECUTING
 Plan: 3 of 3
-Status: Ready to execute
+Status: Phase complete — ready for verification
 Last activity: 2026-04-06

 Progress: [░░░░░░░░░░] 0%
@@ -57,6 +57,7 @@ Progress: [░░░░░░░░░░] 0%
 | Phase 01 P04 | 2 | 2 tasks | 3 files |
 | Phase 02-audio-engine P01 | 525603 | 2 tasks | 5 files |
 | Phase 02-audio-engine P02 | 8 | 2 tasks | 3 files |
+| Phase 02-audio-engine P03 | 8min | 2 tasks | 5 files |

 ## Accumulated Context

@@ -82,6 +83,8 @@ Recent decisions affecting current work:
 - [Phase 02-audio-engine]: broadcast_loop accepts get_engine callable instead of direct import — avoids ws.py/main.py circular import
 - [Phase 02-audio-engine]: asyncio.to_thread for engine.load in REST endpoint — python-mpv-jsonipc load is synchronous, must not block event loop
 - [Phase 02-audio-engine]: AUD-02 (YouTube URL loading) deferred to Phase 6 — noted in docstring, not implemented in audio.py
+- [Phase 02-audio-engine]: ffmpeg subprocess pipe for MP3 waveform extraction (libsndfile/soundfile does not support MP3 natively)
+- [Phase 02-audio-engine]: Transport state driven by WebSocket tick — no frontend polling

 ### Pending Todos

@@ -95,8 +98,8 @@ None yet.

 ## Session Continuity

-Last session: 2026-04-06T13:08:39.818Z
-Stopped at: Completed 02-audio-engine/02-02-PLAN.md
+Last session: 2026-04-06T13:15:05.973Z
+Stopped at: Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation
 Resume file: None

 Next action: `/gsd:plan-phase 1`
--- a/.planning/phases/02-audio-engine/02-03-SUMMARY.md
+++ b/.planning/phases/02-audio-engine/02-03-SUMMARY.md
@@ -0,0 +1,129 @@
+---
+phase: 02-audio-engine
+plan: "03"
+subsystem: audio
+
+tags: [waveform, soundfile, numpy, ffmpeg, transport-ui, canvas, websocket]
+
+requires:
+  - phase: 02-audio-engine/02-01
+    provides: MPVEngine with get_state(), play/pause/seek via WebSocket
+  - phase: 02-audio-engine/02-02
+    provides: WebSocket broadcast loop (tick messages), REST audio endpoints in lightsync/api/audio.py
+
+provides:
+  - extract_peaks() function in lightsync/audio/waveform.py (AUD-05)
+  - GET /api/audio/waveform endpoint returning downsampled peak array
+  - Transport bar UI with play/pause/stop/seek/load controls
+  - Waveform canvas in timeline panel with position cursor
+  - Real-time position cursor via WebSocket tick dispatch in handleMessage
+
+affects: [03-timeline-editor, 04-show-builder]
+
+tech-stack:
+  added: [soundfile, numpy (waveform extraction), ffmpeg (mp3 via subprocess pipe)]
+  patterns:
+    - "MP3 via ffmpeg pipe: subprocess.run(['ffmpeg', '-f', 'f32le', 'pipe:1']) + np.frombuffer"
+    - "asyncio.to_thread for waveform extraction (seconds-long operation)"
+    - "Transport button visibility toggle: btnPlay/btnPause display swap on tick paused state"
+    - "Seek bar _dragging flag prevents position jumps while user scrubs"
+    - "Waveform click-to-seek: clientX/rect.left normalized to audioDuration"
+
+key-files:
+  created:
+    - lightsync/audio/waveform.py
+  modified:
+    - lightsync/api/audio.py
+    - lightsync/frontend/index.html
+    - lightsync/frontend/app.js
+    - lightsync/frontend/style.css
+
+key-decisions:
+  - "ffmpeg subprocess pipe for MP3 (libsndfile/soundfile does not support MP3 natively)"
+  - "peaks param clamped 100-5000 server-side; client requests 2000 for waveform display"
+  - "CSS variables from existing palette used unchanged — --accent, --bg-primary, --bg-panel, --border-dim"
+  - "Playback cursor uses CSS transition: left 0.1s linear for smooth 10Hz tick interpolation"
+
+patterns-established:
+  - "Pattern: waveform extraction returns list[float] 0.0-1.0, normalized to peak amplitude per chunk"
+  - "Pattern: transport state driven by WebSocket tick — no polling from frontend"
+
+requirements-completed: [AUD-05]
+
+duration: 8min
+completed: 2026-04-06
+---
+
+# Phase 02 Plan 03: Waveform Display and Transport UI Summary
+
+**soundfile/ffmpeg waveform extractor, GET /api/audio/waveform endpoint, and full transport bar with play/pause/seek controls and real-time position cursor on canvas**
+
+## Performance
+
+- **Duration:** ~8 min
+- **Started:** 2026-04-06T13:10:00Z
+- **Completed:** 2026-04-06T13:18:00Z
+- **Tasks:** 2 auto + 1 checkpoint (human-verify)
+- **Files modified:** 5 (1 created)
+
+## Accomplishments
+
+- Waveform extraction module handles MP3 via ffmpeg pipe and WAV/FLAC/OGG via soundfile+numpy (AUD-05)
+- GET /api/audio/waveform endpoint wraps extraction in asyncio.to_thread, returns {peaks, count, file}
+- Transport bar replaces placeholder: play/pause/stop buttons, seek slider, time display (M:SS.s), audio path input + LOAD button
+- Timeline panel waveform canvas draws peak bars using --accent color at device pixel ratio
+- Position cursor moves at 10Hz from WebSocket tick dispatch, play/pause button visibility toggles with state
+- Click-to-seek on waveform canvas, seek bar dragging, window resize redraws waveform
+
+## Task Commits
+
+Each task was committed atomically:
+
+1. **Task 1: Waveform extraction module and HTTP endpoint** - `da474d9` (feat)
+2. **Task 2: Transport UI controls and waveform canvas** - `148f16b` (feat)
+3. **Task 3: Human verify** - awaiting checkpoint approval
+
+## Files Created/Modified
+
+- `lightsync/audio/waveform.py` — extract_peaks(), _extract_peaks_mp3(), _extract_peaks_soundfile(), _downsample_peaks()
+- `lightsync/api/audio.py` — Added GET /waveform endpoint with asyncio.to_thread + peaks clamp
+- `lightsync/frontend/index.html` — Timeline panel with waveform canvas + cursor; transport bar with all controls
+- `lightsync/frontend/app.js` — drawWaveform(), loadWaveform(), formatTime(), updatePosition(), transport event handlers, handleMessage tick dispatch
+- `lightsync/frontend/style.css` — .transport-btn, .transport-time, .seek-slider, .transport-file, .waveform-container, .playback-cursor
+
+## Decisions Made
+
+- Used ffmpeg subprocess pipe for MP3 extraction (soundfile/libsndfile doesn't support MP3 natively)
+- peaks query param server-side clamped to 100-5000; frontend requests 2000 for display
+- CSS used existing variable names unchanged (--accent, --bg-primary, --bg-panel, --border-dim) — no new color variables added
+- CSS transition on playback-cursor (left 0.1s linear) interpolates between 10Hz ticks for smooth movement
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+None — plan executed exactly as written, with CSS variable name adaptation (plan used --bg/--surface/--border, existing CSS uses --bg-primary/--bg-panel/--border-dim — adapted to match actual variable names).
+
+---
+
+**Total deviations:** 0
+**Impact on plan:** CSS variable name adaptation was minor and necessary for correctness. No scope creep.
+
+## Issues Encountered
+
+None during auto-tasks. Human verification checkpoint (Task 3) is pending.
+
+## Known Stubs
+
+None — waveform extraction and transport UI are fully wired. The position cursor and transport controls depend on WebSocket tick messages which are broadcast by Phase 02-02's broadcast loop.
+
+## Next Phase Readiness
+
+- GET /api/audio/waveform usable by any future phase needing waveform data
+- Transport controls drive audio via WebSocket commands (play/pause/seek/load)
+- Position cursor provides real-time visual feedback for timeline editor (Phase 3+)
+- All Phase 1 UI (devices panel, device form) preserved
+
+---
+*Phase: 02-audio-engine*
+*Completed: 2026-04-06*