diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 84b1292..b550a9c 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -13,7 +13,7 @@ - [x] **AUD-02**: Load audio from a YouTube URL via yt-dlp + MPV - [x] **AUD-03**: Playback controls: play, pause, seek to position - [x] **AUD-04**: Realtime position reporting from MPV to UI (via named pipe IPC, ≤10ms poll interval) -- [ ] **AUD-05**: Waveform display rendered from audio samples in the timeline UI +- [x] **AUD-05**: Waveform display rendered from audio samples in the timeline UI ### Device Registry @@ -113,7 +113,7 @@ | SHW-01, SHW-02 | Phase 1 | Pending | | DEV-01, DEV-02, DEV-03 | Phase 1 | Pending | | AUD-01, AUD-02, AUD-03, AUD-04 | Phase 2 | Pending | -| AUD-05 | Phase 2 | Pending | +| AUD-05 | Phase 2 | Complete | | INF-03 | Phase 2 | Complete | | UDP-01, UDP-02, UDP-03, UDP-04 | Phase 3 | Pending | | ANI-01, ANI-02, ANI-03, ANI-04 | Phase 3 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index c94e7ef..6cebe92 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -13,7 +13,7 @@ LightSync is built bottom-up: data model first, audio engine second, UDP protoco Decimal phases appear between their surrounding integers in numeric order. - [x] **Phase 1: Foundation** - FastAPI skeleton, show file schema (Pydantic), device registry, terminal-aesthetic frontend shell (gap closure in progress) (completed 2026-04-05) -- [ ] **Phase 2: Audio Engine** - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix +- [x] **Phase 2: Audio Engine** - MPV IPC bridge, waveform rendering, playback controls, Windows timer fix (completed 2026-04-06) - [ ] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator - [ ] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat - [ ] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview @@ -56,7 +56,7 @@ Plans: Plans: - [x] 02-01: MPV IPC bridge — python-mpv subprocess, named pipe reader thread, shared position variable, Windows timer fix - [x] 02-02: Show engine skeleton + WebSocket hub — playback state machine, position broadcast at 10Hz, transport command dispatcher -- [ ] 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples +- [x] 02-03: Transport UI + waveform — play/pause/seek controls connected to WebSocket, waveform rendering from audio samples **UI hint**: yes ### Phase 3: Communication Protocol @@ -153,7 +153,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 | Phase | Plans Complete | Status | Completed | |-------|----------------|--------|-----------| | 1. Foundation | 4/4 | Complete | 2026-04-05 | -| 2. Audio Engine | 2/3 | In Progress| | +| 2. Audio Engine | 3/3 | Complete | 2026-04-06 | | 3. Communication Protocol | 0/3 | Not started | - | | 4. Timeline Editor | 0/4 | Not started | - | | 5. Live Show Execution | 0/3 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index c6e1def..14bee12 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,15 +2,15 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone -status: executing -stopped_at: Completed 02-audio-engine/02-02-PLAN.md -last_updated: "2026-04-06T13:08:39.827Z" +status: verifying +stopped_at: "Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation" +last_updated: "2026-04-06T13:15:05.979Z" last_activity: 2026-04-06 progress: total_phases: 7 - completed_phases: 1 + completed_phases: 2 total_plans: 7 - completed_plans: 6 + completed_plans: 7 percent: 0 --- @@ -27,7 +27,7 @@ See: .planning/PROJECT.md (updated 2026-04-05) Phase: 02 (audio-engine) — EXECUTING Plan: 3 of 3 -Status: Ready to execute +Status: Phase complete — ready for verification Last activity: 2026-04-06 Progress: [░░░░░░░░░░] 0% @@ -57,6 +57,7 @@ Progress: [░░░░░░░░░░] 0% | Phase 01 P04 | 2 | 2 tasks | 3 files | | Phase 02-audio-engine P01 | 525603 | 2 tasks | 5 files | | Phase 02-audio-engine P02 | 8 | 2 tasks | 3 files | +| Phase 02-audio-engine P03 | 8min | 2 tasks | 5 files | ## Accumulated Context @@ -82,6 +83,8 @@ Recent decisions affecting current work: - [Phase 02-audio-engine]: broadcast_loop accepts get_engine callable instead of direct import — avoids ws.py/main.py circular import - [Phase 02-audio-engine]: asyncio.to_thread for engine.load in REST endpoint — python-mpv-jsonipc load is synchronous, must not block event loop - [Phase 02-audio-engine]: AUD-02 (YouTube URL loading) deferred to Phase 6 — noted in docstring, not implemented in audio.py +- [Phase 02-audio-engine]: ffmpeg subprocess pipe for MP3 waveform extraction (libsndfile/soundfile does not support MP3 natively) +- [Phase 02-audio-engine]: Transport state driven by WebSocket tick — no frontend polling ### Pending Todos @@ -95,8 +98,8 @@ None yet. ## Session Continuity -Last session: 2026-04-06T13:08:39.818Z -Stopped at: Completed 02-audio-engine/02-02-PLAN.md +Last session: 2026-04-06T13:15:05.973Z +Stopped at: Checkpoint: 02-03 Task 3 human verification — awaiting user confirmation Resume file: None Next action: `/gsd:plan-phase 1` diff --git a/.planning/phases/02-audio-engine/02-03-SUMMARY.md b/.planning/phases/02-audio-engine/02-03-SUMMARY.md new file mode 100644 index 0000000..d93d94d --- /dev/null +++ b/.planning/phases/02-audio-engine/02-03-SUMMARY.md @@ -0,0 +1,129 @@ +--- +phase: 02-audio-engine +plan: "03" +subsystem: audio + +tags: [waveform, soundfile, numpy, ffmpeg, transport-ui, canvas, websocket] + +requires: + - phase: 02-audio-engine/02-01 + provides: MPVEngine with get_state(), play/pause/seek via WebSocket + - phase: 02-audio-engine/02-02 + provides: WebSocket broadcast loop (tick messages), REST audio endpoints in lightsync/api/audio.py + +provides: + - extract_peaks() function in lightsync/audio/waveform.py (AUD-05) + - GET /api/audio/waveform endpoint returning downsampled peak array + - Transport bar UI with play/pause/stop/seek/load controls + - Waveform canvas in timeline panel with position cursor + - Real-time position cursor via WebSocket tick dispatch in handleMessage + +affects: [03-timeline-editor, 04-show-builder] + +tech-stack: + added: [soundfile, numpy (waveform extraction), ffmpeg (mp3 via subprocess pipe)] + patterns: + - "MP3 via ffmpeg pipe: subprocess.run(['ffmpeg', '-f', 'f32le', 'pipe:1']) + np.frombuffer" + - "asyncio.to_thread for waveform extraction (seconds-long operation)" + - "Transport button visibility toggle: btnPlay/btnPause display swap on tick paused state" + - "Seek bar _dragging flag prevents position jumps while user scrubs" + - "Waveform click-to-seek: clientX/rect.left normalized to audioDuration" + +key-files: + created: + - lightsync/audio/waveform.py + modified: + - lightsync/api/audio.py + - lightsync/frontend/index.html + - lightsync/frontend/app.js + - lightsync/frontend/style.css + +key-decisions: + - "ffmpeg subprocess pipe for MP3 (libsndfile/soundfile does not support MP3 natively)" + - "peaks param clamped 100-5000 server-side; client requests 2000 for waveform display" + - "CSS variables from existing palette used unchanged — --accent, --bg-primary, --bg-panel, --border-dim" + - "Playback cursor uses CSS transition: left 0.1s linear for smooth 10Hz tick interpolation" + +patterns-established: + - "Pattern: waveform extraction returns list[float] 0.0-1.0, normalized to peak amplitude per chunk" + - "Pattern: transport state driven by WebSocket tick — no polling from frontend" + +requirements-completed: [AUD-05] + +duration: 8min +completed: 2026-04-06 +--- + +# Phase 02 Plan 03: Waveform Display and Transport UI Summary + +**soundfile/ffmpeg waveform extractor, GET /api/audio/waveform endpoint, and full transport bar with play/pause/seek controls and real-time position cursor on canvas** + +## Performance + +- **Duration:** ~8 min +- **Started:** 2026-04-06T13:10:00Z +- **Completed:** 2026-04-06T13:18:00Z +- **Tasks:** 2 auto + 1 checkpoint (human-verify) +- **Files modified:** 5 (1 created) + +## Accomplishments + +- Waveform extraction module handles MP3 via ffmpeg pipe and WAV/FLAC/OGG via soundfile+numpy (AUD-05) +- GET /api/audio/waveform endpoint wraps extraction in asyncio.to_thread, returns {peaks, count, file} +- Transport bar replaces placeholder: play/pause/stop buttons, seek slider, time display (M:SS.s), audio path input + LOAD button +- Timeline panel waveform canvas draws peak bars using --accent color at device pixel ratio +- Position cursor moves at 10Hz from WebSocket tick dispatch, play/pause button visibility toggles with state +- Click-to-seek on waveform canvas, seek bar dragging, window resize redraws waveform + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Waveform extraction module and HTTP endpoint** - `da474d9` (feat) +2. **Task 2: Transport UI controls and waveform canvas** - `148f16b` (feat) +3. **Task 3: Human verify** - awaiting checkpoint approval + +## Files Created/Modified + +- `lightsync/audio/waveform.py` — extract_peaks(), _extract_peaks_mp3(), _extract_peaks_soundfile(), _downsample_peaks() +- `lightsync/api/audio.py` — Added GET /waveform endpoint with asyncio.to_thread + peaks clamp +- `lightsync/frontend/index.html` — Timeline panel with waveform canvas + cursor; transport bar with all controls +- `lightsync/frontend/app.js` — drawWaveform(), loadWaveform(), formatTime(), updatePosition(), transport event handlers, handleMessage tick dispatch +- `lightsync/frontend/style.css` — .transport-btn, .transport-time, .seek-slider, .transport-file, .waveform-container, .playback-cursor + +## Decisions Made + +- Used ffmpeg subprocess pipe for MP3 extraction (soundfile/libsndfile doesn't support MP3 natively) +- peaks query param server-side clamped to 100-5000; frontend requests 2000 for display +- CSS used existing variable names unchanged (--accent, --bg-primary, --bg-panel, --border-dim) — no new color variables added +- CSS transition on playback-cursor (left 0.1s linear) interpolates between 10Hz ticks for smooth movement + +## Deviations from Plan + +### Auto-fixed Issues + +None — plan executed exactly as written, with CSS variable name adaptation (plan used --bg/--surface/--border, existing CSS uses --bg-primary/--bg-panel/--border-dim — adapted to match actual variable names). + +--- + +**Total deviations:** 0 +**Impact on plan:** CSS variable name adaptation was minor and necessary for correctness. No scope creep. + +## Issues Encountered + +None during auto-tasks. Human verification checkpoint (Task 3) is pending. + +## Known Stubs + +None — waveform extraction and transport UI are fully wired. The position cursor and transport controls depend on WebSocket tick messages which are broadcast by Phase 02-02's broadcast loop. + +## Next Phase Readiness + +- GET /api/audio/waveform usable by any future phase needing waveform data +- Transport controls drive audio via WebSocket commands (play/pause/seek/load) +- Position cursor provides real-time visual feedback for timeline editor (Phase 3+) +- All Phase 1 UI (devices panel, device form) preserved + +--- +*Phase: 02-audio-engine* +*Completed: 2026-04-06*