2fef43dcc3
- Create 06-02-SUMMARY.md with full execution record - Update STATE.md: plan advanced to 3/3, progress 95%, decisions logged - Update ROADMAP.md: phase 6 now 2/3 summaries - Mark SYNC-03 complete in REQUIREMENTS.md
103 lines
5.5 KiB
Markdown
103 lines
5.5 KiB
Markdown
---
|
|
phase: 06-ai-sync
|
|
plan: 02
|
|
subsystem: audio-analysis
|
|
tags: [librosa, segmentation, feature-extraction, canvas-overlay, timeline]
|
|
dependency_graph:
|
|
requires: [06-01]
|
|
provides: [audio-features, structural-segments, segment-overlay-ui]
|
|
affects: [lightsync/audio, lightsync/api/audio.py, lightsync/frontend/timeline]
|
|
tech_stack:
|
|
added: []
|
|
patterns: [asyncio.to_thread for CPU-bound librosa analysis, canvas overlay layer, agglomerative clustering]
|
|
key_files:
|
|
created:
|
|
- lightsync/audio/features.py
|
|
- lightsync/audio/segments.py
|
|
- lightsync/frontend/timeline/segments.js
|
|
modified:
|
|
- lightsync/api/audio.py
|
|
- lightsync/main.py
|
|
- lightsync/frontend/timeline/timeline.js
|
|
- lightsync/frontend/app.js
|
|
decisions:
|
|
- "Segment band drawing inserted between step 2 (track row banding) and step 3 (waveform) in render() for correct Z-order"
|
|
- "agglomerative returns k boundaries yielding k+1 sections — all_times = [0.0] + bound_times + [duration] pattern"
|
|
- "Auto segment count heuristic: max(4, min(12, int(duration / 30))) — 4-12 sections for typical songs"
|
|
- "Segment click handling added on empty-canvas clicks only — block interactions take priority"
|
|
- "BAND_COLORS alternates 3 colors (cyan, purple, darker-cyan) for visual variety"
|
|
metrics:
|
|
duration: "~8 min"
|
|
completed: "2026-04-07"
|
|
tasks_completed: 2
|
|
files_changed: 7
|
|
requirements_satisfied: [SYNC-03]
|
|
---
|
|
|
|
# Phase 06 Plan 02: Audio Feature Extraction and Segment Overlay Summary
|
|
|
|
**One-liner:** Librosa-based audio feature extraction (onsets, chroma, RMS) and structural segmentation with interactive translucent section bands on the timeline canvas.
|
|
|
|
## What Was Built
|
|
|
|
Added structural audio analysis foundation for the AI sync engine and visible segment overlay UI:
|
|
|
|
1. **Feature extraction backend (Task 1, partial):** New `lightsync/audio/features.py` module:
|
|
- `extract_features(path)` extracts onset_times, rms_1s, chroma_1s, duration, sr
|
|
- Uses librosa onset detection, chroma_cqt, and RMS energy; reduces to 1s resolution
|
|
- `extract_features_async` wraps in asyncio.to_thread
|
|
|
|
2. **Segmentation backend (Task 1, partial):** New `lightsync/audio/segments.py` module:
|
|
- `detect_segments(path, n_segments=None)` uses chroma + recurrence matrix + agglomerative clustering
|
|
- Auto segment count heuristic: `max(4, min(12, int(duration / 30)))` for 4-12 sections
|
|
- Returns `[{"start": float, "end": float, "label": "SEC N"}, ...]`
|
|
- `detect_segments_async` wraps in asyncio.to_thread
|
|
|
|
3. **API endpoints (Task 1, partial):**
|
|
- `GET /api/audio/segments?path=...` — returns cached structural segments
|
|
- `GET /api/audio/features?path=...` — returns onsets, chroma, RMS
|
|
- Both endpoints added to `lightsync/api/audio.py`
|
|
- `app.state.segments = {}` cache initialized in `main.py` lifespan
|
|
|
|
4. **Segment overlay UI (Task 2):** New `lightsync/frontend/timeline/segments.js`:
|
|
- `SegmentOverlay` class manages segment data, drawing, and click interaction
|
|
- `draw()` renders translucent colored bands using BAND_COLORS (cyan/purple alternation)
|
|
- `handleClick(time)` selects segment containing that time, deselects on empty click
|
|
- `getSelected()` returns currently selected segment or null
|
|
- Labels render as `SEC 1`, `SEC 2`, etc. in 9px monospace at top-left of each band
|
|
- Selected band highlighted with 1px `rgba(0,255,255,1.0)` border
|
|
|
|
5. **Timeline integration (Task 2):**
|
|
- `import { SegmentOverlay }` added to `timeline.js`
|
|
- `this.segmentOverlay = new SegmentOverlay()` in constructor
|
|
- `segmentOverlay.draw()` inserted between step 2 (row banding) and step 3 (waveform) in render()
|
|
- `async loadSegments(audioPath)` method added to TimelineCanvas
|
|
- Segment click handling in mousedown: empty-canvas clicks update segment selection
|
|
- `timeline.loadSegments(path)` called in `app.js` loadedmetadata handler
|
|
|
|
## Commits
|
|
|
|
| Task | Commit | Description |
|
|
|------|--------|-------------|
|
|
| 1 | `d339ba5` | feat(06-02): add feature extraction and segmentation backend modules + API |
|
|
| 2 | `48e15ee` | feat(06-02): add segment overlay canvas rendering and click interaction |
|
|
|
|
## Deviations from Plan
|
|
|
|
None - plan executed exactly as written.
|
|
|
|
## Known Stubs
|
|
|
|
None - all functionality is fully wired.
|
|
|
|
## Self-Check: PASSED
|
|
|
|
- `lightsync/audio/features.py` exists: contains `def extract_features`, `async def extract_features_async`, `librosa.onset.onset_detect`, `librosa.feature.chroma_cqt`, `librosa.feature.rms`, returns keys `onset_times`, `rms_1s`, `chroma_1s`, `duration`, `sr`
|
|
- `lightsync/audio/segments.py` exists: contains `def detect_segments`, `async def detect_segments_async`, `librosa.segment.agglomerative`, `librosa.segment.recurrence_matrix`, `SEC {i + 1}`, `max(4, min(12,`
|
|
- `lightsync/api/audio.py` contains `async def get_segments`, `async def get_features`, `from lightsync.audio.segments import detect_segments_async`, `from lightsync.audio.features import extract_features_async`
|
|
- `lightsync/main.py` contains `app.state.segments = {}`
|
|
- `lightsync/frontend/timeline/segments.js` exists: contains `class SegmentOverlay`, `export { SegmentOverlay }`, `BAND_COLORS`, `rgba(0,255,255,0.15)`, `rgba(180,0,255,0.15)`, `handleClick`, `getSelected`, `seg.label`, `9px monospace`
|
|
- `lightsync/frontend/timeline/timeline.js` contains `import { SegmentOverlay }`, `this.segmentOverlay = new SegmentOverlay()`, `segmentOverlay.draw(`, `async loadSegments(audioPath)`
|
|
- `lightsync/frontend/app.js` contains `loadSegments(path)` in loadedmetadata handler
|
|
- All task commits exist: `d339ba5`, `48e15ee`
|