---
phase: 06-ai-sync
plan: 02
subsystem: audio-analysis
tags: [librosa, segmentation, feature-extraction, canvas-overlay, timeline]
dependency_graph:
  requires: [06-01]
  provides: [audio-features, structural-segments, segment-overlay-ui]
  affects: [lightsync/audio, lightsync/api/audio.py, lightsync/frontend/timeline]
tech_stack:
  added: []
  patterns: [asyncio.to_thread for CPU-bound librosa analysis, canvas overlay layer, agglomerative clustering]
key_files:
  created:
    - lightsync/audio/features.py
    - lightsync/audio/segments.py
    - lightsync/frontend/timeline/segments.js
  modified:
    - lightsync/api/audio.py
    - lightsync/main.py
    - lightsync/frontend/timeline/timeline.js
    - lightsync/frontend/app.js
decisions:
  - "Segment band drawing inserted between step 2 (track row banding) and step 3 (waveform) in render() for correct Z-order"
  - "agglomerative returns k boundaries yielding k+1 sections — all_times = [0.0] + bound_times + [duration] pattern"
  - "Auto segment count heuristic: max(4, min(12, int(duration / 30))) — 4-12 sections for typical songs"
  - "Segment click handling added on empty-canvas clicks only — block interactions take priority"
  - "BAND_COLORS alternates 3 colors (cyan, purple, darker-cyan) for visual variety"
metrics:
  duration: "~8 min"
  completed: "2026-04-07"
  tasks_completed: 2
  files_changed: 7
requirements_satisfied: [SYNC-03]
---

# Phase 06 Plan 02: Audio Feature Extraction and Segment Overlay Summary

**One-liner:** Librosa-based audio feature extraction (onsets, chroma, RMS) and structural segmentation with interactive translucent section bands on the timeline canvas.

## What Was Built

Added structural audio analysis foundation for the AI sync engine and visible segment overlay UI:

1. **Feature extraction backend (Task 1, partial):** New `lightsync/audio/features.py` module:
   - `extract_features(path)` extracts onset_times, rms_1s, chroma_1s, duration, sr
   - Uses librosa onset detection, chroma_cqt, and RMS energy; reduces to 1s resolution
   - `extract_features_async` wraps in asyncio.to_thread

2. **Segmentation backend (Task 1, partial):** New `lightsync/audio/segments.py` module:
   - `detect_segments(path, n_segments=None)` uses chroma + recurrence matrix + agglomerative clustering
   - Auto segment count heuristic: `max(4, min(12, int(duration / 30)))` for 4-12 sections
   - Returns `[{"start": float, "end": float, "label": "SEC N"}, ...]`
   - `detect_segments_async` wraps in asyncio.to_thread

3. **API endpoints (Task 1, partial):**
   - `GET /api/audio/segments?path=...` — returns cached structural segments
   - `GET /api/audio/features?path=...` — returns onsets, chroma, RMS
   - Both endpoints added to `lightsync/api/audio.py`
   - `app.state.segments = {}` cache initialized in `main.py` lifespan

4. **Segment overlay UI (Task 2):** New `lightsync/frontend/timeline/segments.js`:
   - `SegmentOverlay` class manages segment data, drawing, and click interaction
   - `draw()` renders translucent colored bands using BAND_COLORS (cyan/purple alternation)
   - `handleClick(time)` selects segment containing that time, deselects on empty click
   - `getSelected()` returns currently selected segment or null
   - Labels render as `SEC 1`, `SEC 2`, etc. in 9px monospace at top-left of each band
   - Selected band highlighted with 1px `rgba(0,255,255,1.0)` border

5. **Timeline integration (Task 2):**
   - `import { SegmentOverlay }` added to `timeline.js`
   - `this.segmentOverlay = new SegmentOverlay()` in constructor
   - `segmentOverlay.draw()` inserted between step 2 (row banding) and step 3 (waveform) in render()
   - `async loadSegments(audioPath)` method added to TimelineCanvas
   - Segment click handling in mousedown: empty-canvas clicks update segment selection
   - `timeline.loadSegments(path)` called in `app.js` loadedmetadata handler

## Commits

| Task | Commit | Description |
|------|--------|-------------|
| 1 | `d339ba5` | feat(06-02): add feature extraction and segmentation backend modules + API |
| 2 | `48e15ee` | feat(06-02): add segment overlay canvas rendering and click interaction |

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all functionality is fully wired.

## Self-Check: PASSED

- `lightsync/audio/features.py` exists: contains `def extract_features`, `async def extract_features_async`, `librosa.onset.onset_detect`, `librosa.feature.chroma_cqt`, `librosa.feature.rms`, returns keys `onset_times`, `rms_1s`, `chroma_1s`, `duration`, `sr`
- `lightsync/audio/segments.py` exists: contains `def detect_segments`, `async def detect_segments_async`, `librosa.segment.agglomerative`, `librosa.segment.recurrence_matrix`, `SEC {i + 1}`, `max(4, min(12,`
- `lightsync/api/audio.py` contains `async def get_segments`, `async def get_features`, `from lightsync.audio.segments import detect_segments_async`, `from lightsync.audio.features import extract_features_async`
- `lightsync/main.py` contains `app.state.segments = {}`
- `lightsync/frontend/timeline/segments.js` exists: contains `class SegmentOverlay`, `export { SegmentOverlay }`, `BAND_COLORS`, `rgba(0,255,255,0.15)`, `rgba(180,0,255,0.15)`, `handleClick`, `getSelected`, `seg.label`, `9px monospace`
- `lightsync/frontend/timeline/timeline.js` contains `import { SegmentOverlay }`, `this.segmentOverlay = new SegmentOverlay()`, `segmentOverlay.draw(`, `async loadSegments(audioPath)`
- `lightsync/frontend/app.js` contains `loadSegments(path)` in loadedmetadata handler
- All task commits exist: `d339ba5`, `48e15ee`