docs(06): capture phase context

.planning/phases/06-ai-sync/06-CONTEXT.md

@@ -0,0 +1,121 @@
+# Phase 6: AI Sync - Context
+
+**Gathered:** 2026-04-07
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+YouTube URL loading (yt-dlp → local file → existing audio stack), AI-assisted timeline auto-fill from multi-feature audio analysis (beats/onsets/chroma/RMS heuristics + optional LLM generation), and structural segmentation overlaid on the waveform as colored labeled bands. No microcontroller firmware (Phase 7).
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### YouTube Loading (06-01)
+- **D-01:** yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download completes, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
+- **D-02:** During download, show a terminal-style progress bar with download percentage. Parse yt-dlp `--progress` output and stream updates to the browser. Disable the URL input during download; re-enable and auto-load when complete.
+- **D-03:** URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners).
+
+### AI Sync Engine (06-02)
+- **D-04:** Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
+- **D-05:** Heuristic features to extract and use:
+  - Beats — from existing `beats.py` `beat_track`
+  - Onsets — `librosa.onset.onset_detect()` for note attacks and transients
+  - Chroma — `librosa.feature.chroma_cqt()` for tonal/harmonic content → drives color palette selection
+  - RMS energy — `librosa.feature.rms()` for loudness envelope → drives animation intensity
+- **D-06:** LLM path: generate from scratch. The LLM receives raw audio features (beat list, segment boundaries, onset times, energy profile, chroma summary) and generates a complete block list — timestamps, duration, animation type, params per device. The heuristic result is NOT passed to the LLM; it generates independently.
+- **D-07:** Conflict handling: prompt before replacing. If the timeline has existing blocks when AI Sync is triggered, show a confirmation dialog: "Replace all existing blocks?" Yes clears and re-fills, No cancels. Undo/redo (Phase 4) can recover from replacements.
+- **D-08:** AI Sync trigger lives in a dedicated panel or context menu — not in the main toolbar or transport bar. Keeps the primary UI uncluttered.
+
+### Segmentation Overlay (06-03)
+- **D-09:** Translucent colored band overlays drawn on the waveform canvas — one band per detected section, spanning the full waveform height. Alternating colors (e.g. cyan/purple variants from the existing palette).
+- **D-10:** Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no attempt to infer VERSE/CHORUS. Simple, always correct, terminal-readable.
+- **D-11:** Bands are interactive: clicking a section band selects it (highlight border). When AI Sync is triggered with a section selected, it fills only that section's time range. This allows per-section AI fill rather than whole-show fill.
+
+### Claude's Discretion
+- **librosa segmentation algorithm:** Choose the best-fit algorithm from librosa's segment module (e.g. `librosa.segment.agglomerative` with recurrence matrix, or MSAF-style structural analysis). Target 4–12 sections for a typical 3–5 minute song. Planner decides implementation details.
+- **LLM prompt design:** Structure the feature payload as a compact JSON (beat timestamps, RMS samples at 1s resolution, chroma average per section, segment boundaries). Planner designs the prompt. Use existing Anthropic SDK patterns from the codebase if available.
+- **yt-dlp progress parsing:** Parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach.
+- **Section color palette:** Pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside the existing terminal palette.
+
+</decisions>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Requirements
+- `.planning/REQUIREMENTS.md` — SYNC-03 (AI-assisted show generation), AUD-02 (YouTube URL loading)
+- `.planning/ROADMAP.md` §Phase 6 — Success criteria (3 items), plan breakdown (06-01 through 06-03)
+
+### Existing audio analysis
+- `lightsync/audio/beats.py` — `analyze(path)` → `{tempo, beats}` using `librosa.beat.beat_track`. Phase 6 extends this module (or adds `segments.py`) for onset/chroma/RMS/segmentation.
+- `lightsync/api/audio.py` — `/api/audio/upload`, `/api/audio/beats`, `/api/audio/waveform` endpoints. YouTube download adds a new `/api/audio/fetch` (or similar) endpoint here.
+
+### Existing frontend wiring
+- `lightsync/frontend/app.js` — Audio load flow: file selected → `audio.src = /shows/filename` → `loadedmetadata` fires → `timeline.loadWaveform(path)` + `timeline.loadBeats(path)`. YouTube download reuses the same post-load path.
+- `lightsync/frontend/timeline/timeline.js` — `loadWaveform()` and `loadBeats()` methods; segmentation overlay drawn on the same waveform canvas.
+- `lightsync/frontend/index.html` — Transport bar layout (file selector, upload, LOAD button); URL input + FETCH button are added to this row.
+- `lightsync/frontend/style.css` — CSS variables; all new UI elements must use `--accent`, `--bg`, `--border`, `--text-dim`.
+
+### Data models
+- `lightsync/models/show.py` — `source_type: Literal["file", "youtube"]` already present; set this when a YouTube URL is used.
+
+### Prior context
+- `.planning/phases/01-foundation/01-CONTEXT.md` — Visual aesthetic (flat, sharp, cyan/teal, no rounded corners, no drop shadows, monospace uppercase labels)
+- `.planning/phases/05-live-show-execution/05-CONTEXT.md` — WebSocket protocol patterns, transport bar layout, CSS variable usage
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- `beats.py` `analyze_async()` — thread-pool wrapper for librosa analysis; new segmentation/feature extraction follows the same `asyncio.to_thread` pattern
+- `audio.py` upload endpoint — POST handler pattern; YouTube fetch endpoint mirrors this (async subprocess + cache beat data after download)
+- `app.state.beats` cache — keyed by filepath; add `app.state.segments` for segmentation cache in the same lifespan setup
+- `timeline.js` waveform canvas — already drawing waveform peaks; segmentation bands drawn as canvas `fillRect` calls on the same canvas layer, below the waveform
+
+### Established Patterns
+- Librosa analysis runs in `asyncio.to_thread` — never blocks the event loop
+- New API endpoints go into `lightsync/api/audio.py` (or a new `lightsync/api/sync.py` router)
+- Frontend ES modules — new JS files must use `export`/`import`
+- WebSocket message protocol: `{"type": "...", ...}` — AI sync progress can use a `sync_progress` message type
+- CSS panel strip pattern from Phase 4/5: compact horizontal strips, `1px solid var(--border)` borders
+
+### Integration Points
+- Transport bar (`index.html`) — add URL text input + FETCH button next to existing file controls
+- `audio.py` router — add `/api/audio/fetch` (POST, takes `{url: string}`) for yt-dlp download; `/api/audio/segments` for structural analysis
+- `timeline.js` `loadWaveform()` / `loadBeats()` — add `loadSegments()` method to render colored bands
+- `main.py` lifespan — add `app.state.segments = {}` alongside existing `app.state.beats`
+- AI Sync panel — new dedicated panel or popover; triggers `/api/sync/generate` (POST) with `{path, use_llm: bool, section_range?: [start, end]}`
+
+</code_context>
+
+<specifics>
+## Specific Ideas
+
+- Progress bar during YouTube download should feel terminal: `DOWNLOADING ██████░░░░ 62%` — monospace, fills left to right, uses `--accent` color. Not a smooth CSS animation, a stepped block fill.
+- Section bands: translucent (opacity ~0.15-0.20), drawn below the waveform fill so the waveform is still readable. Labels in small monospace uppercase at the top-left of each band.
+- The AI Sync panel could look like a terminal dialog — dark background, border, `AI SYNC` header, feature toggles (use beats / onsets / chroma / RMS all checked by default), a `GENERATE` button, and an optional `USE LLM` toggle.
+- When a section band is selected, draw a 1px bright border around it (using `--accent` at full opacity). Deselect when clicking elsewhere.
+
+</specifics>
+
+<deferred>
+## Deferred Ideas
+
+- Per-section manual label renaming (e.g. rename SEC 2 to CHORUS) — out of Phase 6 scope
+- LLM-named sections — deferred; generic numbering is sufficient for v1
+- Full MPV backend (streaming YouTube without download) — out of scope, download approach is cleaner
+- AI Sync with partial fill (gap-only mode) — deferred; prompt+replace is sufficient for Phase 6
+
+</deferred>
+
+---
+
+*Phase: 06-ai-sync*
+*Context gathered: 2026-04-07*

.planning/phases/06-ai-sync/06-DISCUSSION-LOG.md

@@ -0,0 +1,161 @@
+# Phase 6: AI Sync - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-04-07
+**Phase:** 06-ai-sync
+**Areas discussed:** YouTube Loading, AI Sync Engine, Segmentation Overlay UX
+
+---
+
+## YouTube Loading
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| yt-dlp download | Server runs yt-dlp to download to /app/shows/; reuses all existing wiring | ✓ |
+| yt-dlp extract + stream | Extract direct audio URL, pass to browser; no disk write | |
+| Add MPV backend | Introduce MPV as audio engine with --ytdl flag; replaces transport layer | |
+
+**User's choice:** yt-dlp download to /app/shows/
+**Notes:** No MPV backend change. Download reuses the existing local file flow entirely.
+
+---
+
+### Loading UI
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Spinner + progress text | 'DOWNLOADING... [title]' in transport bar | |
+| Progress bar | Terminal-style % bar via yt-dlp --progress parsing | ✓ |
+| Just disable + wait | Disable UI, no feedback | |
+
+**User's choice:** Terminal-style progress bar with download percentage
+
+---
+
+### URL input placement
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Transport bar | Inline with existing file controls | ✓ |
+| Separate panel / modal | Dedicated overlay | |
+
+**User's choice:** Transport bar — inline text input + FETCH button
+
+---
+
+## AI Sync Engine
+
+### Heuristics vs. LLM
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Heuristics only | beat/onset/chroma → rules, no API key needed | |
+| Heuristics + optional LLM | Heuristic base + optional LLM enhance step | ✓ |
+| LLM-first | LLM generates full show, heuristics as fallback | |
+
+**User's choice:** Heuristics + optional LLM
+
+---
+
+### Audio features
+
+| Feature | Description | Selected |
+|---------|-------------|----------|
+| Beats | librosa beat_track (existing) | ✓ |
+| Onsets | librosa.onset.onset_detect() | ✓ |
+| Chroma / key | librosa.feature.chroma_cqt() | ✓ |
+| RMS energy | librosa.feature.rms() | ✓ |
+
+**User's choice:** All four features
+
+---
+
+### LLM path behavior
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Remap + rename | LLM rewrites heuristic animation assignments | |
+| Generate from scratch | LLM gets raw features, generates complete block list | ✓ |
+| Describe + confirm | LLM describes song, user reviews, heuristics execute | |
+
+**User's choice:** Generate from scratch — LLM receives raw audio features, produces a full block list independently
+
+---
+
+### Conflict handling
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Prompt first | Confirmation dialog before replacing existing blocks | ✓ |
+| Always replace | Clear and fill, no prompt (undo recovers) | |
+| Fill gaps only | Only fill time ranges with no existing blocks | |
+
+**User's choice:** Prompt first
+
+---
+
+### AI Sync button location
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Toolbar / timeline header | Button in timeline controls area | |
+| Transport bar | Next to play/pause | |
+| Context menu / separate panel | Hidden behind menu or dedicated panel | ✓ |
+
+**User's choice:** Context menu / separate panel — keeps main UI clean
+
+---
+
+## Segmentation Overlay UX
+
+### Visual style
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Colored bands with labels | Translucent colored rectangles + SEC 1, SEC 2... labels | ✓ |
+| Color bands only | Same but no labels | |
+| Region markers only | Vertical tick marks at section boundaries | |
+
+**User's choice:** Colored bands with labels
+
+---
+
+### Label naming
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Generic numbered | SEC 1, SEC 2, SEC 3... | ✓ |
+| Structural guess | INTRO / VERSE / CHORUS / OUTRO heuristics | |
+| LLM-named | Feed segments to LLM for naming | |
+
+**User's choice:** Generic numbered (SEC 1, SEC 2, ...)
+
+---
+
+### Interactivity
+
+| Option | Description | Selected |
+|--------|-------------|----------|
+| Purely visual | No click behavior | |
+| Click to seek | Click band → seek to section start | |
+| Click to select + fill | Click selects section; AI Sync can fill only that section | ✓ |
+
+**User's choice:** Click to select + fill — AI Sync respects selected section as target range
+
+---
+
+## Claude's Discretion
+
+- librosa segmentation algorithm choice
+- LLM prompt design and feature payload structure
+- yt-dlp progress output parsing approach
+- Section color palette (from existing CSS variables)
+
+## Deferred Ideas
+
+- Per-section manual label renaming
+- LLM-named sections
+- Full MPV backend / streaming YouTube
+- Gap-only fill mode for AI Sync