4.7 KiB
Phase 6: AI Sync - Discussion Log
Audit trail only. Do not use as input to planning, research, or execution agents. Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
Date: 2026-04-07 Phase: 06-ai-sync Areas discussed: YouTube Loading, AI Sync Engine, Segmentation Overlay UX
YouTube Loading
| Option | Description | Selected |
|---|---|---|
| yt-dlp download | Server runs yt-dlp to download to /app/shows/; reuses all existing wiring | ✓ |
| yt-dlp extract + stream | Extract direct audio URL, pass to browser; no disk write | |
| Add MPV backend | Introduce MPV as audio engine with --ytdl flag; replaces transport layer |
User's choice: yt-dlp download to /app/shows/ Notes: No MPV backend change. Download reuses the existing local file flow entirely.
Loading UI
| Option | Description | Selected |
|---|---|---|
| Spinner + progress text | 'DOWNLOADING... [title]' in transport bar | |
| Progress bar | Terminal-style % bar via yt-dlp --progress parsing | ✓ |
| Just disable + wait | Disable UI, no feedback |
User's choice: Terminal-style progress bar with download percentage
URL input placement
| Option | Description | Selected |
|---|---|---|
| Transport bar | Inline with existing file controls | ✓ |
| Separate panel / modal | Dedicated overlay |
User's choice: Transport bar — inline text input + FETCH button
AI Sync Engine
Heuristics vs. LLM
| Option | Description | Selected |
|---|---|---|
| Heuristics only | beat/onset/chroma → rules, no API key needed | |
| Heuristics + optional LLM | Heuristic base + optional LLM enhance step | ✓ |
| LLM-first | LLM generates full show, heuristics as fallback |
User's choice: Heuristics + optional LLM
Audio features
| Feature | Description | Selected |
|---|---|---|
| Beats | librosa beat_track (existing) | ✓ |
| Onsets | librosa.onset.onset_detect() | ✓ |
| Chroma / key | librosa.feature.chroma_cqt() | ✓ |
| RMS energy | librosa.feature.rms() | ✓ |
User's choice: All four features
LLM path behavior
| Option | Description | Selected |
|---|---|---|
| Remap + rename | LLM rewrites heuristic animation assignments | |
| Generate from scratch | LLM gets raw features, generates complete block list | ✓ |
| Describe + confirm | LLM describes song, user reviews, heuristics execute |
User's choice: Generate from scratch — LLM receives raw audio features, produces a full block list independently
Conflict handling
| Option | Description | Selected |
|---|---|---|
| Prompt first | Confirmation dialog before replacing existing blocks | ✓ |
| Always replace | Clear and fill, no prompt (undo recovers) | |
| Fill gaps only | Only fill time ranges with no existing blocks |
User's choice: Prompt first
AI Sync button location
| Option | Description | Selected |
|---|---|---|
| Toolbar / timeline header | Button in timeline controls area | |
| Transport bar | Next to play/pause | |
| Context menu / separate panel | Hidden behind menu or dedicated panel | ✓ |
User's choice: Context menu / separate panel — keeps main UI clean
Segmentation Overlay UX
Visual style
| Option | Description | Selected |
|---|---|---|
| Colored bands with labels | Translucent colored rectangles + SEC 1, SEC 2... labels | ✓ |
| Color bands only | Same but no labels | |
| Region markers only | Vertical tick marks at section boundaries |
User's choice: Colored bands with labels
Label naming
| Option | Description | Selected |
|---|---|---|
| Generic numbered | SEC 1, SEC 2, SEC 3... | ✓ |
| Structural guess | INTRO / VERSE / CHORUS / OUTRO heuristics | |
| LLM-named | Feed segments to LLM for naming |
User's choice: Generic numbered (SEC 1, SEC 2, ...)
Interactivity
| Option | Description | Selected |
|---|---|---|
| Purely visual | No click behavior | |
| Click to seek | Click band → seek to section start | |
| Click to select + fill | Click selects section; AI Sync can fill only that section | ✓ |
User's choice: Click to select + fill — AI Sync respects selected section as target range
Claude's Discretion
- librosa segmentation algorithm choice
- LLM prompt design and feature payload structure
- yt-dlp progress output parsing approach
- Section color palette (from existing CSS variables)
Deferred Ideas
- Per-section manual label renaming
- LLM-named sections
- Full MPV backend / streaming YouTube
- Gap-only fill mode for AI Sync