162 lines
4.7 KiB
Markdown
162 lines
4.7 KiB
Markdown
# Phase 6: AI Sync - Discussion Log
|
|
|
|
> **Audit trail only.** Do not use as input to planning, research, or execution agents.
|
|
> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
|
|
|
|
**Date:** 2026-04-07
|
|
**Phase:** 06-ai-sync
|
|
**Areas discussed:** YouTube Loading, AI Sync Engine, Segmentation Overlay UX
|
|
|
|
---
|
|
|
|
## YouTube Loading
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| yt-dlp download | Server runs yt-dlp to download to /app/shows/; reuses all existing wiring | ✓ |
|
|
| yt-dlp extract + stream | Extract direct audio URL, pass to browser; no disk write | |
|
|
| Add MPV backend | Introduce MPV as audio engine with --ytdl flag; replaces transport layer | |
|
|
|
|
**User's choice:** yt-dlp download to /app/shows/
|
|
**Notes:** No MPV backend change. Download reuses the existing local file flow entirely.
|
|
|
|
---
|
|
|
|
### Loading UI
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Spinner + progress text | 'DOWNLOADING... [title]' in transport bar | |
|
|
| Progress bar | Terminal-style % bar via yt-dlp --progress parsing | ✓ |
|
|
| Just disable + wait | Disable UI, no feedback | |
|
|
|
|
**User's choice:** Terminal-style progress bar with download percentage
|
|
|
|
---
|
|
|
|
### URL input placement
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Transport bar | Inline with existing file controls | ✓ |
|
|
| Separate panel / modal | Dedicated overlay | |
|
|
|
|
**User's choice:** Transport bar — inline text input + FETCH button
|
|
|
|
---
|
|
|
|
## AI Sync Engine
|
|
|
|
### Heuristics vs. LLM
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Heuristics only | beat/onset/chroma → rules, no API key needed | |
|
|
| Heuristics + optional LLM | Heuristic base + optional LLM enhance step | ✓ |
|
|
| LLM-first | LLM generates full show, heuristics as fallback | |
|
|
|
|
**User's choice:** Heuristics + optional LLM
|
|
|
|
---
|
|
|
|
### Audio features
|
|
|
|
| Feature | Description | Selected |
|
|
|---------|-------------|----------|
|
|
| Beats | librosa beat_track (existing) | ✓ |
|
|
| Onsets | librosa.onset.onset_detect() | ✓ |
|
|
| Chroma / key | librosa.feature.chroma_cqt() | ✓ |
|
|
| RMS energy | librosa.feature.rms() | ✓ |
|
|
|
|
**User's choice:** All four features
|
|
|
|
---
|
|
|
|
### LLM path behavior
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Remap + rename | LLM rewrites heuristic animation assignments | |
|
|
| Generate from scratch | LLM gets raw features, generates complete block list | ✓ |
|
|
| Describe + confirm | LLM describes song, user reviews, heuristics execute | |
|
|
|
|
**User's choice:** Generate from scratch — LLM receives raw audio features, produces a full block list independently
|
|
|
|
---
|
|
|
|
### Conflict handling
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Prompt first | Confirmation dialog before replacing existing blocks | ✓ |
|
|
| Always replace | Clear and fill, no prompt (undo recovers) | |
|
|
| Fill gaps only | Only fill time ranges with no existing blocks | |
|
|
|
|
**User's choice:** Prompt first
|
|
|
|
---
|
|
|
|
### AI Sync button location
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Toolbar / timeline header | Button in timeline controls area | |
|
|
| Transport bar | Next to play/pause | |
|
|
| Context menu / separate panel | Hidden behind menu or dedicated panel | ✓ |
|
|
|
|
**User's choice:** Context menu / separate panel — keeps main UI clean
|
|
|
|
---
|
|
|
|
## Segmentation Overlay UX
|
|
|
|
### Visual style
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Colored bands with labels | Translucent colored rectangles + SEC 1, SEC 2... labels | ✓ |
|
|
| Color bands only | Same but no labels | |
|
|
| Region markers only | Vertical tick marks at section boundaries | |
|
|
|
|
**User's choice:** Colored bands with labels
|
|
|
|
---
|
|
|
|
### Label naming
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Generic numbered | SEC 1, SEC 2, SEC 3... | ✓ |
|
|
| Structural guess | INTRO / VERSE / CHORUS / OUTRO heuristics | |
|
|
| LLM-named | Feed segments to LLM for naming | |
|
|
|
|
**User's choice:** Generic numbered (SEC 1, SEC 2, ...)
|
|
|
|
---
|
|
|
|
### Interactivity
|
|
|
|
| Option | Description | Selected |
|
|
|--------|-------------|----------|
|
|
| Purely visual | No click behavior | |
|
|
| Click to seek | Click band → seek to section start | |
|
|
| Click to select + fill | Click selects section; AI Sync can fill only that section | ✓ |
|
|
|
|
**User's choice:** Click to select + fill — AI Sync respects selected section as target range
|
|
|
|
---
|
|
|
|
## Claude's Discretion
|
|
|
|
- librosa segmentation algorithm choice
|
|
- LLM prompt design and feature payload structure
|
|
- yt-dlp progress output parsing approach
|
|
- Section color palette (from existing CSS variables)
|
|
|
|
## Deferred Ideas
|
|
|
|
- Per-section manual label renaming
|
|
- LLM-named sections
|
|
- Full MPV backend / streaming YouTube
|
|
- Gap-only fill mode for AI Sync
|