From b0e13bfab88a05b9a09ada0ddb24abeb0e1dacd3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Apr 2026 10:57:18 +0000 Subject: [PATCH] docs(06): capture phase context --- .planning/phases/06-ai-sync/06-CONTEXT.md | 121 +++++++++++++ .../phases/06-ai-sync/06-DISCUSSION-LOG.md | 161 ++++++++++++++++++ 2 files changed, 282 insertions(+) create mode 100644 .planning/phases/06-ai-sync/06-CONTEXT.md create mode 100644 .planning/phases/06-ai-sync/06-DISCUSSION-LOG.md diff --git a/.planning/phases/06-ai-sync/06-CONTEXT.md b/.planning/phases/06-ai-sync/06-CONTEXT.md new file mode 100644 index 0000000..b5becc2 --- /dev/null +++ b/.planning/phases/06-ai-sync/06-CONTEXT.md @@ -0,0 +1,121 @@ +# Phase 6: AI Sync - Context + +**Gathered:** 2026-04-07 +**Status:** Ready for planning + + +## Phase Boundary + +YouTube URL loading (yt-dlp → local file → existing audio stack), AI-assisted timeline auto-fill from multi-feature audio analysis (beats/onsets/chroma/RMS heuristics + optional LLM generation), and structural segmentation overlaid on the waveform as colored labeled bands. No microcontroller firmware (Phase 7). + + + + +## Implementation Decisions + +### YouTube Loading (06-01) +- **D-01:** yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download completes, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change. +- **D-02:** During download, show a terminal-style progress bar with download percentage. Parse yt-dlp `--progress` output and stream updates to the browser. Disable the URL input during download; re-enable and auto-load when complete. +- **D-03:** URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners). + +### AI Sync Engine (06-02) +- **D-04:** Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly). +- **D-05:** Heuristic features to extract and use: + - Beats — from existing `beats.py` `beat_track` + - Onsets — `librosa.onset.onset_detect()` for note attacks and transients + - Chroma — `librosa.feature.chroma_cqt()` for tonal/harmonic content → drives color palette selection + - RMS energy — `librosa.feature.rms()` for loudness envelope → drives animation intensity +- **D-06:** LLM path: generate from scratch. The LLM receives raw audio features (beat list, segment boundaries, onset times, energy profile, chroma summary) and generates a complete block list — timestamps, duration, animation type, params per device. The heuristic result is NOT passed to the LLM; it generates independently. +- **D-07:** Conflict handling: prompt before replacing. If the timeline has existing blocks when AI Sync is triggered, show a confirmation dialog: "Replace all existing blocks?" Yes clears and re-fills, No cancels. Undo/redo (Phase 4) can recover from replacements. +- **D-08:** AI Sync trigger lives in a dedicated panel or context menu — not in the main toolbar or transport bar. Keeps the primary UI uncluttered. + +### Segmentation Overlay (06-03) +- **D-09:** Translucent colored band overlays drawn on the waveform canvas — one band per detected section, spanning the full waveform height. Alternating colors (e.g. cyan/purple variants from the existing palette). +- **D-10:** Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no attempt to infer VERSE/CHORUS. Simple, always correct, terminal-readable. +- **D-11:** Bands are interactive: clicking a section band selects it (highlight border). When AI Sync is triggered with a section selected, it fills only that section's time range. This allows per-section AI fill rather than whole-show fill. + +### Claude's Discretion +- **librosa segmentation algorithm:** Choose the best-fit algorithm from librosa's segment module (e.g. `librosa.segment.agglomerative` with recurrence matrix, or MSAF-style structural analysis). Target 4–12 sections for a typical 3–5 minute song. Planner decides implementation details. +- **LLM prompt design:** Structure the feature payload as a compact JSON (beat timestamps, RMS samples at 1s resolution, chroma average per section, segment boundaries). Planner designs the prompt. Use existing Anthropic SDK patterns from the codebase if available. +- **yt-dlp progress parsing:** Parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach. +- **Section color palette:** Pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside the existing terminal palette. + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Requirements +- `.planning/REQUIREMENTS.md` — SYNC-03 (AI-assisted show generation), AUD-02 (YouTube URL loading) +- `.planning/ROADMAP.md` §Phase 6 — Success criteria (3 items), plan breakdown (06-01 through 06-03) + +### Existing audio analysis +- `lightsync/audio/beats.py` — `analyze(path)` → `{tempo, beats}` using `librosa.beat.beat_track`. Phase 6 extends this module (or adds `segments.py`) for onset/chroma/RMS/segmentation. +- `lightsync/api/audio.py` — `/api/audio/upload`, `/api/audio/beats`, `/api/audio/waveform` endpoints. YouTube download adds a new `/api/audio/fetch` (or similar) endpoint here. + +### Existing frontend wiring +- `lightsync/frontend/app.js` — Audio load flow: file selected → `audio.src = /shows/filename` → `loadedmetadata` fires → `timeline.loadWaveform(path)` + `timeline.loadBeats(path)`. YouTube download reuses the same post-load path. +- `lightsync/frontend/timeline/timeline.js` — `loadWaveform()` and `loadBeats()` methods; segmentation overlay drawn on the same waveform canvas. +- `lightsync/frontend/index.html` — Transport bar layout (file selector, upload, LOAD button); URL input + FETCH button are added to this row. +- `lightsync/frontend/style.css` — CSS variables; all new UI elements must use `--accent`, `--bg`, `--border`, `--text-dim`. + +### Data models +- `lightsync/models/show.py` — `source_type: Literal["file", "youtube"]` already present; set this when a YouTube URL is used. + +### Prior context +- `.planning/phases/01-foundation/01-CONTEXT.md` — Visual aesthetic (flat, sharp, cyan/teal, no rounded corners, no drop shadows, monospace uppercase labels) +- `.planning/phases/05-live-show-execution/05-CONTEXT.md` — WebSocket protocol patterns, transport bar layout, CSS variable usage + + + + +## Existing Code Insights + +### Reusable Assets +- `beats.py` `analyze_async()` — thread-pool wrapper for librosa analysis; new segmentation/feature extraction follows the same `asyncio.to_thread` pattern +- `audio.py` upload endpoint — POST handler pattern; YouTube fetch endpoint mirrors this (async subprocess + cache beat data after download) +- `app.state.beats` cache — keyed by filepath; add `app.state.segments` for segmentation cache in the same lifespan setup +- `timeline.js` waveform canvas — already drawing waveform peaks; segmentation bands drawn as canvas `fillRect` calls on the same canvas layer, below the waveform + +### Established Patterns +- Librosa analysis runs in `asyncio.to_thread` — never blocks the event loop +- New API endpoints go into `lightsync/api/audio.py` (or a new `lightsync/api/sync.py` router) +- Frontend ES modules — new JS files must use `export`/`import` +- WebSocket message protocol: `{"type": "...", ...}` — AI sync progress can use a `sync_progress` message type +- CSS panel strip pattern from Phase 4/5: compact horizontal strips, `1px solid var(--border)` borders + +### Integration Points +- Transport bar (`index.html`) — add URL text input + FETCH button next to existing file controls +- `audio.py` router — add `/api/audio/fetch` (POST, takes `{url: string}`) for yt-dlp download; `/api/audio/segments` for structural analysis +- `timeline.js` `loadWaveform()` / `loadBeats()` — add `loadSegments()` method to render colored bands +- `main.py` lifespan — add `app.state.segments = {}` alongside existing `app.state.beats` +- AI Sync panel — new dedicated panel or popover; triggers `/api/sync/generate` (POST) with `{path, use_llm: bool, section_range?: [start, end]}` + + + + +## Specific Ideas + +- Progress bar during YouTube download should feel terminal: `DOWNLOADING ██████░░░░ 62%` — monospace, fills left to right, uses `--accent` color. Not a smooth CSS animation, a stepped block fill. +- Section bands: translucent (opacity ~0.15-0.20), drawn below the waveform fill so the waveform is still readable. Labels in small monospace uppercase at the top-left of each band. +- The AI Sync panel could look like a terminal dialog — dark background, border, `AI SYNC` header, feature toggles (use beats / onsets / chroma / RMS all checked by default), a `GENERATE` button, and an optional `USE LLM` toggle. +- When a section band is selected, draw a 1px bright border around it (using `--accent` at full opacity). Deselect when clicking elsewhere. + + + + +## Deferred Ideas + +- Per-section manual label renaming (e.g. rename SEC 2 to CHORUS) — out of Phase 6 scope +- LLM-named sections — deferred; generic numbering is sufficient for v1 +- Full MPV backend (streaming YouTube without download) — out of scope, download approach is cleaner +- AI Sync with partial fill (gap-only mode) — deferred; prompt+replace is sufficient for Phase 6 + + + +--- + +*Phase: 06-ai-sync* +*Context gathered: 2026-04-07* diff --git a/.planning/phases/06-ai-sync/06-DISCUSSION-LOG.md b/.planning/phases/06-ai-sync/06-DISCUSSION-LOG.md new file mode 100644 index 0000000..2f6f71c --- /dev/null +++ b/.planning/phases/06-ai-sync/06-DISCUSSION-LOG.md @@ -0,0 +1,161 @@ +# Phase 6: AI Sync - Discussion Log + +> **Audit trail only.** Do not use as input to planning, research, or execution agents. +> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered. + +**Date:** 2026-04-07 +**Phase:** 06-ai-sync +**Areas discussed:** YouTube Loading, AI Sync Engine, Segmentation Overlay UX + +--- + +## YouTube Loading + +| Option | Description | Selected | +|--------|-------------|----------| +| yt-dlp download | Server runs yt-dlp to download to /app/shows/; reuses all existing wiring | ✓ | +| yt-dlp extract + stream | Extract direct audio URL, pass to browser; no disk write | | +| Add MPV backend | Introduce MPV as audio engine with --ytdl flag; replaces transport layer | | + +**User's choice:** yt-dlp download to /app/shows/ +**Notes:** No MPV backend change. Download reuses the existing local file flow entirely. + +--- + +### Loading UI + +| Option | Description | Selected | +|--------|-------------|----------| +| Spinner + progress text | 'DOWNLOADING... [title]' in transport bar | | +| Progress bar | Terminal-style % bar via yt-dlp --progress parsing | ✓ | +| Just disable + wait | Disable UI, no feedback | | + +**User's choice:** Terminal-style progress bar with download percentage + +--- + +### URL input placement + +| Option | Description | Selected | +|--------|-------------|----------| +| Transport bar | Inline with existing file controls | ✓ | +| Separate panel / modal | Dedicated overlay | | + +**User's choice:** Transport bar — inline text input + FETCH button + +--- + +## AI Sync Engine + +### Heuristics vs. LLM + +| Option | Description | Selected | +|--------|-------------|----------| +| Heuristics only | beat/onset/chroma → rules, no API key needed | | +| Heuristics + optional LLM | Heuristic base + optional LLM enhance step | ✓ | +| LLM-first | LLM generates full show, heuristics as fallback | | + +**User's choice:** Heuristics + optional LLM + +--- + +### Audio features + +| Feature | Description | Selected | +|---------|-------------|----------| +| Beats | librosa beat_track (existing) | ✓ | +| Onsets | librosa.onset.onset_detect() | ✓ | +| Chroma / key | librosa.feature.chroma_cqt() | ✓ | +| RMS energy | librosa.feature.rms() | ✓ | + +**User's choice:** All four features + +--- + +### LLM path behavior + +| Option | Description | Selected | +|--------|-------------|----------| +| Remap + rename | LLM rewrites heuristic animation assignments | | +| Generate from scratch | LLM gets raw features, generates complete block list | ✓ | +| Describe + confirm | LLM describes song, user reviews, heuristics execute | | + +**User's choice:** Generate from scratch — LLM receives raw audio features, produces a full block list independently + +--- + +### Conflict handling + +| Option | Description | Selected | +|--------|-------------|----------| +| Prompt first | Confirmation dialog before replacing existing blocks | ✓ | +| Always replace | Clear and fill, no prompt (undo recovers) | | +| Fill gaps only | Only fill time ranges with no existing blocks | | + +**User's choice:** Prompt first + +--- + +### AI Sync button location + +| Option | Description | Selected | +|--------|-------------|----------| +| Toolbar / timeline header | Button in timeline controls area | | +| Transport bar | Next to play/pause | | +| Context menu / separate panel | Hidden behind menu or dedicated panel | ✓ | + +**User's choice:** Context menu / separate panel — keeps main UI clean + +--- + +## Segmentation Overlay UX + +### Visual style + +| Option | Description | Selected | +|--------|-------------|----------| +| Colored bands with labels | Translucent colored rectangles + SEC 1, SEC 2... labels | ✓ | +| Color bands only | Same but no labels | | +| Region markers only | Vertical tick marks at section boundaries | | + +**User's choice:** Colored bands with labels + +--- + +### Label naming + +| Option | Description | Selected | +|--------|-------------|----------| +| Generic numbered | SEC 1, SEC 2, SEC 3... | ✓ | +| Structural guess | INTRO / VERSE / CHORUS / OUTRO heuristics | | +| LLM-named | Feed segments to LLM for naming | | + +**User's choice:** Generic numbered (SEC 1, SEC 2, ...) + +--- + +### Interactivity + +| Option | Description | Selected | +|--------|-------------|----------| +| Purely visual | No click behavior | | +| Click to seek | Click band → seek to section start | | +| Click to select + fill | Click selects section; AI Sync can fill only that section | ✓ | + +**User's choice:** Click to select + fill — AI Sync respects selected section as target range + +--- + +## Claude's Discretion + +- librosa segmentation algorithm choice +- LLM prompt design and feature payload structure +- yt-dlp progress output parsing approach +- Section color palette (from existing CSS variables) + +## Deferred Ideas + +- Per-section manual label renaming +- LLM-named sections +- Full MPV backend / streaming YouTube +- Gap-only fill mode for AI Sync