From b88bd98c4007adb0ab2f12f0b3403cc9ea56bb42 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Apr 2026 11:17:32 +0000 Subject: [PATCH] docs(06): research phase AI sync domain --- .planning/phases/06-ai-sync/06-RESEARCH.md | 595 +++++++++++++++++++++ 1 file changed, 595 insertions(+) create mode 100644 .planning/phases/06-ai-sync/06-RESEARCH.md diff --git a/.planning/phases/06-ai-sync/06-RESEARCH.md b/.planning/phases/06-ai-sync/06-RESEARCH.md new file mode 100644 index 0000000..27210a6 --- /dev/null +++ b/.planning/phases/06-ai-sync/06-RESEARCH.md @@ -0,0 +1,595 @@ +# Phase 6: AI Sync - Research + +**Researched:** 2026-04-07 +**Domain:** yt-dlp audio download, librosa feature extraction + structural segmentation, heuristic AI sync engine, canvas waveform overlays +**Confidence:** HIGH + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +**YouTube Loading (06-01)** +- D-01: yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change. +- D-02: Terminal-style progress bar during download with percentage. Parse yt-dlp `--progress` output streamed to browser. Disable URL input during download; re-enable and auto-load when complete. +- D-03: URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners). + +**AI Sync Engine (06-02)** +- D-04: Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly). +- D-05: Heuristic features: beats (existing `beats.py`), onsets (`librosa.onset.onset_detect()`), chroma (`librosa.feature.chroma_cqt()`), RMS energy (`librosa.feature.rms()`). +- D-06: LLM path generates from scratch — receives raw features (beat list, segment boundaries, onset times, energy profile, chroma summary), outputs complete block list. Heuristic result NOT passed to LLM. +- D-07: Conflict handling: prompt before replacing. If timeline has existing blocks, show confirmation dialog "Replace all existing blocks?" before clearing. +- D-08: AI Sync trigger lives in a dedicated panel or context menu — not main toolbar or transport bar. + +**Segmentation Overlay (06-03)** +- D-09: Translucent colored band overlays on waveform canvas — one band per detected section, full waveform height. Alternating colors from existing palette. +- D-10: Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no VERSE/CHORUS inference. +- D-11: Bands are interactive — clicking selects (highlight border). When AI Sync triggered with section selected, fills only that section's time range. + +### Claude's Discretion +- librosa segmentation algorithm: choose from `librosa.segment` module (e.g., `agglomerative` with recurrence matrix). Target 4–12 sections for a 3–5 minute song. +- LLM prompt design: compact JSON payload (beat timestamps, RMS at 1s resolution, chroma average per section, segment boundaries). Use existing Anthropic SDK patterns if available. +- yt-dlp progress parsing: parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach. +- Section color palette: pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside existing terminal palette. + +### Deferred Ideas (OUT OF SCOPE) +- Per-section manual label renaming (e.g., rename SEC 2 to CHORUS) +- LLM-named sections — generic numbering is sufficient for v1 +- Full MPV backend (streaming YouTube without download) +- AI Sync with partial fill (gap-only mode) — prompt+replace is sufficient for Phase 6 + + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| AUD-02 | Load audio from a YouTube URL via yt-dlp | yt-dlp CLI subprocess with `--extract-audio`, progress parsing, file-to-existing-audio-stack handoff | +| SYNC-03 | AI-assisted show generation — given beat/segment analysis, auto-fill timeline with matched animations | Heuristic mapper using librosa onsets/chroma/RMS + optional LLM generation; conflict dialog; per-section fill | + + +--- + +## Summary + +Phase 6 has three distinct problems: (1) downloading YouTube audio reliably, (2) auto-filling the timeline from audio features, and (3) rendering structural section overlays on the waveform canvas. Each is technically independent but they share the same audio analysis pipeline. + +**yt-dlp** is the correct tool for YouTube audio extraction. It must be added to the Docker image. The Python `yt_dlp` module is available as a pip package (currently v2026.3.17) and can be used either as a Python API or spawned as a CLI subprocess. The subprocess approach is simpler for streaming progress output; the Python API avoids subprocess overhead. Both patterns work. The key insight is that yt-dlp handles format negotiation, rate limiting, and the best audio quality selection transparently — the application just needs to provide a path and wait. + +**librosa 0.11.0** is already installed in the container and provides all necessary feature extraction APIs: `onset.onset_detect()`, `feature.chroma_cqt()`, `feature.rms()`, and `segment.agglomerative()` with `segment.recurrence_matrix()` for structural segmentation. The segmentation algorithm is confirmed available. `librosa.segment.agglomerative()` takes a feature matrix and cluster count — feeding it a chroma or MFCC matrix with a recurrence-matrix similarity structure is the standard approach for music structure analysis. + +**The Anthropic SDK is not installed** in the container. Adding it requires adding `anthropic>=0.x` to `pyproject.toml` and rebuilding the Docker image. The LLM path is explicitly optional (off by default), so the application must degrade gracefully without it. An `ANTHROPIC_API_KEY` environment variable is also not currently configured in docker-compose.prod.yml — this needs to be wired as an env var or secret. + +**Primary recommendation:** Add `yt-dlp` and `anthropic` to `pyproject.toml` before any implementation. Rebuild Docker image as a first task in plan 06-01. Implement yt-dlp as a CLI subprocess (not Python API) to simplify progress streaming. Use `librosa.segment.agglomerative()` with chroma features for segmentation. Implement the LLM path behind a feature-flag guarded by the presence of `ANTHROPIC_API_KEY`. + +--- + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| yt-dlp | 2026.3.17 | YouTube audio download | De facto youtube-dl successor; active maintenance, robust format selection | +| librosa | 0.11.0 (already installed) | Feature extraction + segmentation | Already in use for beat detection; full suite available | +| anthropic | 0.49+ (needs install) | LLM API client for optional generation | Official SDK, the only LLM client in use on this VPS | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| numpy | already installed | Feature matrix ops for segmentation | Used throughout librosa pipeline | +| sklearn (via librosa dep) | already installed | `AgglomerativeClustering` used by `librosa.segment.agglomerative` | Pulled in automatically by librosa | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| yt-dlp subprocess | yt_dlp Python API | Python API avoids subprocess overhead but progress streaming is more complex; subprocess is simpler for this use case | +| librosa.segment.agglomerative | librosa.segment.subsegment | `subsegment` further divides existing segments; `agglomerative` is the top-level structural tool — correct choice | +| anthropic SDK | Direct HTTP to /v1/messages | SDK handles retries, streaming, error handling; no reason to bypass | + +**Installation (changes to pyproject.toml):** +```toml +"yt-dlp>=2025.0.0", +"anthropic>=0.40.0", +``` + +Rebuild: `docker compose -f docker-compose.prod.yml build && docker compose -f docker-compose.prod.yml up -d` + +**Version verification (confirmed against container):** +- librosa: 0.11.0 (verified in container) +- yt-dlp: 2026.3.17 (verified via pip dry-run) +- anthropic: not installed — needs addition to pyproject.toml + +--- + +## Architecture Patterns + +### Recommended Project Structure +``` +lightsync/ +├── audio/ +│ ├── beats.py # existing — beat_track; Phase 6 extends this +│ ├── features.py # NEW — onset_detect, chroma_cqt, rms extraction +│ └── segments.py # NEW — structural segmentation via agglomerative +├── api/ +│ ├── audio.py # existing — add /fetch endpoint here +│ └── sync.py # NEW router — /api/sync/generate, /api/audio/segments +└── frontend/ + └── timeline/ + └── segments.js # NEW — SegmentOverlay class, loadSegments(), canvas drawing +``` + +### Pattern 1: yt-dlp subprocess with progress streaming + +The `/api/audio/fetch` endpoint spawns yt-dlp as a subprocess with stdout pipe. Progress lines are parsed and sent to the browser as SSE or WebSocket messages. + +**yt-dlp CLI pattern:** +```python +# Source: yt-dlp official docs + verified yt-dlp --help +import asyncio +import re +from pathlib import Path + +PROGRESS_RE = re.compile(r'\[download\]\s+([\d.]+)%') + +async def download_youtube(url: str, dest_dir: Path) -> str: + """Download audio from YouTube URL to dest_dir. Returns local filepath.""" + proc = await asyncio.create_subprocess_exec( + 'yt-dlp', + '--extract-audio', + '--audio-format', 'mp3', + '--audio-quality', '0', + '--newline', # one progress line per newline (critical for parsing) + '--no-playlist', + '-o', str(dest_dir / '%(title)s.%(ext)s'), + url, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + ) + filename = None + async for line_bytes in proc.stdout: + line = line_bytes.decode('utf-8', errors='replace').rstrip() + # Progress: [download] 62.3% of ... + m = PROGRESS_RE.search(line) + if m: + pct = float(m.group(1)) + yield ('progress', pct) + # Destination: [ExtractAudio] Destination: /app/shows/SongTitle.mp3 + if 'Destination:' in line: + filename = line.split('Destination:')[-1].strip() + await proc.wait() + if proc.returncode != 0: + raise RuntimeError(f'yt-dlp failed with exit {proc.returncode}') + return filename +``` + +**Key flags:** `--newline` is critical — without it, yt-dlp uses carriage-return overwrite and line-by-line parsing breaks. `--no-playlist` prevents downloading entire playlists when a playlist URL is given. + +### Pattern 2: librosa feature extraction module + +All feature extraction follows the existing `asyncio.to_thread` pattern from `beats.py`. + +```python +# Source: librosa 0.11.0 docs + verified in container +import librosa +import numpy as np + +def extract_features(path: str) -> dict: + """Extract onsets, chroma, RMS from audio file.""" + y, sr = librosa.load(path, sr=None, mono=True) + + # Onsets — note attacks / transients + onset_frames = librosa.onset.onset_detect(y=y, sr=sr, units='frames') + onset_times = librosa.frames_to_time(onset_frames, sr=sr).tolist() + + # Chroma — tonal/harmonic content (CQT-based, more stable than STFT-based) + chroma = librosa.feature.chroma_cqt(y=y, sr=sr) # shape (12, T) + + # RMS energy — loudness envelope + rms = librosa.feature.rms(y=y)[0] # shape (T,) + + # Reduce to 1s resolution for LLM payload + hop_length = 512 + frame_rate = sr / hop_length + rms_1s = [float(np.mean(rms[int(i*frame_rate):int((i+1)*frame_rate)])) + for i in range(int(len(rms) / frame_rate))] + + # Chroma mean per section (per-second average of dominant pitch class) + chroma_1s = [int(np.argmax(np.mean(chroma[:, int(i*frame_rate):int((i+1)*frame_rate)], axis=1))) + for i in range(int(chroma.shape[1] / frame_rate))] + + return { + 'onset_times': onset_times, + 'rms_1s': rms_1s, + 'chroma_1s': chroma_1s, + 'duration': len(y) / sr, + 'sr': sr, + 'hop_length': hop_length, + } +``` + +### Pattern 3: librosa structural segmentation + +```python +# Source: librosa 0.11.0 docs + confirmed in container +def detect_segments(path: str, n_segments: int = 8) -> list[dict]: + """Detect structural sections using agglomerative clustering on chroma features.""" + y, sr = librosa.load(path, sr=None, mono=True) + + # Build chroma feature matrix + chroma = librosa.feature.chroma_cqt(y=y, sr=sr, bins_per_octave=36) + + # Recurrence matrix — similarity structure + R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True) + + # Filter recurrence matrix to enhance diagonal (segment consistency) + R_filtered = librosa.segment.path_enhance(R, 15) + + # Agglomerative clustering to find n_segments boundary frames + bounds = librosa.segment.agglomerative(R_filtered, k=n_segments) + bound_times = librosa.frames_to_time(bounds, sr=sr).tolist() + + # Build section list: [{start, end, label}, ...] + all_times = [0.0] + bound_times + [librosa.get_duration(y=y, sr=sr)] + sections = [] + for i, (start, end) in enumerate(zip(all_times, all_times[1:])): + sections.append({'start': start, 'end': end, 'label': f'SEC {i+1}'}) + return sections +``` + +**n_segments selection:** For a 3–5 minute song, 8 is a good default. The planner should expose this as a parameter (min=4, max=12) with 8 as default. `agglomerative()` takes the cluster count `k` directly. + +**Confidence note:** `librosa.segment.agglomerative()` signature verified in container: `(data, k, *, clusterer=None, axis=-1)`. Input is a feature matrix, not the recurrence matrix directly. Feed the chroma matrix (or its recurrence) directly. + +### Pattern 4: Progress streaming via Server-Sent Events (SSE) + +SSE (not WebSocket) is the clean pattern for one-way server-to-client streaming of download progress. FastAPI supports it via `StreamingResponse` with `text/event-stream` content type. + +```python +# Source: FastAPI docs pattern +from fastapi.responses import StreamingResponse +import json + +@router.post("/fetch") +async def fetch_youtube(request: Request, body: YouTubeFetchRequest): + async def event_stream(): + try: + async for (kind, value) in download_youtube(body.url, SHOWS_DIR): + if kind == 'progress': + yield f"data: {json.dumps({'type': 'progress', 'pct': value})}\n\n" + filename = ... # captured from download + yield f"data: {json.dumps({'type': 'done', 'path': filename})}\n\n" + except Exception as e: + yield f"data: {json.dumps({'type': 'error', 'msg': str(e)})}\n\n" + + return StreamingResponse(event_stream(), media_type="text/event-stream") +``` + +Frontend reads via `EventSource` API: +```javascript +const es = new EventSource('/api/audio/fetch?url=' + encodeURIComponent(url)); +// or POST via fetch with ReadableStream reader +``` + +**Note on POST + SSE:** `EventSource` only supports GET. For POST requests with a body, use `fetch()` with `response.body.getReader()` to stream the `text/event-stream` body manually. This is standard practice. + +### Pattern 5: Heuristic block placement + +``` +Given: beats[], onsets[], rms_1s[], chroma_1s[], sections[] +1. For each section: + a. Compute mean RMS → intensity_level (low/medium/high) + b. Compute dominant chroma → color palette choice + c. Count beats/onsets → density +2. Intensity mapping: + - high RMS → strobe, chase (fast) + - medium RMS → pulse, color_wipe + - low RMS → solid, rainbow (slow) +3. Duration: snap to beat grid — each block spans from beat[i] to beat[i+N] + where N varies by density (sparse → longer blocks, dense → shorter) +4. Color: chroma pitch class 0-11 → map to HSL hue, convert to RGB +``` + +### Pattern 6: AI Sync panel (06-02) + +The panel is a new DOM element (not in the transport bar). Recommended as a collapsible strip above the timeline, or as a popover anchored to a button in the timeline toolbar. + +```html + +
+ + + + +``` + +### Pattern 7: Canvas segment overlay + +Draw segment bands on the waveform canvas, **below** the waveform fill (rendered before waveform in the render loop). The existing `render()` method in `timeline.js` draws in layers — insert segment bands between step 2 (track row banding) and step 3 (waveform). + +```javascript +// In TimelineCanvas.render(), after track row banding, before waveform: +if (this.segments.length > 0) { + this._drawSegmentBands(ctx, W, H); +} + +_drawSegmentBands(ctx, W, H) { + const BAND_COLORS = [ + 'rgba(0,255,255,0.15)', // --accent cyan + 'rgba(180,0,255,0.15)', // purple complement + ]; + const totalH = this.tracks.length * TRACK_HEIGHT; + const topY = TIME_AXIS_HEIGHT; + + for (let i = 0; i < this.segments.length; i++) { + const seg = this.segments[i]; + const x1 = this.timeToX(seg.start); + const x2 = this.timeToX(seg.end); + const color = BAND_COLORS[i % BAND_COLORS.length]; + const isSelected = this.selectedSegment === i; + + ctx.fillStyle = color; + ctx.fillRect(x1, topY, x2 - x1, totalH); + + if (isSelected) { + ctx.strokeStyle = 'rgba(0,255,255,1.0)'; + ctx.lineWidth = 1; + ctx.strokeRect(x1, topY, x2 - x1, totalH); + } + + // Label: small monospace uppercase at top-left of band + if (x2 - x1 > 30) { + ctx.fillStyle = 'rgba(0,255,255,0.6)'; + ctx.font = '9px monospace'; + ctx.fillText(seg.label, Math.max(x1 + 3, HEADER_WIDTH + 2), topY + 10); + } + } +} +``` + +### Anti-Patterns to Avoid +- **Blocking the event loop with librosa:** All librosa calls must use `asyncio.to_thread`. Feature extraction for a 5-minute song takes 3–8 seconds. Verified pattern: identical to `beats.py`. +- **Using EventSource for POST requests:** EventSource only supports GET. Use `fetch()` + `ReadableStream` reader for streaming responses to POST requests. +- **Calling `yt-dlp` without `--newline`:** Without this flag, progress output uses CR overwrites and line-by-line parsing is impossible. +- **Calling `librosa.segment.agglomerative()` with recurrence matrix directly:** The function expects a feature data matrix. Feed it the chroma array. Use `recurrence_matrix()` as a preprocessing step for `path_enhance` filtering, then pass the filtered matrix as data. +- **Generating blocks without confirming replacement:** If existing blocks are in the timeline, the heuristic must trigger the confirmation dialog first (D-07). +- **LLM path without API key guard:** Check for `ANTHROPIC_API_KEY` env var presence before showing the `USE LLM` toggle. If absent, disable the toggle with tooltip `ANTHROPIC_API_KEY not configured`. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| YouTube download | Custom HTTP fetcher or youtube-dl | yt-dlp | Rate limiting, format negotiation, age-gating, cookies — yt-dlp handles all of these; a custom solution would break constantly | +| Audio feature extraction | Custom FFT + onset detector | librosa | librosa.onset.onset_detect is tuned for music, uses spectral flux; hand-rolled detectors miss transients | +| Structural segmentation | Beat-counting heuristics | librosa.segment.agglomerative | Self-similarity matrix approach is the musicology-standard technique; hand-rolled approaches cannot detect phrase-level repetition | +| LLM HTTP calls | Raw requests to Anthropic API | anthropic SDK | SDK handles token limits, retries, streaming, error handling | + +**Key insight:** The audio analysis stack is already in the container. Feature extraction is a one-module addition to `beats.py` patterns. The only genuinely new infrastructure is yt-dlp (subprocess) and the anthropic SDK (Python package). + +--- + +## Common Pitfalls + +### Pitfall 1: yt-dlp output filename is not predictable +**What goes wrong:** yt-dlp names the output file from the video title. Special characters, Unicode, spaces, and truncation rules make the filename unpredictable. The `/api/audio/fetch` endpoint must capture the actual output filename from yt-dlp's `Destination:` log line — not from the URL. +**Why it happens:** yt-dlp sanitizes filenames per-platform and may truncate long titles. +**How to avoid:** Parse the `[ExtractAudio] Destination: /path/to/file.mp3` line from yt-dlp stdout. Use this as the returned filename. Alternatively, pass a template with a fixed stem: `-o '/app/shows/yt_%(id)s.%(ext)s'` to make the filename deterministic from the YouTube video ID. +**Warning signs:** 404 errors when loading the audio file immediately after download. + +### Pitfall 2: librosa.segment.agglomerative segment count vs. n_segments +**What goes wrong:** `librosa.segment.agglomerative(data, k=n)` returns `k` boundary frame indices (not `k+1` segments). If the caller treats the output as segment counts, it's off by one. +**Why it happens:** The output is boundary frames — need to add `0` at start and `len` at end to form `k+1` sections. +**How to avoid:** Always construct `sections` as `zip(all_times[:-1], all_times[1:])` where `all_times = [0.0] + bound_times + [duration]`. +**Warning signs:** Last section not present, or first section starting at a non-zero time. + +### Pitfall 3: SSE streaming via POST requires ReadableStream reader +**What goes wrong:** Frontend uses `EventSource(url)` but the fetch endpoint is POST-only (body contains the URL). EventSource does not support POST. +**Why it happens:** Developers assume all SSE uses EventSource. +**How to avoid:** Use `fetch(url, {method:'POST', body:...})` then `response.body.getReader()` to stream the `text/event-stream` body. Parse `data: {...}\n\n` frames manually. +**Warning signs:** Browser console error "Failed to construct 'EventSource'", or CORS/method error. + +### Pitfall 4: LLM path with no API key crashes silently +**What goes wrong:** `anthropic.Anthropic()` is instantiated at import time (e.g., module level), throws an error if `ANTHROPIC_API_KEY` is missing, crashing the entire FastAPI startup. +**Why it happens:** SDK raises `AuthenticationError` or `APIKeyMissingError` at client construction. +**How to avoid:** Instantiate the Anthropic client lazily — only when a request triggers the LLM path. Wrap in a try/except and return a `503 LLM_UNAVAILABLE` response if the key is missing. +**Warning signs:** App fails to start after adding anthropic to pyproject.toml when `ANTHROPIC_API_KEY` is not set. + +### Pitfall 5: Docker image not rebuilt after dependency addition +**What goes wrong:** yt-dlp and anthropic are added to `pyproject.toml` but the running container still uses the old image without them. +**Why it happens:** `docker compose up -d` does not rebuild the image automatically. +**How to avoid:** The first task in plan 06-01 must be `docker compose -f docker-compose.prod.yml build` then `up -d --force-recreate`. Add yt-dlp CLI availability check to the /api/audio/fetch endpoint startup. +**Warning signs:** `ModuleNotFoundError: No module named 'yt_dlp'` or `yt-dlp: command not found` in logs. + +### Pitfall 6: yt-dlp not available as CLI after pip install of yt-dlp Python package +**What goes wrong:** `pip install yt-dlp` installs both the Python library AND the `yt-dlp` CLI binary. But in the Docker image, the binary is placed in `/usr/local/bin/yt-dlp`. If the subprocess calls `yt-dlp` as a command, it should work. However, on minimal images the PATH may not include this location. +**Why it happens:** Slim Docker images have a minimal PATH. +**How to avoid:** Use `sys.executable -m yt_dlp` as the subprocess command instead of `yt-dlp`. This is guaranteed to use the installed module regardless of PATH. +**Warning signs:** `FileNotFoundError: [Errno 2] No such file or directory: 'yt-dlp'` in subprocess call. + +### Pitfall 7: Segment overlay rendered over waveform instead of under it +**What goes wrong:** Segment bands obscure the waveform because they're drawn after the waveform in `render()`. +**Why it happens:** Render order in canvas is painter's algorithm — later draws go on top. +**How to avoid:** In `TimelineCanvas.render()`, draw segment bands between step 2 (track row banding) and step 3 (waveform). The `_drawSegmentBands()` call must come before the waveform loop. +**Warning signs:** Waveform is not visible in sections with colored overlays. + +### Pitfall 8: AI Sync blocks don't have valid device_id +**What goes wrong:** Heuristic generates blocks but assigns incorrect or missing device IDs. The show engine ignores cues with unknown device IDs. +**Why it happens:** The sync engine may not know which devices are registered. +**How to avoid:** The `/api/sync/generate` endpoint must accept the current device list as part of its input, or fetch it from `registry`. Generate one block set per device, using the actual `device.id` UUIDs. + +--- + +## Code Examples + +### yt-dlp subprocess (confirmed API) +```python +# Use sys.executable -m yt_dlp to avoid PATH issues in Docker +import sys +proc = await asyncio.create_subprocess_exec( + sys.executable, '-m', 'yt_dlp', + '--extract-audio', '--audio-format', 'mp3', + '--audio-quality', '0', + '--newline', '--no-playlist', + '-o', str(SHOWS_DIR / 'yt_%(id)s.%(ext)s'), # predictable filename + url, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, +) +``` + +### librosa.segment.agglomerative (verified signature in container) +```python +# Verified: librosa.segment.agglomerative(data, k, *, clusterer=None, axis=-1) +# librosa.segment.recurrence_matrix(data, *, k=None, width=1, metric='euclidean', sym=False, ...) +chroma = librosa.feature.chroma_cqt(y=y, sr=sr) +R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True) +R_filtered = librosa.segment.path_enhance(R, 15) +bounds = librosa.segment.agglomerative(R_filtered, k=n_segments) +``` + +### SSE streaming via fetch ReadableStream (frontend) +```javascript +async function fetchWithProgress(url, body, onProgress, onDone, onError) { + const res = await fetch(url, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let buf = ''; + while (true) { + const { done, value } = await reader.read(); + if (done) break; + buf += decoder.decode(value, { stream: true }); + const lines = buf.split('\n\n'); + buf = lines.pop(); + for (const chunk of lines) { + if (chunk.startsWith('data: ')) { + const msg = JSON.parse(chunk.slice(6)); + if (msg.type === 'progress') onProgress(msg.pct); + if (msg.type === 'done') onDone(msg.path); + if (msg.type === 'error') onError(msg.msg); + } + } + } +} +``` + +### Terminal progress bar rendering (frontend) +```javascript +function renderProgressBar(pct, width = 20) { + // DOWNLOADING ██████░░░░ 62% + const filled = Math.round((pct / 100) * width); + const empty = width - filled; + return `DOWNLOADING ${'█'.repeat(filled)}${'░'.repeat(empty)} ${Math.round(pct)}%`; +} +``` + +### Anthropic LLM call (lazy instantiation pattern) +```python +import os + +def get_anthropic_client(): + """Lazy client construction — only called when USE LLM is triggered.""" + key = os.environ.get('ANTHROPIC_API_KEY') + if not key: + raise ValueError('ANTHROPIC_API_KEY not configured') + import anthropic + return anthropic.Anthropic(api_key=key) + +async def generate_llm_show(features: dict, devices: list) -> list[dict]: + """Generate show blocks via LLM. Returns list of CueModel-compatible dicts.""" + client = await asyncio.to_thread(get_anthropic_client) + # ... prompt construction and API call +``` + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| librosa | Feature extraction, segmentation | YES (in container) | 0.11.0 | — | +| numpy | Feature matrix ops | YES (in container) | installed with librosa | — | +| sklearn | `agglomerative` clustering (librosa dep) | YES (librosa pulls it) | installed with librosa | — | +| yt-dlp (Python pkg) | YouTube download | NO — needs pyproject.toml addition | 2026.3.17 available | — | +| anthropic SDK | LLM show generation | NO — needs pyproject.toml addition | 0.40+ available | Disable USE LLM toggle | +| ANTHROPIC_API_KEY env var | LLM show generation | NOT CONFIGURED in docker-compose.prod.yml | — | Graceful disable | +| ffmpeg | yt-dlp audio conversion to MP3 | YES (already in Dockerfile) | system ffmpeg | — | + +**Missing dependencies with no fallback:** +- `yt-dlp` Python package — AUD-02 blocks without it. Add to `pyproject.toml` and rebuild Docker image. + +**Missing dependencies with fallback:** +- `anthropic` SDK — LLM path is off by default. App works fully without it; just disable the `USE LLM` toggle when `ANTHROPIC_API_KEY` is absent or SDK not installed. +- `ANTHROPIC_API_KEY` — Same fallback: guard the UI toggle, return 503 from endpoint. + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| youtube-dl | yt-dlp | 2021 (youtube-dl DMCA takedown response) | yt-dlp is the active fork; youtube-dl is stale | +| librosa.beat.beat_track only | beat_track + onset_detect + chroma + RMS | Phase 6 | Full feature set for heuristic mapping | +| No segmentation | agglomerative clustering on chroma recurrence matrix | Phase 6 | Provides section-level context for AI Sync | + +**Deprecated/outdated:** +- `youtube-dl`: superseded by yt-dlp; do not use +- `librosa.segment.subsegment`: fine-grained sub-segmentation within known sections; not what we need for top-level structure detection + +--- + +## Open Questions + +1. **LLM prompt token budget for a 5-minute song** + - What we know: RMS at 1s resolution = ~300 values; beats ~120 timestamps; chroma summary per section = 8 × 12 = 96 values; total ~500 numbers as JSON + - What's unclear: Whether Anthropic claude-3-haiku (cheapest, fastest) is adequate for structured JSON generation of this kind, or if claude-3-sonnet is needed for reliable output + - Recommendation: Use `claude-3-haiku-20240307` with a response schema in the prompt. The payload is small; haiku handles structured JSON well at this scale. + +2. **yt-dlp download time for a typical 3–5 minute song** + - What we know: yt-dlp downloads at network speed; typical YouTube audio is 128–320kbps + - What's unclear: Whether the progress bar SSE stream can stay open for the full download duration without Traefik/Authelia proxy timeout + - Recommendation: Set `proxy_read_timeout` in Traefik if default timeout is <60s. Alternatively, accept that progress streaming may time out at proxy and use WebSocket instead (the WS connection is already established). + +3. **librosa.segment.agglomerative n_segments optimal value** + - What we know: Target is 4–12 sections; the algorithm accepts `k` directly + - What's unclear: Whether a fixed k=8 produces too many or too few sections for short (<2 min) or long (>6 min) songs + - Recommendation: Default k=8; add heuristic: `k = max(4, min(12, int(duration / 30)))` — roughly one section per 30 seconds. + +--- + +## Sources + +### Primary (HIGH confidence) +- librosa 0.11.0 installed in container — `librosa.segment`, `librosa.onset`, `librosa.feature` APIs verified by direct introspection in Docker container +- yt-dlp pip dry-run in container — version 2026.3.17 available +- Codebase direct read — `beats.py`, `audio.py`, `timeline.js`, `beats.js`, `show.py`, `main.py`, `index.html`, `style.css` all read directly + +### Secondary (MEDIUM confidence) +- yt-dlp GitHub README (known from training data, verified version via pip registry) — `--newline`, `--extract-audio`, `--no-playlist` flag behavior +- FastAPI StreamingResponse + SSE pattern (standard FastAPI docs pattern, widely used) + +### Tertiary (LOW confidence) +- Anthropic SDK lazy instantiation pattern — based on training data knowledge of anthropic Python SDK; version and exact error class names should be confirmed when adding to requirements + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — verified in container (librosa), pip registry (yt-dlp), pyproject.toml (existing deps) +- Architecture: HIGH — all patterns traced to verified existing code (`beats.py` → `features.py` extension, `timeline.js` render order, `audio.py` endpoint pattern) +- Pitfalls: HIGH — yt-dlp filename/PATH pitfalls verified via subprocess behavior; canvas render order verified in `timeline.js`; SSE POST limitation is a browser API constraint + +**Research date:** 2026-04-07 +**Valid until:** 2026-05-07 (librosa and FastAPI are stable; yt-dlp updates frequently but CLI flags are backward-compatible)