# Phase 6: AI Sync - Research

**Researched:** 2026-04-07
**Domain:** yt-dlp audio download, librosa feature extraction + structural segmentation, heuristic AI sync engine, canvas waveform overlays
**Confidence:** HIGH

---

<user_constraints>
## User Constraints (from CONTEXT.md)

### Locked Decisions

**YouTube Loading (06-01)**
- D-01: yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
- D-02: Terminal-style progress bar during download with percentage. Parse yt-dlp `--progress` output streamed to browser. Disable URL input during download; re-enable and auto-load when complete.
- D-03: URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners).

**AI Sync Engine (06-02)**
- D-04: Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
- D-05: Heuristic features: beats (existing `beats.py`), onsets (`librosa.onset.onset_detect()`), chroma (`librosa.feature.chroma_cqt()`), RMS energy (`librosa.feature.rms()`).
- D-06: LLM path generates from scratch — receives raw features (beat list, segment boundaries, onset times, energy profile, chroma summary), outputs complete block list. Heuristic result NOT passed to LLM.
- D-07: Conflict handling: prompt before replacing. If timeline has existing blocks, show confirmation dialog "Replace all existing blocks?" before clearing.
- D-08: AI Sync trigger lives in a dedicated panel or context menu — not main toolbar or transport bar.

**Segmentation Overlay (06-03)**
- D-09: Translucent colored band overlays on waveform canvas — one band per detected section, full waveform height. Alternating colors from existing palette.
- D-10: Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no VERSE/CHORUS inference.
- D-11: Bands are interactive — clicking selects (highlight border). When AI Sync triggered with section selected, fills only that section's time range.

### Claude's Discretion
- librosa segmentation algorithm: choose from `librosa.segment` module (e.g., `agglomerative` with recurrence matrix). Target 4–12 sections for a 3–5 minute song.
- LLM prompt design: compact JSON payload (beat timestamps, RMS at 1s resolution, chroma average per section, segment boundaries). Use existing Anthropic SDK patterns if available.
- yt-dlp progress parsing: parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach.
- Section color palette: pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside existing terminal palette.

### Deferred Ideas (OUT OF SCOPE)
- Per-section manual label renaming (e.g., rename SEC 2 to CHORUS)
- LLM-named sections — generic numbering is sufficient for v1
- Full MPV backend (streaming YouTube without download)
- AI Sync with partial fill (gap-only mode) — prompt+replace is sufficient for Phase 6
</user_constraints>

---

<phase_requirements>
## Phase Requirements

| ID | Description | Research Support |
|----|-------------|------------------|
| AUD-02 | Load audio from a YouTube URL via yt-dlp | yt-dlp CLI subprocess with `--extract-audio`, progress parsing, file-to-existing-audio-stack handoff |
| SYNC-03 | AI-assisted show generation — given beat/segment analysis, auto-fill timeline with matched animations | Heuristic mapper using librosa onsets/chroma/RMS + optional LLM generation; conflict dialog; per-section fill |
</phase_requirements>

---

## Summary

Phase 6 has three distinct problems: (1) downloading YouTube audio reliably, (2) auto-filling the timeline from audio features, and (3) rendering structural section overlays on the waveform canvas. Each is technically independent but they share the same audio analysis pipeline.

**yt-dlp** is the correct tool for YouTube audio extraction. It must be added to the Docker image. The Python `yt_dlp` module is available as a pip package (currently v2026.3.17) and can be used either as a Python API or spawned as a CLI subprocess. The subprocess approach is simpler for streaming progress output; the Python API avoids subprocess overhead. Both patterns work. The key insight is that yt-dlp handles format negotiation, rate limiting, and the best audio quality selection transparently — the application just needs to provide a path and wait.

**librosa 0.11.0** is already installed in the container and provides all necessary feature extraction APIs: `onset.onset_detect()`, `feature.chroma_cqt()`, `feature.rms()`, and `segment.agglomerative()` with `segment.recurrence_matrix()` for structural segmentation. The segmentation algorithm is confirmed available. `librosa.segment.agglomerative()` takes a feature matrix and cluster count — feeding it a chroma or MFCC matrix with a recurrence-matrix similarity structure is the standard approach for music structure analysis.

**The Anthropic SDK is not installed** in the container. Adding it requires adding `anthropic>=0.x` to `pyproject.toml` and rebuilding the Docker image. The LLM path is explicitly optional (off by default), so the application must degrade gracefully without it. An `ANTHROPIC_API_KEY` environment variable is also not currently configured in docker-compose.prod.yml — this needs to be wired as an env var or secret.

**Primary recommendation:** Add `yt-dlp` and `anthropic` to `pyproject.toml` before any implementation. Rebuild Docker image as a first task in plan 06-01. Implement yt-dlp as a CLI subprocess (not Python API) to simplify progress streaming. Use `librosa.segment.agglomerative()` with chroma features for segmentation. Implement the LLM path behind a feature-flag guarded by the presence of `ANTHROPIC_API_KEY`.

---

## Standard Stack

### Core
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| yt-dlp | 2026.3.17 | YouTube audio download | De facto youtube-dl successor; active maintenance, robust format selection |
| librosa | 0.11.0 (already installed) | Feature extraction + segmentation | Already in use for beat detection; full suite available |
| anthropic | 0.49+ (needs install) | LLM API client for optional generation | Official SDK, the only LLM client in use on this VPS |

### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| numpy | already installed | Feature matrix ops for segmentation | Used throughout librosa pipeline |
| sklearn (via librosa dep) | already installed | `AgglomerativeClustering` used by `librosa.segment.agglomerative` | Pulled in automatically by librosa |

### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| yt-dlp subprocess | yt_dlp Python API | Python API avoids subprocess overhead but progress streaming is more complex; subprocess is simpler for this use case |
| librosa.segment.agglomerative | librosa.segment.subsegment | `subsegment` further divides existing segments; `agglomerative` is the top-level structural tool — correct choice |
| anthropic SDK | Direct HTTP to /v1/messages | SDK handles retries, streaming, error handling; no reason to bypass |

**Installation (changes to pyproject.toml):**
```toml
"yt-dlp>=2025.0.0",
"anthropic>=0.40.0",
```

Rebuild: `docker compose -f docker-compose.prod.yml build && docker compose -f docker-compose.prod.yml up -d`

**Version verification (confirmed against container):**
- librosa: 0.11.0 (verified in container)
- yt-dlp: 2026.3.17 (verified via pip dry-run)
- anthropic: not installed — needs addition to pyproject.toml

---

## Architecture Patterns

### Recommended Project Structure
```
lightsync/
├── audio/
│   ├── beats.py          # existing — beat_track; Phase 6 extends this
│   ├── features.py       # NEW — onset_detect, chroma_cqt, rms extraction
│   └── segments.py       # NEW — structural segmentation via agglomerative
├── api/
│   ├── audio.py          # existing — add /fetch endpoint here
│   └── sync.py           # NEW router — /api/sync/generate, /api/audio/segments
└── frontend/
    └── timeline/
        └── segments.js   # NEW — SegmentOverlay class, loadSegments(), canvas drawing
```

### Pattern 1: yt-dlp subprocess with progress streaming

The `/api/audio/fetch` endpoint spawns yt-dlp as a subprocess with stdout pipe. Progress lines are parsed and sent to the browser as SSE or WebSocket messages.

**yt-dlp CLI pattern:**
```python
# Source: yt-dlp official docs + verified yt-dlp --help
import asyncio
import re
from pathlib import Path

PROGRESS_RE = re.compile(r'\[download\]\s+([\d.]+)%')

async def download_youtube(url: str, dest_dir: Path) -> str:
    """Download audio from YouTube URL to dest_dir. Returns local filepath."""
    proc = await asyncio.create_subprocess_exec(
        'yt-dlp',
        '--extract-audio',
        '--audio-format', 'mp3',
        '--audio-quality', '0',
        '--newline',             # one progress line per newline (critical for parsing)
        '--no-playlist',
        '-o', str(dest_dir / '%(title)s.%(ext)s'),
        url,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.STDOUT,
    )
    filename = None
    async for line_bytes in proc.stdout:
        line = line_bytes.decode('utf-8', errors='replace').rstrip()
        # Progress: [download]  62.3% of ...
        m = PROGRESS_RE.search(line)
        if m:
            pct = float(m.group(1))
            yield ('progress', pct)
        # Destination: [ExtractAudio] Destination: /app/shows/SongTitle.mp3
        if 'Destination:' in line:
            filename = line.split('Destination:')[-1].strip()
    await proc.wait()
    if proc.returncode != 0:
        raise RuntimeError(f'yt-dlp failed with exit {proc.returncode}')
    return filename
```

**Key flags:** `--newline` is critical — without it, yt-dlp uses carriage-return overwrite and line-by-line parsing breaks. `--no-playlist` prevents downloading entire playlists when a playlist URL is given.

### Pattern 2: librosa feature extraction module

All feature extraction follows the existing `asyncio.to_thread` pattern from `beats.py`.

```python
# Source: librosa 0.11.0 docs + verified in container
import librosa
import numpy as np

def extract_features(path: str) -> dict:
    """Extract onsets, chroma, RMS from audio file."""
    y, sr = librosa.load(path, sr=None, mono=True)

    # Onsets — note attacks / transients
    onset_frames = librosa.onset.onset_detect(y=y, sr=sr, units='frames')
    onset_times = librosa.frames_to_time(onset_frames, sr=sr).tolist()

    # Chroma — tonal/harmonic content (CQT-based, more stable than STFT-based)
    chroma = librosa.feature.chroma_cqt(y=y, sr=sr)  # shape (12, T)

    # RMS energy — loudness envelope
    rms = librosa.feature.rms(y=y)[0]  # shape (T,)

    # Reduce to 1s resolution for LLM payload
    hop_length = 512
    frame_rate = sr / hop_length
    rms_1s = [float(np.mean(rms[int(i*frame_rate):int((i+1)*frame_rate)]))
              for i in range(int(len(rms) / frame_rate))]

    # Chroma mean per section (per-second average of dominant pitch class)
    chroma_1s = [int(np.argmax(np.mean(chroma[:, int(i*frame_rate):int((i+1)*frame_rate)], axis=1)))
                 for i in range(int(chroma.shape[1] / frame_rate))]

    return {
        'onset_times': onset_times,
        'rms_1s': rms_1s,
        'chroma_1s': chroma_1s,
        'duration': len(y) / sr,
        'sr': sr,
        'hop_length': hop_length,
    }
```

### Pattern 3: librosa structural segmentation

```python
# Source: librosa 0.11.0 docs + confirmed in container
def detect_segments(path: str, n_segments: int = 8) -> list[dict]:
    """Detect structural sections using agglomerative clustering on chroma features."""
    y, sr = librosa.load(path, sr=None, mono=True)

    # Build chroma feature matrix
    chroma = librosa.feature.chroma_cqt(y=y, sr=sr, bins_per_octave=36)

    # Recurrence matrix — similarity structure
    R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)

    # Filter recurrence matrix to enhance diagonal (segment consistency)
    R_filtered = librosa.segment.path_enhance(R, 15)

    # Agglomerative clustering to find n_segments boundary frames
    bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
    bound_times = librosa.frames_to_time(bounds, sr=sr).tolist()

    # Build section list: [{start, end, label}, ...]
    all_times = [0.0] + bound_times + [librosa.get_duration(y=y, sr=sr)]
    sections = []
    for i, (start, end) in enumerate(zip(all_times, all_times[1:])):
        sections.append({'start': start, 'end': end, 'label': f'SEC {i+1}'})
    return sections
```

**n_segments selection:** For a 3–5 minute song, 8 is a good default. The planner should expose this as a parameter (min=4, max=12) with 8 as default. `agglomerative()` takes the cluster count `k` directly.

**Confidence note:** `librosa.segment.agglomerative()` signature verified in container: `(data, k, *, clusterer=None, axis=-1)`. Input is a feature matrix, not the recurrence matrix directly. Feed the chroma matrix (or its recurrence) directly.

### Pattern 4: Progress streaming via Server-Sent Events (SSE)

SSE (not WebSocket) is the clean pattern for one-way server-to-client streaming of download progress. FastAPI supports it via `StreamingResponse` with `text/event-stream` content type.

```python
# Source: FastAPI docs pattern
from fastapi.responses import StreamingResponse
import json

@router.post("/fetch")
async def fetch_youtube(request: Request, body: YouTubeFetchRequest):
    async def event_stream():
        try:
            async for (kind, value) in download_youtube(body.url, SHOWS_DIR):
                if kind == 'progress':
                    yield f"data: {json.dumps({'type': 'progress', 'pct': value})}\n\n"
            filename = ...  # captured from download
            yield f"data: {json.dumps({'type': 'done', 'path': filename})}\n\n"
        except Exception as e:
            yield f"data: {json.dumps({'type': 'error', 'msg': str(e)})}\n\n"

    return StreamingResponse(event_stream(), media_type="text/event-stream")
```

Frontend reads via `EventSource` API:
```javascript
const es = new EventSource('/api/audio/fetch?url=' + encodeURIComponent(url));
// or POST via fetch with ReadableStream reader
```

**Note on POST + SSE:** `EventSource` only supports GET. For POST requests with a body, use `fetch()` with `response.body.getReader()` to stream the `text/event-stream` body manually. This is standard practice.

### Pattern 5: Heuristic block placement

```
Given: beats[], onsets[], rms_1s[], chroma_1s[], sections[]
1. For each section:
   a. Compute mean RMS → intensity_level (low/medium/high)
   b. Compute dominant chroma → color palette choice
   c. Count beats/onsets → density
2. Intensity mapping:
   - high RMS → strobe, chase (fast)
   - medium RMS → pulse, color_wipe
   - low RMS → solid, rainbow (slow)
3. Duration: snap to beat grid — each block spans from beat[i] to beat[i+N]
   where N varies by density (sparse → longer blocks, dense → shorter)
4. Color: chroma pitch class 0-11 → map to HSL hue, convert to RGB
```

### Pattern 6: AI Sync panel (06-02)

The panel is a new DOM element (not in the transport bar). Recommended as a collapsible strip above the timeline, or as a popover anchored to a button in the timeline toolbar.

```html
<!-- Recommended: add to timeline-toolbar div in index.html -->
<div class="toolbar-sep"></div>
<button id="btn-ai-sync-open" class="transport-btn">AI SYNC</button>

<!-- Popover panel (shown/hidden by btn-ai-sync-open) -->
<div id="ai-sync-panel" class="ai-sync-panel" style="display:none;">
  <div class="panel-header">AI SYNC</div>
  <!-- feature toggles, GENERATE button, USE LLM toggle -->
</div>
```

### Pattern 7: Canvas segment overlay

Draw segment bands on the waveform canvas, **below** the waveform fill (rendered before waveform in the render loop). The existing `render()` method in `timeline.js` draws in layers — insert segment bands between step 2 (track row banding) and step 3 (waveform).

```javascript
// In TimelineCanvas.render(), after track row banding, before waveform:
if (this.segments.length > 0) {
    this._drawSegmentBands(ctx, W, H);
}

_drawSegmentBands(ctx, W, H) {
    const BAND_COLORS = [
        'rgba(0,255,255,0.15)',   // --accent cyan
        'rgba(180,0,255,0.15)',   // purple complement
    ];
    const totalH = this.tracks.length * TRACK_HEIGHT;
    const topY = TIME_AXIS_HEIGHT;

    for (let i = 0; i < this.segments.length; i++) {
        const seg = this.segments[i];
        const x1 = this.timeToX(seg.start);
        const x2 = this.timeToX(seg.end);
        const color = BAND_COLORS[i % BAND_COLORS.length];
        const isSelected = this.selectedSegment === i;

        ctx.fillStyle = color;
        ctx.fillRect(x1, topY, x2 - x1, totalH);

        if (isSelected) {
            ctx.strokeStyle = 'rgba(0,255,255,1.0)';
            ctx.lineWidth = 1;
            ctx.strokeRect(x1, topY, x2 - x1, totalH);
        }

        // Label: small monospace uppercase at top-left of band
        if (x2 - x1 > 30) {
            ctx.fillStyle = 'rgba(0,255,255,0.6)';
            ctx.font = '9px monospace';
            ctx.fillText(seg.label, Math.max(x1 + 3, HEADER_WIDTH + 2), topY + 10);
        }
    }
}
```

### Anti-Patterns to Avoid
- **Blocking the event loop with librosa:** All librosa calls must use `asyncio.to_thread`. Feature extraction for a 5-minute song takes 3–8 seconds. Verified pattern: identical to `beats.py`.
- **Using EventSource for POST requests:** EventSource only supports GET. Use `fetch()` + `ReadableStream` reader for streaming responses to POST requests.
- **Calling `yt-dlp` without `--newline`:** Without this flag, progress output uses CR overwrites and line-by-line parsing is impossible.
- **Calling `librosa.segment.agglomerative()` with recurrence matrix directly:** The function expects a feature data matrix. Feed it the chroma array. Use `recurrence_matrix()` as a preprocessing step for `path_enhance` filtering, then pass the filtered matrix as data.
- **Generating blocks without confirming replacement:** If existing blocks are in the timeline, the heuristic must trigger the confirmation dialog first (D-07).
- **LLM path without API key guard:** Check for `ANTHROPIC_API_KEY` env var presence before showing the `USE LLM` toggle. If absent, disable the toggle with tooltip `ANTHROPIC_API_KEY not configured`.

---

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| YouTube download | Custom HTTP fetcher or youtube-dl | yt-dlp | Rate limiting, format negotiation, age-gating, cookies — yt-dlp handles all of these; a custom solution would break constantly |
| Audio feature extraction | Custom FFT + onset detector | librosa | librosa.onset.onset_detect is tuned for music, uses spectral flux; hand-rolled detectors miss transients |
| Structural segmentation | Beat-counting heuristics | librosa.segment.agglomerative | Self-similarity matrix approach is the musicology-standard technique; hand-rolled approaches cannot detect phrase-level repetition |
| LLM HTTP calls | Raw requests to Anthropic API | anthropic SDK | SDK handles token limits, retries, streaming, error handling |

**Key insight:** The audio analysis stack is already in the container. Feature extraction is a one-module addition to `beats.py` patterns. The only genuinely new infrastructure is yt-dlp (subprocess) and the anthropic SDK (Python package).

---

## Common Pitfalls

### Pitfall 1: yt-dlp output filename is not predictable
**What goes wrong:** yt-dlp names the output file from the video title. Special characters, Unicode, spaces, and truncation rules make the filename unpredictable. The `/api/audio/fetch` endpoint must capture the actual output filename from yt-dlp's `Destination:` log line — not from the URL.
**Why it happens:** yt-dlp sanitizes filenames per-platform and may truncate long titles.
**How to avoid:** Parse the `[ExtractAudio] Destination: /path/to/file.mp3` line from yt-dlp stdout. Use this as the returned filename. Alternatively, pass a template with a fixed stem: `-o '/app/shows/yt_%(id)s.%(ext)s'` to make the filename deterministic from the YouTube video ID.
**Warning signs:** 404 errors when loading the audio file immediately after download.

### Pitfall 2: librosa.segment.agglomerative segment count vs. n_segments
**What goes wrong:** `librosa.segment.agglomerative(data, k=n)` returns `k` boundary frame indices (not `k+1` segments). If the caller treats the output as segment counts, it's off by one.
**Why it happens:** The output is boundary frames — need to add `0` at start and `len` at end to form `k+1` sections.
**How to avoid:** Always construct `sections` as `zip(all_times[:-1], all_times[1:])` where `all_times = [0.0] + bound_times + [duration]`.
**Warning signs:** Last section not present, or first section starting at a non-zero time.

### Pitfall 3: SSE streaming via POST requires ReadableStream reader
**What goes wrong:** Frontend uses `EventSource(url)` but the fetch endpoint is POST-only (body contains the URL). EventSource does not support POST.
**Why it happens:** Developers assume all SSE uses EventSource.
**How to avoid:** Use `fetch(url, {method:'POST', body:...})` then `response.body.getReader()` to stream the `text/event-stream` body. Parse `data: {...}\n\n` frames manually.
**Warning signs:** Browser console error "Failed to construct 'EventSource'", or CORS/method error.

### Pitfall 4: LLM path with no API key crashes silently
**What goes wrong:** `anthropic.Anthropic()` is instantiated at import time (e.g., module level), throws an error if `ANTHROPIC_API_KEY` is missing, crashing the entire FastAPI startup.
**Why it happens:** SDK raises `AuthenticationError` or `APIKeyMissingError` at client construction.
**How to avoid:** Instantiate the Anthropic client lazily — only when a request triggers the LLM path. Wrap in a try/except and return a `503 LLM_UNAVAILABLE` response if the key is missing.
**Warning signs:** App fails to start after adding anthropic to pyproject.toml when `ANTHROPIC_API_KEY` is not set.

### Pitfall 5: Docker image not rebuilt after dependency addition
**What goes wrong:** yt-dlp and anthropic are added to `pyproject.toml` but the running container still uses the old image without them.
**Why it happens:** `docker compose up -d` does not rebuild the image automatically.
**How to avoid:** The first task in plan 06-01 must be `docker compose -f docker-compose.prod.yml build` then `up -d --force-recreate`. Add yt-dlp CLI availability check to the /api/audio/fetch endpoint startup.
**Warning signs:** `ModuleNotFoundError: No module named 'yt_dlp'` or `yt-dlp: command not found` in logs.

### Pitfall 6: yt-dlp not available as CLI after pip install of yt-dlp Python package
**What goes wrong:** `pip install yt-dlp` installs both the Python library AND the `yt-dlp` CLI binary. But in the Docker image, the binary is placed in `/usr/local/bin/yt-dlp`. If the subprocess calls `yt-dlp` as a command, it should work. However, on minimal images the PATH may not include this location.
**Why it happens:** Slim Docker images have a minimal PATH.
**How to avoid:** Use `sys.executable -m yt_dlp` as the subprocess command instead of `yt-dlp`. This is guaranteed to use the installed module regardless of PATH.
**Warning signs:** `FileNotFoundError: [Errno 2] No such file or directory: 'yt-dlp'` in subprocess call.

### Pitfall 7: Segment overlay rendered over waveform instead of under it
**What goes wrong:** Segment bands obscure the waveform because they're drawn after the waveform in `render()`.
**Why it happens:** Render order in canvas is painter's algorithm — later draws go on top.
**How to avoid:** In `TimelineCanvas.render()`, draw segment bands between step 2 (track row banding) and step 3 (waveform). The `_drawSegmentBands()` call must come before the waveform loop.
**Warning signs:** Waveform is not visible in sections with colored overlays.

### Pitfall 8: AI Sync blocks don't have valid device_id
**What goes wrong:** Heuristic generates blocks but assigns incorrect or missing device IDs. The show engine ignores cues with unknown device IDs.
**Why it happens:** The sync engine may not know which devices are registered.
**How to avoid:** The `/api/sync/generate` endpoint must accept the current device list as part of its input, or fetch it from `registry`. Generate one block set per device, using the actual `device.id` UUIDs.

---

## Code Examples

### yt-dlp subprocess (confirmed API)
```python
# Use sys.executable -m yt_dlp to avoid PATH issues in Docker
import sys
proc = await asyncio.create_subprocess_exec(
    sys.executable, '-m', 'yt_dlp',
    '--extract-audio', '--audio-format', 'mp3',
    '--audio-quality', '0',
    '--newline', '--no-playlist',
    '-o', str(SHOWS_DIR / 'yt_%(id)s.%(ext)s'),  # predictable filename
    url,
    stdout=asyncio.subprocess.PIPE,
    stderr=asyncio.subprocess.STDOUT,
)
```

### librosa.segment.agglomerative (verified signature in container)
```python
# Verified: librosa.segment.agglomerative(data, k, *, clusterer=None, axis=-1)
# librosa.segment.recurrence_matrix(data, *, k=None, width=1, metric='euclidean', sym=False, ...)
chroma = librosa.feature.chroma_cqt(y=y, sr=sr)
R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)
R_filtered = librosa.segment.path_enhance(R, 15)
bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
```

### SSE streaming via fetch ReadableStream (frontend)
```javascript
async function fetchWithProgress(url, body, onProgress, onDone, onError) {
    const res = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(body),
    });
    const reader = res.body.getReader();
    const decoder = new TextDecoder();
    let buf = '';
    while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buf += decoder.decode(value, { stream: true });
        const lines = buf.split('\n\n');
        buf = lines.pop();
        for (const chunk of lines) {
            if (chunk.startsWith('data: ')) {
                const msg = JSON.parse(chunk.slice(6));
                if (msg.type === 'progress') onProgress(msg.pct);
                if (msg.type === 'done') onDone(msg.path);
                if (msg.type === 'error') onError(msg.msg);
            }
        }
    }
}
```

### Terminal progress bar rendering (frontend)
```javascript
function renderProgressBar(pct, width = 20) {
    // DOWNLOADING ██████░░░░ 62%
    const filled = Math.round((pct / 100) * width);
    const empty = width - filled;
    return `DOWNLOADING ${'█'.repeat(filled)}${'░'.repeat(empty)} ${Math.round(pct)}%`;
}
```

### Anthropic LLM call (lazy instantiation pattern)
```python
import os

def get_anthropic_client():
    """Lazy client construction — only called when USE LLM is triggered."""
    key = os.environ.get('ANTHROPIC_API_KEY')
    if not key:
        raise ValueError('ANTHROPIC_API_KEY not configured')
    import anthropic
    return anthropic.Anthropic(api_key=key)

async def generate_llm_show(features: dict, devices: list) -> list[dict]:
    """Generate show blocks via LLM. Returns list of CueModel-compatible dicts."""
    client = await asyncio.to_thread(get_anthropic_client)
    # ... prompt construction and API call
```

---

## Environment Availability

| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| librosa | Feature extraction, segmentation | YES (in container) | 0.11.0 | — |
| numpy | Feature matrix ops | YES (in container) | installed with librosa | — |
| sklearn | `agglomerative` clustering (librosa dep) | YES (librosa pulls it) | installed with librosa | — |
| yt-dlp (Python pkg) | YouTube download | NO — needs pyproject.toml addition | 2026.3.17 available | — |
| anthropic SDK | LLM show generation | NO — needs pyproject.toml addition | 0.40+ available | Disable USE LLM toggle |
| ANTHROPIC_API_KEY env var | LLM show generation | NOT CONFIGURED in docker-compose.prod.yml | — | Graceful disable |
| ffmpeg | yt-dlp audio conversion to MP3 | YES (already in Dockerfile) | system ffmpeg | — |

**Missing dependencies with no fallback:**
- `yt-dlp` Python package — AUD-02 blocks without it. Add to `pyproject.toml` and rebuild Docker image.

**Missing dependencies with fallback:**
- `anthropic` SDK — LLM path is off by default. App works fully without it; just disable the `USE LLM` toggle when `ANTHROPIC_API_KEY` is absent or SDK not installed.
- `ANTHROPIC_API_KEY` — Same fallback: guard the UI toggle, return 503 from endpoint.

---

## State of the Art

| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| youtube-dl | yt-dlp | 2021 (youtube-dl DMCA takedown response) | yt-dlp is the active fork; youtube-dl is stale |
| librosa.beat.beat_track only | beat_track + onset_detect + chroma + RMS | Phase 6 | Full feature set for heuristic mapping |
| No segmentation | agglomerative clustering on chroma recurrence matrix | Phase 6 | Provides section-level context for AI Sync |

**Deprecated/outdated:**
- `youtube-dl`: superseded by yt-dlp; do not use
- `librosa.segment.subsegment`: fine-grained sub-segmentation within known sections; not what we need for top-level structure detection

---

## Open Questions

1. **LLM prompt token budget for a 5-minute song**
   - What we know: RMS at 1s resolution = ~300 values; beats ~120 timestamps; chroma summary per section = 8 × 12 = 96 values; total ~500 numbers as JSON
   - What's unclear: Whether Anthropic claude-3-haiku (cheapest, fastest) is adequate for structured JSON generation of this kind, or if claude-3-sonnet is needed for reliable output
   - Recommendation: Use `claude-3-haiku-20240307` with a response schema in the prompt. The payload is small; haiku handles structured JSON well at this scale.

2. **yt-dlp download time for a typical 3–5 minute song**
   - What we know: yt-dlp downloads at network speed; typical YouTube audio is 128–320kbps
   - What's unclear: Whether the progress bar SSE stream can stay open for the full download duration without Traefik/Authelia proxy timeout
   - Recommendation: Set `proxy_read_timeout` in Traefik if default timeout is <60s. Alternatively, accept that progress streaming may time out at proxy and use WebSocket instead (the WS connection is already established).

3. **librosa.segment.agglomerative n_segments optimal value**
   - What we know: Target is 4–12 sections; the algorithm accepts `k` directly
   - What's unclear: Whether a fixed k=8 produces too many or too few sections for short (<2 min) or long (>6 min) songs
   - Recommendation: Default k=8; add heuristic: `k = max(4, min(12, int(duration / 30)))` — roughly one section per 30 seconds.

---

## Sources

### Primary (HIGH confidence)
- librosa 0.11.0 installed in container — `librosa.segment`, `librosa.onset`, `librosa.feature` APIs verified by direct introspection in Docker container
- yt-dlp pip dry-run in container — version 2026.3.17 available
- Codebase direct read — `beats.py`, `audio.py`, `timeline.js`, `beats.js`, `show.py`, `main.py`, `index.html`, `style.css` all read directly

### Secondary (MEDIUM confidence)
- yt-dlp GitHub README (known from training data, verified version via pip registry) — `--newline`, `--extract-audio`, `--no-playlist` flag behavior
- FastAPI StreamingResponse + SSE pattern (standard FastAPI docs pattern, widely used)

### Tertiary (LOW confidence)
- Anthropic SDK lazy instantiation pattern — based on training data knowledge of anthropic Python SDK; version and exact error class names should be confirmed when adding to requirements

---

## Metadata

**Confidence breakdown:**
- Standard stack: HIGH — verified in container (librosa), pip registry (yt-dlp), pyproject.toml (existing deps)
- Architecture: HIGH — all patterns traced to verified existing code (`beats.py` → `features.py` extension, `timeline.js` render order, `audio.py` endpoint pattern)
- Pitfalls: HIGH — yt-dlp filename/PATH pitfalls verified via subprocess behavior; canvas render order verified in `timeline.js`; SSE POST limitation is a browser API constraint

**Research date:** 2026-04-07
**Valid until:** 2026-05-07 (librosa and FastAPI are stable; yt-dlp updates frequently but CLI flags are backward-compatible)