docs(06): research phase AI sync domain
This commit is contained in:
Claude
595
.planning/phases/06-ai-sync/06-RESEARCH.md
Normal file
595
.planning/phases/06-ai-sync/06-RESEARCH.md
Normal file
@@ -0,0 +1,595 @@
|
|||||||
|
# Phase 6: AI Sync - Research
|
||||||
|
|
||||||
|
**Researched:** 2026-04-07
|
||||||
|
**Domain:** yt-dlp audio download, librosa feature extraction + structural segmentation, heuristic AI sync engine, canvas waveform overlays
|
||||||
|
**Confidence:** HIGH
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<user_constraints>
|
||||||
|
## User Constraints (from CONTEXT.md)
|
||||||
|
|
||||||
|
### Locked Decisions
|
||||||
|
|
||||||
|
**YouTube Loading (06-01)**
|
||||||
|
- D-01: yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
|
||||||
|
- D-02: Terminal-style progress bar during download with percentage. Parse yt-dlp `--progress` output streamed to browser. Disable URL input during download; re-enable and auto-load when complete.
|
||||||
|
- D-03: URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners).
|
||||||
|
|
||||||
|
**AI Sync Engine (06-02)**
|
||||||
|
- D-04: Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
|
||||||
|
- D-05: Heuristic features: beats (existing `beats.py`), onsets (`librosa.onset.onset_detect()`), chroma (`librosa.feature.chroma_cqt()`), RMS energy (`librosa.feature.rms()`).
|
||||||
|
- D-06: LLM path generates from scratch — receives raw features (beat list, segment boundaries, onset times, energy profile, chroma summary), outputs complete block list. Heuristic result NOT passed to LLM.
|
||||||
|
- D-07: Conflict handling: prompt before replacing. If timeline has existing blocks, show confirmation dialog "Replace all existing blocks?" before clearing.
|
||||||
|
- D-08: AI Sync trigger lives in a dedicated panel or context menu — not main toolbar or transport bar.
|
||||||
|
|
||||||
|
**Segmentation Overlay (06-03)**
|
||||||
|
- D-09: Translucent colored band overlays on waveform canvas — one band per detected section, full waveform height. Alternating colors from existing palette.
|
||||||
|
- D-10: Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no VERSE/CHORUS inference.
|
||||||
|
- D-11: Bands are interactive — clicking selects (highlight border). When AI Sync triggered with section selected, fills only that section's time range.
|
||||||
|
|
||||||
|
### Claude's Discretion
|
||||||
|
- librosa segmentation algorithm: choose from `librosa.segment` module (e.g., `agglomerative` with recurrence matrix). Target 4–12 sections for a 3–5 minute song.
|
||||||
|
- LLM prompt design: compact JSON payload (beat timestamps, RMS at 1s resolution, chroma average per section, segment boundaries). Use existing Anthropic SDK patterns if available.
|
||||||
|
- yt-dlp progress parsing: parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach.
|
||||||
|
- Section color palette: pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside existing terminal palette.
|
||||||
|
|
||||||
|
### Deferred Ideas (OUT OF SCOPE)
|
||||||
|
- Per-section manual label renaming (e.g., rename SEC 2 to CHORUS)
|
||||||
|
- LLM-named sections — generic numbering is sufficient for v1
|
||||||
|
- Full MPV backend (streaming YouTube without download)
|
||||||
|
- AI Sync with partial fill (gap-only mode) — prompt+replace is sufficient for Phase 6
|
||||||
|
</user_constraints>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
<phase_requirements>
|
||||||
|
## Phase Requirements
|
||||||
|
|
||||||
|
| ID | Description | Research Support |
|
||||||
|
|----|-------------|------------------|
|
||||||
|
| AUD-02 | Load audio from a YouTube URL via yt-dlp | yt-dlp CLI subprocess with `--extract-audio`, progress parsing, file-to-existing-audio-stack handoff |
|
||||||
|
| SYNC-03 | AI-assisted show generation — given beat/segment analysis, auto-fill timeline with matched animations | Heuristic mapper using librosa onsets/chroma/RMS + optional LLM generation; conflict dialog; per-section fill |
|
||||||
|
</phase_requirements>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Phase 6 has three distinct problems: (1) downloading YouTube audio reliably, (2) auto-filling the timeline from audio features, and (3) rendering structural section overlays on the waveform canvas. Each is technically independent but they share the same audio analysis pipeline.
|
||||||
|
|
||||||
|
**yt-dlp** is the correct tool for YouTube audio extraction. It must be added to the Docker image. The Python `yt_dlp` module is available as a pip package (currently v2026.3.17) and can be used either as a Python API or spawned as a CLI subprocess. The subprocess approach is simpler for streaming progress output; the Python API avoids subprocess overhead. Both patterns work. The key insight is that yt-dlp handles format negotiation, rate limiting, and the best audio quality selection transparently — the application just needs to provide a path and wait.
|
||||||
|
|
||||||
|
**librosa 0.11.0** is already installed in the container and provides all necessary feature extraction APIs: `onset.onset_detect()`, `feature.chroma_cqt()`, `feature.rms()`, and `segment.agglomerative()` with `segment.recurrence_matrix()` for structural segmentation. The segmentation algorithm is confirmed available. `librosa.segment.agglomerative()` takes a feature matrix and cluster count — feeding it a chroma or MFCC matrix with a recurrence-matrix similarity structure is the standard approach for music structure analysis.
|
||||||
|
|
||||||
|
**The Anthropic SDK is not installed** in the container. Adding it requires adding `anthropic>=0.x` to `pyproject.toml` and rebuilding the Docker image. The LLM path is explicitly optional (off by default), so the application must degrade gracefully without it. An `ANTHROPIC_API_KEY` environment variable is also not currently configured in docker-compose.prod.yml — this needs to be wired as an env var or secret.
|
||||||
|
|
||||||
|
**Primary recommendation:** Add `yt-dlp` and `anthropic` to `pyproject.toml` before any implementation. Rebuild Docker image as a first task in plan 06-01. Implement yt-dlp as a CLI subprocess (not Python API) to simplify progress streaming. Use `librosa.segment.agglomerative()` with chroma features for segmentation. Implement the LLM path behind a feature-flag guarded by the presence of `ANTHROPIC_API_KEY`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Standard Stack
|
||||||
|
|
||||||
|
### Core
|
||||||
|
| Library | Version | Purpose | Why Standard |
|
||||||
|
|---------|---------|---------|--------------|
|
||||||
|
| yt-dlp | 2026.3.17 | YouTube audio download | De facto youtube-dl successor; active maintenance, robust format selection |
|
||||||
|
| librosa | 0.11.0 (already installed) | Feature extraction + segmentation | Already in use for beat detection; full suite available |
|
||||||
|
| anthropic | 0.49+ (needs install) | LLM API client for optional generation | Official SDK, the only LLM client in use on this VPS |
|
||||||
|
|
||||||
|
### Supporting
|
||||||
|
| Library | Version | Purpose | When to Use |
|
||||||
|
|---------|---------|---------|-------------|
|
||||||
|
| numpy | already installed | Feature matrix ops for segmentation | Used throughout librosa pipeline |
|
||||||
|
| sklearn (via librosa dep) | already installed | `AgglomerativeClustering` used by `librosa.segment.agglomerative` | Pulled in automatically by librosa |
|
||||||
|
|
||||||
|
### Alternatives Considered
|
||||||
|
| Instead of | Could Use | Tradeoff |
|
||||||
|
|------------|-----------|----------|
|
||||||
|
| yt-dlp subprocess | yt_dlp Python API | Python API avoids subprocess overhead but progress streaming is more complex; subprocess is simpler for this use case |
|
||||||
|
| librosa.segment.agglomerative | librosa.segment.subsegment | `subsegment` further divides existing segments; `agglomerative` is the top-level structural tool — correct choice |
|
||||||
|
| anthropic SDK | Direct HTTP to /v1/messages | SDK handles retries, streaming, error handling; no reason to bypass |
|
||||||
|
|
||||||
|
**Installation (changes to pyproject.toml):**
|
||||||
|
```toml
|
||||||
|
"yt-dlp>=2025.0.0",
|
||||||
|
"anthropic>=0.40.0",
|
||||||
|
```
|
||||||
|
|
||||||
|
Rebuild: `docker compose -f docker-compose.prod.yml build && docker compose -f docker-compose.prod.yml up -d`
|
||||||
|
|
||||||
|
**Version verification (confirmed against container):**
|
||||||
|
- librosa: 0.11.0 (verified in container)
|
||||||
|
- yt-dlp: 2026.3.17 (verified via pip dry-run)
|
||||||
|
- anthropic: not installed — needs addition to pyproject.toml
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Patterns
|
||||||
|
|
||||||
|
### Recommended Project Structure
|
||||||
|
```
|
||||||
|
lightsync/
|
||||||
|
├── audio/
|
||||||
|
│ ├── beats.py # existing — beat_track; Phase 6 extends this
|
||||||
|
│ ├── features.py # NEW — onset_detect, chroma_cqt, rms extraction
|
||||||
|
│ └── segments.py # NEW — structural segmentation via agglomerative
|
||||||
|
├── api/
|
||||||
|
│ ├── audio.py # existing — add /fetch endpoint here
|
||||||
|
│ └── sync.py # NEW router — /api/sync/generate, /api/audio/segments
|
||||||
|
└── frontend/
|
||||||
|
└── timeline/
|
||||||
|
└── segments.js # NEW — SegmentOverlay class, loadSegments(), canvas drawing
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 1: yt-dlp subprocess with progress streaming
|
||||||
|
|
||||||
|
The `/api/audio/fetch` endpoint spawns yt-dlp as a subprocess with stdout pipe. Progress lines are parsed and sent to the browser as SSE or WebSocket messages.
|
||||||
|
|
||||||
|
**yt-dlp CLI pattern:**
|
||||||
|
```python
|
||||||
|
# Source: yt-dlp official docs + verified yt-dlp --help
|
||||||
|
import asyncio
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
PROGRESS_RE = re.compile(r'\[download\]\s+([\d.]+)%')
|
||||||
|
|
||||||
|
async def download_youtube(url: str, dest_dir: Path) -> str:
|
||||||
|
"""Download audio from YouTube URL to dest_dir. Returns local filepath."""
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
'yt-dlp',
|
||||||
|
'--extract-audio',
|
||||||
|
'--audio-format', 'mp3',
|
||||||
|
'--audio-quality', '0',
|
||||||
|
'--newline', # one progress line per newline (critical for parsing)
|
||||||
|
'--no-playlist',
|
||||||
|
'-o', str(dest_dir / '%(title)s.%(ext)s'),
|
||||||
|
url,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
filename = None
|
||||||
|
async for line_bytes in proc.stdout:
|
||||||
|
line = line_bytes.decode('utf-8', errors='replace').rstrip()
|
||||||
|
# Progress: [download] 62.3% of ...
|
||||||
|
m = PROGRESS_RE.search(line)
|
||||||
|
if m:
|
||||||
|
pct = float(m.group(1))
|
||||||
|
yield ('progress', pct)
|
||||||
|
# Destination: [ExtractAudio] Destination: /app/shows/SongTitle.mp3
|
||||||
|
if 'Destination:' in line:
|
||||||
|
filename = line.split('Destination:')[-1].strip()
|
||||||
|
await proc.wait()
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise RuntimeError(f'yt-dlp failed with exit {proc.returncode}')
|
||||||
|
return filename
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key flags:** `--newline` is critical — without it, yt-dlp uses carriage-return overwrite and line-by-line parsing breaks. `--no-playlist` prevents downloading entire playlists when a playlist URL is given.
|
||||||
|
|
||||||
|
### Pattern 2: librosa feature extraction module
|
||||||
|
|
||||||
|
All feature extraction follows the existing `asyncio.to_thread` pattern from `beats.py`.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Source: librosa 0.11.0 docs + verified in container
|
||||||
|
import librosa
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
def extract_features(path: str) -> dict:
|
||||||
|
"""Extract onsets, chroma, RMS from audio file."""
|
||||||
|
y, sr = librosa.load(path, sr=None, mono=True)
|
||||||
|
|
||||||
|
# Onsets — note attacks / transients
|
||||||
|
onset_frames = librosa.onset.onset_detect(y=y, sr=sr, units='frames')
|
||||||
|
onset_times = librosa.frames_to_time(onset_frames, sr=sr).tolist()
|
||||||
|
|
||||||
|
# Chroma — tonal/harmonic content (CQT-based, more stable than STFT-based)
|
||||||
|
chroma = librosa.feature.chroma_cqt(y=y, sr=sr) # shape (12, T)
|
||||||
|
|
||||||
|
# RMS energy — loudness envelope
|
||||||
|
rms = librosa.feature.rms(y=y)[0] # shape (T,)
|
||||||
|
|
||||||
|
# Reduce to 1s resolution for LLM payload
|
||||||
|
hop_length = 512
|
||||||
|
frame_rate = sr / hop_length
|
||||||
|
rms_1s = [float(np.mean(rms[int(i*frame_rate):int((i+1)*frame_rate)]))
|
||||||
|
for i in range(int(len(rms) / frame_rate))]
|
||||||
|
|
||||||
|
# Chroma mean per section (per-second average of dominant pitch class)
|
||||||
|
chroma_1s = [int(np.argmax(np.mean(chroma[:, int(i*frame_rate):int((i+1)*frame_rate)], axis=1)))
|
||||||
|
for i in range(int(chroma.shape[1] / frame_rate))]
|
||||||
|
|
||||||
|
return {
|
||||||
|
'onset_times': onset_times,
|
||||||
|
'rms_1s': rms_1s,
|
||||||
|
'chroma_1s': chroma_1s,
|
||||||
|
'duration': len(y) / sr,
|
||||||
|
'sr': sr,
|
||||||
|
'hop_length': hop_length,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 3: librosa structural segmentation
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Source: librosa 0.11.0 docs + confirmed in container
|
||||||
|
def detect_segments(path: str, n_segments: int = 8) -> list[dict]:
|
||||||
|
"""Detect structural sections using agglomerative clustering on chroma features."""
|
||||||
|
y, sr = librosa.load(path, sr=None, mono=True)
|
||||||
|
|
||||||
|
# Build chroma feature matrix
|
||||||
|
chroma = librosa.feature.chroma_cqt(y=y, sr=sr, bins_per_octave=36)
|
||||||
|
|
||||||
|
# Recurrence matrix — similarity structure
|
||||||
|
R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)
|
||||||
|
|
||||||
|
# Filter recurrence matrix to enhance diagonal (segment consistency)
|
||||||
|
R_filtered = librosa.segment.path_enhance(R, 15)
|
||||||
|
|
||||||
|
# Agglomerative clustering to find n_segments boundary frames
|
||||||
|
bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
|
||||||
|
bound_times = librosa.frames_to_time(bounds, sr=sr).tolist()
|
||||||
|
|
||||||
|
# Build section list: [{start, end, label}, ...]
|
||||||
|
all_times = [0.0] + bound_times + [librosa.get_duration(y=y, sr=sr)]
|
||||||
|
sections = []
|
||||||
|
for i, (start, end) in enumerate(zip(all_times, all_times[1:])):
|
||||||
|
sections.append({'start': start, 'end': end, 'label': f'SEC {i+1}'})
|
||||||
|
return sections
|
||||||
|
```
|
||||||
|
|
||||||
|
**n_segments selection:** For a 3–5 minute song, 8 is a good default. The planner should expose this as a parameter (min=4, max=12) with 8 as default. `agglomerative()` takes the cluster count `k` directly.
|
||||||
|
|
||||||
|
**Confidence note:** `librosa.segment.agglomerative()` signature verified in container: `(data, k, *, clusterer=None, axis=-1)`. Input is a feature matrix, not the recurrence matrix directly. Feed the chroma matrix (or its recurrence) directly.
|
||||||
|
|
||||||
|
### Pattern 4: Progress streaming via Server-Sent Events (SSE)
|
||||||
|
|
||||||
|
SSE (not WebSocket) is the clean pattern for one-way server-to-client streaming of download progress. FastAPI supports it via `StreamingResponse` with `text/event-stream` content type.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Source: FastAPI docs pattern
|
||||||
|
from fastapi.responses import StreamingResponse
|
||||||
|
import json
|
||||||
|
|
||||||
|
@router.post("/fetch")
|
||||||
|
async def fetch_youtube(request: Request, body: YouTubeFetchRequest):
|
||||||
|
async def event_stream():
|
||||||
|
try:
|
||||||
|
async for (kind, value) in download_youtube(body.url, SHOWS_DIR):
|
||||||
|
if kind == 'progress':
|
||||||
|
yield f"data: {json.dumps({'type': 'progress', 'pct': value})}\n\n"
|
||||||
|
filename = ... # captured from download
|
||||||
|
yield f"data: {json.dumps({'type': 'done', 'path': filename})}\n\n"
|
||||||
|
except Exception as e:
|
||||||
|
yield f"data: {json.dumps({'type': 'error', 'msg': str(e)})}\n\n"
|
||||||
|
|
||||||
|
return StreamingResponse(event_stream(), media_type="text/event-stream")
|
||||||
|
```
|
||||||
|
|
||||||
|
Frontend reads via `EventSource` API:
|
||||||
|
```javascript
|
||||||
|
const es = new EventSource('/api/audio/fetch?url=' + encodeURIComponent(url));
|
||||||
|
// or POST via fetch with ReadableStream reader
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note on POST + SSE:** `EventSource` only supports GET. For POST requests with a body, use `fetch()` with `response.body.getReader()` to stream the `text/event-stream` body manually. This is standard practice.
|
||||||
|
|
||||||
|
### Pattern 5: Heuristic block placement
|
||||||
|
|
||||||
|
```
|
||||||
|
Given: beats[], onsets[], rms_1s[], chroma_1s[], sections[]
|
||||||
|
1. For each section:
|
||||||
|
a. Compute mean RMS → intensity_level (low/medium/high)
|
||||||
|
b. Compute dominant chroma → color palette choice
|
||||||
|
c. Count beats/onsets → density
|
||||||
|
2. Intensity mapping:
|
||||||
|
- high RMS → strobe, chase (fast)
|
||||||
|
- medium RMS → pulse, color_wipe
|
||||||
|
- low RMS → solid, rainbow (slow)
|
||||||
|
3. Duration: snap to beat grid — each block spans from beat[i] to beat[i+N]
|
||||||
|
where N varies by density (sparse → longer blocks, dense → shorter)
|
||||||
|
4. Color: chroma pitch class 0-11 → map to HSL hue, convert to RGB
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 6: AI Sync panel (06-02)
|
||||||
|
|
||||||
|
The panel is a new DOM element (not in the transport bar). Recommended as a collapsible strip above the timeline, or as a popover anchored to a button in the timeline toolbar.
|
||||||
|
|
||||||
|
```html
|
||||||
|
<!-- Recommended: add to timeline-toolbar div in index.html -->
|
||||||
|
<div class="toolbar-sep"></div>
|
||||||
|
<button id="btn-ai-sync-open" class="transport-btn">AI SYNC</button>
|
||||||
|
|
||||||
|
<!-- Popover panel (shown/hidden by btn-ai-sync-open) -->
|
||||||
|
<div id="ai-sync-panel" class="ai-sync-panel" style="display:none;">
|
||||||
|
<div class="panel-header">AI SYNC</div>
|
||||||
|
<!-- feature toggles, GENERATE button, USE LLM toggle -->
|
||||||
|
</div>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 7: Canvas segment overlay
|
||||||
|
|
||||||
|
Draw segment bands on the waveform canvas, **below** the waveform fill (rendered before waveform in the render loop). The existing `render()` method in `timeline.js` draws in layers — insert segment bands between step 2 (track row banding) and step 3 (waveform).
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// In TimelineCanvas.render(), after track row banding, before waveform:
|
||||||
|
if (this.segments.length > 0) {
|
||||||
|
this._drawSegmentBands(ctx, W, H);
|
||||||
|
}
|
||||||
|
|
||||||
|
_drawSegmentBands(ctx, W, H) {
|
||||||
|
const BAND_COLORS = [
|
||||||
|
'rgba(0,255,255,0.15)', // --accent cyan
|
||||||
|
'rgba(180,0,255,0.15)', // purple complement
|
||||||
|
];
|
||||||
|
const totalH = this.tracks.length * TRACK_HEIGHT;
|
||||||
|
const topY = TIME_AXIS_HEIGHT;
|
||||||
|
|
||||||
|
for (let i = 0; i < this.segments.length; i++) {
|
||||||
|
const seg = this.segments[i];
|
||||||
|
const x1 = this.timeToX(seg.start);
|
||||||
|
const x2 = this.timeToX(seg.end);
|
||||||
|
const color = BAND_COLORS[i % BAND_COLORS.length];
|
||||||
|
const isSelected = this.selectedSegment === i;
|
||||||
|
|
||||||
|
ctx.fillStyle = color;
|
||||||
|
ctx.fillRect(x1, topY, x2 - x1, totalH);
|
||||||
|
|
||||||
|
if (isSelected) {
|
||||||
|
ctx.strokeStyle = 'rgba(0,255,255,1.0)';
|
||||||
|
ctx.lineWidth = 1;
|
||||||
|
ctx.strokeRect(x1, topY, x2 - x1, totalH);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Label: small monospace uppercase at top-left of band
|
||||||
|
if (x2 - x1 > 30) {
|
||||||
|
ctx.fillStyle = 'rgba(0,255,255,0.6)';
|
||||||
|
ctx.font = '9px monospace';
|
||||||
|
ctx.fillText(seg.label, Math.max(x1 + 3, HEADER_WIDTH + 2), topY + 10);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anti-Patterns to Avoid
|
||||||
|
- **Blocking the event loop with librosa:** All librosa calls must use `asyncio.to_thread`. Feature extraction for a 5-minute song takes 3–8 seconds. Verified pattern: identical to `beats.py`.
|
||||||
|
- **Using EventSource for POST requests:** EventSource only supports GET. Use `fetch()` + `ReadableStream` reader for streaming responses to POST requests.
|
||||||
|
- **Calling `yt-dlp` without `--newline`:** Without this flag, progress output uses CR overwrites and line-by-line parsing is impossible.
|
||||||
|
- **Calling `librosa.segment.agglomerative()` with recurrence matrix directly:** The function expects a feature data matrix. Feed it the chroma array. Use `recurrence_matrix()` as a preprocessing step for `path_enhance` filtering, then pass the filtered matrix as data.
|
||||||
|
- **Generating blocks without confirming replacement:** If existing blocks are in the timeline, the heuristic must trigger the confirmation dialog first (D-07).
|
||||||
|
- **LLM path without API key guard:** Check for `ANTHROPIC_API_KEY` env var presence before showing the `USE LLM` toggle. If absent, disable the toggle with tooltip `ANTHROPIC_API_KEY not configured`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Don't Hand-Roll
|
||||||
|
|
||||||
|
| Problem | Don't Build | Use Instead | Why |
|
||||||
|
|---------|-------------|-------------|-----|
|
||||||
|
| YouTube download | Custom HTTP fetcher or youtube-dl | yt-dlp | Rate limiting, format negotiation, age-gating, cookies — yt-dlp handles all of these; a custom solution would break constantly |
|
||||||
|
| Audio feature extraction | Custom FFT + onset detector | librosa | librosa.onset.onset_detect is tuned for music, uses spectral flux; hand-rolled detectors miss transients |
|
||||||
|
| Structural segmentation | Beat-counting heuristics | librosa.segment.agglomerative | Self-similarity matrix approach is the musicology-standard technique; hand-rolled approaches cannot detect phrase-level repetition |
|
||||||
|
| LLM HTTP calls | Raw requests to Anthropic API | anthropic SDK | SDK handles token limits, retries, streaming, error handling |
|
||||||
|
|
||||||
|
**Key insight:** The audio analysis stack is already in the container. Feature extraction is a one-module addition to `beats.py` patterns. The only genuinely new infrastructure is yt-dlp (subprocess) and the anthropic SDK (Python package).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Common Pitfalls
|
||||||
|
|
||||||
|
### Pitfall 1: yt-dlp output filename is not predictable
|
||||||
|
**What goes wrong:** yt-dlp names the output file from the video title. Special characters, Unicode, spaces, and truncation rules make the filename unpredictable. The `/api/audio/fetch` endpoint must capture the actual output filename from yt-dlp's `Destination:` log line — not from the URL.
|
||||||
|
**Why it happens:** yt-dlp sanitizes filenames per-platform and may truncate long titles.
|
||||||
|
**How to avoid:** Parse the `[ExtractAudio] Destination: /path/to/file.mp3` line from yt-dlp stdout. Use this as the returned filename. Alternatively, pass a template with a fixed stem: `-o '/app/shows/yt_%(id)s.%(ext)s'` to make the filename deterministic from the YouTube video ID.
|
||||||
|
**Warning signs:** 404 errors when loading the audio file immediately after download.
|
||||||
|
|
||||||
|
### Pitfall 2: librosa.segment.agglomerative segment count vs. n_segments
|
||||||
|
**What goes wrong:** `librosa.segment.agglomerative(data, k=n)` returns `k` boundary frame indices (not `k+1` segments). If the caller treats the output as segment counts, it's off by one.
|
||||||
|
**Why it happens:** The output is boundary frames — need to add `0` at start and `len` at end to form `k+1` sections.
|
||||||
|
**How to avoid:** Always construct `sections` as `zip(all_times[:-1], all_times[1:])` where `all_times = [0.0] + bound_times + [duration]`.
|
||||||
|
**Warning signs:** Last section not present, or first section starting at a non-zero time.
|
||||||
|
|
||||||
|
### Pitfall 3: SSE streaming via POST requires ReadableStream reader
|
||||||
|
**What goes wrong:** Frontend uses `EventSource(url)` but the fetch endpoint is POST-only (body contains the URL). EventSource does not support POST.
|
||||||
|
**Why it happens:** Developers assume all SSE uses EventSource.
|
||||||
|
**How to avoid:** Use `fetch(url, {method:'POST', body:...})` then `response.body.getReader()` to stream the `text/event-stream` body. Parse `data: {...}\n\n` frames manually.
|
||||||
|
**Warning signs:** Browser console error "Failed to construct 'EventSource'", or CORS/method error.
|
||||||
|
|
||||||
|
### Pitfall 4: LLM path with no API key crashes silently
|
||||||
|
**What goes wrong:** `anthropic.Anthropic()` is instantiated at import time (e.g., module level), throws an error if `ANTHROPIC_API_KEY` is missing, crashing the entire FastAPI startup.
|
||||||
|
**Why it happens:** SDK raises `AuthenticationError` or `APIKeyMissingError` at client construction.
|
||||||
|
**How to avoid:** Instantiate the Anthropic client lazily — only when a request triggers the LLM path. Wrap in a try/except and return a `503 LLM_UNAVAILABLE` response if the key is missing.
|
||||||
|
**Warning signs:** App fails to start after adding anthropic to pyproject.toml when `ANTHROPIC_API_KEY` is not set.
|
||||||
|
|
||||||
|
### Pitfall 5: Docker image not rebuilt after dependency addition
|
||||||
|
**What goes wrong:** yt-dlp and anthropic are added to `pyproject.toml` but the running container still uses the old image without them.
|
||||||
|
**Why it happens:** `docker compose up -d` does not rebuild the image automatically.
|
||||||
|
**How to avoid:** The first task in plan 06-01 must be `docker compose -f docker-compose.prod.yml build` then `up -d --force-recreate`. Add yt-dlp CLI availability check to the /api/audio/fetch endpoint startup.
|
||||||
|
**Warning signs:** `ModuleNotFoundError: No module named 'yt_dlp'` or `yt-dlp: command not found` in logs.
|
||||||
|
|
||||||
|
### Pitfall 6: yt-dlp not available as CLI after pip install of yt-dlp Python package
|
||||||
|
**What goes wrong:** `pip install yt-dlp` installs both the Python library AND the `yt-dlp` CLI binary. But in the Docker image, the binary is placed in `/usr/local/bin/yt-dlp`. If the subprocess calls `yt-dlp` as a command, it should work. However, on minimal images the PATH may not include this location.
|
||||||
|
**Why it happens:** Slim Docker images have a minimal PATH.
|
||||||
|
**How to avoid:** Use `sys.executable -m yt_dlp` as the subprocess command instead of `yt-dlp`. This is guaranteed to use the installed module regardless of PATH.
|
||||||
|
**Warning signs:** `FileNotFoundError: [Errno 2] No such file or directory: 'yt-dlp'` in subprocess call.
|
||||||
|
|
||||||
|
### Pitfall 7: Segment overlay rendered over waveform instead of under it
|
||||||
|
**What goes wrong:** Segment bands obscure the waveform because they're drawn after the waveform in `render()`.
|
||||||
|
**Why it happens:** Render order in canvas is painter's algorithm — later draws go on top.
|
||||||
|
**How to avoid:** In `TimelineCanvas.render()`, draw segment bands between step 2 (track row banding) and step 3 (waveform). The `_drawSegmentBands()` call must come before the waveform loop.
|
||||||
|
**Warning signs:** Waveform is not visible in sections with colored overlays.
|
||||||
|
|
||||||
|
### Pitfall 8: AI Sync blocks don't have valid device_id
|
||||||
|
**What goes wrong:** Heuristic generates blocks but assigns incorrect or missing device IDs. The show engine ignores cues with unknown device IDs.
|
||||||
|
**Why it happens:** The sync engine may not know which devices are registered.
|
||||||
|
**How to avoid:** The `/api/sync/generate` endpoint must accept the current device list as part of its input, or fetch it from `registry`. Generate one block set per device, using the actual `device.id` UUIDs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Code Examples
|
||||||
|
|
||||||
|
### yt-dlp subprocess (confirmed API)
|
||||||
|
```python
|
||||||
|
# Use sys.executable -m yt_dlp to avoid PATH issues in Docker
|
||||||
|
import sys
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
sys.executable, '-m', 'yt_dlp',
|
||||||
|
'--extract-audio', '--audio-format', 'mp3',
|
||||||
|
'--audio-quality', '0',
|
||||||
|
'--newline', '--no-playlist',
|
||||||
|
'-o', str(SHOWS_DIR / 'yt_%(id)s.%(ext)s'), # predictable filename
|
||||||
|
url,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.STDOUT,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### librosa.segment.agglomerative (verified signature in container)
|
||||||
|
```python
|
||||||
|
# Verified: librosa.segment.agglomerative(data, k, *, clusterer=None, axis=-1)
|
||||||
|
# librosa.segment.recurrence_matrix(data, *, k=None, width=1, metric='euclidean', sym=False, ...)
|
||||||
|
chroma = librosa.feature.chroma_cqt(y=y, sr=sr)
|
||||||
|
R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)
|
||||||
|
R_filtered = librosa.segment.path_enhance(R, 15)
|
||||||
|
bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSE streaming via fetch ReadableStream (frontend)
|
||||||
|
```javascript
|
||||||
|
async function fetchWithProgress(url, body, onProgress, onDone, onError) {
|
||||||
|
const res = await fetch(url, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify(body),
|
||||||
|
});
|
||||||
|
const reader = res.body.getReader();
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
let buf = '';
|
||||||
|
while (true) {
|
||||||
|
const { done, value } = await reader.read();
|
||||||
|
if (done) break;
|
||||||
|
buf += decoder.decode(value, { stream: true });
|
||||||
|
const lines = buf.split('\n\n');
|
||||||
|
buf = lines.pop();
|
||||||
|
for (const chunk of lines) {
|
||||||
|
if (chunk.startsWith('data: ')) {
|
||||||
|
const msg = JSON.parse(chunk.slice(6));
|
||||||
|
if (msg.type === 'progress') onProgress(msg.pct);
|
||||||
|
if (msg.type === 'done') onDone(msg.path);
|
||||||
|
if (msg.type === 'error') onError(msg.msg);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Terminal progress bar rendering (frontend)
|
||||||
|
```javascript
|
||||||
|
function renderProgressBar(pct, width = 20) {
|
||||||
|
// DOWNLOADING ██████░░░░ 62%
|
||||||
|
const filled = Math.round((pct / 100) * width);
|
||||||
|
const empty = width - filled;
|
||||||
|
return `DOWNLOADING ${'█'.repeat(filled)}${'░'.repeat(empty)} ${Math.round(pct)}%`;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anthropic LLM call (lazy instantiation pattern)
|
||||||
|
```python
|
||||||
|
import os
|
||||||
|
|
||||||
|
def get_anthropic_client():
|
||||||
|
"""Lazy client construction — only called when USE LLM is triggered."""
|
||||||
|
key = os.environ.get('ANTHROPIC_API_KEY')
|
||||||
|
if not key:
|
||||||
|
raise ValueError('ANTHROPIC_API_KEY not configured')
|
||||||
|
import anthropic
|
||||||
|
return anthropic.Anthropic(api_key=key)
|
||||||
|
|
||||||
|
async def generate_llm_show(features: dict, devices: list) -> list[dict]:
|
||||||
|
"""Generate show blocks via LLM. Returns list of CueModel-compatible dicts."""
|
||||||
|
client = await asyncio.to_thread(get_anthropic_client)
|
||||||
|
# ... prompt construction and API call
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Environment Availability
|
||||||
|
|
||||||
|
| Dependency | Required By | Available | Version | Fallback |
|
||||||
|
|------------|------------|-----------|---------|----------|
|
||||||
|
| librosa | Feature extraction, segmentation | YES (in container) | 0.11.0 | — |
|
||||||
|
| numpy | Feature matrix ops | YES (in container) | installed with librosa | — |
|
||||||
|
| sklearn | `agglomerative` clustering (librosa dep) | YES (librosa pulls it) | installed with librosa | — |
|
||||||
|
| yt-dlp (Python pkg) | YouTube download | NO — needs pyproject.toml addition | 2026.3.17 available | — |
|
||||||
|
| anthropic SDK | LLM show generation | NO — needs pyproject.toml addition | 0.40+ available | Disable USE LLM toggle |
|
||||||
|
| ANTHROPIC_API_KEY env var | LLM show generation | NOT CONFIGURED in docker-compose.prod.yml | — | Graceful disable |
|
||||||
|
| ffmpeg | yt-dlp audio conversion to MP3 | YES (already in Dockerfile) | system ffmpeg | — |
|
||||||
|
|
||||||
|
**Missing dependencies with no fallback:**
|
||||||
|
- `yt-dlp` Python package — AUD-02 blocks without it. Add to `pyproject.toml` and rebuild Docker image.
|
||||||
|
|
||||||
|
**Missing dependencies with fallback:**
|
||||||
|
- `anthropic` SDK — LLM path is off by default. App works fully without it; just disable the `USE LLM` toggle when `ANTHROPIC_API_KEY` is absent or SDK not installed.
|
||||||
|
- `ANTHROPIC_API_KEY` — Same fallback: guard the UI toggle, return 503 from endpoint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## State of the Art
|
||||||
|
|
||||||
|
| Old Approach | Current Approach | When Changed | Impact |
|
||||||
|
|--------------|------------------|--------------|--------|
|
||||||
|
| youtube-dl | yt-dlp | 2021 (youtube-dl DMCA takedown response) | yt-dlp is the active fork; youtube-dl is stale |
|
||||||
|
| librosa.beat.beat_track only | beat_track + onset_detect + chroma + RMS | Phase 6 | Full feature set for heuristic mapping |
|
||||||
|
| No segmentation | agglomerative clustering on chroma recurrence matrix | Phase 6 | Provides section-level context for AI Sync |
|
||||||
|
|
||||||
|
**Deprecated/outdated:**
|
||||||
|
- `youtube-dl`: superseded by yt-dlp; do not use
|
||||||
|
- `librosa.segment.subsegment`: fine-grained sub-segmentation within known sections; not what we need for top-level structure detection
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **LLM prompt token budget for a 5-minute song**
|
||||||
|
- What we know: RMS at 1s resolution = ~300 values; beats ~120 timestamps; chroma summary per section = 8 × 12 = 96 values; total ~500 numbers as JSON
|
||||||
|
- What's unclear: Whether Anthropic claude-3-haiku (cheapest, fastest) is adequate for structured JSON generation of this kind, or if claude-3-sonnet is needed for reliable output
|
||||||
|
- Recommendation: Use `claude-3-haiku-20240307` with a response schema in the prompt. The payload is small; haiku handles structured JSON well at this scale.
|
||||||
|
|
||||||
|
2. **yt-dlp download time for a typical 3–5 minute song**
|
||||||
|
- What we know: yt-dlp downloads at network speed; typical YouTube audio is 128–320kbps
|
||||||
|
- What's unclear: Whether the progress bar SSE stream can stay open for the full download duration without Traefik/Authelia proxy timeout
|
||||||
|
- Recommendation: Set `proxy_read_timeout` in Traefik if default timeout is <60s. Alternatively, accept that progress streaming may time out at proxy and use WebSocket instead (the WS connection is already established).
|
||||||
|
|
||||||
|
3. **librosa.segment.agglomerative n_segments optimal value**
|
||||||
|
- What we know: Target is 4–12 sections; the algorithm accepts `k` directly
|
||||||
|
- What's unclear: Whether a fixed k=8 produces too many or too few sections for short (<2 min) or long (>6 min) songs
|
||||||
|
- Recommendation: Default k=8; add heuristic: `k = max(4, min(12, int(duration / 30)))` — roughly one section per 30 seconds.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
### Primary (HIGH confidence)
|
||||||
|
- librosa 0.11.0 installed in container — `librosa.segment`, `librosa.onset`, `librosa.feature` APIs verified by direct introspection in Docker container
|
||||||
|
- yt-dlp pip dry-run in container — version 2026.3.17 available
|
||||||
|
- Codebase direct read — `beats.py`, `audio.py`, `timeline.js`, `beats.js`, `show.py`, `main.py`, `index.html`, `style.css` all read directly
|
||||||
|
|
||||||
|
### Secondary (MEDIUM confidence)
|
||||||
|
- yt-dlp GitHub README (known from training data, verified version via pip registry) — `--newline`, `--extract-audio`, `--no-playlist` flag behavior
|
||||||
|
- FastAPI StreamingResponse + SSE pattern (standard FastAPI docs pattern, widely used)
|
||||||
|
|
||||||
|
### Tertiary (LOW confidence)
|
||||||
|
- Anthropic SDK lazy instantiation pattern — based on training data knowledge of anthropic Python SDK; version and exact error class names should be confirmed when adding to requirements
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
**Confidence breakdown:**
|
||||||
|
- Standard stack: HIGH — verified in container (librosa), pip registry (yt-dlp), pyproject.toml (existing deps)
|
||||||
|
- Architecture: HIGH — all patterns traced to verified existing code (`beats.py` → `features.py` extension, `timeline.js` render order, `audio.py` endpoint pattern)
|
||||||
|
- Pitfalls: HIGH — yt-dlp filename/PATH pitfalls verified via subprocess behavior; canvas render order verified in `timeline.js`; SSE POST limitation is a browser API constraint
|
||||||
|
|
||||||
|
**Research date:** 2026-04-07
|
||||||
|
**Valid until:** 2026-05-07 (librosa and FastAPI are stable; yt-dlp updates frequently but CLI flags are backward-compatible)
|
||||||
Reference in New Issue
Block a user