Gitea-Admin/led2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b88bd98c4007adb0ab2f12f0b3403cc9ea56bb42
led2/.planning/phases/06-ai-sync/06-RESEARCH.md
Claude b88bd98c40 docs(06): research phase AI sync domain
2026-04-07 11:17:32 +00:00

32 KiB
Raw Blame History

Phase 6: AI Sync - Research

Researched: 2026-04-07 Domain: yt-dlp audio download, librosa feature extraction + structural segmentation, heuristic AI sync engine, canvas waveform overlays Confidence: HIGH

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

YouTube Loading (06-01)

  • D-01: yt-dlp downloads audio to /app/shows/ — same directory as uploaded files. After download, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
  • D-02: Terminal-style progress bar during download with percentage. Parse yt-dlp --progress output streamed to browser. Disable URL input during download; re-enable and auto-load when complete.
  • D-03: URL input lives in the transport bar — inline with the existing file selector controls. Text input + FETCH button, same row, terminal aesthetic (flat, monospace, no rounded corners).

AI Sync Engine (06-02)

  • D-04: Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
  • D-05: Heuristic features: beats (existing beats.py), onsets (librosa.onset.onset_detect()), chroma (librosa.feature.chroma_cqt()), RMS energy (librosa.feature.rms()).
  • D-06: LLM path generates from scratch — receives raw features (beat list, segment boundaries, onset times, energy profile, chroma summary), outputs complete block list. Heuristic result NOT passed to LLM.
  • D-07: Conflict handling: prompt before replacing. If timeline has existing blocks, show confirmation dialog "Replace all existing blocks?" before clearing.
  • D-08: AI Sync trigger lives in a dedicated panel or context menu — not main toolbar or transport bar.

Segmentation Overlay (06-03)

  • D-09: Translucent colored band overlays on waveform canvas — one band per detected section, full waveform height. Alternating colors from existing palette.
  • D-10: Section labels are generic numbered: SEC 1, SEC 2, SEC 3, ... — no VERSE/CHORUS inference.
  • D-11: Bands are interactive — clicking selects (highlight border). When AI Sync triggered with section selected, fills only that section's time range.

Claude's Discretion

  • librosa segmentation algorithm: choose from librosa.segment module (e.g., agglomerative with recurrence matrix). Target 4–12 sections for a 3–5 minute song.
  • LLM prompt design: compact JSON payload (beat timestamps, RMS at 1s resolution, chroma average per section, segment boundaries). Use existing Anthropic SDK patterns if available.
  • yt-dlp progress parsing: parse --progress output lines via subprocess pipe. Planner decides exact regex/parsing approach.
  • Section color palette: pick 2–3 alternating colors from existing CSS variables (--accent, complementary hues). No new colors outside existing terminal palette.

Deferred Ideas (OUT OF SCOPE)

  • Per-section manual label renaming (e.g., rename SEC 2 to CHORUS)
  • LLM-named sections — generic numbering is sufficient for v1
  • Full MPV backend (streaming YouTube without download)
  • AI Sync with partial fill (gap-only mode) — prompt+replace is sufficient for Phase 6 </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
AUD-02 Load audio from a YouTube URL via yt-dlp yt-dlp CLI subprocess with --extract-audio, progress parsing, file-to-existing-audio-stack handoff
SYNC-03 AI-assisted show generation — given beat/segment analysis, auto-fill timeline with matched animations Heuristic mapper using librosa onsets/chroma/RMS + optional LLM generation; conflict dialog; per-section fill
</phase_requirements>

Summary

Phase 6 has three distinct problems: (1) downloading YouTube audio reliably, (2) auto-filling the timeline from audio features, and (3) rendering structural section overlays on the waveform canvas. Each is technically independent but they share the same audio analysis pipeline.

yt-dlp is the correct tool for YouTube audio extraction. It must be added to the Docker image. The Python yt_dlp module is available as a pip package (currently v2026.3.17) and can be used either as a Python API or spawned as a CLI subprocess. The subprocess approach is simpler for streaming progress output; the Python API avoids subprocess overhead. Both patterns work. The key insight is that yt-dlp handles format negotiation, rate limiting, and the best audio quality selection transparently — the application just needs to provide a path and wait.

librosa 0.11.0 is already installed in the container and provides all necessary feature extraction APIs: onset.onset_detect(), feature.chroma_cqt(), feature.rms(), and segment.agglomerative() with segment.recurrence_matrix() for structural segmentation. The segmentation algorithm is confirmed available. librosa.segment.agglomerative() takes a feature matrix and cluster count — feeding it a chroma or MFCC matrix with a recurrence-matrix similarity structure is the standard approach for music structure analysis.

The Anthropic SDK is not installed in the container. Adding it requires adding anthropic>=0.x to pyproject.toml and rebuilding the Docker image. The LLM path is explicitly optional (off by default), so the application must degrade gracefully without it. An ANTHROPIC_API_KEY environment variable is also not currently configured in docker-compose.prod.yml — this needs to be wired as an env var or secret.

Primary recommendation: Add yt-dlp and anthropic to pyproject.toml before any implementation. Rebuild Docker image as a first task in plan 06-01. Implement yt-dlp as a CLI subprocess (not Python API) to simplify progress streaming. Use librosa.segment.agglomerative() with chroma features for segmentation. Implement the LLM path behind a feature-flag guarded by the presence of ANTHROPIC_API_KEY.

Standard Stack

Core

Library Version Purpose Why Standard
yt-dlp 2026.3.17 YouTube audio download De facto youtube-dl successor; active maintenance, robust format selection
librosa 0.11.0 (already installed) Feature extraction + segmentation Already in use for beat detection; full suite available
anthropic 0.49+ (needs install) LLM API client for optional generation Official SDK, the only LLM client in use on this VPS

Supporting

Library Version Purpose When to Use
numpy already installed Feature matrix ops for segmentation Used throughout librosa pipeline
sklearn (via librosa dep) already installed AgglomerativeClustering used by librosa.segment.agglomerative Pulled in automatically by librosa

Alternatives Considered

Instead of Could Use Tradeoff
yt-dlp subprocess yt_dlp Python API Python API avoids subprocess overhead but progress streaming is more complex; subprocess is simpler for this use case
librosa.segment.agglomerative librosa.segment.subsegment subsegment further divides existing segments; agglomerative is the top-level structural tool — correct choice
anthropic SDK Direct HTTP to /v1/messages SDK handles retries, streaming, error handling; no reason to bypass

Installation (changes to pyproject.toml):

"yt-dlp>=2025.0.0",
"anthropic>=0.40.0",

Rebuild: docker compose -f docker-compose.prod.yml build && docker compose -f docker-compose.prod.yml up -d

Version verification (confirmed against container):

  • librosa: 0.11.0 (verified in container)
  • yt-dlp: 2026.3.17 (verified via pip dry-run)
  • anthropic: not installed — needs addition to pyproject.toml

Architecture Patterns

Recommended Project Structure

lightsync/
├── audio/
│   ├── beats.py          # existing — beat_track; Phase 6 extends this
│   ├── features.py       # NEW — onset_detect, chroma_cqt, rms extraction
│   └── segments.py       # NEW — structural segmentation via agglomerative
├── api/
│   ├── audio.py          # existing — add /fetch endpoint here
│   └── sync.py           # NEW router — /api/sync/generate, /api/audio/segments
└── frontend/
    └── timeline/
        └── segments.js   # NEW — SegmentOverlay class, loadSegments(), canvas drawing

Pattern 1: yt-dlp subprocess with progress streaming

The /api/audio/fetch endpoint spawns yt-dlp as a subprocess with stdout pipe. Progress lines are parsed and sent to the browser as SSE or WebSocket messages.

yt-dlp CLI pattern:

# Source: yt-dlp official docs + verified yt-dlp --help
import asyncio
import re
from pathlib import Path

PROGRESS_RE = re.compile(r'\[download\]\s+([\d.]+)%')

async def download_youtube(url: str, dest_dir: Path) -> str:
    """Download audio from YouTube URL to dest_dir. Returns local filepath."""
    proc = await asyncio.create_subprocess_exec(
        'yt-dlp',
        '--extract-audio',
        '--audio-format', 'mp3',
        '--audio-quality', '0',
        '--newline',             # one progress line per newline (critical for parsing)
        '--no-playlist',
        '-o', str(dest_dir / '%(title)s.%(ext)s'),
        url,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.STDOUT,
    )
    filename = None
    async for line_bytes in proc.stdout:
        line = line_bytes.decode('utf-8', errors='replace').rstrip()
        # Progress: [download]  62.3% of ...
        m = PROGRESS_RE.search(line)
        if m:
            pct = float(m.group(1))
            yield ('progress', pct)
        # Destination: [ExtractAudio] Destination: /app/shows/SongTitle.mp3
        if 'Destination:' in line:
            filename = line.split('Destination:')[-1].strip()
    await proc.wait()
    if proc.returncode != 0:
        raise RuntimeError(f'yt-dlp failed with exit {proc.returncode}')
    return filename

Key flags: --newline is critical — without it, yt-dlp uses carriage-return overwrite and line-by-line parsing breaks. --no-playlist prevents downloading entire playlists when a playlist URL is given.

Pattern 2: librosa feature extraction module

All feature extraction follows the existing asyncio.to_thread pattern from beats.py.

# Source: librosa 0.11.0 docs + verified in container
import librosa
import numpy as np

def extract_features(path: str) -> dict:
    """Extract onsets, chroma, RMS from audio file."""
    y, sr = librosa.load(path, sr=None, mono=True)

    # Onsets — note attacks / transients
    onset_frames = librosa.onset.onset_detect(y=y, sr=sr, units='frames')
    onset_times = librosa.frames_to_time(onset_frames, sr=sr).tolist()

    # Chroma — tonal/harmonic content (CQT-based, more stable than STFT-based)
    chroma = librosa.feature.chroma_cqt(y=y, sr=sr)  # shape (12, T)

    # RMS energy — loudness envelope
    rms = librosa.feature.rms(y=y)[0]  # shape (T,)

    # Reduce to 1s resolution for LLM payload
    hop_length = 512
    frame_rate = sr / hop_length
    rms_1s = [float(np.mean(rms[int(i*frame_rate):int((i+1)*frame_rate)]))
              for i in range(int(len(rms) / frame_rate))]

    # Chroma mean per section (per-second average of dominant pitch class)
    chroma_1s = [int(np.argmax(np.mean(chroma[:, int(i*frame_rate):int((i+1)*frame_rate)], axis=1)))
                 for i in range(int(chroma.shape[1] / frame_rate))]

    return {
        'onset_times': onset_times,
        'rms_1s': rms_1s,
        'chroma_1s': chroma_1s,
        'duration': len(y) / sr,
        'sr': sr,
        'hop_length': hop_length,
    }

Pattern 3: librosa structural segmentation

# Source: librosa 0.11.0 docs + confirmed in container
def detect_segments(path: str, n_segments: int = 8) -> list[dict]:
    """Detect structural sections using agglomerative clustering on chroma features."""
    y, sr = librosa.load(path, sr=None, mono=True)

    # Build chroma feature matrix
    chroma = librosa.feature.chroma_cqt(y=y, sr=sr, bins_per_octave=36)

    # Recurrence matrix — similarity structure
    R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)

    # Filter recurrence matrix to enhance diagonal (segment consistency)
    R_filtered = librosa.segment.path_enhance(R, 15)

    # Agglomerative clustering to find n_segments boundary frames
    bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
    bound_times = librosa.frames_to_time(bounds, sr=sr).tolist()

    # Build section list: [{start, end, label}, ...]
    all_times = [0.0] + bound_times + [librosa.get_duration(y=y, sr=sr)]
    sections = []
    for i, (start, end) in enumerate(zip(all_times, all_times[1:])):
        sections.append({'start': start, 'end': end, 'label': f'SEC {i+1}'})
    return sections

n_segments selection: For a 3–5 minute song, 8 is a good default. The planner should expose this as a parameter (min=4, max=12) with 8 as default. agglomerative() takes the cluster count k directly.

Confidence note: librosa.segment.agglomerative() signature verified in container: (data, k, *, clusterer=None, axis=-1). Input is a feature matrix, not the recurrence matrix directly. Feed the chroma matrix (or its recurrence) directly.

Pattern 4: Progress streaming via Server-Sent Events (SSE)

SSE (not WebSocket) is the clean pattern for one-way server-to-client streaming of download progress. FastAPI supports it via StreamingResponse with text/event-stream content type.

# Source: FastAPI docs pattern
from fastapi.responses import StreamingResponse
import json

@router.post("/fetch")
async def fetch_youtube(request: Request, body: YouTubeFetchRequest):
    async def event_stream():
        try:
            async for (kind, value) in download_youtube(body.url, SHOWS_DIR):
                if kind == 'progress':
                    yield f"data: {json.dumps({'type': 'progress', 'pct': value})}\n\n"
            filename = ...  # captured from download
            yield f"data: {json.dumps({'type': 'done', 'path': filename})}\n\n"
        except Exception as e:
            yield f"data: {json.dumps({'type': 'error', 'msg': str(e)})}\n\n"

    return StreamingResponse(event_stream(), media_type="text/event-stream")

Frontend reads via EventSource API:

const es = new EventSource('/api/audio/fetch?url=' + encodeURIComponent(url));
// or POST via fetch with ReadableStream reader

Note on POST + SSE: EventSource only supports GET. For POST requests with a body, use fetch() with response.body.getReader() to stream the text/event-stream body manually. This is standard practice.

Pattern 5: Heuristic block placement

Given: beats[], onsets[], rms_1s[], chroma_1s[], sections[]
1. For each section:
   a. Compute mean RMS → intensity_level (low/medium/high)
   b. Compute dominant chroma → color palette choice
   c. Count beats/onsets → density
2. Intensity mapping:
   - high RMS → strobe, chase (fast)
   - medium RMS → pulse, color_wipe
   - low RMS → solid, rainbow (slow)
3. Duration: snap to beat grid — each block spans from beat[i] to beat[i+N]
   where N varies by density (sparse → longer blocks, dense → shorter)
4. Color: chroma pitch class 0-11 → map to HSL hue, convert to RGB

Pattern 6: AI Sync panel (06-02)

The panel is a new DOM element (not in the transport bar). Recommended as a collapsible strip above the timeline, or as a popover anchored to a button in the timeline toolbar.

<!-- Recommended: add to timeline-toolbar div in index.html -->
<div class="toolbar-sep"></div>
<button id="btn-ai-sync-open" class="transport-btn">AI SYNC</button>

<!-- Popover panel (shown/hidden by btn-ai-sync-open) -->
<div id="ai-sync-panel" class="ai-sync-panel" style="display:none;">
  <div class="panel-header">AI SYNC</div>
  <!-- feature toggles, GENERATE button, USE LLM toggle -->
</div>

Pattern 7: Canvas segment overlay

Draw segment bands on the waveform canvas, below the waveform fill (rendered before waveform in the render loop). The existing render() method in timeline.js draws in layers — insert segment bands between step 2 (track row banding) and step 3 (waveform).

// In TimelineCanvas.render(), after track row banding, before waveform:
if (this.segments.length > 0) {
    this._drawSegmentBands(ctx, W, H);
}

_drawSegmentBands(ctx, W, H) {
    const BAND_COLORS = [
        'rgba(0,255,255,0.15)',   // --accent cyan
        'rgba(180,0,255,0.15)',   // purple complement
    ];
    const totalH = this.tracks.length * TRACK_HEIGHT;
    const topY = TIME_AXIS_HEIGHT;

    for (let i = 0; i < this.segments.length; i++) {
        const seg = this.segments[i];
        const x1 = this.timeToX(seg.start);
        const x2 = this.timeToX(seg.end);
        const color = BAND_COLORS[i % BAND_COLORS.length];
        const isSelected = this.selectedSegment === i;

        ctx.fillStyle = color;
        ctx.fillRect(x1, topY, x2 - x1, totalH);

        if (isSelected) {
            ctx.strokeStyle = 'rgba(0,255,255,1.0)';
            ctx.lineWidth = 1;
            ctx.strokeRect(x1, topY, x2 - x1, totalH);
        }

        // Label: small monospace uppercase at top-left of band
        if (x2 - x1 > 30) {
            ctx.fillStyle = 'rgba(0,255,255,0.6)';
            ctx.font = '9px monospace';
            ctx.fillText(seg.label, Math.max(x1 + 3, HEADER_WIDTH + 2), topY + 10);
        }
    }
}

Anti-Patterns to Avoid

  • Blocking the event loop with librosa: All librosa calls must use asyncio.to_thread. Feature extraction for a 5-minute song takes 3–8 seconds. Verified pattern: identical to beats.py.
  • Using EventSource for POST requests: EventSource only supports GET. Use fetch() + ReadableStream reader for streaming responses to POST requests.
  • Calling yt-dlp without --newline: Without this flag, progress output uses CR overwrites and line-by-line parsing is impossible.
  • Calling librosa.segment.agglomerative() with recurrence matrix directly: The function expects a feature data matrix. Feed it the chroma array. Use recurrence_matrix() as a preprocessing step for path_enhance filtering, then pass the filtered matrix as data.
  • Generating blocks without confirming replacement: If existing blocks are in the timeline, the heuristic must trigger the confirmation dialog first (D-07).
  • LLM path without API key guard: Check for ANTHROPIC_API_KEY env var presence before showing the USE LLM toggle. If absent, disable the toggle with tooltip ANTHROPIC_API_KEY not configured.

Don't Hand-Roll

Problem Don't Build Use Instead Why
YouTube download Custom HTTP fetcher or youtube-dl yt-dlp Rate limiting, format negotiation, age-gating, cookies — yt-dlp handles all of these; a custom solution would break constantly
Audio feature extraction Custom FFT + onset detector librosa librosa.onset.onset_detect is tuned for music, uses spectral flux; hand-rolled detectors miss transients
Structural segmentation Beat-counting heuristics librosa.segment.agglomerative Self-similarity matrix approach is the musicology-standard technique; hand-rolled approaches cannot detect phrase-level repetition
LLM HTTP calls Raw requests to Anthropic API anthropic SDK SDK handles token limits, retries, streaming, error handling

Key insight: The audio analysis stack is already in the container. Feature extraction is a one-module addition to beats.py patterns. The only genuinely new infrastructure is yt-dlp (subprocess) and the anthropic SDK (Python package).

Common Pitfalls

Pitfall 1: yt-dlp output filename is not predictable

What goes wrong: yt-dlp names the output file from the video title. Special characters, Unicode, spaces, and truncation rules make the filename unpredictable. The /api/audio/fetch endpoint must capture the actual output filename from yt-dlp's Destination: log line — not from the URL. Why it happens: yt-dlp sanitizes filenames per-platform and may truncate long titles. How to avoid: Parse the [ExtractAudio] Destination: /path/to/file.mp3 line from yt-dlp stdout. Use this as the returned filename. Alternatively, pass a template with a fixed stem: -o '/app/shows/yt_%(id)s.%(ext)s' to make the filename deterministic from the YouTube video ID. Warning signs: 404 errors when loading the audio file immediately after download.

Pitfall 2: librosa.segment.agglomerative segment count vs. n_segments

What goes wrong: librosa.segment.agglomerative(data, k=n) returns k boundary frame indices (not k+1 segments). If the caller treats the output as segment counts, it's off by one. Why it happens: The output is boundary frames — need to add 0 at start and len at end to form k+1 sections. How to avoid: Always construct sections as zip(all_times[:-1], all_times[1:]) where all_times = [0.0] + bound_times + [duration]. Warning signs: Last section not present, or first section starting at a non-zero time.

Pitfall 3: SSE streaming via POST requires ReadableStream reader

What goes wrong: Frontend uses EventSource(url) but the fetch endpoint is POST-only (body contains the URL). EventSource does not support POST. Why it happens: Developers assume all SSE uses EventSource. How to avoid: Use fetch(url, {method:'POST', body:...}) then response.body.getReader() to stream the text/event-stream body. Parse data: {...}\n\n frames manually. Warning signs: Browser console error "Failed to construct 'EventSource'", or CORS/method error.

Pitfall 4: LLM path with no API key crashes silently

What goes wrong: anthropic.Anthropic() is instantiated at import time (e.g., module level), throws an error if ANTHROPIC_API_KEY is missing, crashing the entire FastAPI startup. Why it happens: SDK raises AuthenticationError or APIKeyMissingError at client construction. How to avoid: Instantiate the Anthropic client lazily — only when a request triggers the LLM path. Wrap in a try/except and return a 503 LLM_UNAVAILABLE response if the key is missing. Warning signs: App fails to start after adding anthropic to pyproject.toml when ANTHROPIC_API_KEY is not set.

Pitfall 5: Docker image not rebuilt after dependency addition

What goes wrong: yt-dlp and anthropic are added to pyproject.toml but the running container still uses the old image without them. Why it happens: docker compose up -d does not rebuild the image automatically. How to avoid: The first task in plan 06-01 must be docker compose -f docker-compose.prod.yml build then up -d --force-recreate. Add yt-dlp CLI availability check to the /api/audio/fetch endpoint startup. Warning signs: ModuleNotFoundError: No module named 'yt_dlp' or yt-dlp: command not found in logs.

Pitfall 6: yt-dlp not available as CLI after pip install of yt-dlp Python package

What goes wrong: pip install yt-dlp installs both the Python library AND the yt-dlp CLI binary. But in the Docker image, the binary is placed in /usr/local/bin/yt-dlp. If the subprocess calls yt-dlp as a command, it should work. However, on minimal images the PATH may not include this location. Why it happens: Slim Docker images have a minimal PATH. How to avoid: Use sys.executable -m yt_dlp as the subprocess command instead of yt-dlp. This is guaranteed to use the installed module regardless of PATH. Warning signs: FileNotFoundError: [Errno 2] No such file or directory: 'yt-dlp' in subprocess call.

Pitfall 7: Segment overlay rendered over waveform instead of under it

What goes wrong: Segment bands obscure the waveform because they're drawn after the waveform in render(). Why it happens: Render order in canvas is painter's algorithm — later draws go on top. How to avoid: In TimelineCanvas.render(), draw segment bands between step 2 (track row banding) and step 3 (waveform). The _drawSegmentBands() call must come before the waveform loop. Warning signs: Waveform is not visible in sections with colored overlays.

Pitfall 8: AI Sync blocks don't have valid device_id

What goes wrong: Heuristic generates blocks but assigns incorrect or missing device IDs. The show engine ignores cues with unknown device IDs. Why it happens: The sync engine may not know which devices are registered. How to avoid: The /api/sync/generate endpoint must accept the current device list as part of its input, or fetch it from registry. Generate one block set per device, using the actual device.id UUIDs.

Code Examples

yt-dlp subprocess (confirmed API)

# Use sys.executable -m yt_dlp to avoid PATH issues in Docker
import sys
proc = await asyncio.create_subprocess_exec(
    sys.executable, '-m', 'yt_dlp',
    '--extract-audio', '--audio-format', 'mp3',
    '--audio-quality', '0',
    '--newline', '--no-playlist',
    '-o', str(SHOWS_DIR / 'yt_%(id)s.%(ext)s'),  # predictable filename
    url,
    stdout=asyncio.subprocess.PIPE,
    stderr=asyncio.subprocess.STDOUT,
)

librosa.segment.agglomerative (verified signature in container)

# Verified: librosa.segment.agglomerative(data, k, *, clusterer=None, axis=-1)
# librosa.segment.recurrence_matrix(data, *, k=None, width=1, metric='euclidean', sym=False, ...)
chroma = librosa.feature.chroma_cqt(y=y, sr=sr)
R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)
R_filtered = librosa.segment.path_enhance(R, 15)
bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)

SSE streaming via fetch ReadableStream (frontend)

async function fetchWithProgress(url, body, onProgress, onDone, onError) {
    const res = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(body),
    });
    const reader = res.body.getReader();
    const decoder = new TextDecoder();
    let buf = '';
    while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buf += decoder.decode(value, { stream: true });
        const lines = buf.split('\n\n');
        buf = lines.pop();
        for (const chunk of lines) {
            if (chunk.startsWith('data: ')) {
                const msg = JSON.parse(chunk.slice(6));
                if (msg.type === 'progress') onProgress(msg.pct);
                if (msg.type === 'done') onDone(msg.path);
                if (msg.type === 'error') onError(msg.msg);
            }
        }
    }
}

Terminal progress bar rendering (frontend)

function renderProgressBar(pct, width = 20) {
    // DOWNLOADING ██████░░░░ 62%
    const filled = Math.round((pct / 100) * width);
    const empty = width - filled;
    return `DOWNLOADING ${'█'.repeat(filled)}${'░'.repeat(empty)} ${Math.round(pct)}%`;
}

Anthropic LLM call (lazy instantiation pattern)

import os

def get_anthropic_client():
    """Lazy client construction — only called when USE LLM is triggered."""
    key = os.environ.get('ANTHROPIC_API_KEY')
    if not key:
        raise ValueError('ANTHROPIC_API_KEY not configured')
    import anthropic
    return anthropic.Anthropic(api_key=key)

async def generate_llm_show(features: dict, devices: list) -> list[dict]:
    """Generate show blocks via LLM. Returns list of CueModel-compatible dicts."""
    client = await asyncio.to_thread(get_anthropic_client)
    # ... prompt construction and API call

Environment Availability

Dependency Required By Available Version Fallback
librosa Feature extraction, segmentation YES (in container) 0.11.0 —
numpy Feature matrix ops YES (in container) installed with librosa —
sklearn agglomerative clustering (librosa dep) YES (librosa pulls it) installed with librosa —
yt-dlp (Python pkg) YouTube download NO — needs pyproject.toml addition 2026.3.17 available —
anthropic SDK LLM show generation NO — needs pyproject.toml addition 0.40+ available Disable USE LLM toggle
ANTHROPIC_API_KEY env var LLM show generation NOT CONFIGURED in docker-compose.prod.yml — Graceful disable
ffmpeg yt-dlp audio conversion to MP3 YES (already in Dockerfile) system ffmpeg —

Missing dependencies with no fallback:

  • yt-dlp Python package — AUD-02 blocks without it. Add to pyproject.toml and rebuild Docker image.

Missing dependencies with fallback:

  • anthropic SDK — LLM path is off by default. App works fully without it; just disable the USE LLM toggle when ANTHROPIC_API_KEY is absent or SDK not installed.
  • ANTHROPIC_API_KEY — Same fallback: guard the UI toggle, return 503 from endpoint.

State of the Art

Old Approach Current Approach When Changed Impact
youtube-dl yt-dlp 2021 (youtube-dl DMCA takedown response) yt-dlp is the active fork; youtube-dl is stale
librosa.beat.beat_track only beat_track + onset_detect + chroma + RMS Phase 6 Full feature set for heuristic mapping
No segmentation agglomerative clustering on chroma recurrence matrix Phase 6 Provides section-level context for AI Sync

Deprecated/outdated:

  • youtube-dl: superseded by yt-dlp; do not use
  • librosa.segment.subsegment: fine-grained sub-segmentation within known sections; not what we need for top-level structure detection

Open Questions

  1. LLM prompt token budget for a 5-minute song

    • What we know: RMS at 1s resolution = ~300 values; beats ~120 timestamps; chroma summary per section = 8 × 12 = 96 values; total ~500 numbers as JSON
    • What's unclear: Whether Anthropic claude-3-haiku (cheapest, fastest) is adequate for structured JSON generation of this kind, or if claude-3-sonnet is needed for reliable output
    • Recommendation: Use claude-3-haiku-20240307 with a response schema in the prompt. The payload is small; haiku handles structured JSON well at this scale.

  2. yt-dlp download time for a typical 3–5 minute song

    • What we know: yt-dlp downloads at network speed; typical YouTube audio is 128–320kbps
    • What's unclear: Whether the progress bar SSE stream can stay open for the full download duration without Traefik/Authelia proxy timeout
    • Recommendation: Set proxy_read_timeout in Traefik if default timeout is <60s. Alternatively, accept that progress streaming may time out at proxy and use WebSocket instead (the WS connection is already established).

  3. librosa.segment.agglomerative n_segments optimal value

    • What we know: Target is 4–12 sections; the algorithm accepts k directly
    • What's unclear: Whether a fixed k=8 produces too many or too few sections for short (<2 min) or long (>6 min) songs
    • Recommendation: Default k=8; add heuristic: k = max(4, min(12, int(duration / 30))) — roughly one section per 30 seconds.

Sources

Primary (HIGH confidence)

  • librosa 0.11.0 installed in container — librosa.segment, librosa.onset, librosa.feature APIs verified by direct introspection in Docker container
  • yt-dlp pip dry-run in container — version 2026.3.17 available
  • Codebase direct read — beats.py, audio.py, timeline.js, beats.js, show.py, main.py, index.html, style.css all read directly

Secondary (MEDIUM confidence)

  • yt-dlp GitHub README (known from training data, verified version via pip registry) — --newline, --extract-audio, --no-playlist flag behavior
  • FastAPI StreamingResponse + SSE pattern (standard FastAPI docs pattern, widely used)

Tertiary (LOW confidence)

  • Anthropic SDK lazy instantiation pattern — based on training data knowledge of anthropic Python SDK; version and exact error class names should be confirmed when adding to requirements

Metadata

Confidence breakdown:

  • Standard stack: HIGH — verified in container (librosa), pip registry (yt-dlp), pyproject.toml (existing deps)
  • Architecture: HIGH — all patterns traced to verified existing code (beats.py → features.py extension, timeline.js render order, audio.py endpoint pattern)
  • Pitfalls: HIGH — yt-dlp filename/PATH pitfalls verified via subprocess behavior; canvas render order verified in timeline.js; SSE POST limitation is a browser API constraint

Research date: 2026-04-07 Valid until: 2026-05-07 (librosa and FastAPI are stable; yt-dlp updates frequently but CLI flags are backward-compatible)

Reference in New Issue View Git Blame Copy Permalink