led2/.planning/phases/06-ai-sync/06-CONTEXT.md

Phase 6: AI Sync - Context

Gathered: 2026-04-07 Status: Ready for planning

## Phase Boundary

YouTube URL loading (yt-dlp → local file → existing audio stack), AI-assisted timeline auto-fill from multi-feature audio analysis (beats/onsets/chroma/RMS heuristics + optional LLM generation), and structural segmentation overlaid on the waveform as colored labeled bands. No microcontroller firmware (Phase 7).

## Implementation Decisions

YouTube Loading (06-01)

• D-01: yt-dlp downloads audio to /app/shows/ — same directory as uploaded files. After download completes, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
• D-02: During download, show a terminal-style progress bar with download percentage. Parse yt-dlp --progress output and stream updates to the browser. Disable the URL input during download; re-enable and auto-load when complete.
• D-03: URL input lives in the transport bar — inline with the existing file selector controls. Text input + FETCH button, same row, terminal aesthetic (flat, monospace, no rounded corners).

AI Sync Engine (06-02)

• D-04: Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
• D-05: Heuristic features to extract and use:
  • Beats — from existing beats.py beat_track
  • Onsets — librosa.onset.onset_detect() for note attacks and transients
  • Chroma — librosa.feature.chroma_cqt() for tonal/harmonic content → drives color palette selection
  • RMS energy — librosa.feature.rms() for loudness envelope → drives animation intensity
• D-06: LLM path: generate from scratch. The LLM receives raw audio features (beat list, segment boundaries, onset times, energy profile, chroma summary) and generates a complete block list — timestamps, duration, animation type, params per device. The heuristic result is NOT passed to the LLM; it generates independently.
• D-07: Conflict handling: prompt before replacing. If the timeline has existing blocks when AI Sync is triggered, show a confirmation dialog: "Replace all existing blocks?" Yes clears and re-fills, No cancels. Undo/redo (Phase 4) can recover from replacements.
• D-08: AI Sync trigger lives in a dedicated panel or context menu — not in the main toolbar or transport bar. Keeps the primary UI uncluttered.

Segmentation Overlay (06-03)

• D-09: Translucent colored band overlays drawn on the waveform canvas — one band per detected section, spanning the full waveform height. Alternating colors (e.g. cyan/purple variants from the existing palette).
• D-10: Section labels are generic numbered: SEC 1, SEC 2, SEC 3, ... — no attempt to infer VERSE/CHORUS. Simple, always correct, terminal-readable.
• D-11: Bands are interactive: clicking a section band selects it (highlight border). When AI Sync is triggered with a section selected, it fills only that section's time range. This allows per-section AI fill rather than whole-show fill.

Claude's Discretion

• librosa segmentation algorithm: Choose the best-fit algorithm from librosa's segment module (e.g. librosa.segment.agglomerative with recurrence matrix, or MSAF-style structural analysis). Target 4–12 sections for a typical 3–5 minute song. Planner decides implementation details.
• LLM prompt design: Structure the feature payload as a compact JSON (beat timestamps, RMS samples at 1s resolution, chroma average per section, segment boundaries). Planner designs the prompt. Use existing Anthropic SDK patterns from the codebase if available.
• yt-dlp progress parsing: Parse --progress output lines via subprocess pipe. Planner decides exact regex/parsing approach.
• Section color palette: Pick 2–3 alternating colors from existing CSS variables (--accent, complementary hues). No new colors outside the existing terminal palette.

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

Requirements

• .planning/REQUIREMENTS.md — SYNC-03 (AI-assisted show generation), AUD-02 (YouTube URL loading)
• .planning/ROADMAP.md §Phase 6 — Success criteria (3 items), plan breakdown (06-01 through 06-03)

Existing audio analysis

• lightsync/audio/beats.py — analyze(path) → {tempo, beats} using librosa.beat.beat_track. Phase 6 extends this module (or adds segments.py) for onset/chroma/RMS/segmentation.
• lightsync/api/audio.py — /api/audio/upload, /api/audio/beats, /api/audio/waveform endpoints. YouTube download adds a new /api/audio/fetch (or similar) endpoint here.

Existing frontend wiring

• lightsync/frontend/app.js — Audio load flow: file selected → audio.src = /shows/filename → loadedmetadata fires → timeline.loadWaveform(path) + timeline.loadBeats(path). YouTube download reuses the same post-load path.
• lightsync/frontend/timeline/timeline.js — loadWaveform() and loadBeats() methods; segmentation overlay drawn on the same waveform canvas.
• lightsync/frontend/index.html — Transport bar layout (file selector, upload, LOAD button); URL input + FETCH button are added to this row.
• lightsync/frontend/style.css — CSS variables; all new UI elements must use --accent, --bg, --border, --text-dim.

Data models

• lightsync/models/show.py — source_type: Literal["file", "youtube"] already present; set this when a YouTube URL is used.

Prior context

• .planning/phases/01-foundation/01-CONTEXT.md — Visual aesthetic (flat, sharp, cyan/teal, no rounded corners, no drop shadows, monospace uppercase labels)
• .planning/phases/05-live-show-execution/05-CONTEXT.md — WebSocket protocol patterns, transport bar layout, CSS variable usage

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• beats.py analyze_async() — thread-pool wrapper for librosa analysis; new segmentation/feature extraction follows the same asyncio.to_thread pattern
• audio.py upload endpoint — POST handler pattern; YouTube fetch endpoint mirrors this (async subprocess + cache beat data after download)
• app.state.beats cache — keyed by filepath; add app.state.segments for segmentation cache in the same lifespan setup
• timeline.js waveform canvas — already drawing waveform peaks; segmentation bands drawn as canvas fillRect calls on the same canvas layer, below the waveform

Established Patterns

• Librosa analysis runs in asyncio.to_thread — never blocks the event loop
• New API endpoints go into lightsync/api/audio.py (or a new lightsync/api/sync.py router)
• Frontend ES modules — new JS files must use export/import
• WebSocket message protocol: {"type": "...", ...} — AI sync progress can use a sync_progress message type
• CSS panel strip pattern from Phase 4/5: compact horizontal strips, 1px solid var(--border) borders

Integration Points

• Transport bar (index.html) — add URL text input + FETCH button next to existing file controls
• audio.py router — add /api/audio/fetch (POST, takes {url: string}) for yt-dlp download; /api/audio/segments for structural analysis
• timeline.js loadWaveform() / loadBeats() — add loadSegments() method to render colored bands
• main.py lifespan — add app.state.segments = {} alongside existing app.state.beats
• AI Sync panel — new dedicated panel or popover; triggers /api/sync/generate (POST) with {path, use_llm: bool, section_range?: [start, end]}

</code_context>

## Specific Ideas

• Progress bar during YouTube download should feel terminal: DOWNLOADING ██████░░░░ 62% — monospace, fills left to right, uses --accent color. Not a smooth CSS animation, a stepped block fill.
• Section bands: translucent (opacity ~0.15-0.20), drawn below the waveform fill so the waveform is still readable. Labels in small monospace uppercase at the top-left of each band.
• The AI Sync panel could look like a terminal dialog — dark background, border, AI SYNC header, feature toggles (use beats / onsets / chroma / RMS all checked by default), a GENERATE button, and an optional USE LLM toggle.
• When a section band is selected, draw a 1px bright border around it (using --accent at full opacity). Deselect when clicking elsewhere.

## Deferred Ideas

• Per-section manual label renaming (e.g. rename SEC 2 to CHORUS) — out of Phase 6 scope
• LLM-named sections — deferred; generic numbering is sufficient for v1
• Full MPV backend (streaming YouTube without download) — out of scope, download approach is cleaner
• AI Sync with partial fill (gap-only mode) — deferred; prompt+replace is sufficient for Phase 6

---

Phase: 06-ai-sync Context gathered: 2026-04-07