Gitea-Admin/led2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(06-ai-sync): create phase plan

Browse Source
This commit is contained in:
Claude
2026-04-07 11:31:04 +00:00
parent b88bd98c40
commit 57bd91d4e7
4 changed files with 2059 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

515
.planning/phases/06-ai-sync/06-01-PLAN.md Normal file
View File

@@ -0,0 +1,515 @@
---
phase: 06-ai-sync
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pyproject.toml
- docker-compose.prod.yml
- lightsync/api/audio.py
- lightsync/frontend/index.html
- lightsync/frontend/app.js
- lightsync/frontend/style.css
autonomous: true
requirements:
- AUD-02
must_haves:
truths:
- "A YouTube URL pasted into the transport bar downloads the audio to /app/shows/ as an MP3 file"
- "Download progress is displayed as a terminal-style block progress bar in the transport bar"
- "After download completes, the audio file auto-loads into the player and waveform/beats load identically to a local upload"
- "The URL input is disabled during download and re-enabled on completion or error"
artifacts:
- path: "pyproject.toml"
provides: "yt-dlp and anthropic dependencies"
contains: "yt-dlp"
- path: "lightsync/api/audio.py"
provides: "POST /api/audio/fetch endpoint with SSE progress streaming"
contains: "async def fetch_youtube"
- path: "lightsync/frontend/index.html"
provides: "YouTube URL text input and FETCH button in transport bar"
contains: "yt-url"
- path: "lightsync/frontend/app.js"
provides: "SSE ReadableStream reader, progress bar rendering, auto-load on completion"
contains: "fetchWithProgress"
key_links:
- from: "lightsync/frontend/app.js"
to: "/api/audio/fetch"
via: "fetch POST with ReadableStream reader"
pattern: "api/audio/fetch"
- from: "lightsync/api/audio.py"
to: "yt-dlp subprocess"
via: "asyncio.create_subprocess_exec"
pattern: "yt_dlp"
- from: "lightsync/frontend/app.js"
to: "audio element"
via: "auto-load after download done event"
pattern: "audio\\.src.*shows/"
user_setup:
- service: anthropic
why: "Optional LLM show generation (Plan 06-03)"
env_vars:
- name: ANTHROPIC_API_KEY
source: "Anthropic Console -> API Keys (https://console.anthropic.com/settings/keys)"
dashboard_config: []
---
<objective>
YouTube audio loading via yt-dlp with terminal-style progress streaming.
Purpose: Enables loading songs from YouTube URLs (AUD-02), the primary new input method for Phase 6. After download, the file enters the existing audio pipeline unchanged (waveform, beats, show saving all reuse existing wiring).
Output: Working YouTube fetch endpoint with SSE progress, transport bar URL input, and Docker image with yt-dlp + anthropic dependencies installed.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-ai-sync/06-CONTEXT.md
@.planning/phases/06-ai-sync/06-RESEARCH.md
<interfaces>
<!-- Key types and contracts the executor needs -->
From lightsync/api/audio.py:
```python
router = APIRouter()
SHOWS_DIR = Path("/app/shows")
SUPPORTED_EXTENSIONS = {".mp3", ".wav", ".flac", ".ogg"}
@router.post("/upload")
async def upload_audio(request: Request, file: UploadFile = File(...)):
# Uploads file to SHOWS_DIR, runs beat analysis, caches in app.state.beats
...
@router.get("/beats")
async def get_beats(request: Request, path: str):
...
@router.get("/waveform")
async def get_waveform(path: str, peaks: int = 1000):
...
```
From lightsync/models/show.py:
```python
class AudioRef(BaseModel):
source_type: Literal["file", "youtube"] = "file"
path: str | None = None
yt_url: str | None = None
duration_seconds: float | None = None
```
From lightsync/main.py:
```python
app.include_router(audio.router, prefix="/api/audio")
# app.state.beats = {} in lifespan
```
From lightsync/frontend/app.js:
```javascript
// Audio load flow after file selection:
// audio.src = `/shows/${filename}`; audio.load();
// Then loadedmetadata fires -> timeline.loadWaveform(path) + timeline.loadBeats(path)
async function refreshFiles(selectPath) { ... }
// Refreshes audio-select dropdown from /api/audio/files
```
From lightsync/frontend/index.html (transport bar, lines 84-101):
```html
<div class="transport-sep"></div>
<label class="transport-btn" title="Upload MP3/WAV/FLAC/OGG" style="cursor:pointer;">
+ UPLOAD
<input type="file" id="audio-upload" accept=".mp3,.wav,.flac,.ogg" style="display:none;">
</label>
<select id="audio-select" class="transport-file-select">
<option value="">&#8212; no files &#8212;</option>
</select>
<button id="btn-load" class="transport-btn" title="Load selected file">LOAD</button>
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Add dependencies and rebuild Docker image</name>
<files>pyproject.toml, docker-compose.prod.yml</files>
<read_first>
- pyproject.toml (current dependencies list)
- docker-compose.prod.yml (current environment config)
- Dockerfile (build steps)
</read_first>
<action>
1. Edit `pyproject.toml` dependencies array — add these two lines after the existing `"python-multipart>=0.0.9"` entry:
```
"yt-dlp>=2025.0.0",
"anthropic>=0.40.0",
```
2. Edit `docker-compose.prod.yml` — add `ANTHROPIC_API_KEY` to the environment section (empty default so the app starts without it):
```yaml
environment:
HOST: 0.0.0.0
PORT: "8000"
ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY:-}"
```
3. Rebuild and restart the Docker container:
```bash
cd /home/claude/led2
docker compose -f docker-compose.prod.yml build
docker compose -f docker-compose.prod.yml up -d --force-recreate
```
4. Verify yt-dlp is available inside the container:
```bash
docker exec lightsync python -c "import yt_dlp; print(yt_dlp.version.__version__)"
docker exec lightsync python -m yt_dlp --version
```
</action>
<verify>
<automated>docker exec lightsync python -c "import yt_dlp; print('yt-dlp OK')" && docker exec lightsync python -c "import anthropic; print('anthropic OK')"</automated>
</verify>
<acceptance_criteria>
- pyproject.toml contains `"yt-dlp>=2025.0.0"`
- pyproject.toml contains `"anthropic>=0.40.0"`
- docker-compose.prod.yml contains `ANTHROPIC_API_KEY`
- `docker exec lightsync python -c "import yt_dlp"` exits 0
- `docker exec lightsync python -c "import anthropic"` exits 0
</acceptance_criteria>
<done>Both yt-dlp and anthropic are importable inside the running container. ANTHROPIC_API_KEY env var is wired (empty by default).</done>
</task>
<task type="auto">
<name>Task 2: YouTube download endpoint with SSE progress streaming</name>
<files>lightsync/api/audio.py</files>
<read_first>
- lightsync/api/audio.py (existing endpoints and patterns)
- lightsync/audio/beats.py (analyze_async pattern for post-download beat analysis)
- lightsync/main.py (app.state.beats cache)
</read_first>
<action>
Add a POST `/fetch` endpoint to `lightsync/api/audio.py` that:
1. Add imports at the top of the file:
```python
import asyncio
import json
import re
import sys
from fastapi import Request
from fastapi.responses import StreamingResponse
```
2. Add constants:
```python
PROGRESS_RE = re.compile(r'\[download\]\s+([\d.]+)%')
```
3. Add Pydantic model for the request body:
```python
from pydantic import BaseModel
class YouTubeFetchRequest(BaseModel):
url: str
```
4. Add the endpoint:
```python
@router.post("/fetch")
async def fetch_youtube(request: Request, body: YouTubeFetchRequest):
"""Download audio from YouTube URL via yt-dlp. Streams SSE progress events."""
async def event_stream():
filename = None
try:
proc = await asyncio.create_subprocess_exec(
sys.executable, '-m', 'yt_dlp',
'--extract-audio',
'--audio-format', 'mp3',
'--audio-quality', '0',
'--newline',
'--no-playlist',
'-o', str(SHOWS_DIR / 'yt_%(id)s.%(ext)s'),
body.url,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
)
async for line_bytes in proc.stdout:
line = line_bytes.decode('utf-8', errors='replace').rstrip()
# Progress line
m = PROGRESS_RE.search(line)
if m:
pct = float(m.group(1))
yield f"data: {json.dumps({'type': 'progress', 'pct': pct})}\n\n"
# Destination line — captures actual output filename
if 'Destination:' in line:
filename = line.split('Destination:')[-1].strip()
await proc.wait()
if proc.returncode != 0:
yield f"data: {json.dumps({'type': 'error', 'msg': f'yt-dlp exited with code {proc.returncode}'})}\n\n"
return
if not filename:
# Fallback: find most recently created mp3 in SHOWS_DIR
mp3s = sorted(SHOWS_DIR.glob('yt_*.mp3'), key=lambda p: p.stat().st_mtime, reverse=True)
if mp3s:
filename = str(mp3s[0])
if filename:
# Run beat analysis in background and cache
try:
beat_data = await analyze_async(filename)
if not hasattr(request.app.state, 'beats') or request.app.state.beats is None:
request.app.state.beats = {}
request.app.state.beats[filename] = beat_data
except Exception:
pass # Non-fatal — beats can be loaded later
fname = Path(filename).name
yield f"data: {json.dumps({'type': 'done', 'path': filename, 'filename': fname})}\n\n"
else:
yield f"data: {json.dumps({'type': 'error', 'msg': 'Download completed but output file not found'})}\n\n"
except Exception as e:
yield f"data: {json.dumps({'type': 'error', 'msg': str(e)})}\n\n"
return StreamingResponse(event_stream(), media_type="text/event-stream")
```
Key design decisions per CONTEXT.md:
- D-01: Downloads to SHOWS_DIR (`/app/shows/`) — same directory as uploaded files
- Uses `sys.executable -m yt_dlp` to avoid PATH issues in Docker (Pitfall 6 from RESEARCH.md)
- Uses `yt_%(id)s.%(ext)s` output template for predictable filenames (Pitfall 1 from RESEARCH.md)
- `--newline` flag is critical for line-by-line progress parsing (anti-pattern from RESEARCH.md)
- Runs beat analysis after download completes, caches in app.state.beats
</action>
<verify>
<automated>docker exec lightsync python -c "from lightsync.api.audio import fetch_youtube; print('endpoint imported OK')"</automated>
</verify>
<acceptance_criteria>
- lightsync/api/audio.py contains `async def fetch_youtube`
- lightsync/api/audio.py contains `PROGRESS_RE = re.compile`
- lightsync/api/audio.py contains `sys.executable, '-m', 'yt_dlp'`
- lightsync/api/audio.py contains `--newline`
- lightsync/api/audio.py contains `yt_%(id)s.%(ext)s`
- lightsync/api/audio.py contains `media_type="text/event-stream"`
- lightsync/api/audio.py contains `class YouTubeFetchRequest`
- lightsync/api/audio.py contains `analyze_async(filename)` (beat analysis after download)
</acceptance_criteria>
<done>POST /api/audio/fetch accepts {url: string}, spawns yt-dlp subprocess, streams SSE progress events, runs beat analysis on completion, returns done event with filename.</done>
</task>
<task type="auto">
<name>Task 3: YouTube URL input UI with terminal progress bar</name>
<files>lightsync/frontend/index.html, lightsync/frontend/app.js, lightsync/frontend/style.css</files>
<read_first>
- lightsync/frontend/index.html (transport bar layout, line ~84-104)
- lightsync/frontend/app.js (wireAudio, refreshFiles, upload handler patterns)
- lightsync/frontend/style.css (transport-btn, transport-file-select, transport-sep classes)
</read_first>
<action>
**index.html** — Add YouTube URL input + FETCH button inline in the transport bar. Insert BEFORE the `<div style="flex:1"></div>` spacer (after the LOAD button for audio files, around line 101). Per D-03: same row, terminal aesthetic, flat, monospace, no rounded corners.
```html
<div class="transport-sep"></div>
<input type="text" id="yt-url" class="yt-url-input" placeholder="YOUTUBE URL" disabled>
<button id="btn-fetch" class="transport-btn" title="Download from YouTube">FETCH</button>
<span id="yt-progress" class="yt-progress" style="display:none;"></span>
```
Insert this block between `<button id="btn-load" ...>LOAD</button>` and `<div style="flex:1"></div>`.
**style.css** — Add styles at the end of the file. Use existing CSS variables only (--accent, --bg-panel, --border-dim, --text-dim, --text-primary, --font-mono). Per D-03: flat, monospace, no rounded corners.
```css
/* YouTube URL input (Phase 6) */
.yt-url-input {
background: var(--bg-panel);
border: 1px solid var(--border-dim);
color: var(--text-primary);
font-family: var(--font-mono);
font-size: 12px;
padding: 2px 6px;
width: 180px;
outline: none;
}
.yt-url-input:focus {
border-color: var(--accent);
}
.yt-url-input:disabled {
opacity: 0.4;
cursor: not-allowed;
}
.yt-progress {
font-family: var(--font-mono);
font-size: 11px;
color: var(--accent);
white-space: nowrap;
min-width: 200px;
}
```
**app.js** — Add YouTube fetch handler. Insert after the upload handler (`audio-upload` change listener) and before the `btn-load` click listener section. Add these two functions and the event wiring:
1. Add the `fetchWithProgress` function (from RESEARCH.md Pattern 4 — SSE via fetch ReadableStream):
```javascript
async function fetchWithProgress(url, body, onProgress, onDone, onError) {
const res = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buf += decoder.decode(value, { stream: true });
const lines = buf.split('\n\n');
buf = lines.pop();
for (const chunk of lines) {
if (chunk.startsWith('data: ')) {
const msg = JSON.parse(chunk.slice(6));
if (msg.type === 'progress') onProgress(msg.pct);
if (msg.type === 'done') onDone(msg);
if (msg.type === 'error') onError(msg.msg);
}
}
}
}
```
2. Add the terminal progress bar renderer (per D-02 — stepped block fill, monospace):
```javascript
function renderProgressBar(pct, width = 20) {
const filled = Math.round((pct / 100) * width);
const empty = width - filled;
return `DOWNLOADING ${'█'.repeat(filled)}${'░'.repeat(empty)} ${Math.round(pct)}%`;
}
```
3. Wire the FETCH button click handler:
```javascript
document.getElementById('btn-fetch')?.addEventListener('click', async () => {
const urlInput = document.getElementById('yt-url');
const btn = document.getElementById('btn-fetch');
const progress = document.getElementById('yt-progress');
const url = urlInput?.value.trim();
if (!url) return;
// D-02: Disable input during download
urlInput.disabled = true;
btn.disabled = true;
btn.textContent = '...';
progress.style.display = 'inline';
progress.textContent = 'DOWNLOADING ░░░░░░░░░░░░░░░░░░░░ 0%';
try {
await fetchWithProgress('/api/audio/fetch', { url },
(pct) => {
progress.textContent = renderProgressBar(pct);
},
(msg) => {
// D-01: After download, load identically to local upload
progress.textContent = 'COMPLETE';
const audio = document.getElementById('player');
if (audio && msg.filename) {
audio.src = `/shows/${msg.filename}`;
audio.load();
}
refreshFiles(msg.path);
setTimeout(() => { progress.style.display = 'none'; }, 2000);
},
(errMsg) => {
progress.textContent = `ERROR: ${errMsg}`;
progress.style.color = '#ff4444';
setTimeout(() => {
progress.style.display = 'none';
progress.style.color = '';
}, 5000);
}
);
} catch (e) {
progress.textContent = `ERROR: ${e.message}`;
setTimeout(() => { progress.style.display = 'none'; }, 5000);
} finally {
urlInput.disabled = false;
urlInput.value = '';
btn.disabled = false;
btn.textContent = 'FETCH';
}
});
```
4. Enable the URL input on page load (it starts disabled in HTML as a placeholder state, but should be enabled once JS is ready). Add after `wireAudio()`:
```javascript
document.getElementById('yt-url').disabled = false;
```
</action>
<verify>
<automated>docker exec lightsync python -c "
import pathlib
html = pathlib.Path('/app/lightsync/frontend/index.html').read_text()
assert 'yt-url' in html, 'missing yt-url input'
assert 'btn-fetch' in html, 'missing btn-fetch'
js = pathlib.Path('/app/lightsync/frontend/app.js').read_text()
assert 'fetchWithProgress' in js, 'missing fetchWithProgress'
assert 'renderProgressBar' in js, 'missing renderProgressBar'
assert 'api/audio/fetch' in js, 'missing fetch endpoint call'
css = pathlib.Path('/app/lightsync/frontend/style.css').read_text()
assert 'yt-url-input' in css, 'missing yt-url-input style'
assert 'yt-progress' in css, 'missing yt-progress style'
print('All frontend checks passed')
"</automated>
</verify>
<acceptance_criteria>
- lightsync/frontend/index.html contains `id="yt-url"` input element
- lightsync/frontend/index.html contains `id="btn-fetch"` button element
- lightsync/frontend/index.html contains `id="yt-progress"` span element
- lightsync/frontend/app.js contains `function fetchWithProgress(`
- lightsync/frontend/app.js contains `function renderProgressBar(`
- lightsync/frontend/app.js contains string literal `DOWNLOADING`
- lightsync/frontend/app.js contains `/api/audio/fetch`
- lightsync/frontend/app.js contains `urlInput.disabled = true` (D-02 disable during download)
- lightsync/frontend/app.js contains `audio.src = ` followed by shows path (D-01 auto-load)
- lightsync/frontend/style.css contains `.yt-url-input`
- lightsync/frontend/style.css contains `.yt-progress`
- No rounded corners in new CSS (border-radius must not appear in yt-url-input or yt-progress rules)
</acceptance_criteria>
<done>YouTube URL input field and FETCH button appear in the transport bar. Clicking FETCH downloads audio with a terminal-style progress bar. After download, the audio auto-loads into the player and appears in the file selector.</done>
</task>
</tasks>
<verification>
1. Paste a YouTube URL into the transport bar input, click FETCH — progress bar fills with block characters
2. After download completes, audio loads automatically — waveform and beats appear on timeline
3. The downloaded file appears in the audio file selector dropdown
4. Entering an invalid URL shows an error message that auto-clears
5. During download, the URL input and FETCH button are disabled
</verification>
<success_criteria>
- YouTube URL → audio file download works end-to-end
- Terminal-style progress bar displays during download
- Downloaded audio enters existing pipeline (waveform + beats + show saving)
- yt-dlp and anthropic packages available in Docker container
</success_criteria>
<output>
After completion, create `.planning/phases/06-ai-sync/06-01-SUMMARY.md`
</output>

632
.planning/phases/06-ai-sync/06-02-PLAN.md Normal file
View File

@@ -0,0 +1,632 @@
---
phase: 06-ai-sync
plan: 02
type: execute
wave: 2
depends_on:
- 06-01
files_modified:
- lightsync/audio/features.py
- lightsync/audio/segments.py
- lightsync/api/audio.py
- lightsync/main.py
- lightsync/frontend/timeline/segments.js
- lightsync/frontend/timeline/timeline.js
- lightsync/frontend/app.js
autonomous: true
requirements:
- SYNC-03
must_haves:
truths:
- "After loading audio, structural sections appear as translucent colored bands on the waveform canvas"
- "Section labels SEC 1, SEC 2, etc. display in monospace uppercase at the top-left of each band"
- "Clicking a section band selects it with an accent-color border highlight"
- "Clicking outside a section deselects it"
- "Sections alternate between two colors from the existing palette"
artifacts:
- path: "lightsync/audio/features.py"
provides: "Onset, chroma, RMS feature extraction"
exports: ["extract_features", "extract_features_async"]
- path: "lightsync/audio/segments.py"
provides: "Structural segmentation via librosa agglomerative clustering"
exports: ["detect_segments", "detect_segments_async"]
- path: "lightsync/api/audio.py"
provides: "/api/audio/segments endpoint"
contains: "async def get_segments"
- path: "lightsync/frontend/timeline/segments.js"
provides: "SegmentOverlay class for canvas rendering and click interaction"
exports: ["SegmentOverlay"]
key_links:
- from: "lightsync/frontend/timeline/timeline.js"
to: "lightsync/frontend/timeline/segments.js"
via: "import SegmentOverlay, call overlay.draw() in render()"
pattern: "SegmentOverlay"
- from: "lightsync/frontend/app.js"
to: "/api/audio/segments"
via: "fetch after loadedmetadata"
pattern: "api/audio/segments"
- from: "lightsync/api/audio.py"
to: "lightsync/audio/segments.py"
via: "detect_segments_async"
pattern: "detect_segments_async"
---
<objective>
Audio feature extraction, structural segmentation, and waveform segment overlay UI.
Purpose: Provides the audio analysis foundation (onsets, chroma, RMS, segments) that the AI sync engine (Plan 06-03) consumes, plus the visible segment overlay on the waveform canvas that enables per-section AI fill (D-11).
Output: `features.py` and `segments.py` backend modules, `/api/audio/segments` endpoint, and interactive segment band overlay on the timeline canvas.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-ai-sync/06-CONTEXT.md
@.planning/phases/06-ai-sync/06-RESEARCH.md
@.planning/phases/06-ai-sync/06-01-SUMMARY.md
<interfaces>
<!-- Existing interfaces the executor needs -->
From lightsync/audio/beats.py:
```python
def analyze(path: str) -> dict[str, Any]:
"""Returns {"tempo": float, "beats": [float, ...]}"""
import librosa
y, sr = librosa.load(path, sr=None, mono=True)
...
async def analyze_async(path: str) -> dict[str, Any]:
return await asyncio.to_thread(analyze, path)
```
From lightsync/main.py (lifespan):
```python
app.state.beats = {}
# Add app.state.segments = {} here
```
From lightsync/api/audio.py (after Plan 06-01):
```python
router = APIRouter()
SHOWS_DIR = Path("/app/shows")
# Existing endpoints: /files, /upload, /beats, /waveform, /fetch
```
From lightsync/frontend/timeline/timeline.js:
```javascript
class TimelineCanvas {
constructor(canvasEl, audioEl) { ... }
render() {
// 1. Background
// 2. Track row banding (lines 117-121)
// >>> INSERT segment bands HERE (between step 2 and step 3) <<<
// 3. Waveform background (lines 124-136)
// 4. Beat marks (lines 139-177)
// 5. Block rendering (lines 180+)
}
timeToX(t) { ... }
xToTime(x) { ... }
async loadWaveform(audioPath) { ... }
async loadBeats(audioPath) { ... }
}
export { TimelineCanvas };
```
Constants from timeline.js:
```javascript
const TRACK_HEIGHT = 48;
const HEADER_WIDTH = 140;
const TIME_AXIS_HEIGHT = 24;
```
From lightsync/frontend/style.css (CSS variables):
```css
--accent: #00ffff;
--border-dim: #1a4a4a;
--text-dim: #555555;
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: Feature extraction and segmentation backend modules + API</name>
<files>lightsync/audio/features.py, lightsync/audio/segments.py, lightsync/api/audio.py, lightsync/main.py</files>
<read_first>
- lightsync/audio/beats.py (pattern for librosa analysis + async wrapper)
- lightsync/api/audio.py (existing endpoints, router pattern, app.state.beats cache pattern)
- lightsync/main.py (lifespan — where to add app.state.segments cache)
</read_first>
<action>
**Create `lightsync/audio/features.py`** — Feature extraction module following the `beats.py` async pattern. All librosa calls use deferred import.
```python
"""Audio feature extraction: onsets, chroma, RMS energy (Phase 6).
extract_features(path) -> {
"onset_times": [float, ...],
"rms_1s": [float, ...],
"chroma_1s": [int, ...],
"duration": float,
"sr": int,
}
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import Any
def extract_features(path: str) -> dict[str, Any]:
"""Extract onsets, chroma, RMS from audio file.
Args:
path: Absolute path to audio file.
Returns:
Dictionary with onset_times (seconds), rms_1s (per-second mean RMS),
chroma_1s (per-second dominant pitch class 0-11), duration, sr.
"""
import librosa
import numpy as np
p = Path(path)
if not p.exists():
raise FileNotFoundError(f"Audio file not found: {path}")
y, sr = librosa.load(path, sr=None, mono=True)
# Onsets — note attacks / transients
onset_frames = librosa.onset.onset_detect(y=y, sr=sr, units='frames')
onset_times = librosa.frames_to_time(onset_frames, sr=sr).tolist()
# Chroma — tonal/harmonic content (CQT-based)
chroma = librosa.feature.chroma_cqt(y=y, sr=sr) # shape (12, T)
# RMS energy — loudness envelope
rms = librosa.feature.rms(y=y)[0] # shape (T,)
# Reduce to 1s resolution for downstream consumers (heuristic + LLM)
hop_length = 512
frame_rate = sr / hop_length
duration = len(y) / sr
n_seconds = int(duration)
rms_1s = []
chroma_1s = []
for i in range(n_seconds):
start_frame = int(i * frame_rate)
end_frame = int((i + 1) * frame_rate)
# RMS mean for this second
rms_slice = rms[start_frame:min(end_frame, len(rms))]
rms_1s.append(float(np.mean(rms_slice)) if len(rms_slice) > 0 else 0.0)
# Dominant pitch class for this second
chroma_slice = chroma[:, start_frame:min(end_frame, chroma.shape[1])]
if chroma_slice.shape[1] > 0:
chroma_1s.append(int(np.argmax(np.mean(chroma_slice, axis=1))))
else:
chroma_1s.append(0)
return {
"onset_times": onset_times,
"rms_1s": rms_1s,
"chroma_1s": chroma_1s,
"duration": duration,
"sr": sr,
}
async def extract_features_async(path: str) -> dict[str, Any]:
"""Async wrapper — runs in thread pool."""
return await asyncio.to_thread(extract_features, path)
```
**Create `lightsync/audio/segments.py`** — Structural segmentation module.
```python
"""Structural segmentation via librosa agglomerative clustering (Phase 6).
detect_segments(path, n_segments=None) -> [{"start": float, "end": float, "label": str}, ...]
Uses chroma features + recurrence matrix + agglomerative clustering.
Target 4-12 sections for a typical 3-5 minute song.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import Any
def detect_segments(path: str, n_segments: int | None = None) -> list[dict[str, Any]]:
"""Detect structural sections using agglomerative clustering on chroma features.
Args:
path: Absolute path to audio file.
n_segments: Number of segments (default: auto based on duration, ~1 per 30s).
Returns:
List of dicts: [{"start": float, "end": float, "label": "SEC 1"}, ...]
"""
import librosa
import numpy as np
p = Path(path)
if not p.exists():
raise FileNotFoundError(f"Audio file not found: {path}")
y, sr = librosa.load(path, sr=None, mono=True)
duration = len(y) / sr
# Auto-select segment count if not specified: ~1 per 30s, clamped to 4-12
if n_segments is None:
n_segments = max(4, min(12, int(duration / 30)))
# Build chroma feature matrix
chroma = librosa.feature.chroma_cqt(y=y, sr=sr, bins_per_octave=36)
# Recurrence matrix — similarity structure
R = librosa.segment.recurrence_matrix(chroma, width=3, mode='affinity', sym=True)
# Filter recurrence matrix to enhance diagonal consistency
R_filtered = librosa.segment.path_enhance(R, 15)
# Agglomerative clustering — returns boundary frame indices
bounds = librosa.segment.agglomerative(R_filtered, k=n_segments)
bound_times = librosa.frames_to_time(bounds, sr=sr).tolist()
# Build section list: [{start, end, label}, ...]
# Pitfall 2 from RESEARCH: agglomerative returns k boundaries, yielding k+1 sections
all_times = [0.0] + bound_times + [duration]
sections = []
for i in range(len(all_times) - 1):
sections.append({
"start": all_times[i],
"end": all_times[i + 1],
"label": f"SEC {i + 1}",
})
return sections
async def detect_segments_async(path: str, n_segments: int | None = None) -> list[dict[str, Any]]:
"""Async wrapper — runs in thread pool."""
return await asyncio.to_thread(detect_segments, path, n_segments)
```
**Edit `lightsync/main.py`** — Add `app.state.segments = {}` in the lifespan function, right after `app.state.beats = {}` (line 27):
```python
app.state.segments = {}
```
**Edit `lightsync/api/audio.py`** — Add segments endpoint. Add import at top:
```python
from lightsync.audio.segments import detect_segments_async
from lightsync.audio.features import extract_features_async
```
Add endpoints after the existing `get_waveform` endpoint:
```python
@router.get("/segments")
async def get_segments(request: Request, path: str, n_segments: int | None = None):
"""Return structural segments for an audio file. Cached in app.state.segments."""
segments_cache: dict = getattr(request.app.state, "segments", None) or {}
cache_key = f"{path}:{n_segments or 'auto'}"
if cache_key not in segments_cache:
p = Path(path)
if not p.exists():
raise HTTPException(status_code=404, detail=f"File not found: {path}")
try:
segments = await detect_segments_async(path, n_segments)
if not hasattr(request.app.state, "segments") or request.app.state.segments is None:
request.app.state.segments = {}
request.app.state.segments[cache_key] = segments
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Segmentation failed: {exc}") from exc
else:
segments = segments_cache[cache_key]
return {"segments": segments, "count": len(segments), "file": path}
@router.get("/features")
async def get_features(request: Request, path: str):
"""Return extracted audio features (onsets, chroma, RMS) for a file."""
p = Path(path)
if not p.exists():
raise HTTPException(status_code=404, detail=f"File not found: {path}")
try:
features = await extract_features_async(path)
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Feature extraction failed: {exc}") from exc
return {**features, "file": path}
```
</action>
<verify>
<automated>docker exec lightsync python -c "
from lightsync.audio.features import extract_features, extract_features_async
from lightsync.audio.segments import detect_segments, detect_segments_async
print('features.py OK')
print('segments.py OK')
from lightsync.api.audio import get_segments, get_features
print('API endpoints OK')
"</automated>
</verify>
<acceptance_criteria>
- lightsync/audio/features.py exists and contains `def extract_features(path: str)`
- lightsync/audio/features.py contains `async def extract_features_async`
- lightsync/audio/features.py contains `librosa.onset.onset_detect`
- lightsync/audio/features.py contains `librosa.feature.chroma_cqt`
- lightsync/audio/features.py contains `librosa.feature.rms`
- lightsync/audio/features.py returns keys `onset_times`, `rms_1s`, `chroma_1s`, `duration`, `sr`
- lightsync/audio/segments.py exists and contains `def detect_segments(path: str`
- lightsync/audio/segments.py contains `async def detect_segments_async`
- lightsync/audio/segments.py contains `librosa.segment.agglomerative`
- lightsync/audio/segments.py contains `librosa.segment.recurrence_matrix`
- lightsync/audio/segments.py contains `SEC {i + 1}` (D-10 generic numbering)
- lightsync/audio/segments.py contains `max(4, min(12,` (auto segment count heuristic)
- lightsync/api/audio.py contains `async def get_segments`
- lightsync/api/audio.py contains `async def get_features`
- lightsync/api/audio.py contains `from lightsync.audio.segments import detect_segments_async`
- lightsync/api/audio.py contains `from lightsync.audio.features import extract_features_async`
- lightsync/main.py contains `app.state.segments = {}`
</acceptance_criteria>
<done>Backend feature extraction and segmentation modules exist with async wrappers. API endpoints /api/audio/segments and /api/audio/features serve cached analysis results. Segments cache initialized in app lifespan.</done>
</task>
<task type="auto">
<name>Task 2: Segment overlay canvas rendering and click interaction</name>
<files>lightsync/frontend/timeline/segments.js, lightsync/frontend/timeline/timeline.js, lightsync/frontend/app.js</files>
<read_first>
- lightsync/frontend/timeline/timeline.js (render() method — steps 1-5, timeToX, constructor, interaction handlers, loadWaveform/loadBeats patterns)
- lightsync/frontend/app.js (initTimeline function — loadedmetadata handler, timeline instance usage)
- lightsync/frontend/style.css (CSS variables for colors)
</read_first>
<action>
**Create `lightsync/frontend/timeline/segments.js`** — SegmentOverlay class that manages segment data and canvas rendering.
```javascript
// SegmentOverlay — draws structural section bands on the timeline canvas (Phase 6)
// Bands are translucent, drawn below waveform. Clicking selects a section.
const BAND_COLORS = [
'rgba(0,255,255,0.15)', // --accent cyan variant
'rgba(180,0,255,0.15)', // purple complement
'rgba(0,200,200,0.12)', // darker cyan variant
];
class SegmentOverlay {
constructor() {
this.segments = []; // [{start, end, label}, ...]
this.selectedIndex = -1; // -1 = no selection
this.loading = false;
this.error = null;
}
/**
* Load segments from the API.
* @param {string} audioPath — absolute path to audio file (e.g. /app/shows/file.mp3)
*/
async load(audioPath) {
this.loading = true;
this.error = null;
this.segments = [];
this.selectedIndex = -1;
try {
const res = await fetch(`/api/audio/segments?path=${encodeURIComponent(audioPath)}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
this.segments = data.segments || [];
} catch (e) {
this.error = `SEGMENTATION FAILED: ${e.message}`;
console.warn('[segments]', e);
} finally {
this.loading = false;
}
}
/**
* Get the currently selected segment, or null.
* @returns {{start: number, end: number, label: string} | null}
*/
getSelected() {
if (this.selectedIndex >= 0 && this.selectedIndex < this.segments.length) {
return this.segments[this.selectedIndex];
}
return null;
}
/**
* Handle a click at a given time position. Selects the segment containing that time,
* or deselects if clicking outside any segment.
* @param {number} time — time in seconds
* @returns {boolean} — true if selection changed
*/
handleClick(time) {
const prev = this.selectedIndex;
this.selectedIndex = -1;
for (let i = 0; i < this.segments.length; i++) {
if (time >= this.segments[i].start && time < this.segments[i].end) {
this.selectedIndex = i;
break;
}
}
return this.selectedIndex !== prev;
}
/**
* Draw segment bands on the canvas context.
* Must be called BETWEEN track row banding and waveform in the render pipeline.
*
* @param {CanvasRenderingContext2D} ctx
* @param {number} W — canvas logical width
* @param {number} H — canvas logical height
* @param {number} trackCount — number of tracks
* @param {Function} timeToX — converts time (seconds) to canvas X coordinate
*/
draw(ctx, W, H, trackCount, timeToX) {
if (this.segments.length === 0) return;
const totalH = trackCount * 48; // TRACK_HEIGHT
const topY = 24; // TIME_AXIS_HEIGHT
const headerW = 140; // HEADER_WIDTH
for (let i = 0; i < this.segments.length; i++) {
const seg = this.segments[i];
const x1 = timeToX(seg.start);
const x2 = timeToX(seg.end);
// Skip segments entirely off-screen
if (x2 < headerW || x1 > W) continue;
// Clamp to visible area
const drawX1 = Math.max(x1, headerW);
const drawX2 = Math.min(x2, W);
const drawW = drawX2 - drawX1;
const isSelected = this.selectedIndex === i;
// D-09: Translucent colored band, full waveform height, alternating colors
ctx.fillStyle = BAND_COLORS[i % BAND_COLORS.length];
ctx.fillRect(drawX1, topY, drawW, totalH);
// D-11: Selected section gets accent-color border highlight
if (isSelected) {
ctx.strokeStyle = 'rgba(0,255,255,1.0)';
ctx.lineWidth = 1;
ctx.strokeRect(drawX1, topY, drawW, totalH);
}
// D-10: Label — small monospace uppercase at top-left of band
if (drawW > 30) {
ctx.fillStyle = 'rgba(0,255,255,0.6)';
ctx.font = '9px monospace';
ctx.fillText(seg.label, Math.max(x1 + 3, headerW + 2), topY + 10);
}
}
}
}
export { SegmentOverlay };
```
**Edit `lightsync/frontend/timeline/timeline.js`** — Integrate SegmentOverlay into the timeline.
1. Add import at the top (after existing imports, around line 7):
```javascript
import { SegmentOverlay } from './segments.js';
```
2. In the `constructor()` method, add after `this.beatGrid = new BeatGrid();` (around line 30):
```javascript
this.segmentOverlay = new SegmentOverlay();
```
3. In the `render()` method, insert segment band drawing BETWEEN step 2 (track row banding, after the `forEach` at line ~121) and step 3 (waveform background, starting at line ~124). Add this code:
```javascript
// 2.5 Segment overlay bands (drawn below waveform)
if (this.segmentOverlay.segments.length > 0) {
this.segmentOverlay.draw(ctx, W, H, this.tracks.length, (t) => this.timeToX(t));
}
```
4. Add a `loadSegments` method after `loadBeats` (around line 624):
```javascript
async loadSegments(audioPath) {
await this.segmentOverlay.load(audioPath);
}
```
5. In the existing `_setupInteractions()` method, add segment click handling in the mousedown handler. In the existing mousedown logic, BEFORE the block selection/move/resize logic, add:
```javascript
// Check for segment click (only if click is in track area, not on a block)
// This should be checked and the overlay updated; block interactions take priority
```
Actually, segment selection should happen on clicks that do NOT land on a block. The existing mousedown handler already checks for block hits. Add segment overlay click handling in the existing mousedown flow — after the block hit detection fails (no block found), call:
```javascript
const clickTime = this.xToTime(mx);
this.segmentOverlay.handleClick(clickTime);
```
This ensures: click on block = select block (existing behavior), click on empty track area = select segment.
**Edit `lightsync/frontend/app.js`** — Load segments when audio loads.
In the `initTimeline()` function, inside the `audio.addEventListener('loadedmetadata', ...)` callback (around line 404-417), add after `timeline.loadBeats(path).then(...)`:
```javascript
// Load structural segments for overlay
timeline.loadSegments(path);
```
This triggers segment detection (or cache hit) when audio loads, displaying the overlay as soon as analysis completes.
</action>
<verify>
<automated>docker exec lightsync python -c "
import pathlib
seg_js = pathlib.Path('/app/lightsync/frontend/timeline/segments.js').read_text()
assert 'class SegmentOverlay' in seg_js, 'missing SegmentOverlay class'
assert 'export { SegmentOverlay }' in seg_js, 'missing export'
assert 'BAND_COLORS' in seg_js, 'missing BAND_COLORS'
assert 'handleClick' in seg_js, 'missing handleClick'
tl_js = pathlib.Path('/app/lightsync/frontend/timeline/timeline.js').read_text()
assert 'SegmentOverlay' in tl_js, 'missing SegmentOverlay import'
assert 'segmentOverlay' in tl_js, 'missing segmentOverlay instance'
assert 'loadSegments' in tl_js, 'missing loadSegments method'
app_js = pathlib.Path('/app/lightsync/frontend/app.js').read_text()
assert 'loadSegments' in app_js, 'missing loadSegments call'
print('All segment overlay checks passed')
"</automated>
</verify>
<acceptance_criteria>
- lightsync/frontend/timeline/segments.js exists and contains `class SegmentOverlay`
- lightsync/frontend/timeline/segments.js contains `export { SegmentOverlay }`
- lightsync/frontend/timeline/segments.js contains `BAND_COLORS` with at least 2 colors
- lightsync/frontend/timeline/segments.js contains `rgba(0,255,255,0.15)` (cyan band)
- lightsync/frontend/timeline/segments.js contains `rgba(180,0,255,0.15)` (purple band)
- lightsync/frontend/timeline/segments.js contains `handleClick(time)` method
- lightsync/frontend/timeline/segments.js contains `getSelected()` method
- lightsync/frontend/timeline/segments.js contains `seg.label` (renders D-10 labels)
- lightsync/frontend/timeline/segments.js contains `9px monospace` (small monospace font for labels)
- lightsync/frontend/timeline/timeline.js contains `import { SegmentOverlay }` from `./segments.js`
- lightsync/frontend/timeline/timeline.js contains `this.segmentOverlay = new SegmentOverlay()`
- lightsync/frontend/timeline/timeline.js contains `segmentOverlay.draw(` call in render()
- lightsync/frontend/timeline/timeline.js contains `async loadSegments(audioPath)`
- lightsync/frontend/timeline/timeline.js render() calls segmentOverlay.draw BEFORE waveform drawing
- lightsync/frontend/app.js contains `loadSegments(path)` in the loadedmetadata handler
</acceptance_criteria>
<done>Segment bands appear as translucent colored overlays on the waveform canvas when audio is loaded. Clicking a band selects it (highlighted border). Clicking outside deselects. Labels show SEC 1, SEC 2, etc.</done>
</task>
</tasks>
<verification>
1. Load an audio file — after a few seconds, colored section bands appear on the waveform canvas
2. Bands alternate between cyan and purple translucent fills
3. Each band shows its label (SEC 1, SEC 2, ...) in small monospace at top-left
4. Clicking a band highlights it with a 1px cyan border
5. Clicking empty space deselects the band
6. GET /api/audio/segments?path=/app/shows/file.mp3 returns JSON with segments array
7. GET /api/audio/features?path=/app/shows/file.mp3 returns onset_times, rms_1s, chroma_1s
</verification>
<success_criteria>
- Structural segmentation detects 4-12 sections from audio
- Segment bands render below waveform in correct Z-order
- Section selection works via click interaction
- Features API returns onsets, chroma, RMS at 1s resolution
- All analysis runs in asyncio.to_thread, never blocks the event loop
</success_criteria>
<output>
After completion, create `.planning/phases/06-ai-sync/06-02-SUMMARY.md`
</output>

909
.planning/phases/06-ai-sync/06-03-PLAN.md Normal file
View File

@@ -0,0 +1,909 @@
---
phase: 06-ai-sync
plan: 03
type: execute
wave: 3
depends_on:
- 06-01
- 06-02
files_modified:
- lightsync/api/sync.py
- lightsync/main.py
- lightsync/frontend/index.html
- lightsync/frontend/app.js
- lightsync/frontend/style.css
autonomous: true
requirements:
- SYNC-03
must_haves:
truths:
- "Clicking GENERATE in the AI Sync panel fills the timeline with animation blocks matched to beats, sections, and energy levels"
- "The heuristic mapper produces a playable show without any LLM dependency"
- "High-energy sections get fast animations (strobe, chase); low-energy sections get slow animations (solid, rainbow)"
- "Block colors are derived from chroma analysis (pitch class to HSL hue)"
- "If the timeline has existing blocks, a confirmation dialog asks before replacing"
- "When a section is selected in the segment overlay, AI Sync fills only that section range"
- "The USE LLM toggle is disabled when ANTHROPIC_API_KEY is not configured"
artifacts:
- path: "lightsync/api/sync.py"
provides: "POST /api/sync/generate endpoint with heuristic + LLM paths"
exports: ["router"]
contains: "def heuristic_generate"
- path: "lightsync/frontend/index.html"
provides: "AI Sync panel with feature toggles and GENERATE button"
contains: "ai-sync-panel"
- path: "lightsync/frontend/app.js"
provides: "AI Sync panel wiring — generate button, LLM toggle, section range"
contains: "ai-sync"
key_links:
- from: "lightsync/frontend/app.js"
to: "/api/sync/generate"
via: "fetch POST with features config"
pattern: "api/sync/generate"
- from: "lightsync/api/sync.py"
to: "lightsync/audio/features.py"
via: "extract_features_async"
pattern: "extract_features_async"
- from: "lightsync/api/sync.py"
to: "lightsync/audio/segments.py"
via: "detect_segments_async"
pattern: "detect_segments_async"
- from: "lightsync/frontend/app.js"
to: "timeline.tracks"
via: "populate track cues from API response"
pattern: "track\\.cues"
---
<objective>
AI-assisted show generation — heuristic rule mapper and optional LLM path, with dedicated AI Sync panel UI.
Purpose: Fulfills SYNC-03 — auto-fill the timeline with animation blocks from audio analysis. The heuristic base always works; the LLM path provides a richer alternative when configured. This is the capstone feature of Phase 6.
Output: `/api/sync/generate` endpoint, AI Sync panel in the UI, working heuristic show generator, optional LLM show generator.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-ai-sync/06-CONTEXT.md
@.planning/phases/06-ai-sync/06-RESEARCH.md
@.planning/phases/06-ai-sync/06-01-SUMMARY.md
@.planning/phases/06-ai-sync/06-02-SUMMARY.md
<interfaces>
<!-- Key types and contracts the executor needs -->
From lightsync/audio/features.py (created in 06-02):
```python
def extract_features(path: str) -> dict:
"""Returns {"onset_times": [...], "rms_1s": [...], "chroma_1s": [...], "duration": float, "sr": int}"""
async def extract_features_async(path: str) -> dict:
...
```
From lightsync/audio/segments.py (created in 06-02):
```python
def detect_segments(path: str, n_segments: int | None = None) -> list[dict]:
"""Returns [{"start": float, "end": float, "label": "SEC 1"}, ...]"""
async def detect_segments_async(path: str, n_segments: int | None = None) -> list[dict]:
...
```
From lightsync/audio/beats.py:
```python
def analyze(path: str) -> dict:
"""Returns {"tempo": float, "beats": [float, ...]}"""
async def analyze_async(path: str) -> dict:
...
```
From lightsync/models/show.py:
```python
class CueModel(BaseModel):
id: UUID = Field(default_factory=uuid4)
timestamp: float
duration: float = 4.0
mode: Literal["animation", "frame_sequence"] = "animation"
animation: str | None = None
params: dict[str, Any] = Field(default_factory=dict)
```
From lightsync/frontend/timeline/timeline.js:
```javascript
class TimelineCanvas {
tracks: [{device_id, device_name, strip_type, cues: []}]
history: CommandHistory
segmentOverlay: SegmentOverlay // from 06-02
}
```
From lightsync/frontend/timeline/segments.js (created in 06-02):
```javascript
class SegmentOverlay {
getSelected() -> {start, end, label} | null
}
```
From lightsync/frontend/timeline/commands.js:
```javascript
class PlaceBlockCommand { ... }
// Used for undo/redo — but AI sync replaces all blocks, so we need a bulk replace approach
```
From lightsync/frontend/app.js:
```javascript
const client = new LightSyncClient(); // WebSocket client
let timeline = null; // TimelineCanvas instance
// Animation types available: chase, pulse, rainbow, strobe, color_wipe, fire, solid
```
From lightsync/main.py:
```python
def create_app() -> FastAPI:
from lightsync.api import shows, devices, ws, audio, timeline
app.include_router(audio.router, prefix="/api/audio")
# Add: app.include_router(sync.router, prefix="/api/sync")
```
</interfaces>
</context>
<tasks>
<task type="auto">
<name>Task 1: AI sync backend — heuristic mapper + LLM generator + API endpoint</name>
<files>lightsync/api/sync.py, lightsync/main.py</files>
<read_first>
- lightsync/audio/features.py (extract_features_async return format)
- lightsync/audio/segments.py (detect_segments_async return format)
- lightsync/audio/beats.py (analyze_async return format)
- lightsync/models/show.py (CueModel fields: timestamp, duration, animation, params)
- lightsync/main.py (router registration pattern, lifespan)
- lightsync/api/audio.py (endpoint patterns, app.state cache access)
</read_first>
<action>
**Create `lightsync/api/sync.py`** — New API router with the AI sync generation endpoint.
```python
"""AI-assisted show generation endpoint (Phase 6, SYNC-03).
POST /api/sync/generate — generates animation blocks from audio analysis.
Two modes: heuristic (always available) and LLM (requires ANTHROPIC_API_KEY).
"""
from __future__ import annotations
import asyncio
import colorsys
import logging
import os
from typing import Any
from uuid import uuid4
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from lightsync.audio.beats import analyze_async
from lightsync.audio.features import extract_features_async
from lightsync.audio.segments import detect_segments_async
logger = logging.getLogger(__name__)
router = APIRouter()
ANIMATION_TYPES = ['chase', 'pulse', 'rainbow', 'strobe', 'color_wipe', 'fire', 'solid']
class SyncGenerateRequest(BaseModel):
path: str # audio file path
device_ids: list[str] # device UUIDs to generate for
use_llm: bool = False # optional LLM generation
section_range: list[float] | None = None # [start, end] to fill only a section (D-11)
class GeneratedBlock(BaseModel):
id: str
device_id: str
timestamp: float
duration: float
animation: str
params: dict[str, Any]
def _chroma_to_rgb(pitch_class: int) -> str:
"""Convert pitch class (0-11) to hex RGB color via HSL mapping.
Pitch class 0 (C) = 0deg hue, each class advances 30deg (360/12).
"""
hue = pitch_class / 12.0
r, g, b = colorsys.hls_to_rgb(hue, 0.5, 0.9)
return f"#{int(r*255):02x}{int(g*255):02x}{int(b*255):02x}"
def _intensity_to_animation(rms_mean: float, rms_max: float) -> str:
"""Map RMS intensity level to animation type.
High energy -> fast/flashy, low energy -> slow/ambient.
"""
if rms_max == 0:
return 'solid'
ratio = rms_mean / rms_max
if ratio > 0.7:
# High energy — fast animations
return 'strobe' if ratio > 0.85 else 'chase'
elif ratio > 0.4:
# Medium energy
return 'pulse' if ratio > 0.55 else 'color_wipe'
else:
# Low energy — ambient
return 'rainbow' if ratio > 0.2 else 'solid'
def _speed_from_tempo(tempo: float, intensity: float) -> float:
"""Derive animation speed from tempo and intensity. Range: 0.5 - 5.0."""
base = tempo / 60.0 # beats per second
return max(0.5, min(5.0, base * (0.5 + intensity)))
def heuristic_generate(
beats: list[float],
tempo: float,
features: dict[str, Any],
segments: list[dict[str, Any]],
device_ids: list[str],
section_range: list[float] | None = None,
) -> list[dict[str, Any]]:
"""Generate animation blocks using heuristic rules.
Algorithm per RESEARCH.md Pattern 5:
1. For each section: compute mean RMS -> intensity, dominant chroma -> color
2. Intensity maps to animation type (high=strobe/chase, medium=pulse/wipe, low=solid/rainbow)
3. Duration: snap to beat grid — each block spans from beat[i] to beat[i+N]
4. Color: chroma pitch class -> HSL hue -> RGB
Args:
beats: beat timestamps in seconds
tempo: BPM
features: {"onset_times", "rms_1s", "chroma_1s", "duration"}
segments: [{"start", "end", "label"}, ...]
device_ids: list of device UUID strings
section_range: optional [start, end] to restrict generation
Returns:
List of block dicts: [{"id", "device_id", "timestamp", "duration", "animation", "params"}, ...]
"""
rms_1s = features.get('rms_1s', [])
chroma_1s = features.get('chroma_1s', [])
duration = features.get('duration', 0)
rms_max = max(rms_1s) if rms_1s else 1.0
# Filter beats and segments by section_range if specified
if section_range and len(section_range) == 2:
range_start, range_end = section_range
beats_in_range = [b for b in beats if range_start <= b < range_end]
segments_in_range = [s for s in segments if s['end'] > range_start and s['start'] < range_end]
else:
range_start, range_end = 0.0, duration
beats_in_range = beats
segments_in_range = segments
blocks = []
for seg in segments_in_range:
seg_start = max(seg['start'], range_start)
seg_end = min(seg['end'], range_end)
# Compute section-level features
sec_start_idx = int(seg_start)
sec_end_idx = min(int(seg_end), len(rms_1s))
if sec_end_idx <= sec_start_idx:
continue
rms_section = rms_1s[sec_start_idx:sec_end_idx]
chroma_section = chroma_1s[sec_start_idx:sec_end_idx] if chroma_1s else [0]
rms_mean = sum(rms_section) / len(rms_section) if rms_section else 0
dominant_chroma = max(set(chroma_section), key=chroma_section.count) if chroma_section else 0
# Derive animation type and color
anim_type = _intensity_to_animation(rms_mean, rms_max)
color = _chroma_to_rgb(dominant_chroma)
intensity = rms_mean / rms_max if rms_max > 0 else 0.5
speed = _speed_from_tempo(tempo, intensity)
# Get beats within this section
section_beats = [b for b in beats_in_range if seg_start <= b < seg_end]
if len(section_beats) < 2:
# Too few beats — place one block spanning the section
for device_id in device_ids:
blocks.append({
"id": str(uuid4()),
"device_id": device_id,
"timestamp": seg_start,
"duration": seg_end - seg_start,
"animation": anim_type,
"params": {"color": color, "speed": round(speed, 2)},
})
continue
# Group beats into blocks — N beats per block based on density
# Dense sections (>3 beats/sec) = shorter blocks, sparse = longer blocks
density = len(section_beats) / (seg_end - seg_start) if seg_end > seg_start else 1
beats_per_block = max(2, min(8, int(8 / max(density, 0.5))))
i = 0
while i < len(section_beats):
block_start = section_beats[i]
end_idx = min(i + beats_per_block, len(section_beats) - 1)
block_end = section_beats[end_idx] if end_idx > i else block_start + 2.0
block_duration = max(0.5, block_end - block_start)
for device_id in device_ids:
blocks.append({
"id": str(uuid4()),
"device_id": device_id,
"timestamp": round(block_start, 3),
"duration": round(block_duration, 3),
"animation": anim_type,
"params": {"color": color, "speed": round(speed, 2)},
})
i += beats_per_block
return blocks
async def llm_generate(
beats: list[float],
tempo: float,
features: dict[str, Any],
segments: list[dict[str, Any]],
device_ids: list[str],
section_range: list[float] | None = None,
) -> list[dict[str, Any]]:
"""Generate animation blocks via Anthropic LLM (D-06: generates from scratch, not from heuristic).
Lazy-initializes the Anthropic client to avoid startup crash if API key is missing (Pitfall 4).
"""
import json as json_mod
key = os.environ.get('ANTHROPIC_API_KEY')
if not key:
raise HTTPException(status_code=503, detail='ANTHROPIC_API_KEY not configured')
try:
import anthropic
except ImportError:
raise HTTPException(status_code=503, detail='anthropic package not installed')
client = anthropic.Anthropic(api_key=key)
# Build compact feature payload for LLM (D-06)
rms_1s = features.get('rms_1s', [])
chroma_1s = features.get('chroma_1s', [])
# Filter by section range if specified
if section_range and len(section_range) == 2:
range_start, range_end = section_range
filtered_segments = [s for s in segments if s['end'] > range_start and s['start'] < range_end]
filtered_beats = [b for b in beats if range_start <= b < range_end]
else:
range_start = 0.0
range_end = features.get('duration', 0)
filtered_segments = segments
filtered_beats = beats
feature_payload = {
"tempo_bpm": tempo,
"duration_seconds": range_end - range_start,
"time_range": {"start": range_start, "end": range_end},
"beat_times": filtered_beats[:200], # cap for token budget
"segments": filtered_segments,
"rms_1s": rms_1s[int(range_start):int(range_end)],
"chroma_1s": chroma_1s[int(range_start):int(range_end)],
"device_ids": device_ids,
"available_animations": ANIMATION_TYPES,
}
prompt = f"""You are a music-to-light show generator. Given audio analysis features, generate a list of LED animation blocks that sync with the music.
INPUT FEATURES:
{json_mod.dumps(feature_payload, indent=2)}
RULES:
- Output a JSON array of block objects
- Each block: {{"device_id": "<from device_ids>", "timestamp": <seconds>, "duration": <seconds>, "animation": "<from available_animations>", "params": {{"color": "#rrggbb", "speed": <0.5-5.0>}}}}
- High-energy sections (high RMS) should use strobe, chase (fast)
- Low-energy sections should use solid, rainbow (slow)
- Blocks should align to beat timestamps where possible
- Generate blocks for ALL devices in device_ids
- Colors should reflect the harmonic content (chroma values)
- Ensure full coverage of the time range with no large gaps
OUTPUT: Only the JSON array, no explanation."""
def _call_api():
return client.messages.create(
model="claude-3-haiku-20240307",
max_tokens=4096,
messages=[{"role": "user", "content": prompt}],
)
response = await asyncio.to_thread(_call_api)
# Parse LLM response
text = response.content[0].text.strip()
# Strip markdown code fences if present
if text.startswith('```'):
text = text.split('\n', 1)[1]
if text.endswith('```'):
text = text.rsplit('```', 1)[0]
text = text.strip()
try:
raw_blocks = json_mod.loads(text)
except json_mod.JSONDecodeError as e:
raise HTTPException(status_code=500, detail=f'LLM returned invalid JSON: {e}')
# Validate and normalize blocks
blocks = []
for rb in raw_blocks:
if not isinstance(rb, dict):
continue
if rb.get('device_id') not in device_ids:
continue
if rb.get('animation') not in ANIMATION_TYPES:
rb['animation'] = 'solid'
blocks.append({
"id": str(uuid4()),
"device_id": rb.get('device_id', device_ids[0]),
"timestamp": float(rb.get('timestamp', 0)),
"duration": float(rb.get('duration', 2.0)),
"animation": rb.get('animation', 'solid'),
"params": rb.get('params', {"color": "#00ffff", "speed": 1.0}),
})
return blocks
@router.post("/generate")
async def generate_sync(request: Request, body: SyncGenerateRequest):
"""Generate animation blocks from audio analysis.
D-04: Heuristic base always runs. LLM is optional (use_llm flag).
D-06: LLM generates from scratch, not from heuristic output.
D-11: section_range limits generation to a time window.
"""
from pathlib import Path
p = Path(body.path)
if not p.exists():
raise HTTPException(status_code=404, detail=f"File not found: {body.path}")
if not body.device_ids:
raise HTTPException(status_code=400, detail="No device_ids provided")
# Gather all analysis data (cached or computed)
try:
beats_data = await analyze_async(body.path)
features = await extract_features_async(body.path)
segments = await detect_segments_async(body.path)
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Audio analysis failed: {exc}") from exc
beats = beats_data.get('beats', [])
tempo = beats_data.get('tempo', 120.0)
if body.use_llm:
blocks = await llm_generate(
beats, tempo, features, segments,
body.device_ids, body.section_range,
)
else:
blocks = heuristic_generate(
beats, tempo, features, segments,
body.device_ids, body.section_range,
)
return {"blocks": blocks, "count": len(blocks), "mode": "llm" if body.use_llm else "heuristic"}
@router.get("/llm-available")
async def llm_available():
"""Check if LLM generation is available (API key configured)."""
key = os.environ.get('ANTHROPIC_API_KEY', '')
return {"available": bool(key)}
```
**Edit `lightsync/main.py`** — Register the sync router. Add import and router registration in `create_app()`.
After the existing `from lightsync.api import shows, devices, ws, audio, timeline` line (around line 45), add:
```python
from lightsync.api import sync
```
After the existing `app.include_router(timeline.router, prefix="/api")` line (around line 50), add:
```python
app.include_router(sync.router, prefix="/api/sync")
```
</action>
<verify>
<automated>docker exec lightsync python -c "
from lightsync.api.sync import router, heuristic_generate, llm_generate, SyncGenerateRequest
print('sync.py imported OK')
# Test heuristic with minimal data
blocks = heuristic_generate(
beats=[0.5, 1.0, 1.5, 2.0, 2.5, 3.0],
tempo=120.0,
features={'onset_times': [0.5, 1.5], 'rms_1s': [0.3, 0.5, 0.7], 'chroma_1s': [0, 4, 7], 'duration': 3.0, 'sr': 22050},
segments=[{'start': 0.0, 'end': 3.0, 'label': 'SEC 1'}],
device_ids=['dev-1'],
)
assert len(blocks) > 0, 'heuristic produced no blocks'
assert all('animation' in b for b in blocks), 'blocks missing animation'
assert all('timestamp' in b for b in blocks), 'blocks missing timestamp'
assert all('params' in b for b in blocks), 'blocks missing params'
print(f'Heuristic generated {len(blocks)} blocks OK')
"</automated>
</verify>
<acceptance_criteria>
- lightsync/api/sync.py exists and contains `router = APIRouter()`
- lightsync/api/sync.py contains `def heuristic_generate(`
- lightsync/api/sync.py contains `async def llm_generate(`
- lightsync/api/sync.py contains `async def generate_sync(`
- lightsync/api/sync.py contains `async def llm_available(`
- lightsync/api/sync.py contains `class SyncGenerateRequest`
- lightsync/api/sync.py contains `section_range: list[float] | None` (D-11 support)
- lightsync/api/sync.py contains `use_llm: bool = False` (D-04 off by default)
- lightsync/api/sync.py contains `ANTHROPIC_API_KEY` environment check (Pitfall 4 guard)
- lightsync/api/sync.py contains `claude-3-haiku` (LLM model selection)
- lightsync/api/sync.py contains `_chroma_to_rgb` (pitch class to color mapping)
- lightsync/api/sync.py contains `_intensity_to_animation` (RMS to animation type mapping)
- lightsync/main.py contains `from lightsync.api import sync` (or equivalent import)
- lightsync/main.py contains `sync.router` in include_router call
- Heuristic generates blocks with valid animation types from ANIMATION_TYPES list
- Heuristic blocks have params with color (hex) and speed (float) keys
</acceptance_criteria>
<done>POST /api/sync/generate produces animation blocks from audio analysis. Heuristic works without LLM. LLM path gated by ANTHROPIC_API_KEY. Section range filtering supported.</done>
</task>
<task type="auto">
<name>Task 2: AI Sync panel UI + frontend wiring</name>
<files>lightsync/frontend/index.html, lightsync/frontend/app.js, lightsync/frontend/style.css</files>
<read_first>
- lightsync/frontend/index.html (full layout — toolbar area, main area, transport bar)
- lightsync/frontend/app.js (timeline instance, initTimeline, client WebSocket, show loading patterns)
- lightsync/frontend/style.css (panel, panel-header, transport-btn, existing toggle patterns)
- lightsync/frontend/timeline/commands.js (PlaceBlockCommand, setCurrentShowId — for undo/redo context)
- lightsync/frontend/timeline/timeline.js (tracks structure, segmentOverlay.getSelected)
</read_first>
<action>
**index.html** — Add AI Sync panel and trigger button. Per D-08: dedicated panel, not in toolbar or transport.
1. In the `#timeline-toolbar` div (around line 53-66), add the AI SYNC button before the closing `</div>`. Insert after the SNAP button:
```html
<div class="toolbar-sep"></div>
<button id="btn-ai-sync-open" class="transport-btn" title="AI-assisted show generation">AI SYNC</button>
```
2. Add the AI Sync panel as a new div AFTER the `#timeline-toolbar` div and BEFORE the `#timeline-canvas` canvas element. This panel is shown/hidden by the button:
```html
<div id="ai-sync-panel" class="ai-sync-panel" style="display:none;">
<div class="ai-sync-header">AI SYNC</div>
<div class="ai-sync-body">
<div class="ai-sync-info" id="ai-sync-info">SELECT AUDIO FILE FIRST</div>
<div class="ai-sync-toggles">
<label class="ai-sync-toggle"><input type="checkbox" id="ai-use-beats" checked> BEATS</label>
<label class="ai-sync-toggle"><input type="checkbox" id="ai-use-onsets" checked> ONSETS</label>
<label class="ai-sync-toggle"><input type="checkbox" id="ai-use-chroma" checked> CHROMA</label>
<label class="ai-sync-toggle"><input type="checkbox" id="ai-use-rms" checked> RMS</label>
</div>
<div class="ai-sync-actions">
<button id="btn-ai-generate" class="transport-btn ai-sync-generate" disabled>GENERATE</button>
<label class="ai-sync-toggle ai-sync-llm-toggle">
<input type="checkbox" id="ai-use-llm" disabled>
<span id="ai-llm-label">USE LLM</span>
</label>
</div>
</div>
</div>
```
**style.css** — Add AI Sync panel styles at the end. Terminal aesthetic per D-08 and context specifics.
```css
/* AI Sync panel (Phase 6) */
.ai-sync-panel {
background: var(--bg-panel-dark);
border: 1px solid var(--border-dim);
margin: 0 4px;
padding: 0;
font-family: var(--font-mono);
}
.ai-sync-header {
background: var(--bg-panel);
padding: 4px 8px;
font-size: 11px;
color: var(--accent);
text-transform: uppercase;
letter-spacing: 1px;
border-bottom: 1px solid var(--border-dim);
}
.ai-sync-body {
padding: 6px 8px;
display: flex;
align-items: center;
gap: 12px;
}
.ai-sync-info {
font-size: 11px;
color: var(--text-dim);
min-width: 140px;
}
.ai-sync-toggles {
display: flex;
gap: 8px;
}
.ai-sync-toggle {
font-size: 11px;
color: var(--text-primary);
cursor: pointer;
display: flex;
align-items: center;
gap: 3px;
}
.ai-sync-toggle input[type="checkbox"] {
accent-color: var(--accent);
}
.ai-sync-actions {
display: flex;
align-items: center;
gap: 8px;
margin-left: auto;
}
.ai-sync-generate {
background: var(--accent);
color: var(--bg-primary);
font-weight: bold;
padding: 3px 12px;
}
.ai-sync-generate:disabled {
opacity: 0.3;
cursor: not-allowed;
}
.ai-sync-llm-toggle {
opacity: 0.4;
}
.ai-sync-llm-toggle:not(.disabled) {
opacity: 1;
}
```
**app.js** — Wire AI Sync panel interactions. Add after the `initTimeline()` function call (end of file, before or after the show selector wiring).
1. Add the AI Sync panel open/close toggle:
```javascript
// ── AI Sync panel ───────────────────────────────────────────────────────────
document.getElementById('btn-ai-sync-open')?.addEventListener('click', () => {
const panel = document.getElementById('ai-sync-panel');
if (!panel) return;
const visible = panel.style.display !== 'none';
panel.style.display = visible ? 'none' : 'block';
});
```
2. Check LLM availability on page load and enable/disable the toggle:
```javascript
fetch('/api/sync/llm-available')
.then(r => r.json())
.then(data => {
const checkbox = document.getElementById('ai-use-llm');
const label = document.getElementById('ai-llm-label');
const container = checkbox?.closest('.ai-sync-llm-toggle');
if (data.available) {
checkbox.disabled = false;
if (container) container.classList.remove('disabled');
} else {
checkbox.disabled = true;
if (label) label.title = 'ANTHROPIC_API_KEY not configured';
if (container) container.classList.add('disabled');
}
})
.catch(() => {});
```
3. Enable GENERATE button when audio is loaded. In the existing `audio.addEventListener('loadedmetadata', ...)` inside `initTimeline()`, add:
```javascript
// Enable AI Sync generate button
const genBtn = document.getElementById('btn-ai-generate');
if (genBtn) genBtn.disabled = false;
const infoEl = document.getElementById('ai-sync-info');
if (infoEl) {
const seg = timeline.segmentOverlay?.getSelected();
infoEl.textContent = seg ? `SECTION: ${seg.label}` : 'FULL SHOW';
}
```
4. Update info text when segment selection changes. Add a periodic check or hook into the segment overlay click. The simplest approach: add a `requestAnimationFrame` based checker in initTimeline, or wire it into the canvas click. Add after the timeline.startRenderLoop() call:
```javascript
// Update AI Sync info when segment selection changes
let _lastSegIdx = -1;
setInterval(() => {
if (!timeline?.segmentOverlay) return;
const idx = timeline.segmentOverlay.selectedIndex;
if (idx !== _lastSegIdx) {
_lastSegIdx = idx;
const infoEl = document.getElementById('ai-sync-info');
if (infoEl) {
const seg = timeline.segmentOverlay.getSelected();
infoEl.textContent = seg ? `SECTION: ${seg.label}` : 'FULL SHOW';
}
}
}, 200);
```
5. Wire the GENERATE button. This is the main interaction:
```javascript
document.getElementById('btn-ai-generate')?.addEventListener('click', async () => {
if (!timeline) return;
const audio = document.getElementById('player');
if (!audio?.src) return;
// D-07: Confirm replacement if existing blocks
const hasBlocks = timeline.tracks.some(t => t.cues.length > 0);
if (hasBlocks) {
if (!confirm('Replace all existing blocks?')) return;
}
const btn = document.getElementById('btn-ai-generate');
const infoEl = document.getElementById('ai-sync-info');
btn.disabled = true;
btn.textContent = 'GENERATING...';
if (infoEl) infoEl.textContent = 'ANALYZING...';
try {
const filename = audio.src.split('/').pop();
const path = `/app/shows/${filename}`;
// D-11: If a section is selected, only fill that range
const selectedSeg = timeline.segmentOverlay?.getSelected();
const sectionRange = selectedSeg ? [selectedSeg.start, selectedSeg.end] : null;
// Collect device IDs from timeline tracks
const deviceIds = timeline.tracks.map(t => t.device_id);
const useLlm = document.getElementById('ai-use-llm')?.checked || false;
const res = await fetch('/api/sync/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
path: path,
device_ids: deviceIds,
use_llm: useLlm,
section_range: sectionRange,
}),
});
if (!res.ok) {
const err = await res.json();
throw new Error(err.detail || `HTTP ${res.status}`);
}
const data = await res.json();
// Clear existing blocks (or section range only)
if (sectionRange) {
// Clear only blocks in the section range
for (const track of timeline.tracks) {
track.cues = track.cues.filter(c =>
c.timestamp + (c.duration || 0) < sectionRange[0] ||
c.timestamp >= sectionRange[1]
);
}
} else {
// Clear all blocks
for (const track of timeline.tracks) {
track.cues = [];
}
}
// Place generated blocks onto tracks
for (const block of data.blocks) {
const track = timeline.tracks.find(t => String(t.device_id) === String(block.device_id));
if (track) {
track.cues.push({
id: block.id,
timestamp: block.timestamp,
duration: block.duration,
animation: block.animation,
params: block.params,
mode: 'animation',
});
}
}
if (infoEl) infoEl.textContent = `GENERATED ${data.count} BLOCKS (${data.mode.toUpperCase()})`;
console.log(`[ai-sync] Generated ${data.count} blocks via ${data.mode}`);
} catch (e) {
console.error('[ai-sync]', e);
if (infoEl) infoEl.textContent = `ERROR: ${e.message}`;
} finally {
btn.disabled = false;
btn.textContent = 'GENERATE';
}
});
```
</action>
<verify>
<automated>docker exec lightsync python -c "
import pathlib
html = pathlib.Path('/app/lightsync/frontend/index.html').read_text()
assert 'ai-sync-panel' in html, 'missing ai-sync-panel'
assert 'btn-ai-sync-open' in html, 'missing AI SYNC button'
assert 'btn-ai-generate' in html, 'missing GENERATE button'
assert 'ai-use-llm' in html, 'missing USE LLM toggle'
assert 'ai-use-beats' in html, 'missing feature toggles'
js = pathlib.Path('/app/lightsync/frontend/app.js').read_text()
assert 'api/sync/generate' in js, 'missing sync generate call'
assert 'api/sync/llm-available' in js, 'missing llm-available check'
assert 'Replace all existing blocks' in js, 'missing D-07 confirmation dialog'
assert 'section_range' in js, 'missing D-11 section range'
css = pathlib.Path('/app/lightsync/frontend/style.css').read_text()
assert '.ai-sync-panel' in css, 'missing ai-sync-panel style'
assert '.ai-sync-header' in css, 'missing ai-sync-header style'
assert '.ai-sync-generate' in css, 'missing generate button style'
print('All AI Sync UI checks passed')
"</automated>
</verify>
<acceptance_criteria>
- lightsync/frontend/index.html contains `id="ai-sync-panel"` div
- lightsync/frontend/index.html contains `id="btn-ai-sync-open"` button with text `AI SYNC`
- lightsync/frontend/index.html contains `id="btn-ai-generate"` button with text `GENERATE`
- lightsync/frontend/index.html contains `id="ai-use-llm"` checkbox
- lightsync/frontend/index.html contains checkboxes for `ai-use-beats`, `ai-use-onsets`, `ai-use-chroma`, `ai-use-rms`
- lightsync/frontend/index.html AI sync panel has `style="display:none;"` (starts hidden, D-08)
- lightsync/frontend/app.js contains `api/sync/generate` fetch call
- lightsync/frontend/app.js contains `api/sync/llm-available` fetch call
- lightsync/frontend/app.js contains `confirm('Replace all existing blocks?')` (D-07)
- lightsync/frontend/app.js contains `section_range` in the generate request body (D-11)
- lightsync/frontend/app.js contains `segmentOverlay?.getSelected()` to detect section selection
- lightsync/frontend/app.js contains `GENERATING...` text during generation
- lightsync/frontend/style.css contains `.ai-sync-panel` rule
- lightsync/frontend/style.css contains `.ai-sync-header` rule with `var(--accent)` color
- lightsync/frontend/style.css contains `.ai-sync-generate` rule with `var(--accent)` background
- No border-radius in ai-sync CSS rules (flat terminal aesthetic)
</acceptance_criteria>
<done>AI Sync panel appears when clicking AI SYNC button. GENERATE fills timeline with heuristic blocks. USE LLM toggle is active only when API key is configured. Confirmation dialog appears before replacing existing blocks. Per-section fill works when a section is selected.</done>
</task>
</tasks>
<verification>
1. Click AI SYNC in toolbar — panel appears with feature toggles and GENERATE button
2. With audio loaded, click GENERATE — timeline fills with animation blocks
3. Blocks match the music: fast animations in loud sections, slow in quiet sections
4. Block colors vary based on harmonic content
5. If blocks exist, confirmation dialog appears before replacing
6. Select a section band, then GENERATE — only that section fills
7. USE LLM toggle shows disabled state when ANTHROPIC_API_KEY is not set
8. Press Play — generated show plays with cues firing at correct timestamps
</verification>
<success_criteria>
- Heuristic show generation produces a playable show from any audio file
- Blocks are musically meaningful: tempo-aligned, energy-matched, harmonically colored
- LLM path works when ANTHROPIC_API_KEY is configured; gracefully disabled otherwise
- Per-section fill works with segment overlay selection
- Confirmation dialog prevents accidental block replacement
</success_criteria>
<output>
After completion, create `.planning/phases/06-ai-sync/06-03-SUMMARY.md`
</output>