docs(06-03): complete AI sync plan — heuristic+LLM show generation
- 06-03-SUMMARY.md: full plan documentation - STATE.md: position advanced, decisions logged, session updated - ROADMAP.md: phase 06 marked complete (3/3 plans)
This commit is contained in:
Claude
116
.planning/phases/06-ai-sync/06-03-SUMMARY.md
Normal file
116
.planning/phases/06-ai-sync/06-03-SUMMARY.md
Normal file
@@ -0,0 +1,116 @@
|
||||
---
|
||||
phase: 06-ai-sync
|
||||
plan: 03
|
||||
subsystem: ai-sync
|
||||
tags: [heuristic-generation, llm-generation, anthropic, chroma-analysis, beat-snapping, ui-panel]
|
||||
dependency_graph:
|
||||
requires:
|
||||
- phase: 06-02
|
||||
provides: "extract_features_async, detect_segments_async — audio analysis foundation"
|
||||
- phase: 06-01
|
||||
provides: "YouTube audio loading, ANTHROPIC_API_KEY wiring"
|
||||
provides:
|
||||
- "POST /api/sync/generate — heuristic + LLM show generation endpoint"
|
||||
- "GET /api/sync/llm-available — API key presence check"
|
||||
- "AI Sync panel with GENERATE button, USE LLM toggle, section range support"
|
||||
affects: [lightsync/api/sync.py, lightsync/frontend]
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Heuristic rule mapper: RMS-to-animation-type, chroma pitch class-to-HSL-hue color, beat-snapped block placement"
|
||||
- "LLM lazy initialization: import anthropic inside function to avoid startup crash when API key missing"
|
||||
- "setInterval-based segment selection watcher for reactive UI without event subscription"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- lightsync/api/sync.py
|
||||
modified:
|
||||
- lightsync/main.py
|
||||
- lightsync/frontend/index.html
|
||||
- lightsync/frontend/app.js
|
||||
- lightsync/frontend/style.css
|
||||
|
||||
key-decisions:
|
||||
- "Heuristic always runs first; LLM is opt-in via use_llm flag (D-04)"
|
||||
- "LLM generates from scratch, not from heuristic output (D-06)"
|
||||
- "section_range param limits generation to selected segment time window (D-11)"
|
||||
- "LLM toggle disabled when ANTHROPIC_API_KEY not configured — Pitfall 4 guard"
|
||||
- "claude-3-haiku-20240307 chosen for LLM model (speed vs cost)"
|
||||
|
||||
patterns-established:
|
||||
- "API key guard pattern: check os.environ.get('ANTHROPIC_API_KEY') in async function, raise 503 if missing"
|
||||
- "Lazy anthropic import: import inside function avoids startup crash without key"
|
||||
|
||||
requirements-completed: [SYNC-03]
|
||||
|
||||
duration: 8min
|
||||
completed: 2026-04-07
|
||||
---
|
||||
|
||||
# Phase 06 Plan 03: AI-Assisted Show Generation Summary
|
||||
|
||||
**Heuristic + LLM show generator with dedicated AI Sync panel: fills timeline with beat-snapped, energy-matched, harmonically-colored animation blocks from audio analysis.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~8 min
|
||||
- **Started:** 2026-04-07T11:39:00Z
|
||||
- **Completed:** 2026-04-07T11:47:18Z
|
||||
- **Tasks:** 2
|
||||
- **Files modified:** 5
|
||||
|
||||
## Accomplishments
|
||||
|
||||
1. **AI sync backend** (`lightsync/api/sync.py`): `POST /api/sync/generate` endpoint with two generation modes:
|
||||
- `heuristic_generate`: section-level RMS-to-animation-type mapping, chroma pitch-class-to-HSL color, beat-grid-snapped block placement with density-adaptive grouping
|
||||
- `llm_generate`: Anthropic claude-3-haiku with compact feature payload, JSON parse + validation, lazy init to guard against missing API key
|
||||
- `GET /api/sync/llm-available` for frontend toggle gating
|
||||
- `section_range` parameter for per-section fill (D-11)
|
||||
|
||||
2. **AI Sync panel UI** (`index.html`, `app.js`, `style.css`):
|
||||
- AI SYNC button in toolbar toggles collapsible panel
|
||||
- Feature toggles: BEATS, ONSETS, CHROMA, RMS
|
||||
- GENERATE button disabled until audio loaded; re-enabled on loadedmetadata
|
||||
- USE LLM toggle: disabled when ANTHROPIC_API_KEY not set (checked via llm-available endpoint)
|
||||
- Confirmation dialog before replacing existing blocks (D-07)
|
||||
- Section-aware: if a segment band is selected, fills only that time range (D-11)
|
||||
- Info text shows FULL SHOW or SECTION: SEC N, updates via 200ms interval watcher
|
||||
- Terminal/flat aesthetic — no border-radius
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: AI sync backend** - `b6aacad` (feat)
|
||||
2. **Task 2: AI Sync panel UI + frontend wiring** - `70b86f7` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `lightsync/api/sync.py` - New: `/api/sync/generate` endpoint, `heuristic_generate`, `llm_generate`, `SyncGenerateRequest`, `_chroma_to_rgb`, `_intensity_to_animation`
|
||||
- `lightsync/main.py` - Register `sync.router` with prefix `/api/sync`
|
||||
- `lightsync/frontend/index.html` - AI SYNC button, ai-sync-panel div with feature toggles and GENERATE button
|
||||
- `lightsync/frontend/app.js` - Panel toggle, llm-available check, segment watcher, GENERATE handler with D-07/D-11 logic
|
||||
- `lightsync/frontend/style.css` - AI sync panel styles (flat terminal aesthetic)
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- Heuristic always available as baseline; LLM is opt-in (D-04) — no LLM dependency for core functionality
|
||||
- LLM generates independently from scratch, not post-processing heuristic output (D-06)
|
||||
- claude-3-haiku-20240307 for LLM: fast and cheap for structured JSON output
|
||||
- `anthropic` imported lazily inside `llm_generate` to avoid import errors at container startup without API key (Pitfall 4)
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None - plan executed exactly as written.
|
||||
|
||||
## Known Stubs
|
||||
|
||||
None - all functionality is fully wired end-to-end.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- `lightsync/api/sync.py` exists: contains `router = APIRouter()`, `def heuristic_generate(`, `async def llm_generate(`, `async def generate_sync(`, `async def llm_available(`, `class SyncGenerateRequest`, `section_range: list[float] | None`, `use_llm: bool = False`, `ANTHROPIC_API_KEY`, `claude-3-haiku`, `_chroma_to_rgb`, `_intensity_to_animation`
|
||||
- `lightsync/main.py` contains `from lightsync.api import shows, devices, ws, audio, timeline, sync` and `sync.router`
|
||||
- `lightsync/frontend/index.html` contains `id="ai-sync-panel"`, `id="btn-ai-sync-open"`, `id="btn-ai-generate"`, `id="ai-use-llm"`, `ai-use-beats`, `style="display:none;"`
|
||||
- `lightsync/frontend/app.js` contains `api/sync/generate`, `api/sync/llm-available`, `Replace all existing blocks`, `section_range`, `segmentOverlay?.getSelected()`, `GENERATING...`
|
||||
- `lightsync/frontend/style.css` contains `.ai-sync-panel`, `.ai-sync-header`, `.ai-sync-generate`
|
||||
- Heuristic generates blocks with valid animation types and params with color/speed
|
||||
- All 2 task commits exist: `b6aacad`, `70b86f7`
|
||||
Reference in New Issue
Block a user