diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 76281c8..89fd150 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -17,7 +17,7 @@ Decimal phases appear between their surrounding integers in numeric order. - [x] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator (completed 2026-04-06) - [x] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat (completed 2026-04-06) - [x] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview (completed 2026-04-07) -- [ ] **Phase 6: AI Sync** - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation +- [x] **Phase 6: AI Sync** - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation (completed 2026-04-07) - [ ] **Phase 7: Microcontroller Firmware** - MicroPython firmware for ESP32/Pico/RPi, animation command + frame receivers for SK6812 and WS2801 ## Phase Details @@ -128,7 +128,7 @@ Plans: Plans: - [x] 06-01: YouTube loading — yt-dlp download to /app/shows/, SSE progress streaming, transport bar URL input - [x] 06-02: Feature extraction + segmentation — onset/chroma/RMS extraction, structural segmentation API, colored section band overlay on waveform -- [ ] 06-03: AI sync engine — heuristic block mapper + optional LLM generation, AI Sync panel UI, per-section fill +- [x] 06-03: AI sync engine — heuristic block mapper + optional LLM generation, AI Sync panel UI, per-section fill ### Phase 7: Microcontroller Firmware **Goal**: Physical LED hardware responds to UDP packets from the show system — both animation command mode and raw frame mode work on real strips @@ -158,5 +158,5 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 | 3. Communication Protocol | 3/3 | Complete | 2026-04-06 | | 4. Timeline Editor | 4/4 | Complete | 2026-04-06 | | 5. Live Show Execution | 4/4 | Complete | 2026-04-07 | -| 6. AI Sync | 2/3 | In Progress| | +| 6. AI Sync | 3/3 | Complete | 2026-04-07 | | 7. Microcontroller Firmware | 0/3 | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 5528e0b..dd123ee 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,15 +2,15 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: milestone -status: executing -stopped_at: Completed 06-ai-sync/06-02-PLAN.md -last_updated: "2026-04-07T11:42:33.193Z" +status: verifying +stopped_at: Completed 06-ai-sync/06-03-PLAN.md +last_updated: "2026-04-07T11:48:26.614Z" last_activity: 2026-04-07 progress: total_phases: 7 - completed_phases: 5 + completed_phases: 6 total_plans: 21 - completed_plans: 20 + completed_plans: 21 percent: 0 --- @@ -27,7 +27,7 @@ See: .planning/PROJECT.md (updated 2026-04-05) Phase: 06 (ai-sync) — EXECUTING Plan: 3 of 3 -Status: Ready to execute +Status: Phase complete — ready for verification Last activity: 2026-04-07 Progress: [░░░░░░░░░░] 0% @@ -72,6 +72,7 @@ Progress: [░░░░░░░░░░] 0% | Phase 05-live-show-execution P04 | 1 | 1 tasks | 1 files | | Phase 06-ai-sync P01 | 8 | 3 tasks | 6 files | | Phase 06-ai-sync P02 | 8 | 2 tasks | 7 files | +| Phase 06-ai-sync P03 | 8 | 2 tasks | 5 files | ## Accumulated Context @@ -128,6 +129,9 @@ Recent decisions affecting current work: - [Phase 06]: yt-dlp --newline flag required for line-by-line progress parsing - [Phase 06-ai-sync]: Segment band drawing inserted between row banding and waveform in render() for correct Z-order - [Phase 06-ai-sync]: Auto segment count: max(4, min(12, int(duration / 30))) for typical songs +- [Phase 06-ai-sync]: Heuristic always available as baseline; LLM opt-in via use_llm flag (D-04) +- [Phase 06-ai-sync]: LLM generates from scratch, not post-processing heuristic output (D-06) +- [Phase 06-ai-sync]: anthropic imported lazily inside llm_generate to avoid startup crash without API key ### Pending Todos @@ -141,8 +145,8 @@ None yet. ## Session Continuity -Last session: 2026-04-07T11:42:33.186Z -Stopped at: Completed 06-ai-sync/06-02-PLAN.md +Last session: 2026-04-07T11:48:26.607Z +Stopped at: Completed 06-ai-sync/06-03-PLAN.md Resume file: None Next action: `/gsd:plan-phase 1` diff --git a/.planning/phases/06-ai-sync/06-03-SUMMARY.md b/.planning/phases/06-ai-sync/06-03-SUMMARY.md new file mode 100644 index 0000000..86d05c6 --- /dev/null +++ b/.planning/phases/06-ai-sync/06-03-SUMMARY.md @@ -0,0 +1,116 @@ +--- +phase: 06-ai-sync +plan: 03 +subsystem: ai-sync +tags: [heuristic-generation, llm-generation, anthropic, chroma-analysis, beat-snapping, ui-panel] +dependency_graph: + requires: + - phase: 06-02 + provides: "extract_features_async, detect_segments_async — audio analysis foundation" + - phase: 06-01 + provides: "YouTube audio loading, ANTHROPIC_API_KEY wiring" + provides: + - "POST /api/sync/generate — heuristic + LLM show generation endpoint" + - "GET /api/sync/llm-available — API key presence check" + - "AI Sync panel with GENERATE button, USE LLM toggle, section range support" + affects: [lightsync/api/sync.py, lightsync/frontend] +tech-stack: + added: [] + patterns: + - "Heuristic rule mapper: RMS-to-animation-type, chroma pitch class-to-HSL-hue color, beat-snapped block placement" + - "LLM lazy initialization: import anthropic inside function to avoid startup crash when API key missing" + - "setInterval-based segment selection watcher for reactive UI without event subscription" + +key-files: + created: + - lightsync/api/sync.py + modified: + - lightsync/main.py + - lightsync/frontend/index.html + - lightsync/frontend/app.js + - lightsync/frontend/style.css + +key-decisions: + - "Heuristic always runs first; LLM is opt-in via use_llm flag (D-04)" + - "LLM generates from scratch, not from heuristic output (D-06)" + - "section_range param limits generation to selected segment time window (D-11)" + - "LLM toggle disabled when ANTHROPIC_API_KEY not configured — Pitfall 4 guard" + - "claude-3-haiku-20240307 chosen for LLM model (speed vs cost)" + +patterns-established: + - "API key guard pattern: check os.environ.get('ANTHROPIC_API_KEY') in async function, raise 503 if missing" + - "Lazy anthropic import: import inside function avoids startup crash without key" + +requirements-completed: [SYNC-03] + +duration: 8min +completed: 2026-04-07 +--- + +# Phase 06 Plan 03: AI-Assisted Show Generation Summary + +**Heuristic + LLM show generator with dedicated AI Sync panel: fills timeline with beat-snapped, energy-matched, harmonically-colored animation blocks from audio analysis.** + +## Performance + +- **Duration:** ~8 min +- **Started:** 2026-04-07T11:39:00Z +- **Completed:** 2026-04-07T11:47:18Z +- **Tasks:** 2 +- **Files modified:** 5 + +## Accomplishments + +1. **AI sync backend** (`lightsync/api/sync.py`): `POST /api/sync/generate` endpoint with two generation modes: + - `heuristic_generate`: section-level RMS-to-animation-type mapping, chroma pitch-class-to-HSL color, beat-grid-snapped block placement with density-adaptive grouping + - `llm_generate`: Anthropic claude-3-haiku with compact feature payload, JSON parse + validation, lazy init to guard against missing API key + - `GET /api/sync/llm-available` for frontend toggle gating + - `section_range` parameter for per-section fill (D-11) + +2. **AI Sync panel UI** (`index.html`, `app.js`, `style.css`): + - AI SYNC button in toolbar toggles collapsible panel + - Feature toggles: BEATS, ONSETS, CHROMA, RMS + - GENERATE button disabled until audio loaded; re-enabled on loadedmetadata + - USE LLM toggle: disabled when ANTHROPIC_API_KEY not set (checked via llm-available endpoint) + - Confirmation dialog before replacing existing blocks (D-07) + - Section-aware: if a segment band is selected, fills only that time range (D-11) + - Info text shows FULL SHOW or SECTION: SEC N, updates via 200ms interval watcher + - Terminal/flat aesthetic — no border-radius + +## Task Commits + +1. **Task 1: AI sync backend** - `b6aacad` (feat) +2. **Task 2: AI Sync panel UI + frontend wiring** - `70b86f7` (feat) + +## Files Created/Modified + +- `lightsync/api/sync.py` - New: `/api/sync/generate` endpoint, `heuristic_generate`, `llm_generate`, `SyncGenerateRequest`, `_chroma_to_rgb`, `_intensity_to_animation` +- `lightsync/main.py` - Register `sync.router` with prefix `/api/sync` +- `lightsync/frontend/index.html` - AI SYNC button, ai-sync-panel div with feature toggles and GENERATE button +- `lightsync/frontend/app.js` - Panel toggle, llm-available check, segment watcher, GENERATE handler with D-07/D-11 logic +- `lightsync/frontend/style.css` - AI sync panel styles (flat terminal aesthetic) + +## Decisions Made + +- Heuristic always available as baseline; LLM is opt-in (D-04) — no LLM dependency for core functionality +- LLM generates independently from scratch, not post-processing heuristic output (D-06) +- claude-3-haiku-20240307 for LLM: fast and cheap for structured JSON output +- `anthropic` imported lazily inside `llm_generate` to avoid import errors at container startup without API key (Pitfall 4) + +## Deviations from Plan + +None - plan executed exactly as written. + +## Known Stubs + +None - all functionality is fully wired end-to-end. + +## Self-Check: PASSED + +- `lightsync/api/sync.py` exists: contains `router = APIRouter()`, `def heuristic_generate(`, `async def llm_generate(`, `async def generate_sync(`, `async def llm_available(`, `class SyncGenerateRequest`, `section_range: list[float] | None`, `use_llm: bool = False`, `ANTHROPIC_API_KEY`, `claude-3-haiku`, `_chroma_to_rgb`, `_intensity_to_animation` +- `lightsync/main.py` contains `from lightsync.api import shows, devices, ws, audio, timeline, sync` and `sync.router` +- `lightsync/frontend/index.html` contains `id="ai-sync-panel"`, `id="btn-ai-sync-open"`, `id="btn-ai-generate"`, `id="ai-use-llm"`, `ai-use-beats`, `style="display:none;"` +- `lightsync/frontend/app.js` contains `api/sync/generate`, `api/sync/llm-available`, `Replace all existing blocks`, `section_range`, `segmentOverlay?.getSelected()`, `GENERATING...` +- `lightsync/frontend/style.css` contains `.ai-sync-panel`, `.ai-sync-header`, `.ai-sync-generate` +- Heuristic generates blocks with valid animation types and params with color/speed +- All 2 task commits exist: `b6aacad`, `70b86f7`