docs(06-03): complete AI sync plan — heuristic+LLM show generation
- 06-03-SUMMARY.md: full plan documentation - STATE.md: position advanced, decisions logged, session updated - ROADMAP.md: phase 06 marked complete (3/3 plans)
This commit is contained in:
@@ -17,7 +17,7 @@ Decimal phases appear between their surrounding integers in numeric order.
|
|||||||
- [x] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator (completed 2026-04-06)
|
- [x] **Phase 3: Communication Protocol** - UDP packet spec, DRGB/DNRGB/DRGBW encoders, animation library, software simulator (completed 2026-04-06)
|
||||||
- [x] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat (completed 2026-04-06)
|
- [x] **Phase 4: Timeline Editor** - DAW UI (tracks, drag-place blocks, resize, delete), undo/redo, beat detection, snap-to-beat (completed 2026-04-06)
|
||||||
- [x] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview (completed 2026-04-07)
|
- [x] **Phase 5: Live Show Execution** - End-to-end show playback, seek-safe cue scheduling, show save/load, keyboard shortcuts, live preview (completed 2026-04-07)
|
||||||
- [ ] **Phase 6: AI Sync** - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation
|
- [x] **Phase 6: AI Sync** - YouTube loading, AI-assisted show generation, beat calibration UI, structural segmentation (completed 2026-04-07)
|
||||||
- [ ] **Phase 7: Microcontroller Firmware** - MicroPython firmware for ESP32/Pico/RPi, animation command + frame receivers for SK6812 and WS2801
|
- [ ] **Phase 7: Microcontroller Firmware** - MicroPython firmware for ESP32/Pico/RPi, animation command + frame receivers for SK6812 and WS2801
|
||||||
|
|
||||||
## Phase Details
|
## Phase Details
|
||||||
@@ -128,7 +128,7 @@ Plans:
|
|||||||
Plans:
|
Plans:
|
||||||
- [x] 06-01: YouTube loading — yt-dlp download to /app/shows/, SSE progress streaming, transport bar URL input
|
- [x] 06-01: YouTube loading — yt-dlp download to /app/shows/, SSE progress streaming, transport bar URL input
|
||||||
- [x] 06-02: Feature extraction + segmentation — onset/chroma/RMS extraction, structural segmentation API, colored section band overlay on waveform
|
- [x] 06-02: Feature extraction + segmentation — onset/chroma/RMS extraction, structural segmentation API, colored section band overlay on waveform
|
||||||
- [ ] 06-03: AI sync engine — heuristic block mapper + optional LLM generation, AI Sync panel UI, per-section fill
|
- [x] 06-03: AI sync engine — heuristic block mapper + optional LLM generation, AI Sync panel UI, per-section fill
|
||||||
|
|
||||||
### Phase 7: Microcontroller Firmware
|
### Phase 7: Microcontroller Firmware
|
||||||
**Goal**: Physical LED hardware responds to UDP packets from the show system — both animation command mode and raw frame mode work on real strips
|
**Goal**: Physical LED hardware responds to UDP packets from the show system — both animation command mode and raw frame mode work on real strips
|
||||||
@@ -158,5 +158,5 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7
|
|||||||
| 3. Communication Protocol | 3/3 | Complete | 2026-04-06 |
|
| 3. Communication Protocol | 3/3 | Complete | 2026-04-06 |
|
||||||
| 4. Timeline Editor | 4/4 | Complete | 2026-04-06 |
|
| 4. Timeline Editor | 4/4 | Complete | 2026-04-06 |
|
||||||
| 5. Live Show Execution | 4/4 | Complete | 2026-04-07 |
|
| 5. Live Show Execution | 4/4 | Complete | 2026-04-07 |
|
||||||
| 6. AI Sync | 2/3 | In Progress| |
|
| 6. AI Sync | 3/3 | Complete | 2026-04-07 |
|
||||||
| 7. Microcontroller Firmware | 0/3 | Not started | - |
|
| 7. Microcontroller Firmware | 0/3 | Not started | - |
|
||||||
|
|||||||
@@ -2,15 +2,15 @@
|
|||||||
gsd_state_version: 1.0
|
gsd_state_version: 1.0
|
||||||
milestone: v1.0
|
milestone: v1.0
|
||||||
milestone_name: milestone
|
milestone_name: milestone
|
||||||
status: executing
|
status: verifying
|
||||||
stopped_at: Completed 06-ai-sync/06-02-PLAN.md
|
stopped_at: Completed 06-ai-sync/06-03-PLAN.md
|
||||||
last_updated: "2026-04-07T11:42:33.193Z"
|
last_updated: "2026-04-07T11:48:26.614Z"
|
||||||
last_activity: 2026-04-07
|
last_activity: 2026-04-07
|
||||||
progress:
|
progress:
|
||||||
total_phases: 7
|
total_phases: 7
|
||||||
completed_phases: 5
|
completed_phases: 6
|
||||||
total_plans: 21
|
total_plans: 21
|
||||||
completed_plans: 20
|
completed_plans: 21
|
||||||
percent: 0
|
percent: 0
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -27,7 +27,7 @@ See: .planning/PROJECT.md (updated 2026-04-05)
|
|||||||
|
|
||||||
Phase: 06 (ai-sync) — EXECUTING
|
Phase: 06 (ai-sync) — EXECUTING
|
||||||
Plan: 3 of 3
|
Plan: 3 of 3
|
||||||
Status: Ready to execute
|
Status: Phase complete — ready for verification
|
||||||
Last activity: 2026-04-07
|
Last activity: 2026-04-07
|
||||||
|
|
||||||
Progress: [░░░░░░░░░░] 0%
|
Progress: [░░░░░░░░░░] 0%
|
||||||
@@ -72,6 +72,7 @@ Progress: [░░░░░░░░░░] 0%
|
|||||||
| Phase 05-live-show-execution P04 | 1 | 1 tasks | 1 files |
|
| Phase 05-live-show-execution P04 | 1 | 1 tasks | 1 files |
|
||||||
| Phase 06-ai-sync P01 | 8 | 3 tasks | 6 files |
|
| Phase 06-ai-sync P01 | 8 | 3 tasks | 6 files |
|
||||||
| Phase 06-ai-sync P02 | 8 | 2 tasks | 7 files |
|
| Phase 06-ai-sync P02 | 8 | 2 tasks | 7 files |
|
||||||
|
| Phase 06-ai-sync P03 | 8 | 2 tasks | 5 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -128,6 +129,9 @@ Recent decisions affecting current work:
|
|||||||
- [Phase 06]: yt-dlp --newline flag required for line-by-line progress parsing
|
- [Phase 06]: yt-dlp --newline flag required for line-by-line progress parsing
|
||||||
- [Phase 06-ai-sync]: Segment band drawing inserted between row banding and waveform in render() for correct Z-order
|
- [Phase 06-ai-sync]: Segment band drawing inserted between row banding and waveform in render() for correct Z-order
|
||||||
- [Phase 06-ai-sync]: Auto segment count: max(4, min(12, int(duration / 30))) for typical songs
|
- [Phase 06-ai-sync]: Auto segment count: max(4, min(12, int(duration / 30))) for typical songs
|
||||||
|
- [Phase 06-ai-sync]: Heuristic always available as baseline; LLM opt-in via use_llm flag (D-04)
|
||||||
|
- [Phase 06-ai-sync]: LLM generates from scratch, not post-processing heuristic output (D-06)
|
||||||
|
- [Phase 06-ai-sync]: anthropic imported lazily inside llm_generate to avoid startup crash without API key
|
||||||
|
|
||||||
### Pending Todos
|
### Pending Todos
|
||||||
|
|
||||||
@@ -141,8 +145,8 @@ None yet.
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-04-07T11:42:33.186Z
|
Last session: 2026-04-07T11:48:26.607Z
|
||||||
Stopped at: Completed 06-ai-sync/06-02-PLAN.md
|
Stopped at: Completed 06-ai-sync/06-03-PLAN.md
|
||||||
Resume file: None
|
Resume file: None
|
||||||
|
|
||||||
Next action: `/gsd:plan-phase 1`
|
Next action: `/gsd:plan-phase 1`
|
||||||
|
|||||||
116
.planning/phases/06-ai-sync/06-03-SUMMARY.md
Normal file
116
.planning/phases/06-ai-sync/06-03-SUMMARY.md
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
---
|
||||||
|
phase: 06-ai-sync
|
||||||
|
plan: 03
|
||||||
|
subsystem: ai-sync
|
||||||
|
tags: [heuristic-generation, llm-generation, anthropic, chroma-analysis, beat-snapping, ui-panel]
|
||||||
|
dependency_graph:
|
||||||
|
requires:
|
||||||
|
- phase: 06-02
|
||||||
|
provides: "extract_features_async, detect_segments_async — audio analysis foundation"
|
||||||
|
- phase: 06-01
|
||||||
|
provides: "YouTube audio loading, ANTHROPIC_API_KEY wiring"
|
||||||
|
provides:
|
||||||
|
- "POST /api/sync/generate — heuristic + LLM show generation endpoint"
|
||||||
|
- "GET /api/sync/llm-available — API key presence check"
|
||||||
|
- "AI Sync panel with GENERATE button, USE LLM toggle, section range support"
|
||||||
|
affects: [lightsync/api/sync.py, lightsync/frontend]
|
||||||
|
tech-stack:
|
||||||
|
added: []
|
||||||
|
patterns:
|
||||||
|
- "Heuristic rule mapper: RMS-to-animation-type, chroma pitch class-to-HSL-hue color, beat-snapped block placement"
|
||||||
|
- "LLM lazy initialization: import anthropic inside function to avoid startup crash when API key missing"
|
||||||
|
- "setInterval-based segment selection watcher for reactive UI without event subscription"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- lightsync/api/sync.py
|
||||||
|
modified:
|
||||||
|
- lightsync/main.py
|
||||||
|
- lightsync/frontend/index.html
|
||||||
|
- lightsync/frontend/app.js
|
||||||
|
- lightsync/frontend/style.css
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "Heuristic always runs first; LLM is opt-in via use_llm flag (D-04)"
|
||||||
|
- "LLM generates from scratch, not from heuristic output (D-06)"
|
||||||
|
- "section_range param limits generation to selected segment time window (D-11)"
|
||||||
|
- "LLM toggle disabled when ANTHROPIC_API_KEY not configured — Pitfall 4 guard"
|
||||||
|
- "claude-3-haiku-20240307 chosen for LLM model (speed vs cost)"
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "API key guard pattern: check os.environ.get('ANTHROPIC_API_KEY') in async function, raise 503 if missing"
|
||||||
|
- "Lazy anthropic import: import inside function avoids startup crash without key"
|
||||||
|
|
||||||
|
requirements-completed: [SYNC-03]
|
||||||
|
|
||||||
|
duration: 8min
|
||||||
|
completed: 2026-04-07
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 06 Plan 03: AI-Assisted Show Generation Summary
|
||||||
|
|
||||||
|
**Heuristic + LLM show generator with dedicated AI Sync panel: fills timeline with beat-snapped, energy-matched, harmonically-colored animation blocks from audio analysis.**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** ~8 min
|
||||||
|
- **Started:** 2026-04-07T11:39:00Z
|
||||||
|
- **Completed:** 2026-04-07T11:47:18Z
|
||||||
|
- **Tasks:** 2
|
||||||
|
- **Files modified:** 5
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
1. **AI sync backend** (`lightsync/api/sync.py`): `POST /api/sync/generate` endpoint with two generation modes:
|
||||||
|
- `heuristic_generate`: section-level RMS-to-animation-type mapping, chroma pitch-class-to-HSL color, beat-grid-snapped block placement with density-adaptive grouping
|
||||||
|
- `llm_generate`: Anthropic claude-3-haiku with compact feature payload, JSON parse + validation, lazy init to guard against missing API key
|
||||||
|
- `GET /api/sync/llm-available` for frontend toggle gating
|
||||||
|
- `section_range` parameter for per-section fill (D-11)
|
||||||
|
|
||||||
|
2. **AI Sync panel UI** (`index.html`, `app.js`, `style.css`):
|
||||||
|
- AI SYNC button in toolbar toggles collapsible panel
|
||||||
|
- Feature toggles: BEATS, ONSETS, CHROMA, RMS
|
||||||
|
- GENERATE button disabled until audio loaded; re-enabled on loadedmetadata
|
||||||
|
- USE LLM toggle: disabled when ANTHROPIC_API_KEY not set (checked via llm-available endpoint)
|
||||||
|
- Confirmation dialog before replacing existing blocks (D-07)
|
||||||
|
- Section-aware: if a segment band is selected, fills only that time range (D-11)
|
||||||
|
- Info text shows FULL SHOW or SECTION: SEC N, updates via 200ms interval watcher
|
||||||
|
- Terminal/flat aesthetic — no border-radius
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
1. **Task 1: AI sync backend** - `b6aacad` (feat)
|
||||||
|
2. **Task 2: AI Sync panel UI + frontend wiring** - `70b86f7` (feat)
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `lightsync/api/sync.py` - New: `/api/sync/generate` endpoint, `heuristic_generate`, `llm_generate`, `SyncGenerateRequest`, `_chroma_to_rgb`, `_intensity_to_animation`
|
||||||
|
- `lightsync/main.py` - Register `sync.router` with prefix `/api/sync`
|
||||||
|
- `lightsync/frontend/index.html` - AI SYNC button, ai-sync-panel div with feature toggles and GENERATE button
|
||||||
|
- `lightsync/frontend/app.js` - Panel toggle, llm-available check, segment watcher, GENERATE handler with D-07/D-11 logic
|
||||||
|
- `lightsync/frontend/style.css` - AI sync panel styles (flat terminal aesthetic)
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
- Heuristic always available as baseline; LLM is opt-in (D-04) — no LLM dependency for core functionality
|
||||||
|
- LLM generates independently from scratch, not post-processing heuristic output (D-06)
|
||||||
|
- claude-3-haiku-20240307 for LLM: fast and cheap for structured JSON output
|
||||||
|
- `anthropic` imported lazily inside `llm_generate` to avoid import errors at container startup without API key (Pitfall 4)
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
None - plan executed exactly as written.
|
||||||
|
|
||||||
|
## Known Stubs
|
||||||
|
|
||||||
|
None - all functionality is fully wired end-to-end.
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
- `lightsync/api/sync.py` exists: contains `router = APIRouter()`, `def heuristic_generate(`, `async def llm_generate(`, `async def generate_sync(`, `async def llm_available(`, `class SyncGenerateRequest`, `section_range: list[float] | None`, `use_llm: bool = False`, `ANTHROPIC_API_KEY`, `claude-3-haiku`, `_chroma_to_rgb`, `_intensity_to_animation`
|
||||||
|
- `lightsync/main.py` contains `from lightsync.api import shows, devices, ws, audio, timeline, sync` and `sync.router`
|
||||||
|
- `lightsync/frontend/index.html` contains `id="ai-sync-panel"`, `id="btn-ai-sync-open"`, `id="btn-ai-generate"`, `id="ai-use-llm"`, `ai-use-beats`, `style="display:none;"`
|
||||||
|
- `lightsync/frontend/app.js` contains `api/sync/generate`, `api/sync/llm-available`, `Replace all existing blocks`, `section_range`, `segmentOverlay?.getSelected()`, `GENERATING...`
|
||||||
|
- `lightsync/frontend/style.css` contains `.ai-sync-panel`, `.ai-sync-header`, `.ai-sync-generate`
|
||||||
|
- Heuristic generates blocks with valid animation types and params with color/speed
|
||||||
|
- All 2 task commits exist: `b6aacad`, `70b86f7`
|
||||||
Reference in New Issue
Block a user