docs(06): capture phase context
This commit is contained in:
Claude
121
.planning/phases/06-ai-sync/06-CONTEXT.md
Normal file
121
.planning/phases/06-ai-sync/06-CONTEXT.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# Phase 6: AI Sync - Context
|
||||
|
||||
**Gathered:** 2026-04-07
|
||||
**Status:** Ready for planning
|
||||
|
||||
<domain>
|
||||
## Phase Boundary
|
||||
|
||||
YouTube URL loading (yt-dlp → local file → existing audio stack), AI-assisted timeline auto-fill from multi-feature audio analysis (beats/onsets/chroma/RMS heuristics + optional LLM generation), and structural segmentation overlaid on the waveform as colored labeled bands. No microcontroller firmware (Phase 7).
|
||||
|
||||
</domain>
|
||||
|
||||
<decisions>
|
||||
## Implementation Decisions
|
||||
|
||||
### YouTube Loading (06-01)
|
||||
- **D-01:** yt-dlp downloads audio to `/app/shows/` — same directory as uploaded files. After download completes, load the file identically to a local upload (beat analysis, waveform, show saving all reuse existing wiring unchanged). No MPV backend change.
|
||||
- **D-02:** During download, show a terminal-style progress bar with download percentage. Parse yt-dlp `--progress` output and stream updates to the browser. Disable the URL input during download; re-enable and auto-load when complete.
|
||||
- **D-03:** URL input lives in the transport bar — inline with the existing file selector controls. Text input + `FETCH` button, same row, terminal aesthetic (flat, monospace, no rounded corners).
|
||||
|
||||
### AI Sync Engine (06-02)
|
||||
- **D-04:** Two-layer architecture — heuristic base always runs; LLM enhancement is optional (off by default, user can trigger it explicitly).
|
||||
- **D-05:** Heuristic features to extract and use:
|
||||
- Beats — from existing `beats.py` `beat_track`
|
||||
- Onsets — `librosa.onset.onset_detect()` for note attacks and transients
|
||||
- Chroma — `librosa.feature.chroma_cqt()` for tonal/harmonic content → drives color palette selection
|
||||
- RMS energy — `librosa.feature.rms()` for loudness envelope → drives animation intensity
|
||||
- **D-06:** LLM path: generate from scratch. The LLM receives raw audio features (beat list, segment boundaries, onset times, energy profile, chroma summary) and generates a complete block list — timestamps, duration, animation type, params per device. The heuristic result is NOT passed to the LLM; it generates independently.
|
||||
- **D-07:** Conflict handling: prompt before replacing. If the timeline has existing blocks when AI Sync is triggered, show a confirmation dialog: "Replace all existing blocks?" Yes clears and re-fills, No cancels. Undo/redo (Phase 4) can recover from replacements.
|
||||
- **D-08:** AI Sync trigger lives in a dedicated panel or context menu — not in the main toolbar or transport bar. Keeps the primary UI uncluttered.
|
||||
|
||||
### Segmentation Overlay (06-03)
|
||||
- **D-09:** Translucent colored band overlays drawn on the waveform canvas — one band per detected section, spanning the full waveform height. Alternating colors (e.g. cyan/purple variants from the existing palette).
|
||||
- **D-10:** Section labels are generic numbered: `SEC 1`, `SEC 2`, `SEC 3`, ... — no attempt to infer VERSE/CHORUS. Simple, always correct, terminal-readable.
|
||||
- **D-11:** Bands are interactive: clicking a section band selects it (highlight border). When AI Sync is triggered with a section selected, it fills only that section's time range. This allows per-section AI fill rather than whole-show fill.
|
||||
|
||||
### Claude's Discretion
|
||||
- **librosa segmentation algorithm:** Choose the best-fit algorithm from librosa's segment module (e.g. `librosa.segment.agglomerative` with recurrence matrix, or MSAF-style structural analysis). Target 4–12 sections for a typical 3–5 minute song. Planner decides implementation details.
|
||||
- **LLM prompt design:** Structure the feature payload as a compact JSON (beat timestamps, RMS samples at 1s resolution, chroma average per section, segment boundaries). Planner designs the prompt. Use existing Anthropic SDK patterns from the codebase if available.
|
||||
- **yt-dlp progress parsing:** Parse `--progress` output lines via subprocess pipe. Planner decides exact regex/parsing approach.
|
||||
- **Section color palette:** Pick 2–3 alternating colors from existing CSS variables (`--accent`, complementary hues). No new colors outside the existing terminal palette.
|
||||
|
||||
</decisions>
|
||||
|
||||
<canonical_refs>
|
||||
## Canonical References
|
||||
|
||||
**Downstream agents MUST read these before planning or implementing.**
|
||||
|
||||
### Requirements
|
||||
- `.planning/REQUIREMENTS.md` — SYNC-03 (AI-assisted show generation), AUD-02 (YouTube URL loading)
|
||||
- `.planning/ROADMAP.md` §Phase 6 — Success criteria (3 items), plan breakdown (06-01 through 06-03)
|
||||
|
||||
### Existing audio analysis
|
||||
- `lightsync/audio/beats.py` — `analyze(path)` → `{tempo, beats}` using `librosa.beat.beat_track`. Phase 6 extends this module (or adds `segments.py`) for onset/chroma/RMS/segmentation.
|
||||
- `lightsync/api/audio.py` — `/api/audio/upload`, `/api/audio/beats`, `/api/audio/waveform` endpoints. YouTube download adds a new `/api/audio/fetch` (or similar) endpoint here.
|
||||
|
||||
### Existing frontend wiring
|
||||
- `lightsync/frontend/app.js` — Audio load flow: file selected → `audio.src = /shows/filename` → `loadedmetadata` fires → `timeline.loadWaveform(path)` + `timeline.loadBeats(path)`. YouTube download reuses the same post-load path.
|
||||
- `lightsync/frontend/timeline/timeline.js` — `loadWaveform()` and `loadBeats()` methods; segmentation overlay drawn on the same waveform canvas.
|
||||
- `lightsync/frontend/index.html` — Transport bar layout (file selector, upload, LOAD button); URL input + FETCH button are added to this row.
|
||||
- `lightsync/frontend/style.css` — CSS variables; all new UI elements must use `--accent`, `--bg`, `--border`, `--text-dim`.
|
||||
|
||||
### Data models
|
||||
- `lightsync/models/show.py` — `source_type: Literal["file", "youtube"]` already present; set this when a YouTube URL is used.
|
||||
|
||||
### Prior context
|
||||
- `.planning/phases/01-foundation/01-CONTEXT.md` — Visual aesthetic (flat, sharp, cyan/teal, no rounded corners, no drop shadows, monospace uppercase labels)
|
||||
- `.planning/phases/05-live-show-execution/05-CONTEXT.md` — WebSocket protocol patterns, transport bar layout, CSS variable usage
|
||||
|
||||
</canonical_refs>
|
||||
|
||||
<code_context>
|
||||
## Existing Code Insights
|
||||
|
||||
### Reusable Assets
|
||||
- `beats.py` `analyze_async()` — thread-pool wrapper for librosa analysis; new segmentation/feature extraction follows the same `asyncio.to_thread` pattern
|
||||
- `audio.py` upload endpoint — POST handler pattern; YouTube fetch endpoint mirrors this (async subprocess + cache beat data after download)
|
||||
- `app.state.beats` cache — keyed by filepath; add `app.state.segments` for segmentation cache in the same lifespan setup
|
||||
- `timeline.js` waveform canvas — already drawing waveform peaks; segmentation bands drawn as canvas `fillRect` calls on the same canvas layer, below the waveform
|
||||
|
||||
### Established Patterns
|
||||
- Librosa analysis runs in `asyncio.to_thread` — never blocks the event loop
|
||||
- New API endpoints go into `lightsync/api/audio.py` (or a new `lightsync/api/sync.py` router)
|
||||
- Frontend ES modules — new JS files must use `export`/`import`
|
||||
- WebSocket message protocol: `{"type": "...", ...}` — AI sync progress can use a `sync_progress` message type
|
||||
- CSS panel strip pattern from Phase 4/5: compact horizontal strips, `1px solid var(--border)` borders
|
||||
|
||||
### Integration Points
|
||||
- Transport bar (`index.html`) — add URL text input + FETCH button next to existing file controls
|
||||
- `audio.py` router — add `/api/audio/fetch` (POST, takes `{url: string}`) for yt-dlp download; `/api/audio/segments` for structural analysis
|
||||
- `timeline.js` `loadWaveform()` / `loadBeats()` — add `loadSegments()` method to render colored bands
|
||||
- `main.py` lifespan — add `app.state.segments = {}` alongside existing `app.state.beats`
|
||||
- AI Sync panel — new dedicated panel or popover; triggers `/api/sync/generate` (POST) with `{path, use_llm: bool, section_range?: [start, end]}`
|
||||
|
||||
</code_context>
|
||||
|
||||
<specifics>
|
||||
## Specific Ideas
|
||||
|
||||
- Progress bar during YouTube download should feel terminal: `DOWNLOADING ██████░░░░ 62%` — monospace, fills left to right, uses `--accent` color. Not a smooth CSS animation, a stepped block fill.
|
||||
- Section bands: translucent (opacity ~0.15-0.20), drawn below the waveform fill so the waveform is still readable. Labels in small monospace uppercase at the top-left of each band.
|
||||
- The AI Sync panel could look like a terminal dialog — dark background, border, `AI SYNC` header, feature toggles (use beats / onsets / chroma / RMS all checked by default), a `GENERATE` button, and an optional `USE LLM` toggle.
|
||||
- When a section band is selected, draw a 1px bright border around it (using `--accent` at full opacity). Deselect when clicking elsewhere.
|
||||
|
||||
</specifics>
|
||||
|
||||
<deferred>
|
||||
## Deferred Ideas
|
||||
|
||||
- Per-section manual label renaming (e.g. rename SEC 2 to CHORUS) — out of Phase 6 scope
|
||||
- LLM-named sections — deferred; generic numbering is sufficient for v1
|
||||
- Full MPV backend (streaming YouTube without download) — out of scope, download approach is cleaner
|
||||
- AI Sync with partial fill (gap-only mode) — deferred; prompt+replace is sufficient for Phase 6
|
||||
|
||||
</deferred>
|
||||
|
||||
---
|
||||
|
||||
*Phase: 06-ai-sync*
|
||||
*Context gathered: 2026-04-07*
|
||||
Reference in New Issue
Block a user