Gitea-Admin/led2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(phase-04): add phase plans (4 plans, 3 waves)

Browse Source
This commit is contained in:
Claude
2026-04-06 23:24:46 +00:00
parent e8e0532972
commit 78c5d6256d
2 changed files with 920 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

602
.planning/phases/04-timeline-editor/04-RESEARCH.md Normal file
View File

@@ -0,0 +1,602 @@
# Phase 4: Timeline Editor - Research
**Researched:** 2026-04-06
**Domain:** HTML5 Canvas DAW timeline, command-pattern undo/redo, beat detection calibration, block editor UI
**Confidence:** HIGH
---
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
- **D-01:** Fixed inspector panel between the timeline and the transport bar — always visible when a block is selected, hidden/collapsed when nothing is selected
- **D-02:** Inspector shows block label (animation type), color picker, and animation params (speed, direction, length, etc.) in a single horizontal row — compact strip style matching the terminal aesthetic
- **D-03:** Layout from top to bottom: TIMELINE → BLOCK INSPECTOR (conditional) → TRANSPORT BAR
### Claude's Discretion
- **Block data model**: Add `duration: float = 4.0` to existing `CueModel` — backward-compatible, `timestamp` stays as start time. Schema_version stays at 1 (duration defaults gracefully for old files). Renaming `timestamp` to `start_time` would be cleaner but requires migration — defer to v2.
- **Timeline canvas technology**: Choose based on what's best for the interaction model. HTML5 Canvas is recommended (custom hit-testing, DAW-like perf, consistent with waveform rendering approach). DOM divs are acceptable if the planner has strong reasons. Mixed (canvas background + DOM blocks) is also valid.
- **State ownership for undo/redo**: Frontend-owned state recommended — all timeline mutations live in a JS command history stack, backend updated on explicit save or after each mutation via async API call. This matches how the audio clock already lives in the browser. The command pattern must support at least 20 undo steps.
- **Beat detection UI placement**: Beat marks overlaid directly on the timeline (same horizontal axis as blocks). Calibration offset control can live in a settings/toolbar area or inline near the beat mark display — planner decides.
- **Waveform canvas**: Phase 2 backend (`/api/audio/waveform`, `lightsync/audio/waveform.py`) and frontend beat analysis are implemented, but there's currently no `<canvas>` element in `index.html`. Phase 4 must add the waveform canvas to the timeline panel and wire it up.
### Deferred Ideas (OUT OF SCOPE)
- Live UDP firing while timeline plays — Phase 5
- AI-assisted block placement — Phase 6
- Structural segmentation overlay (verse/chorus/bridge regions) — Phase 6 / v2
- Keyboard shortcuts for all primary operations — Phase 5 (UI-03)
- Show save/load persistence across sessions — Phase 5
</user_constraints>
---
<phase_requirements>
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| TL-01 | Per-device horizontal tracks on a time axis | Canvas rendering loop with pixelsPerSecond coordinate system; track headers from DeviceConfig.name |
| TL-02 | Drag-and-place animation blocks onto a track | Mousedown on animation palette item + mousemove ghost + mouseup drop; hit-test track Y-coord to determine target track |
| TL-03 | Move, resize, and delete animation blocks | Hit-test canvas for block rect; drag handle regions for resize (left/right edge); Delete key or context action for delete |
| TL-04 | Color picker per animation block | `<input type="color">` hidden, triggered by inspector color swatch click; value stored as hex string in CueModel.params.color |
| TL-05 | Playback cursor (vertical line) tracking audio position in realtime | `audio.timeupdate` event → compute cursorX = position * pixelsPerSecond; redraw cursor on each tick |
| TL-06 | Undo/redo for all timeline mutations (min 20 steps, command pattern) | Command objects with execute()/undo() methods; two stacks (undoStack, redoStack); Ctrl+Z / Ctrl+Shift+Z bindings |
| TL-07 | Beat detection marks overlaid on timeline (librosa beat_track) | GET /api/audio/beats → beat_times array; render thin vertical tick marks at beat_time * pixelsPerSecond |
| TL-08 | Snap-to-beat when placing/resizing blocks | On drop/resize: find nearest beat_time within snap threshold; clamp block.timestamp to that beat |
| SYNC-01 | Beat detection via librosa — produces beat timestamps and onset marks | Already implemented in lightsync/audio/beats.py and /api/audio/beats endpoint (Phase 2); Phase 4 consumes it |
| SYNC-02 | Beat calibration offset (UI control) — compensate for librosa's systematic 20-60ms latency bias | Slider or numeric input stored as `calibrationOffset` float (seconds); applied as `displayTime = beat_time - calibrationOffset` before rendering beat marks |
</phase_requirements>
---
## Summary
Phase 4 transforms the main area of the LightSync UI from a static animation tile grid into a fully interactive DAW-style timeline editor. The core problem is implementing DAW-quality block manipulation (place, drag, resize, delete) with undo/redo on an HTML5 Canvas — a well-understood but non-trivial engineering problem that requires careful coordinate system design, hit testing, and command pattern discipline.
The technical stack is entirely frontend: a new `<canvas>` element in `main-area`, new JavaScript modules for the timeline controller, command history, and block editor inspector. The backend gains only three new REST endpoints (block CRUD) and one model field addition (`CueModel.duration`). All existing Phase 2 infrastructure (waveform extraction, beat analysis API) is consumed as-is; Phase 4 just adds the frontend canvas that renders it.
The approach recommended by the context — pure HTML5 Canvas with custom hit-testing — is the correct choice for this use case. It matches the waveform rendering approach already established, avoids the z-index and overflow complexity of DOM-div approaches, and gives complete rendering control needed for the terminal aesthetic (no browser-default drag handles or selection UI that would clash with the dark theme).
**Primary recommendation:** Pure HTML5 Canvas timeline with a `pixelsPerSecond` coordinate system, a minimal hand-rolled command stack (not a library), and `<input type="color">` for the color picker. No third-party canvas framework needed.
---
## Standard Stack
### Core (already in project — Phase 4 consumes)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| librosa | >=0.10.0 | Beat detection (already runs via /api/audio/beats) | Established in Phase 2; beat_track returns beat_times array |
| FastAPI | >=0.115.0 | New block CRUD endpoints | Already the project backend |
| Pydantic v2 | >=2.0.0 | CueModel + duration field validation | Already the data model layer |
| HTML5 Canvas 2D API | Browser native | Timeline rendering, waveform, cursor, beat marks | No dependency; consistent with waveform approach |
### No New Libraries Required
This phase adds no new Python or JS dependencies. Everything needed is:
- Browser-native Canvas 2D API (hit testing, drawing)
- Browser-native `<input type="color">` (color picker)
- Browser-native `requestAnimationFrame` (cursor animation)
- Existing `/api/audio/beats` and `/api/audio/waveform` endpoints
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| Custom Canvas hit-testing | Konva.js or Fabric.js | Libraries add 100-300KB, impose object model overhead, and render default handles that clash with terminal aesthetic. Not worth it for this focused use case. |
| Hand-rolled command stack | Immer.js or Redux | Overkill — the state is a flat array of block objects, not a complex reactive tree. A 40-line command class covers all requirements. |
| `<input type="color">` | Custom HSV canvas picker | Native input is sufficient and consistent across Chromium (Docker serves to desktop browser). No custom widget needed. |
| requestAnimationFrame cursor loop | setInterval tick | rAF is smoother and automatically pauses when tab is hidden |
---
## Architecture Patterns
### Recommended File Structure
```
lightsync/frontend/
├── timeline/
│ ├── timeline.js # TimelineCanvas class — rendering loop, hit testing, coordinate math
│ ├── commands.js # Command classes: PlaceBlock, MoveBlock, ResizeBlock, DeleteBlock
│ ├── history.js # CommandHistory — undoStack, redoStack, execute(), undo(), redo()
│ ├── inspector.js # BlockInspector — renders/hides the inspector strip, color picker wiring
│ └── beats.js # BeatGrid — loads beat data, applies calibration offset, snap logic
├── app.js # (existing) — imports timeline modules, wires up on DOMContentLoaded
├── index.html # (modified) — adds #timeline-canvas, #block-inspector, restructures main-area
└── style.css # (modified) — timeline layout, inspector strip, beat mark styles
lightsync/api/
└── timeline.py # New router: POST/PATCH/DELETE /api/shows/{id}/blocks
lightsync/models/
└── show.py # Add duration: float = 4.0 to CueModel
```
### Pattern 1: Canvas Coordinate System
**What:** A single `pixelsPerSecond` float (e.g. 100.0 = 100px per second) translates between time and screen X. Tracks are stacked vertically with fixed `TRACK_HEIGHT` (e.g. 48px) and a fixed `HEADER_WIDTH` (e.g. 140px) for device labels.
**Coordinate math:**
```javascript
// Source: established DAW UI pattern (verified across multiple open-source timeline implementations)
const PIXELS_PER_SECOND = 100; // zoom level
const TRACK_HEIGHT = 48;
const HEADER_WIDTH = 140;
const TIME_AXIS_HEIGHT = 24;
function timeToX(t) {
return HEADER_WIDTH + t * PIXELS_PER_SECOND - scrollX;
}
function xToTime(x) {
return (x - HEADER_WIDTH + scrollX) / PIXELS_PER_SECOND;
}
function trackIndexToY(i) {
return TIME_AXIS_HEIGHT + i * TRACK_HEIGHT;
}
function yToTrackIndex(y) {
return Math.floor((y - TIME_AXIS_HEIGHT) / TRACK_HEIGHT);
}
```
### Pattern 2: Hit Testing on Canvas
**What:** Since Canvas has no DOM event targets per shape, hit testing is manual. On every `mousemove`/`mousedown`, iterate blocks and check if the cursor falls within the block rect. A small edge region (e.g. 8px) on left/right triggers resize cursor.
```javascript
// Source: standard canvas hit-testing pattern
function hitTest(mouseX, mouseY, blocks, tracks) {
const EDGE = 8;
for (const block of blocks) {
const trackIdx = tracks.findIndex(t => t.device_id === block.device_id);
const x = timeToX(block.timestamp);
const w = block.duration * PIXELS_PER_SECOND;
const y = trackIndexToY(trackIdx);
if (mouseY < y || mouseY > y + TRACK_HEIGHT) continue;
if (mouseX < x || mouseX > x + w) continue;
if (mouseX < x + EDGE) return { block, handle: 'resize-left' };
if (mouseX > x + w - EDGE) return { block, handle: 'resize-right' };
return { block, handle: 'move' };
}
return null;
}
```
### Pattern 3: Command Pattern for Undo/Redo (HIGH confidence)
**What:** Each mutation is a command object with `execute()` and `undo()` methods. The history manager maintains two stacks. On execute: push to undoStack, clear redoStack. On undo: pop from undoStack, push to redoStack. On redo: pop from redoStack, push to undoStack.
```javascript
// Source: command pattern, verified against multiple authoritative references
class PlaceBlockCommand {
constructor(block, tracks) {
this.block = block;
this.tracks = tracks; // reference to mutable state
}
execute() {
const track = this.tracks.find(t => t.device_id === this.block.device_id);
track.cues.push(this.block);
syncToBackend(this.block, 'create');
}
undo() {
const track = this.tracks.find(t => t.device_id === this.block.device_id);
track.cues = track.cues.filter(c => c.id !== this.block.id);
syncToBackend(this.block, 'delete');
}
}
class CommandHistory {
constructor(maxSteps = 50) {
this.undoStack = [];
this.redoStack = [];
this.maxSteps = maxSteps;
}
execute(cmd) {
cmd.execute();
this.undoStack.push(cmd);
if (this.undoStack.length > this.maxSteps)
this.undoStack.shift();
this.redoStack = []; // branching history — clear redo
}
undo() {
const cmd = this.undoStack.pop();
if (!cmd) return;
cmd.undo();
this.redoStack.push(cmd);
}
redo() {
const cmd = this.redoStack.pop();
if (!cmd) return;
cmd.execute();
this.undoStack.push(cmd);
}
}
```
### Pattern 4: Beat Snap Logic
**What:** When a block is dropped or a resize handle released, find the nearest beat time (after applying calibration offset). If within a snap threshold, clamp to that beat time.
```javascript
// Source: standard DAW snap pattern
const SNAP_THRESHOLD_SEC = 0.1; // 100ms snap radius
function snapToBeat(t, beatTimes, calibrationOffset) {
const adjustedBeats = beatTimes.map(b => b - calibrationOffset);
let nearest = null;
let minDist = Infinity;
for (const beat of adjustedBeats) {
const d = Math.abs(t - beat);
if (d < minDist) { minDist = d; nearest = beat; }
}
return (nearest !== null && minDist < SNAP_THRESHOLD_SEC) ? nearest : t;
}
```
### Pattern 5: Playback Cursor Animation
**What:** Use `requestAnimationFrame` to read `audio.currentTime` and draw the cursor line. This is smoother than the 100ms WebSocket tick and stays in sync with the HTML5 audio element directly.
```javascript
// Source: standard rAF audio sync pattern
function drawCursor(ctx, audio, canvas) {
if (audio.paused) return;
const x = timeToX(audio.currentTime);
ctx.save();
ctx.strokeStyle = '#00ffff';
ctx.lineWidth = 1;
ctx.beginPath();
ctx.moveTo(x, TIME_AXIS_HEIGHT);
ctx.lineTo(x, canvas.height);
ctx.stroke();
ctx.restore();
}
function renderLoop() {
redraw();
requestAnimationFrame(renderLoop);
}
```
### Pattern 6: Waveform Canvas Integration
**What:** Phase 2 produced `/api/audio/waveform?path=...&peaks=1000`. Phase 4 must add a `<canvas>` element and draw the peaks as vertical bars behind the timeline blocks. The waveform is a background layer; blocks render on top.
```javascript
// Source: derived from existing waveform.py extract_peaks() contract
async function loadWaveform(audioPath) {
const res = await fetch(`/api/audio/waveform?path=${encodeURIComponent(audioPath)}&peaks=1000`);
const { peaks } = await res.json();
return peaks; // float[] 0.0-1.0
}
function drawWaveform(ctx, peaks, totalDuration, canvasHeight) {
const midY = TIME_AXIS_HEIGHT + (canvasHeight - TIME_AXIS_HEIGHT) / 2;
ctx.strokeStyle = '#1a3a3a'; // very dim cyan — behind blocks
for (let i = 0; i < peaks.length; i++) {
const t = (i / peaks.length) * totalDuration;
const x = timeToX(t);
const h = peaks[i] * (canvasHeight - TIME_AXIS_HEIGHT) * 0.4;
ctx.beginPath();
ctx.moveTo(x, midY - h);
ctx.lineTo(x, midY + h);
ctx.stroke();
}
}
```
### Anti-Patterns to Avoid
- **DOM div blocks on canvas background:** Mixing DOM and canvas for block rendering creates z-index and positioning hell when the canvas scrolls. Use one or the other; this phase uses canvas-only.
- **Mutation without command wrapping:** Any direct mutation of `track.cues` that bypasses `CommandHistory.execute()` breaks undo. All state changes must go through commands.
- **Synchronous backend calls on every mouse move:** Backend sync should happen on `mouseup` (after drag ends), not during drag. During drag, only the local canvas state is updated.
- **redoStack not cleared on new execute:** This is the classic command-pattern bug — forgetting to clear `redoStack` when a new action is executed creates branching history corruption.
- **Storing calibrationOffset in beat_times:** The raw beat times from the API must remain uncalibrated. Apply the offset only at render/snap time so changing the offset immediately recalculates without re-fetching.
---
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Beat detection | Custom onset algorithm | Existing `/api/audio/beats` endpoint (librosa) | Already implemented and cached in Phase 2 |
| Waveform data | Custom audio decoding | Existing `/api/audio/waveform` endpoint | Already handles MP3 via ffmpeg pipe, returns normalized peaks |
| Color picker UI | Custom HSV canvas widget | `<input type="color">` (browser native) | Sufficient fidelity; avoids 50+ lines of canvas widget code |
| Undo/redo | External library (immer, etc.) | 40-line CommandHistory class | State is flat; library overhead is unjustified |
| Canvas object model | Konva.js / Fabric.js | Raw Canvas 2D API | Libraries fight terminal aesthetic, add weight, impose event model overhead |
| Block persistence | Custom file format | Extend existing ShowModel/shows.py | Show file is already the persistence layer; add block endpoints to shows API |
**Key insight:** The heaviest work (audio processing, beat detection, waveform extraction) is already done in Phase 2. Phase 4 is primarily a rendering and interaction problem — all the data is available, the challenge is drawing it and wiring up mouse events correctly.
---
## Data Model Changes
### CueModel — add `duration` field
```python
# In lightsync/models/show.py
class CueModel(BaseModel):
id: UUID = Field(default_factory=uuid4)
timestamp: float # block start time in seconds (unchanged)
duration: float = 4.0 # NEW — block length in seconds; defaults to 4.0 for old files
mode: Literal["animation", "frame_sequence"] = "animation"
animation: str | None = None
params: dict[str, Any] = Field(default_factory=dict)
# params stores: color (hex str), speed (float), direction (str), length (float), etc.
```
`schema_version` stays at 1. Old show files without `duration` will deserialize with `duration=4.0` via Pydantic default — no migration needed.
### New Backend Endpoints (in lightsync/api/timeline.py or shows.py)
```
POST /api/shows/{show_id}/tracks/{device_id}/blocks — create block (returns CueModel)
PATCH /api/shows/{show_id}/tracks/{device_id}/blocks/{block_id} — update block (move/resize/params)
DELETE /api/shows/{show_id}/tracks/{device_id}/blocks/{block_id} — delete block
```
These endpoints mutate the in-memory show and persist via `show_store.save()`. They are called from command `execute()` and `undo()` methods asynchronously after local state update.
---
## Beat Calibration Detail
**Root cause (verified via GitHub issue #1052):** librosa's `beat_track` uses centered STFT frames (with padding), causing beat timestamps to be systematically 20-60ms late compared to perceptual beat positions.
**Phase 4 implementation:** A numeric input (or range slider) labeled "BEAT OFFSET" in a toolbar above the timeline or inline. Stored as `calibrationOffset` float (seconds, default 0.03 = 30ms). Applied only at display and snap time:
```javascript
const displayBeatTime = rawBeatTime - calibrationOffset;
```
The raw beat times from the API cache are never modified. The offset is stored in frontend state and persisted in the show file as `show.analysis.calibration_offset` (add field to `AnalysisBlock` model).
---
## Layout Changes to index.html
The `main-area` div currently holds the animation tile grid. Phase 4 replaces this content with:
```
main-area (flex column)
├── #timeline-toolbar — zoom control, beat offset slider, snap toggle (flex row, 32px height)
├── #timeline-canvas — <canvas> fills remaining flex space, scrollable via overflow + scroll listener
├── #block-inspector — hidden by default; shown as compact 40px horizontal strip when block selected
└── (transport-bar is the footer — unchanged)
```
The animation library panel moves to a section within the sidebar (below DEVICES) or as a collapsible panel in the sidebar. This is the main structural change to `index.html`.
---
## Common Pitfalls
### Pitfall 1: Canvas devicePixelRatio (Blurry Canvas on HiDPI)
**What goes wrong:** Canvas drawn at CSS size without accounting for `window.devicePixelRatio` appears blurry on retina/HiDPI displays.
**Why it happens:** Canvas has intrinsic pixel dimensions separate from CSS dimensions.
**How to avoid:**
```javascript
function resizeCanvas(canvas) {
const dpr = window.devicePixelRatio || 1;
const rect = canvas.getBoundingClientRect();
canvas.width = rect.width * dpr;
canvas.height = rect.height * dpr;
const ctx = canvas.getContext('2d');
ctx.scale(dpr, dpr);
}
// Call on init and on window resize
```
**Warning signs:** Block edges appear fuzzy; text on canvas is not crisp.
### Pitfall 2: Drag State Leaks (mouseup Outside Canvas)
**What goes wrong:** User starts dragging a block, moves mouse outside the canvas, releases — the block stays in "dragging" state indefinitely because the canvas never receives `mouseup`.
**Why it happens:** Canvas only receives events when mouse is over it by default.
**How to avoid:** Attach `mousemove` and `mouseup` listeners to `document` (not canvas) while dragging, remove them on `mouseup`.
```javascript
canvas.addEventListener('mousedown', (e) => {
startDrag(e);
document.addEventListener('mousemove', onDrag);
document.addEventListener('mouseup', () => {
endDrag();
document.removeEventListener('mousemove', onDrag);
}, { once: true });
});
```
### Pitfall 3: Block Overlap and Z-ordering
**What goes wrong:** Two blocks on the same track can be placed at overlapping times. The drawing order matters for click selection.
**Why it happens:** No constraint prevents overlapping placements.
**How to avoid:** During drop, check for time-range overlap with existing blocks on the same track. Reject or nudge the placement. Draw blocks in reverse iteration order (last placed = on top = hit-tested first).
### Pitfall 4: Backend Sync Out of Order on Fast Undo/Redo
**What goes wrong:** Rapid Ctrl+Z presses fire multiple PATCH/DELETE requests; if responses arrive out of order, the server state diverges from client state.
**Why it happens:** Async fetch calls without sequencing.
**How to avoid:** For Phase 4, backend sync is fire-and-forget on each command. The client is the source of truth. Full save is triggered on explicit "save" action (Phase 5). For undo/redo, `await` the API call before allowing the next command to prevent races, or use a mutation queue.
### Pitfall 5: Animation Library Panel Source of Truth
**What goes wrong:** The sidebar currently has hardcoded animation tiles in HTML. Phase 4 needs to drag from a dynamic list backed by `ANIMATION_REGISTRY`. If the HTML tiles stay hardcoded, new animation types added to the registry won't appear.
**How to avoid:** On init, fetch `/api/animations` (or derive from known registry keys) and render the palette dynamically. The 7 types are known and stable for Phase 4 — hardcoding is acceptable, but a note in code to make it dynamic in Phase 5 is good practice.
### Pitfall 6: Beat Times Not Loaded Before Canvas Renders
**What goes wrong:** Timeline renders before the beat analysis fetch completes; beat marks never appear or appear late.
**Why it happens:** Beat analysis can take 2-5 seconds for a long file.
**How to avoid:** Beat marks are optional overlays. Show a loading indicator in the time axis area. When beat data arrives, call `beatGrid.load(data)` and trigger a full redraw. This is an async overlay, not a blocking dependency.
---
## Runtime State Inventory
Step 2.5 SKIPPED — this is a greenfield UI/feature phase, not a rename/refactor/migration phase. No runtime string renaming or data migration required. The only data model change is adding a `duration` field with a default value (backward-compatible, no existing record migration needed).
---
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Docker | App deployment | Yes | 28.2.2 | — |
| lightsync container | Running app | Yes (Up 5h) | led2-lightsync | — |
| librosa | Beat detection | Yes (in container) | >=0.10.0 | — |
| ffmpeg | Waveform MP3 extraction | Yes (in container, Phase 2 validated) | — | — |
| HTML5 Canvas 2D | Timeline rendering | Yes (browser native) | All modern browsers | — |
| `<input type="color">` | Color picker | Yes (browser native, Chromium) | — | — |
**No missing dependencies.** All required capabilities are available in the existing container or browser. Phase 4 adds no new pip packages or npm packages.
---
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| DOM divs for timeline blocks | HTML5 Canvas with custom hit-testing | Standard in DAW UIs circa 2015+ | Full rendering control, no CSS fighting, consistent with waveform approach |
| Global undo stack (Memento/snapshots) | Command pattern with execute/undo per operation | Established pattern | Supports side effects (API calls), lower memory than full state snapshots |
| Separate playback position poll | `audio.timeupdate` + `requestAnimationFrame` | Phase 2 decision | Sub-frame cursor accuracy without WebSocket latency |
**Deprecated/outdated in this codebase:**
- The hardcoded animation tile buttons in `index.html` (`.anim-tile` elements) — Phase 4 replaces this section with the timeline canvas. The sidebar animation library panel should be dynamically rendered from the registry.
- The `selectedAnim` variable and `select_animation` WebSocket event in app.js — repurposed in Phase 4 as the drag source for placing blocks (not a global "active animation" concept).
---
## Code Examples
### Rendering Loop Skeleton
```javascript
// Source: established Canvas 2D rendering pattern for timelines
class TimelineCanvas {
constructor(canvasEl, audioEl) {
this.canvas = canvasEl;
this.ctx = canvasEl.getContext('2d');
this.audio = audioEl;
this.pixelsPerSecond = 100;
this.scrollX = 0;
this.tracks = []; // TrackModel[]
this.blocks = []; // CueModel[] (flattened for rendering)
this.beatTimes = []; // float[] from /api/audio/beats
this.waveformPeaks = []; // float[] from /api/audio/waveform
this.calibrationOffset = 0.03;
this.selectedBlock = null;
this._dragState = null;
}
render() {
const ctx = this.ctx;
const { width, height } = this.canvas.getBoundingClientRect();
ctx.clearRect(0, 0, width, height);
this._drawBackground(ctx, width, height);
this._drawWaveform(ctx, width, height);
this._drawTimeAxis(ctx, width);
this._drawTrackHeaders(ctx, height);
this._drawBeatMarks(ctx, height);
this._drawBlocks(ctx);
this._drawCursor(ctx, height);
}
startRenderLoop() {
const loop = () => { this.render(); requestAnimationFrame(loop); };
requestAnimationFrame(loop);
}
}
```
### Inspector Strip (Compact Horizontal Layout)
```javascript
// Source: D-02 from CONTEXT.md — compact strip matching terminal aesthetic
function renderInspector(block) {
const el = document.getElementById('block-inspector');
if (!block) { el.style.display = 'none'; return; }
el.style.display = 'flex';
el.innerHTML = `
<span class="inspector-label">${block.animation?.toUpperCase() ?? 'BLOCK'}</span>
<label class="inspector-field">
COLOR
<input type="color" id="block-color"
value="${block.params.color ?? '#00ffff'}">
</label>
<label class="inspector-field">
SPEED
<input type="number" id="block-speed" step="0.1" min="0.1" max="10"
value="${block.params.speed ?? 1.0}">
</label>
<label class="inspector-field">
DIR
<select id="block-direction">
<option value="forward" ${block.params.direction === 'forward' ? 'selected' : ''}>FWD</option>
<option value="reverse" ${block.params.direction === 'reverse' ? 'selected' : ''}>REV</option>
</select>
</label>
<label class="inspector-field">
LEN
<input type="number" id="block-length" step="0.05" min="0.05" max="1"
value="${block.params.length ?? 0.3}">
</label>
`;
wireInspectorEvents(block);
}
```
### Beat Calibration Control
```javascript
// Source: CONTEXT.md SYNC-02 + librosa issue #1052 research
// In timeline toolbar HTML:
// <label class="toolbar-field">
// BEAT OFFSET <input type="number" id="beat-offset" step="0.005" min="-0.2" max="0.2" value="0.030">ms
// </label>
document.getElementById('beat-offset')?.addEventListener('input', (e) => {
const offsetSec = parseFloat(e.target.value);
timeline.calibrationOffset = offsetSec;
// No API call needed — display-only calculation applied at render time
});
```
---
## Open Questions
1. **Animation library panel placement in sidebar**
- What we know: Current sidebar has only DEVICES. The animation palette needs to be accessible for drag-to-track.
- What's unclear: Whether it fits in the sidebar (200px wide may be cramped) or should be a collapsible toolbar above the timeline.
- Recommendation: Planner decides. Sidebar second panel (`ANIMATIONS` below `DEVICES`) is simplest and avoids layout changes. Items can be compact single-line rows.
2. **Show persistence during Phase 4 (before Phase 5)**
- What we know: Phase 5 is designated for full save/load persistence. Phase 4 blocks added to `track.cues` in memory only.
- What's unclear: Whether Phase 4 should call `show_store.save()` after each mutation or defer to Phase 5.
- Recommendation: Phase 4 should fire-and-forget `PATCH /api/shows/{id}/tracks/{device_id}/blocks/{block_id}` on each command (to keep the running show state current in memory), but full show file persistence is Phase 5. Add a "SAVE" button stub in Phase 4 that logs "not yet implemented."
3. **Scroll behavior — horizontal scrolling of timeline**
- What we know: Long songs (5+ min at 100px/s = 30,000px wide) require horizontal scrolling.
- What's unclear: Whether to use native CSS overflow-x or a virtual scroll implemented in Canvas.
- Recommendation: Virtual scroll in Canvas — maintain a `scrollX` offset, map all time→X via `timeToX(t) = HEADER_WIDTH + t * PPS - scrollX`. Wheel event updates scrollX and triggers redraw. This is simpler than dealing with scrollable canvas containers.
---
## Sources
### Primary (HIGH confidence)
- Existing codebase files — `lightsync/audio/beats.py`, `lightsync/api/audio.py`, `lightsync/models/show.py`, `lightsync/frontend/app.js`, `lightsync/frontend/index.html`, `lightsync/frontend/style.css` — read directly; all patterns verified against live code
- `.planning/phases/04-timeline-editor/04-CONTEXT.md` — locked decisions D-01 through D-03 and discretion areas
- GitHub librosa/librosa issue #1052 — confirms 20-60ms systematic beat_track latency bias and `center=False` mitigation
### Secondary (MEDIUM confidence)
- MDN Web Docs — `<input type="color">` browser support, Canvas 2D API — standard browser APIs, highly stable
- Command pattern undo/redo — multiple sources agree on undoStack/redoStack structure with execute()/undo() per command; verified against Redux docs and JitBlox TypeScript example
### Tertiary (LOW confidence — not used in prescriptive recommendations)
- Konva.js docs — considered and rejected in favor of raw Canvas
- WebSearch results on DAW timeline patterns — used only to confirm that custom Canvas hit-testing is the established approach, not a novel invention
---
## Metadata
**Confidence breakdown:**
- Standard stack: HIGH — no new dependencies; all libraries already in container and verified
- Architecture (coordinate system, hit testing, command pattern): HIGH — established patterns verified across multiple sources and cross-checked against existing codebase patterns
- Beat calibration offset: HIGH — root cause confirmed via official librosa GitHub issue
- Pitfalls: HIGH — all based on reading the actual existing codebase and known browser/canvas behavior
- Inspector layout: HIGH — locked by D-01/D-02/D-03 in CONTEXT.md
**Research date:** 2026-04-06
**Valid until:** 2026-06-01 (stable browser APIs and Python libs; no fast-moving dependencies)

318
.planning/phases/04-timeline-editor/04-UI-SPEC.md Normal file
View File

@@ -0,0 +1,318 @@
---
phase: 4
slug: timeline-editor
status: draft
shadcn_initialized: false
preset: none
created: 2026-04-06
---
# Phase 4 — UI Design Contract
> Visual and interaction contract for the Timeline Editor phase. Generated by gsd-ui-researcher.
> Stack is plain HTML5/CSS/JS served by FastAPI — no React, no shadcn, no build step.
---
## Design System
| Property | Value |
|----------|-------|
| Tool | none — hand-rolled CSS variables |
| Preset | not applicable |
| Component library | none — plain DOM elements |
| Icon library | Unicode glyphs inline (▶, ■, ▼, ✕, >) — no icon font |
| Font | JetBrains Mono, Fira Code, Consolas, monospace (system fallback chain) |
Source: `lightsync/frontend/style.css` — CSS variable declarations at `:root`.
---
## Spacing Scale
Declared values (must be multiples of 4):
| Token | Value | Usage |
|-------|-------|-------|
| xs | 4px | Gap between inline inspector fields, scrollbar width |
| sm | 8px | Panel content padding, element gaps |
| md | 16px | Default spacing between grouped elements |
| lg | 24px | Section-level padding |
| xl | 32px | Layout gaps |
| 2xl | 48px | TRACK_HEIGHT — per-device track row height on canvas |
| 3xl | 64px | Reserved; not used in this phase |
Exceptions:
- TIME_AXIS_HEIGHT: 24px — time ruler height above tracks
- HEADER_WIDTH: 140px — track label column width (device names)
- Timeline toolbar height: 32px — compact single row above canvas
- Block inspector strip height: 40px — compact horizontal strip (D-02)
- Resize handle hit zone: 8px on left/right edge of each canvas block
- Snap threshold: 100ms (not a pixel dimension, but a time threshold for beat snap)
Source: RESEARCH.md Pattern 1 (coordinate system constants); CONTEXT.md D-02.
---
## Typography
| Role | Size | Weight | Line Height | Usage |
|------|------|--------|-------------|-------|
| Body | 13px | 400 | 1.4 | Default text, form inputs, device names |
| Label | 10px | 400 | 1.2 | Panel headers (DEVICES, ANIMATIONS, TIMELINE), toolbar labels, canvas track names |
| Compact | 12px | 400 | 1.3 | Transport controls, animation names, inspector field values, beat tick labels |
| Heading | 14px | 600 | 1.2 | Header title (LIGHTSYNC), block animation type label in inspector |
All text: font-family `var(--font-mono)` — monospace throughout.
Text transform: uppercase for all panel headers, labels, and inspector field names.
Letter spacing: 0.14em on panel headers; 0.06em on animation names; 0.15em on header title.
Source: `style.css` — `.panel-header` (10px/0.14em), `.header-title` (14px/600), `.transport-time` (12px), `:root` (13px base).
---
## Color
| Role | Value | Usage |
|------|-------|-------|
| Dominant (60%) | `#0a0a0a` | Body background, main-area background, canvas background |
| Secondary (30%) | `#0f0f0f` (`--bg-panel`) / `#080808` (`--bg-panel-dark`) | Sidebar, header, transport bar, block inspector strip, timeline toolbar |
| Accent (10%) | `#00ffff` | See reserved list below |
| Destructive | `#ff3333` | Delete block hover state on remove controls only |
| Dim border | `#1a4a4a` (`--border-dim`) | All panel borders, track separator lines, time axis tick marks |
| Waveform background | `#1a3a3a` | Waveform peaks drawn behind blocks (very dim cyan — not accent) |
| Beat marks | `#00ffff` at 40% opacity | Thin vertical beat tick marks on timeline canvas |
| Playback cursor | `#00ffff` | 1px vertical line spanning full canvas height |
| Selected block | `#00ffff` border at 1px | Selected block outline |
| Block fill (default) | `#0a2a2a` | Unselected animation block fill |
| Block fill (selected) | `#0d3a3a` | Selected animation block fill |
| Ghost block (drag) | `#00ffff` at 20% opacity | Block preview during drag-and-place |
Accent (`#00ffff`) reserved for:
- Playback cursor line
- Selected block border
- Active animation type label in inspector (`.inspector-label`)
- Panel header text (`--text-accent`)
- Transport button hover state (accent fills button bg, primary bg for text)
- Beat indicator flash in header
- Timeline toolbar labels in focus/active state
- `>` cursor glyph in active animation palette row
Accent is NOT used for:
- Block fills (blocks use `#0a2a2a` / `#0d3a3a`)
- Waveform strokes (use `#1a3a3a`)
- Time axis background ticks (use `--border-dim` at reduced opacity)
Source: `style.css` CSS variables; CONTEXT.md "terminal aesthetic carries forward"; RESEARCH.md Pattern 6 (waveform color `#1a3a3a`).
---
## Component Inventory
### New in Phase 4
| Component | Type | Description |
|-----------|------|-------------|
| `#timeline-toolbar` | HTML div (flex row) | 32px strip above canvas: zoom control, beat offset input, snap toggle |
| `#timeline-canvas` | HTML5 Canvas | Fills remaining flex space in main-area; custom hit-testing, virtual scroll |
| `#block-inspector` | HTML div (flex row) | 40px horizontal strip between canvas and transport-bar; hidden when no block selected |
| Animation palette rows | HTML buttons in sidebar | Compact single-line rows below DEVICES panel; drag source for placing blocks |
### Modified in Phase 4
| Component | Change |
|-----------|--------|
| `.main-area` | Replaces animation tile grid with: toolbar + canvas + inspector (flex column) |
| `.sidebar` | Adds ANIMATIONS panel below DEVICES; items are compact drag-source rows |
### Retained Unchanged
Transport bar, header, device panel, all CSS variables, all form element styles.
---
## Canvas Rendering Contract
All canvas rendering uses HTML5 Canvas 2D API. No third-party canvas library.
### Coordinate System
```
HEADER_WIDTH = 140px // track label column
TIME_AXIS_HEIGHT = 24px
TRACK_HEIGHT = 48px
PIXELS_PER_SECOND = 100 // default zoom; adjustable via toolbar
```
### Layer Draw Order (back to front)
1. Background fill `#0a0a0a`
2. Track row alternating banding: even `#0a0a0a`, odd `#0c0c0c` (2px brightness difference only)
3. Waveform peaks: `#1a3a3a` vertical bars behind blocks
4. Beat marks: `#00ffff` at 40% opacity, 1px wide vertical lines at calibrated beat times
5. Animation blocks: filled rects `#0a2a2a`, 1px border `#1a4a4a`; selected: `#0d3a3a` fill, `#00ffff` 1px border
6. Block labels: animation name in `#00ffff`, 10px, uppercase, clipped to block width
7. Drag ghost: 1px border `#00ffff`, `rgba(0,255,255,0.15)` fill
8. Track headers: `#0f0f0f` fill, device name in `--text-primary` 12px
9. Time axis: `#080808` fill, time labels `--text-dim` 10px, tick marks `--border-dim`
10. Playback cursor: `#00ffff` 1px line, full height
### devicePixelRatio Scaling
Canvas must call `resizeCanvas()` on init and on `window resize`. Scale ctx by `window.devicePixelRatio` to prevent blur on HiDPI.
---
## Interaction Contract
### Block Placement (TL-02)
1. User drags from animation palette row in sidebar.
2. Ghost block follows mouse across canvas at target track Y position.
3. On `mouseup` over a track: snap to nearest beat if within 100ms threshold; place block at snapped time.
4. On `mouseup` outside canvas: cancel drag, remove ghost.
5. Attach `mousemove` and `mouseup` to `document` during drag; remove on `mouseup`. (Pitfall 2 prevention.)
### Block Move (TL-03)
1. `mousedown` on block center region (not within 8px edge): enter move mode.
2. `mousemove`: update block start time live on canvas (visual only, no API call during drag).
3. `mouseup`: snap to beat; fire `MoveBlockCommand.execute()` → updates local state → async PATCH to backend.
### Block Resize (TL-03)
1. `mousedown` within 8px of left or right block edge: enter resize mode; cursor changes to `ew-resize`.
2. `mousemove`: update duration or start time live on canvas.
3. `mouseup`: snap to beat; fire `ResizeBlockCommand.execute()`.
### Block Delete (TL-03)
- `Delete` key when block selected: fire `DeleteBlockCommand.execute()` immediately. No confirmation dialog for individual block delete.
- Block remove button in inspector (`✕`): same behavior as Delete key.
### Block Selection
- `mousedown` on block body (not dragging): select block, show inspector strip.
- `mousedown` on canvas background: deselect, hide inspector strip.
- Only one block selected at a time (no multi-select in Phase 4).
### Undo/Redo (TL-06)
- `Ctrl+Z`: `CommandHistory.undo()`
- `Ctrl+Shift+Z` or `Ctrl+Y`: `CommandHistory.redo()`
- Minimum 50 undo steps (CommandHistory `maxSteps = 50`).
- All mutations (place, move, resize, delete, param change) must go through `CommandHistory.execute()`.
### Beat Snap (TL-08)
- Snap active by default; toggle via "SNAP" button in toolbar (lit accent when on, dim when off).
- Snap threshold: 100ms.
- Apply calibration offset before snap: `displayBeat = rawBeat - calibrationOffset`.
### Timeline Scroll
- Horizontal scroll via mouse wheel on canvas: updates `scrollX` offset.
- No native CSS overflow scrollbar on canvas — virtual scroll only.
- Auto-scroll: when playback cursor would exit the visible window, scroll to keep cursor at 20% from left edge.
---
## Inspector Strip Contract (D-01, D-02, D-03)
Height: 40px. Display: flex row. Background: `--bg-panel-dark`. Border-top: `1px solid var(--border-dim)`.
Hidden (`display: none`) when no block selected. Shown (`display: flex`) when block selected.
Field order (left to right):
1. `LABEL` — animation type name, `--text-accent`, 14px, 600 weight, uppercase, `flex-shrink: 0`, min-width 120px
2. `COLOR` — label + `<input type="color">` hidden, visible swatch 16×16px with 1px `--border-dim` border; click swatch triggers color input
3. `SPEED` — label + `<input type="number" step="0.1" min="0.1" max="10">`, width 60px
4. `DIR` — label + `<select>` with FWD / REV options
5. `LEN` — label + `<input type="number" step="0.05" min="0.05" max="1">`, width 60px
6. Spacer (`flex: 1`)
7. `✕` — remove button, `--text-dim`, hover: `#ff3333`
All field labels: 10px, uppercase, `--text-dim`, letter-spacing 0.1em.
All inputs: `var(--font-mono)`, 12px, `--bg-panel`, `--border-dim` border, no border-radius.
---
## Timeline Toolbar Contract
Height: 32px. Background: `--bg-panel-dark`. Border-bottom: `1px solid var(--border-dim)`.
Display: flex row, align-center, gap 16px, padding 0 12px.
Fields (left to right):
1. Zoom: `ZOOM` label + `<input type="range" min="20" max="400" step="10" value="100">` — updates `pixelsPerSecond`
2. Separator: 1px `--border-dim` vertical line, 20px tall
3. Beat offset: `BEAT OFFSET` label + `<input type="number" step="0.005" min="-0.2" max="0.2" value="0.030">` width 70px + `s` unit label
4. Separator
5. Snap toggle: `SNAP` button — `.transport-btn` style, lit (accent bg, primary color text) when active, dim when inactive
All labels: 10px, uppercase, `--text-dim`, letter-spacing 0.1em.
---
## Animation Palette Panel (Sidebar)
Placed below DEVICES panel in sidebar. Panel header: `ANIMATIONS` — same `.panel-header` style.
Items: one row per animation type (7 total). Each row is a draggable element styled as `.anim-tile`.
Compact layout: `>` glyph + animation name + brief description, same as existing `.anim-tile` pattern.
Drag cursor: `grab` on hover, `grabbing` during drag.
---
## Copywriting Contract
| Element | Copy |
|---------|------|
| Primary CTA (place block) | PLACE BLOCK |
| Animation palette panel header | ANIMATIONS |
| Timeline toolbar label | TIMELINE |
| Empty timeline state (no devices) | NO DEVICES — add a device in the sidebar to create tracks |
| Empty timeline state (devices, no audio) | LOAD AUDIO — use the transport bar to load a file |
| Empty track state (device exists, no blocks) | (no label — empty track row is visually self-evident) |
| Beat offset field label | BEAT OFFSET |
| Snap toggle | SNAP |
| Beat marks loading state | DETECTING BEATS... (shown as dim text in time axis area) |
| Beat marks load error | BEAT DETECTION FAILED — check audio file |
| Block delete (no confirmation dialog needed — undo covers it) | (immediate, no confirm) |
| Inspector: block removed | (inspector hides; no toast) |
| Undo available indicator | (Ctrl+Z visible in toolbar tooltip only — no persistent UI) |
No modal confirmations for any Phase 4 action. All destructive actions are undoable via Ctrl+Z.
---
## Error States
| Scenario | Visual Treatment | Copy |
|----------|-----------------|------|
| Beat detection API call fails | Dim status text in time axis | BEAT DETECTION FAILED |
| Block PATCH/DELETE API call fails | Console warning only (silent in UI — client is source of truth) | — |
| Block placed over existing block on same track | Placement rejected; ghost turns `#ff3333` briefly | OVERLAP — cannot place here |
| Audio not loaded when timeline renders | Dim placeholder text in canvas center | LOAD AUDIO — transport bar |
| No devices registered | Dim placeholder text in canvas center | NO DEVICES — add a device |
---
## Registry Safety
| Registry | Blocks Used | Safety Gate |
|----------|-------------|-------------|
| shadcn official | none | not applicable — no shadcn |
| third-party | none | not applicable |
No third-party component registries. All UI is hand-rolled HTML/CSS/JS.
---
## Checker Sign-Off
- [ ] Dimension 1 Copywriting: PASS
- [ ] Dimension 2 Visuals: PASS
- [ ] Dimension 3 Color: PASS
- [ ] Dimension 4 Typography: PASS
- [ ] Dimension 5 Spacing: PASS
- [ ] Dimension 6 Registry Safety: PASS
**Approval:** pending