603 lines
32 KiB
Markdown
603 lines
32 KiB
Markdown
# Phase 4: Timeline Editor - Research
|
|
|
|
**Researched:** 2026-04-06
|
|
**Domain:** HTML5 Canvas DAW timeline, command-pattern undo/redo, beat detection calibration, block editor UI
|
|
**Confidence:** HIGH
|
|
|
|
---
|
|
|
|
<user_constraints>
|
|
## User Constraints (from CONTEXT.md)
|
|
|
|
### Locked Decisions
|
|
- **D-01:** Fixed inspector panel between the timeline and the transport bar — always visible when a block is selected, hidden/collapsed when nothing is selected
|
|
- **D-02:** Inspector shows block label (animation type), color picker, and animation params (speed, direction, length, etc.) in a single horizontal row — compact strip style matching the terminal aesthetic
|
|
- **D-03:** Layout from top to bottom: TIMELINE → BLOCK INSPECTOR (conditional) → TRANSPORT BAR
|
|
|
|
### Claude's Discretion
|
|
- **Block data model**: Add `duration: float = 4.0` to existing `CueModel` — backward-compatible, `timestamp` stays as start time. Schema_version stays at 1 (duration defaults gracefully for old files). Renaming `timestamp` to `start_time` would be cleaner but requires migration — defer to v2.
|
|
- **Timeline canvas technology**: Choose based on what's best for the interaction model. HTML5 Canvas is recommended (custom hit-testing, DAW-like perf, consistent with waveform rendering approach). DOM divs are acceptable if the planner has strong reasons. Mixed (canvas background + DOM blocks) is also valid.
|
|
- **State ownership for undo/redo**: Frontend-owned state recommended — all timeline mutations live in a JS command history stack, backend updated on explicit save or after each mutation via async API call. This matches how the audio clock already lives in the browser. The command pattern must support at least 20 undo steps.
|
|
- **Beat detection UI placement**: Beat marks overlaid directly on the timeline (same horizontal axis as blocks). Calibration offset control can live in a settings/toolbar area or inline near the beat mark display — planner decides.
|
|
- **Waveform canvas**: Phase 2 backend (`/api/audio/waveform`, `lightsync/audio/waveform.py`) and frontend beat analysis are implemented, but there's currently no `<canvas>` element in `index.html`. Phase 4 must add the waveform canvas to the timeline panel and wire it up.
|
|
|
|
### Deferred Ideas (OUT OF SCOPE)
|
|
- Live UDP firing while timeline plays — Phase 5
|
|
- AI-assisted block placement — Phase 6
|
|
- Structural segmentation overlay (verse/chorus/bridge regions) — Phase 6 / v2
|
|
- Keyboard shortcuts for all primary operations — Phase 5 (UI-03)
|
|
- Show save/load persistence across sessions — Phase 5
|
|
</user_constraints>
|
|
|
|
---
|
|
|
|
<phase_requirements>
|
|
## Phase Requirements
|
|
|
|
| ID | Description | Research Support |
|
|
|----|-------------|------------------|
|
|
| TL-01 | Per-device horizontal tracks on a time axis | Canvas rendering loop with pixelsPerSecond coordinate system; track headers from DeviceConfig.name |
|
|
| TL-02 | Drag-and-place animation blocks onto a track | Mousedown on animation palette item + mousemove ghost + mouseup drop; hit-test track Y-coord to determine target track |
|
|
| TL-03 | Move, resize, and delete animation blocks | Hit-test canvas for block rect; drag handle regions for resize (left/right edge); Delete key or context action for delete |
|
|
| TL-04 | Color picker per animation block | `<input type="color">` hidden, triggered by inspector color swatch click; value stored as hex string in CueModel.params.color |
|
|
| TL-05 | Playback cursor (vertical line) tracking audio position in realtime | `audio.timeupdate` event → compute cursorX = position * pixelsPerSecond; redraw cursor on each tick |
|
|
| TL-06 | Undo/redo for all timeline mutations (min 20 steps, command pattern) | Command objects with execute()/undo() methods; two stacks (undoStack, redoStack); Ctrl+Z / Ctrl+Shift+Z bindings |
|
|
| TL-07 | Beat detection marks overlaid on timeline (librosa beat_track) | GET /api/audio/beats → beat_times array; render thin vertical tick marks at beat_time * pixelsPerSecond |
|
|
| TL-08 | Snap-to-beat when placing/resizing blocks | On drop/resize: find nearest beat_time within snap threshold; clamp block.timestamp to that beat |
|
|
| SYNC-01 | Beat detection via librosa — produces beat timestamps and onset marks | Already implemented in lightsync/audio/beats.py and /api/audio/beats endpoint (Phase 2); Phase 4 consumes it |
|
|
| SYNC-02 | Beat calibration offset (UI control) — compensate for librosa's systematic 20-60ms latency bias | Slider or numeric input stored as `calibrationOffset` float (seconds); applied as `displayTime = beat_time - calibrationOffset` before rendering beat marks |
|
|
</phase_requirements>
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
Phase 4 transforms the main area of the LightSync UI from a static animation tile grid into a fully interactive DAW-style timeline editor. The core problem is implementing DAW-quality block manipulation (place, drag, resize, delete) with undo/redo on an HTML5 Canvas — a well-understood but non-trivial engineering problem that requires careful coordinate system design, hit testing, and command pattern discipline.
|
|
|
|
The technical stack is entirely frontend: a new `<canvas>` element in `main-area`, new JavaScript modules for the timeline controller, command history, and block editor inspector. The backend gains only three new REST endpoints (block CRUD) and one model field addition (`CueModel.duration`). All existing Phase 2 infrastructure (waveform extraction, beat analysis API) is consumed as-is; Phase 4 just adds the frontend canvas that renders it.
|
|
|
|
The approach recommended by the context — pure HTML5 Canvas with custom hit-testing — is the correct choice for this use case. It matches the waveform rendering approach already established, avoids the z-index and overflow complexity of DOM-div approaches, and gives complete rendering control needed for the terminal aesthetic (no browser-default drag handles or selection UI that would clash with the dark theme).
|
|
|
|
**Primary recommendation:** Pure HTML5 Canvas timeline with a `pixelsPerSecond` coordinate system, a minimal hand-rolled command stack (not a library), and `<input type="color">` for the color picker. No third-party canvas framework needed.
|
|
|
|
---
|
|
|
|
## Standard Stack
|
|
|
|
### Core (already in project — Phase 4 consumes)
|
|
| Library | Version | Purpose | Why Standard |
|
|
|---------|---------|---------|--------------|
|
|
| librosa | >=0.10.0 | Beat detection (already runs via /api/audio/beats) | Established in Phase 2; beat_track returns beat_times array |
|
|
| FastAPI | >=0.115.0 | New block CRUD endpoints | Already the project backend |
|
|
| Pydantic v2 | >=2.0.0 | CueModel + duration field validation | Already the data model layer |
|
|
| HTML5 Canvas 2D API | Browser native | Timeline rendering, waveform, cursor, beat marks | No dependency; consistent with waveform approach |
|
|
|
|
### No New Libraries Required
|
|
This phase adds no new Python or JS dependencies. Everything needed is:
|
|
- Browser-native Canvas 2D API (hit testing, drawing)
|
|
- Browser-native `<input type="color">` (color picker)
|
|
- Browser-native `requestAnimationFrame` (cursor animation)
|
|
- Existing `/api/audio/beats` and `/api/audio/waveform` endpoints
|
|
|
|
### Alternatives Considered
|
|
| Instead of | Could Use | Tradeoff |
|
|
|------------|-----------|----------|
|
|
| Custom Canvas hit-testing | Konva.js or Fabric.js | Libraries add 100-300KB, impose object model overhead, and render default handles that clash with terminal aesthetic. Not worth it for this focused use case. |
|
|
| Hand-rolled command stack | Immer.js or Redux | Overkill — the state is a flat array of block objects, not a complex reactive tree. A 40-line command class covers all requirements. |
|
|
| `<input type="color">` | Custom HSV canvas picker | Native input is sufficient and consistent across Chromium (Docker serves to desktop browser). No custom widget needed. |
|
|
| requestAnimationFrame cursor loop | setInterval tick | rAF is smoother and automatically pauses when tab is hidden |
|
|
|
|
---
|
|
|
|
## Architecture Patterns
|
|
|
|
### Recommended File Structure
|
|
```
|
|
lightsync/frontend/
|
|
├── timeline/
|
|
│ ├── timeline.js # TimelineCanvas class — rendering loop, hit testing, coordinate math
|
|
│ ├── commands.js # Command classes: PlaceBlock, MoveBlock, ResizeBlock, DeleteBlock
|
|
│ ├── history.js # CommandHistory — undoStack, redoStack, execute(), undo(), redo()
|
|
│ ├── inspector.js # BlockInspector — renders/hides the inspector strip, color picker wiring
|
|
│ └── beats.js # BeatGrid — loads beat data, applies calibration offset, snap logic
|
|
├── app.js # (existing) — imports timeline modules, wires up on DOMContentLoaded
|
|
├── index.html # (modified) — adds #timeline-canvas, #block-inspector, restructures main-area
|
|
└── style.css # (modified) — timeline layout, inspector strip, beat mark styles
|
|
lightsync/api/
|
|
└── timeline.py # New router: POST/PATCH/DELETE /api/shows/{id}/blocks
|
|
lightsync/models/
|
|
└── show.py # Add duration: float = 4.0 to CueModel
|
|
```
|
|
|
|
### Pattern 1: Canvas Coordinate System
|
|
**What:** A single `pixelsPerSecond` float (e.g. 100.0 = 100px per second) translates between time and screen X. Tracks are stacked vertically with fixed `TRACK_HEIGHT` (e.g. 48px) and a fixed `HEADER_WIDTH` (e.g. 140px) for device labels.
|
|
|
|
**Coordinate math:**
|
|
```javascript
|
|
// Source: established DAW UI pattern (verified across multiple open-source timeline implementations)
|
|
|
|
const PIXELS_PER_SECOND = 100; // zoom level
|
|
const TRACK_HEIGHT = 48;
|
|
const HEADER_WIDTH = 140;
|
|
const TIME_AXIS_HEIGHT = 24;
|
|
|
|
function timeToX(t) {
|
|
return HEADER_WIDTH + t * PIXELS_PER_SECOND - scrollX;
|
|
}
|
|
|
|
function xToTime(x) {
|
|
return (x - HEADER_WIDTH + scrollX) / PIXELS_PER_SECOND;
|
|
}
|
|
|
|
function trackIndexToY(i) {
|
|
return TIME_AXIS_HEIGHT + i * TRACK_HEIGHT;
|
|
}
|
|
|
|
function yToTrackIndex(y) {
|
|
return Math.floor((y - TIME_AXIS_HEIGHT) / TRACK_HEIGHT);
|
|
}
|
|
```
|
|
|
|
### Pattern 2: Hit Testing on Canvas
|
|
**What:** Since Canvas has no DOM event targets per shape, hit testing is manual. On every `mousemove`/`mousedown`, iterate blocks and check if the cursor falls within the block rect. A small edge region (e.g. 8px) on left/right triggers resize cursor.
|
|
|
|
```javascript
|
|
// Source: standard canvas hit-testing pattern
|
|
function hitTest(mouseX, mouseY, blocks, tracks) {
|
|
const EDGE = 8;
|
|
for (const block of blocks) {
|
|
const trackIdx = tracks.findIndex(t => t.device_id === block.device_id);
|
|
const x = timeToX(block.timestamp);
|
|
const w = block.duration * PIXELS_PER_SECOND;
|
|
const y = trackIndexToY(trackIdx);
|
|
|
|
if (mouseY < y || mouseY > y + TRACK_HEIGHT) continue;
|
|
if (mouseX < x || mouseX > x + w) continue;
|
|
|
|
if (mouseX < x + EDGE) return { block, handle: 'resize-left' };
|
|
if (mouseX > x + w - EDGE) return { block, handle: 'resize-right' };
|
|
return { block, handle: 'move' };
|
|
}
|
|
return null;
|
|
}
|
|
```
|
|
|
|
### Pattern 3: Command Pattern for Undo/Redo (HIGH confidence)
|
|
**What:** Each mutation is a command object with `execute()` and `undo()` methods. The history manager maintains two stacks. On execute: push to undoStack, clear redoStack. On undo: pop from undoStack, push to redoStack. On redo: pop from redoStack, push to undoStack.
|
|
|
|
```javascript
|
|
// Source: command pattern, verified against multiple authoritative references
|
|
|
|
class PlaceBlockCommand {
|
|
constructor(block, tracks) {
|
|
this.block = block;
|
|
this.tracks = tracks; // reference to mutable state
|
|
}
|
|
execute() {
|
|
const track = this.tracks.find(t => t.device_id === this.block.device_id);
|
|
track.cues.push(this.block);
|
|
syncToBackend(this.block, 'create');
|
|
}
|
|
undo() {
|
|
const track = this.tracks.find(t => t.device_id === this.block.device_id);
|
|
track.cues = track.cues.filter(c => c.id !== this.block.id);
|
|
syncToBackend(this.block, 'delete');
|
|
}
|
|
}
|
|
|
|
class CommandHistory {
|
|
constructor(maxSteps = 50) {
|
|
this.undoStack = [];
|
|
this.redoStack = [];
|
|
this.maxSteps = maxSteps;
|
|
}
|
|
execute(cmd) {
|
|
cmd.execute();
|
|
this.undoStack.push(cmd);
|
|
if (this.undoStack.length > this.maxSteps)
|
|
this.undoStack.shift();
|
|
this.redoStack = []; // branching history — clear redo
|
|
}
|
|
undo() {
|
|
const cmd = this.undoStack.pop();
|
|
if (!cmd) return;
|
|
cmd.undo();
|
|
this.redoStack.push(cmd);
|
|
}
|
|
redo() {
|
|
const cmd = this.redoStack.pop();
|
|
if (!cmd) return;
|
|
cmd.execute();
|
|
this.undoStack.push(cmd);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Pattern 4: Beat Snap Logic
|
|
**What:** When a block is dropped or a resize handle released, find the nearest beat time (after applying calibration offset). If within a snap threshold, clamp to that beat time.
|
|
|
|
```javascript
|
|
// Source: standard DAW snap pattern
|
|
const SNAP_THRESHOLD_SEC = 0.1; // 100ms snap radius
|
|
|
|
function snapToBeat(t, beatTimes, calibrationOffset) {
|
|
const adjustedBeats = beatTimes.map(b => b - calibrationOffset);
|
|
let nearest = null;
|
|
let minDist = Infinity;
|
|
for (const beat of adjustedBeats) {
|
|
const d = Math.abs(t - beat);
|
|
if (d < minDist) { minDist = d; nearest = beat; }
|
|
}
|
|
return (nearest !== null && minDist < SNAP_THRESHOLD_SEC) ? nearest : t;
|
|
}
|
|
```
|
|
|
|
### Pattern 5: Playback Cursor Animation
|
|
**What:** Use `requestAnimationFrame` to read `audio.currentTime` and draw the cursor line. This is smoother than the 100ms WebSocket tick and stays in sync with the HTML5 audio element directly.
|
|
|
|
```javascript
|
|
// Source: standard rAF audio sync pattern
|
|
function drawCursor(ctx, audio, canvas) {
|
|
if (audio.paused) return;
|
|
const x = timeToX(audio.currentTime);
|
|
ctx.save();
|
|
ctx.strokeStyle = '#00ffff';
|
|
ctx.lineWidth = 1;
|
|
ctx.beginPath();
|
|
ctx.moveTo(x, TIME_AXIS_HEIGHT);
|
|
ctx.lineTo(x, canvas.height);
|
|
ctx.stroke();
|
|
ctx.restore();
|
|
}
|
|
|
|
function renderLoop() {
|
|
redraw();
|
|
requestAnimationFrame(renderLoop);
|
|
}
|
|
```
|
|
|
|
### Pattern 6: Waveform Canvas Integration
|
|
**What:** Phase 2 produced `/api/audio/waveform?path=...&peaks=1000`. Phase 4 must add a `<canvas>` element and draw the peaks as vertical bars behind the timeline blocks. The waveform is a background layer; blocks render on top.
|
|
|
|
```javascript
|
|
// Source: derived from existing waveform.py extract_peaks() contract
|
|
async function loadWaveform(audioPath) {
|
|
const res = await fetch(`/api/audio/waveform?path=${encodeURIComponent(audioPath)}&peaks=1000`);
|
|
const { peaks } = await res.json();
|
|
return peaks; // float[] 0.0-1.0
|
|
}
|
|
|
|
function drawWaveform(ctx, peaks, totalDuration, canvasHeight) {
|
|
const midY = TIME_AXIS_HEIGHT + (canvasHeight - TIME_AXIS_HEIGHT) / 2;
|
|
ctx.strokeStyle = '#1a3a3a'; // very dim cyan — behind blocks
|
|
for (let i = 0; i < peaks.length; i++) {
|
|
const t = (i / peaks.length) * totalDuration;
|
|
const x = timeToX(t);
|
|
const h = peaks[i] * (canvasHeight - TIME_AXIS_HEIGHT) * 0.4;
|
|
ctx.beginPath();
|
|
ctx.moveTo(x, midY - h);
|
|
ctx.lineTo(x, midY + h);
|
|
ctx.stroke();
|
|
}
|
|
}
|
|
```
|
|
|
|
### Anti-Patterns to Avoid
|
|
- **DOM div blocks on canvas background:** Mixing DOM and canvas for block rendering creates z-index and positioning hell when the canvas scrolls. Use one or the other; this phase uses canvas-only.
|
|
- **Mutation without command wrapping:** Any direct mutation of `track.cues` that bypasses `CommandHistory.execute()` breaks undo. All state changes must go through commands.
|
|
- **Synchronous backend calls on every mouse move:** Backend sync should happen on `mouseup` (after drag ends), not during drag. During drag, only the local canvas state is updated.
|
|
- **redoStack not cleared on new execute:** This is the classic command-pattern bug — forgetting to clear `redoStack` when a new action is executed creates branching history corruption.
|
|
- **Storing calibrationOffset in beat_times:** The raw beat times from the API must remain uncalibrated. Apply the offset only at render/snap time so changing the offset immediately recalculates without re-fetching.
|
|
|
|
---
|
|
|
|
## Don't Hand-Roll
|
|
|
|
| Problem | Don't Build | Use Instead | Why |
|
|
|---------|-------------|-------------|-----|
|
|
| Beat detection | Custom onset algorithm | Existing `/api/audio/beats` endpoint (librosa) | Already implemented and cached in Phase 2 |
|
|
| Waveform data | Custom audio decoding | Existing `/api/audio/waveform` endpoint | Already handles MP3 via ffmpeg pipe, returns normalized peaks |
|
|
| Color picker UI | Custom HSV canvas widget | `<input type="color">` (browser native) | Sufficient fidelity; avoids 50+ lines of canvas widget code |
|
|
| Undo/redo | External library (immer, etc.) | 40-line CommandHistory class | State is flat; library overhead is unjustified |
|
|
| Canvas object model | Konva.js / Fabric.js | Raw Canvas 2D API | Libraries fight terminal aesthetic, add weight, impose event model overhead |
|
|
| Block persistence | Custom file format | Extend existing ShowModel/shows.py | Show file is already the persistence layer; add block endpoints to shows API |
|
|
|
|
**Key insight:** The heaviest work (audio processing, beat detection, waveform extraction) is already done in Phase 2. Phase 4 is primarily a rendering and interaction problem — all the data is available, the challenge is drawing it and wiring up mouse events correctly.
|
|
|
|
---
|
|
|
|
## Data Model Changes
|
|
|
|
### CueModel — add `duration` field
|
|
```python
|
|
# In lightsync/models/show.py
|
|
class CueModel(BaseModel):
|
|
id: UUID = Field(default_factory=uuid4)
|
|
timestamp: float # block start time in seconds (unchanged)
|
|
duration: float = 4.0 # NEW — block length in seconds; defaults to 4.0 for old files
|
|
mode: Literal["animation", "frame_sequence"] = "animation"
|
|
animation: str | None = None
|
|
params: dict[str, Any] = Field(default_factory=dict)
|
|
# params stores: color (hex str), speed (float), direction (str), length (float), etc.
|
|
```
|
|
|
|
`schema_version` stays at 1. Old show files without `duration` will deserialize with `duration=4.0` via Pydantic default — no migration needed.
|
|
|
|
### New Backend Endpoints (in lightsync/api/timeline.py or shows.py)
|
|
```
|
|
POST /api/shows/{show_id}/tracks/{device_id}/blocks — create block (returns CueModel)
|
|
PATCH /api/shows/{show_id}/tracks/{device_id}/blocks/{block_id} — update block (move/resize/params)
|
|
DELETE /api/shows/{show_id}/tracks/{device_id}/blocks/{block_id} — delete block
|
|
```
|
|
|
|
These endpoints mutate the in-memory show and persist via `show_store.save()`. They are called from command `execute()` and `undo()` methods asynchronously after local state update.
|
|
|
|
---
|
|
|
|
## Beat Calibration Detail
|
|
|
|
**Root cause (verified via GitHub issue #1052):** librosa's `beat_track` uses centered STFT frames (with padding), causing beat timestamps to be systematically 20-60ms late compared to perceptual beat positions.
|
|
|
|
**Phase 4 implementation:** A numeric input (or range slider) labeled "BEAT OFFSET" in a toolbar above the timeline or inline. Stored as `calibrationOffset` float (seconds, default 0.03 = 30ms). Applied only at display and snap time:
|
|
|
|
```javascript
|
|
const displayBeatTime = rawBeatTime - calibrationOffset;
|
|
```
|
|
|
|
The raw beat times from the API cache are never modified. The offset is stored in frontend state and persisted in the show file as `show.analysis.calibration_offset` (add field to `AnalysisBlock` model).
|
|
|
|
---
|
|
|
|
## Layout Changes to index.html
|
|
|
|
The `main-area` div currently holds the animation tile grid. Phase 4 replaces this content with:
|
|
|
|
```
|
|
main-area (flex column)
|
|
├── #timeline-toolbar — zoom control, beat offset slider, snap toggle (flex row, 32px height)
|
|
├── #timeline-canvas — <canvas> fills remaining flex space, scrollable via overflow + scroll listener
|
|
├── #block-inspector — hidden by default; shown as compact 40px horizontal strip when block selected
|
|
└── (transport-bar is the footer — unchanged)
|
|
```
|
|
|
|
The animation library panel moves to a section within the sidebar (below DEVICES) or as a collapsible panel in the sidebar. This is the main structural change to `index.html`.
|
|
|
|
---
|
|
|
|
## Common Pitfalls
|
|
|
|
### Pitfall 1: Canvas devicePixelRatio (Blurry Canvas on HiDPI)
|
|
**What goes wrong:** Canvas drawn at CSS size without accounting for `window.devicePixelRatio` appears blurry on retina/HiDPI displays.
|
|
**Why it happens:** Canvas has intrinsic pixel dimensions separate from CSS dimensions.
|
|
**How to avoid:**
|
|
```javascript
|
|
function resizeCanvas(canvas) {
|
|
const dpr = window.devicePixelRatio || 1;
|
|
const rect = canvas.getBoundingClientRect();
|
|
canvas.width = rect.width * dpr;
|
|
canvas.height = rect.height * dpr;
|
|
const ctx = canvas.getContext('2d');
|
|
ctx.scale(dpr, dpr);
|
|
}
|
|
// Call on init and on window resize
|
|
```
|
|
**Warning signs:** Block edges appear fuzzy; text on canvas is not crisp.
|
|
|
|
### Pitfall 2: Drag State Leaks (mouseup Outside Canvas)
|
|
**What goes wrong:** User starts dragging a block, moves mouse outside the canvas, releases — the block stays in "dragging" state indefinitely because the canvas never receives `mouseup`.
|
|
**Why it happens:** Canvas only receives events when mouse is over it by default.
|
|
**How to avoid:** Attach `mousemove` and `mouseup` listeners to `document` (not canvas) while dragging, remove them on `mouseup`.
|
|
```javascript
|
|
canvas.addEventListener('mousedown', (e) => {
|
|
startDrag(e);
|
|
document.addEventListener('mousemove', onDrag);
|
|
document.addEventListener('mouseup', () => {
|
|
endDrag();
|
|
document.removeEventListener('mousemove', onDrag);
|
|
}, { once: true });
|
|
});
|
|
```
|
|
|
|
### Pitfall 3: Block Overlap and Z-ordering
|
|
**What goes wrong:** Two blocks on the same track can be placed at overlapping times. The drawing order matters for click selection.
|
|
**Why it happens:** No constraint prevents overlapping placements.
|
|
**How to avoid:** During drop, check for time-range overlap with existing blocks on the same track. Reject or nudge the placement. Draw blocks in reverse iteration order (last placed = on top = hit-tested first).
|
|
|
|
### Pitfall 4: Backend Sync Out of Order on Fast Undo/Redo
|
|
**What goes wrong:** Rapid Ctrl+Z presses fire multiple PATCH/DELETE requests; if responses arrive out of order, the server state diverges from client state.
|
|
**Why it happens:** Async fetch calls without sequencing.
|
|
**How to avoid:** For Phase 4, backend sync is fire-and-forget on each command. The client is the source of truth. Full save is triggered on explicit "save" action (Phase 5). For undo/redo, `await` the API call before allowing the next command to prevent races, or use a mutation queue.
|
|
|
|
### Pitfall 5: Animation Library Panel Source of Truth
|
|
**What goes wrong:** The sidebar currently has hardcoded animation tiles in HTML. Phase 4 needs to drag from a dynamic list backed by `ANIMATION_REGISTRY`. If the HTML tiles stay hardcoded, new animation types added to the registry won't appear.
|
|
**How to avoid:** On init, fetch `/api/animations` (or derive from known registry keys) and render the palette dynamically. The 7 types are known and stable for Phase 4 — hardcoding is acceptable, but a note in code to make it dynamic in Phase 5 is good practice.
|
|
|
|
### Pitfall 6: Beat Times Not Loaded Before Canvas Renders
|
|
**What goes wrong:** Timeline renders before the beat analysis fetch completes; beat marks never appear or appear late.
|
|
**Why it happens:** Beat analysis can take 2-5 seconds for a long file.
|
|
**How to avoid:** Beat marks are optional overlays. Show a loading indicator in the time axis area. When beat data arrives, call `beatGrid.load(data)` and trigger a full redraw. This is an async overlay, not a blocking dependency.
|
|
|
|
---
|
|
|
|
## Runtime State Inventory
|
|
|
|
Step 2.5 SKIPPED — this is a greenfield UI/feature phase, not a rename/refactor/migration phase. No runtime string renaming or data migration required. The only data model change is adding a `duration` field with a default value (backward-compatible, no existing record migration needed).
|
|
|
|
---
|
|
|
|
## Environment Availability
|
|
|
|
| Dependency | Required By | Available | Version | Fallback |
|
|
|------------|------------|-----------|---------|----------|
|
|
| Docker | App deployment | Yes | 28.2.2 | — |
|
|
| lightsync container | Running app | Yes (Up 5h) | led2-lightsync | — |
|
|
| librosa | Beat detection | Yes (in container) | >=0.10.0 | — |
|
|
| ffmpeg | Waveform MP3 extraction | Yes (in container, Phase 2 validated) | — | — |
|
|
| HTML5 Canvas 2D | Timeline rendering | Yes (browser native) | All modern browsers | — |
|
|
| `<input type="color">` | Color picker | Yes (browser native, Chromium) | — | — |
|
|
|
|
**No missing dependencies.** All required capabilities are available in the existing container or browser. Phase 4 adds no new pip packages or npm packages.
|
|
|
|
---
|
|
|
|
## State of the Art
|
|
|
|
| Old Approach | Current Approach | When Changed | Impact |
|
|
|--------------|------------------|--------------|--------|
|
|
| DOM divs for timeline blocks | HTML5 Canvas with custom hit-testing | Standard in DAW UIs circa 2015+ | Full rendering control, no CSS fighting, consistent with waveform approach |
|
|
| Global undo stack (Memento/snapshots) | Command pattern with execute/undo per operation | Established pattern | Supports side effects (API calls), lower memory than full state snapshots |
|
|
| Separate playback position poll | `audio.timeupdate` + `requestAnimationFrame` | Phase 2 decision | Sub-frame cursor accuracy without WebSocket latency |
|
|
|
|
**Deprecated/outdated in this codebase:**
|
|
- The hardcoded animation tile buttons in `index.html` (`.anim-tile` elements) — Phase 4 replaces this section with the timeline canvas. The sidebar animation library panel should be dynamically rendered from the registry.
|
|
- The `selectedAnim` variable and `select_animation` WebSocket event in app.js — repurposed in Phase 4 as the drag source for placing blocks (not a global "active animation" concept).
|
|
|
|
---
|
|
|
|
## Code Examples
|
|
|
|
### Rendering Loop Skeleton
|
|
```javascript
|
|
// Source: established Canvas 2D rendering pattern for timelines
|
|
|
|
class TimelineCanvas {
|
|
constructor(canvasEl, audioEl) {
|
|
this.canvas = canvasEl;
|
|
this.ctx = canvasEl.getContext('2d');
|
|
this.audio = audioEl;
|
|
this.pixelsPerSecond = 100;
|
|
this.scrollX = 0;
|
|
this.tracks = []; // TrackModel[]
|
|
this.blocks = []; // CueModel[] (flattened for rendering)
|
|
this.beatTimes = []; // float[] from /api/audio/beats
|
|
this.waveformPeaks = []; // float[] from /api/audio/waveform
|
|
this.calibrationOffset = 0.03;
|
|
this.selectedBlock = null;
|
|
this._dragState = null;
|
|
}
|
|
|
|
render() {
|
|
const ctx = this.ctx;
|
|
const { width, height } = this.canvas.getBoundingClientRect();
|
|
ctx.clearRect(0, 0, width, height);
|
|
|
|
this._drawBackground(ctx, width, height);
|
|
this._drawWaveform(ctx, width, height);
|
|
this._drawTimeAxis(ctx, width);
|
|
this._drawTrackHeaders(ctx, height);
|
|
this._drawBeatMarks(ctx, height);
|
|
this._drawBlocks(ctx);
|
|
this._drawCursor(ctx, height);
|
|
}
|
|
|
|
startRenderLoop() {
|
|
const loop = () => { this.render(); requestAnimationFrame(loop); };
|
|
requestAnimationFrame(loop);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Inspector Strip (Compact Horizontal Layout)
|
|
```javascript
|
|
// Source: D-02 from CONTEXT.md — compact strip matching terminal aesthetic
|
|
|
|
function renderInspector(block) {
|
|
const el = document.getElementById('block-inspector');
|
|
if (!block) { el.style.display = 'none'; return; }
|
|
|
|
el.style.display = 'flex';
|
|
el.innerHTML = `
|
|
<span class="inspector-label">${block.animation?.toUpperCase() ?? 'BLOCK'}</span>
|
|
<label class="inspector-field">
|
|
COLOR
|
|
<input type="color" id="block-color"
|
|
value="${block.params.color ?? '#00ffff'}">
|
|
</label>
|
|
<label class="inspector-field">
|
|
SPEED
|
|
<input type="number" id="block-speed" step="0.1" min="0.1" max="10"
|
|
value="${block.params.speed ?? 1.0}">
|
|
</label>
|
|
<label class="inspector-field">
|
|
DIR
|
|
<select id="block-direction">
|
|
<option value="forward" ${block.params.direction === 'forward' ? 'selected' : ''}>FWD</option>
|
|
<option value="reverse" ${block.params.direction === 'reverse' ? 'selected' : ''}>REV</option>
|
|
</select>
|
|
</label>
|
|
<label class="inspector-field">
|
|
LEN
|
|
<input type="number" id="block-length" step="0.05" min="0.05" max="1"
|
|
value="${block.params.length ?? 0.3}">
|
|
</label>
|
|
`;
|
|
wireInspectorEvents(block);
|
|
}
|
|
```
|
|
|
|
### Beat Calibration Control
|
|
```javascript
|
|
// Source: CONTEXT.md SYNC-02 + librosa issue #1052 research
|
|
|
|
// In timeline toolbar HTML:
|
|
// <label class="toolbar-field">
|
|
// BEAT OFFSET <input type="number" id="beat-offset" step="0.005" min="-0.2" max="0.2" value="0.030">ms
|
|
// </label>
|
|
|
|
document.getElementById('beat-offset')?.addEventListener('input', (e) => {
|
|
const offsetSec = parseFloat(e.target.value);
|
|
timeline.calibrationOffset = offsetSec;
|
|
// No API call needed — display-only calculation applied at render time
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Open Questions
|
|
|
|
1. **Animation library panel placement in sidebar**
|
|
- What we know: Current sidebar has only DEVICES. The animation palette needs to be accessible for drag-to-track.
|
|
- What's unclear: Whether it fits in the sidebar (200px wide may be cramped) or should be a collapsible toolbar above the timeline.
|
|
- Recommendation: Planner decides. Sidebar second panel (`ANIMATIONS` below `DEVICES`) is simplest and avoids layout changes. Items can be compact single-line rows.
|
|
|
|
2. **Show persistence during Phase 4 (before Phase 5)**
|
|
- What we know: Phase 5 is designated for full save/load persistence. Phase 4 blocks added to `track.cues` in memory only.
|
|
- What's unclear: Whether Phase 4 should call `show_store.save()` after each mutation or defer to Phase 5.
|
|
- Recommendation: Phase 4 should fire-and-forget `PATCH /api/shows/{id}/tracks/{device_id}/blocks/{block_id}` on each command (to keep the running show state current in memory), but full show file persistence is Phase 5. Add a "SAVE" button stub in Phase 4 that logs "not yet implemented."
|
|
|
|
3. **Scroll behavior — horizontal scrolling of timeline**
|
|
- What we know: Long songs (5+ min at 100px/s = 30,000px wide) require horizontal scrolling.
|
|
- What's unclear: Whether to use native CSS overflow-x or a virtual scroll implemented in Canvas.
|
|
- Recommendation: Virtual scroll in Canvas — maintain a `scrollX` offset, map all time→X via `timeToX(t) = HEADER_WIDTH + t * PPS - scrollX`. Wheel event updates scrollX and triggers redraw. This is simpler than dealing with scrollable canvas containers.
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
### Primary (HIGH confidence)
|
|
- Existing codebase files — `lightsync/audio/beats.py`, `lightsync/api/audio.py`, `lightsync/models/show.py`, `lightsync/frontend/app.js`, `lightsync/frontend/index.html`, `lightsync/frontend/style.css` — read directly; all patterns verified against live code
|
|
- `.planning/phases/04-timeline-editor/04-CONTEXT.md` — locked decisions D-01 through D-03 and discretion areas
|
|
- GitHub librosa/librosa issue #1052 — confirms 20-60ms systematic beat_track latency bias and `center=False` mitigation
|
|
|
|
### Secondary (MEDIUM confidence)
|
|
- MDN Web Docs — `<input type="color">` browser support, Canvas 2D API — standard browser APIs, highly stable
|
|
- Command pattern undo/redo — multiple sources agree on undoStack/redoStack structure with execute()/undo() per command; verified against Redux docs and JitBlox TypeScript example
|
|
|
|
### Tertiary (LOW confidence — not used in prescriptive recommendations)
|
|
- Konva.js docs — considered and rejected in favor of raw Canvas
|
|
- WebSearch results on DAW timeline patterns — used only to confirm that custom Canvas hit-testing is the established approach, not a novel invention
|
|
|
|
---
|
|
|
|
## Metadata
|
|
|
|
**Confidence breakdown:**
|
|
- Standard stack: HIGH — no new dependencies; all libraries already in container and verified
|
|
- Architecture (coordinate system, hit testing, command pattern): HIGH — established patterns verified across multiple sources and cross-checked against existing codebase patterns
|
|
- Beat calibration offset: HIGH — root cause confirmed via official librosa GitHub issue
|
|
- Pitfalls: HIGH — all based on reading the actual existing codebase and known browser/canvas behavior
|
|
- Inspector layout: HIGH — locked by D-01/D-02/D-03 in CONTEXT.md
|
|
|
|
**Research date:** 2026-04-06
|
|
**Valid until:** 2026-06-01 (stable browser APIs and Python libs; no fast-moving dependencies)
|