led-sync-studio/.planning/phases/02-app-core-audio/02-CONTEXT.md

Phase 2: App Core + Audio - Context

Gathered: 2026-04-03 Status: Ready for planning

## Phase Boundary

Headless Pi-side Python application: song playback (MP3/FLAC/WAV), beat detection from system audio, choreography data model with JSON persistence, UDP transport client to ESP32, and a monotonic-clock event scheduler. Everything testable from the command line without any TUI.

## Implementation Decisions

Choreography File Format

• D-01: Choreography files are Pydantic-validated JSON. Schema: {"song_path": str, "bpm": float|null, "events": [{"timestamp": float, "zone": str, "animation": str, "params": {...}, "loop": int|null, "duration": float|null}]}. File-level metadata includes song path and optional BPM.
• D-02: Events are an ordered array sorted by timestamp. Each event specifies zone, animation name, params (matching ESP32 protocol), optional loop count, and optional duration.
• D-03: File extension: .choreo.json for choreography files.

Audio Pipeline

• D-04: Two separate threads: miniaudio for song playback (with sample-accurate position tracking), sounddevice+aubio for system audio capture and beat detection. Neither thread touches the other's audio device.
• D-05: Beat detection thread uses call_from_thread() bridge to post beat events into the asyncio event loop. Beat events carry the detected timestamp.
• D-06: Audio capture uses PipeWire's PulseAudio compatibility layer via sounddevice. Monitor source for system audio loopback.

Playback Clock

• D-07: Absolute monotonic timestamp scheduling. Record start_time = time.monotonic() at play start. Each event fires at start_time + event.timestamp. Recalculate remaining sleep each iteration — never accumulate relative delays.
• D-08: Drift tolerance: must stay under 20ms over 5 minutes of playback. Validate with test.
• D-09: Pause/resume adjusts start_time offset so event scheduling remains correct after resume.

UDP Transport Client

• D-10: Asyncio DatagramProtocol wrapping UDP socket. Sends JSON commands to ESP32 on port 4210. Matches the protocol defined in docs/protocol.md exactly.
• D-11: Heartbeat: optional periodic ping to detect ESP32 availability (not blocking — fire-and-forget).
• D-12: Connection state tracked (connected/disconnected) based on ESP32 STATUS responses. Displayed in CLI.

CLI Testing Interface

• D-13: Simple asyncio REPL for testing. Commands: play <file>, pause, resume, seek <seconds>, stop, load <choreo.json>, add <timestamp> <zone> <animation> [params_json], save <file>, status, quit.
• D-14: Not a production interface — Phase 3 TUI replaces this. Minimal error handling, no fancy output.

Claude's Discretion

• Python package structure (src layout vs flat)
• asyncio task organization and cancellation patterns
• Pydantic model field validation details
• Error handling granularity in transport layer
• aubio configuration (buffer size, hop size, threshold)

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

ESP32 Protocol

• docs/protocol.md — UDP JSON protocol spec (port 4210, command structure, animation names, zone values). Pi-side transport MUST match this exactly.

Firmware Source (for integration understanding)

• firmware/src/protocol.h — LedCommand struct, CmdType enum, Zone enum
• firmware/src/wifi_server.h — UDP server interface (port, queue)

Project Context

• .planning/PROJECT.md — Project vision, hardware specs
• .planning/REQUIREMENTS.md — AUD-01..04, CHR-04, CHR-05

Research

• .planning/research/STACK.md — miniaudio, sounddevice, aubio recommendations
• .planning/research/ARCHITECTURE.md — Component boundaries, async/thread architecture
• .planning/research/PITFALLS.md — Timeline drift, audio thread safety, PipeWire compatibility

Prior Phase Context

• .planning/phases/01-esp32-firmware/01-CONTEXT.md — Firmware decisions (JSON protocol, animation names, zone addressing)

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• docs/protocol.md — Complete protocol specification, Pi-side client must implement this
• firmware/src/protocol.h — Reference for command structure (LedCommand, AnimParams, Zone, CmdType)

Established Patterns

• JSON command format: {"v":1, "zone":"...", "animation":"...", "params":{...}}
• Zone values: "schrank", "wand", "all"
• 8 animation names: chase, pulse, rainbow, strobe, color_wash, breathe, sparkle, gradient_sweep
• UDP port: 4210

Integration Points

• UDP socket to ESP32 (port 4210) — fire-and-forget commands
• Choreography JSON files — load/save persistence
• System audio via PipeWire — beat detection input

</code_context>

## Specific Ideas

• This phase produces a headless app that Phase 3 wraps with Textual TUI
• The asyncio event loop and playback clock must be designed for TUI integration from the start (Textual is asyncio-native)
• Beat detection must be a standalone module that Phase 5 (Live Reactive) can reuse
• Choreography data model must support the loop/repeat feature from CHR-03 (Phase 3) even if the CLI doesn't expose full loop editing

## Deferred Ideas

None — discussion stayed within phase scope (auto mode)

---

Phase: 02-app-core-audio Context gathered: 2026-04-03