116 lines
5.4 KiB
Markdown
116 lines
5.4 KiB
Markdown
# Phase 2: App Core + Audio - Context
|
|
|
|
**Gathered:** 2026-04-03
|
|
**Status:** Ready for planning
|
|
|
|
<domain>
|
|
## Phase Boundary
|
|
|
|
Headless Pi-side Python application: song playback (MP3/FLAC/WAV), beat detection from system audio, choreography data model with JSON persistence, UDP transport client to ESP32, and a monotonic-clock event scheduler. Everything testable from the command line without any TUI.
|
|
|
|
</domain>
|
|
|
|
<decisions>
|
|
## Implementation Decisions
|
|
|
|
### Choreography File Format
|
|
- **D-01:** Choreography files are Pydantic-validated JSON. Schema: `{"song_path": str, "bpm": float|null, "events": [{"timestamp": float, "zone": str, "animation": str, "params": {...}, "loop": int|null, "duration": float|null}]}`. File-level metadata includes song path and optional BPM.
|
|
- **D-02:** Events are an ordered array sorted by timestamp. Each event specifies zone, animation name, params (matching ESP32 protocol), optional loop count, and optional duration.
|
|
- **D-03:** File extension: `.choreo.json` for choreography files.
|
|
|
|
### Audio Pipeline
|
|
- **D-04:** Two separate threads: miniaudio for song playback (with sample-accurate position tracking), sounddevice+aubio for system audio capture and beat detection. Neither thread touches the other's audio device.
|
|
- **D-05:** Beat detection thread uses `call_from_thread()` bridge to post beat events into the asyncio event loop. Beat events carry the detected timestamp.
|
|
- **D-06:** Audio capture uses PipeWire's PulseAudio compatibility layer via sounddevice. Monitor source for system audio loopback.
|
|
|
|
### Playback Clock
|
|
- **D-07:** Absolute monotonic timestamp scheduling. Record `start_time = time.monotonic()` at play start. Each event fires at `start_time + event.timestamp`. Recalculate remaining sleep each iteration — never accumulate relative delays.
|
|
- **D-08:** Drift tolerance: must stay under 20ms over 5 minutes of playback. Validate with test.
|
|
- **D-09:** Pause/resume adjusts `start_time` offset so event scheduling remains correct after resume.
|
|
|
|
### UDP Transport Client
|
|
- **D-10:** Asyncio DatagramProtocol wrapping UDP socket. Sends JSON commands to ESP32 on port 4210. Matches the protocol defined in `docs/protocol.md` exactly.
|
|
- **D-11:** Heartbeat: optional periodic ping to detect ESP32 availability (not blocking — fire-and-forget).
|
|
- **D-12:** Connection state tracked (connected/disconnected) based on ESP32 STATUS responses. Displayed in CLI.
|
|
|
|
### CLI Testing Interface
|
|
- **D-13:** Simple asyncio REPL for testing. Commands: `play <file>`, `pause`, `resume`, `seek <seconds>`, `stop`, `load <choreo.json>`, `add <timestamp> <zone> <animation> [params_json]`, `save <file>`, `status`, `quit`.
|
|
- **D-14:** Not a production interface — Phase 3 TUI replaces this. Minimal error handling, no fancy output.
|
|
|
|
### Claude's Discretion
|
|
- Python package structure (src layout vs flat)
|
|
- asyncio task organization and cancellation patterns
|
|
- Pydantic model field validation details
|
|
- Error handling granularity in transport layer
|
|
- aubio configuration (buffer size, hop size, threshold)
|
|
|
|
</decisions>
|
|
|
|
<canonical_refs>
|
|
## Canonical References
|
|
|
|
**Downstream agents MUST read these before planning or implementing.**
|
|
|
|
### ESP32 Protocol
|
|
- `docs/protocol.md` — UDP JSON protocol spec (port 4210, command structure, animation names, zone values). Pi-side transport MUST match this exactly.
|
|
|
|
### Firmware Source (for integration understanding)
|
|
- `firmware/src/protocol.h` — LedCommand struct, CmdType enum, Zone enum
|
|
- `firmware/src/wifi_server.h` — UDP server interface (port, queue)
|
|
|
|
### Project Context
|
|
- `.planning/PROJECT.md` — Project vision, hardware specs
|
|
- `.planning/REQUIREMENTS.md` — AUD-01..04, CHR-04, CHR-05
|
|
|
|
### Research
|
|
- `.planning/research/STACK.md` — miniaudio, sounddevice, aubio recommendations
|
|
- `.planning/research/ARCHITECTURE.md` — Component boundaries, async/thread architecture
|
|
- `.planning/research/PITFALLS.md` — Timeline drift, audio thread safety, PipeWire compatibility
|
|
|
|
### Prior Phase Context
|
|
- `.planning/phases/01-esp32-firmware/01-CONTEXT.md` — Firmware decisions (JSON protocol, animation names, zone addressing)
|
|
|
|
</canonical_refs>
|
|
|
|
<code_context>
|
|
## Existing Code Insights
|
|
|
|
### Reusable Assets
|
|
- `docs/protocol.md` — Complete protocol specification, Pi-side client must implement this
|
|
- `firmware/src/protocol.h` — Reference for command structure (LedCommand, AnimParams, Zone, CmdType)
|
|
|
|
### Established Patterns
|
|
- JSON command format: `{"v":1, "zone":"...", "animation":"...", "params":{...}}`
|
|
- Zone values: "schrank", "wand", "all"
|
|
- 8 animation names: chase, pulse, rainbow, strobe, color_wash, breathe, sparkle, gradient_sweep
|
|
- UDP port: 4210
|
|
|
|
### Integration Points
|
|
- UDP socket to ESP32 (port 4210) — fire-and-forget commands
|
|
- Choreography JSON files — load/save persistence
|
|
- System audio via PipeWire — beat detection input
|
|
|
|
</code_context>
|
|
|
|
<specifics>
|
|
## Specific Ideas
|
|
|
|
- This phase produces a headless app that Phase 3 wraps with Textual TUI
|
|
- The asyncio event loop and playback clock must be designed for TUI integration from the start (Textual is asyncio-native)
|
|
- Beat detection must be a standalone module that Phase 5 (Live Reactive) can reuse
|
|
- Choreography data model must support the loop/repeat feature from CHR-03 (Phase 3) even if the CLI doesn't expose full loop editing
|
|
|
|
</specifics>
|
|
|
|
<deferred>
|
|
## Deferred Ideas
|
|
|
|
None — discussion stayed within phase scope (auto mode)
|
|
|
|
</deferred>
|
|
|
|
---
|
|
|
|
*Phase: 02-app-core-audio*
|
|
*Context gathered: 2026-04-03*
|