From c3680306ee76ad00f3293cdd9197a4b52d7b4b8f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Apr 2026 12:14:03 +0200 Subject: [PATCH] docs: research complete --- .planning/research/ARCHITECTURE.md | 395 +++++++++++++++++++++++++++++ .planning/research/FEATURES.md | 179 +++++++++++++ .planning/research/PITFALLS.md | 283 +++++++++++++++++++++ .planning/research/STACK.md | 212 ++++++++++++++++ .planning/research/SUMMARY.md | 227 +++++++++++++++++ 5 files changed, 1296 insertions(+) create mode 100644 .planning/research/ARCHITECTURE.md create mode 100644 .planning/research/FEATURES.md create mode 100644 .planning/research/PITFALLS.md create mode 100644 .planning/research/STACK.md create mode 100644 .planning/research/SUMMARY.md diff --git a/.planning/research/ARCHITECTURE.md b/.planning/research/ARCHITECTURE.md new file mode 100644 index 0000000..65480d9 --- /dev/null +++ b/.planning/research/ARCHITECTURE.md @@ -0,0 +1,395 @@ +# Architecture Patterns: LED Sync Studio + +**Domain:** Terminal-based LED choreography system (Raspberry Pi + ESP32) +**Researched:** 2026-04-03 +**Confidence:** HIGH for component boundaries and data flow; MEDIUM for exact protocol choices (verified against ESP32 docs and WLED/xLights ecosystem) + +--- + +## System Overview + +LED Sync Studio splits cleanly into two physical contexts: the Raspberry Pi runs the user-facing application and all intelligence, while the ESP32-C3 is a dumb executor — it receives animation commands and runs them locally on the LED strips. The Pi never streams raw pixel frames. This is the defining architectural constraint from which everything else follows. + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Raspberry Pi 4B │ +│ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ TUI Layer │ │ App Core │ │ Audio Layer │ │ +│ │ (Textual) │◄──►│ (Async) │◄──►│ (PipeWire) │ │ +│ └──────────────┘ └──────┬───────┘ └──────────────┘ │ +│ │ │ +│ ┌──────▼───────┐ │ +│ │ Transport │ │ +│ │ (UDP/WiFi) │ │ +│ └──────┬───────┘ │ +└─────────────────────────────┼───────────────────────────────┘ + │ JSON commands + ┌─────────▼─────────┐ + │ ESP32-C3 │ + │ │ + │ ┌─────────────┐ │ + │ │ WiFi Server │ │ + │ └──────┬──────┘ │ + │ │ │ + │ ┌──────▼──────┐ │ + │ │ Anim Engine │ │ + │ └──────┬──────┘ │ + │ │ │ + │ ┌──────▼──────┐ │ + │ │ LED Drivers │ │ + │ │ WS2801(SPI) │ │ + │ │ SK6812(RMT) │ │ + │ └─────────────┘ │ + └───────────────────┘ +``` + +--- + +## Component Boundaries + +### Pi-Side Components + +| Component | Responsibility | Communicates With | +|-----------|---------------|-------------------| +| TUI Layer | Render all UI views, accept keystrokes, display timeline/event list | App Core (reactive messages) | +| App Core | State machine, playback clock, choreography scheduler | TUI Layer, Audio Layer, Transport | +| Choreography Engine | Load/save JSON files, manage event sequences, calculate playback position | App Core, file system | +| Audio Layer | Song playback (MP3/FLAC/WAV), real-time beat detection from system audio | App Core (beat events) | +| Transport | JSON serialization, UDP/TCP socket to ESP32, connection health | App Core | + +### ESP32-Side Components + +| Component | Responsibility | Communicates With | +|-----------|---------------|-------------------| +| WiFi Server | Listen for UDP datagrams or TCP connections, parse JSON | Animation Engine | +| Animation Engine | Maintain animation state, run effect loops, handle parameter updates | WiFi Server, LED Drivers | +| LED Drivers | Hardware-specific output: WS2801 via SPI, SK6812 via RMT peripheral | Hardware (GPIO/SPI) | +| Zone Manager | Route animation commands to correct strip (zone A = WS2801, zone B = SK6812) | Animation Engine, LED Drivers | + +--- + +## Data Flow + +### Choreography Playback Mode + +``` +Song file → Audio decoder → PulseAudio/PipeWire sink + │ + elapsed time + │ + Playback Clock ──► Choreography Engine + │ + scheduled event fires + │ + JSON command built + │ + UDP datagram → ESP32 + │ + Animation Engine starts + │ + LEDs execute effect +``` + +### Live Reactive Mode + +``` +System audio → PyAudio/sounddevice → Audio buffer + │ + FFT / onset detection + │ + beat event (aubio) + │ + App Core receives event + │ + mapped animation selected + │ + JSON command → UDP → ESP32 + │ + LEDs react in near-realtime +``` + +### User Interaction Flow + +``` +SSH terminal → Textual event loop → UI widget action → App Core message + │ + state mutation + │ + reactive UI update + (no direct DOM) +``` + +### Choreography File Data Model + +``` +Choreography (file: .json / .yaml) +├── metadata +│ ├── song_path: str +│ ├── song_duration_s: float +│ └── bpm: float | null +└── events: [ + { + time_s: float, # offset from song start + zone: "ws2801" | "sk6812" | "both", + animation: str, # registered name on ESP32 + params: { # animation-specific + color: [r, g, b], + speed: float, + intensity: float, + loop: bool + }, + duration_s: float | null # null = run until replaced + } + ] +``` + +### JSON Command Protocol (Pi → ESP32) + +```json +{ + "zone": "ws2801", + "animation": "pulse", + "params": { + "color": [255, 0, 128], + "speed": 0.8, + "intensity": 1.0 + } +} +``` + +A separate command handles stop/clear: + +```json +{ + "zone": "all", + "animation": "off" +} +``` + +--- + +## Recommended Architecture Pattern: Event-Driven Async Core + +The App Core is an asyncio event loop (Textual already runs on asyncio). Everything hangs off it: + +- **Playback clock** — asyncio task, fires scheduled events from choreography +- **Audio processor** — runs in a thread (audio callbacks are synchronous), posts beat events into the asyncio loop via `call_soon_threadsafe` +- **Network sender** — async UDP socket (asyncio DatagramProtocol), non-blocking +- **File I/O** — async file reads (aiofiles) to avoid blocking the TUI + +This pattern avoids the thread-safety trap: only one asyncio task ever mutates state, threads post events inward via thread-safe bridges. + +### Critical Pattern: Beat Detection Threading + +``` +Thread: audio callback (sounddevice/PyAudio) + └── aubio onset detector + └── beat detected + └── loop.call_soon_threadsafe(post_beat_event) + └── asyncio coroutine handles it + └── sends UDP command +``` + +Audio must run in a thread — audio callbacks cannot be async. This is the one mandatory thread boundary in the system. + +--- + +## Transport Protocol Decision: UDP over TCP + +**Use UDP for animation commands.** Rationale: + +- Animation commands are stateless triggers, not streams. A dropped packet means a missed animation start — tolerable. A delayed retransmission (TCP) means a late animation start — worse for musical sync. +- UDP latency on a local WiFi network is typically 1-5ms vs TCP 10-30ms (with Nagle's algorithm; TCP_NODELAY helps but UDP is still lower). +- ESP32-C3's single RISC-V core handles UDP receive with less overhead than full TCP stack maintenance. +- Command payloads are small (< 200 bytes), well under UDP MTU (1470 bytes on typical WiFi). +- WLED (the leading ESP32 LED firmware) uses UDP for all real-time control protocols (E1.31, DDP, Art-Net). This is industry standard for LED sync. + +**Heartbeat mechanism:** Pi sends a UDP heartbeat every 5s; ESP32 detects loss and enters "safe mode" (off or last known state). + +**Confidence:** HIGH — verified against WLED protocol architecture and ESP32 forum discussions on latency optimization. + +--- + +## ESP32-C3 Constraints and Firmware Architecture + +The ESP32-C3 SuperMini has: +- 400 KB SRAM (tight) +- 4 MB Flash +- Single RISC-V core at 160 MHz +- No hardware FPU (floating-point operations are slower) + +### FreeRTOS Task Allocation + +``` +Core 0 (only core): + Task 1 (high priority): LED output — drives SPI for WS2801, RMT for SK6812 + Task 2 (medium priority): WiFi + JSON receive + Task 3 (low priority): Animation tick — updates frame buffer, feeds to LED task +``` + +WiFi and LED output cannot share timing perfectly on one core. The approach: LED task uses FreeRTOS queue to receive new frame buffers from the animation task. Animation task runs at a fixed interval (e.g., 20ms = 50fps) and posts computed frames into the queue. LED task drains the queue and writes to hardware. + +### LED Driver Architecture + +WS2801 (SPI): +- Uses ESP-IDF SPI master driver +- Clock + Data pins, speed ~1 MHz +- Blocking write per frame is acceptable (160 LEDs * 3 bytes = 480 bytes, fast at 1 MHz) + +SK6812 (single-wire): +- Uses ESP-IDF RMT (Remote Control Transceiver) peripheral +- RMT generates precise timing pulses without CPU involvement +- Non-blocking: CPU queues frame data, RMT transmits via DMA +- SK6812 is RGBW (4 bytes/LED) → 300 LEDs * 4 bytes = 1200 bytes per frame + +Both strips can update concurrently: SPI is blocking but fast, RMT is DMA-driven. Sequence: start RMT (non-blocking), write SPI (blocking), both finish in roughly the same window. + +--- + +## Textual TUI Architecture + +Textual uses a reactive widget tree with message passing. For this app: + +``` +App (root) +├── Header (song info, BPM, mode) +├── TransportBar (play/pause/stop, position scrubber) +├── MainContent (switches between views) +│ ├── TimelineView (horizontal scrollable canvas) +│ │ ├── TimeRuler +│ │ ├── ZoneTrack (WS2801) +│ │ └── ZoneTrack (SK6812) +│ └── EventListView (DataTable of events) +├── AnimationPanel (sidebar: select animation, set params) +└── StatusBar (ESP32 connection, audio status, confidence) +``` + +**Reactive data binding:** Textual's `reactive` descriptor triggers re-renders when playback position changes. The playback clock posts position updates at ~10 Hz, which drives the timeline cursor without flooding the event loop. + +**Timeline rendering:** Custom Textual Canvas or Rich markup — render events as colored blocks at proportional x positions. Horizontal scroll follows playback cursor. + +--- + +## Suggested Build Order (Dependencies) + +The dependency graph drives this order: + +``` +1. ESP32 Firmware (foundation — everything depends on having a working LED executor) + └── LED drivers (WS2801 SPI, SK6812 RMT) + └── Animation engine (effect registry, parameter system) + └── WiFi UDP server + JSON parser + +2. Transport Layer (Pi side — needed to test firmware without UI) + └── UDP socket client + └── JSON command builder + └── Connection management + heartbeat + +3. App Core + State (no UI yet — testable via CLI) + └── Choreography data model (JSON load/save) + └── Playback clock + └── Event scheduler + +4. Audio Layer (can be developed in parallel with App Core after firmware done) + └── Song playback (pygame/miniaudio) + └── Beat detection (aubio + sounddevice) + └── Audio→App Core event bridge + +5. TUI Layer (depends on App Core having stable API) + └── Transport controls + status + └── Event list view + └── Timeline view (most complex — build last within TUI) + └── Animation parameter panel + +6. Integration + Polish + └── Cyberpunk visual theme + └── Choreography file management (browse, rename, delete) + └── Live reactive mode (ties Audio Layer to Animation dispatch) +``` + +**Why firmware first:** Every other component is validated against a real ESP32. Developing the protocol without firmware means writing a mock, then potentially changing everything when the real hardware behaves differently. Ship firmware first, test with curl/netcat, then build the Pi application around proven commands. + +**Why Audio Layer parallel to App Core:** Beat detection and song playback have no dependency on the UI. They can be prototyped as standalone scripts, then integrated. + +--- + +## Cross-Cutting Concerns + +### Latency Budget for Musical Sync + +Musical sync at 120 BPM has 500ms between beats. A cue that fires 50ms early or late is perceptible. Target latency budget: + +| Stage | Budget | Mechanism | +|-------|--------|-----------| +| Playback clock tick accuracy | < 10ms | asyncio event loop, avoid blocking | +| JSON serialization | < 1ms | ujson (faster than stdlib json) | +| WiFi UDP transit | < 10ms | local network, UDP | +| ESP32 receive to animation start | < 5ms | FreeRTOS high-priority LED task | +| **Total** | **< 26ms** | Well within perceptible threshold | + +### Zone Independence + +WS2801 (cabinet, 160 LEDs) and SK6812 (wall, 300 LEDs) must be independently addressable. The JSON protocol always specifies a zone. The ESP32 Zone Manager routes to the correct driver. Both zones can run different animations simultaneously — this is a first-class requirement, not an afterthought. + +### Animation Registry + +Animations are named strings registered in firmware. The Pi does not know animation internals — it only sends a name + parameters. This keeps the protocol stable: new animations are added to firmware without changing the Pi application, as long as the parameter schema is consistent. + +Define a shared animation manifest (JSON file on Pi, #define registry in firmware) so the TUI can offer valid animation names and parameters without hardcoding them in two places. + +--- + +## Anti-Patterns to Avoid + +### Anti-Pattern 1: Frame Streaming from Pi to ESP32 + +**What:** Sending raw pixel arrays (160*3 + 300*4 = 1680 bytes) at 50fps = 84 KB/s continuously over WiFi UDP +**Why bad:** Saturates single-core ESP32 receive pipeline. WiFi on ESP32-C3 has practical throughput around 1-2 Mbps but latency spikes under load. 84 KB/s is feasible but leaves no headroom for WiFi overhead. Any retransmit or congestion causes visible glitch. Also burns CPU on both ends doing serialization. +**Instead:** Commands only. The project already decided this correctly — ESP32 runs animations locally. + +### Anti-Pattern 2: Blocking Audio Processing on Asyncio Loop + +**What:** Running aubio beat detection in an `async def` handler directly +**Why bad:** aubio's Python bindings are synchronous C extensions. Blocking the asyncio event loop for even 10ms causes TUI lag and missed events. +**Instead:** Thread worker with `call_soon_threadsafe` bridge (documented above). + +### Anti-Pattern 3: Polling Playback Position from TUI + +**What:** TUI widget calling `time.time()` on every render to calculate playback position +**Why bad:** Playback position should be authoritative state in App Core. Multiple widgets reading independently diverge. If audio playback pauses, widgets don't know. +**Instead:** App Core maintains `current_position_s` as reactive state. TUI subscribes to updates. + +### Anti-Pattern 4: JSON Parsing on ESP32 ISR + +**What:** Parsing received UDP JSON inside the WiFi receive interrupt handler +**Why bad:** JSON parsing is slow (malloc, string iteration) and will cause watchdog resets from ISR +**Instead:** WiFi ISR copies raw bytes to a FreeRTOS queue. Separate task drains queue and parses JSON with enough stack space. + +--- + +## Scalability Considerations + +This is a single-user, single-ESP32 system by design. Scalability means: + +| Concern | Current | Future if needed | +|---------|---------|-----------------| +| More LED strips | Add second ESP32, Pi sends duplicate commands | Multi-target transport layer | +| More animations | Add to ESP32 firmware + manifest file | Plugin architecture on ESP32 | +| Longer songs | JSON file size grows linearly — no issue | No change needed | +| Multiple choreographies | File system already handles this | Directory browser UI | + +--- + +## Sources + +- [WLED Project Documentation](https://kno.wled.ge/) — Industry reference for ESP32 LED control architecture, UDP protocol patterns +- [WLED Control Interfaces (DeepWiki)](https://deepwiki.com/wled/WLED-Docs/6-control-interfaces) — Protocol specifics (E1.31, DDP, UDP) +- [ESP32-C3 Technical Reference Manual](https://www.espressif.com/sites/default/files/documentation/esp32-c3_technical_reference_manual_en.pdf) — RMT peripheral, SPI, single-core constraints +- [Getting Started with ESP32-C3 Super Mini](https://randomnerdtutorials.com/getting-started-esp32-c3-super-mini/) — Hardware specs (400KB RAM, 4MB Flash, 160MHz) +- [Textual Workers Guide](https://textual.textualize.io/guide/workers/) — Background task architecture, thread safety +- [Textual Events and Messages](https://textual.textualize.io/guide/events/) — Async message passing patterns +- [Real-Time Beat Tracking (TISMIR 2024)](https://transactions.ismir.net/articles/10.5334/tismir.189) — Latency sources in beat tracking pipelines +- [dancyPi: Audio-reactive LEDs on RPi + ESP](https://github.com/naztronaut/dancyPi-audio-reactive-led) — Reference architecture for similar system +- [ESP32 UDP Latency Discussion](https://github.com/espressif/arduino-esp32/issues/1283) — UDP vs TCP latency on ESP32 +- [aubio Real-Time Beat Prediction](https://www.maxhaesslein.de/notes/real-time-beat-prediction-with-aubio/) — aubio integration pattern for live beat detection diff --git a/.planning/research/FEATURES.md b/.planning/research/FEATURES.md new file mode 100644 index 0000000..10cbe0d --- /dev/null +++ b/.planning/research/FEATURES.md @@ -0,0 +1,179 @@ +# Feature Landscape + +**Domain:** LED Choreography / Music-Synced Light Show Controller +**Researched:** 2026-04-03 +**Confidence:** MEDIUM — based on xLights, WLED, LedFx, Jinx! analysis; ESP32/Pi-specific constraints from community sources + +--- + +## Reference Products Surveyed + +| Product | Model | Relevance | +|---------|-------|-----------| +| xLights | Timeline-based sequencer, open source | Direct analog — timeline + audio + multi-model | +| WLED | ESP32 firmware + web UI | What runs on the ESP side; preset/segment model | +| LedFx | Network audio-reactive engine | Live reactive reference implementation | +| Jinx! | LED matrix control, audio/MIDI sync | Beat detection + sound-to-light baseline | +| VenueMagic | Professional show control | High-end cue-list model | + +--- + +## Table Stakes + +Features users expect. Missing = product feels incomplete or unusable. + +| Feature | Why Expected | Complexity | Notes | +|---------|--------------|------------|-------| +| Audio waveform display on timeline | Every LED sequencer shows this; without it, manual timing is blind | Medium | Display audio amplitude/waveform as scrubbing reference | +| Play / pause / seek | Fundamental transport controls | Low | Must work while animation preview is live on LEDs | +| Per-zone independent control | Both zones (WS2801 + SK6812) behave differently; single global control frustrates | Low | Two named zones minimum | +| Block-based timeline editor | xLights, Vixen, and all sequencers use drag-drop blocks on a horizontal track | High | Blocks represent animation events at timestamps | +| Animation library (built-in effects) | Users expect ready-made animations: chase, pulse, rainbow, strobe, color wash | Medium | At minimum 8–12 distinct named effects per zone | +| Per-animation parameter config | Speed, color, intensity — parameterization is baseline since Jinx! v1 | Medium | Each effect has configurable fields exposed in sidebar/panel | +| Save / load choreography files | Without persistence, no session survives a restart | Low | JSON or YAML; portable between sessions | +| Live reactive mode (beat detection) | Users familiar with WLED/LedFx expect this as a parallel mode | High | System audio capture via PipeWire/PulseAudio; onset/beat detection | +| Visual feedback of current position | Playhead showing current song position during playback | Low | Mandatory for sequencing work | +| Loop / repeat control for blocks | Avoids manually placing repeated blocks; xLights, WLED playlists do this | Low | Repeat N times or until next event | +| Manual timing mark placement | xLights "tapper" workflow: press key on beat to stamp marks | Low | Core ergonomic feature for choreography | + +--- + +## Differentiators + +Features that set the product apart. Not universally expected, but create strong value. + +| Feature | Value Proposition | Complexity | Notes | +|---------|-------------------|------------|-------| +| Cyberpunk TUI (Textual, SSH-native) | No existing LED sequencer runs as a terminal UI over SSH; this is genuinely unique | High | xLights/LedFx all require a desktop or browser on the same machine | +| Dual-mode in one tool (timeline + live reactive) | Most tools are one or the other; switching modes without restarting is rare | High | Unified session state; reactive mode as a layer on top of timeline | +| Zone-aware beat-reactive routing | Route bass frequencies → Zone A, treble → Zone B | Medium | LedFx does this with virtuals; no TUI equivalent exists | +| Auto-beat detection timing marks | VAMP-style beat analysis that stamps timing marks automatically, then user refines | High | xLights uses VAMP plugins; librosa or aubio in Python are viable | +| Streaming audio source (Spotify) | Playing from Spotify while syncing LEDs to it bridges the "my music is streaming" gap | Very High | Requires Spotify Connect or OS audio loopback — treat as stretch goal | +| Animation preview in-terminal | ASCII/block-character real-time preview of what LEDs will do, without needing to look at physical strip | High | Textual widget, color simulation | +| Per-block transition effects | Crossfade, cut, dissolve between adjacent animation blocks | Medium | WLED playlists have transition time; xLights has layer blending | +| Export / import xLights sequences | Import existing xLights .xsq files to reuse community-built shows | Very High | Complex format; defer unless strong demand | +| Scripted effect language | Jinx!Script-style: define custom animation logic in a mini-DSL | Very High | Not needed for V1; advanced power-user feature | +| Undo / redo | Timeline editors without undo are painful; all quality DAWs have it | Medium | Required for comfortable editing; Textual app state management | + +--- + +## Anti-Features + +Features to deliberately NOT build. + +| Anti-Feature | Why Avoid | What to Do Instead | +|--------------|-----------|-------------------| +| Web UI / browser interface | Contradicts the entire terminal-SSH design; adds a full stack (server, frontend) | SSH + Textual TUI is the whole point | +| Frame-by-frame pixel streaming from Pi to ESP | WiFi jitter (47ms+ on ESP8266, 12ms on ESP32) makes real-time frame data unreliable; kills bandwidth | ESP executes animations locally; Pi sends only named commands + params | +| Support for more than one ESP32 | Multiplies coordination complexity exponentially; no hardware for it | Keep single ESP32-C3 as sole controller | +| Cloud sync / multi-user collaboration | No network dependency is a feature for a local-first tool | Local files only (JSON/YAML) | +| DMX / E1.31 / sACN protocol support | xLights is already excellent at this; out of hardware scope | JSON-over-WiFi to ESP32 is the transport layer | +| Mobile app | Out of scope, contradicts SSH-terminal model | Terminal is accessible anywhere SSH works | +| 2D matrix effects | Hardware is 1D strips; 2D matrix effects (WLED matrix, xLights matrix) are irrelevant | All effects are 1D strip effects | +| Automatic Spotify token management (OAuth flow) | Extremely complex auth flow through terminal; security risk | If Spotify is implemented, use device auth flow or OS-level audio capture instead | + +--- + +## Feature Dependencies + +``` +Audio waveform display ──────────────────> Audio file loading (MP3/FLAC/WAV) + | +Manual timing mark placement ──────────────> Audio waveform display + | +Auto-beat detection timing marks ──────────> Audio waveform display + librosa/aubio + +Block-based timeline editor ───────────────> Per-zone independent control + | +Per-block transition effects ──────────────> Block-based timeline editor + | +Loop / repeat control ─────────────────────> Block-based timeline editor + +Live reactive mode ────────────────────────> Beat detection (onset/BPM) + | +Zone-aware beat-reactive routing ──────────> Live reactive mode + per-zone control + +Animation library ─────────────────────────> (standalone, no deps) + | +Per-animation parameter config ────────────> Animation library + | +Animation preview in-terminal ─────────────> Animation library + Per-animation parameter config + +Save / load choreography ─────────────────> Block-based timeline editor (saves timeline state) + +Undo / redo ───────────────────────────────> Block-based timeline editor (wraps edit operations) + +Dual-mode (timeline + live reactive) ─────> Timeline editor + Live reactive mode (both complete first) +``` + +--- + +## MVP Recommendation + +Build in this order — each tier unblocks the next: + +**Tier 1 — Foundation (without this, nothing works):** +1. Audio file loading and playback (MP3/FLAC/WAV) +2. Play / pause / seek transport +3. Per-zone control model (WS2801 + SK6812 as named zones) +4. Animation library (8–12 built-in effects with parameter config) +5. Pi → ESP JSON command protocol + +**Tier 2 — Core Editor (the actual product):** +6. Audio waveform display in timeline +7. Block-based timeline editor (place/move/resize/delete animation blocks) +8. Manual timing mark placement (tapper) +9. Loop / repeat per block +10. Save / load choreography (JSON) + +**Tier 3 — Live Reactive (second major mode):** +11. System audio capture (PipeWire/PulseAudio) +12. Beat / onset detection +13. Live reactive mode (beat → trigger animation) + +**Tier 4 — Polish (quality of life):** +14. Undo / redo +15. Zone-aware beat-reactive routing +16. Auto-beat detection timing marks (librosa/aubio) +17. Per-block transition effects + +**Defer indefinitely:** +- Spotify integration (very high complexity, unclear if system audio loopback is simpler) +- Animation preview in-terminal (nice to have, not critical when you have physical LEDs) +- Export/import xLights sequences + +--- + +## Key Observations + +**What xLights does that this tool must do:** +- Waveform display with zoom and timing marks is non-negotiable for manual sync work +- Timing tracks with auto-generated beat marks (xLights uses VAMP plugins; Python has `librosa.beat.beat_track`) dramatically speed up choreography + +**What WLED does that informs ESP firmware design:** +- Segments = zones; presets = named animation configs; playlists = ordered preset sequences with timing +- The WLED mental model (preset per zone, triggered by command) closely matches what this project's JSON protocol needs + +**What LedFx does that informs live reactive mode:** +- Melbank (mel-scale filterbank) over raw FFT for audio features is the right approach — perceptually uniform frequency buckets +- Latency target for reactive mode: ~16ms audio-to-LED pipeline (LedFx achieves this) +- Virtual device layering (effect → post-process → device) is a useful architecture even for two zones + +**Latency reality check (MEDIUM confidence — community sources):** +- WiFi jitter on ESP32: ~12ms variance (acceptable for music sync at ~70ms total budget) +- System audio monitoring buffer latency: 100–300ms (must be accounted for in beat detection offset) +- Command-based approach (vs frame streaming) sidesteps most latency problems — the animation runs autonomously on ESP, only start/stop/param-change commands cross WiFi + +--- + +## Sources + +- xLights Manual (timing tracks, waveform, effects): https://manual.xlights.org/xlights/ +- xLights 2025 release notes: https://xlights.org/2025/ +- WLED segments/presets/playlists: https://kno.wled.ge/features/segments/ and https://kno.wled.ge/features/presets/ +- LedFx documentation: https://docs.ledfx.app/en/latest/ +- LedFx DeepWiki architecture: https://deepwiki.com/LedFx/LedFx +- Jinx! LED Matrix Control user manual: https://www.hbksaar.de/vorlesungen/details/raumstrategien-mit-kabelgebundenen-farben?file=files/user_accounts/user_46/jinx-usermanual-2.4.pdf +- DIY WiFi LED sync pitfalls (latency/jitter): https://www.oreateai.com/blog/diy-wifi-light-shows-syncing-your-leds-without-breaking-the-bank/b284c3a86d52b27010cd88fe5646e558 +- WLED audio reactive (WLED MoonModules): https://mm.kno.wled.ge/WLEDSR/Reactive-Animations/ +- ESP32 audio reactive LED strip: https://www.esp32s.com/blog/ultimate-guide-to-building-a-sound-reactive-led-system-with-esp32-and-wled/ diff --git a/.planning/research/PITFALLS.md b/.planning/research/PITFALLS.md new file mode 100644 index 0000000..c48d861 --- /dev/null +++ b/.planning/research/PITFALLS.md @@ -0,0 +1,283 @@ +# Domain Pitfalls: LED Sync Studio + +**Domain:** Terminal-based LED choreography with ESP32 + WiFi + audio sync +**Researched:** 2026-04-03 +**Overall confidence:** HIGH — most pitfalls are well-documented in ESP32 community forums and official Espressif issue trackers + +--- + +## Critical Pitfalls + +Mistakes that cause rewrites, hardware damage, or make the project non-functional. + +--- + +### Pitfall 1: WiFi Interrupts Corrupt RMT/Single-Wire LED Timing on ESP32-C3 + +**What goes wrong:** The SK6812 strip (single-wire NeoPixel-style protocol) is driven via the ESP32's RMT peripheral. On the ESP32-C3 (single core, RISC-V), the WiFi driver shares the same CPU core as RMT interrupt handling. Every ~1ms, the WiFi subsystem inserts an approximately 5 µs delay into the RMT signal stream — far beyond the 150 ns tolerance of WS2812/SK6812 timing. The result is flickering, wrong colors, or partial frame corruption. This is a documented, unsolved problem that appears repeatedly in FastLED, ESPHome, and ESP-IDF issue trackers as recently as February 2026. + +**Why it happens:** The ESP32-C3 has one CPU core. The dual-core ESP32 can pin the LED task to core 1 and WiFi to core 0 — that workaround does not exist on the C3. RMT DMA helps but the DMA buffer can be too small, causing the RMT transmission to break into fragments with WiFi-induced gaps between them. + +**Consequences:** Visible LED flicker whenever a WiFi packet is sent or received. Particularly bad when command acknowledgements and animation frames coincide. Makes music-reactive LED effects look broken even when the logic is correct. + +**Prevention:** +- Use the RMT peripheral with DMA enabled and set `rmt_symbols` (buffer size) large enough to hold an entire full frame in one transmission, avoiding mid-frame DMA refills. +- Disable WiFi power saving (`esp_wifi_set_ps(WIFI_PS_NONE)`) — this reduces the most common interrupt spike pattern. +- Consider a dedicated LED controller chip (e.g., WS2801 over SPI does not have this problem — it is clock-driven and tolerates interruptions). +- For SK6812 specifically: test with WiFi transmitting at maximum rate before accepting the hardware works. Do not test LED timing with WiFi idle. +- Watch the FastLED issue [#1657](https://github.com/FastLED/FastLED/issues/1657) for ESP32-C3 specific fixes as the library evolves. + +**Detection:** Flash a static color on SK6812, then send rapid JSON commands from Pi. If LEDs flicker during command bursts but are stable in silence, this pitfall is active. + +**Phase:** Address in firmware foundation phase (Phase 1 of ESP32 work). Validate the RMT+WiFi coexistence before building any animation logic on top. + +--- + +### Pitfall 2: WS2801 SPI Speed vs. Long Cable Signal Integrity + +**What goes wrong:** The WS2801 uses a clock-and-data SPI protocol. The 5m cable run (U-turn around a wardrobe = potentially 10m round trip for the signal ground loop) introduces capacitive load that degrades signal edges at higher SPI clock speeds. Additionally, the WS2801 5V version is documented as "low speed" — writing too fast causes the chip to hang or produce wrong colors. WiFi RF noise from the ESP32's antenna can also couple into the SPI data line if the cable is nearby and unshielded. + +**Why it happens:** Long cables act as antennas and RC low-pass filters. SPI has no built-in re-synchronization — a degraded clock edge causes a wrong bit, and the error propagates through all subsequent LEDs in the chain. + +**Consequences:** Corrupt colors on LEDs at the far end of the strip. Intermittent failures that appear at random (correlated with WiFi transmissions). Very hard to debug without an oscilloscope. + +**Prevention:** +- Set SPI clock to 1 MHz or lower for the WS2801 strip. The chip supports up to 25 MHz in theory, but 1–2 MHz is safe with long cables. +- Use a 33Ω series resistor at the ESP32 end of the data and clock lines to dampen ringing. +- Add a 100–470 µF capacitor across the 5V/GND power rails at the strip's input connector. +- Route SPI cables away from the ESP32's antenna and WiFi power stages. +- Use twisted pair for clock and data lines if possible. + +**Detection:** Set all LEDs to a solid color; if far-end LEDs show wrong color or flicker only at the strip tail, this is signal integrity, not timing. Reduce SPI speed by half and retest. + +**Phase:** Address in hardware wiring before firmware development. Document the working SPI clock speed in firmware config before writing animation code. + +--- + +### Pitfall 3: Sub-Second Choreography Sync Requires Absolute Timestamps, Not Relative Delays + +**What goes wrong:** The intuitive approach to timeline execution is: "start playing the song, set a timer, fire events when timer reaches the event timestamp." In Python, `time.sleep()` drifts. `asyncio` event loop delays accumulate. Over a 3-minute song, drift of 100–200ms per minute is typical, meaning by the end of the song, animations fire 300–600ms late. This makes the choreography feel broken even though the logic is correct. + +**Why it happens:** `time.sleep()` is a minimum sleep — the OS can wake the process later. Python's GIL, garbage collector pauses, and SSH I/O buffering all add latency. The Textual event loop also processes render and input events on the same loop, competing with timer callbacks. + +**Consequences:** Animations gradually fall out of sync with song playback. The longer the song, the worse the drift. A ballistic "fire and forget" approach accumulates error permanently. + +**Prevention:** +- Use absolute timestamps throughout: record `start_wall_time = time.monotonic()` when playback begins, and for each event compute `fire_at = start_wall_time + event.timestamp`. Schedule based on `fire_at - time.monotonic()` to get the remaining sleep, recalculated fresh each time. +- Use `time.monotonic()` not `time.time()` — monotonic is immune to NTP adjustments. +- Run the choreography engine as a dedicated `asyncio` task that polls remaining time in a tight loop (sleep 1–5ms at a time) rather than a single long sleep. +- Validate by logging actual fire times vs. expected: measure drift at 30s, 60s, 120s into a test song. Reject any implementation with drift > 20ms/minute. + +**Detection:** Log `time.monotonic()` at each event fire and compare to the intended song timestamp. Drift that grows linearly over time confirms this pitfall. + +**Phase:** Address in choreography engine milestone before adding any visual polish to the timeline editor. + +--- + +### Pitfall 4: WiFi Packet Jitter Makes "Fire-and-Forget" JSON Commands Unreliable for Music Sync + +**What goes wrong:** JSON commands from the Pi to the ESP32 over WiFi arrive with variable latency. On a home network, typical UDP jitter is ±500 µs under idle conditions but can spike to 10–50ms when other WiFi clients are active or the ESP32's WiFi stack clusters packets. For commands tied to beat events (kick drum, chorus drop), a 50ms late arrival is perceptible as a missed sync. + +**Why it happens:** The ESP32 WiFi stack has documented behavior where UDP packets cluster: they queue for 10–20ms then burst. Home 2.4GHz networks have contention with phones, smart home devices, and neighboring APs. There is no TCP-level retry in a UDP choreography protocol without explicit implementation. + +**Consequences:** Commands arrive too late for beat-locked animations. If TCP is used instead, the TCP ACK round-trip adds another 5–15ms and TCP's Nagle algorithm can buffer small JSON packets, adding additional latency. The system works in testing (quiet network, idle ESP32) but fails in realistic conditions. + +**Prevention:** +- Use UDP, not TCP, for command delivery. TCP's Nagle algorithm and ACK round-trips are worse than UDP jitter for real-time choreography. +- Disable Nagle if TCP is used: `socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)`. +- Design the animation command protocol to be "early arrival tolerant": include the intended start timestamp in the JSON payload and have the ESP32 defer execution until that timestamp (using its own `millis()` clock synchronized via NTP or initial handshake). +- Keep commands small (< 200 bytes) to avoid fragmentation. +- Disable WiFi power saving on the ESP32: `esp_wifi_set_ps(WIFI_PS_NONE)`. +- For beat-detection live mode, accept that ~20ms jitter exists and design animations that look good with that tolerance (avoid hard-cut color changes on beat; prefer quick fades that mask ±20ms arrival variance). + +**Detection:** Log send timestamp on Pi and receive timestamp on ESP32 (compare `millis()` after initial NTP sync). Plot the distribution. If 95th-percentile latency exceeds 50ms, the protocol needs redesign. + +**Phase:** Address in communication protocol design before the live reactive mode milestone. + +--- + +## Moderate Pitfalls + +--- + +### Pitfall 5: ESP32-C3 Has No Second Core — Standard Dual-Core WiFi/LED Isolation Advice Does Not Apply + +**What goes wrong:** Almost all ESP32 LED + WiFi tutorials, StackOverflow answers, and FastLED documentation recommend "pin the LED task to core 1, pin WiFi to core 0" as the solution for WiFi-induced LED flicker. The ESP32-C3 SuperMini has one core (RISC-V). This advice silently does nothing — `xTaskCreatePinnedToCore(..., 1)` on a single-core chip is accepted without error but has no effect. + +**Why it happens:** ESP32-C3 is a cut-down variant. Its documentation is sometimes mixed with the dual-core ESP32 docs in tutorials. + +**Prevention:** +- Do not follow dual-core pinning advice for this hardware. +- The correct mitigations for C3 are: RMT DMA with large buffers, `WIFI_PS_NONE`, and SPI (WS2801) for the strip that needs the most reliability. +- Double-check any library or tutorial's target chip before applying its advice. + +**Detection:** If `CONFIG_FREERTOS_UNICORE` is set in sdkconfig, you are on a single-core chip and core pinning is a no-op. + +**Phase:** Firmware setup — verify chip variant before writing any FreeRTOS task pinning code. + +--- + +### Pitfall 6: Textual's Async Event Loop Blocks on Audio I/O if Not Isolated + +**What goes wrong:** PipeWire/PulseAudio audio capture (via `pyaudio` or `sounddevice`) uses blocking reads or callbacks on a native thread. If the audio capture thread tries to update Textual widgets directly (e.g., calling `self.app.query_one(...).update()`), it bypasses Textual's thread safety model, causing race conditions, corrupted widget state, or deadlocks. The bug is intermittent and often appears only under load. + +**Why it happens:** Textual runs on Python asyncio. Its widgets and reactive variables are not thread-safe. Audio libraries operate on OS-level audio threads outside asyncio. + +**Consequences:** The app works during development (small audio buffers, idle system) but crashes or freezes in performance conditions (long choreography run, many beat events per second). + +**Prevention:** +- Run audio capture in a dedicated thread using Textual's `self.run_worker(..., thread=True)`. +- Never call Textual widget methods from the audio thread. Instead, use `self.app.call_from_thread(callback)` to marshal updates back to the asyncio event loop. +- Use a `queue.Queue` to pass beat detection events from audio thread to the asyncio event loop, polled by an asyncio task. +- Set the PipeWire quantum to 512 or 256 samples for low-latency capture (configures via `pw-metadata` or `PIPEWIRE_QUANTUM` environment variable). + +**Detection:** Enable Python threading warnings. If `RuntimeError: Non-thread-safe operation invoked on the event loop` appears under load, this pitfall is active. + +**Phase:** Audio integration milestone. Design the thread boundary upfront, not as a fix after the fact. + +--- + +### Pitfall 7: SK6812 is RGBW (4 bytes per LED), Not RGB (3 bytes) + +**What goes wrong:** SK6812 RGBW LEDs have a white channel in addition to RGB, requiring 4 bytes of data per LED instead of 3. Code written assuming 3-byte RGB (as for WS2812B) will send incorrect data, corrupting color output for the entire strip after the first LED that is calculated wrong. + +**Why it happens:** SK6812 is often listed alongside WS2812 as "NeoPixel-compatible." That is true for the protocol timing, but not for the data width. Library defaults may vary — some assume RGBW, some assume RGB. + +**Consequences:** Colors appear shifted or incorrect. Animations that look correct for the first few LEDs gradually degrade. White channel stuck on or off. + +**Prevention:** +- Explicitly configure LED type as `SK6812` (RGBW, 4 bytes/LED) in the firmware, not as `WS2812` (RGB, 3 bytes/LED). +- When designing animation color representations on the Pi side, include a W (white) channel in the JSON protocol from day one, even if it is always 0 initially. Adding it later requires a protocol version bump. +- Total strip data frame size: 300 LEDs × 4 bytes = 1,200 bytes per frame. Account for this in RMT buffer sizing. + +**Detection:** Set the first LED to pure red (255, 0, 0). If the second LED glows slightly (white channel bleed), the byte layout is wrong. + +**Phase:** Firmware LED protocol definition — before any animation code is written. + +--- + +### Pitfall 8: Power Budget for 300 SK6812 + 160 WS2801 LEDs is Substantial + +**What goes wrong:** SK6812 RGBW at full white: ~60 mA per LED. 300 LEDs × 60 mA = 18A peak. WS2801 RGB at full white: ~60 mA per LED. 160 LEDs × 60 mA = 9.6A peak. Combined theoretical maximum: ~27A at 5V. Even at 30% brightness, that is 8A. A single 5V/2A supply (or USB power) will cause brown-out resets on the ESP32, corrupt SPI/RMT signals due to voltage drooping, and potentially damage the power supply. + +**Why it happens:** LED power calculations are done at 100% white brightness as a worst case. Developers assume "I'll keep brightness low" without enforcing it in firmware — but demo patterns often briefly hit full white. + +**Consequences:** ESP32 brown-out reset mid-performance. Voltage droop corrupts LED data signals. Intermittent failures that look like software bugs. + +**Prevention:** +- Power each LED strip from a dedicated 5V supply rated for the strip's expected current, not from the ESP32's USB or VIN pin. +- Implement a firmware-level brightness cap (e.g., max 40% = ~7A combined). This is a hard limit, not a default. +- Add a 1000 µF capacitor at each strip's power input to absorb transient draws. +- Run a separate ground wire from power supply to ESP32 GND alongside the LED power cables to avoid ground loops. +- Use adequately sized wire: at 10A over 1m, use at minimum 20 AWG; prefer 18 AWG. + +**Detection:** Measure voltage at the far end of the LED strip under full-on load. If it drops below 4.5V, the power delivery is inadequate. + +**Phase:** Hardware setup — before any firmware is written. A brown-out during firmware development wastes hours of debugging time on a hardware problem. + +--- + +### Pitfall 9: JSON Protocol Evolution Without Versioning Causes Firmware/App Desync + +**What goes wrong:** The Pi app and ESP32 firmware are developed in parallel. Early in development, the JSON command schema is simple. As features are added (new animation types, RGBW white channel, zone targeting, timing payloads), the schema grows. Without a version field, old firmware silently ignores unknown fields or misparses them — and there is no way to detect the firmware is out of date. + +**Why it happens:** JSON is schema-less. Missing fields cause no error in most parsers. The developer assumes firmware and app are always in sync because they are updated together — until they are not. + +**Consequences:** New animation commands do nothing on ESP32 without any error output. Debugging requires checking both Pi logs and ESP32 serial output simultaneously, which is slow over SSH. + +**Prevention:** +- Include a `"protocol_version": 2` field in every command from day one. +- Have the ESP32 respond with its supported protocol version on connection or on version mismatch. +- Version the command schema in a shared document (even a comment in the firmware source) that is updated whenever the JSON structure changes. +- Design the protocol to be forward-compatible: ESP32 ignores unknown fields rather than crashing, but logs a warning. + +**Detection:** If adding a new animation parameter on the Pi side produces no change in LED behavior with no error, suspect a protocol version mismatch. + +**Phase:** Communication protocol design milestone — add versioning from the first command sent. + +--- + +## Minor Pitfalls + +--- + +### Pitfall 10: Beat Detection Accuracy Degrades with Complex Mixes + +**What goes wrong:** Simple FFT energy-threshold beat detection works well for 4/4 kick-heavy electronic music but fails on jazz, classical, or poly-rhythmic tracks. The beat detector fires on loud transients (cymbals, snare reverb) rather than the "felt" beat. False positives cause the LED to flash randomly, which looks worse than no sync at all. + +**Prevention:** +- Use a moving energy average comparison (current band energy vs. recent window average) rather than a global energy threshold. This adapts to track loudness. +- Focus the beat detection on the sub-bass frequency band (20–200 Hz) for kick detection specifically. +- Implement a "beat hold" period (minimum 200ms between beats) to filter double-trigger from reverb. +- Expose the sensitivity and frequency band as user-adjustable parameters in the TUI, so the user can tune per song. +- Accept that reactive mode is "best effort" and document it — the timeline/choreography mode is where precise sync is guaranteed. + +**Phase:** Beat detection milestone — include a calibration UI from the start. + +--- + +### Pitfall 11: SSH Terminal Rendering Over Slow/High-Latency Link Causes Textual Frame Drops + +**What goes wrong:** Textual renders full-screen TUI updates over SSH. On a high-quality local network SSH connection, this is fast. But Textual's default refresh rate may generate more terminal output than the SSH pipe can deliver, causing visible tearing, delayed input response, and excessive CPU usage on the Pi for SSH I/O. + +**Prevention:** +- Set a reasonable Textual refresh rate (e.g., 30 fps maximum) — Textual's `REFRESH_RATE = 30` class variable. +- Profile SSH I/O bandwidth: the Pi's `sshd` and the client terminal emulator both have buffer limits. +- Test over a simulated degraded link (e.g., `tc qdisc add dev lo root netem delay 20ms`) to expose rendering issues before deployment. + +**Phase:** TUI development — set refresh rate early; do not default to Textual's maximum. + +--- + +### Pitfall 12: ESP32-C3 SuperMini Flash is Only 4MB with ~1.25MB Usable for Firmware + +**What goes wrong:** The ESP32-C3 SuperMini has 4MB flash but the default partition table allocates only approximately 1.25MB for the application. A firmware that includes WiFi, JSON parsing (ArduinoJson), FastLED/NeoPixel library, and multiple animation routines can approach this limit. Adding OTA update support requires a second app partition, halving available firmware space to ~600KB. + +**Prevention:** +- Build the firmware early and check its size before writing many animation effects. Use `idf.py size` or Arduino IDE's compilation output. +- Use a custom partition table to allocate more flash to the app partition if OTA is not required. +- Prefer lean libraries: ArduinoJson v7 is more compact than v6; avoid including unused animation effects. +- Note: Some clone ESP32-C3 SuperMini boards have been found with no external flash at all (flash-less variants). Verify the chip markings match the ESP32-C3FH4 (which has embedded 4MB flash). + +**Phase:** Firmware setup — check flash size and partition table before writing animation code. + +--- + +## Phase-Specific Warnings + +| Phase Topic | Likely Pitfall | Mitigation | +|-------------|---------------|------------| +| ESP32 firmware init | WiFi+RMT coexistence (Pitfall 1) | Test WiFi+LED simultaneously on day 1 of firmware work | +| Hardware wiring | Power budget / brown-out (Pitfall 8) | Use dedicated 5V supply per strip before first flash | +| Hardware wiring | WS2801 SPI signal integrity (Pitfall 2) | Reduce SPI clock to 1 MHz, add series resistor | +| LED protocol definition | SK6812 as RGBW not RGB (Pitfall 7) | Explicit 4-byte configuration before any animation | +| Communication protocol | JSON versioning (Pitfall 9) | Version field in first command | +| Communication protocol | WiFi jitter (Pitfall 4) | UDP with timestamp payload, disable power saving | +| Choreography engine | Timeline drift (Pitfall 3) | Absolute timestamp scheduling, validate at 120s | +| Audio integration | Textual thread safety (Pitfall 6) | `call_from_thread`, `queue.Queue` boundary design | +| Beat detection | False positives on complex tracks (Pitfall 10) | Moving average + sub-bass focus + user tuning | +| TUI development | SSH rendering throughput (Pitfall 11) | Cap refresh rate at 30 fps | +| Firmware feature expansion | Flash size limit (Pitfall 12) | Check binary size early, custom partition table | + +--- + +## Sources + +- [ESP32 Forum: RMT transaction corrupted by WiFi interrupts](https://esp32.com/viewtopic.php?t=41170) +- [ESP32 Forum: RMT NeoPixels flicker when WiFi is used (3 pages)](https://esp32.com/viewtopic.php?t=3980) +- [FastLED GitHub Issue #1657: Serious flickering on ESP32-C3](https://github.com/FastLED/FastLED/issues/1657) +- [FastLED GitHub Issue #1873: Multiple strip declaration causes ESP32-C3 reboot](https://github.com/FastLED/FastLED/issues/1873) +- [ESPHome GitHub Issue #10335: esp32_rmt_led_strip flicker](https://github.com/esphome/esphome/issues/10335) +- [Adafruit NeoPixel GitHub Issue #139: ESP32 timing error every 1ms](https://github.com/adafruit/Adafruit_NeoPixel/issues/139) +- [ESP-IDF GitHub Issue #15345: Making UDP send latency more predictable](https://github.com/espressif/esp-idf/issues/15345) +- [ESP32 Forum: UDP packet delays and clustering](https://esp32.com/viewtopic.php?t=27665) +- [ESP32 Forum: WS2801 SPI output inconsistent](https://www.esp32.com/viewtopic.php?t=37673) +- [Arduino Forum: WS2801 SPI schemes and speeds](https://forum.arduino.cc/t/ws2801-and-different-spi-schemes-speeds/153756) +- [Textual Workers documentation](https://textual.textualize.io/guide/workers/) +- [Textual: The Heisenbug lurking in your async code](https://textual.textualize.io/blog/2023/02/11/the-heisenbug-lurking-in-your-async-code/) +- [ESPHome ESP32 RMT LED Strip component](https://esphome.io/components/light/esp32_rmt_led_strip/) +- [ESP32-C3 SuperMini specs: 4MB flash, ~1.25MB usable](https://www.espboards.dev/esp32/esp32-c3-super-mini/) +- [DONE.LAND: ESP32-C3 Super Mini overview](https://done.land/components/microcontroller/families/esp/esp32/developmentboards/esp32-c3/c3supermini/) +- [PipeWire low-latency configuration guide](https://oneuptime.com/blog/post/2026-03-02-configure-pipewire-low-latency-audio-ubuntu/view) +- [ESP32 brownout reset causes and prevention](https://www.espboards.dev/troubleshooting/issues/power/esp32-brownout-reset/) +- [SK6812 RGBW flickering with RMT driver](https://esp32.com/viewtopic.php?t=5066) diff --git a/.planning/research/STACK.md b/.planning/research/STACK.md new file mode 100644 index 0000000..c436676 --- /dev/null +++ b/.planning/research/STACK.md @@ -0,0 +1,212 @@ +# Technology Stack + +**Project:** LED Sync Studio +**Researched:** 2026-04-03 +**Confidence:** MEDIUM (hardware-specific constraints limit definitive verification for some choices) + +--- + +## Recommended Stack + +### TUI Layer (Raspberry Pi Host App) + +| Technology | Version | Purpose | Why | +|------------|---------|---------|-----| +| Python | 3.11+ | Runtime | Ships on Raspberry Pi OS bookworm; 3.11 asyncio improvements matter for this use case | +| Textual | 8.2.1 | TUI framework | Latest stable (March 2026). CSS-based theming with custom color variables maps cleanly to cyberpunk/neon aesthetics. asyncio-native — no threading hacks needed alongside audio callbacks. Runs cleanly over SSH. | +| Rich | (Textual dependency) | Terminal rendering | Pulled in by Textual; use directly for log panels and progress bars | + +**Why Textual and not alternatives:** +- `curses` / `urwid`: No CSS, no async, painful for custom widgets like a timeline ruler +- `blessed`: Imperative, no widget system, would require building everything from scratch +- `prompt_toolkit`: Good for REPL-style apps, not layout-based dashboards + +### Audio Layer + +| Technology | Version | Purpose | Why | +|------------|---------|---------|-----| +| sounddevice | 0.5.x | Real-time audio capture from PipeWire/PulseAudio | Uses PortAudio under the hood; supports asyncio generators for block-by-block audio callbacks; works with PipeWire's PulseAudio compatibility layer without extra config | +| aubio | 0.4.9 | Beat detection + onset detection | Written in C, Python bindings via NumPy arrays, designed explicitly for causal real-time processing. Configurable hop size controls latency (512 samples at 44.1kHz = ~11ms per frame). | +| numpy | 1.26+ | Audio buffer manipulation, FFT for spectrum display | Required by aubio; use `numpy.fft.rfft` directly for real-time frequency band analysis without librosa overhead | +| miniaudio | 1.61 | MP3/FLAC/WAV file decoding and playback with position tracking | Pure C backend, no SDL dependency (unlike pygame), exposes playback position in samples, supports seeking. `just_playback` wraps it but miniaudio directly gives more control. | + +**Why aubio over librosa for real-time:** +- librosa is offline-first — designed for batch analysis of whole files, not streaming callbacks +- aubio's C core processes audio frames in-place with ~11ms latency at 44.1kHz/512-sample hop +- The `aubio.tempo` object maintains internal beat-tracking state across frames + +**Why sounddevice over pyaudio:** +- pyaudio requires ALSA headers to compile and has no asyncio support +- sounddevice provides `sd.InputStream` with a Python callback that receives NumPy arrays directly +- PipeWire exposes a PulseAudio-compatible socket on Raspberry Pi OS bookworm — sounddevice connects without special config + +**Why miniaudio over pygame for playback:** +- pygame's audio system conflicts with sounddevice when both need ALSA/PulseAudio simultaneously +- miniaudio exposes exact sample position, enabling precise choreography timeline sync +- No SDL dependency = no display server requirement (SSH-only environment) + +### Communication Layer (Pi → ESP32) + +| Technology | Version | Purpose | Why | +|------------|---------|---------|-----| +| asyncio (stdlib) | Python 3.11+ | UDP datagram sending | No external dependency; `loop.create_datagram_endpoint()` gives fire-and-forget UDP with ~1-5ms overhead | +| UDP (not TCP) | — | Transport protocol | TCP Nagle algorithm adds 40-200ms latency on small packets without tuning; UDP is connectionless and sub-5ms RTT on local WiFi. Commands are idempotent (animation start/stop) so lost packets are acceptable. | +| JSON (stdlib `json`) | — | Command serialization | Already specified in requirements; human-readable for debugging; ArduinoJson v7 on ESP side handles deserialization efficiently | + +**Protocol design note:** Keep payload small. `{"cmd":"start","anim":"pulse","zone":0,"color":"#FF00FF","speed":1.5}` fits in a single UDP datagram (MTU safe under 512 bytes). No fragmentation, no connection overhead. + +**Why not WebSocket/HTTP:** +- HTTP adds 100-300ms per request (connection setup + headers) +- WebSocket persistent connection is viable but adds complexity on ESP side and has known ESP32-C3 crash issues with me-no-dev library +- UDP fire-and-forget matches the use case: beat hits → send command → ESP acts immediately + +### ESP32-C3 Firmware + +| Technology | Version | Purpose | Why | +|------------|---------|---------|-----| +| PlatformIO | latest | Build system | Far superior to Arduino IDE for this project: proper dependency management, `platformio.ini` reproducible builds, VS Code integration, OTA flash support | +| Arduino framework for ESP32 | 3.x (espressif32) | Hardware abstraction | Mature, large library ecosystem, good ESP32-C3 support in core 3.x | +| NeoPixelBus by Makuna | 2.8.x | SK6812 single-wire control via RMT | Uses ESP32-C3 RMT peripheral (channels 0-1 available), hardware-timed — no CPU spin waiting. Handles RGBW SK6812 correctly including white channel. | +| Adafruit DotStar / raw SPI | — | WS2801 SPI control | WS2801 is a clock+data SPI protocol, not single-wire. Use Arduino `SPI.h` directly or Adafruit DotStar library which supports WS2801 via hardware SPI. NeoPixelBus also supports SPI method for WS2801. | +| ArduinoJson | 7.4.x | JSON deserialization of commands | Version 7 uses heap-allocated elastic documents — no need to pre-size buffers. Benchmarked at <1ms for typical command payloads on ESP32. | +| AsyncUDP (ESP32 Arduino stdlib) | — | UDP packet reception | Built into arduino-esp32 core, no external dependency, runs in background task. More stable on C3 than ESPAsyncWebServer HTTP. | + +**LED library strategy — two strips, two protocols:** + +The ESP32-C3 SuperMini has 2 RMT channels and hardware SPI. Use them independently: +- **SK6812 (Wand, 300 LEDs):** NeoPixelBus with `NeoEsp32RmtMethodBase` on GPIO4 (RMT channel 0) +- **WS2801 (Schrank, 160 LEDs):** Hardware SPI via `SPI.h` on GPIO6 (MOSI) + GPIO4 (SCK) — but check GPIO conflicts with SK6812 RMT pin assignment. Use GPIO pin mapping carefully: avoid GPIO2, GPIO8, GPIO9 (boot-critical), GPIO12-17 (internal flash). + +**Concrete pin assignment to validate on hardware:** +- SK6812 data: GPIO3 (RMT channel 0) +- WS2801 MOSI: GPIO7, WS2801 CLK: GPIO6 (hardware SPI bus 1) +- Leave GPIO8 (onboard LED/WS2812) unused to avoid conflicts + +**Why not WLED:** +- WLED is a complete opinionated firmware — cannot receive custom JSON animation commands +- Animation logic must run on ESP32 with own state machine; WLED's architecture doesn't expose low-level animation API +- Building custom firmware is the right call here + +**Why not me-no-dev ESPAsyncWebServer:** +- Archived January 2025, read-only +- Known crash bugs on ESP32-C3 single-core with WebSocket connections +- Successor: ESP32Async/ESPAsyncWebServer — but unnecessary complexity for this project. Plain AsyncUDP is simpler and lower latency. + +### Data Storage (Choreography Files) + +| Technology | Version | Purpose | Why | +|------------|---------|---------|-----| +| JSON (stdlib) | — | Choreography file format | Human-readable, easily version-controlled, Python `json` module needs zero deps. YAML adds a dependency and parsing overhead with no benefit for machine-generated files. | +| Pydantic | 2.x | Schema validation for choreography files | V2 is Rust-backed, fast. Validates animation parameter ranges, timeline consistency. Generates JSON schemas for free. Worth the dependency for data integrity. | + +### Development Tooling + +| Technology | Purpose | +|------------|---------| +| uv | Python package manager — significantly faster than pip, proper lockfiles, works on Raspberry Pi ARM | +| PlatformIO | ESP32 firmware build, dependency resolution, OTA upload | +| pytest + pytest-asyncio | Test suite for choreography engine and protocol logic | + +--- + +## Alternatives Considered + +| Category | Recommended | Alternative | Why Not | +|----------|-------------|-------------|---------| +| TUI | Textual 8.2.1 | curses, urwid, blessed | No CSS theming, no asyncio integration, verbose widget construction | +| Beat detection | aubio | librosa | librosa is offline/batch, not real-time streaming; adds heavy scipy/sklearn deps | +| Beat detection | aubio | essentia | Essentia is ML-heavy, not pip-installable cleanly on Raspberry Pi ARM without build hassles | +| Audio capture | sounddevice | pyaudio | pyaudio needs ALSA headers, no asyncio, harder to pip install | +| Audio playback | miniaudio | pygame | pygame conflicts with sounddevice for audio device access; no playback position API | +| Audio playback | miniaudio | gstreamer | gstreamer Python bindings (gi.repository) are fragile, heavyweight | +| Transport | UDP | HTTP/REST | HTTP adds 100-300ms latency per command, unacceptable for beat-sync | +| Transport | UDP | WebSocket | ESP32-C3 WebSocket stability issues; more complex firmware; no benefit for one-directional command stream | +| LED (SK6812) | NeoPixelBus (RMT) | Adafruit NeoPixel | NeoPixelBus RMT is hardware-timed and interrupt-free; Adafruit NeoPixel disables interrupts during transfer, causing WiFi packet drops | +| Firmware build | PlatformIO | Arduino IDE | Arduino IDE has no proper dependency management, no CLI, no lockfiles | +| Choreography format | JSON | YAML | Zero extra dependency for JSON; no user-facing config editing needed | +| Config validation | Pydantic v2 | dataclasses | Pydantic gives JSON schema, field validators, and serialization; dataclasses don't | + +--- + +## Installation + +### Raspberry Pi Host App + +```bash +# Install uv (fast Python package manager) +curl -LsSf https://astral.sh/uv/install.sh | sh + +# Create project environment +uv init led-sync-studio +cd led-sync-studio +uv add textual==8.2.1 +uv add sounddevice miniaudio aubio numpy pydantic + +# System dependency for sounddevice (PortAudio) +sudo apt install -y libportaudio2 +``` + +### ESP32-C3 Firmware (PlatformIO) + +```ini +; platformio.ini +[env:esp32c3_supermini] +platform = espressif32 +board = esp32-c3-devkitm-1 +board_build.mcu = esp32c3 +framework = arduino +build_flags = + -DARDUINO_USB_CDC_ON_BOOT=1 + -DARDUINO_USB_MODE=1 +monitor_speed = 115200 + +lib_deps = + makuna/NeoPixelBus @ ^2.8.0 + bblanchon/ArduinoJson @ ^7.4.0 +``` + +WS2801 via hardware SPI uses only Arduino core `SPI.h` — no external library needed. + +--- + +## Confidence Assessment + +| Component | Confidence | Notes | +|-----------|------------|-------| +| Textual 8.2.1 | HIGH | Verified via GitHub releases page | +| aubio for real-time | HIGH | Official docs confirm causal/streaming design | +| sounddevice + PipeWire | MEDIUM | PipeWire PulseAudio compat layer works in practice; not officially documented for sounddevice | +| miniaudio for playback | MEDIUM | Position tracking confirmed in docs; Raspberry Pi ARM support confirmed via PyPI | +| UDP over TCP | HIGH | Latency characteristics well-documented; ESP32 TCP Nagle issue confirmed in arduino-esp32 issues | +| NeoPixelBus RMT for SK6812 | HIGH | RMT method confirmed for ESP32-C3; RGBW SK6812 support documented | +| WS2801 via SPI | MEDIUM | SPI method works in NeoPixelBus or raw SPI; GPIO pin assignments need hardware verification | +| ArduinoJson 7.4.x | HIGH | Latest version confirmed via ESP Component Registry | +| ESP32-C3 GPIO constraints | MEDIUM | SuperMini pinout documented; specific GPIO conflicts need hardware validation | +| PlatformIO build | HIGH | Well-documented, community-confirmed for ESP32-C3 | + +--- + +## Critical Constraints to Validate Early + +1. **GPIO conflict: SK6812 RMT + WS2801 SPI on ESP32-C3 SuperMini** — The SuperMini has limited broken-out pins. Verify that chosen RMT GPIO and SPI GPIO for simultaneous operation don't conflict. NeoPixelBus RMT does not block SPI, so hardware parallelism is possible. + +2. **sounddevice + miniaudio simultaneous device access** — Both will request the default PulseAudio/PipeWire sink. On Raspberry Pi OS bookworm (PipeWire default), multiple readers on a loopback monitor source should work, but test early. + +3. **aubio version on Raspberry Pi ARM** — aubio 0.4.9 is the last stable release (2019). The pip wheel for ARM may not exist; may need to compile from source. Test `pip install aubio` on target Pi before committing. + +4. **WiFi latency floor** — UDP on local WiFi to ESP32 has measured RTT of 20-60ms depending on router/environment. Commands are fire-and-forget so one-way latency is 10-30ms. Musical sync at 120BPM has 500ms between beats — 30ms latency is well within tolerance for beat-level sync, but not sub-beat. + +--- + +## Sources + +- Textual releases: https://github.com/Textualize/textual/releases +- aubio real-time design: https://aubio.org/doc/ — https://github.com/zak-45/WLEDAudioSyncRTBeat +- sounddevice docs: https://python-sounddevice.readthedocs.io/ +- miniaudio Python: https://github.com/irmen/pyminiaudio +- NeoPixelBus ESP32-C3 RMT: https://github.com/Makuna/NeoPixelBus/issues/477 +- ArduinoJson 7.4.x: https://components.espressif.com/components/bblanchon/arduinojson/versions/7.4.2 +- ESPAsyncWebServer archived + successor: https://github.com/mathieucarbou/ESPAsyncWebServer +- ESP32-C3 SuperMini pinout: https://www.espboards.dev/esp32/esp32-c3-super-mini/ +- PlatformIO ESP32-C3: https://docs.platformio.org/en/latest/boards/espressif32/esp32-c3-devkitm-1.html +- UDP latency on ESP32: https://github.com/espressif/arduino-esp32/issues/1283 diff --git a/.planning/research/SUMMARY.md b/.planning/research/SUMMARY.md new file mode 100644 index 0000000..e289344 --- /dev/null +++ b/.planning/research/SUMMARY.md @@ -0,0 +1,227 @@ +# Project Research Summary + +**Project:** LED Sync Studio +**Domain:** Terminal-based LED choreography / music-synced light show controller (Raspberry Pi + ESP32) +**Researched:** 2026-04-03 +**Confidence:** MEDIUM-HIGH + +## Executive Summary + +LED Sync Studio is a music-synchronized LED choreography system built around two physical components: a Raspberry Pi running a cyberpunk-themed TUI application (over SSH), and an ESP32-C3 SuperMini executing animation commands on two LED strips (300x SK6812 RGBW and 160x WS2801 RGB). The defining architectural constraint is that the Pi never streams raw pixel frames — it sends only named animation commands with parameters over UDP, and the ESP32 runs animations autonomously. This command-based approach sidesteps the WiFi jitter problem that makes frame-streaming systems unreliable for real-time music sync. + +The recommended stack is: Python 3.11 + Textual 8.2.1 for the TUI (asyncio-native, CSS-themeable, SSH-compatible); sounddevice + aubio for audio capture and beat detection; miniaudio for song playback with exact position tracking; JSON-over-UDP as the Pi-to-ESP32 protocol; and PlatformIO + NeoPixelBus (RMT) + ArduinoJson on the ESP32 side. This combination was selected to avoid known failure modes: pygame conflicts with sounddevice, librosa is batch-only not real-time, WebSocket has documented crash bugs on single-core ESP32-C3, and ESPAsyncWebServer was archived in January 2025. + +The highest-risk area is the ESP32-C3's single RISC-V core — standard dual-core WiFi/LED isolation advice does not apply, and WiFi interrupts are documented to corrupt RMT timing for SK6812 strips without careful DMA buffer sizing and power-save disabling. Power delivery is a close second: 300 SK6812 + 160 WS2801 LEDs peak at ~27A at 5V and require dedicated supplies per strip with a firmware-enforced brightness cap. Both issues must be validated in hardware before any application-layer development begins. + +--- + +## Key Findings + +### Recommended Stack + +The Pi-side application is pure Python with minimal dependencies: Textual handles the full TUI without threading complexity (asyncio-native), sounddevice captures system audio via PipeWire's PulseAudio compatibility layer, aubio processes audio frames in C with ~11ms latency at 512-sample hop size, and miniaudio decodes and plays song files with sample-accurate position tracking (avoiding the pygame/sounddevice device conflict). Pydantic v2 validates choreography JSON files with zero-cost schema generation. + +On the ESP32, PlatformIO with the Arduino framework provides reproducible builds. NeoPixelBus uses the RMT peripheral (hardware-timed, DMA-driven) for SK6812 RGBW output. WS2801 uses hardware SPI directly from Arduino's `SPI.h`. ArduinoJson v7 parses command payloads in under 1ms. AsyncUDP (built into arduino-esp32 core) receives commands without the stability issues of the archived ESPAsyncWebServer. + +**Core technologies:** +- Python 3.11 + Textual 8.2.1: TUI runtime — asyncio-native, CSS-themeable, runs cleanly over SSH +- sounddevice 0.5.x: System audio capture — PipeWire-compatible, asyncio callback support, no ALSA header build issues +- aubio 0.4.9: Beat/onset detection — C-backed, streaming/causal design, 11ms per-frame latency +- miniaudio: Song playback — sample-accurate position, no SDL, no device conflict with sounddevice +- UDP + stdlib json: Pi-to-ESP32 transport — fire-and-forget, 1-5ms local WiFi latency +- NeoPixelBus 2.8.x (RMT): SK6812 output — hardware-timed via DMA, interrupt-safe +- ArduinoJson 7.4.x: ESP32 JSON parsing — heap-elastic documents, sub-1ms for command payloads +- PlatformIO: ESP32 build system — dependency lockfiles, CLI, OTA support +- Pydantic v2: Choreography file validation — Rust-backed, auto JSON schema + +### Expected Features + +The feature landscape is informed by xLights (timeline sequencer), WLED (ESP32 preset/segment model), LedFx (live reactive audio engine), and Jinx! (beat detection baseline). Choreography-mode features are the higher-value deliverable; live reactive mode is secondary but expected by users familiar with WLED/LedFx. + +**Must have (table stakes):** +- Audio waveform display on timeline — blind timing without it; every sequencer has this +- Play / pause / seek transport — fundamental, must work with live LED preview +- Per-zone independent control (WS2801 + SK6812 as named zones) +- Block-based timeline editor — drag/place/resize animation blocks on a horizontal track +- Animation library (8-12 built-in effects: chase, pulse, rainbow, strobe, color wash) +- Per-animation parameter configuration (speed, color, intensity per block) +- Save / load choreography files (JSON) +- Live reactive mode (system audio capture + beat detection → animation trigger) +- Playback position cursor (playhead) +- Loop / repeat per block +- Manual timing mark placement (tapper: press key on beat to stamp marks) + +**Should have (competitive differentiators):** +- Cyberpunk TUI aesthetics over SSH — no equivalent exists in LED sequencer space +- Dual-mode in one tool (timeline + live reactive without restart) +- Zone-aware beat-reactive routing (bass → Zone A, treble → Zone B) +- Auto-beat detection timing marks (aubio stamps marks automatically, user refines) +- Undo / redo (painful without it; all quality DAWs have it) +- Per-block transition effects (crossfade, cut, dissolve) +- Animation preview in-terminal (ASCII block-character LED simulation) + +**Defer (v2+):** +- Spotify integration — complex auth, unclear system-level audio loopback is simpler +- xLights .xsq import/export — complex format, unclear demand +- Scripted effect DSL (Jinx!Script-style) — advanced power-user feature + +### Architecture Approach + +The system divides cleanly into Pi (all intelligence) and ESP32 (dumb executor). The Pi runs an asyncio event loop anchored by Textual; audio processing runs in a thread with a `call_soon_threadsafe` bridge into the async loop; the playback clock is an asyncio task that fires choreography events using absolute `time.monotonic()` timestamps (not relative sleep chains); the UDP transport is a non-blocking asyncio DatagramProtocol. The ESP32 runs three FreeRTOS tasks: high-priority LED output, medium-priority WiFi/JSON receive, and low-priority animation tick at 50fps posting frame buffers to a queue. + +**Major components:** +1. TUI Layer (Textual) — renders all views, accepts keystrokes, subscribes to reactive state from App Core +2. App Core — asyncio state machine: playback clock, choreography scheduler, event dispatch +3. Audio Layer — song playback (miniaudio) + beat detection (sounddevice + aubio), thread-isolated +4. Transport — async UDP socket, JSON serialization, heartbeat, connection health +5. Choreography Engine — load/save JSON files, event sequence management +6. ESP32 Animation Engine — effect registry, parameter system, zone routing, FreeRTOS task structure +7. ESP32 LED Drivers — NeoPixelBus RMT for SK6812, hardware SPI for WS2801 + +### Critical Pitfalls + +1. **WiFi interrupts corrupt SK6812 RMT timing on single-core ESP32-C3** — Enable DMA with full-frame buffer size, disable WiFi power saving (`WIFI_PS_NONE`), test WiFi+LED simultaneously on day 1 of firmware work. This is actively documented in FastLED #1657 (February 2026). + +2. **Choreography timeline drift over long songs** — Never use relative sleep chains. Record `start_wall_time = time.monotonic()` at play, compute `fire_at = start_wall_time + event.timestamp`, recalculate remaining sleep each iteration. Validate drift stays under 20ms/minute at 60s and 120s. + +3. **SK6812 is RGBW (4 bytes/LED), not RGB** — Explicit 4-byte configuration in firmware before any animation code. Include white channel in JSON protocol from day one; adding it later requires a protocol version bump. + +4. **Power budget: 300 SK6812 + 160 WS2801 = ~27A peak at 5V** — Dedicated 5V supply per strip (not from ESP32 VIN/USB), firmware-enforced brightness cap at 40%, 1000µF capacitor at each strip input. Brown-out resets mid-performance otherwise. + +5. **Audio thread safety with Textual** — Audio callbacks are OS-thread-level; never call Textual widget methods from audio thread. Use `self.app.call_from_thread()` or a `queue.Queue` polled by an asyncio task. + +--- + +## Implications for Roadmap + +Based on the dependency graph in FEATURES.md and build order in ARCHITECTURE.md, six phases are recommended. Hardware must be validated first because every other component is validated against a real ESP32. + +### Phase 1: Hardware Foundation + ESP32 Firmware +**Rationale:** The ESP32 is the validation target for all subsequent work. Developing the Pi app against a mock ESP32 risks discovering protocol mismatches only after building everything. Ship firmware first, test with netcat/curl, then build the Pi app around proven commands. +**Delivers:** Working LED hardware: both strips driving correctly with WiFi receiving JSON commands. All hardware risks retired. +**Addresses:** Per-zone control model, animation library (8-12 effects), JSON command protocol +**Avoids:** +- Pitfall 1 (WiFi+RMT coexistence): test simultaneously on day 1 +- Pitfall 2 (WS2801 SPI signal integrity): set clock to 1MHz with series resistors before firmware +- Pitfall 7 (SK6812 RGBW 4-byte framing): configure explicitly before animation code +- Pitfall 8 (power budget): dedicated supplies before first flash +- Pitfall 12 (flash size): check binary size before adding many animation effects +**Research flag:** Needs phase research — ESP32-C3 RMT+WiFi coexistence has conflicting documentation; GPIO pin assignment for simultaneous RMT+SPI needs hardware validation. + +### Phase 2: Transport + App Core (Headless) +**Rationale:** A testable Pi-side core without TUI allows validating the protocol and choreography engine against real hardware before the UI complexity is introduced. CLI-testable with simple scripts. +**Delivers:** UDP transport client, JSON command builder, heartbeat, choreography data model (Pydantic), playback clock with absolute timestamp scheduling, event scheduler. +**Addresses:** Save/load choreography files, per-zone command routing, protocol versioning +**Avoids:** +- Pitfall 3 (timeline drift): implement absolute monotonic timestamp scheduling from the start +- Pitfall 4 (WiFi jitter): UDP with `WIFI_PS_NONE`, small payloads, timestamp in payload +- Pitfall 9 (JSON protocol versioning): `protocol_version` field in first command +**Research flag:** Standard patterns — asyncio UDP client and Pydantic data models are well-documented. + +### Phase 3: Audio Layer +**Rationale:** Beat detection and song playback have no UI dependency. Prototype as standalone scripts, then integrate. This keeps the complex audio threading isolated before TUI is introduced. +**Delivers:** Song playback (MP3/FLAC/WAV via miniaudio with position tracking), system audio capture (sounddevice), beat/onset detection (aubio), thread-safe event bridge to asyncio. +**Addresses:** Audio waveform data source, beat events for live reactive mode, playback position for choreography clock +**Avoids:** +- Pitfall 6 (Textual thread safety): design the `call_from_thread` bridge before TUI exists +- Pitfall 10 (beat detection false positives): moving energy average + sub-bass focus + sensitivity params +**Research flag:** Needs phase research — aubio 0.4.9 ARM pip wheel availability on Raspberry Pi OS bookworm is unverified; sounddevice + miniaudio simultaneous PipeWire device access needs early testing. + +### Phase 4: TUI Layer +**Rationale:** App Core and Audio Layer must have stable APIs before the TUI is built on top of them. The timeline view is the most complex widget — build transport controls and event list first. +**Delivers:** Full Textual application: Header, TransportBar, TimelineView (TimeRuler + ZoneTrack x2), EventListView, AnimationPanel sidebar, StatusBar. Cyberpunk visual theme. +**Addresses:** Audio waveform display, block-based timeline editor, playback cursor, animation parameter configuration panel, manual timing mark (tapper) +**Avoids:** +- Pitfall 6 (Textual thread safety): use established audio bridge from Phase 3 +- Pitfall 11 (SSH rendering throughput): set `REFRESH_RATE = 30` from the start; do not default to Textual maximum +- Anti-pattern: polling playback position from TUI (App Core owns authoritative state) +**Research flag:** Standard patterns — Textual docs are comprehensive for reactive widgets and worker threads. + +### Phase 5: Live Reactive Mode +**Rationale:** Depends on both the Audio Layer (beat events) and the TUI (mode switching). Both must be complete first. This is the "second major mode" that users familiar with WLED/LedFx expect. +**Delivers:** System audio capture → beat detection → animation dispatch pipeline. Zone-aware routing (bass → Zone A, treble → Zone B). Mode toggle in TUI without restart. Calibration UI for sensitivity. +**Addresses:** Live reactive mode, zone-aware beat-reactive routing +**Avoids:** +- Pitfall 10 (beat detection false positives): expose sensitivity and frequency band in calibration UI +- Pitfall 4 (WiFi jitter): design animations with ±20ms tolerance (quick fades mask arrival variance) +**Research flag:** Standard patterns — LedFx architecture and WLED audio-reactive docs provide clear precedent. + +### Phase 6: Polish + Editor Completeness +**Rationale:** Quality-of-life features that make the tool pleasant to use long-term, not required for core function. +**Delivers:** Undo/redo, loop/repeat per block, per-block transition effects, auto-beat detection timing marks (aubio stamps on analysis, user refines), choreography file browser. +**Addresses:** Undo/redo, loop/repeat, transitions, auto-beat marks +**Research flag:** Standard patterns — undo/redo command stack is well-documented; aubio batch analysis for timing mark generation uses same library already integrated. + +### Phase Ordering Rationale + +- Hardware first because every other component validates against real ESP32 behavior, not a mock +- Transport + App Core second because the CLI-testable layer validates the protocol before UI complexity +- Audio Layer third because it has no UI dependency and threading design must precede TUI integration +- TUI fourth because it depends on stable App Core and Audio Layer APIs +- Live Reactive fifth because it requires both Audio Layer and TUI to be complete +- Polish last because it adds no new system capability, only ergonomics + +The groupings follow the architectural component boundaries in ARCHITECTURE.md: firmware → headless Pi core → audio → UI → integration → refinement. This order ensures each phase has a working validation target from the phase before it. + +### Research Flags + +Phases needing deeper research during planning: +- **Phase 1 (ESP32 Firmware):** ESP32-C3 RMT+WiFi coexistence has conflicting community reports; GPIO pin mapping for simultaneous RMT (SK6812) + SPI (WS2801) on the specific SuperMini board needs hardware validation. Firmware size budget vs. animation library scope needs early sizing. +- **Phase 3 (Audio Layer):** aubio 0.4.9 ARM pip wheel on Raspberry Pi OS bookworm — may need to build from source. Simultaneous sounddevice + miniaudio access to PipeWire — needs early testing before building the full layer. + +Phases with standard patterns (skip research-phase): +- **Phase 2 (Transport + App Core):** asyncio UDP, Pydantic data models, JSON choreography format — all well-documented with abundant examples. +- **Phase 4 (TUI Layer):** Textual has comprehensive official documentation; reactive widget patterns and worker thread safety are explicitly covered. +- **Phase 5 (Live Reactive):** LedFx architecture and WLED audio-reactive docs provide clear, tested precedent for the audio-to-LED pipeline. +- **Phase 6 (Polish):** Standard patterns for undo/redo (command stack) and loop control. + +--- + +## Confidence Assessment + +| Area | Confidence | Notes | +|------|------------|-------| +| Stack | MEDIUM-HIGH | Core choices verified against official docs and GitHub releases. Two gaps: aubio ARM pip wheel availability, sounddevice + miniaudio PipeWire simultaneous access. | +| Features | MEDIUM | Informed by xLights/WLED/LedFx analysis; this is a novel TUI product so user expectations are partially inferred. Table stakes list is solid; differentiator value is harder to confirm without users. | +| Architecture | HIGH | Component boundaries and data flow patterns are well-established. Event-driven async core with thread-isolated audio is the correct pattern; WLED validates UDP command architecture at scale. | +| Pitfalls | HIGH | Most pitfalls are actively documented in ESP32 issue trackers with recent activity (February 2026). The RMT+WiFi coexistence issue is particularly well-evidenced across multiple library ecosystems. | + +**Overall confidence:** MEDIUM-HIGH + +### Gaps to Address + +- **aubio ARM wheel:** Test `pip install aubio` on actual Raspberry Pi hardware before Phase 3 begins. If it fails, build from source (documented procedure exists, adds ~30min to setup). +- **Simultaneous PipeWire device access:** sounddevice (capture) + miniaudio (playback) both accessing PipeWire's default sink. Test early in Phase 3 — multiple readers on a PipeWire loopback monitor source should work but is not officially documented for this combination. +- **GPIO pin assignment:** ESP32-C3 SuperMini has limited broken-out pins. Exact RMT GPIO + SPI GPIO combination for simultaneous operation needs hands-on hardware validation in Phase 1. +- **WiFi latency floor:** Measured RTT on actual home network environment may exceed the 10-30ms estimate. If 95th-percentile one-way latency exceeds 50ms, the timestamp-in-payload approach (Phase 2) becomes mandatory rather than optional. +- **Animation library scope vs. flash budget:** ESP32-C3 SuperMini has ~1.25MB usable firmware space. Scope the animation library (targeted 8-12 effects) against actual compiled firmware size early in Phase 1. + +--- + +## Sources + +### Primary (HIGH confidence) +- https://github.com/Textualize/textual/releases — Textual 8.2.1 confirmed latest stable +- https://aubio.org/doc/ — aubio causal/streaming design confirmed +- https://components.espressif.com/components/bblanchon/arduinojson/versions/7.4.2 — ArduinoJson 7.4.x confirmed +- https://docs.platformio.org/en/latest/boards/espressif32/esp32-c3-devkitm-1.html — PlatformIO ESP32-C3 support +- https://kno.wled.ge/ — WLED protocol architecture (industry reference for ESP32 LED control) +- https://textual.textualize.io/guide/workers/ — Textual thread safety patterns +- https://www.espressif.com/sites/default/files/documentation/esp32-c3_technical_reference_manual_en.pdf — ESP32-C3 hardware specs + +### Secondary (MEDIUM confidence) +- https://github.com/FastLED/FastLED/issues/1657 — ESP32-C3 RMT flicker with WiFi (February 2026) +- https://esp32.com/viewtopic.php?t=41170 — RMT transaction corrupted by WiFi interrupts +- https://python-sounddevice.readthedocs.io/ — sounddevice docs (PipeWire PulseAudio compat layer not explicitly documented) +- https://github.com/irmen/pyminiaudio — miniaudio position tracking confirmed +- https://docs.ledfx.app/en/latest/ — LedFx reactive architecture reference +- https://manual.xlights.org/xlights/ — xLights feature baseline +- https://www.espboards.dev/esp32/esp32-c3-super-mini/ — ESP32-C3 SuperMini pinout and flash specs + +### Tertiary (LOW confidence — community sources) +- https://www.oreateai.com/blog/diy-wifi-light-shows-syncing-your-leds-without-breaking-the-bank — WiFi jitter figures (~12ms variance on ESP32) +- https://deepwiki.com/LedFx/LedFx — LedFx architecture (DeepWiki auto-generated, verify critical claims) + +--- +*Research completed: 2026-04-03* +*Ready for roadmap: yes*