docs: research complete
This commit is contained in:
Claude
227
.planning/research/SUMMARY.md
Normal file
227
.planning/research/SUMMARY.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# Project Research Summary
|
||||
|
||||
**Project:** LED Sync Studio
|
||||
**Domain:** Terminal-based LED choreography / music-synced light show controller (Raspberry Pi + ESP32)
|
||||
**Researched:** 2026-04-03
|
||||
**Confidence:** MEDIUM-HIGH
|
||||
|
||||
## Executive Summary
|
||||
|
||||
LED Sync Studio is a music-synchronized LED choreography system built around two physical components: a Raspberry Pi running a cyberpunk-themed TUI application (over SSH), and an ESP32-C3 SuperMini executing animation commands on two LED strips (300x SK6812 RGBW and 160x WS2801 RGB). The defining architectural constraint is that the Pi never streams raw pixel frames — it sends only named animation commands with parameters over UDP, and the ESP32 runs animations autonomously. This command-based approach sidesteps the WiFi jitter problem that makes frame-streaming systems unreliable for real-time music sync.
|
||||
|
||||
The recommended stack is: Python 3.11 + Textual 8.2.1 for the TUI (asyncio-native, CSS-themeable, SSH-compatible); sounddevice + aubio for audio capture and beat detection; miniaudio for song playback with exact position tracking; JSON-over-UDP as the Pi-to-ESP32 protocol; and PlatformIO + NeoPixelBus (RMT) + ArduinoJson on the ESP32 side. This combination was selected to avoid known failure modes: pygame conflicts with sounddevice, librosa is batch-only not real-time, WebSocket has documented crash bugs on single-core ESP32-C3, and ESPAsyncWebServer was archived in January 2025.
|
||||
|
||||
The highest-risk area is the ESP32-C3's single RISC-V core — standard dual-core WiFi/LED isolation advice does not apply, and WiFi interrupts are documented to corrupt RMT timing for SK6812 strips without careful DMA buffer sizing and power-save disabling. Power delivery is a close second: 300 SK6812 + 160 WS2801 LEDs peak at ~27A at 5V and require dedicated supplies per strip with a firmware-enforced brightness cap. Both issues must be validated in hardware before any application-layer development begins.
|
||||
|
||||
---
|
||||
|
||||
## Key Findings
|
||||
|
||||
### Recommended Stack
|
||||
|
||||
The Pi-side application is pure Python with minimal dependencies: Textual handles the full TUI without threading complexity (asyncio-native), sounddevice captures system audio via PipeWire's PulseAudio compatibility layer, aubio processes audio frames in C with ~11ms latency at 512-sample hop size, and miniaudio decodes and plays song files with sample-accurate position tracking (avoiding the pygame/sounddevice device conflict). Pydantic v2 validates choreography JSON files with zero-cost schema generation.
|
||||
|
||||
On the ESP32, PlatformIO with the Arduino framework provides reproducible builds. NeoPixelBus uses the RMT peripheral (hardware-timed, DMA-driven) for SK6812 RGBW output. WS2801 uses hardware SPI directly from Arduino's `SPI.h`. ArduinoJson v7 parses command payloads in under 1ms. AsyncUDP (built into arduino-esp32 core) receives commands without the stability issues of the archived ESPAsyncWebServer.
|
||||
|
||||
**Core technologies:**
|
||||
- Python 3.11 + Textual 8.2.1: TUI runtime — asyncio-native, CSS-themeable, runs cleanly over SSH
|
||||
- sounddevice 0.5.x: System audio capture — PipeWire-compatible, asyncio callback support, no ALSA header build issues
|
||||
- aubio 0.4.9: Beat/onset detection — C-backed, streaming/causal design, 11ms per-frame latency
|
||||
- miniaudio: Song playback — sample-accurate position, no SDL, no device conflict with sounddevice
|
||||
- UDP + stdlib json: Pi-to-ESP32 transport — fire-and-forget, 1-5ms local WiFi latency
|
||||
- NeoPixelBus 2.8.x (RMT): SK6812 output — hardware-timed via DMA, interrupt-safe
|
||||
- ArduinoJson 7.4.x: ESP32 JSON parsing — heap-elastic documents, sub-1ms for command payloads
|
||||
- PlatformIO: ESP32 build system — dependency lockfiles, CLI, OTA support
|
||||
- Pydantic v2: Choreography file validation — Rust-backed, auto JSON schema
|
||||
|
||||
### Expected Features
|
||||
|
||||
The feature landscape is informed by xLights (timeline sequencer), WLED (ESP32 preset/segment model), LedFx (live reactive audio engine), and Jinx! (beat detection baseline). Choreography-mode features are the higher-value deliverable; live reactive mode is secondary but expected by users familiar with WLED/LedFx.
|
||||
|
||||
**Must have (table stakes):**
|
||||
- Audio waveform display on timeline — blind timing without it; every sequencer has this
|
||||
- Play / pause / seek transport — fundamental, must work with live LED preview
|
||||
- Per-zone independent control (WS2801 + SK6812 as named zones)
|
||||
- Block-based timeline editor — drag/place/resize animation blocks on a horizontal track
|
||||
- Animation library (8-12 built-in effects: chase, pulse, rainbow, strobe, color wash)
|
||||
- Per-animation parameter configuration (speed, color, intensity per block)
|
||||
- Save / load choreography files (JSON)
|
||||
- Live reactive mode (system audio capture + beat detection → animation trigger)
|
||||
- Playback position cursor (playhead)
|
||||
- Loop / repeat per block
|
||||
- Manual timing mark placement (tapper: press key on beat to stamp marks)
|
||||
|
||||
**Should have (competitive differentiators):**
|
||||
- Cyberpunk TUI aesthetics over SSH — no equivalent exists in LED sequencer space
|
||||
- Dual-mode in one tool (timeline + live reactive without restart)
|
||||
- Zone-aware beat-reactive routing (bass → Zone A, treble → Zone B)
|
||||
- Auto-beat detection timing marks (aubio stamps marks automatically, user refines)
|
||||
- Undo / redo (painful without it; all quality DAWs have it)
|
||||
- Per-block transition effects (crossfade, cut, dissolve)
|
||||
- Animation preview in-terminal (ASCII block-character LED simulation)
|
||||
|
||||
**Defer (v2+):**
|
||||
- Spotify integration — complex auth, unclear system-level audio loopback is simpler
|
||||
- xLights .xsq import/export — complex format, unclear demand
|
||||
- Scripted effect DSL (Jinx!Script-style) — advanced power-user feature
|
||||
|
||||
### Architecture Approach
|
||||
|
||||
The system divides cleanly into Pi (all intelligence) and ESP32 (dumb executor). The Pi runs an asyncio event loop anchored by Textual; audio processing runs in a thread with a `call_soon_threadsafe` bridge into the async loop; the playback clock is an asyncio task that fires choreography events using absolute `time.monotonic()` timestamps (not relative sleep chains); the UDP transport is a non-blocking asyncio DatagramProtocol. The ESP32 runs three FreeRTOS tasks: high-priority LED output, medium-priority WiFi/JSON receive, and low-priority animation tick at 50fps posting frame buffers to a queue.
|
||||
|
||||
**Major components:**
|
||||
1. TUI Layer (Textual) — renders all views, accepts keystrokes, subscribes to reactive state from App Core
|
||||
2. App Core — asyncio state machine: playback clock, choreography scheduler, event dispatch
|
||||
3. Audio Layer — song playback (miniaudio) + beat detection (sounddevice + aubio), thread-isolated
|
||||
4. Transport — async UDP socket, JSON serialization, heartbeat, connection health
|
||||
5. Choreography Engine — load/save JSON files, event sequence management
|
||||
6. ESP32 Animation Engine — effect registry, parameter system, zone routing, FreeRTOS task structure
|
||||
7. ESP32 LED Drivers — NeoPixelBus RMT for SK6812, hardware SPI for WS2801
|
||||
|
||||
### Critical Pitfalls
|
||||
|
||||
1. **WiFi interrupts corrupt SK6812 RMT timing on single-core ESP32-C3** — Enable DMA with full-frame buffer size, disable WiFi power saving (`WIFI_PS_NONE`), test WiFi+LED simultaneously on day 1 of firmware work. This is actively documented in FastLED #1657 (February 2026).
|
||||
|
||||
2. **Choreography timeline drift over long songs** — Never use relative sleep chains. Record `start_wall_time = time.monotonic()` at play, compute `fire_at = start_wall_time + event.timestamp`, recalculate remaining sleep each iteration. Validate drift stays under 20ms/minute at 60s and 120s.
|
||||
|
||||
3. **SK6812 is RGBW (4 bytes/LED), not RGB** — Explicit 4-byte configuration in firmware before any animation code. Include white channel in JSON protocol from day one; adding it later requires a protocol version bump.
|
||||
|
||||
4. **Power budget: 300 SK6812 + 160 WS2801 = ~27A peak at 5V** — Dedicated 5V supply per strip (not from ESP32 VIN/USB), firmware-enforced brightness cap at 40%, 1000µF capacitor at each strip input. Brown-out resets mid-performance otherwise.
|
||||
|
||||
5. **Audio thread safety with Textual** — Audio callbacks are OS-thread-level; never call Textual widget methods from audio thread. Use `self.app.call_from_thread()` or a `queue.Queue` polled by an asyncio task.
|
||||
|
||||
---
|
||||
|
||||
## Implications for Roadmap
|
||||
|
||||
Based on the dependency graph in FEATURES.md and build order in ARCHITECTURE.md, six phases are recommended. Hardware must be validated first because every other component is validated against a real ESP32.
|
||||
|
||||
### Phase 1: Hardware Foundation + ESP32 Firmware
|
||||
**Rationale:** The ESP32 is the validation target for all subsequent work. Developing the Pi app against a mock ESP32 risks discovering protocol mismatches only after building everything. Ship firmware first, test with netcat/curl, then build the Pi app around proven commands.
|
||||
**Delivers:** Working LED hardware: both strips driving correctly with WiFi receiving JSON commands. All hardware risks retired.
|
||||
**Addresses:** Per-zone control model, animation library (8-12 effects), JSON command protocol
|
||||
**Avoids:**
|
||||
- Pitfall 1 (WiFi+RMT coexistence): test simultaneously on day 1
|
||||
- Pitfall 2 (WS2801 SPI signal integrity): set clock to 1MHz with series resistors before firmware
|
||||
- Pitfall 7 (SK6812 RGBW 4-byte framing): configure explicitly before animation code
|
||||
- Pitfall 8 (power budget): dedicated supplies before first flash
|
||||
- Pitfall 12 (flash size): check binary size before adding many animation effects
|
||||
**Research flag:** Needs phase research — ESP32-C3 RMT+WiFi coexistence has conflicting documentation; GPIO pin assignment for simultaneous RMT+SPI needs hardware validation.
|
||||
|
||||
### Phase 2: Transport + App Core (Headless)
|
||||
**Rationale:** A testable Pi-side core without TUI allows validating the protocol and choreography engine against real hardware before the UI complexity is introduced. CLI-testable with simple scripts.
|
||||
**Delivers:** UDP transport client, JSON command builder, heartbeat, choreography data model (Pydantic), playback clock with absolute timestamp scheduling, event scheduler.
|
||||
**Addresses:** Save/load choreography files, per-zone command routing, protocol versioning
|
||||
**Avoids:**
|
||||
- Pitfall 3 (timeline drift): implement absolute monotonic timestamp scheduling from the start
|
||||
- Pitfall 4 (WiFi jitter): UDP with `WIFI_PS_NONE`, small payloads, timestamp in payload
|
||||
- Pitfall 9 (JSON protocol versioning): `protocol_version` field in first command
|
||||
**Research flag:** Standard patterns — asyncio UDP client and Pydantic data models are well-documented.
|
||||
|
||||
### Phase 3: Audio Layer
|
||||
**Rationale:** Beat detection and song playback have no UI dependency. Prototype as standalone scripts, then integrate. This keeps the complex audio threading isolated before TUI is introduced.
|
||||
**Delivers:** Song playback (MP3/FLAC/WAV via miniaudio with position tracking), system audio capture (sounddevice), beat/onset detection (aubio), thread-safe event bridge to asyncio.
|
||||
**Addresses:** Audio waveform data source, beat events for live reactive mode, playback position for choreography clock
|
||||
**Avoids:**
|
||||
- Pitfall 6 (Textual thread safety): design the `call_from_thread` bridge before TUI exists
|
||||
- Pitfall 10 (beat detection false positives): moving energy average + sub-bass focus + sensitivity params
|
||||
**Research flag:** Needs phase research — aubio 0.4.9 ARM pip wheel availability on Raspberry Pi OS bookworm is unverified; sounddevice + miniaudio simultaneous PipeWire device access needs early testing.
|
||||
|
||||
### Phase 4: TUI Layer
|
||||
**Rationale:** App Core and Audio Layer must have stable APIs before the TUI is built on top of them. The timeline view is the most complex widget — build transport controls and event list first.
|
||||
**Delivers:** Full Textual application: Header, TransportBar, TimelineView (TimeRuler + ZoneTrack x2), EventListView, AnimationPanel sidebar, StatusBar. Cyberpunk visual theme.
|
||||
**Addresses:** Audio waveform display, block-based timeline editor, playback cursor, animation parameter configuration panel, manual timing mark (tapper)
|
||||
**Avoids:**
|
||||
- Pitfall 6 (Textual thread safety): use established audio bridge from Phase 3
|
||||
- Pitfall 11 (SSH rendering throughput): set `REFRESH_RATE = 30` from the start; do not default to Textual maximum
|
||||
- Anti-pattern: polling playback position from TUI (App Core owns authoritative state)
|
||||
**Research flag:** Standard patterns — Textual docs are comprehensive for reactive widgets and worker threads.
|
||||
|
||||
### Phase 5: Live Reactive Mode
|
||||
**Rationale:** Depends on both the Audio Layer (beat events) and the TUI (mode switching). Both must be complete first. This is the "second major mode" that users familiar with WLED/LedFx expect.
|
||||
**Delivers:** System audio capture → beat detection → animation dispatch pipeline. Zone-aware routing (bass → Zone A, treble → Zone B). Mode toggle in TUI without restart. Calibration UI for sensitivity.
|
||||
**Addresses:** Live reactive mode, zone-aware beat-reactive routing
|
||||
**Avoids:**
|
||||
- Pitfall 10 (beat detection false positives): expose sensitivity and frequency band in calibration UI
|
||||
- Pitfall 4 (WiFi jitter): design animations with ±20ms tolerance (quick fades mask arrival variance)
|
||||
**Research flag:** Standard patterns — LedFx architecture and WLED audio-reactive docs provide clear precedent.
|
||||
|
||||
### Phase 6: Polish + Editor Completeness
|
||||
**Rationale:** Quality-of-life features that make the tool pleasant to use long-term, not required for core function.
|
||||
**Delivers:** Undo/redo, loop/repeat per block, per-block transition effects, auto-beat detection timing marks (aubio stamps on analysis, user refines), choreography file browser.
|
||||
**Addresses:** Undo/redo, loop/repeat, transitions, auto-beat marks
|
||||
**Research flag:** Standard patterns — undo/redo command stack is well-documented; aubio batch analysis for timing mark generation uses same library already integrated.
|
||||
|
||||
### Phase Ordering Rationale
|
||||
|
||||
- Hardware first because every other component validates against real ESP32 behavior, not a mock
|
||||
- Transport + App Core second because the CLI-testable layer validates the protocol before UI complexity
|
||||
- Audio Layer third because it has no UI dependency and threading design must precede TUI integration
|
||||
- TUI fourth because it depends on stable App Core and Audio Layer APIs
|
||||
- Live Reactive fifth because it requires both Audio Layer and TUI to be complete
|
||||
- Polish last because it adds no new system capability, only ergonomics
|
||||
|
||||
The groupings follow the architectural component boundaries in ARCHITECTURE.md: firmware → headless Pi core → audio → UI → integration → refinement. This order ensures each phase has a working validation target from the phase before it.
|
||||
|
||||
### Research Flags
|
||||
|
||||
Phases needing deeper research during planning:
|
||||
- **Phase 1 (ESP32 Firmware):** ESP32-C3 RMT+WiFi coexistence has conflicting community reports; GPIO pin mapping for simultaneous RMT (SK6812) + SPI (WS2801) on the specific SuperMini board needs hardware validation. Firmware size budget vs. animation library scope needs early sizing.
|
||||
- **Phase 3 (Audio Layer):** aubio 0.4.9 ARM pip wheel on Raspberry Pi OS bookworm — may need to build from source. Simultaneous sounddevice + miniaudio access to PipeWire — needs early testing before building the full layer.
|
||||
|
||||
Phases with standard patterns (skip research-phase):
|
||||
- **Phase 2 (Transport + App Core):** asyncio UDP, Pydantic data models, JSON choreography format — all well-documented with abundant examples.
|
||||
- **Phase 4 (TUI Layer):** Textual has comprehensive official documentation; reactive widget patterns and worker thread safety are explicitly covered.
|
||||
- **Phase 5 (Live Reactive):** LedFx architecture and WLED audio-reactive docs provide clear, tested precedent for the audio-to-LED pipeline.
|
||||
- **Phase 6 (Polish):** Standard patterns for undo/redo (command stack) and loop control.
|
||||
|
||||
---
|
||||
|
||||
## Confidence Assessment
|
||||
|
||||
| Area | Confidence | Notes |
|
||||
|------|------------|-------|
|
||||
| Stack | MEDIUM-HIGH | Core choices verified against official docs and GitHub releases. Two gaps: aubio ARM pip wheel availability, sounddevice + miniaudio PipeWire simultaneous access. |
|
||||
| Features | MEDIUM | Informed by xLights/WLED/LedFx analysis; this is a novel TUI product so user expectations are partially inferred. Table stakes list is solid; differentiator value is harder to confirm without users. |
|
||||
| Architecture | HIGH | Component boundaries and data flow patterns are well-established. Event-driven async core with thread-isolated audio is the correct pattern; WLED validates UDP command architecture at scale. |
|
||||
| Pitfalls | HIGH | Most pitfalls are actively documented in ESP32 issue trackers with recent activity (February 2026). The RMT+WiFi coexistence issue is particularly well-evidenced across multiple library ecosystems. |
|
||||
|
||||
**Overall confidence:** MEDIUM-HIGH
|
||||
|
||||
### Gaps to Address
|
||||
|
||||
- **aubio ARM wheel:** Test `pip install aubio` on actual Raspberry Pi hardware before Phase 3 begins. If it fails, build from source (documented procedure exists, adds ~30min to setup).
|
||||
- **Simultaneous PipeWire device access:** sounddevice (capture) + miniaudio (playback) both accessing PipeWire's default sink. Test early in Phase 3 — multiple readers on a PipeWire loopback monitor source should work but is not officially documented for this combination.
|
||||
- **GPIO pin assignment:** ESP32-C3 SuperMini has limited broken-out pins. Exact RMT GPIO + SPI GPIO combination for simultaneous operation needs hands-on hardware validation in Phase 1.
|
||||
- **WiFi latency floor:** Measured RTT on actual home network environment may exceed the 10-30ms estimate. If 95th-percentile one-way latency exceeds 50ms, the timestamp-in-payload approach (Phase 2) becomes mandatory rather than optional.
|
||||
- **Animation library scope vs. flash budget:** ESP32-C3 SuperMini has ~1.25MB usable firmware space. Scope the animation library (targeted 8-12 effects) against actual compiled firmware size early in Phase 1.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- https://github.com/Textualize/textual/releases — Textual 8.2.1 confirmed latest stable
|
||||
- https://aubio.org/doc/ — aubio causal/streaming design confirmed
|
||||
- https://components.espressif.com/components/bblanchon/arduinojson/versions/7.4.2 — ArduinoJson 7.4.x confirmed
|
||||
- https://docs.platformio.org/en/latest/boards/espressif32/esp32-c3-devkitm-1.html — PlatformIO ESP32-C3 support
|
||||
- https://kno.wled.ge/ — WLED protocol architecture (industry reference for ESP32 LED control)
|
||||
- https://textual.textualize.io/guide/workers/ — Textual thread safety patterns
|
||||
- https://www.espressif.com/sites/default/files/documentation/esp32-c3_technical_reference_manual_en.pdf — ESP32-C3 hardware specs
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- https://github.com/FastLED/FastLED/issues/1657 — ESP32-C3 RMT flicker with WiFi (February 2026)
|
||||
- https://esp32.com/viewtopic.php?t=41170 — RMT transaction corrupted by WiFi interrupts
|
||||
- https://python-sounddevice.readthedocs.io/ — sounddevice docs (PipeWire PulseAudio compat layer not explicitly documented)
|
||||
- https://github.com/irmen/pyminiaudio — miniaudio position tracking confirmed
|
||||
- https://docs.ledfx.app/en/latest/ — LedFx reactive architecture reference
|
||||
- https://manual.xlights.org/xlights/ — xLights feature baseline
|
||||
- https://www.espboards.dev/esp32/esp32-c3-super-mini/ — ESP32-C3 SuperMini pinout and flash specs
|
||||
|
||||
### Tertiary (LOW confidence — community sources)
|
||||
- https://www.oreateai.com/blog/diy-wifi-light-shows-syncing-your-leds-without-breaking-the-bank — WiFi jitter figures (~12ms variance on ESP32)
|
||||
- https://deepwiki.com/LedFx/LedFx — LedFx architecture (DeepWiki auto-generated, verify critical claims)
|
||||
|
||||
---
|
||||
*Research completed: 2026-04-03*
|
||||
*Ready for roadmap: yes*
|
||||
Reference in New Issue
Block a user