## Project **LED Sync Studio** Eine Cyberpunk-Terminal-App (Python/Textual) zur Steuerung von LED-Streifen, die auf einem Raspberry Pi 4B laeuft und per SSH bedient wird. Die App ermoeglicht sowohl Song-Choreografie (Animationen manuell auf Zeitpunkte im Song mappen) als auch Live-Reaktiv-Modus (Beat-Detection aus System-Audio). Die Animationen laufen lokal auf einem ESP32-C3, der per JSON-Kommandos ueber WiFi gesteuert wird. **Core Value:** Songs mit LED-Animationen choreografieren und abspielen — der Nutzer baut Timeline-basierte Lichtshows zu seiner Musik. ### Constraints - **Platform**: Raspberry Pi 4B (ARM, Linux) — App muss performant auf Pi laufen - **Terminal**: Textual TUI Framework (Python) — muss ueber SSH funktionieren - **ESP**: ESP32-C3 SuperMini — begrenzter Speicher, ein Core, WiFi only - **LED-Protokoll**: WS2801 (SPI/Clock+Data) und SK6812 (single-wire like NeoPixel) — unterschiedliche Ansteuerung - **Latenz**: JSON-Kommandos muessen schnell genug sein fuer musikalische Synchronisation ## Technology Stack ## Recommended Stack ### TUI Layer (Raspberry Pi Host App) | Technology | Version | Purpose | Why | |------------|---------|---------|-----| | Python | 3.11+ | Runtime | Ships on Raspberry Pi OS bookworm; 3.11 asyncio improvements matter for this use case | | Textual | 8.2.1 | TUI framework | Latest stable (March 2026). CSS-based theming with custom color variables maps cleanly to cyberpunk/neon aesthetics. asyncio-native — no threading hacks needed alongside audio callbacks. Runs cleanly over SSH. | | Rich | (Textual dependency) | Terminal rendering | Pulled in by Textual; use directly for log panels and progress bars | - `curses` / `urwid`: No CSS, no async, painful for custom widgets like a timeline ruler - `blessed`: Imperative, no widget system, would require building everything from scratch - `prompt_toolkit`: Good for REPL-style apps, not layout-based dashboards ### Audio Layer | Technology | Version | Purpose | Why | |------------|---------|---------|-----| | sounddevice | 0.5.x | Real-time audio capture from PipeWire/PulseAudio | Uses PortAudio under the hood; supports asyncio generators for block-by-block audio callbacks; works with PipeWire's PulseAudio compatibility layer without extra config | | aubio | 0.4.9 | Beat detection + onset detection | Written in C, Python bindings via NumPy arrays, designed explicitly for causal real-time processing. Configurable hop size controls latency (512 samples at 44.1kHz = ~11ms per frame). | | numpy | 1.26+ | Audio buffer manipulation, FFT for spectrum display | Required by aubio; use `numpy.fft.rfft` directly for real-time frequency band analysis without librosa overhead | | miniaudio | 1.61 | MP3/FLAC/WAV file decoding and playback with position tracking | Pure C backend, no SDL dependency (unlike pygame), exposes playback position in samples, supports seeking. `just_playback` wraps it but miniaudio directly gives more control. | - librosa is offline-first — designed for batch analysis of whole files, not streaming callbacks - aubio's C core processes audio frames in-place with ~11ms latency at 44.1kHz/512-sample hop - The `aubio.tempo` object maintains internal beat-tracking state across frames - pyaudio requires ALSA headers to compile and has no asyncio support - sounddevice provides `sd.InputStream` with a Python callback that receives NumPy arrays directly - PipeWire exposes a PulseAudio-compatible socket on Raspberry Pi OS bookworm — sounddevice connects without special config - pygame's audio system conflicts with sounddevice when both need ALSA/PulseAudio simultaneously - miniaudio exposes exact sample position, enabling precise choreography timeline sync - No SDL dependency = no display server requirement (SSH-only environment) ### Communication Layer (Pi → ESP32) | Technology | Version | Purpose | Why | |------------|---------|---------|-----| | asyncio (stdlib) | Python 3.11+ | UDP datagram sending | No external dependency; `loop.create_datagram_endpoint()` gives fire-and-forget UDP with ~1-5ms overhead | | UDP (not TCP) | — | Transport protocol | TCP Nagle algorithm adds 40-200ms latency on small packets without tuning; UDP is connectionless and sub-5ms RTT on local WiFi. Commands are idempotent (animation start/stop) so lost packets are acceptable. | | JSON (stdlib `json`) | — | Command serialization | Already specified in requirements; human-readable for debugging; ArduinoJson v7 on ESP side handles deserialization efficiently | - HTTP adds 100-300ms per request (connection setup + headers) - WebSocket persistent connection is viable but adds complexity on ESP side and has known ESP32-C3 crash issues with me-no-dev library - UDP fire-and-forget matches the use case: beat hits → send command → ESP acts immediately ### ESP32-C3 Firmware | Technology | Version | Purpose | Why | |------------|---------|---------|-----| | PlatformIO | latest | Build system | Far superior to Arduino IDE for this project: proper dependency management, `platformio.ini` reproducible builds, VS Code integration, OTA flash support | | Arduino framework for ESP32 | 3.x (espressif32) | Hardware abstraction | Mature, large library ecosystem, good ESP32-C3 support in core 3.x | | NeoPixelBus by Makuna | 2.8.x | SK6812 single-wire control via RMT | Uses ESP32-C3 RMT peripheral (channels 0-1 available), hardware-timed — no CPU spin waiting. Handles RGBW SK6812 correctly including white channel. | | Adafruit DotStar / raw SPI | — | WS2801 SPI control | WS2801 is a clock+data SPI protocol, not single-wire. Use Arduino `SPI.h` directly or Adafruit DotStar library which supports WS2801 via hardware SPI. NeoPixelBus also supports SPI method for WS2801. | | ArduinoJson | 7.4.x | JSON deserialization of commands | Version 7 uses heap-allocated elastic documents — no need to pre-size buffers. Benchmarked at <1ms for typical command payloads on ESP32. | | AsyncUDP (ESP32 Arduino stdlib) | — | UDP packet reception | Built into arduino-esp32 core, no external dependency, runs in background task. More stable on C3 than ESPAsyncWebServer HTTP. | - **SK6812 (Wand, 300 LEDs):** NeoPixelBus with `NeoEsp32RmtMethodBase` on GPIO4 (RMT channel 0) - **WS2801 (Schrank, 160 LEDs):** Hardware SPI via `SPI.h` on GPIO6 (MOSI) + GPIO4 (SCK) — but check GPIO conflicts with SK6812 RMT pin assignment. Use GPIO pin mapping carefully: avoid GPIO2, GPIO8, GPIO9 (boot-critical), GPIO12-17 (internal flash). - SK6812 data: GPIO3 (RMT channel 0) - WS2801 MOSI: GPIO7, WS2801 CLK: GPIO6 (hardware SPI bus 1) - Leave GPIO8 (onboard LED/WS2812) unused to avoid conflicts - WLED is a complete opinionated firmware — cannot receive custom JSON animation commands - Animation logic must run on ESP32 with own state machine; WLED's architecture doesn't expose low-level animation API - Building custom firmware is the right call here - Archived January 2025, read-only - Known crash bugs on ESP32-C3 single-core with WebSocket connections - Successor: ESP32Async/ESPAsyncWebServer — but unnecessary complexity for this project. Plain AsyncUDP is simpler and lower latency. ### Data Storage (Choreography Files) | Technology | Version | Purpose | Why | |------------|---------|---------|-----| | JSON (stdlib) | — | Choreography file format | Human-readable, easily version-controlled, Python `json` module needs zero deps. YAML adds a dependency and parsing overhead with no benefit for machine-generated files. | | Pydantic | 2.x | Schema validation for choreography files | V2 is Rust-backed, fast. Validates animation parameter ranges, timeline consistency. Generates JSON schemas for free. Worth the dependency for data integrity. | ### Development Tooling | Technology | Purpose | |------------|---------| | uv | Python package manager — significantly faster than pip, proper lockfiles, works on Raspberry Pi ARM | | PlatformIO | ESP32 firmware build, dependency resolution, OTA upload | | pytest + pytest-asyncio | Test suite for choreography engine and protocol logic | ## Alternatives Considered | Category | Recommended | Alternative | Why Not | |----------|-------------|-------------|---------| | TUI | Textual 8.2.1 | curses, urwid, blessed | No CSS theming, no asyncio integration, verbose widget construction | | Beat detection | aubio | librosa | librosa is offline/batch, not real-time streaming; adds heavy scipy/sklearn deps | | Beat detection | aubio | essentia | Essentia is ML-heavy, not pip-installable cleanly on Raspberry Pi ARM without build hassles | | Audio capture | sounddevice | pyaudio | pyaudio needs ALSA headers, no asyncio, harder to pip install | | Audio playback | miniaudio | pygame | pygame conflicts with sounddevice for audio device access; no playback position API | | Audio playback | miniaudio | gstreamer | gstreamer Python bindings (gi.repository) are fragile, heavyweight | | Transport | UDP | HTTP/REST | HTTP adds 100-300ms latency per command, unacceptable for beat-sync | | Transport | UDP | WebSocket | ESP32-C3 WebSocket stability issues; more complex firmware; no benefit for one-directional command stream | | LED (SK6812) | NeoPixelBus (RMT) | Adafruit NeoPixel | NeoPixelBus RMT is hardware-timed and interrupt-free; Adafruit NeoPixel disables interrupts during transfer, causing WiFi packet drops | | Firmware build | PlatformIO | Arduino IDE | Arduino IDE has no proper dependency management, no CLI, no lockfiles | | Choreography format | JSON | YAML | Zero extra dependency for JSON; no user-facing config editing needed | | Config validation | Pydantic v2 | dataclasses | Pydantic gives JSON schema, field validators, and serialization; dataclasses don't | ## Installation ### Raspberry Pi Host App # Install uv (fast Python package manager) # Create project environment # System dependency for sounddevice (PortAudio) ### ESP32-C3 Firmware (PlatformIO) ## Confidence Assessment | Component | Confidence | Notes | |-----------|------------|-------| | Textual 8.2.1 | HIGH | Verified via GitHub releases page | | aubio for real-time | HIGH | Official docs confirm causal/streaming design | | sounddevice + PipeWire | MEDIUM | PipeWire PulseAudio compat layer works in practice; not officially documented for sounddevice | | miniaudio for playback | MEDIUM | Position tracking confirmed in docs; Raspberry Pi ARM support confirmed via PyPI | | UDP over TCP | HIGH | Latency characteristics well-documented; ESP32 TCP Nagle issue confirmed in arduino-esp32 issues | | NeoPixelBus RMT for SK6812 | HIGH | RMT method confirmed for ESP32-C3; RGBW SK6812 support documented | | WS2801 via SPI | MEDIUM | SPI method works in NeoPixelBus or raw SPI; GPIO pin assignments need hardware verification | | ArduinoJson 7.4.x | HIGH | Latest version confirmed via ESP Component Registry | | ESP32-C3 GPIO constraints | MEDIUM | SuperMini pinout documented; specific GPIO conflicts need hardware validation | | PlatformIO build | HIGH | Well-documented, community-confirmed for ESP32-C3 | ## Critical Constraints to Validate Early ## Sources - Textual releases: https://github.com/Textualize/textual/releases - aubio real-time design: https://aubio.org/doc/ — https://github.com/zak-45/WLEDAudioSyncRTBeat - sounddevice docs: https://python-sounddevice.readthedocs.io/ - miniaudio Python: https://github.com/irmen/pyminiaudio - NeoPixelBus ESP32-C3 RMT: https://github.com/Makuna/NeoPixelBus/issues/477 - ArduinoJson 7.4.x: https://components.espressif.com/components/bblanchon/arduinojson/versions/7.4.2 - ESPAsyncWebServer archived + successor: https://github.com/mathieucarbou/ESPAsyncWebServer - ESP32-C3 SuperMini pinout: https://www.espboards.dev/esp32/esp32-c3-super-mini/ - PlatformIO ESP32-C3: https://docs.platformio.org/en/latest/boards/espressif32/esp32-c3-devkitm-1.html - UDP latency on ESP32: https://github.com/espressif/arduino-esp32/issues/1283 ## Conventions Conventions not yet established. Will populate as patterns emerge during development. ## Architecture Architecture not yet mapped. Follow existing patterns found in the codebase. ## GSD Workflow Enforcement Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync. Use these entry points: - `/gsd:quick` for small fixes, doc updates, and ad-hoc tasks - `/gsd:debug` for investigation and bug fixing - `/gsd:execute-phase` for planned phase work Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it. ## Developer Profile > Profile not yet configured. Run `/gsd:profile-user` to generate your developer profile. > This section is managed by `generate-claude-profile` -- do not edit manually.