396 lines
19 KiB
Markdown
396 lines
19 KiB
Markdown
# Architecture Patterns: LED Sync Studio
|
|
|
|
**Domain:** Terminal-based LED choreography system (Raspberry Pi + ESP32)
|
|
**Researched:** 2026-04-03
|
|
**Confidence:** HIGH for component boundaries and data flow; MEDIUM for exact protocol choices (verified against ESP32 docs and WLED/xLights ecosystem)
|
|
|
|
---
|
|
|
|
## System Overview
|
|
|
|
LED Sync Studio splits cleanly into two physical contexts: the Raspberry Pi runs the user-facing application and all intelligence, while the ESP32-C3 is a dumb executor — it receives animation commands and runs them locally on the LED strips. The Pi never streams raw pixel frames. This is the defining architectural constraint from which everything else follows.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Raspberry Pi 4B │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ TUI Layer │ │ App Core │ │ Audio Layer │ │
|
|
│ │ (Textual) │◄──►│ (Async) │◄──►│ (PipeWire) │ │
|
|
│ └──────────────┘ └──────┬───────┘ └──────────────┘ │
|
|
│ │ │
|
|
│ ┌──────▼───────┐ │
|
|
│ │ Transport │ │
|
|
│ │ (UDP/WiFi) │ │
|
|
│ └──────┬───────┘ │
|
|
└─────────────────────────────┼───────────────────────────────┘
|
|
│ JSON commands
|
|
┌─────────▼─────────┐
|
|
│ ESP32-C3 │
|
|
│ │
|
|
│ ┌─────────────┐ │
|
|
│ │ WiFi Server │ │
|
|
│ └──────┬──────┘ │
|
|
│ │ │
|
|
│ ┌──────▼──────┐ │
|
|
│ │ Anim Engine │ │
|
|
│ └──────┬──────┘ │
|
|
│ │ │
|
|
│ ┌──────▼──────┐ │
|
|
│ │ LED Drivers │ │
|
|
│ │ WS2801(SPI) │ │
|
|
│ │ SK6812(RMT) │ │
|
|
│ └─────────────┘ │
|
|
└───────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Component Boundaries
|
|
|
|
### Pi-Side Components
|
|
|
|
| Component | Responsibility | Communicates With |
|
|
|-----------|---------------|-------------------|
|
|
| TUI Layer | Render all UI views, accept keystrokes, display timeline/event list | App Core (reactive messages) |
|
|
| App Core | State machine, playback clock, choreography scheduler | TUI Layer, Audio Layer, Transport |
|
|
| Choreography Engine | Load/save JSON files, manage event sequences, calculate playback position | App Core, file system |
|
|
| Audio Layer | Song playback (MP3/FLAC/WAV), real-time beat detection from system audio | App Core (beat events) |
|
|
| Transport | JSON serialization, UDP/TCP socket to ESP32, connection health | App Core |
|
|
|
|
### ESP32-Side Components
|
|
|
|
| Component | Responsibility | Communicates With |
|
|
|-----------|---------------|-------------------|
|
|
| WiFi Server | Listen for UDP datagrams or TCP connections, parse JSON | Animation Engine |
|
|
| Animation Engine | Maintain animation state, run effect loops, handle parameter updates | WiFi Server, LED Drivers |
|
|
| LED Drivers | Hardware-specific output: WS2801 via SPI, SK6812 via RMT peripheral | Hardware (GPIO/SPI) |
|
|
| Zone Manager | Route animation commands to correct strip (zone A = WS2801, zone B = SK6812) | Animation Engine, LED Drivers |
|
|
|
|
---
|
|
|
|
## Data Flow
|
|
|
|
### Choreography Playback Mode
|
|
|
|
```
|
|
Song file → Audio decoder → PulseAudio/PipeWire sink
|
|
│
|
|
elapsed time
|
|
│
|
|
Playback Clock ──► Choreography Engine
|
|
│
|
|
scheduled event fires
|
|
│
|
|
JSON command built
|
|
│
|
|
UDP datagram → ESP32
|
|
│
|
|
Animation Engine starts
|
|
│
|
|
LEDs execute effect
|
|
```
|
|
|
|
### Live Reactive Mode
|
|
|
|
```
|
|
System audio → PyAudio/sounddevice → Audio buffer
|
|
│
|
|
FFT / onset detection
|
|
│
|
|
beat event (aubio)
|
|
│
|
|
App Core receives event
|
|
│
|
|
mapped animation selected
|
|
│
|
|
JSON command → UDP → ESP32
|
|
│
|
|
LEDs react in near-realtime
|
|
```
|
|
|
|
### User Interaction Flow
|
|
|
|
```
|
|
SSH terminal → Textual event loop → UI widget action → App Core message
|
|
│
|
|
state mutation
|
|
│
|
|
reactive UI update
|
|
(no direct DOM)
|
|
```
|
|
|
|
### Choreography File Data Model
|
|
|
|
```
|
|
Choreography (file: .json / .yaml)
|
|
├── metadata
|
|
│ ├── song_path: str
|
|
│ ├── song_duration_s: float
|
|
│ └── bpm: float | null
|
|
└── events: [
|
|
{
|
|
time_s: float, # offset from song start
|
|
zone: "ws2801" | "sk6812" | "both",
|
|
animation: str, # registered name on ESP32
|
|
params: { # animation-specific
|
|
color: [r, g, b],
|
|
speed: float,
|
|
intensity: float,
|
|
loop: bool
|
|
},
|
|
duration_s: float | null # null = run until replaced
|
|
}
|
|
]
|
|
```
|
|
|
|
### JSON Command Protocol (Pi → ESP32)
|
|
|
|
```json
|
|
{
|
|
"zone": "ws2801",
|
|
"animation": "pulse",
|
|
"params": {
|
|
"color": [255, 0, 128],
|
|
"speed": 0.8,
|
|
"intensity": 1.0
|
|
}
|
|
}
|
|
```
|
|
|
|
A separate command handles stop/clear:
|
|
|
|
```json
|
|
{
|
|
"zone": "all",
|
|
"animation": "off"
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Recommended Architecture Pattern: Event-Driven Async Core
|
|
|
|
The App Core is an asyncio event loop (Textual already runs on asyncio). Everything hangs off it:
|
|
|
|
- **Playback clock** — asyncio task, fires scheduled events from choreography
|
|
- **Audio processor** — runs in a thread (audio callbacks are synchronous), posts beat events into the asyncio loop via `call_soon_threadsafe`
|
|
- **Network sender** — async UDP socket (asyncio DatagramProtocol), non-blocking
|
|
- **File I/O** — async file reads (aiofiles) to avoid blocking the TUI
|
|
|
|
This pattern avoids the thread-safety trap: only one asyncio task ever mutates state, threads post events inward via thread-safe bridges.
|
|
|
|
### Critical Pattern: Beat Detection Threading
|
|
|
|
```
|
|
Thread: audio callback (sounddevice/PyAudio)
|
|
└── aubio onset detector
|
|
└── beat detected
|
|
└── loop.call_soon_threadsafe(post_beat_event)
|
|
└── asyncio coroutine handles it
|
|
└── sends UDP command
|
|
```
|
|
|
|
Audio must run in a thread — audio callbacks cannot be async. This is the one mandatory thread boundary in the system.
|
|
|
|
---
|
|
|
|
## Transport Protocol Decision: UDP over TCP
|
|
|
|
**Use UDP for animation commands.** Rationale:
|
|
|
|
- Animation commands are stateless triggers, not streams. A dropped packet means a missed animation start — tolerable. A delayed retransmission (TCP) means a late animation start — worse for musical sync.
|
|
- UDP latency on a local WiFi network is typically 1-5ms vs TCP 10-30ms (with Nagle's algorithm; TCP_NODELAY helps but UDP is still lower).
|
|
- ESP32-C3's single RISC-V core handles UDP receive with less overhead than full TCP stack maintenance.
|
|
- Command payloads are small (< 200 bytes), well under UDP MTU (1470 bytes on typical WiFi).
|
|
- WLED (the leading ESP32 LED firmware) uses UDP for all real-time control protocols (E1.31, DDP, Art-Net). This is industry standard for LED sync.
|
|
|
|
**Heartbeat mechanism:** Pi sends a UDP heartbeat every 5s; ESP32 detects loss and enters "safe mode" (off or last known state).
|
|
|
|
**Confidence:** HIGH — verified against WLED protocol architecture and ESP32 forum discussions on latency optimization.
|
|
|
|
---
|
|
|
|
## ESP32-C3 Constraints and Firmware Architecture
|
|
|
|
The ESP32-C3 SuperMini has:
|
|
- 400 KB SRAM (tight)
|
|
- 4 MB Flash
|
|
- Single RISC-V core at 160 MHz
|
|
- No hardware FPU (floating-point operations are slower)
|
|
|
|
### FreeRTOS Task Allocation
|
|
|
|
```
|
|
Core 0 (only core):
|
|
Task 1 (high priority): LED output — drives SPI for WS2801, RMT for SK6812
|
|
Task 2 (medium priority): WiFi + JSON receive
|
|
Task 3 (low priority): Animation tick — updates frame buffer, feeds to LED task
|
|
```
|
|
|
|
WiFi and LED output cannot share timing perfectly on one core. The approach: LED task uses FreeRTOS queue to receive new frame buffers from the animation task. Animation task runs at a fixed interval (e.g., 20ms = 50fps) and posts computed frames into the queue. LED task drains the queue and writes to hardware.
|
|
|
|
### LED Driver Architecture
|
|
|
|
WS2801 (SPI):
|
|
- Uses ESP-IDF SPI master driver
|
|
- Clock + Data pins, speed ~1 MHz
|
|
- Blocking write per frame is acceptable (160 LEDs * 3 bytes = 480 bytes, fast at 1 MHz)
|
|
|
|
SK6812 (single-wire):
|
|
- Uses ESP-IDF RMT (Remote Control Transceiver) peripheral
|
|
- RMT generates precise timing pulses without CPU involvement
|
|
- Non-blocking: CPU queues frame data, RMT transmits via DMA
|
|
- SK6812 is RGBW (4 bytes/LED) → 300 LEDs * 4 bytes = 1200 bytes per frame
|
|
|
|
Both strips can update concurrently: SPI is blocking but fast, RMT is DMA-driven. Sequence: start RMT (non-blocking), write SPI (blocking), both finish in roughly the same window.
|
|
|
|
---
|
|
|
|
## Textual TUI Architecture
|
|
|
|
Textual uses a reactive widget tree with message passing. For this app:
|
|
|
|
```
|
|
App (root)
|
|
├── Header (song info, BPM, mode)
|
|
├── TransportBar (play/pause/stop, position scrubber)
|
|
├── MainContent (switches between views)
|
|
│ ├── TimelineView (horizontal scrollable canvas)
|
|
│ │ ├── TimeRuler
|
|
│ │ ├── ZoneTrack (WS2801)
|
|
│ │ └── ZoneTrack (SK6812)
|
|
│ └── EventListView (DataTable of events)
|
|
├── AnimationPanel (sidebar: select animation, set params)
|
|
└── StatusBar (ESP32 connection, audio status, confidence)
|
|
```
|
|
|
|
**Reactive data binding:** Textual's `reactive` descriptor triggers re-renders when playback position changes. The playback clock posts position updates at ~10 Hz, which drives the timeline cursor without flooding the event loop.
|
|
|
|
**Timeline rendering:** Custom Textual Canvas or Rich markup — render events as colored blocks at proportional x positions. Horizontal scroll follows playback cursor.
|
|
|
|
---
|
|
|
|
## Suggested Build Order (Dependencies)
|
|
|
|
The dependency graph drives this order:
|
|
|
|
```
|
|
1. ESP32 Firmware (foundation — everything depends on having a working LED executor)
|
|
└── LED drivers (WS2801 SPI, SK6812 RMT)
|
|
└── Animation engine (effect registry, parameter system)
|
|
└── WiFi UDP server + JSON parser
|
|
|
|
2. Transport Layer (Pi side — needed to test firmware without UI)
|
|
└── UDP socket client
|
|
└── JSON command builder
|
|
└── Connection management + heartbeat
|
|
|
|
3. App Core + State (no UI yet — testable via CLI)
|
|
└── Choreography data model (JSON load/save)
|
|
└── Playback clock
|
|
└── Event scheduler
|
|
|
|
4. Audio Layer (can be developed in parallel with App Core after firmware done)
|
|
└── Song playback (pygame/miniaudio)
|
|
└── Beat detection (aubio + sounddevice)
|
|
└── Audio→App Core event bridge
|
|
|
|
5. TUI Layer (depends on App Core having stable API)
|
|
└── Transport controls + status
|
|
└── Event list view
|
|
└── Timeline view (most complex — build last within TUI)
|
|
└── Animation parameter panel
|
|
|
|
6. Integration + Polish
|
|
└── Cyberpunk visual theme
|
|
└── Choreography file management (browse, rename, delete)
|
|
└── Live reactive mode (ties Audio Layer to Animation dispatch)
|
|
```
|
|
|
|
**Why firmware first:** Every other component is validated against a real ESP32. Developing the protocol without firmware means writing a mock, then potentially changing everything when the real hardware behaves differently. Ship firmware first, test with curl/netcat, then build the Pi application around proven commands.
|
|
|
|
**Why Audio Layer parallel to App Core:** Beat detection and song playback have no dependency on the UI. They can be prototyped as standalone scripts, then integrated.
|
|
|
|
---
|
|
|
|
## Cross-Cutting Concerns
|
|
|
|
### Latency Budget for Musical Sync
|
|
|
|
Musical sync at 120 BPM has 500ms between beats. A cue that fires 50ms early or late is perceptible. Target latency budget:
|
|
|
|
| Stage | Budget | Mechanism |
|
|
|-------|--------|-----------|
|
|
| Playback clock tick accuracy | < 10ms | asyncio event loop, avoid blocking |
|
|
| JSON serialization | < 1ms | ujson (faster than stdlib json) |
|
|
| WiFi UDP transit | < 10ms | local network, UDP |
|
|
| ESP32 receive to animation start | < 5ms | FreeRTOS high-priority LED task |
|
|
| **Total** | **< 26ms** | Well within perceptible threshold |
|
|
|
|
### Zone Independence
|
|
|
|
WS2801 (cabinet, 160 LEDs) and SK6812 (wall, 300 LEDs) must be independently addressable. The JSON protocol always specifies a zone. The ESP32 Zone Manager routes to the correct driver. Both zones can run different animations simultaneously — this is a first-class requirement, not an afterthought.
|
|
|
|
### Animation Registry
|
|
|
|
Animations are named strings registered in firmware. The Pi does not know animation internals — it only sends a name + parameters. This keeps the protocol stable: new animations are added to firmware without changing the Pi application, as long as the parameter schema is consistent.
|
|
|
|
Define a shared animation manifest (JSON file on Pi, #define registry in firmware) so the TUI can offer valid animation names and parameters without hardcoding them in two places.
|
|
|
|
---
|
|
|
|
## Anti-Patterns to Avoid
|
|
|
|
### Anti-Pattern 1: Frame Streaming from Pi to ESP32
|
|
|
|
**What:** Sending raw pixel arrays (160*3 + 300*4 = 1680 bytes) at 50fps = 84 KB/s continuously over WiFi UDP
|
|
**Why bad:** Saturates single-core ESP32 receive pipeline. WiFi on ESP32-C3 has practical throughput around 1-2 Mbps but latency spikes under load. 84 KB/s is feasible but leaves no headroom for WiFi overhead. Any retransmit or congestion causes visible glitch. Also burns CPU on both ends doing serialization.
|
|
**Instead:** Commands only. The project already decided this correctly — ESP32 runs animations locally.
|
|
|
|
### Anti-Pattern 2: Blocking Audio Processing on Asyncio Loop
|
|
|
|
**What:** Running aubio beat detection in an `async def` handler directly
|
|
**Why bad:** aubio's Python bindings are synchronous C extensions. Blocking the asyncio event loop for even 10ms causes TUI lag and missed events.
|
|
**Instead:** Thread worker with `call_soon_threadsafe` bridge (documented above).
|
|
|
|
### Anti-Pattern 3: Polling Playback Position from TUI
|
|
|
|
**What:** TUI widget calling `time.time()` on every render to calculate playback position
|
|
**Why bad:** Playback position should be authoritative state in App Core. Multiple widgets reading independently diverge. If audio playback pauses, widgets don't know.
|
|
**Instead:** App Core maintains `current_position_s` as reactive state. TUI subscribes to updates.
|
|
|
|
### Anti-Pattern 4: JSON Parsing on ESP32 ISR
|
|
|
|
**What:** Parsing received UDP JSON inside the WiFi receive interrupt handler
|
|
**Why bad:** JSON parsing is slow (malloc, string iteration) and will cause watchdog resets from ISR
|
|
**Instead:** WiFi ISR copies raw bytes to a FreeRTOS queue. Separate task drains queue and parses JSON with enough stack space.
|
|
|
|
---
|
|
|
|
## Scalability Considerations
|
|
|
|
This is a single-user, single-ESP32 system by design. Scalability means:
|
|
|
|
| Concern | Current | Future if needed |
|
|
|---------|---------|-----------------|
|
|
| More LED strips | Add second ESP32, Pi sends duplicate commands | Multi-target transport layer |
|
|
| More animations | Add to ESP32 firmware + manifest file | Plugin architecture on ESP32 |
|
|
| Longer songs | JSON file size grows linearly — no issue | No change needed |
|
|
| Multiple choreographies | File system already handles this | Directory browser UI |
|
|
|
|
---
|
|
|
|
## Sources
|
|
|
|
- [WLED Project Documentation](https://kno.wled.ge/) — Industry reference for ESP32 LED control architecture, UDP protocol patterns
|
|
- [WLED Control Interfaces (DeepWiki)](https://deepwiki.com/wled/WLED-Docs/6-control-interfaces) — Protocol specifics (E1.31, DDP, UDP)
|
|
- [ESP32-C3 Technical Reference Manual](https://www.espressif.com/sites/default/files/documentation/esp32-c3_technical_reference_manual_en.pdf) — RMT peripheral, SPI, single-core constraints
|
|
- [Getting Started with ESP32-C3 Super Mini](https://randomnerdtutorials.com/getting-started-esp32-c3-super-mini/) — Hardware specs (400KB RAM, 4MB Flash, 160MHz)
|
|
- [Textual Workers Guide](https://textual.textualize.io/guide/workers/) — Background task architecture, thread safety
|
|
- [Textual Events and Messages](https://textual.textualize.io/guide/events/) — Async message passing patterns
|
|
- [Real-Time Beat Tracking (TISMIR 2024)](https://transactions.ismir.net/articles/10.5334/tismir.189) — Latency sources in beat tracking pipelines
|
|
- [dancyPi: Audio-reactive LEDs on RPi + ESP](https://github.com/naztronaut/dancyPi-audio-reactive-led) — Reference architecture for similar system
|
|
- [ESP32 UDP Latency Discussion](https://github.com/espressif/arduino-esp32/issues/1283) — UDP vs TCP latency on ESP32
|
|
- [aubio Real-Time Beat Prediction](https://www.maxhaesslein.de/notes/real-time-beat-prediction-with-aubio/) — aubio integration pattern for live beat detection
|