diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 130491a1..e3c44659 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -129,7 +129,7 @@ jobs: ~/.espressif ~/esp/esp-idf # Bump the suffix when changing the IDF version to invalidate. - key: esp-idf-v6.1-beta1-${{ runner.os }}-v1 + key: esp-idf-v6.1-rc1-${{ runner.os }}-v1 # The release-channel tag burned into the binary as MM_RELEASE (shown by # SystemModule alongside the semver). Same resolution as the release @@ -172,24 +172,22 @@ jobs: - name: Build firmware uses: espressif/esp-idf-ci-action@v1 with: - # Match the local development IDF (v6.1-beta1, commit b1d13e9f). + # Match the local development IDF (v6.1-rc1, commit 44f0c59f). # The project uses v6-era APIs — `esp_eth_phy_new_generic`, the new # mDNS component manager, `int clock_gpio` in the EMAC config (v5.x # had a strongly-typed enum). v5.4 fails to compile platform_esp32.cpp. - # `v6.1-beta1` is the espressif/idf Docker tag pinned to the same - # tagged beta the local pin uses — a fixed, signed pre-release (not the + # `v6.1-rc1` is the espressif/idf Docker tag pinned to the same + # tagged pre-release the local pin uses — fixed and signed (not the # rolling `release-v6.1` branch tag), so CI and local build the exact # same tree. This tracks the v6.1 line toward GA; re-pin to the `v6.1` # tag once it ships (a deliberate re-test pass, see docs/building.md). - esp_idf_version: v6.1-beta1 + esp_idf_version: v6.1-rc1 # The IDF target follows the firmware-key prefix: esp32s31* → esp32s31 # (checked BEFORE esp32s3 — esp32s31 also startsWith 'esp32s3'), # esp32s3* → esp32s3, esp32p4* → esp32p4 (the only target that pulls # the ip101 PHY + esp_hosted, both manifest-gated on target == esp32p4), # everything - # else → esp32. (The matrix is the `ships` subset of firmwares.json; - # esp32p4-eth-wifi has ships=False in build_esp32.py so it stays out — - # its C6-slave Kconfig defaults don't survive a plain CI build.) + # else → esp32. (The matrix is the `ships` subset of firmwares.json.) target: ${{ startsWith(matrix.firmware, 'esp32s31') && 'esp32s31' || startsWith(matrix.firmware, 'esp32s3') && 'esp32s3' || startsWith(matrix.firmware, 'esp32p4') && 'esp32p4' || 'esp32' }} path: 'esp32' # We run our own builder (not the action's default `idf.py build`) diff --git a/docs/architecture.md b/docs/architecture.md index c3599d0b..1cc2355c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -257,7 +257,7 @@ Three distinct things, kept distinct in the vocabulary: - **deviceModel** — the whole assembled product, identified by its catalog name (`Olimex ESP32-Gateway Rev G`). This is *which hardware this is*. It is distinct from **`deviceName`**, *which individual unit this is* (per-unit identity the user sets — see [§ Device name](#device-name-one-identity-every-network-name-derives-from-it)); a **device** (the umbrella term) has a `deviceName` and a `deviceModel`. - **board** — the bare PCB *only*. The word survives in its literal sense: **on-board** LED, **on-board** peripherals, board-soldered pins — things physically *on the PCB*. (A deviceModel is a board plus whatever is wired onto it.) -**Firmware** is the compiled binary: chip target plus which radios/peripherals/sdkconfig fragments are included. Today's variants: `esp32` (classic, WiFi **and** RMII Ethernet in one binary — Ethernet comes up only when a PHY is present, pins/PHY per deviceModel), `esp32-eth` (classic, Ethernet only, WiFi excluded), `esp32-16mb` (classic with 16 MB flash, WiFi + Ethernet), `esp32s3-n16r8` / `esp32s3-n8r8` (S3 with WiFi + W5500 SPI Ethernet), `esp32p4-eth` (Waveshare ESP32-P4-NANO, Ethernet only), `esp32p4-eth-wifi` (the same P4 hardware with WiFi via its on-board ESP32-C6 over esp_hosted). Each chip's firmware carries the Ethernet *driver(s)* it can host (RMII EMAC for classic/P4, W5500 SPI for S3); which PHY/pins a deviceModel uses is runtime config. Selected by `build_esp32.py --firmware `, reported by `SystemModule.firmware`, used as the contract target key in scenarios. +**Firmware** is the compiled binary: chip target plus which radios/peripherals/sdkconfig fragments are included. Today's variants: `esp32` (classic, WiFi **and** RMII Ethernet in one binary — Ethernet comes up only when a PHY is present, pins/PHY per deviceModel), `esp32-eth` (classic, Ethernet only, WiFi excluded), `esp32-16mb` (classic with 16 MB flash, WiFi + Ethernet), `esp32s3-n16r8` / `esp32s3-n8r8` (S3 with WiFi + W5500 SPI Ethernet), `esp32p4rev1-eth` (Waveshare ESP32-P4-NANO, Ethernet only), `esp32p4rev1-eth-wifi` (the same P4 hardware with WiFi via its on-board ESP32-C6 over esp_hosted), `esp32p4rev3-eth` / `esp32p4rev3-eth-wifi` (the same two images built for P4 **v3.x** silicon, which is not binary-compatible with rev <3.0 — untested, no v3 board on the bench). Each chip's firmware carries the Ethernet *driver(s)* it can host (RMII EMAC for classic/P4, W5500 SPI for S3); which PHY/pins a deviceModel uses is runtime config. Selected by `build_esp32.py --firmware `, reported by `SystemModule.firmware`, used as the contract target key in scenarios. **deviceModel** is the physical hardware: chip + PCB + on-board peripherals (PHY, USB-serial, PSRAM, antenna), identified by its product name. Examples: `Olimex ESP32-Gateway Rev G`, `LOLIN D32`, `Generic ESP32 Dev`. A unit cannot identify its own deviceModel (no readable PCB ID on classic ESP32), so MoonDeck deduces it from the firmware where unambiguous (`esp32-eth*` ⇒ Olimex) and otherwise lets the user pick. It is stored on the unit as SystemModule's `deviceModel` Text control (display-only in the UI; HTTP `/api/control` writes still apply). MoonDeck mirrors the picked / deduced value to the unit via `POST /api/control` after each discover and after every dropdown change. The catalog of valid deviceModels lives at [web-installer/deviceModels.json](../web-installer/deviceModels.json), shared between MoonDeck and the web installer: MoonDeck reads it for its dropdown and HTTP push (plain REST on the LAN); the web installer reads it for its picker and pushes the whole entry — deviceModel plus every module/control — over serial during provisioning as REST ops (**"Improv = REST over serial"**, the `APPLY_OP` vendor RPC; see [ImprovProvisioningModule.md](moonmodules/core/moxygen/ImprovProvisioningModule.md)). Pushing over serial sidesteps the mixed-content block that stops an HTTPS installer page from POSTing to an `http://` device; an already-running device is re-configured via MoonDeck on the LAN. diff --git a/docs/backlog/backlog-core.md b/docs/backlog/backlog-core.md index f6f7d913..e19acf67 100644 --- a/docs/backlog/backlog-core.md +++ b/docs/backlog/backlog-core.md @@ -8,9 +8,9 @@ Forward-looking to-build items for the **core / infrastructure** domain (`src/co 1.0 ships ESP32 firmware (4 variants) + macOS arm64 + Windows x64. Still to add: -- **ESP32-P4** firmware variant — **`esp32p4-eth` (Ethernet-only) shipped**: in `build_esp32.py`'s `FIRMWARES`, the `deviceModels.json` catalog (Waveshare P4-NANO), and CI builds + publishes it to the web installer + releases. **Still to ship: `esp32p4-eth-wifi`** (the C6-WiFi variant) — it doesn't build reproducibly in CI yet (the `CONFIG_WIFI_RMT_*` Kconfig defaults don't survive a plain build without a fresh `set-target`), so it's held out of the release matrix until that's fixed; see § ESP32-P4 round 3. +- **ESP32-P4** firmware variant — **`esp32p4rev1-eth` (Ethernet-only) shipped**: in `build_esp32.py`'s `FIRMWARES`, the `deviceModels.json` catalog (Waveshare P4-NANO), and CI builds + publishes it to the web installer + releases. **`esp32p4rev1-eth-wifi` now ships too** (2026-08-19): it boots and associates on IDF v6.1-rc1, so it is out of the experimental set in the installer and carries a normal description. Its open defect is throughput, not shipping — see § ESP32-P4 round 3, open issue 0. - **ESP32-S31 web-flash (waiting on esptool-js)** — the `esp32s31` firmware ships (build, catalog, CI matrix, web installer listing), and CLI flashing works (`flash_esp32.py` → esptool.py, which has S31 support since v5.2.0). **Browser flashing does not**: the web installer's `esptool-js` (pinned 0.5.7) has no S31 chip class. Worse than a missing entry — the S31's ROM magic (`15736195`) *collides* with the classic ESP32's; esptool.py disambiguates with secondary register detection (S31 `USES_MAGIC_VALUE=False`), but esptool-js has only the magic table, so it would mis-identify the RISC-V S31 as a classic Xtensa ESP32 and flash the wrong stub/params. `install.js`'s `WEB_FLASH_UNSUPPORTED_CHIPS` guard catches an S31 connect-flash failure and points the user at the CLI. **No upstream timeline**: as of 2026-06 the esptool-js repo has zero S31 issues/PRs/commits and its last release was 2026-03 (it lags esptool.py on new chips by months). **Removal trigger**: when esptool-js ships S31 support *with* the secondary detection (not just a magic-table entry — re-check the chip-detect switch, not the version number), bump the esptool-js pin in `install-orchestrator.js` and drop `ESP32-S31` from `WEB_FLASH_UNSUPPORTED_CHIPS`. -- **ESP32-P4 v3.x silicon variant (backlog)** — `esp32p4-eth` is built for pre-v3 P4 (`CONFIG_ESP32P4_SELECTS_REV_LESS_V3` + `REV_MIN_0`), because the v6.1 IDF default (v3.1) refused to boot on the bench/field v1.x P4 and rev <3.0 vs >=3.0 are "huge hardware difference" (one binary can't cover both). The field is pre-v3 P4 today, so the single image is correct for now. When v3.x P4 silicon arrives, add an `esp32p4-eth-v3` firmware variant (REV_MIN_300+) + the catalog/device-model mapping so each P4 board points at the matching image, rather than the pre-v3 `esp32p4-eth` for everything. (CodeRabbit flagged the single-variant exposure; deferred until v3.x P4 is actually in play.) +- **ESP32-P4 v3.x silicon variant (backlog)** — `esp32p4rev1-eth` is built for pre-v3 P4 (`CONFIG_ESP32P4_SELECTS_REV_LESS_V3` + `REV_MIN_0`), because the v6.1 IDF default (v3.1) refused to boot on the bench/field v1.x P4 and rev <3.0 vs >=3.0 are "huge hardware difference" (one binary can't cover both). **DONE (2026-08-19): `esp32p4rev3-eth` and `esp32p4rev3-eth-wifi` ship** (`REV_MIN_300`, which covers v3.0-v3.99), reusing the rev1 board fragment so the partition table and EMAC config stay in one place. **Still open: neither has ever been booted** — both bench boards are v1.3 engineering samples, so the rev3 images are flagged experimental in the installer and need a v3 board to verify. Espressif does not recommend v0.x/v1.x for new designs, so a board bought today is v3.x and needs these. - **Linux desktop binary** — third desktop job in `release.yml`, static-linked libstdc++. - **Teensy 4.1** — toolchain-file build, `.hex` for Teensy Loader. - **Raspberry Pi** — ARM64, cross-built or native. @@ -19,8 +19,20 @@ Forward-looking to-build items for the **core / infrastructure** domain (`src/co - **Live RMII Ethernet reconfigure** — runtime PHY/pin config shipped (`ethType` + pin controls in NetworkModule, per-board defaults in `deviceModels.json`, `platform::setEthConfig`/`ethInit` dispatch). W5500 (SPI) on S3 applies **live** — `ethStop()` tears down the SPI bus and `ethInit()` re-runs on the next `loop1s()` with no reboot. RMII (classic/P4 internal EMAC) still saves config and asks for a restart to apply, because the EMAC bring-up is fiddlier to hot-cycle cleanly. Make RMII live too: a hot `esp_eth_stop` + EMAC/netif teardown + re-init on config change, matching the W5500 path, so every interface honours the no-reboot principle. - **Installer UX polish** — clear "Pre-release (beta)" warning on RC/latest picks, yank-by-asset-tag instead of yank-by-release-deletion. - **Offer projectMM/MoonLight as a library** — a downstream sketch where another firmware/app consumes the light pipeline (or a subset) as an embeddable dependency rather than running the whole binary. `library.json` is already a PlatformIO *library* manifest, so the seed exists. When this is designed, give it a small public **identity surface**: one runtime constant the consumer reads (a `kProjectName`, likely a `ProjectInfo` bundle of name + version + url) that the network wire-strings (ArtNet/E1.31 source-name + CID), the UI banner, and any "About" string all *derive from* — the one place a consumer queries "what am I embedding." This is the genuine home for the name-centralisation that the rename ([rename-to-moonlight.md § Phase 1.3](rename-to-moonlight.md)) deliberately *didn't* do: the rename is a one-time sweep (a constant would just split it), but a library consumer references the identity ongoing and widely, which is the test a constant must pass. Build it *then*, against the real library API, not speculatively now. +- **A uint16 script member reads back 0 on the DESKTOP backend (2026-08-20).** `addUint16` works + end-to-end on hardware — S3 (Xtensa) and S31 (RISC-V) both publish ember's `cycle` control and + drive it to 2000 and back, and the arena holds the right bytes on desktop too (`232,3` = 1000). + But re-running a compiled script after writing the slot renders 0 rather than the new value on + the host JIT, so the emitted `LoadCtrl16` either is not implemented on the arm64 backend or the + read is folded to the initializer. Found while adding the runtime assertion CodeRabbit asked for; + no existing test covered a wide member's runtime READ (the one that mentions `uint16_t x = 300` + only compiles it). Not user-visible today — the feature ships on the device backends where it is + verified — but it means the desktop suite cannot pin wide-member reads, so the next wide-member + codegen bug would only show on hardware. Reproduce with a two-line script and a slot write; the + assertion is written out in `unit_moonlive_fill.cpp` under "a uint16_t member is published". + - **ESP32-P4 panics with `Cache error` every few minutes, pre-existing** (2026-08-19): the bench - P4 (Waveshare P4-NANO, `esp32p4-eth`) reboots roughly every four minutes while IDLE, with + P4 (Waveshare P4-NANO, `esp32p4rev1-eth`) reboots roughly every four minutes while IDLE, with `Guru Meditation Error: Core 0 panic'ed (Cache error)`, sometimes followed by an `Illegal instruction` and a `CHIP_LP_WDT_RESET` on the way down. @@ -556,25 +568,77 @@ This is hardening, not a known bug — the shipped fix is correct for the cases ### ESP32-P4 support — rounds 3-4 (in progress) Rounds 1 (board + Ethernet-only) and 2 (Parlio LED driver) have landed. Remaining rounds, each its own plan + commit: -- **Round 3 — WiFi via the C6 co-processor. PARTIALLY PROVEN — C6 link up, STA failover not yet working.** The P4 has no native radio (`SOC_WIFI_SUPPORTED` absent); WiFi comes from the on-board ESP32-C6 over SDIO via `esp_wifi_remote` / esp_hosted. Landed as the `esp32p4-eth-wifi` firmware variant: components pulled P4-only (`rules:` gate in `idf_component.yml`), and `ensureWifiInit()` adds an `esp_hosted_init` + `connect_to_slave` prelude before `esp_wifi_init` (gated on `platform::usesRemoteWifi`). The rest of the WiFi seam is unchanged because `esp_wifi_remote` is API-compatible. A deliberate, documented [v6.0-floor exception](../building.md#esp-idf-version); C6 config via `CONFIG_SLAVE_IDF_TARGET_ESP32C6` + `CONFIG_ESP_HOSTED_CP_TARGET_ESP32C6` + the `CONFIG_ESP_HOSTED_P4_DEV_BOARD_FUNC_BOARD` SDIO-pin preset. +- **Round 3 — WiFi via the C6 co-processor. WORKING, BUT THE LINK IS SLOW (2026-08-19).** Boots and associates on IDF v6.1-rc1 (see round 4); the remaining defect is throughput, bench-bisected below. The P4 has no native radio (`SOC_WIFI_SUPPORTED` absent); WiFi comes from the on-board ESP32-C6 over SDIO via `esp_wifi_remote` / esp_hosted. Landed as the `esp32p4rev1-eth-wifi` firmware variant: components pulled P4-only (`rules:` gate in `idf_component.yml`), and `ensureWifiInit()` needs no hosted bring-up of its own: esp_hosted self-initialises at boot via a constructor (`ESP_SYSTEM_INIT_FN`), which sets up the SDIO transport, RPC and wifi-remote channels before `app_main`. Calling `esp_hosted_init`/`connect_to_slave` there would be worse than redundant — `connect_to_slave` is a transport *reconfigure* that resets the slave and re-inits SDIO, which fails on a live link. The rest of the WiFi seam is unchanged because `esp_wifi_remote` is API-compatible. A deliberate, documented [v6.0-floor exception](../building.md#esp-idf-version); C6 config via `CONFIG_SLAVE_IDF_TARGET_ESP32C6` + `CONFIG_ESP_HOSTED_CP_TARGET_ESP32C6` + the `CONFIG_ESP_HOSTED_P4_DEV_BOARD_FUNC_BOARD` SDIO-pin preset. **Hardware results (bench, P4-NANO, 2026-06-12):** - ✅ **esp_hosted / C6 SDIO comes up at boot.** `host_init: ESP Hosted`, `H_API: ESP-Hosted starting`, `add_esp_wifi_remote_channels`, `H_SDIO_DRV: sdio_data_to_rx_buf_task started`. No NVS error / assert / panic / hang. Device boots fully (~57-60 FPS), `hasWiFi` true, WiFi controls present. esp_hosted **self-initialises at boot via a constructor** (`ESP_SYSTEM_INIT_FN` → `esp_hosted_init`), so no bring-up code is needed in our platform layer — an earlier explicit `esp_hosted_init` + `esp_hosted_connect_to_slave` prelude was *removed*: init was a redundant no-op and `connect_to_slave` is actually a transport *reconfigure* (slave GPIO-54 reset + SDIO re-init). SDIO config confirmed correct on the wire: `CLK[18] CMD[19] D0[14] D1[15] D2[16] D3[17] Slave_Reset[54]`, 4-bit 40 MHz. - ❌ **WiFi STA connect fails on the SDIO re-init.** The cascade DOES fire correctly (`Ethernet no link, cascading` → STA path), but `esp_wifi_init()` (forwarded to esp_wifi_remote) internally triggers `esp_hosted_reconfigure` → `Reset slave using GPIO[54]` → **`sdmmc_card_init failed` (×15) → `card init failed` → `esp_wifi_init failed: ESP_FAIL`**. So: the boot-time SDIO init succeeds, but a **runtime slave reset can't re-establish the SDIO link**. The C6 doesn't come back after the GPIO-54 reset during operation. This is an esp_hosted/SDIO/C6-slave-firmware level issue (reset timing or slave image), below our application code — the pins and Kconfig are correct. **Open issues before this is done:** - 1. **Runtime SDIO re-init of the C6 fails — CONFIRMED a C6 slave-firmware problem (not a guess).** SystemModule now exposes a `wifiCoproc` read-only control (via `platform::coprocessorWifi()` → `esp_hosted_get_coprocessor_fwversion()`), and on the bench it reads **`not detected`** — the C6 returns no valid firmware version (0.0.0 / handshake never completes), which is exactly the documented signature of absent / incompatible C6 slave firmware. So this is proven off the device, not inferred. Likely a version mismatch on top of that: The host pulled esp_hosted **2.12.9**; Espressif's P4-Function-EV-Board ships its C6 pre-flashed with esp_hosted slave **v0.0.6**, and the **Waveshare NANO is a different board that may carry a different / absent C6 slave image**. The symptom fits: boot inits the host SDIO master fine, but resetting the C6 (GPIO 54) and re-enumerating it as a slave fails (`sdmmc_card_init failed`) because the C6 has no compatible slave firmware responding. **Primary next step: build + flash the version-matched esp_hosted slave firmware onto the NANO's C6.** The slave project is already vendored at `esp32/managed_components/espressif__esp_hosted/slave/` (`sdkconfig.defaults.esp32c6`, `partitions.esp32c6.csv`); `idf.py create-project-from-example "espressif/esp_hosted:slave"` → `set-target esp32c6` → flash. **Caveat / needs PO + bench hardware:** flashing the C6 on the EV board uses an **ESP-Prog wired to the `PROG_C6` header** with the P4 held in bootloader mode (esp_hosted `docs/esp32_p4_function_ev_board.md` §5.2) — the NANO's C6-flash path must be confirmed (separate USB? equivalent header? ESP-Prog?), and an ESP-Prog may be needed. An OTA slave-update path exists but needs a *working* link first (chicken-and-egg here). This is a hardware-provisioning task, not application code. Secondary fallbacks if firmware-match doesn't fix it: an esp_hosted option to skip the reconfigure/slave-reset when the transport is already up at boot; a slower SDIO freq or 1-bit mode; verify GPIO 54 reset polarity/timing for the NANO. **(Note: EIM — the building.md v6.0-adoption item — does NOT help here; it's a host-machine installer, unrelated to device-side C6 firmware.)** + + 0. **The esp_hosted build is ~17x slower on HTTP, with a ~1 s stutter — BISECTED to the hosted link, not to us (2026-08-19).** Same board, same commit, same application code, measured over **Ethernet on both builds** so the radio is not in the path: + + | build | per-request (`/api/system`) | throughput (73 KB `app.js`) | + |---|---|---| + | `esp32p4rev1-eth-wifi` | alternating 0.434 / 0.788 / 0.446 / 0.813 s | 42 KB/s | + | `esp32p4rev1-eth` | 13 ms flat, 12 consecutive fetches | 708 KB/s | + + Render is healthy in both (121-134 fps), so this is not frame-loop contention. Because the two images share every line of application code, the penalty is **esp_hosted being compiled in and its SDIO link serviced**, degrading traffic on an interface it is not even carrying. The cost is **per-request, not per-byte** (a 1 KB fetch cost nearly as much as a 36 KB fetch), which points at a periodic blocker a request must wait out rather than a slow pipe. + + Already excluded on the bench: SDIO width/clock (already 4-bit/40 MHz, the maximum), WiFi buffer counts (match the S3's), ICMP latency (6.6 ms). An earlier `SystemModule` fix (`coprocessorWifi()` retried an unanswered version query every tick: 1,012,344 us → 257 us) was real and is kept, but it was **not** this: the stutter outlived it and vanished only when the C6 path left the image. + + **Per-module tick timing, both builds on the SAME Ethernet interface (2026-08-19).** The measurement that says what this is NOT: + + | module | eth-only | eth-wifi | factor | + |---|---|---|---| + | HttpServer | 113 us | 3441 us | **30x** | + | File Manager | 279 us | 4686 us | 17x | + | Services | 242 us | 1388 us | 5.7x | + | Audio | 239 us | 1376 us | 5.7x | + | Effects | 172 us | 397 us | 2.3x | + | **Network** | 10 us | **92 us** | (negligible in absolute terms) | + | Drivers | 6601 us | 4215 us | **0.64x (FASTER)** | + | ParallelLed | 7186 us | 6569 us | 0.91x (faster) | + + **Two theories are refused by this table.** It is not a blocking WiFi call in `NetworkModule` (`Network` costs 92 us, and `wifiStaRssi`/`wifiTxPower` are state-gated off entirely in `ConnectedEth`); and it is not periodic WiFi scanning, since no module carries anything like a 0.37 s cost. Instead **modules that touch no network at all** (Audio, Noise, Effects, Layer) slow by the same kind of factor as the HTTP path: a broad, roughly proportional slowdown of ordinary code. + + **Working theory: L2 cache contention.** Both builds are byte-identical in every memory setting (`CACHE_L2_CACHE_128KB`, `SPIRAM_SPEED_200M`, `FLASHFREQ_40M`, `SPIRAM_MODE_HEX` — a full sdkconfig diff shows no cache/PSRAM/flash/CPU-freq difference), and code executes from flash through that shared 128 KB L2 in both. Adding esp_hosted's tasks, ISRs and DMA buffers evicts application code that was previously resident, so ordinary work starts missing to flash/PSRAM. This also explains the otherwise odd inversion: the DMA-bandwidth-bound `Drivers`/`ParallelLed` got *faster*, which is what happens when the CPU competes less for the same bus. **Consistent with the data, not yet proven** — separating cache contention from plain CPU stealing needs a cache-hit-rate counter or a per-tick histogram, neither of which we expose today. + + **The cache theory does NOT explain the visible LED hiccup, and there are TWO effects here (PO observation, 2026-08-19).** Uniform cache contention predicts a smooth frame-rate drop (133 → 96 → ~73 fps), which is what the averages show. It does not predict a *stall*, yet a once-per-second hiccup is plainly visible on the fixture. So the throughput loss and the hiccup are separate problems and must be chased separately. + + **Our instrumentation structurally cannot see the hiccup.** `MoonModule::tickTimeUs_` is a MEAN (`accumUs_ / frameCount`, [MoonModule.h:656](../../src/core/MoonModule.h)) and `Scheduler::fps()` is derived from it, so a single 300 ms frame among 70 good ones shifts the average ~4 ms and vanishes. Every per-module number in the table above is an average and none of them can confirm or refute a stall. External evidence of the stall is therefore weak but non-zero: sampling `fps` once per second for 30 s gives a steady 73 with periodic dips to 69-70 (~55 ms lost in those seconds), and the dips are NOT on a clean 1 s period. Ruled out along the way: the degradation is not caused by our own HTTP polling (fps is identical at 76 after 25 s of zero traffic and under continuous polling), and the early 96 → 73 decay is warm-up, not load. + + **Per-TASK CPU, both builds, `--task-cpu-stats` (2026-08-19) — this is the decisive measurement.** Same profiling overhead on both, so the comparison is clean: + + | task | prio | eth-only | eth-wifi | | + |---|---|---|---|---| + | `main` (render) | 1 | **35.4%** | **93.3%** | the same work costs 2.6x the CPU | + | `IDLE0` | 0 | 63.3% | 5.1% | core 0 headroom is gone | + | `ipc1` | 24 | 7.8% | 12.7% | present in BOTH, so not the cause | + | `mmEncode` | 5 | 3.4% | 6.0% | | + | `sdio_read` / `sdio_write` / `sdio_process_rx` / `rpc_rx` / `rpc_tx` | 23 | absent | **0.0-0.1%** | the hosted tasks are IDLE | + + **What this refutes.** Not a busy WiFi task: every SDIO/RPC task sits at 0.0-0.1%. Not periodic WiFi scanning, and not a blocking call in our render path either, since `main` is not *waiting* — it is *running*, and burning 2.6x the cycles for identical work. Not `ipc1` (priority 24, preempts everything), which is present on the eth-only build too. There is no WiFi *activity* to stop, so a "compile it in but don't run it" variant would likely change nothing; the cost is already there with the link idle. + + **What this supports.** The same instruction stream executing 2.6x slower with no extra runnable work is the signature of **memory contention**, consistent with the L2-cache theory above: esp_hosted's footprint evicts application code from the shared 128 KB L2, so ordinary code stalls on flash/PSRAM fetches. Cycles are spent *inside* `main`, which is why every module slowed proportionally and why the DMA-bound drivers (bandwidth-bound, not cache-resident) got faster. + + **Still unexplained: the visible ~1 s LED hiccup.** Cache contention predicts the steady 2.6x, not a stall. Per-task CPU is cumulative-since-boot and cannot show a spike either. So the hiccup remains unmeasured, and the worst-case instrumentation below is still the next step for it specifically. + + **Instrumentation gap to close first — worst-case tick tracking.** Add a per-module and per-scheduler *max* (and ideally a coarse histogram or a "frames over 2x mean" counter) alongside the existing mean, and expose it in `/api/system`. Cheap (one comparison per tick), and it converts "the PO can see a hiccup that no number shows" into a measurable quantity. Without it, every theory below is unfalsifiable from the host side. Reading the P4 console needs `CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG=y` or a UART adapter on GPIO 37/38 (see the dev-loop note in round 4), so serial is not the fast path. + + Then, cheapest first: (a) read the new max/histogram to see whether the cost is uniform or spiky (uniform favors cache contention, spiky favors a blocking call, and the two coexisting is now the leading reading); (b) the P4 cache performance counters for a direct miss-rate read; (c) moving the hot render path to IRAM/internal RAM and re-measuring, which is also a candidate fix rather than only a diagnostic. + 1. **Runtime SDIO re-init of the C6 fails — CONFIRMED a C6 slave-firmware problem (not a guess).** SystemModule now exposes a `wifiCoproc` read-only control (via `platform::coprocessorWifi()` → `esp_hosted_get_coprocessor_fwversion()`), and on the bench it read **`not detected`** at the time (the control now reports **`no version reply`**, and only after a bounded number of attempts): the C6 returns no valid firmware version. **What that means was overstated here.** It was read as the signature of absent / incompatible slave firmware, but the same board later associated and served traffic over that very link while this RPC still went unanswered — so an unanswered version query says the QUERY failed, not that the C6 is absent. The round-3 conclusion below rests on the `sdmmc_card_init failed` re-init evidence, not on this control. Likely a version mismatch on top of that: The host pulled esp_hosted **2.12.9**; Espressif's P4-Function-EV-Board ships its C6 pre-flashed with esp_hosted slave **v0.0.6**, and the **Waveshare NANO is a different board that may carry a different / absent C6 slave image**. The symptom fits: boot inits the host SDIO master fine, but resetting the C6 (GPIO 54) and re-enumerating it as a slave fails (`sdmmc_card_init failed`) because the C6 has no compatible slave firmware responding. **Primary next step: build + flash the version-matched esp_hosted slave firmware onto the NANO's C6.** The slave project is already vendored at `esp32/managed_components/espressif__esp_hosted/slave/` (`sdkconfig.defaults.esp32c6`, `partitions.esp32c6.csv`); `idf.py create-project-from-example "espressif/esp_hosted:slave"` → `set-target esp32c6` → flash. **Caveat / needs PO + bench hardware:** flashing the C6 on the EV board uses an **ESP-Prog wired to the `PROG_C6` header** with the P4 held in bootloader mode (esp_hosted `docs/esp32_p4_function_ev_board.md` §5.2) — the NANO's C6-flash path must be confirmed (separate USB? equivalent header? ESP-Prog?), and an ESP-Prog may be needed. An OTA slave-update path exists but needs a *working* link first (chicken-and-egg here). This is a hardware-provisioning task, not application code. Secondary fallbacks if firmware-match doesn't fix it: an esp_hosted option to skip the reconfigure/slave-reset when the transport is already up at boot; a slower SDIO freq or 1-bit mode; verify GPIO 54 reset polarity/timing for the NANO. **(Note: EIM — the building.md v6.0-adoption item — does NOT help here; it's a host-machine installer, unrelated to device-side C6 firmware.)** **User lead (2026-07-09) — avoid the WiFi teardown/re-init on hosted targets entirely; it may sidestep the slave-reset failure.** A user hit the mirror symptom on a *different* codebase (ESP32-Sveltekit / WLED-MM, not projectMM — its `lib/framework/WiFiSettingsService.cpp` doesn't exist here): after a clean C6-flash the P4-NANO WiFi worked, but their app's boot-time `WiFi.disconnect(true)` fully tore down the WiFi stack, and the later AP bring-up then failed with `esp_hosted_transport_config: Transport already initialized` / `esp_hosted_init failed!` / `AP enable failed!`. Their working fix: on `CONFIG_ESP_WIFI_REMOTE_ENABLED` (= hosted) targets, **don't do a full stack reset** — keep STA enabled, reconnect *without* a full teardown/re-init, and never call `WiFi.disconnect(true)` in the disconnect callback. **Why this is relevant to us even though the file differs:** our own comment at `platform_esp32.cpp:796` already documents that the esp_hosted transport is set up **once at boot** and is fragile to re-init (`connect_to_slave` = a transport reconfigure that resets the slave + re-inits SDIO and fails on a live link). Our disconnect *callback* is already safe (`wifiEventHandler` on `STA_DISCONNECTED` only sets a flag — no teardown), BUT our **failover path is not**: `wifiStaStop()` (`platform_esp32.cpp:915`) calls `esp_wifi_deinit()`, and the STA-retry / AP-fallback then re-runs `ensureWifiInit()` → `esp_wifi_init()` — the exact deinit→reinit cycle that on a hosted target triggers the GPIO-54 slave reset (round-3 open-issue #2's `sdmmc_card_init failed`). So the round-3 failure ("runtime SDIO re-init of the C6 fails") and this user's report may be **the same root cause**: the re-init shouldn't happen at all on a hosted target. **Concrete next step to try on the bench:** guard the teardown/re-init on `platform::hasWifiCoprocessor` (already defined = `isEsp32P4 && hasWiFi`) — on hosted targets, do NOT `esp_wifi_deinit()` in `wifiStaStop()` and do NOT re-`esp_wifi_init()` in `ensureWifiInit()` once the boot-time init is up; instead just `esp_wifi_disconnect()` + `esp_wifi_set_config()` + `esp_wifi_connect()` (STA retry) or `esp_wifi_set_mode(APSTA)` for the fallback, reusing the live transport. This is cheaper than the C6 reflash and independent of #18759, so it's worth trying first once the board boots. If it works, it also removes the slave-reset from the normal failover, not just the AP case. Blocked behind the #18759 boot crash like everything else P4-WiFi, but this is the first thing to try when the board boots again. - 2. **Co-processor components no longer compile into `esp32p4-eth` — FIXED.** The gate is now `rules: if CONFIG_MM_P4_WIFI == True` (a Kconfig option declared in `esp32/main/Kconfig.projbuild`, set only by `sdkconfig.defaults.esp32p4-eth-wifi`), so `esp_hosted` / `esp_wifi_remote` are pulled **only** by the WiFi build, never by eth-only. The old `target == esp32p4` gate pulled them into `esp32p4-eth` too; that wasn't merely build-time waste — esp_hosted self-inits its SDIO master at boot, which on the eth-only build interfered with the EMAC bring-up (a red herring chased during the P4 no-DHCP hunt). The eth-only image dropped 1.36→1.12 MB once gated out. The `wifiCoproc` read-out stays compile-gated on `platform::hasWifiCoprocessor` (`isEsp32P4 && hasWiFi`). + 2. **Co-processor components no longer compile into `esp32p4rev1-eth` — FIXED.** The gate is now `rules: if CONFIG_MM_P4_WIFI == True` (a Kconfig option declared in `esp32/main/Kconfig.projbuild`, set only by `sdkconfig.defaults.esp32p4rev1-eth-wifi`), so `esp_hosted` / `esp_wifi_remote` are pulled **only** by the WiFi build, never by eth-only. The old `target == esp32p4` gate pulled them into `esp32p4rev1-eth` too; that wasn't merely build-time waste — esp_hosted self-inits its SDIO master at boot, which on the eth-only build interfered with the EMAC bring-up (a red herring chased during the P4 no-DHCP hunt). The eth-only image dropped 1.36→1.12 MB once gated out. The `wifiCoproc` read-out stays compile-gated on `platform::hasWifiCoprocessor` (`isEsp32P4 && hasWiFi`). 3. **Build reproducibility.** `build_esp32.py` does not yet build this variant reliably: the C6 slave-target Kconfig `default ... if IDF_TARGET_ESP32P4` only fires on `set-target`, and the reconfigure a plain `build` triggers drops it back to ESP32-H2 (no WiFi) → fails on missing `CONFIG_WIFI_RMT_*`. A clean manual sequence works (`rm -rf ` → `set-target esp32p4` → `build`, all with the same `-DSDKCONFIG`/`-DSDKCONFIG_DEFAULTS`); the wrapper needs a fix so the auto-default sticks across reconfigures (see the KNOWN ISSUE comment in `build_esp32.py`). **Round 4 — the IDF-update regression (2026-07-03).** After the IDF bump to `v6.1-dev-5215-g0d928780081`, the variant stopped building entirely, then stopped booting. Two distinct causes, both IDF/component-manager side (all our config was correct): - ❌→✅ **Build: `esp_hosted.h` not found — the manifest `if` syntax changed.** The component manager (now 3.0.3) silently skipped `esp_hosted`/`esp_wifi_remote` (`NOTICE: Skipping optional dependency`) because our `idf_component.yml` rule used the bare `CONFIG_MM_P4_WIFI == True`. The current manager only recognises a Kconfig variable in the **`$CONFIG{...}`** form (its `KCONFIG_VAR_REGEX = \$CONFIG\{([^}]+)}`); the bare `CONFIG_X` falls through to plain string-eval → false → skipped. **Fixed:** `if: "$CONFIG{MM_P4_WIFI} == True"` (note: NO `CONFIG_` prefix inside the braces). Confirmed by [idf-component-manager #104](https://github.com/espressif/idf-component-manager/issues/104) + the official manifest docs. This also supersedes open-issue #3 above — with the correct syntax the pull is reliable (no `set-target` dance needed for the *dependency*). - - ❌ **Boot: `sleep_clock_icg_startup_init` aborts with `ESP_ERR_NO_MEM` (0x101) → reboot loop.** A KNOWN, OPEN ESP-IDF bug: **[esp-idf #18759 (IDFGH-17859)](https://github.com/espressif/esp-idf/issues/18759)** — on ESP32-P4 + PSRAM, this sleep-clock retention init runs unconditionally at a SECONDARY boot phase (before `app_main`, NOT gated by `CONFIG_PM_ENABLE`) and fails to allocate its REGDMA retention links when early internal DRAM is tight, which it is once esp_hosted's SDIO stack is pulled in (WiFi build only; the eth-only P4 has DRAM to spare). Espressif's guidance on the issue: reduce early internal-DRAM static usage. Bench findings (2026-07-03): `CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP` does NOT help (those buffers allocate after the boot init); `CONFIG_PM_POWER_DOWN_PERIPHERAL_IN_LIGHT_SLEEP=y` drops the ICG file but only moves the failure to the next retention alloc (PCR / int_wdt) then a `sleep_retention.c:914` assert; `.bss`/`.noinit` → PSRAM (`CONFIG_SPIRAM_ALLOW_{BSS,NOINIT}_SEG_EXTERNAL_MEMORY`) still fails — because `MALLOC_CAP_RETENTION` is a *specific reserved memory region*, not general DRAM, so freeing general DRAM doesn't reach it. This is the current blocker; MoonLight ran the same board on IDF 5.5 without it, so it's a 6.1-era regression. + - ❌ **Boot: `sleep_clock_icg_startup_init` aborts with `ESP_ERR_NO_MEM` (0x101) → reboot loop.** A KNOWN, OPEN ESP-IDF bug: **[esp-idf #18759 (IDFGH-17859)](https://github.com/espressif/esp-idf/issues/18759)** — on ESP32-P4 + PSRAM, this sleep-clock retention init runs unconditionally at a SECONDARY boot phase (before `app_main`, NOT gated by `CONFIG_PM_ENABLE`) and fails to allocate its REGDMA retention links when early internal DRAM is tight, which it is once esp_hosted's SDIO stack is pulled in (WiFi build only; the eth-only P4 has DRAM to spare). Espressif's guidance on the issue: reduce early internal-DRAM static usage. Bench findings (2026-07-03): `CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP` does NOT help (those buffers allocate after the boot init); `CONFIG_PM_POWER_DOWN_PERIPHERAL_IN_LIGHT_SLEEP=y` drops the ICG file but only moves the failure to the next retention alloc (PCR / int_wdt) then a `sleep_retention.c:914` assert; `.bss`/`.noinit` → PSRAM (`CONFIG_SPIRAM_ALLOW_{BSS,NOINIT}_SEG_EXTERNAL_MEMORY`) still fails — because `MALLOC_CAP_RETENTION` is a *specific reserved memory region*, not general DRAM, so freeing general DRAM doesn't reach it. MoonLight ran the same board on IDF 5.5 without it, so it's a 6.1-era regression. **RESOLVED for us 2026-08-19 by resolution path 1 below** (`CONFIG_PM_SLEEP_CLK_ICG_ENABLE=n` on IDF v6.1-rc1) — a workaround, since the allocation failure is untouched and #18759 remains open upstream. **Resolution paths (each independent):** - 1. **Upstream fix — SHIPPED on master, needs a v6.1 back-port or a cherry-pick (rechecked 2026-07-27).** #18759 is now **CLOSED (Resolution: Done)**. The fix is exactly the knob we wanted: commit `7d31b82d27d` ("change(esp_pm): add kconfig option for REGDMA sleep clock ICG", 2026-07-06) adds **`CONFIG_PM_SLEEP_CLK_ICG_ENABLE`** (`bool`, default `y`) — set it **`n`** and the crashing `sleep_clock_icg_startup_init` is not built/run. Since a 236-FPS LED controller never light-sleeps, ICG retention is dead weight for us, so `=n` costs nothing. **But it is on `origin/master` only — NOT back-ported to `release/v6.1`** (our pinned IDF `14f663f`/dev-5880 and the current `origin/release/v6.1` are the same commit; neither has it). So bumping the pinned v6.1 IDF does *not* get it yet. **Decision (2026-07-27): wait for the next v6.1 beta** that carries the back-port, rather than cherry-pick `7d31b82d27d` onto the pinned IDF — a manual local IDF patch is bespoke, drifts, and complicates the single-pinned-IDF story for a fix that is a clean one-line sdkconfig change once it's in the branch. When a new v6.1 beta publishes: check it contains `7d31b82d27d` (or the `CONFIG_PM_SLEEP_CLK_ICG_ENABLE` symbol), bump the pinned IDF, add `CONFIG_PM_SLEEP_CLK_ICG_ENABLE=n` to `sdkconfig.defaults.esp32p4-eth-wifi`, and bench-test the P4-WiFi boot. Once it boots, resume round-3 (the runtime SDIO re-init / C6 slave-reset work above). Watch `origin/release/v6.1` for the back-port. + 1. **Upstream fix — SHIPPED on master, needs a v6.1 back-port or a cherry-pick (rechecked 2026-07-27).** #18759 is now **CLOSED (Resolution: Done)**. The fix is exactly the knob we wanted: commit `7d31b82d27d` ("change(esp_pm): add kconfig option for REGDMA sleep clock ICG", 2026-07-06) adds **`CONFIG_PM_SLEEP_CLK_ICG_ENABLE`** (`bool`, default `y`) — set it **`n`** and the crashing `sleep_clock_icg_startup_init` is not built/run. Since a 236-FPS LED controller never light-sleeps, ICG retention is dead weight for us, so `=n` costs nothing. **But it is on `origin/master` only — NOT back-ported to `release/v6.1`** (our pinned IDF `14f663f`/dev-5880 and the current `origin/release/v6.1` are the same commit; neither has it). So bumping the pinned v6.1 IDF does *not* get it yet. **Decision (2026-07-27): wait for the next v6.1 beta** that carries the back-port, rather than cherry-pick `7d31b82d27d` onto the pinned IDF — a manual local IDF patch is bespoke, drifts, and complicates the single-pinned-IDF story for a fix that is a clean one-line sdkconfig change once it's in the branch. **DONE 2026-08-19 — that plan executed exactly as written.** v6.1-rc1 published carrying the option as `4b8e1e87106` (verified present in `components/esp_pm/Kconfig` at the pinned commit `44f0c59f7c8`); the IDF pin was bumped, `CONFIG_PM_SLEEP_CLK_ICG_ENABLE=n` added to `sdkconfig.defaults.esp32p4rev1-eth-wifi`, and the P4-WiFi boot bench-tested: it boots, associates (RSSI -52) and serves the UI. The variant is out of the installer's experimental set and ships as a normal firmware. Reported back on the upstream issue ([comment](https://github.com/espressif/esp-idf/issues/18759#issuecomment-5345660136)), which also told the original reporter the option had landed — nobody had announced it in the thread. **#18759 stays OPEN upstream** and we did not ask for it to be closed: the retention allocation still fails under early-DRAM pressure, the option merely means nothing requests that memory. Round 3 is now unblocked, and its remaining defect is throughput, not boot (see open issue 0 above). 2. **IDF 5.5 fallback — investigated 2026-07-03, NOT a cheap escape.** MoonLight ran this exact board on IDF **5.5** without the crash, so a 5.5 build of just this variant (everything else on 6.1) looked like a timeline-independent path. A bench attempt against IDF 5.5.4 found `src/platform/esp32/` has drifted to genuinely require IDF **6.x**: four distinct 5.5↔6.1 API breaks in `platform_esp32.cpp` / `platform_config.h` — (a) `RMT_LL_TX_CANDIDATES_PER_INST` renamed (5.5: `SOC_RMT_TX_CANDIDATES_PER_GROUP`); (b) `esp_eth_phy_ip101.h` moved (5.5: ctor lives in core `esp_eth_phy.h`, no standalone header); (c) `CHIP_ESP32S31` enum is 6.1-only; (d) `ETH_ESP32_EMAC_DEFAULT_CONFIG()` in 5.5 has out-of-declaration-order designated initializers — a C++ **hard error no compiler flag suppresses** (`-fpermissive` doesn't touch it). So a working 5.5 binary needs permanent `#if`-IDF-version compat branches in the platform layer (an EMAC-init back-port + a second IDF 5.5.4 in the CI matrix) — a real feature, not a throwaway. Only worth it if #18759 stalls long enough that a shippable P4-WiFi is needed sooner. 3. **Version-matched C6 slave reflash** (see round-3 item 1) — may change the boot memory picture, but is blocked behind the boot crash (host must boot to test the C6 handshake), so it only matters once 1 or 2 gets us to `app_main` stably. - **Round 4 — Parlio loopback self-test FIXED (2026-07-09), real long strip still to prove.** The Parlio loopback self-test now passes on P4 hardware at any grid size (verified on MM-P4, jumper GPIO 32↔33, at both 8×8 and 128×128). A real *long* WS2812 strip (not just the bench panel) is the remaining hardware proof. diff --git a/docs/backlog/power-functions-analysis-bottom-up.md b/docs/backlog/power-functions-analysis-bottom-up.md index 7ccd930f..b45e0f04 100644 --- a/docs/backlog/power-functions-analysis-bottom-up.md +++ b/docs/backlog/power-functions-analysis-bottom-up.md @@ -129,7 +129,7 @@ Also worth knowing when reading either codebase: mainline 0.16 replaced FastLED ## The candidate set (synthesis) -These families group functions by *algorithm*, which is how they were discovered. The build order is the five **phases** in [the top-down plan](power-functions-analysis-top-down.md#5-migration-plan-stage-1-and-example-effects-stage-2), which group the same functions by what lands in the repo together and name the families each phase carries; "what are we building next" is answered there, not here. +These families group functions by *algorithm*, which is how they were discovered. The build order is the five **phases** in [the top-down plan](power-functions-analysis-top-down.md#migration-plan), which group the same functions by what lands in the repo together and name the families each phase carries; "what are we building next" is answered there, not here. Merging in-repo demand, WLED's usage counts, and the canon's coverage-per-byte ranking — nine families, ~34 functions (family 9, Projection, was added on review; the gather group below came from the canon survey). Dimension-generic per the PO decision; every entry is integer/fixed-point; *(have)* = exists for compiled effects today, so the work is exposure + adoption, not invention. diff --git a/docs/backlog/power-functions-analysis-top-down.md b/docs/backlog/power-functions-analysis-top-down.md index 93a86680..c7953e7e 100644 --- a/docs/backlog/power-functions-analysis-top-down.md +++ b/docs/backlog/power-functions-analysis-top-down.md @@ -126,6 +126,8 @@ What the builtin surface needs from the engine, recorded for the livescripts wor Until the ABI lands, stages 1–2 proceed compiled-side; nothing here blocks on the engine. + + ## 5. Migration plan and example effects ✅ *(11 of 12 showcases built; VectorBalls landed 2026-08-07)* Order by leverage, cheapest risk first; every batch lands with its tests and the branch stays under ~100 files. **These five phases are the project's one numbering for this work** — the bottom-up document's nine *families* group functions by algorithm, while the phases below group them by what lands in the repo together, so each phase names the families it carries. diff --git a/docs/building.md b/docs/building.md index ce1043a7..18258c73 100644 --- a/docs/building.md +++ b/docs/building.md @@ -84,7 +84,7 @@ Every host needs [uv](https://docs.astral.sh/uv/), CMake 3.20+, and a C++20 comp The ESP32 target uses ESP-IDF directly, not the Arduino framework. -**Tested IDF version:** **v6.1-beta1** (commit `b1d13e9f`). CI builds against the `v6.1-beta1` Docker tag and local builds should match (clone command below). The why, the alternatives, and how to check for a newer one are in [ESP-IDF version](#esp-idf-version) below. +**Tested IDF version:** **v6.1-rc1** (commit `44f0c59f`). CI builds against the `v6.1-rc1` Docker tag and local builds should match (clone command below). The why, the alternatives, and how to check for a newer one are in [ESP-IDF version](#esp-idf-version) below. ### Prerequisites @@ -93,16 +93,16 @@ You need [uv](https://docs.astral.sh/uv/) (Python launcher), CMake 3.20+, and a **macOS / Linux:** ```sh -git clone --depth 1 --branch v6.1-beta1 https://github.com/espressif/esp-idf.git ~/esp/esp-idf +git clone --depth 1 --branch v6.1-rc1 https://github.com/espressif/esp-idf.git ~/esp/esp-idf ``` **Windows** (PowerShell — run once with admin to enable long paths if you haven't already): ```powershell # IDF and its tooling have deeply nested paths; without longpaths the clone -# trips MAX_PATH (260 chars) inside the v6.1-beta1 tree. +# trips MAX_PATH (260 chars) inside the v6.1-rc1 tree. git config --global core.longpaths true -git clone --depth 1 --branch v6.1-beta1 https://github.com/espressif/esp-idf.git "$env:USERPROFILE\esp\esp-idf" +git clone --depth 1 --branch v6.1-rc1 https://github.com/espressif/esp-idf.git "$env:USERPROFILE\esp\esp-idf" ``` Then run the one-time Python environment setup — either open MoonDeck (`uv run moondeck/moondeck.py`), go to the ESP32 tab, and click **Setup ESP-IDF**, or run it directly: @@ -173,9 +173,9 @@ After the driver installs and Windows finishes binding (a few seconds), the boar ### ESP-IDF version -**Pinned to `v6.1-beta1`** (commit `b1d13e9f`, a signed pre-release tag). `setup_esp_idf.py` holds the exact commit in `PINNED_IDF_VERSION`, warns loudly when the installed tree differs, and by default offers to check the pin out so a stray `git pull` or a fresh shallow clone landing on a newer commit converges back rather than silently building against the wrong tree (`--no-checkout` keeps it warn-only). Minimum is ESP-IDF v5.1 (C++20 needs GCC 12+); the project uses v6.x APIs (`esp_eth_phy_new_generic`, the component manager for mDNS, the modern RMT/parlio/LCD drivers) so v5.x would need adjustments. +**Pinned to `v6.1-rc1`** (commit `44f0c59f`, a signed pre-release tag). `setup_esp_idf.py` holds the exact commit in `PINNED_IDF_VERSION`, warns loudly when the installed tree differs, and by default offers to check the pin out so a stray `git pull` or a fresh shallow clone landing on a newer commit converges back rather than silently building against the wrong tree (`--no-checkout` keeps it warn-only). Minimum is ESP-IDF v5.1 (C++20 needs GCC 12+); the project uses v6.x APIs (`esp_eth_phy_new_generic`, the component manager for mDNS, the modern RMT/parlio/LCD drivers) so v5.x would need adjustments. -**Why a v6.1 pre-release and not a stable tag.** The v6.x line is: **v6.0 is the current stable** (GA 2026-02-27); **v6.1 is pre-release** (beta1 2026-06-24, RC1 and GA targeted late July 2026). We pin the `v6.1-beta1` *tag* (a fixed, signed pre-release, not the rolling `release/v6.1` branch) because it carries driver fixes for the newer SoCs (P4 parlio, RMT v2 on every chip) **and is on the earliest IDF line that carries the `esp32s31` preview target** — and because v6.0 vs v6.1 is a small delta. Riding the betas toward GA means breakage from the v6.1 delta surfaces incrementally, not all at once at the GA re-pin. The trade-off is honest: a pre-release gets **no support guarantee**, which is why the pin is a fixed tag, not a floating branch. The clean inflection point is **v6.1 GA**: re-pin to the `v6.1` tag then, which starts the 30-month support clock (see below). Each pin move (beta1 → RC → GA) is a deliberate re-test pass, not a routine pull. Tracked in [backlog](backlog/README.md). +**Why a v6.1 pre-release and not a stable tag.** The v6.x line is: **v6.0 is the current stable** (GA 2026-02-27); **v6.1 is pre-release** (beta1 2026-06-24, rc1 2026-08-14, GA to follow). We pin the `v6.1-rc1` *tag* (a fixed, signed pre-release, not the rolling `release/v6.1` branch) because it carries driver fixes for the newer SoCs (P4 parlio, RMT v2 on every chip) **and is on the earliest IDF line that carries the `esp32s31` preview target** — and because v6.0 vs v6.1 is a small delta. Riding the betas toward GA means breakage from the v6.1 delta surfaces incrementally, not all at once at the GA re-pin. The trade-off is honest: a pre-release gets **no support guarantee**, which is why the pin is a fixed tag, not a floating branch. The clean inflection point is **v6.1 GA**: re-pin to the `v6.1` tag then, which starts the 30-month support clock (see below). Each pin move (beta1 → RC → GA) is a deliberate re-test pass, not a routine pull. Tracked in [backlog](backlog/README.md). **v6.0 is the floor — don't depend on anything newer than it.** Because **v6.0 stable is our fallback** if the v6.1 line proves troublesome, the firmware and build tooling must stay buildable on v6.0. The rule is generic: **use no IDF API, component, Kconfig symbol, or tool that isn't present in v6.0.** A feature that exists only on the v6.1-dev branch (or arrives in a later minor) is off-limits until v6.0 is no longer the fallback. When adopting anything new from the IDF, confirm it shipped in v6.0 first (check the v6.0 docs / release notes, not `latest`); if it's v6.1-only, it waits. @@ -229,7 +229,7 @@ Tracked in [backlog](backlog/README.md). `build_esp32.py --firmware` selects one of the shipping variants. The key combines chip name + feature flags + (for SKU-sensitive chips) module. ("Firmware" here is the compiled binary; the physical product (deviceModel) is a separate concept — see [architecture.md § Firmware vs deviceModel vs board](architecture.md#firmware-vs-devicemodel-vs-board).) `build_esp32.py --help` lists the full set. -The canonical list is the **`FIRMWARES` dict** in [`moondeck/build/build_esp32.py`](../moondeck/build/build_esp32.py) — the single source of truth, carrying each variant's `chip`, sdkconfig `fragments`, `eth_only`, `ships`, and `description`. Its machine-readable projection is [`web-installer/firmwares.json`](../web-installer/firmwares.json) (generated by `generate_firmwares.py`, drift-guarded by `check_firmwares.py`), which the CI release matrix, the ESP Web Tools manifest loops, and MoonDeck all read — so the list lives in exactly one place. `esp32p4-eth-wifi` has `ships: false` (its C6-slave Kconfig isn't reproducible in CI yet), so it builds from the CLI but stays out of the release matrix. +The canonical list is the **`FIRMWARES` dict** in [`moondeck/build/build_esp32.py`](../moondeck/build/build_esp32.py) — the single source of truth, carrying each variant's `chip`, sdkconfig `fragments`, `eth_only`, `ships`, and `description`. Its machine-readable projection is [`web-installer/firmwares.json`](../web-installer/firmwares.json) (generated by `generate_firmwares.py`, drift-guarded by `check_firmwares.py`), which the CI release matrix, the ESP Web Tools manifest loops, and MoonDeck all read — so the list lives in exactly one place. `esp32p4rev1-eth-wifi` has `ships: false` (its C6-slave Kconfig isn't reproducible in CI yet), so it builds from the CLI but stays out of the release matrix. ESP-IDF v6.x has no `CONFIG_ESP_WIFI_ENABLED` switch (the symbol is forced on for WiFi-capable SoCs), so dropping WiFi at compile time happens via `EXCLUDE_COMPONENTS` plus `MM_NO_WIFI` (set when `MM_ETH_ONLY=1`, applied in `esp32/main/CMakeLists.txt`). The `esp32-eth` variant takes this path; the default `esp32` keeps both stacks compiled in and uses the runtime cascade in `NetworkModule` (Ethernet first, WiFi fallback when no PHY responds). diff --git "a/docs/history/plans/Plan-20260813 - MoonLive on a stack machine \342\200\224 the frame is where values live (shipped).md" "b/docs/history/plans/Plan-20260813 - MoonLive on a stack machine \342\200\224 the frame is where values live (shipped).md" index f14bcb22..4aaf5651 100644 --- "a/docs/history/plans/Plan-20260813 - MoonLive on a stack machine \342\200\224 the frame is where values live (shipped).md" +++ "b/docs/history/plans/Plan-20260813 - MoonLive on a stack machine \342\200\224 the frame is where values live (shipped).md" @@ -343,7 +343,7 @@ Each step is independently verifiable, and the branch stays green throughout: Steps 10 to 13, which replaced step 7, moved to their own plan once they grew into a language change rather than a refactor: [Plan-20260817 — MoonLive scripts are -classes](Plan-20260817%20-%20MoonLive%20scripts%20are%20classes.md). +classes](Plan-20260817%20-%20MoonLive%20scripts%20are%20classes%20(shipped).md). 8. ✅ **Bench: S3 and P4**, a scripted layout and a scripted effect, both with nested loops. Done on FOUR boards (S3, classic ESP32, P4, S31), scripted layout + effect, plasma and the heavier @@ -447,7 +447,7 @@ lowering serves every backend, one system-variable vocabulary serves every role, frame contract that blocked the whole thing is fixed and pinned. What a script LOOKS LIKE is the next question, and it continues in -[Plan-20260817 — MoonLive scripts are classes](Plan-20260817%20-%20MoonLive%20scripts%20are%20classes.md). +[Plan-20260817 — MoonLive scripts are classes](Plan-20260817%20-%20MoonLive%20scripts%20are%20classes%20(shipped).md). ## Then, separately diff --git a/docs/history/plans/Plan-20260817 - MoonLive scripts are classes.md b/docs/history/plans/Plan-20260817 - MoonLive scripts are classes (shipped).md similarity index 99% rename from docs/history/plans/Plan-20260817 - MoonLive scripts are classes.md rename to docs/history/plans/Plan-20260817 - MoonLive scripts are classes (shipped).md index e8d84fd2..a93ceb97 100644 --- a/docs/history/plans/Plan-20260817 - MoonLive scripts are classes.md +++ b/docs/history/plans/Plan-20260817 - MoonLive scripts are classes (shipped).md @@ -625,7 +625,7 @@ two. The one concrete use case is a text overlay in a showcase effect, and that on literals plus the numeric vocabulary already present. 10. ✅ **The editing loop, which is the thing people will actually see.** Done, in - [Plan-20260818](Plan-20260818%20-%20A%20file%20editor%20control%20and%20a%20filesystem%20change%20seam.md). + [Plan-20260818](Plan-20260818%20-%20A%20file%20editor%20control%20and%20a%20filesystem%20change%20seam%20(shipped).md). A card carries a file picker and an editor; typing and clicking away recompiles. Built EARLIER than this plan's "last step, against the finished shape" reasoning suggested, and diff --git a/docs/history/plans/Plan-20260818 - A file editor control and a filesystem change seam.md b/docs/history/plans/Plan-20260818 - A file editor control and a filesystem change seam (shipped).md similarity index 100% rename from docs/history/plans/Plan-20260818 - A file editor control and a filesystem change seam.md rename to docs/history/plans/Plan-20260818 - A file editor control and a filesystem change seam (shipped).md diff --git a/docs/logging-an-issue.md b/docs/logging-an-issue.md new file mode 100644 index 00000000..2bc84275 --- /dev/null +++ b/docs/logging-an-issue.md @@ -0,0 +1,111 @@ +# Log an issue + +Something not working? Tell us. You do not need to diagnose it — describe what +you saw, and hand us the state of the device it happened on. The **`{ }`** link +on every module card gives you that in one click, and it is the single most +useful thing you can attach. + +👉 **[Open an issue on GitHub](https://github.com/MoonModules/projectMM/issues)** + +Not sure it's a bug, or want to talk it through first? +**[Discord](https://discord.gg/TC8NSUSCdV)** is the right place for "is this +supposed to work like this?". + +--- + +## Checklist + +- Search the [existing issues](https://github.com/MoonModules/projectMM/issues) + first — it may already be reported, or already fixed +- Check the Firmware card: are you on the current version? +- For bugs, include the `{ }` output (below) +- For feature requests, describe the use case — no template needed + +--- + +## Bug report template + +### 1. Describe the problem + +``` +Steps to reproduce: +1. ... +2. ... + +Expected behavior: +[What should happen] + +Actual behavior: +[What happens instead] +``` + +### 2. Attach the `{ }` output + +⚠️ **This is the most useful diagnostic data.** It is a JSON snapshot of the +module's live state — its type, every control value, its status line and its +timing — exactly as it was when things went wrong. + +**How to get it:** + +1. Find the card that misbehaved — the effect that went dark, the driver that + isn't lighting, the network card that won't connect +2. Click the **`{ }`** link in its title row, next to the `?` +3. Select all in the new tab (Ctrl/Cmd+A), copy, and paste it into the issue + +**Which cards to include:** + +1. **Always: System.** Names the chip, firmware variant, build, SDK version and + the last boot reason — how we tell a board problem from a firmware one +2. **Always: Firmware.** The exact version and build you are running +3. **The affected card itself** — Effects, Drivers, Layouts, Network, whichever + one went wrong +4. **If it is about lights:** the Layer card, which carries the geometry +5. **If it is about pins or wiring:** the Drivers card + +Wrap each one in a fenced code block so the issue stays readable: + +```` +API output: + +- System +```text +{"name":"System","type":"SystemModule","controls":[{"name":"chip","value":"ESP32-S3"},...]} +``` + +- Effects +```text +{"name":"Effects","type":"Effects","children":[...]} +``` +```` + +### 3. Add a photo + +If it is something you can see — a fixture showing the wrong colours, a UI in a +state that looks wrong — a photo or screenshot says it faster than a paragraph. + +--- + +## If the device crashed or won't boot + +The `{ }` link needs a device that still serves its interface. When it doesn't, +tell us instead: + +- **What the LEDs did** — nothing at all, a brief flash, a repeating pattern +- **What changed** just before it started, even if it seems unrelated: a setting, + a script edit, a firmware update, a cable moved +- **Does it repeat?** Power-cycle it. A device that boots once and fails the next + time is a different problem from one that never boots +- **The serial boot log**, if you have the device on USB and can run a monitor. + If you can't, say so — we will not ask you to set up a toolchain to file a bug + +One more thing worth checking yourself: `bootReason` on the System card. `PANIC`, +`TASK_WDT` or `BROWNOUT` means the device crashed or lost power rather than +merely misbehaving, and saying so in the first line of the report saves a round +trip. + +--- + +## Feature requests + +Describe the use case, what you expect it to do, and what you have tried instead. +No template, no diagnostics — just tell us what you want to build. diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 3a52d0b4..c0a4560e 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,22 +1,23 @@ { - "commit": "762676fb", + "commit": "575b4472", "flash": { - "esp32": 1743056, - "esp32p4-eth": 1632720, - "esp32p4-eth-wifi": 1793760, - "esp32s3-n16r8": 1781728, + "esp32": 1752608, + "esp32p4rev1-eth": 1640176, + "esp32p4rev1-eth-wifi": 1928640, + "esp32s3-n16r8": 1791680, "esp32s3-n8r8": 1753232, - "esp32s31": 2054480, - "desktop": 1175560, + "esp32s31": 2070528, "esp32-16mb": 1714608, "esp32-eth": 1324816, "esp32-wrover": 1765504, - "qemu": 1318160 + "qemu": 1318160, + "esp32p4rev3-eth": 1643760, + "desktop": 1192520 }, "perf": { "desktop": { - "tick_us": 260, - "fps": 3846 + "tick_us": 157, + "fps": 6369 }, "esp32": { "tick_us": 2151, @@ -24,54 +25,54 @@ } }, "loc": { - "core": 19185, - "light": 24925, - "platform": 13466, - "ui": 6738, - "test": 43507, - "moondeck": 20949 + "core": 19369, + "light": 24985, + "platform": 13493, + "ui": 6805, + "test": 43746, + "moondeck": 21138 }, "comments": { "core": { - "lines": 7503, - "ratio": 0.424 + "lines": 7589, + "ratio": 0.425 }, "light": { - "lines": 9752, + "lines": 9779, "ratio": 0.432 }, "platform": { - "lines": 4776, - "ratio": 0.391 + "lines": 4798, + "ratio": 0.392 }, "ui": { - "lines": 1736, - "ratio": 0.274 + "lines": 1776, + "ratio": 0.277 }, "test": { - "lines": 7775, + "lines": 7846, "ratio": 0.206 }, "moondeck": { - "lines": 3377, + "lines": 3421, "ratio": 0.185 } }, "tests": { - "cases": 1405, + "cases": 1410, "scenarios": 23 }, "docs": { - "md_files": 180, - "md_lines": 25715, + "md_files": 182, + "md_lines": 26186, "plans_files": 93, - "backlog_lines": 3714, + "backlog_lines": 3817, "lessons_lines": 549, "claude_md_lines": 135 }, "complexity": { - "functions": 2586, - "over_threshold": 162, + "functions": 2588, + "over_threshold": 163, "worst_ccn": 108 } } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 426e7aa8..9dafebf5 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `762676fb`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** +Measured at `575b4472`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -8,59 +8,60 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | |---|---:| -| desktop | 1,148 KB (+0 KB) ⚠ | -| esp32 | 1,702 KB (+7 KB) ⚠ | +| desktop | 1,165 KB (+0 KB) ⚠ | +| esp32 | 1,712 KB (+3 KB) ⚠ | | esp32-16mb | 1,674 KB | | esp32-eth | 1,294 KB | | esp32-wrover | 1,724 KB | -| esp32p4-eth | 1,594 KB (+15 KB) ⚠ | -| esp32p4-eth-wifi | 1,752 KB | -| esp32s3-n16r8 | 1,740 KB (+7 KB) ⚠ | +| esp32p4rev1-eth | 1,602 KB (+1 KB) ⚠ | +| esp32p4rev1-eth-wifi | 1,883 KB | +| esp32p4rev3-eth | 1,605 KB | +| esp32s3-n16r8 | 1,750 KB (+3 KB) ⚠ | | esp32s3-n8r8 | 1,712 KB | -| esp32s31 | 2,006 KB (+7 KB) ⚠ | +| esp32s31 | 2,022 KB (+1 KB) ⚠ | | qemu | 1,287 KB | ## Render performance | Target | Tick | FPS | |---|---:|---:| -| desktop | 260 µs (+74 µs) ⚠ | 3,846 (−1,530) ⚠ | +| desktop | 157 µs (−24 µs) ✓ | 6,369 (+845) ✓ | | esp32 | 2,151 µs | 464 | ## Code | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 19,185 (+85) ⚠ | 7,503 | 42.4 % | -| light | 24,925 (+148) ⚠ | 9,752 | 43.2 % | -| platform | 13,466 | 4,776 | 39.1 % | -| ui | 6,738 (+270) ⚠ | 1,736 | 27.4 % | -| test | 43,507 (+315) ⚠ | 7,775 | 20.6 % (+0.1 %) ⚠ | -| moondeck | 20,949 | 3,377 | 18.5 % | +| core | 19,369 (+59) ⚠ | 7,589 | 42.5 % | +| light | 24,985 | 9,779 | 43.2 % | +| platform | 13,493 (+4) ⚠ | 4,798 | 39.2 % | +| ui | 6,805 (+24) ⚠ | 1,776 | 27.7 % (+0.1 %) ⚠ | +| test | 43,746 (+41) ⚠ | 7,846 | 20.6 % | +| moondeck | 21,138 (+81) ⚠ | 3,421 | 18.5 % (+0.1 %) ⚠ | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,405 (+15) ✓ | +| unit cases | 1,410 | | scenarios | 23 | ## Complexity | Metric | Value | |---|---:| -| functions | 2,586 (+14) ✓ | -| over threshold | 162 (+1) ⚠ | +| functions | 2,588 (+1) ✓ | +| over threshold | 163 (+1) ⚠ | | worst CCN | 108 | ## Documentation | Metric | Value | |---|---:| -| markdown files | 180 (+1) ⚠ | -| markdown lines | 25,715 (+254) ⚠ | -| plan files | 93 (+1) ⚠ | -| backlog lines | 3,714 (+2) ⚠ | +| markdown files | 182 (+2) ⚠ | +| markdown lines | 26,186 (+297) ⚠ | +| plan files | 93 | +| backlog lines | 3,817 (+14) ⚠ | | lessons lines | 549 | | CLAUDE.md lines | 135 | diff --git a/docs/moonmodules/core/ui.md b/docs/moonmodules/core/ui.md index a27dfe9d..e5c204cd 100644 --- a/docs/moonmodules/core/ui.md +++ b/docs/moonmodules/core/ui.md @@ -89,7 +89,7 @@ indentation. Nesting depth shows as progressively lighter backgrounds and a left ```text ┌─ card ──────────────────────────────────┐ -│ [name] [emoji] [timing · 🧠 mem] [enabled toggle] [✎ × ☰] [? help] │ +│ [name] [emoji] [timing · 🧠 mem] [enabled toggle] [✎ × ☰] [? help] [{ } api] │ │ [control rows — one per control] │ │ ┌─ child card ────────────────────────┐ │ │ │ … │ │ @@ -101,6 +101,9 @@ indentation. Nesting depth shows as progressively lighter backgrounds and a left - The parent's own controls render **above** its children; `+ add child` renders **below** them. Child cards live in a `.card-children` wrapper appended into the parent card's DOM node (not flat siblings); `renderModuleTree` recurses into the parent card, not into `main`. +- **`{ }`** opens `GET /api/modules/{name}` in a new tab — that one module's live JSON, for issue + reports (see [Log an issue](../../logging-an-issue.md)). On EVERY card, unlike `✎`/`×` (user-editable + children only) and `?` (types that have a doc page). - **Enabled toggle** in the right-hand action cluster mirrors `MoonModule::enabled()` — a styled `