Code: Select all
## Summary
On an ESP32-S3 product we get an intermittent **hard freeze of both cores at
once**, caught by the interrupt watchdog (INT_WDT) with **no core dump** (the
panic handler never runs because both CPUs are wedged). The freeze is tightly
correlated with **SPI master GDMA activity used for an LCD panel flush**, and it
**disappears completely** when the second core is disabled
(`CONFIG_FREERTOS_UNICORE=y`). We have spent significant time isolating it and
would appreciate guidance on whether this matches a known ESP32-S3 erratum and
whether there is a workaround that preserves both cores.
## Environment
- **Target:** ESP32-S3 (dual-core). *Chip revision: <please fill from `esptool.py chip_id` / boot log>*
- **ESP-IDF:** v5.5.2
- **Flash:** XMC XM25QH32B (JEDEC 0x204016), **DIO @ 40 MHz**
- **PSRAM:** **disabled** (`CONFIG_SPIRAM` not set)
- **Watchdogs:** `CONFIG_ESP_INT_WDT=y` (timeout 2000 ms), `CONFIG_ESP_TASK_WDT_EN=y`
## Relevant peripherals running concurrently
- **LCD:** ILI9488 480×320, driven over **SPI master (GDMA)** via `esp_lcd`
(`esp_lcd_panel_draw_bitmap`), LVGL 8.3.1. LVGL draw buffers are allocated in
**internal DMA-capable SRAM** (`heap_caps_malloc(..., MALLOC_CAP_DMA)`), *not*
PSRAM. Flush is asynchronous; completion via the `on_color_trans_done` ISR
(which is `IRAM_ATTR`).
- **Addressable LED strip:** 51× WS2812 driven by the **RMT peripheral** (tested
both with and without GDMA).
- **7-segment driver** on a second SPI host, **RS-485/Modbus** on UART1,
Wi-Fi (AP/STA).
The LCD and 7-seg SPI hosts can be swapped (SPI2 ↔ SPI3); swapping them did not
change the behaviour.
## Symptom
- **Both cores stop executing simultaneously.** Per-core idle "heartbeat"
timestamps (stamped from each CPU's idle hook into RTC RAM) both stop at the
same time.
- Reset reason is the **interrupt watchdog**; **no core dump** is produced.
- Occurs during **normal operation** while the LCD is being refreshed and the
LED strip / Modbus are active. Intermittent (minutes to hours).
## Diagnostic evidence (what changes the behaviour)
| # | Change | Result |
|---|--------|--------|
| 1 | Baseline: dual-core, LCD SPI-GDMA flush + RMT + Modbus active | **Freeze** (INT_WDT, both cores, no coredump) |
| 2 | `CONFIG_FREERTOS_UNICORE=y` (core 1 held in reset) | **No freeze** — stable for days. Proven repeatedly. |
| 3 | Dual-core, but **every application task pinned to core 0** (core 1 alive but runs only idle/tick/IPC) | **Freeze returns** |
| 4 | Earlier: application tasks split across cores (display tasks→core1, wifi/modbus→core0) | **Freeze** |
| 5 | Disable the LCD flush entirely (no SPI-GDMA traffic), keep everything else | **No freeze** |
| 6 | Move the LED strip from **GDMA-RMT** to **non-DMA RMT** (remove the 2nd GDMA channel) | Freeze got **more frequent** (added RMT-ISR / AHB load) |
| 7 | Software mutex serialising the LCD SPI-GDMA flush against the RMT-GDMA refresh (the two DMA channels never overlap) | **No effect** — still freezes |
| 8 | PSRAM: already **disabled** (draw buffers in internal SRAM) | N/A — see "ruled out" |
The key pair is **#2 vs #3**: disabling core 1 (unicore) fixes it, but merely
**pinning all tasks to core 0 while core 1 remains alive does not**. That points
at the second core's *presence* (its cache/bus activity from idle, the per-core
tick, IPC), not at any particular task, interacting with the SPI-GDMA flush.
## What we have ruled out
- **CACHE-126 ("Cache Hit Error During Cache Write-Backs").** With **PSRAM
disabled** there is no writable cached memory, so there are **no dcache
write-backs** (`esp_cache_msync` C2M is never invoked; the framebuffer is
internal SRAM read directly by GDMA). The freeze still occurs, so this is not
the write-back erratum.
- **Two concurrent GDMA channels.** A mutex that guarantees the LCD SPI-GDMA and
the RMT-GDMA never run at the same time (evidence #7) does not help, so a
simple two-channel GDMA collision is not the whole story — a single SPI-GDMA
flush plus a live second core is enough.
- **A specific task.** Evidence #3/#4 (task placement doesn't matter, only core 1
liveness does).
- **PSRAM contention** (it is off).
Our working hypothesis is contention between **SPI master GDMA** and the
**dual-core cache / instruction-fetch path** that only manifests when both CPUs
are active — but we cannot find a documented erratum or a Kconfig workaround, and
IRAM-placing the flush callbacks does not help because the vulnerable window is
the entire multi-millisecond background DMA transfer (during which arbitrary
flash-resident code runs on both cores), not the flush call itself.
## Secondary issue (may be related)
After such a freeze/watchdog reset, the **XMC flash is left in a state the ROM
bootloader cannot read**: the boot loops with repeated `invalid header: 0x…`
and `rst:0x7 (TG0WDT_SYS_RST)` (bootloader watchdog), and **only a full power
cycle recovers** — a clean `esp_restart()` never triggers this. So a warm/
watchdog reset does not reset the flash chip the way `esp_restart()` does.
`CONFIG_BOOTLOADER_FLASH_XMC_SUPPORT=y` and `CONFIG_SPI_FLASH_BROWNOUT_RESET_XMC=y`
are enabled but do not prevent it. This turns every freeze into a field failure
that needs manual power-cycling.
## Questions for Espressif
1. Is there a **known ESP32-S3 erratum** where **SPI-master GDMA** activity,
concurrent with **both CPU cores active**, can hang the system (INT_WDT,
both cores) — independent of PSRAM and independent of a second DMA channel?
2. If so, is there a **workaround that keeps both cores enabled** (e.g. a
required interrupt/cache configuration, a GDMA setting, an alignment or
memory-placement constraint for the SPI DMA buffers)?
3. Is there anything specific to **`esp_lcd` SPI + GDMA on dual-core S3** we
should configure differently (e.g. the flush buffer placement, transaction
queue depth, or interrupt priority/affinity)?
4. For the secondary issue: is there a supported way to **reset the (XMC) flash
chip on a warm/watchdog reset** so the device self-recovers without a power
cycle?
## What we can provide
We can prepare a **minimal reproducer** (LVGL flush loop on the ILI9488 over SPI-
GDMA + a background load on core 1) on request, plus the full `sdkconfig`,
`idf.py --version`, and exact chip revision. The behaviour is highly repeatable:
unicore = stable, dual-core = freezes.