ESP32-C6 low-latency audio decoding

[kilobyte_ch]

Hello

I'm trying to stream encoded audio to a ESP32-C6, decode it and play it back via I2S.

Currently, I use the OPUS codec in https://github.com/espressif/esp-adf-li ... udio_codec for decoding.
Parameters are Stereo, 48kHz and 5ms samples. The samples are encoded on a x86 machine with libopus.

Unfortunately, the ESP32-C6 struggles with this task. The decoder task, which runs esp_opus_dec_decode(), is taking up nearly all cycles.
It is so bad that the TWDT triggers regularly, which slows down the system even more due to printing to serial.

Code: Select all

decoder         131214982               88%
websocket_clien 4967571         3%
IDLE            4259361         2%
tiT             1947887         1%
esp_timer       267326          <1%
wifi            3709050         2%
soundcard       1212308         <1%
sys_evt         1287            <1%
Tmr Svc         8               <1%

Code: Select all

E (96837) task_wdt: Task watchdog got triggered. The following tasks/users did not reset the watchdog in time:
E (96837) task_wdt:  - IDLE (CPU 0)
E (96837) task_wdt: Tasks currently running:
E (96837) task_wdt: CPU 0: decoder
E (96837) task_wdt: Print CPU 0 (current core) registers
Core  0 register dump:
MEPC    : 0x4201584e  RA      : 0x4201709c  SP      : 0x4084cee0  GP      : 0x4081b924  
--- 0x4201584e: deemphasis at celt_decoder.c.obj:?
--- 0x4201709c: celt_decode_with_ec at ??:?
TP      : 0x4084d920  T0      : 0x0525e995  T1      : 0x00006ccd  T2      : 0xe6530b7e  
S0/FP   : 0x4084cf30  S1      : 0x4084d720  A0      : 0x40851cac  A1      : 0xffc08737  
A2      : 0x000003c0  A3      : 0x00000002  A4      : 0x00000001  A5      : 0x002f85ec  
A6      : 0x40850064  A7      : 0x00000000  S2      : 0x000000f0  S3      : 0x00000001  
S4      : 0x420d28ac  S5      : 0x00000054  S6      : 0x4084d658  S7      : 0x4085448c  
S8      : 0x40854534  S9      : 0x4084d658  S10     : 0x4084da50  S11     : 0x408544e0  
T3      : 0x40853e8c  T4      : 0x0b0c0b0f  T5      : 0x00001000  T6      : 0x00008000  
MSTATUS : 0x00001889  MTVEC   : 0x40800001  MCAUSE  : 0xdeadc0de  MTVAL   : 0xdeadc0de  
--- 0x40800001: _vector_table at /home/user/esp/esp-idf/components/riscv/vectors_intc.S:54
MHARTID : 0x00000000  
Please enable CONFIG_ESP_SYSTEM_USE_FRAME_POINTER option to have a full backtrace.

Are there any ways to optimize this? The network protocol is already optimized quite a bit.

The TWDT messages are gone when running the application on a ESP32-S3.
Unfortunately the ESP32-S3 is not suitable for my application due too big light-sleep current.

The reason for OPUS till now is that is allows for easy synchronization between multiple ESP32 due to the short 5ms samples.

[MicroController]

The "Low Complexity Communication Codec" (LC3) may be what you need: It is made for streaming applications, you can choose from 2.5/5ms ("LC3plus") or 7.5/10ms frame duration, and it is designed to be computationally "low" complexity. (At a glance, the implementation of the codec looks like a trimmed-down version of Opus.)
Depending on who you ask, audio quality is "better" (Fraunhofer Institut, inventors of the codec) or a bit worse than Opus at the same bitrate.
On the PC side, you'd use e.g. liblc3.

[kilobyte_ch]

The "Low Complexity Communication Codec" (LC3) may be what you need: You can chose from 2.5 ("LC3plus")/7.5 to 10ms frame duration and it is designed to be computationally "low" complexity.

Interesting idea! I have now tried LC3 in my setup on ESP32-C6. Unfortunately, I encounter similar problems.

The ESP32-C6 seems unable to keep up decoding 1 channel 48kHz in LC3. 10ms frame duration.
Used liblc3 on x86 as encoder and esp_audio_codec as decoder on ESP32-C6.

It might be that the esp_audio_codec doesn't support all LC3 features, as it is limited to 7.5ms and 10ms frame duration.

Code: Select all

E (50907) task_wdt: Task watchdog got triggered. The following tasks/users did not reset the watchdog in time:
E (50907) task_wdt:  - IDLE (CPU 0)
E (50907) task_wdt: Tasks currently running:
E (50907) task_wdt: CPU 0: decoder
E (50907) task_wdt: Print CPU 0 (current core) backtrace
esp_backtrace_print: Print CPU 0 (current core) registers
Core  0 register dump:
MEPC    : 0x4201a230  RA      : 0x42018b00  SP      : 0x4084cb40  GP      : 0x4081ac74  
--- 0x4201a230: lc3_norm_l at ??:?
--- 0x42018b00: arithmetic_dec at ??:?
TP      : 0x4084cfe0  T0      : 0x00000004  T1      : 0xffffb000  T2      : 0x80690000  
S0/FP   : 0x00002060  S1      : 0x0000000d  A0      : 0x00075000  A1      : 0x00000640  
A2      : 0x4084e444  A3      : 0x00000000  A4      : 0x4084e444  A5      : 0x00000000  
A6      : 0x00000080  A7      : 0xffffd000  S2      : 0x00000027  S3      : 0x4085acbc  
S4      : 0x4084ced8  S5      : 0x40834208  S6      : 0x00000000  S7      : 0x43f00000  
S8      : 0x40340000  S9      : 0x00000000  S10     : 0x40834208  S11     : 0x4084de04  
T3      : 0x00005000  T4      : 0x00003000  T5      : 0x00000008  T6      : 0x00000000  
MSTATUS : 0x00001889  MTVEC   : 0x40800001  MCAUSE  : 0xdeadc0de  MTVAL   : 0xdeadc0de  
--- 0x40800001: _vector_table at /home/user/esp/esp-idf/components/riscv/vectors_intc.S:54
MHARTID : 0x00000000  

This also kind of agrees with the performance numbers published by Espressif for esp_audio_codec
https://github.com/espressif/esp-adf-li ... #decoder-1

[MicroController]

Interesting result.
So much for the "low complexity"
Seeing Espressif's figures, I guess they use the fixed-point version of Opus; while LC3 may not even have a fixed-point implementation (yet).

Seems that someone "forgot" to check in the sources for the adf-libs unfortunately, so hard to tell how optimized each codec implementation actually is.

It might be that a 160MHz RISC-V single-core without an FPU just isn't quite up to the task - but if everything works except that the idle task may complain from time to time... just turn off the TWDT and see how it goes. (Btw, make sure you didn't compile for debug ("-Og")...)

[kilobyte_ch]

Oh no, after playing around a bit more (and switching to raw PCM streaming for testing) I realized that the real problem is in my application (decoder task).

After fixing this, the ESP32-C6 happily decodes OPUS 48kHz with ~60% time consumption in the RTOS and thus a very happy TWDT

I guess the problem didn't occur on ESP32-S3 as it is dual-core and thus happily did a lot more RTOS context switches.