emac insufficient buffer size error on ethernet

[RonWebski]

When running ethernet examples on esp32 ethernet dev board, getting following error:

esp.emac: emac_esp32_transmit(231): insufficient TX buffer size[0m

what could be the cause, using latest ESP-IDF SDK.

[ondrej]

What example do you run? Anyway, it indicates you are trying to transmit more data than the hardware is able to process, i.e. you are reaching to limit of maximum Ethernet bandwidth of 100 Mbps. In reality it little bit less. You need to limit bandwidth. With default configuration of 10 Tx buffers, stable throughput is around 80 Mbps. If you observe it with iperf, you can experiment with bandwidth limit (-b option). You can also try to increase number of Tx buffers in Kconfig.

[RonWebski]

Thanks for the reply, Will try the bandwidth option. The example i tried was "basic" example, also tried iperf example, it also gives the same error after a while. I noticed in iperf, ip addresses are zero.

[ondrej]

That's strange, "insufficient buffer size error" should not occur in "basic" example! Could you please provide more information about your setup?
- What board do you use?
- What is your exact IDF version?
- Please share logs from "basic" example if insufficient buffer size error occurs.

[eiSouthBoy]

I'm getting the same error as well. According to the answer upstairs, the default tx size is 10, and the maximum UDP bandwidth that can be tested is 80Mbps. So if you want to test whether the bandwidth can reach 100Mpbs?

[ondrej]

With newer versions of iperf (not publicly available yet), and with suppressed log of "insufficient buffer size" error (now it's at debug level) we can get close to 95 Mbps which is theoretical maximum when we subtract all the Ethernet frame and IP packet overhead.

I'm also in process of extension of ETH documentation. Below is "work in progress" version:

Memory Considerations when Using Internal MAC

The internal MAC subsystem transfers data to and from the CPU domain via DMA using a linked list of descriptors. There are two types of descriptors: Transmit and Receive. Based on its type, a descriptor holds status information about the received or transmitted frame or provides controls for transmission. Each descriptor also contains pointers to the current data buffer and the next descriptor. As such, a single EMAC DMA descriptor has size of 32 bytes in DMA-capable memory.

The default configuration should cover most use cases. However, certain scenarios may require configuring the Ethernet DMA memory utilization to suit specific needs. Typical problems may arise in the following situations:

* Short and frequent frames dominate network traffic: If your network traffic primarily consists of very short and frequently transmitted/received frames, you may observe issues such as lower-than-expected throughput (despite the rated 100 Mbps) and missed frames during reception. On transmission, the socket send API may return errno equals to ENOMEM, accompanied by the insufficient TX buffer size message (if debug log level is enabled). This is because the default memory configuration is optimized for larger frames; CONFIG_ETH_DMA_BUFFER_SIZE is set to 512 bytes by default to ensure a better data buffer to descriptor size overhead ratio. The solution is to increase CONFIG_ETH_DMA_RX_BUFFER_NUM or CONFIG_ETH_DMA_TX_BUFFER_NUM. Additionally, consider decreasing CONFIG_ETH_DMA_BUFFER_SIZE to match the typical frame size in your network to maintain a reasonable memory footprint of the Ethernet driver.

* High throughput leads to buffer exhaustion: If the socket send API intermittently returns errno equals to ENOMEM, accompanied by the insufficient TX buffer size message (if debug log level is enabled), and the throughput is close to the rated 100 Mbps, this likely indicates nearing hardware limitations. In such case, the hardware cannot keep up with the transmission requests. The solution is to increase CONFIG_ETH_DMA_TX_BUFFER_NUM to buffer more frames and mitigate temporary peaks in transmission requests. However, this will not help if the requested traffic consistently exceeds the rated throughput. In such situations, the only solution is to limit the bandwidth by software means at the application level.