ESP32 Arduino CDC communication unreliable/dropouts on ESP32-S3

[toybuilder]

Hello. I've discovered a problem with errors and dropouts on CDC communication data transfers.
This simple demonstration code shows when data arrive out of order and drops out.
The time value should be steadily increasing, but where it first appears that it only dropped text that spans a few lines, you can see that a later timestamp value is sent before an earlier one.

I first discovered this potential problem while trying to debug errors in CSV data sent from the DUT, and commas and newlines would appear out of order.

You can readily replicate the problem with the following code:

Code: Select all

#include <Arduino.h>

 void setup() {
    Serial.begin(115200);
}

uint32_t counter = 0;
 void loop() {
    Serial.print(micros());
    Serial.print(",");
    Serial.print(counter++);
    Serial.print("\n");
}

This results in the time in micros and a sequence number. Here is a snippet showing the problem happening:

Code: Select all

4445713,110809
4445746,110810
4445789,110811
4445829,110812
4445876,110813
4445917,110814
4445959,110815
4445992,110816
4446026,110817
4446071,110818
4446110,110819
4446151,110820
4446195,110821
4446228,110822
4446270,110823
4446310,110824
4446352,110825
4446396,110826
4446437,110827
4446471,110828
4446513,110829
4446554,110830
4446590,110831
4446633,110832
4446667,110833
4446710,110834
4446752,110835
4446785,110836
4446833,110837
44468664448636,110882
4448670,110883
4448713,110884
4448747,110885
4448795,110886
4448828,110887
4448876,110888
4448917,110889
4448959,110890
4448992,110891
4449026,110892
4449072,110893
4449113,110894
4449154,110895
4449196,110896
4449237,110897

[toybuilder]

Hmm, as I am taking a wider view at this problem, it looks like it might be a Windows 10 / 11 issue.

https://www.reddit.com/r/embedded/comme ... _on_win11/

[RandomInternetGuy]

Of course it's sad that a PC of today can't keep up with a PC running Procomm on a '286 from forty years ago.

But serial is NEVER promised to be a lossless, error-detected and recovered media. If that PC has spam to deliver or malware to run or whatever it is what Windows system might prioririze over receiving data, it's not like you'll ever get another chance at it.

It's up to you to configure handshaking, error detection, correction, retry, and all that annoying other stuff that differentiates reliable network-quality communication from just shouting out the backdoor. That might be SL/IP, PPP, XModem, Kermit, or whatever. For just shouting out "Whee, I'm at __LINE__ of __FILE__" every once in a while, serial is fine, if babies are going to die if there's a missed byte, it's certainly not fine. For most of us, deciding the point in the spectrum between those two points and solving for it is our challenge.

Now the doubly sad part is that USB itself IS such a protocol, but when it's shoved into existing frameworks, things like the return value of the number bytes written (I couldn't fake shock if Arduino simply discards this, as I hold it in lower institutional value than even Windows...) tend to get lost. The USB/Serial drivers tend to take a super reliable USB bytstream and then connect it at the bottom level of some of the oldest and most groady drivers in many operating systems, so it's not shocking that it's lost internally by the OS *after* all the fancy error detection/retry of USB is done and it's then dumped into data structures meant for 8250's from 1981.

Since you're on an S3, if you have enough self-respect to actually care about your data, you can bypass rings and rings of age on these trees and "just" open the USB bulk channel (https://www.beyondlogic.org/usbnutshell/usb4.shtml) that gives you "Error detection via CRC, with guarantee of delivery." Sure, you have to set up host mode (https://docs.espressif.com/projects/esp ... _host.html) and do something smarter than a serial port on the host PC side (hint: libusb), but poof, error detected, corrected, and sequenced frames and packets show up in user space.

Quit treating USB ports like a 1960s UART, and life gets a LOT better.