ESP32-S3 - Device reset loop and USB re-enumeration when trying to run esptool commands over USB serial in own app

[daniel-alsen]

Hi.

I'm building a secure mobile Android firmware updater app that programs ESP32-S3 devices over USB. The idea is to use this in manufacturing of my complete device for fast and secure handling and flashing of own firmware.

I'm using out of the box modules with the native USB-Serial-JTAG peripheral (no external USB-UART bridge), SLIP protocol over USB Serial and esptool-js v0.5.7 (JavaScript port of esptool.py).

It seems like I have to put the device into bootloader mode for the communications to work with my app. This is not needed when I use esptool, which makes the process more convenient and faster for flashing multiple devices.

Is there a way to put the device into bootloader mode automatically, or how does the esptool do it?

Which USB/serial state is the esp32-s3 device in when it's not in bootloader mode?
Which commands can it respond to then?

It would be great if you please can share some technical help here so I can get a more smooth connection process with the esp device.

>> Our connection approach and issue when not in bootloader mode

Our Android app connects to ESP32-S3 via USB-Serial-JTAG using:

USB Access: Android UsbManager API → USB device enumeration and permission
Serial Driver: usb-serial-for-android library → CDC-ACM interface handling
Protocol: SLIP-encoded packets over USB Bulk transfers
Commands: esptool.js SYNC command (opcode 0x08) to detect bootloader

The Problem: When device is NOT in bootloader mode, sending the SYNC packet causes immediate USB re-enumeration (device disconnects and reconnects with new ID), creating a permission loop. Resets?

Serial Port Configuration:

SET_LINE_CODING control transfer (USB request 0x20):
Baud: 115200, Data bits: 8, Stop bits: 1, Parity: None
SET_CONTROL_LINE_STATE control transfer (USB request 0x22):
DTR: false, RTS: false (strategy prevents reset triggers)

SYNC Command Transmission:

Opcode: 0x08 (ESP_SYNC)
Payload: 36 bytes (0x07 0x07 0x12 0x20 + 32 bytes of 0x55)
SLIP Encoding: Wrapped in 0xC0 delimiters, special chars escaped
USB Transfer: bulkTransfer() sends ~48-byte SLIP packet via Bulk OUT endpoint

Issue:
Re-enumeration Event (Scenario 1: Not in Bootloader)
50-200ms after SYNC packet: Device electrically disconnects from USB

Result:
UsbDeviceConnection invalidated
Device re-appears with new ID (deviceId: 43)
Permission lost (Android treats as new device)
Loop repeats: permission → connect → SYNC → re-enumerate → new ID...
No SLIP response received

--> Successful Connection (when in bootloader Mode)
Device held in bootloader (BOOT button during USB plug-in)
Same USB descriptors (VID/PID unchanged)
SYNC packet sent → No re-enumeration
ESP32 responds with 8 SLIP packets (~150ms total)
Chip ID read succeeds, flash operations work perfectly

Many thanks,
Daniel

[Sprite]

There's a little dance with the RTS/DTR lines that you can do to get the chip into bootloader mode. See the technical reference manual bits of the USB-serial-JTAG converter, it's documented there. When not in download mode, it depends on the firmware what the USB port looks like - can be a standard serial port when firmware keeps using USB-serial-JTAG, can be anything when it switches to USB-OTG.

[daniel-alsen]

Great. I'll try it. It only needs to work with the factory shipped firmware and that saves some hassle in the manufacturing line.

[daniel-alsen]

It works well.

Why is the device going into this "reset loop" when it's not in bootloader, delivered from the factory and connected via USB?

What is causing that?

[Sprite]

It works well.

Why is the device going into this "reset loop" when it's not in bootloader, delivered from the factory and connected via USB?

What is causing that?

It's a safety feature. If the ESP32 came up 'weird', in a mode where somehow the communication with the flash fails, we don't want it to just sit there even if the issue is transient. Hence there's a watchdog that hard-resets the chip if we haven't been able to load a program after a given time. Factory-fresh chips don't have a program and as such will demonstrate that behaviour.

[daniel-alsen]

Got it.

I connect my ESP32-S3 device directly to Windows, Mac and Android via USB.

It works well to connect and communicate with your guidance above.

Do I need a USB driver for any of these platforms? Anything available that improves the communications?

The USB connector is just connected directly to D+/D- on the chip. I have the CC pull-down resistors on CC1 and CC2 to indicate that it's a sink and want charging. I use a ESD diode on the data lines as well, for protection.

[Sprite]

No, sounds your setup is correct. You normally don't need a driver (for serial communication) as the ESP32 generally uses the standardized CDC-ACM protocol; if your OS can talk to it, you already have the correct driver.

[daniel-alsen]

I have two final issues that I have encountered:

- Device stuck if not reset via RTS last in write-flash sequence (reset button / pin won't put it in boot mode)

The final step of doing a hard reset via RTS as the last step in the flash sequence seems important. Otherwise, the device seems to be in a state when the code won't run. Even though I reset the device using the reset button / pin. What happens during this step?

- MD5 checksum fails

I tried to add the MD5 checksum validation in the flash sequence as well, but get the message that the Md5 fails. However, the checksum returned is the same after every flash, and the code works after RTS / reset. Is there any magic here to get the MD5 to match?

I saw some post mentioning "write_flash_status --non-volatile 0". Is this needed and when should it be done? Does the flash have to be erases for the MD5 to match?

Thanks for revealing some of the magic here

[Sprite]

Does the flash have to be erases for the MD5 to match?

Absolutely. If you write over already-written flash, the end result will be the logical and of the existing and new contents. You need to erase the flash to reset everything to 0xff.

[daniel-alsen]

Offcourse.

Here is the sequence I used:

Source device
- Erase flash
- Load code
- Extract full 4Mb flash from 0x00 into a firmware file

Target device
- Erase flash
- Write flash from 0x00
- Verify MD5 (start address 0x00, using raw byte array with firmware file)

Which gives me a MD5 mismatch. Perhaps some flags are written after the write flash and before the MD5 verification, or the file is not padded correctly? I will investigate. Any ideas?

Chip is ESP32-S3.