Any decent monitoring setups for ESP32 deployments?

[Grant7]

Hi all,

Working on a small-ish fleet of ESP32s for an IoT setup and starting to feel the pain of not having proper observability in place, Serial prints and basic logs only go so far, especially once you've got a few dozen devices doing their thing,
Wondering if anyone here has tried integrating more structured monitoring tools with ESP32, even lightweight stuff, I'm thinking along the lines of metrics collection, maybe some sort of remote logging or even basic tracing if possible, Not expecting full-blown Prometheus + Grafana stacks on-device obviously, but curious if anyone’s managed to hook ESP32s into something more DevOps-y without going totally overkill.
I’ve come across https://gritt.io/, which looks like it’s geared toward observability for IoT devices, has anyone here tried it, or anything similar, with ESP32s,

Would be great to hear how others are approaching this, even rough hacks or lessons learned would be useful,

Cheers,

[Sprite]

I've done some stuff for my personal home automation stuff. Note that this all on a LAN; your setup would probably a bit more complex if you have machines that connect to servers on the Internet. What I basically did was have some stubs that take the logging data and log it via UDP to a rsyslogd-server; that gives me historic data for if one act up. I also have an OTA setup that checks an internal webserver on startup for newer firmware, including rollback to the previous version if the new one crashes before a certain point of execution. I also use the esp-idf feature to save coredumps to flash; on startup, I check if one exists and then update that, plus the crashed software version, to the OTA server. So if something goes wrong, I have the syslog that leads up to the event, plus a coredump that tells me the location and circumstances of a crash; up to now, that's been enough to find a fair few errors.

(Note that the code for this all is A. not the best quality and B. quite specific to my situation, so sorry, I don't think I want to share it.)

[thefury]

I liked ESP Insights when I played around with it a couple years ago.

Crash dump uploading & analysis, remote logging (errors only by default, logging all info would probably be excessive), and various metrics like WiFi RSSI, free memory, and other good stuff like that.

https://github.com/espressif/esp-insights

[DrMickeyLauer]

This is a very interesting topic for me. Implementation-wise it's almost holistic as it spans a lot of areas (coredump creation, logging, runtime time monitoring) and the question how to retrieve the generated information [push/pull, WiFi, BLE, etc.].