I didn't make any more headway into this.

I tried digging into the esp-idf and I got as far as the checks that no subsystem was holding a radio lock before entering sleep mode.
I couldn't find the sleep mode implementation. I was working off of commit 11b444b of master

So I've been messing around with the modem sleep for ble, and taking measurements with a little test application where the my phone connects to the esp32 and every few seconds the esp sends a notification to the phone.

I've set my clock rate to 80MHz, freeRTOS to tickless idle support, and freeRTOS ...