ESP32-WROOM-32D high current consumption while asleep

[digitalstables]

Hello,
I have designed two boards , a base and a sheild.
The base board has an esp32 wroom 32D, it has a lora module and a rtc. it also has the cp2104 usb chip .
the shield has leds , 2 super capacitor 1.5F each. The shield has an ads1115 which is used to measure
the voltage from the super capacitors. the sheild also has the in219 ic to measure the current coming
out of the capacitors. The boards are powered by a solar panel that has a usb connector.

The problem is that when the esp32 goes to deep sleep, the voltage in the capacitors drops very rapidly.

A Few caveats, the usb chip is isolated by a header on the first board as well as a header on the shield. If the jumper
is not on the header, the usb chip does not get power even when the solar panel is connected to the usb port.

The Lora radio as well as all the i2c devices are powered via high side switches that cut power when not set high.
Right before going to sleep i set all gpios to input.

The voltage regulator has a 35uA Iq

I am including the schematics, any suggestions as to what to try or defects in the design will be very appreciated.