ESP32-S3 GPIO20 remove startup glitch

[yohnsee]

Dear forum,
I designed a custom PCB board for the ESP32 S3 Wroom-1U module, and found an annoying behavior during testing. I configured the GPIO20 (USB D+) as a normal output pin in SETUP, but during chip boot cycle it gets pulled high for about 170ms. Do you know any method to disable this pull-up, as it is a serious problem for my circuit? As a last case scenario I can reroute the PCB and use other pin to control the output, but first I would like to get to the bottom of this.
So far I tried:
burnt DIS_USB_OTG efuse,
DIS_USB_JTAG
USB_PHY_SEL
DIS_USB_OTG_DOWNLOAD_MODE
None of the above helped.
Also tried to redefine #define USBPHY_DM_NUM 19 to an other pin in the first line in my .ino file, hoping it will reroute the GPIO matrix accordingly, but this didn't help either.