You are correct and I don't think there's a way to do this without a lookup table. Per esp-idf/components/hal/esp32/include/hal/gpio_ll.h :

// the address of esp32's IO_MUX_GPIOx_REGs are not incremented as the gpio number increments
(address are out of order)
extern const uint8_t GPIO_PIN_MUX ...

Hello folks,

I'm trying to implement my own baremetal solution to setup the GPIO pins on ESP32, but unfortunately I've come up across a confusion when reading the SoC documentation.

I want to implement a generic function to get the IO_MUX_GPIOX_REG address, in order to properly control it ...