ESP32 with INM441 I2S microphone.

[ow_birds]

No, nonono. Don't get me started

No, please do! Explain how I should be processing this 24 bit sample.

[MicroController]

I don't think one "should" do it in any particular way. You can directly interpret the sample data as 32-bit values (-2^31...2^31-1) or right-shift/divide and then go on with the 24-bit data (-2^23...2^23-1), doesn't make a difference; use whatever is more convenient in your case. When you want 16 bits, you can right-shift the 32-bit data by 16, or by 8 to first get 24-bit values, then by 8 once more, or skip/discard the lower 16 bits in other ways; same result.
I.o.w., you define what value you want to use for "maximum amplitude", be it 2^31, 2^23, 2^15, or some other; and based on that you shift/divide the sample data as needed.

[ow_birds]

Essentially we agree, as I said, it's just an issue of magnitude. In my opinion, if you treat the 24 bit value (which it is, the spec says so) as a 32 bit value, you're seeing it 256x larger than it really is. But that's fine if you're always going to adjust it to whatever is usable. Peace.