I have a bit of exprience using the S3's SIMD instructions so may be able to help :) They are documented in the S3's TRM and also referred to as "PIE" instructions if you want to search for resources online.
(Using the SIMD, mixing 8 16-bit stereo samples down to 8 mono samples, for example ...

Wow! Impressive work!

Not a signal processing guy, but:
Can you use a more directional sound pickup (->beamforming using 2 or more microphones)? This can reduce noise significantly, see e.g. Benn Jordan's video on 'acoustic spying'. Or do you need omni-directionality?

Btw: If you run into ...

Hello!
I need some help to improve audio vocal quality and reduce the background noise in the captured audio.
I am working on a project with an i2s mems microphone and speaker, with two esp32-s3 devkits. I am working on PlatformIO using the Arduino framework.
My mic is configured to capture audio ...