I gave the IDF source code a look and this definitely smells like a bug to me.
A quick search through gitlab issues didn't show anything similar, so I suggest you to open one.

I should especially note that I did not use any interrupt tuning whatsoever , i.e. never called any of those uart_intr_config, uart_set_rx_timeout, uart_enable_rx_intr etc.
Okay, I bit the bullet and made a dive into the confusing world of the aforementioned interrupt functions.
After reading the ...

Is it possible to receive a notification when the TX FIFO becomes empty?

There is a pending feature request about this.

If you need low-latency reaction to Rx, you must always call uart_read_bytes with length of 1.
See an ugly, but working, crutch in this post.
I tested it by connecting UART Rx to the same UART's Tx, and calling esp_timer_get_time both after uart_write_bytes and after the aforementioned crutch (when ...