Consistent inconsistencies with f_read performance
Posted: Thu May 08, 2025 9:47 pm
Hey everyone!
In a recent proyect ive found myself having to really optimize read speeds from an sdcard and have encountered a weird behaveour I just cant fix or even explain.
Introducing the issue, im trying to reproduce GIFs directly from an SD card. So, using the SDMMC driver Ive mounted the card though 4bit SDIO at 35MHz. To avoid having to deal with the internal LZW compression of gif files, ive created a custom binary format that contains the LZW data straight and uncompressed for each frame. The important thing here is that we are working with big, continuous blocks of data to read directly. no kind of seek operation needed, all the frames are stored one after the other. The FILE interface was too slow, clocking in at around 60ms for a 72KB (roughly the size of one frames data) read. Reading straight continuous sectors it clocks barely over 5ms, but this does not consider any fragmentation that might ocurr after using the card for a while.
Context out of the way, im currently trying FatFS though the ff.h driver. Acording to the doc it has way less overhead than FILE and could help me bring down those 10-20ms I need. And here is where the funny behaveour starts. After opening the file heres the sample code Im using:
And here are the results:
First i thought this could be about clusters, that they may be fragmented, even though the card was just formated. Or maybe an internal cache of the card is messing with the results. To test it I bumped the buffer size to hold 3 frames at onece (no PSRAM involved) and tried reading in batches of 3. Here are the results:
Going by the result where we are reading chunks of 3 frames at a time, it doesnt look like the file is fragmented and so no seek operation should be necesary. But just to be sure, ive also tested the case where, after each frame i f_lseek back to the first frame just to see what effect a seek operation has. Heres what happened:
Ive checked the sdcard data itself and verified the file is not fragmented. We also know that theres something weird about the consistency of the timings, and how for example reading frame nº2 takes 40ms in the first example as its in an even read but takes 18ms to read it along side frames 1 and 3. Ive also tried using normal malloc buffers and allinged heap_caps_malloc buffers just in case. Ive also tried the FATFS_USE_FASTSEEK option in menuconfig as the file is mounted in read only mode, but times remain being wack as in the examples avobe. Also, i have 2 files open at once, as Im iterating a directory to find the files. Just in case Ive tried closing it but this did not improve the situation. Heres mi SDIO config just in case Ive messed something up:
I have no clue what is truly going on here or what is causing it, and havent have much luck finding information or other documented cases of this happening.
What do you think of this? have you encountered anything similar?
In a recent proyect ive found myself having to really optimize read speeds from an sdcard and have encountered a weird behaveour I just cant fix or even explain.
Introducing the issue, im trying to reproduce GIFs directly from an SD card. So, using the SDMMC driver Ive mounted the card though 4bit SDIO at 35MHz. To avoid having to deal with the internal LZW compression of gif files, ive created a custom binary format that contains the LZW data straight and uncompressed for each frame. The important thing here is that we are working with big, continuous blocks of data to read directly. no kind of seek operation needed, all the frames are stored one after the other. The FILE interface was too slow, clocking in at around 60ms for a 72KB (roughly the size of one frames data) read. Reading straight continuous sectors it clocks barely over 5ms, but this does not consider any fragmentation that might ocurr after using the card for a while.
Context out of the way, im currently trying FatFS though the ff.h driver. Acording to the doc it has way less overhead than FILE and could help me bring down those 10-20ms I need. And here is where the funny behaveour starts. After opening the file heres the sample code Im using:
Code: Select all
for(int i=0; i<frame_count; i++){
start_time = esp_timer_get_time();
result = f_read(&file, input_buffer, (width * height * sizeof(uint8_t)) + 2, &read_bytes);
end_time = esp_timer_get_time();
if(result != FR_OK || read_bytes != (width * height * sizeof(uint8_t)) + 2){
printf("ERROR: f_read error :(\n");
break;
}
printf("FRAME NUMBER %d\n", i);
printf(" > Done in %f ms!\n", (end_time - start_time) / 1000.0);
reset_watchdog();
}
All throughout the 61 frames my example file has, it consistently takes ~42ms for even frames and 6-7ms for odd frames. Heres where the rant begins, as i tell the things ive tried to understand the issue and how little sense they make.FRAME NUMBER 0
> Done in 42.086000 ms!
FRAME NUMBER 1
> Done in 6.859000 ms!
FRAME NUMBER 2
> Done in 42.310000 ms!
FRAME NUMBER 3
> Done in 6.586000 ms!
FRAME NUMBER 4
> Done in 42.336000 ms!
First i thought this could be about clusters, that they may be fragmented, even though the card was just formated. Or maybe an internal cache of the card is messing with the results. To test it I bumped the buffer size to hold 3 frames at onece (no PSRAM involved) and tried reading in batches of 3. Here are the results:
It consistently maintains either the good or bad performance for the whole 3 frames, just like in the first example, but with times tripled. Seeing this I had the dumb idea of reading just one frame in even (slow) reads and 3 in odd (fast) reads, and to further mess with my current understanding of the issue, it worked:FRAME NUMBER 0 to 2
> Done in 159.425000 ms!
FRAME NUMBER 3 to 5
> Done in 18.807000 ms!
FRAME NUMBER 6 to 8
> Done in 159.398000 ms!
FRAME NUMBER 9 to 11
> Done in 19.627000 ms!
FRAME NUMBER 12 to 14
> Done in 160.346000 ms!
Going by how dumb these results are, Ive tried to fetch the frame data in two reads, first a 1 byte read to "consume the slow read" and then the big read operation for the majority of the data. In this case it behaves like the first example, taking ~40ms for even reads and 6-7ms for odd reads. The initial, small read operation is not affecting the result in any way, shape or form.FRAME NUMBER 0
> Done in 42.411000 ms!
FRAME NUMBER 1 to 3
> Done in 18.804000 ms!
FRAME NUMBER 4
> Done in 42.387000 ms!
FRAME NUMBER 5 to 7
> Done in 18.807000 ms!
FRAME NUMBER 8
> Done in 42.419000 ms!
Going by the result where we are reading chunks of 3 frames at a time, it doesnt look like the file is fragmented and so no seek operation should be necesary. But just to be sure, ive also tested the case where, after each frame i f_lseek back to the first frame just to see what effect a seek operation has. Heres what happened:
Now all frames are slowed down consistently. This makes me suspect a hidden internal seek operation is the cause, but the thing I still cant wrap my head arround is how this happens, what triggers it or if anything can be done to prevent it.FRAME NUMBER 0
> Done in 42.434000 ms!
FRAME NUMBER 1
> Done in 42.405000 ms!
FRAME NUMBER 2
> Done in 42.411000 ms!
Ive checked the sdcard data itself and verified the file is not fragmented. We also know that theres something weird about the consistency of the timings, and how for example reading frame nº2 takes 40ms in the first example as its in an even read but takes 18ms to read it along side frames 1 and 3. Ive also tried using normal malloc buffers and allinged heap_caps_malloc buffers just in case. Ive also tried the FATFS_USE_FASTSEEK option in menuconfig as the file is mounted in read only mode, but times remain being wack as in the examples avobe. Also, i have 2 files open at once, as Im iterating a directory to find the files. Just in case Ive tried closing it but this did not improve the situation. Heres mi SDIO config just in case Ive messed something up:
Code: Select all
esp_vfs_fat_sdmmc_mount_config_t mount_config = {
.format_if_mount_failed = false,
.max_files = 5,
.allocation_unit_size = 16 * 1024,
};
sdmmc_host_t host = SDMMC_HOST_DEFAULT();
host.slot = SDMMC_HOST_SLOT_0;
host.flags = SDMMC_HOST_FLAG_4BIT;
host.max_freq_khz = 35000;
sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
slot_config.width = 4;
slot_config.clk = PIN_CLK;
slot_config.cmd = PIN_CMD;
slot_config.d0 = PIN_DAT_0;
slot_config.d1 = PIN_DAT_1;
slot_config.d2 = PIN_DAT_2;
slot_config.d3 = PIN_DAT_3;
slot_config.flags |= SDMMC_SLOT_FLAG_INTERNAL_PULLUP;
What do you think of this? have you encountered anything similar?