SD_MMC file access time

copych
Posts: 8
Joined: Tue Mar 07, 2023 6:04 pm

SD_MMC file access time

Postby copych » Sun Jan 28, 2024 1:31 pm

  1. SD_MMC.open(filename)

Is there a way to speed up file open operation? I can imagine some kind of caching start sectors location or some sort of keeping indexed directory structure on the app's side or whatever. Or maybe some tuning is available (I'd really prefer to stay in Arduino for compatibility reasons).
I have a code that uses SD_MMC driver for accessing files on a microSD card. The app is time-critical. On the card there are folders and files, hundreds of them. The problem is access time. First files (I mean default sorting) in the folder are about 5ms till first read, but the 200th file would be already ~20ms to open, and so on, which is not acceptable for my goal.
ESP is only performing reading operations, but quite intensively.
So, please, is there a way in ESP32 Arduino to have lower level access to the filesystem or even to the SD, so I could use something instead of SD_MMC.open(filename) for random access? But I wouldn't want to write my custom fatfs )))
Please note, that this must be an ordinary FAT card, not a custom partitioned storage. It must contain files readable/writable on a PC.

lbernstone
Posts: 607
Joined: Mon Jul 22, 2019 3:20 pm

Re: SD_MMC file access time

Postby lbernstone » Thu Feb 01, 2024 5:20 pm

Only thing I can recommend would be to try using the SdFat library. It may do some additional caching of the file tables that would give you faster access.

copych
Posts: 8
Joined: Tue Mar 07, 2023 6:04 pm

Re: SD_MMC file access time

Postby copych » Mon Feb 12, 2024 7:55 am

Oh, thank you so much for the answer and for the link! Meanwhile I wrote a raw-reader class for the FAT32 SD, and now facing a problem with caching, cause data is quite heavy in terms of ESP32.
Please, advise how to make it better:
I use ESP32 / ESP32S3 boards with PSRAM.
When building an index of the directory with both long and short file names, I use std::vector (I need the names at least for some time, cause next I have to parse them and bind indexes to events, and several passes may be needed). And at some point I receive rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT), which I suppose is connected with heap allocation problems during vector growing (wrong?). At least, increasing stack or adding taskYIELD() don't help.
So how can I use PSRAM to allocate std::vector? Should I ps_malloc() the initial vector, and then it'd be growing in PSRAM? Or am I getting it wrong?

lbernstone
Posts: 607
Joined: Mon Jul 22, 2019 3:20 pm

Re: SD_MMC file access time

Postby lbernstone » Mon Feb 12, 2024 12:27 pm

https://github.com/lbernstone/psram_cus ... ocator.ino

That WDT reset is more likely due to slow SPI access blocking the cpu. Make sure you put some delay in between heavy periods of disk reading/writing. If you already have this separated as a task, then lower the priority of your task to use yield.

copych
Posts: 8
Joined: Tue Mar 07, 2023 6:04 pm

Re: SD_MMC file access time

Postby copych » Mon Feb 12, 2024 1:02 pm

Thank you so much.

copych
Posts: 8
Joined: Tue Mar 07, 2023 6:04 pm

Re: SD_MMC file access time

Postby copych » Mon Feb 12, 2024 1:36 pm

That WDT reset is more likely due to slow SPI access blocking the cpu. Make sure you put some delay in between heavy periods of disk reading/writing. If you already have this separated as a task, then lower the priority of your task to use yield.
I have tried to place delay(10) or even delay(100) inbetween intense readings, but no luck. Also, please, specify what SPI access do you mean: PSRAM or SD? Actually I thought that SD_MMC is not SPI, and PSRAM is not engaged as a heap in Arduino IDE, so I don't use it in this part of the app yet (please, forgive my ignorance).

lbernstone
Posts: 607
Joined: Mon Jul 22, 2019 3:20 pm

Re: SD_MMC file access time

Postby lbernstone » Mon Feb 12, 2024 3:19 pm

Ok, it is the sdio bumping the watchdog, not spi. sdio is black box, so you may just have to increase the watchdog timer.

Who is online

Users browsing this forum: No registered users and 24 guests