Page 1 of 1

SPIFFS fast enough for a one second data logger?

Posted: Mon Mar 19, 2018 6:50 pm
by rin67630
SPIFFS or SD-Card?

I am planning a data logger with an ESP32 WROOM.
It should store one byte every second in a file. At the end of the day, a new file will be created. The file will be retrieved by FTP, then erased.

The available SPIFF memory should be enough for the job, but I am woried about the time performance.
I intend to store data in a 60 byte RAM array first and update the SPIFF file every minute.
But will I be able to finish the SPIFF update task within ~ 800mS?

If not, will a SD-Card on a separate shield be faster?

Thank you for your advice.

Re: SPIFFS fast enough for a one second data logger?

Posted: Mon Mar 19, 2018 10:50 pm
by Archibald
I've just tried writing 60 bytes to flash and timed it using the micros() function (before and after). It took 212μs (including some time taken by the 'micros' function calls, but excluding opening of file). I wonder if it may sometimes take longer, such as if the filing system sometimes needs to rewrite a sector.

Re: SPIFFS fast enough for a one second data logger?

Posted: Tue Mar 20, 2018 2:32 am
by ESP_igrr
Both SPIFFS and SD card will have higher write latency as the disk fills up. For SPIFFS, see this issue: https://github.com/pellepl/spiffs/issues/202. For SD cards, from memory we've seen delays as long as 0.5 seconds for erase/write operations on some cards, after writing a few megabytes of data. So make sure you have a sufficiently large buffer.

Re: SPIFFS fast enough for a one second data logger?

Posted: Tue Mar 20, 2018 3:51 am
by ESP_Angus
To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)

Re: SPIFFS fast enough for a one second data logger?

Posted: Tue Mar 20, 2018 10:44 pm
by rin67630
ESP_Angus wrote:To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.
Ouch, that is absolutely low-level fiddling!
Out of my reach, and out of reach of 99% of the people here, isn't it?

Re: SPIFFS fast enough for a one second data logger?

Posted: Wed Mar 21, 2018 3:48 am
by ESP_igrr
That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.

Re: SPIFFS fast enough for a one second data logger?

Posted: Wed Mar 21, 2018 8:46 am
by rin67630
ESP_igrr wrote:That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.
But you want to be able to read the results as a regular file e.g. to send it via FTP. So that's not so straight forward as you pretend.

Re: SPIFFS fast enough for a one second data logger?

Posted: Thu Mar 22, 2018 9:21 am
by ESP_igrr
Ah i see, I missed the FTP requirement. In that case, yes, it's going to be a bit more work to keep track of the beginning and the end of the ring buffer.

Re: SPIFFS fast enough for a one second data logger?

Posted: Tue Sep 17, 2019 6:18 am
by akbarhash
Hi ESP_Angus and ESP_igrr.
Sorry to resurrect this thread, if possible could a simple example be shared of the correct way to initialize and do serial write to the internal flash device using the library. A simple example like writing a small string and reading it back from flash memory would do. SPIFFS and NVS have a lot of overhead and is not suitable for the project I am working on. The spi flash api reference is a bit mixed up and not clear. Thanks.
ESP_igrr wrote: That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.
ESP_Angus wrote:
Tue Mar 20, 2018 3:51 am
To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)