SPIFFS fast enough for a one second data logger?

rin67630
Posts: 99
Joined: Sun Mar 11, 2018 5:13 pm

SPIFFS fast enough for a one second data logger?

Postby rin67630 » Mon Mar 19, 2018 6:50 pm

SPIFFS or SD-Card?

I am planning a data logger with an ESP32 WROOM.
It should store one byte every second in a file. At the end of the day, a new file will be created. The file will be retrieved by FTP, then erased.

The available SPIFF memory should be enough for the job, but I am woried about the time performance.
I intend to store data in a 60 byte RAM array first and update the SPIFF file every minute.
But will I be able to finish the SPIFF update task within ~ 800mS?

If not, will a SD-Card on a separate shield be faster?

Thank you for your advice.

Archibald
Posts: 110
Joined: Mon Mar 05, 2018 12:44 am

Re: SPIFFS fast enough for a one second data logger?

Postby Archibald » Mon Mar 19, 2018 10:50 pm

I've just tried writing 60 bytes to flash and timed it using the micros() function (before and after). It took 212μs (including some time taken by the 'micros' function calls, but excluding opening of file). I wonder if it may sometimes take longer, such as if the filing system sometimes needs to rewrite a sector.

ESP_igrr
Posts: 2067
Joined: Tue Dec 01, 2015 8:37 am

Re: SPIFFS fast enough for a one second data logger?

Postby ESP_igrr » Tue Mar 20, 2018 2:32 am

Both SPIFFS and SD card will have higher write latency as the disk fills up. For SPIFFS, see this issue: https://github.com/pellepl/spiffs/issues/202. For SD cards, from memory we've seen delays as long as 0.5 seconds for erase/write operations on some cards, after writing a few megabytes of data. So make sure you have a sufficiently large buffer.

ESP_Angus
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: SPIFFS fast enough for a one second data logger?

Postby ESP_Angus » Tue Mar 20, 2018 3:51 am

To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)

rin67630
Posts: 99
Joined: Sun Mar 11, 2018 5:13 pm

Re: SPIFFS fast enough for a one second data logger?

Postby rin67630 » Tue Mar 20, 2018 10:44 pm

ESP_Angus wrote:To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.
Ouch, that is absolutely low-level fiddling!
Out of my reach, and out of reach of 99% of the people here, isn't it?

ESP_igrr
Posts: 2067
Joined: Tue Dec 01, 2015 8:37 am

Re: SPIFFS fast enough for a one second data logger?

Postby ESP_igrr » Wed Mar 21, 2018 3:48 am

That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.

rin67630
Posts: 99
Joined: Sun Mar 11, 2018 5:13 pm

Re: SPIFFS fast enough for a one second data logger?

Postby rin67630 » Wed Mar 21, 2018 8:46 am

ESP_igrr wrote:That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.
But you want to be able to read the results as a regular file e.g. to send it via FTP. So that's not so straight forward as you pretend.

ESP_igrr
Posts: 2067
Joined: Tue Dec 01, 2015 8:37 am

Re: SPIFFS fast enough for a one second data logger?

Postby ESP_igrr » Thu Mar 22, 2018 9:21 am

Ah i see, I missed the FTP requirement. In that case, yes, it's going to be a bit more work to keep track of the beginning and the end of the ring buffer.

akbarhash
Posts: 7
Joined: Thu Sep 05, 2019 6:35 am

Re: SPIFFS fast enough for a one second data logger?

Postby akbarhash » Tue Sep 17, 2019 6:18 am

Hi ESP_Angus and ESP_igrr.
Sorry to resurrect this thread, if possible could a simple example be shared of the correct way to initialize and do serial write to the internal flash device using the library. A simple example like writing a small string and reading it back from flash memory would do. SPIFFS and NVS have a lot of overhead and is not suitable for the project I am working on. The spi flash api reference is a bit mixed up and not clear. Thanks.
ESP_igrr wrote: That's actually pretty simple — you have a block device with defined size, and functions to read/write/erase sectors of the block device. Simplest way would be to implement a ring buffer based on this block device.
ESP_Angus wrote:
Tue Mar 20, 2018 3:51 am
To add to igrr's advice, if all you need to ever do is write a lot of sequential fixed size data then you could also create a custom data partition and use the esp_partition_erase_range() and esp_partition_write() functions to write a log formatted journal, one entry at a time.

(It depends on the flash chip, but worst case for a single sector erase tends to be a few hundred milliseconds - and usually much less. Writing to flash after erasing should be only a couple of milliseconds.)

Who is online

Users browsing this forum: Google [Bot] and 109 guests