ESP32 Forum

Hello

I am developing an immutable (read-only) filesystem for esp32, based on TAR archive file format. I need it for my project (lots of PHP scripts and web-content), but may be someone else can find it useful

It is fast, really fast (26 MiB/sec read()/fread() on ESP32S3 DevKit clone in typical scenario: 1k buffer reads in a tight loop)

It supports hardlinks, symlinks and directory junctions. Supports looooong names, UTF8 charset. Adds support for mmap(), statvfs(), dupfd() and fdopendir() - these are extensions to Espressif's VFS.

open() can open directories

Lookup times are O(logN), read operations are O(1), no allocators or mutexes in hot path (open() and opendir() are NOT wait-free).

No central superblock, capable of CRC64 protection (while remains compatible with existing TAR tools). A tool for injecting CRC64 information in exiting tar archive is available (can be compiled for Linux or Windows/Cygwin)

Tarfs is "damage resistant" - tries to mount what is not damaged (i.e. it is almost impossible to critically damage the filesystem).

CRC64 is checked on mount() but I plant to add support for check on each open() - for systems with huge uptime.

Uses integer indexes instead of pointers for VFS contexts, avoids a common race on unregister_fs() + open()

Memory consumption:
Compiled for ESP32S3 (-Os): .text is ~10Kbytes, .bss is 24 bytes, .data: 0 bytes
Inode is 24 bytes in size on ESP32S3.
A filesystem with 1000 files will require ~24KiB of RAM
There are no alloc/free in a hot paths - no memory fragmentation (fdopendir() uses malloc)

It is portable (compiles with -Wall -pedantic, all platform depended code is in one single file) and can be ported with minimal effort to STM32 and any other MCU which has flash memory mapped to MCU's address space (at least readonly)

Although it is an Arduino library, it has no single line of code related to Arduino Core.

Install (stable version but older): available from the Arduino IDE's Library Manager (search for "tarfs")
Project page (latest source code): https://github.com/vvb333007/tarfs
What it is all about (long read): https://github.com/vvb333007/tarfs/blob/main/README.md
How to create and flash your filesystem: https://github.com/vvb333007/tarfs/blob ... ckStart.md

Thanks for for sharing this library, incl elaborate documentation

Some test results:
Filesystem: A TAR file, 12 MiB in size, consisting of 640 entries (of them ~ 20 are directories, the rest are html files)
Hardware: ESP32S3, 16MiB OPI PSRAM, 32MiB FLASH, QUAD, 240MHz CPU clock.
PSRAM is enabled and used by the tarfs (~16000 bytes).
open() call: ~92 microseconds on average. Second open on the same file: 21 microseconds (dunno why, probably cache issue)
tarfs_mount(): 66 milliseconds


It can be compiled for ESP-IDF without any changes - just compile everything inside src/ directory. There is no CMake file, but you can use tarsum/Makefile as a starting point.

After answering questions on Arduino forum I made a FAQ:

https://github.com/vvb333007/tarfs/blob ... ocs/FAQ.md

Damage resistance was tested like this: freshly created FS image (CRC64 applied) was then damaged by five memset(,0x5a, 200): first one applied to the start of the archive, damaging very first entry. Second - at the end, damaging last 200 bytes. Then it was 3 random memsets of 200 bytes. Then this damaged archive was flashed to ESP32S3 and mounted. It mounts, but 1..3 files become inaccessible. Attempt to open() such a file results in errno = EIO.