Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

rfleming
Posts: 21
Joined: Tue Oct 09, 2018 12:30 am

Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby rfleming » Wed Nov 18, 2020 9:03 am

Hi All,

I have been running into "bugs" in the FATFS implmentation of the driver. This is directly caused from the way I write to the driver using it in ways that it shouldn't be used. My project requires me to use 1MB of storage and I would prefer not to use FATFS as I have ran into issues with it on roughly 10% of all my units.

Hopefully without redirecting this thread title too much, let me identify my issues so far with the FATFS emulation. Note, this could be bug related, but is quite difficult to fix on my end I believe as there is too many layers between my code and the silicon.
I am happy to send my code directly to the esp-team if required. As this a commerical project I would prefer not to upload it to a public forum.

----
I am afraid I have a few units in field and this just happens after "sometime" (a few weeks of continuous operation) on roughly 10% of ours units. So I cannot quite identify what is the initial trigger to get the units into this state. My guess is that for some reason I have somehow corrupted a file (no idea how) then when I try to write it, I can't so the system creates a new file and continues to write on that one instead.
Specifically what happens is I have a unit that has the below partition table. I am writing to 2 files in the storage section where 1 is a configuration file that only needs to be ~40 bytes and the other is for measurement logging. I am writing to the file in a circular buffer style where it initially fills up the entire partition (with the one file) minus the configuration file. Then once full, starts to overwrite the oldest data at the top of the file. This was my psuedo wear levelling idea to maximise the storage that the esp has.
_Note: If there is an easier way to write to flash directly that would be the prefered option and would likely solve all my issues._

Code: Select all

# Espressif ESP32 Partition Table
# Name, 	Type, 	SubType, 	Offset, 	Size, 		Flags
nvs,      	data,	nvs,     	0x9000,  	0x4000,
otadata,	data,	ota,		0xd000,	0x2000,
phy_init, 	data,	phy,     	0xf000,  	0x1000,
ota_0,	app,		ota_0,	0x10000,	0x190000,
ota_1,	app,		ota_1, 	,		0x190000,
storage,	data,	fat,		,		0xD0000,
After a while I noticed that randomly that data was missing. Doing a directly listing on my unit provided some insite into this issue:

Code: Select all

"ÿÿÿÿÿÿÿÿ.ÿÿÿ", size -2
"ÿÿÿÿÿÿÿÿ.ÿÿÿ", size -2
...repeat ÿÿÿÿÿÿÿÿ.ÿÿÿ 256 times...
"CONFIG.LOG", size 0
"PARTICLE.LOG", size 0
Looking at the structure of the file name, the ÿ is really just 0xFF converted as ASCII and the xxxxxxxx.xxx is the same file name length and structure as PARTICLE.LOG and still contains the dot for file extension, it is safe to assume this is the root of the issue.

First looking into code, obviously built around the c standard, the readdir reads iterates through files, though I found no way to do a stat() on the file that is returned because it requires a file name, that is identical for every entry.

Code: Select all

void Hal_Storage_FatFs::Debug_ListFilesInDir(const char *dir_name) {
	DIR *dir;
	struct dirent *ent;
	if ((dir = opendir(dir_name)) != NULL) {
		/* print all the files and directories within directory */
		while ((ent = readdir(dir)) != NULL) {
			struct stat stat_buf;
			memset(&stat_buf, 0, sizeof(stat_buf));
			long len = 0;
			char path[128];
			strcpy(path, dir_name);
			strcat(path, "/");
			strcat(path, ent->d_name);

			if (stat(path, &stat_buf) == 0) {
				len = stat_buf.st_size;
			}
			else {
				len = -2;  //error. app would likely give 0 or -1, so just do this instead.
			}
			LogD("\"%s\", size %ld", ent->d_name, len);
		}
		closedir(dir);
	}
	else {
		/* could not open directory */
		LogE("FATFS Error couldn't open dir");
	}
}
I dont quite know what happens internally either in my code or in the driver to cause this.

I open the file itself by doing:

Code: Select all

FILE *Hal_Storage_FatFs::OpenFile_ForModifying(const char *file_name) {
	if (access(file_name, F_OK) == -1) {
		//create new file
		LogD("Creating new file.");
		FILE *fileTmp = fopen(file_name, "wb");
		if (fileTmp != NULL)
			fclose(fileTmp);
	}

	//now actually access the file.
	char write_flags[] = "rb+";

	FILE *file = fopen(file_name, write_flags);

	if (file == NULL) {
		LogE("Couldn't open file for modifying '%s'", file_name == NULL ? "NULL" : file_name);
	}
	return file;
}
and call it by doing:

Code: Select all

#define FILE_NAME_DATALOG "/data/particle.log"
this->fileHandle = fatfs->OpenFile_ForModifying(FILE_NAME_DATALOG);
so the file name itself should NEVER change to a 0xFF crazy mess. Nor do I ever delete individual files.

So at the end of the day, it would be a lot simpler to write directly to flash if possible :)

ESP_Sprite
Posts: 4112
Joined: Thu Nov 26, 2015 4:08 am

Re: Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby ESP_Sprite » Thu Nov 19, 2020 1:46 am

From what I recall, FatFS is only robust to power fails in some configurations. The docs should have more information about this.

It's possible to directly access partitions. You can use the SPI flash api to either do that or write to the entire flash. (Just in case: do make sure you're aware of NOR flash properties wrt wear-out, sector and erase sector sizes etc. Flash without translation layers is quirky if you're used to 'standard' block devices.)

rfleming
Posts: 21
Joined: Tue Oct 09, 2018 12:30 am

Re: Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby rfleming » Thu Nov 19, 2020 6:54 am

Thanks for the info Sprite. Our devices have battery backup and spend most of their life asleep. So while it is possible that a few of these units may have these issues triggered from power cycles or similar, its more likely a cause from something else.

I'll have a crack and writing directly to to the SPI flash then. Everytime I saw this in passing previously I always thought it was only for external ICs. I am typically used to NAND flash from the STM32F7 series, AVR series and SAMD21 series. So the ESP may have some quirks I'll have to get used to.

ESP_Sprite
Posts: 4112
Joined: Thu Nov 26, 2015 4:08 am

Re: Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby ESP_Sprite » Fri Nov 20, 2020 7:39 am

Technically, the flash in an ESP32 module is an 'external IC' as in the flash is not on the die itself, so in that aspect it's more-or-less the same as you're used to. The difference is that the ESP32 has some advanced caching that transparently can allow memory access to the chip, so it's not advisable to roll your own SPI access driver for the flash chip, but as long as you use the partition API I pointed out, you should be able to apply your experience without issues.

rfleming
Posts: 21
Joined: Tue Oct 09, 2018 12:30 am

Re: Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby rfleming » Sun Nov 22, 2020 2:59 am

ESP_Sprite wrote: Technically, the flash in an ESP32 module is an 'external IC' as in the flash is not on the die itself, so in that aspect it's more-or-less the same as you're used to. The difference is that the ESP32 has some advanced caching that transparently can allow memory access to the chip, so it's not advisable to roll your own SPI access driver for the flash chip, but as long as you use the partition API I pointed out, you should be able to apply your experience without issues.
Perfect!

Since I am using the internal spi flash, can I also utilise the same SPI bus used internally?

That is, currently there are 3 SPI buses:
#define SPI_HOST SPI1_HOST
#define HSPI_HOST SPI2_HOST
#define VSPI_HOST SPI3_HOST

As the first SPI_HOST is typically used for internal flash, it previously was not recommended to use this for external use. So an external IC etc. Though since I am using the internal flash now, is it fine to force this SPI channel to be used for this instead?


Edit: I have a free SPI. Though I dont have any free IOs. I couldn't find any example of using the internal Flash directly. The best I found was in "components/spi_flash/test/test_read_write.c" or
https://github.com/espressif/esp-idf/bl ... ad_write.c

Here a good example that just assumes everything is setup (a lot better for me)

Code: Select all

static void IRAM_ATTR test_read(int src_off, int dst_off, int len)
They simply disable/enable cache when writing and no special considerations are required when reading from any memory location. In terms of the partition itself, as its just a memory location, there is no explicit reason to "mount" it as its not really a file system, just a continuous memory allocation.

Anyway, going to have a crack at this hope it works ;)

rfleming
Posts: 21
Joined: Tue Oct 09, 2018 12:30 am

Re: Can I write directly to flash/partition without fatfs/wear lvl etc drivers?

Postby rfleming » Sun Nov 22, 2020 4:34 am

Even better...

examples/storage/partition_api/partition_ops

Here it provides an example how to write/read/erase to a partition. I only a little concerned about how the underlining write operation works though. I'll put this into a new thread as this once has now gone a little out of scope.

Who is online

Users browsing this forum: davidtaille and 43 guests