Bug: esp_flash_erase_region() misbehaves on an encrypted flash

[code4sex]

IDF5.0.1, Flash encryption: DEV MODE
ISSUE: Reading out flash with esp_flash_read_encrypted() right after esp_flash_erase_region() DOES NOT return 0xFF's. Instead, arbitrary garbage is returned.

Same procedure ( esp_flash_erase_region() followed by esp_flash_read() ) on an unencrypted flash returns 0xFF's.

Sample code (DEV_MODE flash encryption):

Code: Select all

#include "esp_flash_spi_init.h"
#define ANY_ADDR 0x60000
static uint8_t test[0x1000];
// (1) 1st go
esp_flash_read_encrypted(0, ANY_ADDR, &test, 0x1000);
printf("\n\n1st check:\n\n"); for(uint16_t i=0; i< 0x1000; i++){ printf("%02X ",test[i]); }
// (2) erasing flash
esp_flash_erase_region(0, ANY_ADDR, 0x1000);
// (3) 2nd go
esp_flash_read_encrypted(0, ANY_ADDR, &test, 0x1000);
printf("\n\n2nd check:\n\n"); for(uint16_t i=0; i< 0x1000; i++){ printf("%02X ",test[i]); }
fflush(0);

I believe this is not the way things should happen.
esp_flash_erase_region() is likely to be revised to work with encrypted flash properly.

[MicroController]

IDF5.0.1, Flash encryption: DEV MODE
ISSUE: Reading out flash with esp_flash_read_encrypted() right after esp_flash_erase_region() DOES NOT return 0xFF's. Instead, arbitrary garbage is returned.

I believe this is not the way things should happen.

Actually, it is the way it's supposed to happen. Erasing a block of flash is not the same as encrypting a block of 0xFF and writing it to flash.

[code4sex]

Yes, technically, it is sure not the same.
And no, logically, it should be the same since we should get a container filled with 0xFF irrespective encrypted/unencrypted flash is used.

Life-case example: I'm fiddling with partition-tables at the moment. For some unknown reason Espressif bootloader fails/stalls in a cycled SW_RST upon accessing a partition-table NOT FILLED with trailing 0xFF's (no matter the vaild content and MD5). Also, the esp_partition_xxx functions abort and reboot esp32 in these conditions.
// Also, some bootloader gets crashed when "something sha256" signatures do not match another "something something whatsoever".... Okay, back to story.

Once again, the partition tables are VALID and MD5 is perfect! Why should Espressif want those shitless kilos of trailing 0xFF?!!!
Anyways, to have everything operate correctly the partition-table is first filled in memory with 0xFF and then gets populated over with records and only after that gets written to flash.
We all know that flash should be erased prior to writing hence esp_flash_erase_regions gets into play.
Also, we know that NAND flash is written by means of zeroing.
Bingo! Further optimisation allows to get rid of the memset(flash, 0xFF, size) call since the erased flash is already filled with 0xFF's.
Afterwards we get down to coding our major super-mega-firmware functionality happilly forgetting about this stuff.
Later on we decide to go for encryption and ... everything gets screwed because of that crap. Wtf?!
Thanks it's not a long way after the partition-table experiments so backwards tracking was quite fast.... But... This shouldn't 've happened at all!

So:
(1) Why stick to 0xFF in the meaningless part of PT ?!
(2) Why not come up with a universal esp_flash_erase_regions() that works reliably under either encryption mode?!
(3) Why not document all this stuff properly before esp32 community gets into trouble ?!

It ain't any rocket science to have all this done properly!

Resume: I keep saying this is a bug rather than a feature.

[MicroController]

Yes, technically, it is sure not the same.
And no, logically, it should be the same since we should get a container filled with 0xFF irrespective encrypted/unencrypted flash is used.

...

(2) Why not come up with a universal esp_flash_erase_regions() that works reliably under either encryption mode?!

Because of the way flash works. When using encryption, you can have either erased flash which you can write any data to but reads as 'garbage', or read valid 0xFF data but cannot overwrite it.

[code4sex]

no problem.
then calls to esp_flash_read_encrypted should tell garbage from erased.
software might be unaware whether it is reading blank (erased) flash or not. especially 3d party libraries.

[MicroController]

no problem.
then calls to esp_flash_read_encrypted should tell garbage from erased.

This is not possible in a general way because, as you know, a 'valid' 0xFF value stored in flash per se is indistinguishable from a (possibly 'invalid') erased value.
And what about partially filled sectors? How would the reader know which bytes of a 4k sector are valid/used and which are erased/garbage?

The 'shortcut' of assuming that erased flash reads back as 0xFF is just not valid with flash encryption.
If there was an easy way around this, the people behind SPIFFS or NVS on the ESP32 would already be using it.

[code4sex]

Although your comments make sense in a general way there's still one thicky thing.

Look, we have a physical array of memory cells having 0xFF's after flash erase. You are right in a way WE DON'T know whether it is a valid 0xFF value or a "blank" cell just erased.

However, esp_flash_read_encrypted() DOES KNOW somehow and makes a difference:
(a) returns garbage when we read erased but uninitialized flash
(b) returns 0xFF's when we have these bytes written explicitly

The bug-proof version of esp_flash_read_encrypted() should have failed with sort of a ERROR_FLASH_NOT_INITIALIZED error code in case [a].

Do you agree?

[MicroController]

However, esp_flash_read_encrypted() DOES KNOW somehow and makes a difference:

No, esp_flash_read_encrypted() and the underlying crypto hardware does not know this.

Here's what happens:
1. You erase some flash memory. When the data is subsequently retrieved from the flash, it comes back as all 0xFF to the ESP.
2. When you use esp_flash_read_encrypted() on that data, it comes as 0xFF out of the flash; that 0xFF then goes into decryption and is unconditionally 'decrypted', inherently resulting in pseudo-random 'garbage', because, unbeknownst to the decryption process, the 0xFF stored in flash is not the result of previously encrypting meaningful data.

Effectively, the decryption actually encrypts in this case, taking 'plaintext' 0xFF from flash and producing cryptographically hard junk as output. It does this specifically because it cannot know whether a 0xFF in flash is a 'meaningful' ciphertext or not.

(Same goes in reverse: If you'd write just the right junk byte sequence to encrypted flash you could end up with all-0xFF as the ciphertext in flash.)

[code4sex]

Oh...My mistake - I DID miss the point that explicilty writing 0xFF to flash in DEV_MODE will become "garbage". Then everything makes sense....
Thanks a lot
Then my post effectively gets shrinked to just accusion of capricious bootloader behaviour without trailing 0xFF in PT