[ESP32] NVS Page Status: Full pages with only 2 written entries and 0 erased — is this normal?

[PETERESP]

Hi everyone,

I'm using NVS on ESP32 to store blob data under the namespace EEPROM. After running the NVS debug tool, I got the following output and have some questions about it.

NVS Partition Output:

Code: Select all

Index: Namespace
 001: [color=#0080FF]EEPROM[/color]
Page no. 0, Status: Full
 EEPROM:blob[0] =

Page no. 1, Status: Full
 EEPROM:blob[1] =

Page no. 2, Status: Active
 EEPROM:blob[2] =

Page Empty

Page Empty

Page Full
  Found entries:
    Written:     2
    Erased:      0
    Empty:       0
    Invalid:     0
    Total:       2

Page Full
  Found entries:
    Written:     1
    Erased:      0
    Empty:       0
    Invalid:     0
    Total:       1

Page Active
  Found entries:
    Written:     2
    Erased:      0
    Empty:     117
    Invalid:     0
    Total:     119

Page Empty
  Found entries:
    Written:     0
    Erased:      0
    Empty:     126
    Invalid:     0
    Total:     126

Page Empty
  Found entries:
    Written:     0
    Erased:      0
    Empty:     126
    Invalid:     0
    Total:     126

Global
  Config:
    Page size:    4096
    Entry size:     32
    Total pages:     5
  Entries:
    Written:     5
    Erased:      0
    Empty:     369
    Invalid:     0
    Total:     374

My Questions:

Why are some pages marked as Full even though they only have 2 or 1 written entries, and the maximum per page should be 126?
Why is the Erased count 0 on all pages? I expected some entries to be marked as erased after updates.
What I understand so far:

According to the NVS documentation, each page maps to one physical flash sector (4096 bytes) and can hold up to 126 entries (32 bytes each). When a key-value pair is updated, NVS appends a new entry at the end and marks the old one as erased — it does not overwrite in place.

My guess:

The Full pages with very few written entries might be namespace index pages, which only need a small number of entries and are then marked Full internally.
The Erased count being 0 suggests the blob data was written only once and never updated or deleted, so no old entries needed to be invalidated.
Is my understanding correct? Any clarification would be greatly appreciated!

Thanks in advance!

1

nvs_8KB测试.png (206.52 KiB) Viewed 82 times

repeat………… untill about 0x2000

2

nvs_8KB测试2.png (226.24 KiB) Viewed 82 times

Here is the relevant NVS-related code from my project for reference:

Configuration & globals:

Code: Select all

#define NVS_NAMESPACE "EEPROM"
#define BLOB_FIXED_SIZE (4096*2)  // Fixed blob size, 2 pages, total 5 pages
uint8_t read_blob[BLOB_FIXED_SIZE];
size_t  length = BLOB_FIXED_SIZE;
int nvs_offset;
uint8_t g_array[BLOB_FIXED_SIZE-2];

NVS initialization (runs once on boot):

Code: Select all

void NVS_chushihua_kaijizhixingyici() {
    esp_err_t err = nvs_flash_init();
    if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        nvs_flash_erase();
        err = nvs_flash_init();
    }
    nvs_handle_t nvs_handle;
    err = nvs_open("EEPROM", NVS_READWRITE, &nvs_handle);
    int32_t counter = 0;
    err = nvs_commit(nvs_handle);
    nvs_close(nvs_handle);
}

Read/Write function (caozuofanshi=1 for read, =2 for write):

Code: Select all

void function_EEPROM_caozuo(int caozuofanshi) {
    esp_err_t err = nvs_flash_init();
    nvs_handle_t nvs_handle;
    err = nvs_open("EEPROM", NVS_READWRITE, &nvs_handle);

    if (caozuofanshi == 1) {
        err = nvs_get_blob(nvs_handle, "blob", read_blob, &length);
    }

    nvs_offset = 0;
    read_blob[0]++; nvs_offset++;
    if (caozuofanshi == 2) { read_blob[1]++; } nvs_offset++;

    // PROCESS_VAR macro copies between read_blob and global variables
    PROCESS_VAR(g_array);

    if (caozuofanshi == 2) {
        err = nvs_set_blob(nvs_handle, "blob", read_blob, length);
        err = nvs_commit(nvs_handle);
    }
    nvs_close(nvs_handle);
}

My questions based on the above:

I use a fixed-size blob of 8192 bytes (4096*2) stored under the key "blob" in namespace "EEPROM". Is this a valid and recommended approach?
I call nvs_flash_init() every time function_EEPROM_caozuo() is called — is this safe, or should it only be called once?
Could the Full pages with very few entries be related to the large blob being split across multiple NVS entries internally?
Any advice is appreciated. Thank you!

ESP_ERR_NVS_NOT_ENOUGH_SPACE
E (1060) SLAVE_TEST: Save failed! (Failed to set NVS blob!)
Key observations from the logs:

On the first boot, nvs_get_blob returns 4354 (ESP_ERR_NVS_NOT_FOUND) — the blob does not exist yet, so the write succeeds with no old copy to retain.
On the second boot, nvs_get_blob returns 0 (ESP_OK) — the blob now exists. When attempting to update it, NVS needs space for both the old and new copy simultaneously, resulting in ESP_ERR_NVS_NOT_ENOUGH_SPACE.
The NVS partition is only 0x5000 = 20480 bytes (5 pages), while the blob alone is 8192 bytes. There is not enough room for two copies during an update.
Any suggestions on the best approach to resolve this would be greatly appreciated!


------
Update: Write succeeds on first boot, fails on second boot

Here are the actual serial logs from two consecutive boots with the same firmware:

First boot — write succeeds:

Code: Select all

I (52) boot: Partition Table:
I (55) boot: ## Label            Usage          Type ST Offset   Length
I (61) boot:  0 nvs              WiFi data        01 02 00009000 00005000
I (68) boot:  1 otadata          OTA data         01 00 0000e000 00002000
I (74) boot:  2 app0             OTA app          00 10 00010000 001e0000
I (81) boot:  3 app1             OTA app          00 11 001f0000 001e0000
I (87) boot:  4 spiffs           Unknown data     01 82 003d0000 00030000
I (94) boot: End of partition table
...
E (810) SLAVE_TEST: nvs_get_blob err = 4354 (ESP_ERR_NVS_NOT_FOUND), length = 8192
I (810) SLAVE_TEST: read_blob[0] = 0
I (810) SLAVE_TEST: read_blob[1] = 0
I (810) SLAVE_TEST: read_blob[2] = 0
I (810) SLAVE_TEST: read_blob[3] = 0
W (840) SLAVE_TEST: before save: read_blob[0] = 2
W (840) SLAVE_TEST: before save: read_blob[1] = 1
W (850) SLAVE_TEST: before save: read_blob[2] = 0
W (850) SLAVE_TEST: before save: read_blob[3] = 1
E (890) SLAVE_TEST: nvs_set_blob: err = 0, ESP_OK

Second boot — write fails:

Code: Select all

E (930) SLAVE_TEST: nvs_get_blob err = 0 (ESP_OK), length = 8192
I (930) SLAVE_TEST: read_blob[0] = 2
I (930) SLAVE_TEST: read_blob[1] = 1
I (930) SLAVE_TEST: read_blob[2] = 0
I (930) SLAVE_TEST: read_blob[3] = 1
W (960) SLAVE_TEST: before save: read_blob[0] = 4
W (960) SLAVE_TEST: before save: read_blob[1] = 2
W (960) SLAVE_TEST: before save: read_blob[2] = 0
W (970) SLAVE_TEST: before save: read_blob[3] = 2
E (1060) SLAVE_TEST: nvs_set_blob: err = 4357, ESP_ERR_NVS_NOT_ENOUGH_SPACE
E (1060) SLAVE_TEST: Save failed! (Failed to set NVS blob!)

Key observations from the logs:

On the first boot, nvs_get_blob returns 4354 (ESP_ERR_NVS_NOT_FOUND) — the blob does not exist yet, so the write succeeds with no old copy to retain.
On the second boot, nvs_get_blob returns 0 (ESP_OK) — the blob now exists. When attempting to update it, NVS needs space for both the old and new copy simultaneously, resulting in ESP_ERR_NVS_NOT_ENOUGH_SPACE.
The NVS partition is only 0x5000 = 20480 bytes (5 pages), while the blob alone is 8192 bytes. There is not enough room for two copies during an update.
Any suggestions on the best approach to resolve this would be greatly appreciated!


------
Correction & Key Question

After reviewing the ESP-IDF NVS documentation more carefully:

The blob size limit is 508000 bytes, or 97.6% of the partition size minus 4000 bytes, whichever is smaller.

For a 20480-byte NVS partition:

Code: Select all

20480 × 97.6% - 4000 ≈ 15989 bytes

This means an 8192-byte blob is well within the documented size limit. There should theoretically be enough remaining space (~7797 bytes) to hold both the old and new blob during an update.

So why does ESP_ERR_NVS_NOT_ENOUGH_SPACE occur on the second write?

Could any of the following be the actual cause?

Does the NVS blob chunking mechanism consume significantly more overhead than expected, leaving less usable space than the raw numbers suggest?
Is there additional space consumed by the namespace entry, chunk index entries, or other internal metadata that is not accounted for in the simple calculation above?
Is there a known issue with updating large blobs in small NVS partitions on ESP32?
Any insight into the actual space accounting would be very helpful. Thank you!

3

nvs_8KB测试3.png (60.14 KiB) Viewed 80 times

piture 4:
# Name Type SubType Offset Size Flags
nvs data nvs 0x9000 0x5000
otadata data ota 0xe000 0x2000
app0 app ota_0 0x10000 0x1E0000
app1 app ota_1 0x1F0000 0x1E0000
nvs_ex data nvs 0x3D0000 0x10000
spiffs data spiffs 0x3E0000 0x20000

One critical concern is whether remaining available space will keep shrinking over time.
Before switching to a new partition layout, I conducted a final test using a blob size of 7000 bytes. After five consecutive device restarts, no out-of-space errors appeared anymore.

Meanwhile, I monitored the global entry usage statistics across all five pages of the NVS partition after each reboot. Normally, the count of empty entries decreases monotonically as reboot times increase, but I observed an unprecedented rebound in empty entry numbers during this downtrend, which serves as clear evidence that the garbage collection mechanism is working properly.

While challenges still persist in ESP32 development, we have found a viable solution in the short term after troubleshooting. Relevant test parameters are shared below for reference:
An NVS partition sized at 0x5000 (equivalent to five full pages, 20 KB total) triggers insufficient storage errors when reading and writing blob_data[8192]. The first write operation completes successfully, yet the second write — namely the first data modification attempt — fails consistently.

Based on the calculation: 95% of total partition size minus 4000 bytes ≈ 19 KB − 4000 bytes ≈ twice the 7000-byte blob size, combined with the safe overwrite rule that new data is written into empty pages before legacy data gets marked invalid and eligible for overwriting in subsequent writes, are we only allowed to utilize half of the total available space?

In the final verification, repeated read and write operations on 7000-byte blobs ran without obvious anomalies.

[MicroController]

I use a fixed-size blob of 8192 bytes (4096*2) stored under the key "blob" in namespace "EEPROM". Is this a valid and recommended approach?

No.
https://docs.espressif.com/projects/esp ... and-values:

NVS works best for storing many small values, rather than a few large values of the type string and blob. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.

[PETERESP]

Thank you！Appreciate！

[ESP_rrtandler]

@PETERESP
Let me clarify the ocupied space calculation for 8192 Bytes BLOB on the NVS partition.

After the first write

Page 1
1 entry - namespace metadata
1 entry - BLOB_INDEX metadata (first value)
1 entry - BLOB_DATA metadata chunk #1
123 entries - chunk #1 payload (123*32 = 3936 B)

Page 2
1 entry - BLOB_DATA metadata chunk #2
125 entries - chunk #2 payload (125*32 = 4000 B)

Page 3
1 entry - BLOB_DATA metadata chunk #3
8 entries - chunk #3 payload (8*32 = 256 B)
117 available entries

Page 4
126 available entries

Page 5
Spare, never considered as the available space


The 8192 Bytes long BLOB will span to at least 3 pages. This means, that on top of 8192/32 = 256 entries for payload, there is always additional space needed - 1 entry for BLOB_INDEX, 1 entry for BLOB_DATA per page occupied by the payload (in this example at least 3 pages will be populated by the payload chunks, on larger and fragmented partitions, the number may be even higher). The overwrite will require a partition with at least 260 (1*BLOB_INDEX, 3*BLOB_DATA + 256*payload) available entries (8320 Bytes) to be able to replace the existing BLOB with a new 8192 Bytes of data. The 5 pages long partition from the example has only 243 entries available for overwrite. This explains why nvs_set_blob reports ESP_ERR_NVS_NOT_ENOUGH_SPACE error.

There exists an API function retrieving the number of available entries - see

https://docs.espressif.com/projects/esp ... vs_stats_t

and observe the .available_entries returned.

Hope it helps better understanding the required space calculation for BLOBS spanning over multiple pages.