Flashing pre-encrypted full binary image ended with invalid header

[dmitrij999]

Hello everyone!

For devices production and make further device service possible, I'd like to make flash encryption and secure boot in this way:
- take binaries with signed app and bootloader
- generate nvs binary file per device from factory provision data
- generate image and nvs keys on host per device
- encrypt the corresponding binaries: bootloader, partition table, otadata initial and app
- build the full image
- burn flash encryption key and secure boot digest into device
- burn the corresponding efuses
- flash the prepared full image

The goals are:
- Prevent firmware and data from leakage
- Prevent firmware from counterfeit
- Remain the ability to readout flash contains encrypted to readout user data by serviceman

After these steps, I got an error:

Code: Select all

invalid header: 0x29207d17

As a reference, I used this guide: https://docs.espressif.com/projects/esp ... flows.html

And I used the following commands:

Code: Select all

# burn keys and efuses
..\..\espefuse.exe --port COM29 --chip esp32s3 burn_key --show-sensitive-info BLOCK_KEY0 keys/image_key.bin XTS_AES_128_KEY BLOCK_KEY1 secure_boot_key_digest.bin SECURE_BOOT_DIGEST0
..\..\espefuse.exe --port COM29 --chip esp32s3 burn_efuse SPI_BOOT_CRYPT_CNT 7 DIS_DOWNLOAD_ICACHE 1 DIS_DOWNLOAD_DCACHE 1 HARD_DIS_JTAG 1 DIS_DIRECT_BOOT 1 DIS_USB_JTAG 1 DIS_DOWNLOAD_MANUAL_ENCRYPT 1 
..\..\espefuse.exe --port COM29 --chip esp32s3 burn_efuse SECURE_BOOT_KEY_REVOKE1 1 SECURE_BOOT_KEY_REVOKE2 1 SECURE_BOOT_EN 1

# Flashing full image
..\..\esptool.exe --port COM29 --baud 921600 --chip esp32s3 --no-stub write_flash 0 full.bin --force

Could you please help me with precising the right steps to implement flashing pre-encrypted images?
Logs are in attachments

[dmitrij999]

Addendum: I generated AES-XTS 128 key, then specified that I burn AES-XTS-128 key, and binary is decrypted incorrectly.
Error message:

Code: Select all

invalid header: 0x29207d17

Encrypted full binary header: FD F1 37 53
Plain full binary header: E9 04 02 3F
Readout header: FD F1 37 53

[dmitrij999]

I tried to implement security features on QEMU, and I get the same results.
Here is the sdkconfig stub regarding to security features:

Code: Select all

#
# Security features
#
CONFIG_SECURE_SIGNED_ON_BOOT=y
CONFIG_SECURE_SIGNED_ON_UPDATE=y
CONFIG_SECURE_SIGNED_APPS=y
CONFIG_SECURE_BOOT_V2_RSA_SUPPORTED=y
CONFIG_SECURE_BOOT_V2_PREFERRED=y
CONFIG_SECURE_BOOT_V2_RSA_ENABLED=y
CONFIG_SECURE_SIGNED_APPS_RSA_SCHEME=y
CONFIG_SECURE_BOOT=y
CONFIG_SECURE_BOOT_V2_ENABLED=y
CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
CONFIG_SECURE_BOOT_SIGNING_KEY="prod_secure_boot_key.pem"
# CONFIG_SECURE_BOOT_ENABLE_AGGRESSIVE_KEY_REVOKE is not set
# CONFIG_SECURE_BOOT_V2_ALLOW_EFUSE_RD_DIS is not set
CONFIG_SECURE_BOOT_FLASH_BOOTLOADER_DEFAULT=y
# CONFIG_SECURE_BOOT_INSECURE is not set
CONFIG_SECURE_FLASH_ENC_ENABLED=y
CONFIG_SECURE_FLASH_ENCRYPTION_AES128=y
# CONFIG_SECURE_FLASH_ENCRYPTION_AES256 is not set
CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT=y
# CONFIG_SECURE_FLASH_ENCRYPTION_MODE_RELEASE is not set
CONFIG_SECURE_FLASH_HAS_WRITE_PROTECTION_CACHE=y

The experiment shows that even in the case of generated flash encryption key, the "ESP32S3" (emulated with QEMU) refuses to boot with it. Hardware bug?

QEMU log regarding to flash encryption & secure boot

Code: Select all

Generating efuse image: /home/dmitrij999/esp32-workspace/v5.4.1/smartvalve32/build.secure/qemu_efuse.bin
Running qemu (bg): qemu-system-xtensa -M esp32s3 -drive file=/home/dmitrij999/esp32-workspace/v5.4.1/smartvalve32/build.secure/qemu_flash.bin,if=mtd,format=raw -drive file=/home/dmitrij999/esp32-workspace/v5.4.1/smartvalve32/build.secure/qemu_efuse.bin,if=none,format=raw,id=efuse -global driver=nvram.esp32c3.efuse,property=drive,value=efuse -global driver=timer.esp32s3.timg,property=wdt_disable,value=true -nic user,model=open_eth -nographic -serial tcp::5555,server
Executing action: monitor
Running idf_monitor in directory /home/dmitrij999/esp32-workspace/v5.4.1/smartvalve32
Executing "/home/dmitrij999/esp/esp-tools/esp-idf-v5.4.1/python_env/idf5.4_py3.12_env/bin/python /home/dmitrij999/esp/esp-idf-v5.4.1/tools/idf_monitor.py -p socket://localhost:5555 -b 115200 --toolchain-prefix xtensa-esp32s3-elf- --target esp32s3 --revision 0 /home/dmitrij999/esp32-workspace/v5.4.1/smartvalve32/build.secure/SmartValve_v3.elf -m '/home/dmitrij999/esp/esp-tools/esp-idf-v5.4.1/python_env/idf5.4_py3.12_env/bin/python' '/home/dmitrij999/esp/esp-idf-v5.4.1/tools/idf.py' '-B' 'build.secure' '-DSDKCONFIG=sdkconfig.secure'"...
--- esp-idf-monitor 1.6.2 on socket://localhost:5555 115200
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
I (139) esp_image: segment 1: paddr=002e188c vaddr=3fca5700 size=05974h ( 22900) load
I (145) esp_image: segment 2: paddr=002e7208 vaddr=40374000 size=08e10h ( 36368) load
I (154) esp_image: segment 3: paddr=002f0020 vaddr=42000020 size=114a1ch (1133084) map
I (351) esp_image: segment 4: paddr=00404a44 vaddr=4037ce10 size=188dch (100572) load
I (374) esp_image: segment 5: paddr=0041d328 vaddr=600fe100 size=0001ch (    28) load
I (375) esp_image: segment 6: paddr=0041d34c vaddr=00000000 size=02c84h ( 11396) 
I (380) esp_image: Verifying image signature...
I (383) secure_boot_v2: Secure boot V2 is not enabled yet and eFuse digest keys are not set
I (384) secure_boot_v2: Verifying with RSA-PSS...
I (388) secure_boot_v2: Signature verified successfully!
I (390) boot: Loaded app from partition at offset 0x270000
I (393) boot: Set actual ota_seq=1 in otadata[0]
I (393) secure_boot_v2: enabling secure boot v2...
I (394) efuse: Batch mode of writing fields is enabled
I (395) esp_image: segment 0: paddr=00000020 vaddr=3fce2980 size=034b8h ( 13496) 
I (402) esp_image: segment 1: paddr=000034e0 vaddr=403c8700 size=00004h (     4) 
I (404) esp_image: segment 2: paddr=000034ec vaddr=403c8704 size=00d20h (  3360) 
I (407) esp_image: segment 3: paddr=00004214 vaddr=403cb700 size=05494h ( 21652) 
I (416) esp_image: Verifying image signature...
I (420) secure_boot_v2: Secure boot V2 is not enabled yet and eFuse digest keys are not set
I (421) secure_boot_v2: Verifying with RSA-PSS...
I (422) secure_boot_v2: Signature verified successfully!
I (424) secure_boot_v2: Secure boot digests absent, generating..
I (441) secure_boot_v2: Digests successfully calculated, 1 valid signatures (image offset 0x0)
I (443) secure_boot_v2: 1 signature block(s) found appended to the bootloader.
I (444) secure_boot_v2: Burning public key hash to eFuse
I (445) efuse: Writing EFUSE_BLK_KEY0 with purpose 9
I (815) secure_boot_v2: Digests successfully calculated, 1 valid signatures (image offset 0x270000)
I (817) secure_boot_v2: 1 signature block(s) found appended to the app.
I (817) secure_boot_v2: Application key(0) matches with bootloader key(0).
I (818) secure_boot_v2: Revoking empty key digest slot (1)...
I (818) secure_boot_v2: Revoking empty key digest slot (2)...
I (819) secure_boot_v2: blowing secure boot efuse...
W (819) secure_boot: UART ROM Download mode kept enabled - SECURITY COMPROMISED
I (820) secure_boot: Disable hardware & software JTAG...
I (822) efuse: BURN BLOCK4
I (827) efuse: BURN BLOCK4 - OK (write block == read block)
I (827) efuse: BURN BLOCK0
I (831) efuse: BURN BLOCK0 - OK (write block == read block)
I (832) efuse: Batch mode. Prepared fields are committed
I (832) secure_boot_v2: Secure boot permanently enabled
I (833) boot: Checking flash encryption...
I (834) efuse: Batch mode of writing fields is enabled
I (834) flash_encrypt: Generating new flash encryption key...
I (835) efuse: Writing EFUSE_BLK_KEY1 with purpose 4
W (836) flash_encrypt: Not disabling UART bootloader encryption
W (836) flash_encrypt: Not disabling UART bootloader cache - SECURITY COMPROMISED
I (837) flash_encrypt: Disable JTAG...
I (837) efuse: BURN BLOCK5
I (841) efuse: BURN BLOCK5 - OK (write block == read block)
I (842) efuse: BURN BLOCK0
I (846) efuse: BURN BLOCK0 - OK (all write block bits are set)
I (847) efuse: Batch mode. Prepared fields are committed
I (847) esp_image: segment 0: paddr=00000020 vaddr=3fce2980 size=034b8h ( 13496) 
I (852) esp_image: segment 1: paddr=000034e0 vaddr=403c8700 size=00004h (     4) 
I (853) esp_image: segment 2: paddr=000034ec vaddr=403c8704 size=00d20h (  3360) 
I (855) esp_image: segment 3: paddr=00004214 vaddr=403cb700 size=05494h ( 21652) 
I (861) esp_image: Verifying image signature...
I (864) secure_boot_v2: Verifying with RSA-PSS...
I (866) secure_boot_v2: Signature verified successfully!
I (942) flash_encrypt: bootloader encrypted successfully
I (951) flash_encrypt: partition table encrypted and loaded successfully
I (951) flash_encrypt: Encrypting partition 1 at offset 0x25e000 (length 0x2000)...
I (965) flash_encrypt: Done encrypting
I (965) flash_encrypt: Encrypting partition 2 at offset 0x260000 (length 0x10000)...
I (1053) flash_encrypt: Done encrypting
I (1054) esp_image: segment 0: paddr=00270020 vaddr=3c120020 size=71864h (464996) map
I (1151) esp_image: segment 1: paddr=002e188c vaddr=3fca5700 size=05974h ( 22900) 
I (1157) esp_image: segment 2: paddr=002e7208 vaddr=40374000 size=08e10h ( 36368) 
I (1167) esp_image: segment 3: paddr=002f0020 vaddr=42000020 size=114a1ch (1133084) map
I (1376) esp_image: segment 4: paddr=00404a44 vaddr=4037ce10 size=188dch (100572) 
I (1397) esp_image: segment 5: paddr=0041d328 vaddr=600fe100 size=0001ch (    28) 
I (1398) esp_image: segment 6: paddr=0041d34c vaddr=00000000 size=02c84h ( 11396) 
I (1402) esp_image: Verifying image signature...
I (1404) secure_boot_v2: Verifying with RSA-PSS...
I (1406) secure_boot_v2: Signature verified successfully!
I (1407) flash_encrypt: Encrypting partition 3 at offset 0x270000 (length 0x2c0000)...
I (4369) flash_encrypt: Done encrypting
E (4369) esp_image: image at 0x530000 has invalid magic byte (nothing flashed here?)
I (4370) flash_encrypt: Encrypting partition 5 at offset 0x7f0000 (length 0x10000)...
I (4446) flash_encrypt: Done encrypting
I (4447) efuse: BURN BLOCK0
I (4451) efuse: BURN BLOCK0 - OK (all write block bits are set)
I (4453) flash_encrypt: Flash encryption completed
I (4453) boot: Resetting with flash encryption enabled...
<< Monitor log freezes here, and after repeated 'idf.py -B "sdkconfig.secure" qemu monitor' returns invalid header >>

As well, I tried to use AES-XTS-128 and AES-XTS-256 keys

[dmitrij999]

I've tested flash encryption externally, and I figured out how to do it right. But in my application, flashing secure boot digest manually is not necessary, and I let the bootloader onboard generate and burn secure boot digest, just because in this case I need to bring secure boot digest binary to my tooling, and I wouldn't like to do this.

Here is the workflow I tested.

Configure the project

• Enable Flash Encryption and Secure boot in project. Optionally, if you need to encrypt NVS as well, you need enable NVS Encryption in project as well
  The exact config values I have:

  Code: Select all

  #
  # Security features
  #
  CONFIG_SECURE_SIGNED_ON_BOOT=y
  CONFIG_SECURE_SIGNED_ON_UPDATE=y
  CONFIG_SECURE_SIGNED_APPS=y
  CONFIG_SECURE_BOOT_V2_RSA_SUPPORTED=y
  CONFIG_SECURE_BOOT_V2_PREFERRED=y
  CONFIG_SECURE_BOOT_V2_RSA_ENABLED=y
  CONFIG_SECURE_SIGNED_APPS_RSA_SCHEME=y
  CONFIG_SECURE_BOOT=y
  CONFIG_SECURE_BOOT_V2_ENABLED=y
  CONFIG_SECURE_BOOT_BUILD_SIGNED_BINARIES=y
  CONFIG_SECURE_BOOT_SIGNING_KEY="prod_secure_boot_key.pem"
  # CONFIG_SECURE_BOOT_ENABLE_AGGRESSIVE_KEY_REVOKE is not set
  # CONFIG_SECURE_BOOT_V2_ALLOW_EFUSE_RD_DIS is not set
  CONFIG_SECURE_BOOT_FLASH_BOOTLOADER_DEFAULT=y
  # CONFIG_SECURE_BOOT_INSECURE is not set
  CONFIG_SECURE_FLASH_ENC_ENABLED=y
  CONFIG_SECURE_FLASH_ENCRYPTION_AES128=y
  # CONFIG_SECURE_FLASH_ENCRYPTION_AES256 is not set
  # CONFIG_SECURE_FLASH_ENCRYPTION_MODE_DEVELOPMENT is not set
  CONFIG_SECURE_FLASH_ENCRYPTION_MODE_RELEASE=y
  CONFIG_SECURE_FLASH_HAS_WRITE_PROTECTION_CACHE=y
  # CONFIG_SECURE_FLASH_ENCRYPT_ONLY_IMAGE_LEN_IN_APP_PART is not set
  CONFIG_SECURE_FLASH_CHECK_ENC_EN_IN_APP=y
  CONFIG_SECURE_ROM_DL_MODE_ENABLED=y
  # CONFIG_SECURE_DISABLE_ROM_DL_MODE is not set
  # CONFIG_SECURE_ENABLE_SECURE_ROM_DL_MODE is not set
  CONFIG_SECURE_INSECURE_ALLOW_DL_MODE=y
  # end of Security features
  
  ...
  
  #
  # Partition Table
  #
  # CONFIG_PARTITION_TABLE_SINGLE_APP is not set
  # CONFIG_PARTITION_TABLE_SINGLE_APP_LARGE is not set
  # CONFIG_PARTITION_TABLE_TWO_OTA is not set
  # CONFIG_PARTITION_TABLE_TWO_OTA_LARGE is not set
  CONFIG_PARTITION_TABLE_CUSTOM=y
  # CONFIG_PARTITION_TABLE_SINGLE_APP_ENCRYPTED_NVS is not set
  # CONFIG_PARTITION_TABLE_SINGLE_APP_LARGE_ENC_NVS is not set
  # CONFIG_PARTITION_TABLE_TWO_OTA_ENCRYPTED_NVS is not set
  CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partition_table_8mb.csv"
  CONFIG_PARTITION_TABLE_FILENAME="partition_table_8mb.csv"
  CONFIG_PARTITION_TABLE_OFFSET=0x40000
  CONFIG_PARTITION_TABLE_MD5=y
  # end of Partition Table
  
  ...
  
  #
  # NVS
  #
  CONFIG_NVS_ENCRYPTION=y
  # CONFIG_NVS_COMPATIBLE_PRE_V4_3_ENCRYPTION_FLAG is not set
  # CONFIG_NVS_ASSERT_ERROR_CHECK is not set
  # CONFIG_NVS_LEGACY_DUP_KEYS_COMPATIBILITY is not set
  CONFIG_NVS_ALLOCATE_CACHE_IN_SPIRAM=y
  # end of NVS
  

• Build the project
• After building, I extract the binaries and put to tooling the following artifacts:
  • bootloader
  • otadata_initial
  • partition-table
  • app itself

Tooling does the following workflow:

• get unique data for device (such as serial number or something)
• create folder for device named with SerialNumber. All the further operations with device package will be done in this folder.
• generate nvs.csv and nvs.bin.plain based on these data
• generate nvs encryption key saving to nvs_key.bin.plain, encrypts nvs.bin.plain to nvs.bin and deletes nvs.bin.plain
• copy artifacts to device package, then add ". plain" to their name extensions
• generate flash encryption key
• encrypt artifacts in package from .bin.plain to just .bin, except nvs.bin (nvs.bin should be encrypted with another key), but we need also encrypt nvs_key.bin.plain into nvs_key.bin
  The problem I had because I forgot —aes-xts flag, and it was specified in docs
• merge all the binaries into full.bin according to partition table (tooling uses its own partitions description format)
• deletes all the .plain files
• archive package to further service usage
• flash device with full.bin
• burn flash encryption key
• burn efuses to make esp32 work with encrypted flash (don't touch secure boot efuses, as well, don't touch DIS_ICACHE).
  The exact efuses set to be burnt varies between esp32 MCU's:

  Code: Select all

  esp32: "FLASH_CRYPT_CNT 127", "FLASH_CRYPT_CONFIG 0xF", "DISABLE_DL_ENCRYPT 1", "DISABLE_DL_DECRYPT 1", "DISABLE_DL_CACHE 1", "DISABLE_JTAG 1"
  esp32s2: "SPI_BOOT_CRYPT_CNT 7", "DIS_BOOT_REMAP 1", "DIS_DOWNLOAD_ICACHE 1", "DIS_DOWNLOAD_DCACHE 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1", "HARD_DIS_JTAG 1", "DIS_LEGACY_SPI_BOOT 1"
  esp32s3: "SPI_BOOT_CRYPT_CNT 7", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1", "DIS_DOWNLOAD_ICACHE 1", "DIS_DOWNLOAD_DCACHE 1", "HARD_DIS_JTAG 1", "DIS_USB_JTAG 1", "DIS_DIRECT_BOOT 1"
  esp32c3: "SPI_BOOT_CRYPT_CNT 7", "DIS_DOWNLOAD_ICACHE 1", "DIS_DIRECT_BOOT 1", "DIS_USB_JTAG 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1"
  esp32c2: "SPI_BOOT_CRYPT_CNT 7", "DIS_DOWNLOAD_ICACHE 1", "DIS_DIRECT_BOOT 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1"
  esp32c5: "SPI_BOOT_CRYPT_CNT 7", "DIS_DIRECT_BOOT 1", "DIS_USB_JTAG 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1", "XTS_DPA_PSEUDO_LEVEL 1"
  esp32c6/esp32c61: "SPI_BOOT_CRYPT_CNT 7", "DIS_DOWNLOAD_ICACHE 1", "DIS_DIRECT_BOOT 1", "DIS_USB_JTAG 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1"
  esp32h2: "SPI_BOOT_CRYPT_CNT 7", "DIS_DIRECT_BOOT 1", "DIS_USB_JTAG 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1", "XTS_DPA_PSEUDO_LEVEL 1"
  esp32p4: "SPI_BOOT_CRYPT_CNT 7", "DIS_DIRECT_BOOT 1", "DIS_USB_JTAG 1", "DIS_PAD_JTAG 1", "DIS_DOWNLOAD_MANUAL_ENCRYPT 1", "DIS_DOWNLOAD_MSPI 1"
  

  device is flashed, move to the testing or move on

During first device start, it runs as the flash encryption is enabled, but it sees that bootloader is in secure boot but device is not configured for that, therefore it generates the digest and burns it and secure boot efuses onboard.

This way, I confirmed that setting just flash encryption efuses and flashing encrypted binary is sufficient even if you need secure boot as well. I worked out the balanced scheme where device will be protected well and easy to recover it (within the factory), in addition, I haven't to do odd operations.