Out of buffer: OTA + AWS S3 + presigned URL

[anees.ahmad.jaffer]

I'm trying to implement OTA updates using presigned URLs to access a AWS S3 bucket. The firmware.bin file is stored in S3. I generate a presigned URL with 15 minute expiry. I send the presigned URL to the ESP over MQTT. The payload is as follows:

Code: Select all

I (60877) OTA_MQTT: MQTT message received (1008 bytes)
I (60877) OTA_MQTT: MQTT message received (210 bytes)
I (60887) OTA_MQTT: Full payload assembled (1218 bytes): {
    "ota": {
        "url": "https://redacted.s3.amazonaws.com/hello-world-test.bin?AWSAccessKeyId=redacted-amz-security-token=redactedExpires=1768991558"
    }
}

My http config is as follows:

Code: Select all

esp_http_client_config_t http_config = {
        .url = url,
        .crt_bundle_attach = esp_crt_bundle_attach,
        .buffer_size = 8*1024,
    };

.buffer_size = 1024 was insuffucient. I know that 8*1024 is overkill, but I wanted to make sure that it will not be an issue. I did this because the ESP was unable to parse the URL. The URL was not being received in full, since it was longer than .buffer_size of 1024. The error was:

Code: Select all

E (90155) HTTP_CLIENT: Error parse url

As you can see, the payload itself is quite long, and I had to break it up in pieces and then re-assemble it. That was successful. However, when the ESP tries to download the firmware file, I get a out of buffer warning on HTTP_CLIENT:

Code: Select all

E (62727) HTTP_CLIENT: Out of buffer
E (62727) esp_https_ota: Failed to open HTTP connection: ESP_FAIL
E (62727) esp_https_ota: Failed to establish HTTP connection
E (62737) OTA_MQTT: OTA failed

In menuconfig, "(Top) → Component config → HTTP Server", I tried changing the "Max HTTP URI Length", but that didn't work. Not sure where to go from here.

Some other menuconfig options which may be relevant are:

Code: Select all

(Top) → Component config → ESP HTTP client                                                                                                                                                    Espressif IoT Development Framework Configuration                                               
[*] Enable https                                                                                                                               
[ ] Enable HTTP Basic Authentication                                                                                                           
[ ] Enable HTTP Digest Authentication                                                                                                          
[ ] Enable custom transport

Code: Select all

(Top) → Component config → HTTP Server                                                                                                                                                        Espressif IoT Development Framework Configuration                                               
(512) Max HTTP Request Header Length                                                                                                           
(2048) Max HTTP URI Length                                                                                                                      
[*] Use TCP_NODELAY socket option when sending HTTP error responses                                                                            
(32) Length of temporary buffer for purging data                                                                                               
[ ] Log purged content data at Debug level                                                                                                     
[ ] WebSocket server support                                                                                                                   
[ ] httpd_queue_work as blocking API   

Code: Select all

(Top) → Component config → ESP HTTPS OTA                                                                                                                                                      Espressif IoT Development Framework Configuration                                               
[ ] Provide decryption callback                                                                                                                
[ ] Allow HTTP for OTA (WARNING: ONLY FOR TESTING PURPOSE, READ HELP)  

Code: Select all

(Top) → Component config → ESP HTTPS server                                                                                                                                                   Espressif IoT Development Framework Configuration                                               
[ ] Enable ESP_HTTPS_SERVER component 

Here is my full code. Certs are stored in a certs folder in the root directory on my pc. I'm using IDF version 5.3.4. Code is being written for ESP32-S3, 16MB Flash, 2MB PSRAM (not using PSRAM, but its there).

Code: Select all

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"

#include "esp_system.h"
#include "esp_log.h"
#include "esp_event.h"
#include "esp_wifi.h"
#include "nvs_flash.h"

#include "mqtt_client.h"
#include "esp_https_ota.h"
#include "esp_http_client.h"
#include "esp_crt_bundle.h"

#include "cJSON.h"

#define WIFI_SSID      "SSIDHERE"
#define WIFI_PASS      "PWDHERE"

#define AWS_IOT_ENDPOINT  "ENDPOINTHERE-ats.iot.eu-west-1.amazonaws.com"
#define AWS_IOT_PORT      8883
#define DEVICE_ID         "DEVICEIDHERE"

#define MQTT_SUB_TOPIC    "devwork/ota"
#define MQTT_PUB_TOPIC    "devwork/heartbeat"

#define MQTT_MAX_PAYLOAD 2048

static char mqtt_payload_buf[MQTT_MAX_PAYLOAD];
static int mqtt_payload_len = 0;

static const char *TAG = "OTA_MQTT";

/* ===========================
   AWS IoT Certificates
   =========================== */

extern const uint8_t aws_root_ca_pem_start[] asm("_binary_AmazonRootCA1_pem_start");
extern const uint8_t aws_root_ca_pem_end[]   asm("_binary_AmazonRootCA1_pem_end");

extern const uint8_t device_cert_pem_start[] asm("_binary_certificate_pem_crt_start");
extern const uint8_t device_cert_pem_end[]   asm("_binary_certificate_pem_crt_end");

extern const uint8_t device_private_key_start[] asm("_binary_private_pem_key_start");
extern const uint8_t device_private_key_end[]   asm("_binary_private_pem_key_end");

/* ===========================
   Event group
   =========================== */

static EventGroupHandle_t wifi_event_group;
#define WIFI_CONNECTED_BIT BIT0

static esp_mqtt_client_handle_t mqtt_client = NULL;

/* ===========================
   WiFi Event Handler
   =========================== */

static void wifi_event_handler(void *arg,
                               esp_event_base_t event_base,
                               int32_t event_id,
                               void *event_data)
{
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
        esp_wifi_connect();
    } else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
        ESP_LOGI(TAG, "WiFi disconnected, retrying...");
        esp_wifi_connect();
    } else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
        ESP_LOGI(TAG, "WiFi connected");
        xEventGroupSetBits(wifi_event_group, WIFI_CONNECTED_BIT);
    }
}

/* ===========================
   WiFi Init
   =========================== */

static void wifi_init(void)
{
    wifi_event_group = xEventGroupCreate();

    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());

    esp_netif_create_default_wifi_sta();

    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK(esp_wifi_init(&cfg));

    ESP_ERROR_CHECK(esp_event_handler_register(WIFI_EVENT,
                                                ESP_EVENT_ANY_ID,
                                                &wifi_event_handler,
                                                NULL));

    ESP_ERROR_CHECK(esp_event_handler_register(IP_EVENT,
                                                IP_EVENT_STA_GOT_IP,
                                                &wifi_event_handler,
                                                NULL));

    wifi_config_t wifi_config = {
        .sta = {
            .ssid = WIFI_SSID,
            .password = WIFI_PASS,
        },
    };

    ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
    ESP_ERROR_CHECK(esp_wifi_start());

    ESP_LOGI(TAG, "Connecting to WiFi...");
    xEventGroupWaitBits(wifi_event_group,
                        WIFI_CONNECTED_BIT,
                        pdFALSE,
                        pdTRUE,
                        portMAX_DELAY);
}

/* ===========================
   OTA Task
   =========================== */

static void ota_task(void *param)
{
    char *url = (char *)param;
    ESP_LOGI(TAG, "Starting OTA from URL: %s", url);

    esp_http_client_config_t http_config = {
        .url = url,
        .crt_bundle_attach = esp_crt_bundle_attach,
        .buffer_size = 4*1024, // default is 1024
    };


    esp_https_ota_config_t ota_config = {
        .http_config = &http_config,
    };

    esp_err_t ret = esp_https_ota(&ota_config);
    if (ret == ESP_OK) {
        ESP_LOGI(TAG, "OTA successful, restarting...");
        esp_restart();
    } else {
        ESP_LOGE(TAG, "OTA failed");
    }

    free(url); // Free the duplicated string
    vTaskDelete(NULL);
}

/* ===========================
   MQTT Event Handler
   =========================== */

static void mqtt_event_handler(void *handler_args,
                               esp_event_base_t base,
                               int32_t event_id,
                               void *event_data)
{
    esp_mqtt_event_handle_t event = event_data;

    switch (event->event_id) {
    case MQTT_EVENT_CONNECTED:
        ESP_LOGI(TAG, "MQTT connected");
        esp_mqtt_client_subscribe(mqtt_client, MQTT_SUB_TOPIC, 1);
        break;

    case MQTT_EVENT_DATA:
    ESP_LOGI(TAG, "MQTT message received (%d bytes)", event->data_len);

    if (strncmp(event->topic, MQTT_SUB_TOPIC, event->topic_len) == 0) {
        // Check buffer overflow
        if (mqtt_payload_len + event->data_len >= MQTT_MAX_PAYLOAD) {
            ESP_LOGE(TAG, "MQTT payload too large, resetting buffer");
            mqtt_payload_len = 0;
            break;
        }

        // Append new chunk
        memcpy(mqtt_payload_buf + mqtt_payload_len, event->data, event->data_len);
        mqtt_payload_len += event->data_len;
        mqtt_payload_buf[mqtt_payload_len] = '\0'; // null terminate for JSON

        // Check if payload contains end of JSON ('}')
        if (strchr(mqtt_payload_buf, '}')) {
            ESP_LOGI(TAG, "Full payload assembled (%d bytes): %s", mqtt_payload_len, mqtt_payload_buf);

            // Parse JSON
            cJSON *root = cJSON_Parse(mqtt_payload_buf);
            if (!root) {
                ESP_LOGE(TAG, "Failed to parse JSON");
                mqtt_payload_len = 0; // reset buffer
                break;
            }

            cJSON *ota_obj = cJSON_GetObjectItem(root, "ota");
            if (ota_obj) {
                cJSON *url_item = cJSON_GetObjectItem(ota_obj, "url");
                if (cJSON_IsString(url_item) && url_item->valuestring) {
                    ESP_LOGI(TAG, "OTA URL extracted: %s", url_item->valuestring);
                    char *url_copy = strdup(url_item->valuestring);
                    if (url_copy) {
                        xTaskCreate(ota_task, "ota_task", 8192, url_copy, 5, NULL);
                    }
                }
            }

            cJSON_Delete(root);
            mqtt_payload_len = 0; // reset buffer for next message
        }
    }
    break;



    default:
        break;
    }
}

/* ===========================
   MQTT Init
   =========================== */

static void mqtt_init(void)
{
    esp_mqtt_client_config_t mqtt_cfg = {
        .broker.address.hostname = AWS_IOT_ENDPOINT,
        .broker.address.port = 8883,
        .broker.address.transport = MQTT_TRANSPORT_OVER_SSL,

        .credentials.client_id = DEVICE_ID,
        .broker.verification.certificate = (const char *)aws_root_ca_pem_start,
        .credentials.authentication.certificate = (const char *)device_cert_pem_start,
        .credentials.authentication.key = (const char *)device_private_key_start,

        .network.reconnect_timeout_ms = 3000,
        .session.keepalive = 240,
    };

    mqtt_client = esp_mqtt_client_init(&mqtt_cfg);
    esp_mqtt_client_register_event(mqtt_client,
                                   ESP_EVENT_ANY_ID,
                                   mqtt_event_handler,
                                   NULL);

    esp_mqtt_client_start(mqtt_client);
}

/* ===========================
   Heartbeat Task
   =========================== */

static void heartbeat_task(void *arg)
{
    while (1) {
        esp_mqtt_client_publish(
            mqtt_client,
            MQTT_PUB_TOPIC,
            "{\"status\":\"alive\"}",
            0,
            1,
            0
        );

        vTaskDelay(pdMS_TO_TICKS(60 * 1000)); // 1 minute
    }
}

/* ===========================
   Main
   =========================== */

void app_main(void)
{
    ESP_ERROR_CHECK(nvs_flash_init());

    wifi_init();
    mqtt_init();

    xTaskCreate(heartbeat_task, "heartbeat_task", 4096, NULL, 5, NULL);
}

[anees.ahmad.jaffer]

Update: I get the same error when using the "advanced_https_ota_example" project , the error being:

Code: Select all

E (10797) HTTP_CLIENT: Out of buffer
E (10797) esp_https_ota: Failed to open HTTP connection: ESP_FAIL
E (10797) esp_https_ota: Failed to establish HTTP connection
E (10807) advanced_https_ota_example: ESP HTTPS OTA Begin failed

The only modifications I did to that was to hardcode my presigned URL, give it the AWS root CA cert, and to change the http config to this:

Code: Select all

    esp_http_client_config_t config = {
        .url = CONFIG2_EXAMPLE_FIRMWARE_UPGRADE_URL,
        .timeout_ms = CONFIG_EXAMPLE_OTA_RECV_TIMEOUT,
        .crt_bundle_attach = esp_crt_bundle_attach,
        .keep_alive_enable = true,
        .buffer_size = 4*1024, // default is 1024
    };

[anees.ahmad.jaffer]

Solved. You need to adjust buffer_size_tx, not buffer_size, in esp_http_client_config_t config