How to improve http client time to download image?

[karunt]

Connected ESP32-S3 chip to node.js server on local machine. Smart phone app instructs ESP32-S3 (over mqtt server) to download image from node server. It takes ~30 seconds for ESP32-S3 to download ~100KB image and display on LCD screen (most of the time consumed by downloading process). Why so slow? I've read something about poor wifi channel utilization, but not sure how to go about implementing wifi channel utilization to improve performance. How else can I improve my code to reduce the time it takes to download and save the image data in a file on spiffs? Here's my code:

Code: Select all

void http_get(char *file){
    char output_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0}; //MAX_HTTP_OUTPUT_BUFFER = 2048 (max value tested ~2300 before stack overflow during image download)
    int data_read;
    char uri[256];
    char binary_filename[strlen(file)+2];
    snprintf(binary_filename, strlen(file)+1, "%s", file);
    strtok(binary_filename, ".");
    strcat(binary_filename, ".bin");

    snprintf(uri, sizeof(uri), "%s://%s:%s/esp?file=%s", httpServer, httpIpAddress, httpPort, file);
    esp_http_client_config_t config = {
        .url = uri
    };

    esp_http_client_handle_t client = esp_http_client_init(&config);

    esp_http_client_set_method(client, HTTP_METHOD_GET);
    esp_err_t err = esp_http_client_open(client, 0);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));
    } else {
        int content_length = esp_http_client_fetch_headers(client);
        if (content_length < 0) {
            ESP_LOGE(TAG, "HTTP client fetch headers failed");
        } else {
            do {
                data_read = esp_http_client_read_response(client, output_buffer, MAX_HTTP_OUTPUT_BUFFER);
                create_file_app(binary_filename, output_buffer, data_read);
                if (data_read >= 0) {
                    esp_http_client_get_status_code(client),
                    esp_http_client_get_content_length(client));
                } else {
                    ESP_LOGE(TAG, "Failed to read response");
                }
            }
            while (data_read > 0);
            xEventGroupSetBits(main_event_group, FILE_DOWNLOAD_COMPLETE_BIT);
            ESP_LOGI(TAG, "Data download compeleted");
            display_img(binary_filename);
        }
    }
    esp_http_client_close(client);
}

Here are my WiFi related sdkconfig settings:

Code: Select all

CONFIG_ESP_WIFI_ENABLED=y
CONFIG_ESP_WIFI_STATIC_RX_BUFFER_NUM=10
CONFIG_ESP_WIFI_DYNAMIC_RX_BUFFER_NUM=32
# CONFIG_ESP_WIFI_STATIC_TX_BUFFER is not set
CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER=y
CONFIG_ESP_WIFI_TX_BUFFER_TYPE=1
CONFIG_ESP_WIFI_CACHE_TX_BUFFER_NUM=32
CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER_NUM=32
# CONFIG_ESP_WIFI_CSI_ENABLED is not set
CONFIG_ESP_WIFI_AMPDU_TX_ENABLED=y
CONFIG_ESP_WIFI_TX_BA_WIN=6
CONFIG_ESP_WIFI_AMPDU_RX_ENABLED=y
CONFIG_ESP_WIFI_RX_BA_WIN=6
# CONFIG_ESP_WIFI_AMSDU_TX_ENABLED is not set
CONFIG_ESP_WIFI_NVS_ENABLED=y
CONFIG_ESP_WIFI_TASK_PINNED_TO_CORE_0=y
# CONFIG_ESP_WIFI_TASK_PINNED_TO_CORE_1 is not set
CONFIG_ESP_WIFI_SOFTAP_BEACON_MAX_LEN=752
CONFIG_ESP_WIFI_MGMT_SBUF_NUM=32
CONFIG_ESP_WIFI_IRAM_OPT=y
# CONFIG_ESP_WIFI_EXTRA_IRAM_OPT is not set
CONFIG_ESP_WIFI_RX_IRAM_OPT=y
CONFIG_ESP_WIFI_ENABLE_WPA3_SAE=y
CONFIG_ESP_WIFI_ENABLE_SAE_PK=y
CONFIG_ESP_WIFI_SOFTAP_SAE_SUPPORT=y
CONFIG_ESP_WIFI_ENABLE_WPA3_OWE_STA=y
# CONFIG_ESP_WIFI_SLP_IRAM_OPT is not set
# CONFIG_ESP_WIFI_FTM_ENABLE is not set
CONFIG_ESP_WIFI_STA_DISCONNECTED_PM_ENABLE=y
# CONFIG_ESP_WIFI_GCMP_SUPPORT is not set
# CONFIG_ESP_WIFI_GMAC_SUPPORT is not set
CONFIG_ESP_WIFI_SOFTAP_SUPPORT=y
# CONFIG_ESP_WIFI_SLP_BEACON_LOST_OPT is not set
CONFIG_ESP_WIFI_ESPNOW_MAX_ENCRYPT_NUM=7
CONFIG_ESP_WIFI_MBEDTLS_CRYPTO=y
CONFIG_ESP_WIFI_MBEDTLS_TLS_CLIENT=y
# CONFIG_ESP_WIFI_WAPI_PSK is not set
# CONFIG_ESP_WIFI_SUITE_B_192 is not set
# CONFIG_ESP_WIFI_11KV_SUPPORT is not set
# CONFIG_ESP_WIFI_MBO_SUPPORT is not set
# CONFIG_ESP_WIFI_DPP_SUPPORT is not set
# CONFIG_ESP_WIFI_11R_SUPPORT is not set
# CONFIG_ESP_WIFI_WPS_SOFTAP_REGISTRAR is not set

Finally, a snippet of my Debug level log:

I (106763) spiffs: Opening file: /spiffs/cup00.bin
D (106813) event: no handlers have been registered for event ESP_HTTP_CLIENT_EVENT:4 posted to loop 0x3fcbf564
I (106833) spiffs: File written
I (106833) HTTP_CLIENT: HTTP GET Status = 200, content_length = 691204
D (106833) HTTP_CLIENT: is_data_remain=1, is_chunked=0, content_length=691204
D (106843) HTTP_CLIENT: need_read=2048, byte_to_read=512, rlen=512, ridx=0
D (106843) HTTP_CLIENT: http_on_body 512
D (106853) HTTP_CLIENT: is_data_remain=1, is_chunked=0, content_length=691204
D (106853) event: no handlers have been registered for event ESP_HTTP_CLIENT_EVENT:4 posted to loop 0x3fcbf564
D (106853) HTTP_CLIENT: need_read=1536, byte_to_read=512, rlen=512, ridx=512
D (106873) HTTP_CLIENT: http_on_body 512
D (106873) HTTP_CLIENT: is_data_remain=1, is_chunked=0, content_length=691204
D (106883) HTTP_CLIENT: need_read=1024, byte_to_read=512, rlen=512, ridx=1024
D (106893) HTTP_CLIENT: http_on_body 512
D (106883) event: no handlers have been registered for event ESP_HTTP_CLIENT_EVENT:4 posted to loop 0x3fcbf564
D (106893) HTTP_CLIENT: is_data_remain=1, is_chunked=0, content_length=691204
D (106913) HTTP_CLIENT: need_read=512, byte_to_read=512, rlen=512, ridx=1536
D (106913) event: no handlers have been registered for event ESP_HTTP_CLIENT_EVENT:4 posted to loop 0x3fcbf564
D (106923) HTTP_CLIENT: http_on_body 512

[MicroController]

Writing to SPIFFS (create_file_app()) may be taking most of the time. A larger output_buffer[] (on the heap) may help reduce its overhead.

[karunt]

Increased output_buffer on the heap with

Code: Select all

char *output_buffer = (char *) calloc(sizeof(char), 8192)

. Also adjusted

Code: Select all

MAX_HTTP_OUTPUT_BUFFER

in

Code: Select all

esp_http_client_read_response(client, output_buffer, MAX_HTTP_OUTPUT_BUFFER)

to 8192. Definitely reduced the number of iterations in the do while loop to retrieve data, but overall increased time to display the image. Content length from the http request is to the tune of 675KB. But allocating that much to output_buffer from heap causes core dump to occur almost instantaneously after the http read process beings.

Guru Meditation Error: Core 0 panic'ed (StoreProhibited). Exception was unhandled.

Core 0 register dump:
PC : 0x40056f60 PS : 0x00060530 A0 : 0x8207cefd A1 : 0x3fcda920
0x40056f60: memcpy in ROM

A2 : 0x00000000 A3 : 0x3fcbe050 A4 : 0x0000009f A5 : 0x00000000
A6 : 0x3c078005 A7 : 0x39ffc639 A8 : 0x00000000 A9 : 0x3fcda8b0
A10 : 0x00000004 A11 : 0x00000000 A12 : 0x00000010 A13 : 0x0000009f
A14 : 0x000000c6 A15 : 0x0000005a SAR : 0x0000001c EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000000 LBEG : 0x40056f5c LEND : 0x40056f72 LCOUNT : 0x00000008
0x40056f5c: memcpy in ROM

0x40056f72: memcpy in ROM

Backtrace: 0x40056f5d:0x3fcda920 |<-CORRUPTED
0x40056f5d: memcpy in ROM

Any other suggestions? Would it make better sense to write to RAM instead and avoid saving data into a spiffs file altogether? This is unfortunately out of my league, so some guidance is appreciated. Thanks.

[MicroController]

Of course, buffering all data only in RAM would speed things up a lot.
However, the basic -S3 has "only" ~500kB of RAM on chip, of which you might be able to use 200-400kB for a buffer. Not enough for 675kB of image data in any case.
There are -S3 variants (chips and modules) available with "external" PSRAM (4-16MB, I believe) which could solve the problem.

Using a different file system (LittleFS or FAT) in flash may also improve speed.