Web server pages timing out / memory issue?

[BivvyCoder]

I'm using an ESP32 in an IoT project, logging data from a number of sensors.
I've included a built in WiFi access point and web server so I can connect a computer to the ESP32 to change settings and also to read the logged data. I've also got MQTT running as, eventually, I want the data sent directly to an MQTT server.
The basic code works fine, but I'm running into problems if the web page delivered by the ESP32 is too large.

If I open the web page immediately after a reboot it loads fine.
In the following log data I'm displaying available heap memory either side of the malloc to reserve memory for the web page

Here is a successful page load

Code: Select all

I (21855) Control Handler: Loading Control Page
W (21865) HeapDebug: Free heap size: 73240
W (21865) HeapDebug: Largest free block: 21504
W (21865) HeapDebug: Heap info: total_free_bytes=71260, largest_free_block=21504, minimum_free_bytes=12028
I (21875) Control Handler: Page Size: 21036: Payload Size: 104, Memory Reserved: 21141
W (21875) HeapDebug: Free heap size: 51668
W (21885) HeapDebug: Largest free block: 20480
W (21885) HeapDebug: Heap info: total_free_bytes=51668, largest_free_block=20480, minimum_free_bytes=12028
I (21915) Control Handler: Page delivered

A few seconds later the MQTT task starts and appears to be very (heap) memory hungry, so there's not enough heap available to load my control page:

Code: Select all

I (81585) Control Handler: Loading Motor Control Page
W (81585) HeapDebug: Free heap size: 35680
W (81585) HeapDebug: Largest free block: 17408
W (81595) HeapDebug: Heap info: total_free_bytes=35680, largest_free_block=17408, minimum_free_bytes=12028
I (81605) Control Handler: Page Size: 21036: Payload Size: 104, Memory Reserved: 21141
W (81605) HeapDebug: Free heap size: 35680
W (81615) HeapDebug: Largest free block: 17408
W (81615) HeapDebug: Heap info: total_free_bytes=35680, largest_free_block=17408, minimum_free_bytes=12028
E (81625) Control Handler: Error allocating memory for control page
W (81635) httpd_uri: httpd_uri: uri handler execution failed

Here's a snippet from the c code to create and deliver the web page:

Code: Select all

	uint32_t payload_size = strlen(settings) + strlen(data1) + strlen(data2);
            
        check_heap_fragmentation();
        ESP_LOGI(TAG,"Page Size: %zu: Payload Size: %lu, Memory Reserved: %lu",c_html_size, payload_size, payload_size + c_html_size +1);
        char *control_page = malloc(c_html_size  +payload_size +1);
        check_heap_fragmentation();
        
        if (control_page == NULL) {
            ESP_LOGE(TAG,"Error allocating memory for control page");
            
            return ESP_ERR_NO_MEM;
        }
        sprintf(control_page, control_page_start, settings, data1, data2);
           
        closeHeader(req);

        esp_err_t out = httpd_resp_send(req, control_page, HTTPD_RESP_USE_STRLEN);
        if (out != ESP_OK) ESP_LOGE(TAG,"Error sending page: %d",out);
        free(control_page);
        ESP_LOGI(TAG,"Page delivered");
        
    } 

    return out;
}

control_page_start is the location in memory of the HTML outline, which includes %s placeholders for settings, data1 and data2.

Here's the memory allocation report after the last build

Code: Select all

                            Memory Type Usage Summary
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Memory Type/Section ┃ Used [bytes] ┃ Used [%] ┃ Remain [bytes] ┃ Total [bytes] ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Flash Code          │       830138 │      9.9 │        7558438 │       8388576 │
│    .text            │       830138 │      9.9 │                │               │
│ Flash Data          │       502248 │      1.5 │       33052152 │      33554400 │
│    .rodata          │       501992 │      1.5 │                │               │
│    .appdesc         │          256 │      0.0 │                │               │
│ DIRAM               │       144688 │    42.34 │         197072 │        341760 │
│    .text            │        92771 │    27.15 │                │               │
│    .data            │        29096 │     8.51 │                │               │
│    .bss             │        22664 │     6.63 │                │               │
│ IRAM                │        16383 │    99.99 │              1 │         16384 │
│    .text            │        15356 │    93.73 │                │               │
│    .vectors         │         1027 │     6.27 │                │               │
│ RTC SLOW            │         6416 │    78.32 │           1776 │          8192 │
│    .data            │         6416 │    78.32 │                │               │
│ RTC FAST            │          308 │     3.76 │           7884 │          8192 │
│    .force_fast      │           28 │     0.34 │                │               │
│    .rtc_reserved    │           24 │     0.29 │                │               │
└─────────────────────┴──────────────┴──────────┴────────────────┴───────────────┘
Total image size: 1477080 bytes (.bin may be padded larger)

Any suggestions on how I can resolve this much appreciated.
I'm using an ESP32-S3-WROOM-1N16. Internal RAM only - no additional PSRAM.

Many thanks

[MicroController]

Option a: Process and send the HTML template in smaller chunks,
option b: Use JavaScript to load the dynamic data from the server and insert it into the document.

[BivvyCoder]

Thanks - I've experimented with both approaches when trying to solve this problem so will refactor my code to make this the standard approach throughout.
The challenge is the way that the pages are built - Treating the HTML as a large string, and loading the latest data into the full html txt in memory to replace %s placeholder.
I'll change the code so the basic HTML is served up as chunks and then use javascript to poll for the latest data.