Sending partition content directly to cloud service without passing through SRAM Memory

[pablocamacho]

We have developed our own board using an Espressif ESP32 (Over the chip we can read: ESP-WROOM-32, but, when flashing to the chip, the ESP-IDF prints to the console: ESP32D0WDQ5).

We have configured the partitions in an external SPI Flash (Winbond product) in such a way that one of those partitions is devoted to storing the core-dump file whenever it is generated.

We want to send that core-dump file to an external repository, which should be a cloud service. But without passing through the SRAM Memory.

We would like to use an Espressif function to execute this process (sending the core-dump file stored in core-dump partition) and we want to do this (I repeat) WITHOUT previously copying the core-dump file (64 KB) to the SRAM.

For example, we might use the function described here:

https://docs.espressif.com/projects/esp ... p_handle_t):

esp_err_t esp_partition_mmap(const esp_partition_t *partition, size_t offset, size_t size, spi_flash_mmap_memory_t memory, const void **out_ptr, spi_flash_mmap_handle_t *out_handle)

Is this possible?

I mean: Can we access the core-dump partition, pick up the core-dump file and send it directly to a cloud service without previously copying that core-dump file to the SRAM memory?

We want: From flash to cloud.
We do not want: From flash to RAM and from RAM to cloud (this is because we have legacy devices, with limited RAM memory).

I have found the following restriction which is regarding the external RAM (not regarding the SPI flash memory):

"External RAM uses the same cache region as the external flash. This means that frequently accessed variables in external RAM can be read and modified almost as quickly as in internal ram. However, when accessing large chunks of data (>32 KB), the cache can be insufficient, and speeds will fall back to the access speed of the external RAM. Moreover, accessing large chunks of data can “push out” cached flash, possibly making the execution of code slower afterwards."

Source: https://docs.espressif.com/projects/esp ... l-ram.html

So, the restriction applies to the external RAM, not to the external SPI flash memory.

Am I correct?

Thus, if this movement of a 64 KB content of a SPI flash partition can be send to cloud server, could you please, provide coding examples and appropriate functions to do this, as well as correct links to Espressif online documentation?

I look forward to your feedback.
Thanks in advance and best regards!

[ESP_Sprite]

In general, yes, you can. Easiest way would be to use spi_flash_mmap to map the partition into memory space, then use that as your buffer to send.

> So, the restriction applies to the external RAM, not to the external SPI flash memory.
The external SPI also has this restriction, as it goes through the same cache. Reading a chunk of >=32K will slow down execution.

[pablocamacho]

Hi, ESP32 Forum!

Again here with a question.

First of all, thanks for your answer, @Esp_Sprite!

I am trying to figure out how to copy to cloud an entire partition, got from SPI Flash memory, already mapped to Caché memory, but without passing through the SRAM.

Again: here is the prototype:

esp_err_t esp_partition_mmap(const esp_partition_t *partition, size_t offset, size_t size, spi_flash_mmap_memory_t memory, const void **out_ptr, spi_flash_mmap_handle_t *out_handle)

However, what I was able to implement successfully, so far, is using a copy to SRAM:

if((esp_partition_mmap(part, 0, part->size, SPI_FLASH_MMAP_DATA, &mypartition_map_ptr, &map_handle)) == ESP_OK)
{
memcpy(mymemory_ptr, mypartition_map_ptr, MYPARTITION_SIZE);
}

esp_http_client_set_post_field(client, mymemory_ptr, MYPARTITION_SIZE);

Where:
mypartition_map_ptr: is the pointer to the Caché memory which contains the partition retrieved from SPI Flash memory.
mymemory_ptr: is a pointer to SRAM (where I have previously allocated the corresponding size required to contain the SPI Flash partition).

As you can see, the http function is using the pointer to SRAM content, mymemory_ptr, instead of mypartition_map_ptr.

Consequently, instead of doing: SPI Flash > Caché > Cloud <----- REQUIRED SOLUTION
I am doing: SPI Flash > Caché > SRAM > Cloud
which is redundant, because I can simply do: SPI Flash > SRAM > Cloud, avoiding one step.

But the point is I need the one I marked as "REQUIRED SOLUTION".

I wonder if the issue I found when trying to use directly mypartition_map_ptr is just a casting one (Note: the code is compiling successfully). For instance:

esp_http_client_set_post_field(client, (casting_here) mypartition_map_ptr, MYPARTITION_SIZE);

Of course, I have attempted doing it, but without success.

Any concrete advice on this, please?

Thanks in advance!!

[boarchuz]

I have attempted doing it, but without success.

What is/isn't happening?
Is mypartition_map_ptr still valid when you esp_http_client_perform?

[pablocamacho]

Hi, ESP-32 Forum.

@boarchuz: Thanks for your interest in providing help!

To respond @boarchuz questions:
What is/isn't happening?
Is mypartition_map_ptr still valid when you esp_http_client_perform?

Yes, mypartition_map_ptr is still valid when I execute esp_http_client_perform.
I have tested it doing a memcpy, and getting the Flash partition content pointed by mypartition_map_ptr into a RAM buffer, and printing out that content to stdout.

Note 1: We are working with 2 versions of hardware. The older version has limited memory. Thus, we cannot use the RAM memory to create a buffer, get a copy of the required Flash memory partition and send it to the cloud. Instead, with the newer version, which has more RAM memory, we do not have this issue: we can create the buffer and use it to send the information to the cloud.

Note 2: I have created a free account here https://pipedream.com and here https://putsreq.com/ and was able to successfully sent to cloud the content of the partition of the newest hardware version.

My questions:
1) In this prototype:

esp_partition_mmap(const esp_partition_t *partition, size_t offset, size_t size, spi_flash_mmap_memory_t memory, const void **out_ptr, spi_flash_mmap_handle_t *out_handle)

the paramente out_ptr is a pointer to pointer.
However, I have not found in the documentation where is pointing to each level of indirection.
Could you please clarify this?

2) ALL the code I have found as an example of mapping the Flash memory, is using the RAM memory to create a buffer. I was not able to find out neither one example on how to properly handle the content of one Flash memory partition and send it to the cloud without creating a buffer on the RAM.

Could you please provide some working example on how to do this?

I suppose it should be something like getting the out_ptr parameter provided in the esp_partition_mmap and using it in this function:

esp_http_client_set_post_field(esp_http_client_handle_t client, const char *data, int len);

So, how to transfer the information pointed by out_ptr to the const char* data parameter, without having a buffer in RAM memory?

Going deeper: I persist on this question because, @ESP_Sprite responded to me (in my firs post) that it is possible to extract information from the Flash, using the Caché memory and sending it to the cloud without using the RAM memory.

If you are sure this is possible, please, provide a concrete example.
If you are sure of the opposite, please, also let me know it clearly.

Thanks in advance for your help!