Sending large data over MQTT

davdav
Posts: 187
Joined: Thu Nov 17, 2016 2:33 pm

Sending large data over MQTT

Postby davdav » Tue Jul 30, 2019 1:18 pm

Hi everybody,

I have to send multiple files (about 1000) content (for a total of about 1 MB or more) via MQTT to a remote broker.

Since ESP32-WROOM-32 has limited RAM I can't read all files and put in a buffer before publish it.

Is there any way to perform a "chuncked/segmented" publishing with esp-mqtt library?
It seems that

Code: Select all

esp_mqtt_client_publish
need to know the length of the message and the pointer to content, but in my case I can't get the total length and I would like to open the file one by one and send the data during the publish connection.


Thanks

agatrost1620
Posts: 2
Joined: Thu Mar 05, 2020 1:24 pm

Re: Sending large data over MQTT

Postby agatrost1620 » Fri May 22, 2020 9:21 pm

@davdav were you able to figure out a solution?

I am trying to send 1 file that is in the flash using mqtt. I do not have enough memory to allocate for the size of the file, nor can a create an array large enough either. the file at this time is 48kB, and I would like to be able to not worry about the size, because it could be as big as 256kB in the future.

Is there a way to use a FILE * to send using mqtt?

Thanks in advance,

PeterR
Posts: 370
Joined: Mon Jun 04, 2018 2:47 pm

Re: Sending large data over MQTT

Postby PeterR » Sat May 23, 2020 1:20 pm

Assuming qos is 0 then it should be possible.
Not entirely clear on your system but if you store your 'file' as raw data in FLASH then you could use the existing API. Of course you would not be able to retain data.
If you want to use a file within a file system then you would have to dig into the IDF library. MQTT is built on a websocket and so you can stream (assuming qos 0) by which there is no need to complete within a single API call. I only saw block transfers in the IDF but there is no reason why not from a file (using qos=0)
Maybe worth a feature request or a little project for you.

Another approach might be to create many segment endpoints e.g. myfile/seg0 ... myfile/seg100 etc.
& I also believe that IDF CAN should be fixed.

davdav
Posts: 187
Joined: Thu Nov 17, 2016 2:33 pm

Re: Sending large data over MQTT

Postby davdav » Mon May 25, 2020 9:19 pm

agatrost1620 wrote:
Fri May 22, 2020 9:21 pm
@davdav were you able to figure out a solution?

I am trying to send 1 file that is in the flash using mqtt. I do not have enough memory to allocate for the size of the file, nor can a create an array large enough either. the file at this time is 48kB, and I would like to be able to not worry about the size, because it could be as big as 256kB in the future.

Is there a way to use a FILE * to send using mqtt?

Thanks in advance,
Hello @agatrost1620,
I managed on "application level", i.e. I implemented a "protocol" between esp32 and the backend of the mqtt broker on the server. In practice it is a sort of segmentation as @PeterR suggested.

I have also raised a question in esp-mqtt github repo. See here:

https://github.com/espressif/esp-mqtt/issues/123

maybe it can help to find a good solution for you.


Regards

Who is online

Users browsing this forum: No registered users and 10 guests