Sending large data over MQTT

Posts: 186
Joined: Thu Nov 17, 2016 2:33 pm

Sending large data over MQTT

Postby davdav » Tue Jul 30, 2019 1:18 pm

Hi everybody,

I have to send multiple files (about 1000) content (for a total of about 1 MB or more) via MQTT to a remote broker.

Since ESP32-WROOM-32 has limited RAM I can't read all files and put in a buffer before publish it.

Is there any way to perform a "chuncked/segmented" publishing with esp-mqtt library?
It seems that

Code: Select all

need to know the length of the message and the pointer to content, but in my case I can't get the total length and I would like to open the file one by one and send the data during the publish connection.


Posts: 2
Joined: Thu Mar 05, 2020 1:24 pm

Re: Sending large data over MQTT

Postby agatrost1620 » Fri May 22, 2020 9:21 pm

@davdav were you able to figure out a solution?

I am trying to send 1 file that is in the flash using mqtt. I do not have enough memory to allocate for the size of the file, nor can a create an array large enough either. the file at this time is 48kB, and I would like to be able to not worry about the size, because it could be as big as 256kB in the future.

Is there a way to use a FILE * to send using mqtt?

Thanks in advance,

Posts: 358
Joined: Mon Jun 04, 2018 2:47 pm

Re: Sending large data over MQTT

Postby PeterR » Sat May 23, 2020 1:20 pm

Assuming qos is 0 then it should be possible.
Not entirely clear on your system but if you store your 'file' as raw data in FLASH then you could use the existing API. Of course you would not be able to retain data.
If you want to use a file within a file system then you would have to dig into the IDF library. MQTT is built on a websocket and so you can stream (assuming qos 0) by which there is no need to complete within a single API call. I only saw block transfers in the IDF but there is no reason why not from a file (using qos=0)
Maybe worth a feature request or a little project for you.

Another approach might be to create many segment endpoints e.g. myfile/seg0 ... myfile/seg100 etc.
& I also believe that IDF CAN should be fixed.

Who is online

Users browsing this forum: Google [Bot] and 18 guests