publish big data amount to google core iot

[ledtom]

Hy,
I am currently developing a project, which is measuring frequencies with a sample rate of max. 500 Hz. The sample rate is realized with the esp_timer. In the callback function I measure the values and send them to a queue. Another task is reading the datas from the queue and saves them to a sd card.
So far so good, the above description works well. But due to saving the data to a sd card, I also want to publish them to google core iot, so I am able to access them from anywhere. Unfortunatelly, I don´t have any experience in sending such amount of datas.
I have already implemented the web framework to access the google cloud iot core and I am already able to receive commands and to publish small amount of datas during the measurement (Created a delayed publish task, that just sends "dummy" every second).
I´ve tried to create another queue, where the values, that should be published are hold. After this, I created a while loop in my delayed publish task, that reads out the values of the queue and publishs them to the cloud. However, the last part is somehow not working, as I always get an "heap out of memmory error".

Ideally I would like to create a task, that is waiting until a value is in the loop and then publishs this value immediately (in the worst case, every 2ms a new value is added to the queue), instead of a delayed publish task.

Does anyone know, if this is even possible? Or do I face the wrong approach do implement this project?

Any help is very appreciated.

Many thanks in advance
Thomas