Transferring STL containers using esp_event_post_to

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Transferring STL containers using esp_event_post_to

Postby istokm » Tue Jul 28, 2020 6:45 am

Is there a feasible way of transferring STL containers (std::set, std::map, etc) using the esp_event_post_to?

The specific container I want to transfer is http_message_t:

Code: Select all

typedef struct http_header_t {
	std::string key;
	std::string val;
	
	bool operator< (const http_header_t& x) const { return key < x.key; };
} http_header_t;

typedef struct {
	std::string		data;
	std::set<http_header_t>	headers;
	int			status_code;
} http_message_t;
I've spent the last 4 days thinking about this, and all solutions except converting the strings and set to native C constructs seem unfeasible. But then there is almost no point in using them in the first place - except for the braindead simple handling of incoming messages - but with no benefit to the user event handler:

Code: Select all

case HTTP_EVENT_ON_HEADER:
	rx_buffer_->headers.insert({evt->header_key, evt->header_value});
	break;
case HTTP_EVENT_ON_DATA:
	rx_buffer_->data += event_data;
	break;
case HTTP_EVENT_ON_FINISH:
	rx_buffer_->status_code = esp_http_client_get_status_code(esp_client_handle_);
	esp_event_post_to(...)
	...

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Re: Transferring STL containers using esp_event_post_to

Postby istokm » Wed Jul 29, 2020 8:47 am

I've just realized that any containers that include pointer types (char* in struct, etc) cannot be used, that is a very strict limitation of this implementation... A much better implementation would be to just treat the pointer like shared_ptr does, and free it once all event handlers exit - the user would have to allocate it on heap, but still, miles better than this, as I would have to serialize all data I want to send through to the handlers.

ESP_jakob
Posts: 47
Joined: Mon Jun 01, 2020 6:28 am

Re: Transferring STL containers using esp_event_post_to

Postby ESP_jakob » Thu Jul 30, 2020 12:14 pm

Hi Marek,

Is it possible to dynamically allocate your data (e.g. http_message_t) on heap? I think this should work, you can initialize the stl container there and just have to make sure that it will be deleted upon event receiving.

What do you mean by:
I've just realized that any containers that include pointer types (char* in struct, etc) cannot be used
?

Best,
Jakob

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Re: Transferring STL containers using esp_event_post_to

Postby istokm » Fri Jul 31, 2020 9:22 am

ESP_jakob wrote:
Thu Jul 30, 2020 12:14 pm
Is it possible to dynamically allocate your data (e.g. http_message_t) on heap? I think this should work, you can initialize the stl container there and just have to make sure that it will be deleted upon event receiving.
Unfortunately no, this is a public event, so the user would have a implicit requirement to delete the data, or there could be an instance where there are multiple event handlers that are going to use this data, so neither of them can delete it as they don't know when it is 'safe' to do so. So yes, it is possible, but not without memory leaks.
I've just realized that any containers that include pointer types (char* in struct, etc) cannot be used
Unless the char* is actually char[], then the same happens as above. Essentialy, you cannot use dynamically allocated data with the event loop system because there is no way to implement the subsequent freeing.

I've also looked at the event loop source, and unfortunately the esp_event_post is using esp_event_post_to so changing anything internally would probably require changing usage in all of the internal modules (wifi, etc) - which are precompiled.

ESP_jakob
Posts: 47
Joined: Mon Jun 01, 2020 6:28 am

Re: Transferring STL containers using esp_event_post_to

Postby ESP_jakob » Fri Jul 31, 2020 11:45 am

Hi Marek,

Yes, the data you're sending basically needs to be a trivial C struct. As you found already, this requirement comes from a design decision in esp_event to being able to send arbitrary data in C. It is simply not intended for this kind of resource management.

I think however, that your use case is very interesting, a while ago I had similar thoughts. If you can, would you like to share some context about your use case? Maybe there's an alternative way to express the use case without esp_event but with a different component.

Thanks,
Jakob

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Re: Transferring STL containers using esp_event_post_to

Postby istokm » Fri Jul 31, 2020 1:24 pm

ESP_jakob wrote:
Fri Jul 31, 2020 11:45 am
If you can, would you like to share some context about your use case?
Yep sure, all I'm doing is a esp_http_client wrapper, so that I don't have to worry about chunked data, initialization, configuration and error checking in every little project I (or others, if I release it) do - and cause modularization is good :)
I could've probably used some more generic FreeRTOS functions, but that seemed a little too cumbersome.

In the end the simplest solution I've come up with is just making the post, get, etc, functions blocking, and cut out events completely. I still left all of my event infrastructure in there, so that when I add opening a connection (esp_http_client_open()) as an option, the user can still receive a 'connected' event - so alteast that wasn't for nothing ;)

In the end, this is the actual implementation that I'll go with for now.

Code: Select all

ApiClient::http_message_t ApiClient::sendRequest(std::string path,
  std::string payload, esp_http_client_method_t http_method) {
  if (state_ == STATE_DEINIT) {
    ESP_LOGW(TAG, "[%s]: ApiClient wasn't initialized!", __func__);
    return {ERR_NOT_INIT,"",{},0};
  }
  else if (state_ != STATE_INIT) {
    ESP_LOGW(TAG, "[%s]: Unable to send request, " \
      "other operations are using the socket", __func__);
    return {ERR_CONN,"",{},0};
  }

  std::string url = api_config_.host + path;

  /* Reset rx buffer & rx done flag */
  rx_buffer_  = {OK, "", {}, 0};
  rx_done_    = false;

  ESP_ERROR_CHECK(esp_http_client_set_method(esp_client_handle_, http_method));
  ESP_ERROR_CHECK(esp_http_client_set_url(esp_client_handle_, url.c_str()));
  ESP_ERROR_CHECK(esp_http_client_set_post_field(esp_client_handle_,
    payload.c_str(), payload.length()));
  ESP_ERROR_CHECK(esp_http_client_perform(esp_client_handle_));

  while (!rx_done_)
    vTaskDelay(10);

  return rx_buffer_;
} //sendRequest

esp_err_t ApiClient::event_handler(esp_http_client_event_t *evt) {
  ApiClient*      api_client = static_cast<ApiClient*>(evt->user_data);
  http_message_t*  rx_buffer = &(api_client->rx_buffer_);

  esp_http_client_event_id_t event_id = evt->event_id;
  std::string event_data(static_cast<char*>(evt->data), evt->data_len);

  ESP_LOGD(TAG, "[%s]: Event id: %u", __func__, event_id);

  switch(event_id) {
    case HTTP_EVENT_ERROR:
            //Todo: find out what errors can happen here & handle them
      break;
    case HTTP_EVENT_ON_CONNECTED:
      esp_event_post_to(api_client->event_handle_,
        STATE_EVENT, EVENT_CONNECTED, nullptr, 0, event_block_ticks);
      break;
    case HTTP_EVENT_ON_HEADER:
      rx_buffer->headers.insert(
        {evt->header_key, evt->header_value});
      break;
    case HTTP_EVENT_ON_DATA:
      rx_buffer->data += event_data;
      break;
    case HTTP_EVENT_ON_FINISH:
      rx_buffer->status_code =
        esp_http_client_get_status_code(api_client->esp_client_handle_);

      api_client->rx_done_ = true;

      esp_event_post_to(api_client->event_handle_,
        STATE_EVENT, EVENT_RECV, nullptr, 0, event_block_ticks);
      break;
    case HTTP_EVENT_DISCONNECTED:
      esp_event_post_to(api_client->event_handle_,
        STATE_EVENT, EVENT_DISCONNECTED, nullptr, 0, event_block_ticks);
      break;
    default:
      break;
  }
  
    return ESP_OK;
} //ApiClient::event_handler

boarchuz
Posts: 559
Joined: Tue Aug 21, 2018 5:28 am

Re: Transferring STL containers using esp_event_post_to

Postby boarchuz » Fri Jul 31, 2020 4:48 pm

Marek Ištok wrote:
Fri Jul 31, 2020 9:22 am
Unfortunately no, this is a public event, so the user would have a implicit requirement to delete the data, or there could be an instance where there are multiple event handlers that are going to use this data, so neither of them can delete it as they don't know when it is 'safe' to do so. So yes, it is possible, but not without memory leaks.
There is esp_event_loop_run, which gives you control of when and from which task the registered handlers run.

Code: Select all

const esp_event_loop_args_t event_loop_args = {
  .queue_size = x,
  .task_name = NULL, // do not create a new task, we will use our own
};

// On Event:

// Allocate, populate, etc
ptr = malloc(size);

// Queue the event
esp_event_post_to(handle, STATE_EVENT, id, ptr, size, portMAX_DELAY);
// Now run all the handlers (users will need to ensure handlers don't block, don't stress this task's stack, etc)
esp_event_loop_run(handle, max_time);

// Handlers done, cleanup
free(ptr)
Your STL containers should be happy with that too.

I use this a lot myself, mainly to take advantage of the events API with each component having its own base and without chewing through memory for a bunch of new event loop tasks. But I'm having a hard time finding documentation for it. Is it only in that comment..?
Marek Ištok wrote:
Fri Jul 31, 2020 9:22 am
Unless the char* is actually char[], then the same happens as above. Essentialy, you cannot use dynamically allocated data with the event loop system because there is no way to implement the subsequent freeing.
AFAIK event handlers are run in order of their registration. I suppose you could register your own 'cleanup handler' immediately prior to dispatching an event. Given that it was registered last, when it runs you know that all other handlers have returned so it's safe to free any heap allocated memory.

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Re: Transferring STL containers using esp_event_post_to

Postby istokm » Sat Aug 01, 2020 1:37 am

boarchuz wrote:
Fri Jul 31, 2020 4:48 pm
Your STL containers should be happy with that too.
I use this a lot myself, mainly to take advantage of the events API with each component having its own base and without chewing through memory for a bunch of new event loop tasks. But I'm having a hard time finding documentation for it. Is it only in that comment..?
Might be an issue in larger projects, but all I'm doing is minimal embedded devices - I'll check how my memory usage looks tomorrow.
Also, as I'm posting events from the http_client event handler, so I'd like to keep it non-blocking for a lot of reasons.
Not sure exactly what documentation you are refering to, but NULL task is mentioned here:
https://docs.espressif.com/projects/esp ... task_nameE
And esp_event_loop_run usage here:
https://docs.espressif.com/projects/esp ... TickType_t
boarchuz wrote:
Fri Jul 31, 2020 4:48 pm
AFAIK event handlers are run in order of their registration.
Yep, that is correct, the whole system is pretty much based on that, because most events on the default loop actually have a internal handler that fires before any of the user ones do.
https://docs.espressif.com/projects/esp ... atch-order
boarchuz wrote:
Fri Jul 31, 2020 4:48 pm
I suppose you could register your own 'cleanup handler' immediately prior to dispatching an event.
Yeah I've thought about it, IMO it's a bit of a hacky solution though

I might revisit this a bit later and rewrite it back to a event - but I don't see an immediate need rn, I don't have any code that should run between a http request and its response, and in general, having that asynchronous is a rare implementation - compared to java, python, etc.

ESP_jakob
Posts: 47
Joined: Mon Jun 01, 2020 6:28 am

Re: Transferring STL containers using esp_event_post_to

Postby ESP_jakob » Fri Aug 07, 2020 6:44 am

Hi Marek!

As I understand, you want to notify the users of your component about the finished task. Why don't you use just simple callbacks? This could give you the control over the memory back since the caller of the callback can free the memory afterwards. This is also type-safe.
If you don't want to block inside some function (at least not for too long), you can use std::async or start a thread manually.

Best,
Jakob

istokm
Posts: 27
Joined: Thu Jun 25, 2020 12:11 pm

Re: Transferring STL containers using esp_event_post_to

Postby istokm » Fri Aug 07, 2020 6:59 am

Thank you for your suggestion, that would indeed work, and I've done components like that - but for this one I've found, that using a blocking function is quite easy to work with. Everything that uses events and callbacks will likely have to become quite a complicated state machine.
Here's a snippet of how simple a blocking syntax is for retrying authentication:

Code: Select all

http_message_t res = sendRequest(path, payload, http_method);

if (res.status_code == 401)
  if (api_auth_method_->reauth())
    res = sendRequest(path, payload, http_method);

return res;
I'm still not done, because of another baseless (http_client this time) limitation (https://github.com/espressif/esp-idf/issues/5707)... but I could work around that - just have to rewrite a lot of stuff again.

The whole point of this thread was, that with a super simple change to the esp_event loop, you could have a much larger range of possibilities. Of course not counting the rewriting and possible bugs this could cause in the rest of the system...

Who is online

Users browsing this forum: No registered users and 108 guests