HTTP client response data corrupted

ukrsms
Posts: 10
Joined: Tue Aug 20, 2019 11:30 am

HTTP client response data corrupted

Postby ukrsms » Fri Aug 23, 2019 7:57 am

I faced and issue with HTTP client when performing POST request.

The code is as followed:

Code: Select all

esp_http_client_config_t config = {
				.method=HTTP_METHOD_POST,
				.url=Request->Query,
				.timeout_ms=CLOUD_LOOKUP_TIMEOUTms,
				.disable_auto_redirect=1,					//no redirection
				.event_handler = HTTP_EventHandler,
			};
		client = esp_http_client_init(&config);
		if(!client)
		{
			break;
		}
		if(esp_http_client_set_header(client, "Content-Type", HTTP_CONT_TYPE_BIN) 	||
				esp_http_client_set_header(client, "Cache-Control", "no-cache") 	||
				esp_http_client_set_header(client, "User-Agent", HTTP_USERAGENT) 	||
				esp_http_client_set_header(client, "Accept", "*/*"))
		{
			break;
		}

		if(Err=esp_http_client_set_post_field(client, Request->BodyData, Request->BodyDataLen))
		{
			break;
		}
		Err = esp_http_client_perform(client);
		if (!Err)
		{
					Request->ReponseCode=esp_http_client_get_status_code(client);
					Request->ResponseDataLen=esp_http_client_get_content_length(client);
		}
		else
		{
			break;
		}
		if(Request->ResponseDataLen>HTTP_MAX_RESPONSE_LEN)
		{
			break;
		}
		Request->ResponseData=zalloc(Request->ResponseDataLen);
		if(!Request->ResponseData)
		{
			break;
		}
		u16 RecDataLen=0;
		if((RecDataLen=esp_http_client_read(client, Request->ResponseData, Request->ResponseDataLen/*max len*/))==-1)
		{
			break;
		}
However, after the request is finished I received response data currupted.
After studying the Esp_HTTP_client code I found that when the first data chunk is received (containing headers and body data begin) and parsed client->response->buffer->raw_data pointer points to the place where the body data begins. But the next data chunk is received before calling esp_http_client_read(). As a result new data is placed to the response buffer beginning , hence, pointer client->response->buffer->raw_data is assign to the buffer beginning causing a case that the first data chunk lays at larger address, while the second one starts at lower address (two chunks become swapped).

Could anyone tell me did I have some mistake while using the client or is there some error in it?

Update: I tried the code with both IDF versions 3.2 and 3.2.2, the result was the same.

chegewara
Posts: 930
Joined: Wed Jun 14, 2017 9:00 pm

Re: HTTP client response data corrupted

Postby chegewara » Mon Aug 26, 2019 12:30 pm

https://github.com/espressif/esp-idf/bl ... mple.c#L55

Code: Select all

	case HTTP_EVENT_ON_DATA:
		ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
		
		if (!esp_http_client_is_chunked_response(evt->client)) {
			// Write out data
			memcpy(response + total_len, (char*)evt->data, evt->data_len);
			total_len += evt->data_len;
			ESP_LOGD(TAG, "%d ---> %d", evt->data_len, total_len);
			// printf("%.*s", evt->data_len, (char*)evt->data);
		}

		break;

pdemianczuk
Posts: 14
Joined: Thu May 30, 2019 12:05 pm

Re: HTTP client response data corrupted

Postby pdemianczuk » Tue Dec 10, 2019 2:55 pm

Hello, I've very similar problem. I tried read 6400B (in the future I would like to read circa about 64kB)
I noticed that when I try read data less than MTU= 1440B is everything correct.
Which parameter (in Kconfig) is responsobile for buffor.
I find out the #define DEFAULT_HTTP_BUF_SIZE (512) but when i set higer I can't see anything better.

@chegewara
Unfortunetlly, when I used .event_handler I have problem with heap. When heap is above 5KB I have problem with wifi or WDT and the program is restarting.

Code: Select all


HEAP 76 kB 0x1302c                                                        
*CHECK TASK no: 1074322300       size: 2352                               
 Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 356                    
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 5592                 
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 808                  
I (25033) Check HEAP ... : free DRAM 22 kB 0x5b8c | IRAM 80 kB 0x1413c    
Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 356                     
Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 2                       
*CHECK TASK no: 1074322300       size: 2320                               
 I (25053) Check HEAP ... : free DRAM 56 kB 0xe0b0 | IRAM 113 kB 0x1c660              
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 5592                 

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 808
I (25473) Check HEAP ... : free DRAM 11 kB 0x2d14 | IRAM 68 kB 0x112c4 

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 5592

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 808
I (25903) Check HEAP ... : free DRAM 11 kB 0x2e20 | IRAM 68 kB 0x113d0 
*CHECK TASK no: 1074322300       size: 2320
anyone knows what was happend with this componnet?

pdemianczuk
Posts: 14
Joined: Thu May 30, 2019 12:05 pm

Re: HTTP client response data corrupted

Postby pdemianczuk » Thu Dec 19, 2019 12:29 pm

Hi, I probably solve the problem. I update IDF to v4.1-dev-1533 from IDF4.0-dec-667. I had to update also toolchain (to esp-2019r2, compiler to 8.2.0), and I havn't problem with memory.
I found out some https://github.com/espressif/esp-idf/co ... 165cf29087
fix, but I am not sure is this fix the problem.
Generally, You should check is the component is updated.
I dunno know why, my app had a problem but the most people wasnt.

atlex92
Posts: 1
Joined: Fri Feb 14, 2020 2:00 pm

Re: HTTP client response data corrupted

Postby atlex92 » Fri Feb 14, 2020 2:14 pm

Hi, i had a same problem as described above, so i've managed with it by setting buffer size of client large enough, By default it will be just 512 bytes
esp_http_client_config_t config = {};
config.url = reqInfo.url.c_str();
config.is_async = false;
config.timeout_ms = reqInfo.timeoutMs;
config.buffer_size = 1024;
_client = esp_http_client_init(&config);

Who is online

Users browsing this forum: Bing [Bot], MSN [Bot] and 21 guests