Azure Blob Storage data fetch fails due to HTTPS Authentification MBed TLS

[adrian-fratila]

Hi everybody,

I try to fetch a binary file from an Azure Blob Storage, using esp_http_client, IDF v5.0.1. The fetch works in Postman or in a browser, which means the url path and the SAS token are correct.

The connection to the Azure server is established, but I get the error message: "Server failed to authenticate the request":

Code: Untitled.txt Select all

I(15113) APP : Opened successfully the HTTP connection

I(15273) HTTP_CLIENT : Body received in fetch header state, 0x3ffbb6da, 166

I(15293) APP : Read data : ���< ? xml version = "1.0" encoding = "utf-8" ? >

<Error><Code>AuthenticationFailed< / Code><Message>Server failed to authenticate the request.Make sure the value of Authorization header is formed correctly including the signature.

RequestId : 01100f54 - 901e-0045 - 18cc - e3e32c000000

I have tried various combinations for the connection setting: the url with and without the SAS at the end, and setting the SAS in the http header "Authorization" field.

I suspect the problem comes from the way the connection data is set / arranged in the esp http client. This is the code that I have tested:

Code: Untitled.c Select all

esp_http_client_config_t config = {

  //.url = "https://name.blob.core.windows.net/test/v0.0.2-fw.bin&sp=r&...0MtMgNI%3D",

  .url = "https://name.blob.core.windows.net/test/v0.0.2-fw.bin",

  .transport_type = HTTP_TRANSPORT_OVER_SSL,

  .event_handler = http_event_handler,

  .cert_pem = rootCaPem,

  .cert_len = rootCaPemLen,

  .timeout_ms = RECEIVE_TIMEOUT,

  .keep_alive_enable = true,

  .skip_cert_common_name_check = true,

  //.use_global_ca_store = true

};



esp_http_client_handle_t client = esp_http_client_init(&config);



if (client == NULL) {

  ESP_LOGE(TAG, "Failed to initialise HTTP connection");

  return;

}



esp_http_client_set_header(client, "Authorization", "SharedAccessSignature sp=r&...0MtMgNI%3D");

//esp_http_client_set_header(client, "ContentType", "application/octet-stream");

esp_http_client_set_header(client, "Accept", "*/*"); // copied from Postman

esp_http_client_set_header(client, "Accept-Encoding", "gzip, deflate, br"); // copied from Postman

esp_http_client_set_header(client, "Host", "name.blob.core.windows.net");

esp_http_client_set_method(client, HTTP_METHOD_GET);



err = esp_http_client_open(client, 0);



if (err != ESP_OK) {

  ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));

  esp_http_client_cleanup(client);

  return;

}

else {

  ESP_LOGI(TAG, "Opened successfully the HTTP connection");

}



esp_http_client_fetch_headers(client);

Any idea on why the Azure authorization fails?

Does anybody have an ESP http client example, for fetching data from an Azure Blob Storage?

I would really appreciate any help!

Thank you!

[harshal]

Hello @adrian-fratila,

I think the http_request needs authentication data which is not included in the config. If you are trying to access the domain "https://name.blob.core.windows.net/test/v0.0.2-fw.bin" without any auth data then this should have been accessible using a browser publicly by me too. But when I try to access this domain, I get the same error message in the browser as the reported error "Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature."
I feel all you need is to add the authentication configs. You can refer to the basic auth example of esp_http_client: https://github.com/espressif/esp-idf/bl ... #L329-L355

[MicroController]

AFAICT, the SAS token goes into the URI of the request. And apparently, a "date" or "x-ms-date" header is mandatory for all authenticated accesses; see e.g. examples here.

Edit:

Code: Select all

//.url = "https://name.blob.core.windows.net/test/v0.0.2-fw.bin&sp=r&...0MtMgNI%3D"

looks almost right. Just the resource-path 'test/v0.0.2-fw.bin' needs to be separated from the query parameters by a ? instead of an &!
Try

Code: Select all

.url = "https://name.blob.core.windows.net/test/v0.0.2-fw.bin?sp=r&...0MtMgNI%3D"

[adrian-fratila]

Hi guys,

Thank you for your input. I should have stated in my previous post that the URL and SAS were dummy. For completeness I have created a new Azure blob and uploaded two .txt files (3kB & 12KB).

The strange thing is that the small one (test.txt) is somehow loaded, even there is an error from the HTTP_CLIENT. In this error the beginning of the text file appears in the Error log:

I (12172) OTA: Opened successfully the HTTP connection
I (12252) HTTP_CLIENT: Body received in fetch header state, 0x3ffbad7a, 486
E (12252) HTTP_CLIENT: This is a test file.
This example demonstrates common operations the OTA tool:
- reading, writing and erasing OTA partitions,
- switching boot partitions, and
- switching to factory partition.

Users taking a look at this example should focus on the contents of the Python script [otatool_example.py](otatool_example.py) or shell script [otatool_example.sh](otatool_example.sh). The scripts contain
programmatic invocation of the tool's functions via the Python API and command-l�?!
I (12302) OTA: Fetching the Image file of length 2840 bytes
I (12302) OTA: Incremented length 1024
I (12312) OTA: Incremented length 2048
I (12312) OTA: Incremented length 2840
I (12322) OTA: Connection closed
I (12322) OTA: Total received data length: 2840
I (12332) HTTP_CLIENT: HTTP_EVENT_DISCONNECTED


The second file: test_long.txt doesn't load at all, and throws this error:

I (14202) OTA: Opened successfully the HTTP connection
E (14282) esp-tls-mbedtls: read error :-0x7100:
E (14282) transport_base: esp_tls_conn_read error, errno=Connection already in progress
E (14292) OTA: HTTP client fetch headers failed
I (14292) HTTP_CLIENT: HTTP_EVENT_DISCONNECTED


The same error appears for any large size file. I have inlined the code below, which contains valid URLs and SAS tokens with the expiration at the end of October, so plenty of time left for testing.

Code: Untitled.c Select all

	  esp_http_client_config_t config = {

	  		.url = "https://dumstorage.blob.core.windows.net/test/test.txt?sp=r&st=2023-09-13T16:10:57Z&se=2023-10-31T01:10:57Z&spr=https&sv=2022-11-02&sr=b&sig=5WjKp2Mp7uJWVJNLr2sdYt17YWuhYpVoJSwhmmBBgAo%3D",

	  		//.url = "https://dumstorage.blob.core.windows.net/test/test_long.txt?sp=r&st=2023-09-13T14:51:07Z&se=2023-10-30T23:51:07Z&spr=https&sv=2022-11-02&sr=b&sig=lK88A5l41ysl%2BxEgwhvOsYZaia3CdJzFchDHnLlGaNE%3D",

				.transport_type = HTTP_TRANSPORT_OVER_SSL,

		    .event_handler = http_event_handler,

				.cert_pem = rootCaPem,

		    .cert_len = rootCaPemLen,

				.timeout_ms = 5000,

				//.auth_type = HTTP_AUTH_TYPE_BASIC,

				.keep_alive_enable = true,

				.skip_cert_common_name_check = true,

		    //.use_global_ca_store = true

		};



		esp_http_client_handle_t client = esp_http_client_init(&config);

		if (client == NULL) {

		   ESP_LOGE(TAG, "Failed to initialise HTTP connection");

		   return;

		}



		//esp_http_client_set_header(client, "Authorization", "");

		//esp_http_client_set_header(client, "ContentType", "application/octet-stream");

		//esp_http_client_set_header(client, "Accept", "*/*");

		esp_http_client_set_method(client, HTTP_METHOD_GET);





		err = esp_http_client_open(client, 0);

		if (err != ESP_OK) {

		  ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));

		  esp_http_client_cleanup(client);

		  return;

		}else{

			ESP_LOGI(TAG, "Opened successfully the HTTP connection");

		}



		content_length = esp_http_client_fetch_headers(client);



		if (content_length <0){

			ESP_LOGE(TAG, "HTTP client fetch headers failed");

			esp_http_client_cleanup(client);

			return;

		}else{

			ESP_LOGI(TAG, "Fetching the Image file of length %d bytes", content_length);

		}



		int binary_file_length = 0;



		while (1) {

		    int data_read = esp_http_client_read_response(client, ota_write_data, BUFFSIZE);



		    if (data_read < 0) {

		        ESP_LOGE(TAG, "Error: SSL data read error");

		        esp_http_client_cleanup(client);

		        return;

		    } else if (data_read > 0) {

		    	  //ESP_LOGI(TAG, "Read data: %s", ota_write_data);

		        binary_file_length += data_read;

		        ESP_LOGI(TAG, "Incremented length %d", binary_file_length);

		    } else if (data_read == 0) {

		        if (errno == ECONNRESET || errno == ENOTCONN) {

		            ESP_LOGE(TAG, "Connection closed, errno = %d", errno);

		            break;

		        }

		        if (esp_http_client_is_complete_data_received(client) == true) {

		            ESP_LOGI(TAG, "Connection closed");

		            break;

		        }

		    }

		}

		ESP_LOGI(TAG, "Total received data length: %d", binary_file_length);

		if (esp_http_client_is_complete_data_received(client) != true) {

		    ESP_LOGE(TAG, "Error in receiving complete file");

		    http_cleanup(client);

		    return;

		}



		esp_http_client_cleanup(client);

Following @ESP_harshal suggestion, I have also set .auth_type = HTTP_AUTH_TYPE_BASIC in config, but that doesn't change the outcome.

I believe there must be some ESP HTTP CLIENT config settings which are missing, in order to get the data from an Azure blob storage. The links in the code above work in a browser or Postman.

Any help will be highly appreciated!

[crodriguez]

Hello Adrian,

Did you manage to solve it? I am facing the same issue...

Thanks in advance