Download and write binary file

jaskod
Posts: 2
Joined: Fri May 22, 2020 6:01 am

Download and write binary file

Postby jaskod » Fri May 22, 2020 6:12 am

Hi there,

i am trying to download binary fila via http request and save it. To achieve this, i used and combined examples "http_request" and "wear_levelling". Everything works fine, while i am downloading some text file (.html,.txt etc.), but when i try to download .bin file, I always get empty response from server.

When i get response, at first I delete the old file if exist:

Code: Select all

if (remove("/spiflash/file.txt") == 0) 
                ESP_LOGI(TAG, "Deleted old file");
then i open a file to write in append mode:

Code: Select all

            FILE *f = fopen("/spiflash/file.txt", "a");
            if (f == NULL) {
                ESP_LOGE(TAG, "Failed to open file for writing");
                return;
            }
and finally i am reading the response and writing to file:

Code: Select all

            do {
                bzero(recv_buf, sizeof(recv_buf));
                r = read(s, recv_buf, sizeof(recv_buf)-1);
                fprintf(f, recv_buf);
                printf(recv_buf);
            } while(r > 0);
Can you pleas give me an advice, how to download and save the .bin file ?

nvannote
Posts: 51
Joined: Thu Nov 14, 2019 10:42 pm

Re: Download and write binary file

Postby nvannote » Mon May 25, 2020 9:07 am

Well, I am not going to get into the download part as you didn’t post any code you're using to do that. But I will address the binary output part you did.

Personally, I prefer to use the system/posix functions for binary data; but the C functions will do just fine here.

First off; For any fprintf (or any format type function), you never want to pass arbitrary unknown data as the format string (even for plain text) which for fprintf would be the second parameter. fprintf is expecting a format string there. That call should be changed to a fputs or fwrite for unformatted TEXT writes. You're just opening yourself up to a world of hurt “when” something fprintf recognizes as a format specification comes through in your recv_buf. If you're lucky, it will just print some garbage and not crash.

Now, for the binary data question (and I am just going on what you had posted).

Change the fopen mode to “wb”; this will open the file for writing/binary; It will also have the side effect of truncating an existing file, so you can “remove” the remove call. If you want to keep the remove, "wb" or “ab” will do the same with the exception that "ap" will automatically append to an existing file. This is true for the text version as well ("w" vs "a").

For binary data you don’t need to bzero the buffer as the contents are irrelevant and we don’t care about string null terminators, so the last argument to your read can be sizeof(recv_buf) without subtracting 1.

You want to check the return value of the input read to ensure it didn't fail.

For the output, Change the fprintf to an fwrite(recv_buf, 1, r, f).

The second (1) parameter is an item size and the third (r) is the item count. So for example, if you received 20 bytes, the fwrite should return 20. You could swap the two arguments and fwrite would return 1 for the 20 bytes; but I prefer the former. If fwrite returns less than the count, something unexpected/bad happened.

Last but not least, I see your trying to print that binary buffer to the console as well. That’s not going to work out well. If you really wan’t to see a human readable representation of the binary data on the console, you can use ESP_LOG_BUFFER_HEXDUMP (or ESP_LOG_BUFFER_HEX or ESP_LOG_BUFFER_HEX_LEVEL) for that.

As always, check return values!

Regards

PeterR
Posts: 621
Joined: Mon Jun 04, 2018 2:47 pm

Re: Download and write binary file

Postby PeterR » Mon May 25, 2020 10:10 pm

Short answer is that fprintf() won't do binary well. Ends at first 0x00.
Try fwrite(), r gives you the length. rtfm wrt to fwrite() ;)
& I also believe that IDF CAN should be fixed.

nvannote
Posts: 51
Joined: Thu Nov 14, 2019 10:42 pm

Re: Download and write binary file

Postby nvannote » Tue May 26, 2020 5:36 am

PeterR wrote:
Mon May 25, 2020 10:10 pm
Short answer is that fprintf() won't do binary well. Ends at first 0x00.
Try fwrite(), r gives you the length. rtfm wrt to fwrite() ;)

Actually the short answer is...

  • Don't pass unknown strings/data to fprintf as a format specification. It will produce unexpected results at best.
  • Open the file with the correct mode (binary in this case).
  • Use fwrite for writing binary data to that file.
  • Use a proper mechanism to output raw binary data to the console.
Regards

jaskod
Posts: 2
Joined: Fri May 22, 2020 6:01 am

Re: Download and write binary file

Postby jaskod » Wed May 27, 2020 8:07 am

Thank you for your advices, i have tried to write binary data from variable to file and it works.

About the downloading file, as i said, i just combined two example codes.
Whole code is there:
  1.  
  2.  
  3. /*************************
  4.     I N C L U D E
  5. *************************/
  6. #include "https_request.h"
  7.  
  8. #include <stdio.h>
  9. #include <string.h>
  10. #include <stdlib.h>
  11. #include <stdbool.h>
  12. #include "freertos/FreeRTOS.h"
  13. #include "freertos/task.h"
  14. #include "freertos/event_groups.h"
  15. #include "esp_event.h"
  16. #include "esp_log.h"
  17. #include "esp_system.h"
  18. #include "esp_vfs_fat.h"
  19. #include "tcpip_adapter.h"
  20.  
  21. #include "lwip/err.h"
  22. #include "lwip/sockets.h"
  23. #include "lwip/sys.h"
  24. #include "lwip/netdb.h"
  25. #include "lwip/dns.h"
  26.  
  27. #include "esp_tls.h"
  28. /*************************
  29.     T Y P E D E F
  30. *************************/
  31. /*************************
  32.     D E F I N E
  33. *************************/
  34.  
  35. /*************************
  36.     V A R I A B L E
  37. *************************/
  38. static const char *TAG = "https_request";
  39.  
  40. static const char *REQUEST = "GET " WEB_PATH " HTTP/1.0\r\n"
  41.     "Host: "WEB_SERVER":"WEB_PORT"\r\n"
  42.     "User-Agent: esp-idf/1.0 esp32\r\n"
  43.     "\r\n";
  44.  
  45. const char *base_path = "/spiflash";
  46.  
  47. static wl_handle_t s_wl_handle = WL_INVALID_HANDLE;
  48.  
  49. static char recv_buf[64];
  50. /*************************
  51.     P R O T O T Y P E
  52. *************************/
  53. void readWrittenData();
  54. /*************************
  55.     F U N C T I O N
  56. *************************/
  57. void https_request(){
  58.     const struct addrinfo hints = {
  59.         .ai_family = AF_INET,
  60.         .ai_socktype = SOCK_STREAM,
  61.     };
  62.     struct addrinfo *res;
  63.     struct in_addr *addr;
  64.     int s, r;
  65.  
  66.     bool success = 0;
  67.     while(!success) {
  68.         int err = getaddrinfo(WEB_SERVER, WEB_PORT, &hints, &res);
  69.  
  70.         if(err != 0 || res == NULL) {
  71.             ESP_LOGE(TAG, "DNS lookup failed err=%d res=%p", err, res);
  72.             vTaskDelay(1000 / portTICK_PERIOD_MS);
  73.             continue;
  74.         }
  75.  
  76.         addr = &((struct sockaddr_in *)res->ai_addr)->sin_addr;
  77.         ESP_LOGI(TAG, "DNS lookup succeeded. IP=%s", inet_ntoa(*addr));
  78.  
  79.         s = socket(res->ai_family, res->ai_socktype, 0);
  80.         if(s < 0) {
  81.             ESP_LOGE(TAG, "... Failed to allocate socket.");
  82.             freeaddrinfo(res);
  83.             vTaskDelay(1000 / portTICK_PERIOD_MS);
  84.             continue;
  85.         }
  86.         ESP_LOGI(TAG, "... allocated socket");
  87.  
  88.         if(connect(s, res->ai_addr, res->ai_addrlen) != 0) {
  89.             ESP_LOGE(TAG, "... socket connect failed errno=%d", errno);
  90.             close(s);
  91.             freeaddrinfo(res);
  92.             vTaskDelay(4000 / portTICK_PERIOD_MS);
  93.             continue;
  94.         }
  95.  
  96.         ESP_LOGI(TAG, "... connected");
  97.         freeaddrinfo(res);
  98.  
  99.         if (write(s, REQUEST, strlen(REQUEST)) < 0) {
  100.             ESP_LOGE(TAG, "... socket send failed");
  101.             close(s);
  102.             vTaskDelay(4000 / portTICK_PERIOD_MS);
  103.             continue;
  104.         }
  105.         ESP_LOGI(TAG, "... socket send success");
  106.  
  107.         struct timeval receiving_timeout;
  108.         receiving_timeout.tv_sec = 5;
  109.         receiving_timeout.tv_usec = 0;
  110.         if (setsockopt(s, SOL_SOCKET, SO_RCVTIMEO, &receiving_timeout,
  111.                 sizeof(receiving_timeout)) < 0) {
  112.             ESP_LOGE(TAG, "... failed to set socket receiving timeout");
  113.             close(s);
  114.             vTaskDelay(4000 / portTICK_PERIOD_MS);
  115.             continue;
  116.         }
  117.         ESP_LOGI(TAG, "... set socket receiving timeout success");
  118.  
  119.  
  120.  
  121.             ESP_LOGI(TAG, "Mounting FAT filesystem");
  122.             const esp_vfs_fat_mount_config_t mount_config = {
  123.                     .max_files = 4,
  124.                     .format_if_mount_failed = true,
  125.                     .allocation_unit_size = CONFIG_WL_SECTOR_SIZE
  126.             };
  127.             esp_err_t err2 = esp_vfs_fat_spiflash_mount(base_path, "storage", &mount_config, &s_wl_handle);
  128.             if (err2 != ESP_OK) {
  129.                 ESP_LOGE(TAG, "Failed to mount FATFS (%s)", esp_err_to_name(err2));
  130.                 return;
  131.             }
  132.             if (remove("/spiflash/file.txt") == 0)
  133.                 ESP_LOGI(TAG, "Deleted old file");
  134.  
  135.             ESP_LOGI(TAG, "Opening file");
  136.             FILE *f = fopen("/spiflash/file.txt", "a");
  137.             if (f == NULL) {
  138.                 ESP_LOGE(TAG, "Failed to open file for writing");
  139.                 return;
  140.             }
  141.  
  142.             do {
  143.                 bzero(recv_buf, sizeof(recv_buf));
  144.                 r = read(s, recv_buf, sizeof(recv_buf)-1);
  145.                 fprintf(f, recv_buf);
  146.                 printf(recv_buf);
  147.             } while(r > 0);
  148.  
  149.             fclose(f);
  150.             ESP_LOGI(TAG, "File written");
  151.             ESP_LOGI(TAG, "Unmounting FAT filesystem");
  152.             ESP_ERROR_CHECK( esp_vfs_fat_spiflash_unmount(base_path, s_wl_handle));
  153.  
  154.         ESP_LOGI(TAG, "... done reading from socket. Last read return=%d errno=%d.", r, errno);
  155.         close(s);
  156.         success = 1;
  157.     }
  158.  
  159.     readWrittenData();
  160. }
  161.  
  162.  
  163. void readWrittenData(){
  164.    
  165.     ESP_LOGI(TAG, "Mounting FAT filesystem");
  166.     const esp_vfs_fat_mount_config_t mount_config = {
  167.             .max_files = 4,
  168.             .format_if_mount_failed = true,
  169.             .allocation_unit_size = CONFIG_WL_SECTOR_SIZE
  170.     };
  171.     esp_err_t err = esp_vfs_fat_spiflash_mount(base_path, "storage", &mount_config, &s_wl_handle);
  172.     if (err != ESP_OK) {
  173.         ESP_LOGE(TAG, "Failed to mount FATFS (%s)", esp_err_to_name(err));
  174.         return;
  175.     }
  176.  
  177.     // Open file for reading
  178.     ESP_LOGI(TAG, "Reading file");
  179.     FILE *f = fopen("/spiflash/file.txt", "rb");
  180.     if (f == NULL) {
  181.         ESP_LOGE(TAG, "Failed to open file for reading");
  182.         return;
  183.     }
  184.     char line[128];
  185.    
  186.    
  187.     while(fgets(line, sizeof(line), f)){
  188.         printf(line);
  189.     }
  190.  
  191.  
  192.     fclose(f);
  193.  
  194.  
  195.     // Unmount FATFS
  196.     ESP_LOGI(TAG, "Unmounting FAT filesystem");
  197.     ESP_ERROR_CHECK( esp_vfs_fat_spiflash_unmount(base_path, s_wl_handle));
  198.  
  199.     ESP_LOGI(TAG, "Done");
  200. }

nvannote
Posts: 51
Joined: Thu Nov 14, 2019 10:42 pm

Re: Download and write binary file

Postby nvannote » Wed May 27, 2020 11:49 am

jaskod wrote:
Wed May 27, 2020 8:07 am
Thank you for your advices, i have tried to write binary data from variable to file and it works.

About the downloading file, as i said, i just combined two example codes.
Whole code is there:

Groan. OK, You certainly did "just combined two example codes", didn't you.

Well, I am not going to rewrite your task; such as only mounting/unmounting the filesystem at proper start/end points. But the code you posted still treats binary data as text and is not going to work. I will also point out that the http_request example as written (IDF v4.0.1) doesn't use fprintf to write arbitrary data to a file, so I am not sure where you picked up that nasty habit. You also don't wan't to use fgets to read raw binary data.

May I kindly suggest that you pursue a better understanding of application design and the C programming language before moving forward. There are thousands of resources for that on the internet. Have fun!

Best Regards

nvannote
Posts: 51
Joined: Thu Nov 14, 2019 10:42 pm

Re: Download and write binary file

Postby nvannote » Wed May 27, 2020 1:44 pm

I might also point out that that http_request example is a plain sockets example. The HTTP protocol has more going on than just receiving the requested resource. You may wan't to look into the ESP HTTP Client APIs to deal with that.

Regards

PeterR
Posts: 621
Joined: Mon Jun 04, 2018 2:47 pm

Re: Download and write binary file

Postby PeterR » Wed May 27, 2020 10:41 pm

This does seem more of a 'C' question than ESP.
You do need to understand the ASCII table & the difference between fwrite()/fread() and fprintf().
Simplistically - fprintf() is about readable text & fwrite() is about binary, what does that mean? So fread() (or socket calls) might return data which (politely) 'mess' with fprintf().
IMHO download Visual Studio and try a few file tutorials. Much easier to get on top of language issues in Studio than an embedded platform; you don't need to setup a Jtag probe etc, etc.
& actually all the cool kids cross platform develop! Well I do, oh well....
& I also believe that IDF CAN should be fixed.

nvannote
Posts: 51
Joined: Thu Nov 14, 2019 10:42 pm

Re: Download and write binary file

Postby nvannote » Fri May 29, 2020 12:40 am

PeterR wrote:
Wed May 27, 2020 10:41 pm
So fread() (or socket calls) might return data which (politely) 'mess' with fprintf().

Depends on what you mean by "politely". ;)

printf/fprintf/sprintf/snprintf (and the like) use the format string as a trustworthy map of the a variable argument list that follows; which it only has a stack pointer to access. So it pretty easy to end up with an access violation (panic LoadProhibited) or worse.

By default; clang issues a security warning if you pass non-literal format strings to these functions; surprisingly gcc does not.

Best Regards

Who is online

Users browsing this forum: mahone and 266 guests