Comparing the ESP-IDF vs Arduino ESP32 component libraries

[kmulier]

1. Context

I'm co-founder of a startup building a new IDE for microcontrollers: Embeetle IDE. We would like to support the ESP32 family in our IDE, so I'm figuring out the ESP32 build system.

While figuring out the build system, I currently focus on how the ESP-IDF tool builds projects, and how Arduino IDE builds ESP projects on the other hand.


2. Questions

The ESP-IDF framework has a set of "component libraries". In the Arduino IDE, they are available as precompiled archives. Comparing both raised a lot of questions to me. I'll explain in the next paragraphs.


3. The Arduino "component libraries"

The Arduino IDE stores the ESP32 component libraries as archive files here:

~/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/sdk/lib/


I'm trying to figure out how these precompiled component libraries were built in the first place. Thanks to my previous question (see viewtopic.php?f=19&t=20396) I now know where to find the source code for them. If you have the esp-idf tool installed, the source code can be found here:

~/esp/esp-idf/components/

Compiling this source code into those component libraries is not straightforward. This is what I do:

1. I launch a "Hello World" project with the esp-idf tool.

2. I replace the file '~/esp/hello_world/sdkconfig' with '~/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/sdk/sdkconfig'. In other words, I try to apply the Arduino configurations onto my "Hello World" project.

3. I build the "Hello World" project with full verbosity to see how the component libraries are built.

Since my "Hello World" project has now the exact same configuration settings as Arduino (thanks to replacing the 'sdkconfig' file), the esp component libraries that appear in my project's build folder should be the same as those in the '~/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/sdk/lib/' folder. Is this correct?

If that's correct, then the three-step-approach described above is sufficient to figure out how the component libraries were built for the Arduino IDE.

4. Linking the "component libraries" with the application code

The Arduino IDE uses its precompiled "component libraries" totally different from how ESP-IDF uses them in the linking step of the final application .elf file. This is how Arduino does it:

Code: Select all

xtensa-esp32-elf-g++ -nostdlib
                     -L<esp-sdk>/lib
                     -L<esp-sdk>/ld
                     -T esp32_out.ld
                     -T esp32.project.ld
                     -T esp32.rom.ld
                     -T esp32.peripherals.ld
                     -T esp32.rom.libgcc.ld
                     -T esp32.rom.spiram_incompatible_fns.ld
                     -u esp_app_desc
                     -u ld_include_panic_highint_hdl
                     -u call_user_start_cpu0
                     -Wl,--gc-sections
                     -Wl,-static
                     -Wl,--undefined=uxTopUsedPriority
                     -u __cxa_guard_dummy
                     -u __cxx_fatal_exception
                     -Wl,--start-group
                     /tmp/arduino_build_852524/sketch/WiFiScan.ino.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/ETH.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFi.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiAP.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiClient.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiGeneric.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiMulti.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiSTA.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiScan.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiServer.cpp.o
                     /tmp/arduino_build_852524/libraries/WiFi/WiFiUdp.cpp.o
                     /tmp/arduino_build_852524/core/core.a
                     -lgcc
                     -lesp_websocket_client
                     -lwpa2
                     -ldetection
                     -lesp_https_server
                     -lwps
                     -lhal
                     -lconsole
                     -lpe
                     -lsoc
                     -lsdmmc
                     -lpthread
                     -llog
                     -lesp_http_client
                     -ljson
                     -lmesh
                     -lesp32-camera
                     -lnet80211
                     -lwpa_supplicant
                     -lc
                     -lmqtt
                     -lcxx
                     -lesp_https_ota
                     -lulp
                     -lefuse
                     -lpp
                     -lmdns
                     -lbt
                     -lwpa
                     -lspiffs
                     -lheap
                     -limage_util
                     -lunity
                     -lrtc
                     -lmbedtls
                     -lface_recognition
                     -lnghttp
                     -ljsmn
                     -lopenssl
                     -lcore
                     -lfatfs
                     -lm
                     -lprotocomm
                     -lsmartconfig
                     -lxtensa-debug-module
                     -ldl
                     -lesp_event
                     -lesp-tls
                     -lfd
                     -lespcoredump
                     -lesp_http_server
                     -lfr
                     -lsmartconfig_ack
                     -lwear_levelling
                     -ltcp_transport
                     -llwip
                     -lphy
                     -lvfs
                     -lcoap
                     -lesp32
                     -llibsodium
                     -lbootloader_support
                     -ldriver
                     -lcoexist
                     -lasio
                     -lod
                     -lmicro-ecc
                     -lesp_ringbuf
                     -ldetection_cat_face
                     -lapp_update
                     -lespnow
                     -lface_detection
                     -lapp_trace
                     -lnewlib
                     -lbtdm_app
                     -lwifi_provisioning
                     -lfreertos
                     -lfreemodbus
                     -lethernet
                     -lnvs_flash
                     -lspi_flash
                     -lc_nano
                     -lexpat
                     -lfb_gfx
                     -lprotobuf-c
                     -lesp_adc_cal
                     -ltcpip_adapter
                     -lstdc++
                     -Wl,--end-group
                     -Wl,-EL
                     -o /tmp/arduino_build_852524/WiFiScan.ino.elf

The precompiled component libraries are sitting in the folder <esp-sdk>/lib (actually in ~/.arduino15/packages/esp32/hardware/esp32/1.0.6/tools/sdk/lib, but I'm using some placeholders here for brevity). This folder is exposed to the linker through the -L<esp-sdk>/lib flag. After listing all the relevant object files of the application, the libraries are then passed to the linker with -l flags. The linker knows where to find them.


This is how the linking happens with the ESP-IDF tool:

Code: Select all

xtensa-esp32-elf-g++ -mlongcalls
                     -Wno-frame-address
                     CMakeFiles/hello-world.elf.dir/project_elf_src_esp32.c.obj
                     -o hello-world.elf
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/asio/libasio.a
                     esp-idf/bt/libbt.a
                     esp-idf/cbor/libcbor.a
                     esp-idf/unity/libunity.a
                     esp-idf/cmock/libcmock.a
                     esp-idf/coap/libcoap.a
                     esp-idf/console/libconsole.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/esp_adc_cal/libesp_adc_cal.a
                     esp-idf/esp_hid/libesp_hid.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/protobuf-c/libprotobuf-c.a
                     esp-idf/protocomm/libprotocomm.a
                     esp-idf/mdns/libmdns.a
                     esp-idf/esp_local_ctrl/libesp_local_ctrl.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/esp_websocket_client/libesp_websocket_client.a
                     esp-idf/expat/libexpat.a
                     esp-idf/wear_levelling/libwear_levelling.a
                     esp-idf/fatfs/libfatfs.a
                     esp-idf/freemodbus/libfreemodbus.a
                     esp-idf/jsmn/libjsmn.a
                     esp-idf/json/libjson.a
                     esp-idf/libsodium/liblibsodium.a
                     esp-idf/mqtt/libmqtt.a
                     esp-idf/openssl/libopenssl.a
                     esp-idf/spiffs/libspiffs.a
                     esp-idf/ulp/libulp.a
                     esp-idf/wifi_provisioning/libwifi_provisioning.a
                     esp-idf/main/libmain.a
                     -Wl,--cref -Wl,--Map=/home/kristof/esp/hello_world/build/hello-world.map -Wl,--gc-sections -fno-rtti -fno-lto esp-idf/asio/libasio.a
                     esp-idf/cbor/libcbor.a
                     esp-idf/cmock/libcmock.a
                     esp-idf/unity/libunity.a
                     esp-idf/coap/libcoap.a
                     esp-idf/esp_adc_cal/libesp_adc_cal.a
                     esp-idf/esp_hid/libesp_hid.a
                     esp-idf/esp_local_ctrl/libesp_local_ctrl.a
                     esp-idf/esp_websocket_client/libesp_websocket_client.a
                     esp-idf/expat/libexpat.a
                     esp-idf/fatfs/libfatfs.a
                     esp-idf/wear_levelling/libwear_levelling.a
                     esp-idf/freemodbus/libfreemodbus.a
                     esp-idf/jsmn/libjsmn.a
                     esp-idf/libsodium/liblibsodium.a
                     esp-idf/mqtt/libmqtt.a
                     esp-idf/openssl/libopenssl.a
                     esp-idf/spiffs/libspiffs.a
                     esp-idf/wifi_provisioning/libwifi_provisioning.a
                     esp-idf/protocomm/libprotocomm.a
                     esp-idf/bt/libbt.a
                     -L/home/kristof/esp/esp-idf/components/bt/controller/lib/esp32 -lbtdm_app esp-idf/protobuf-c/libprotobuf-c.a
                     esp-idf/mdns/libmdns.a
                     esp-idf/console/libconsole.a
                     esp-idf/json/libjson.a
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a
                     esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a
                     esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a
                     esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a
                     esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a
                     esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a
                     esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a
                     esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a
                     esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a
                     esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a
                     esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/ulp/libulp.a
                     esp-idf/mbedtls/mbedtls/library/libmbedtls.a
                     esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
                     esp-idf/mbedtls/mbedtls/library/libmbedx509.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcoexist.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcore.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libespnow.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libmesh.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libnet80211.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libpp.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libsmartconfig.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libwapi.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libphy.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/librtc.a
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a
                     esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a
                     esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a
                     esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a
                     esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a
                     esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a
                     esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a
                     esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a
                     esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a
                     esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a
                     esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/ulp/libulp.a
                     esp-idf/mbedtls/mbedtls/library/libmbedtls.a
                     esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
                     esp-idf/mbedtls/mbedtls/library/libmbedx509.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcoexist.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcore.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libespnow.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libmesh.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libnet80211.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libpp.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libsmartconfig.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libwapi.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libphy.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/librtc.a
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a
                     esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a
                     esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a
                     esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a
                     esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a
                     esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a
                     esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a
                     esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a
                     esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a
                     esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a
                     esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/ulp/libulp.a
                     esp-idf/mbedtls/mbedtls/library/libmbedtls.a
                     esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
                     esp-idf/mbedtls/mbedtls/library/libmbedx509.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcoexist.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcore.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libespnow.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libmesh.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libnet80211.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libpp.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libsmartconfig.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libwapi.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libphy.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/librtc.a
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a
                     esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a
                     esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a
                     esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a
                     esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a
                     esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a
                     esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a
                     esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a
                     esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a
                     esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a
                     esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/ulp/libulp.a
                     esp-idf/mbedtls/mbedtls/library/libmbedtls.a
                     esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
                     esp-idf/mbedtls/mbedtls/library/libmbedx509.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcoexist.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcore.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libespnow.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libmesh.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libnet80211.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libpp.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libsmartconfig.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libwapi.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libphy.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/librtc.a
                     esp-idf/mbedtls/libmbedtls.a
                     esp-idf/efuse/libefuse.a
                     esp-idf/app_update/libapp_update.a
                     esp-idf/bootloader_support/libbootloader_support.a
                     esp-idf/esp_ipc/libesp_ipc.a
                     esp-idf/spi_flash/libspi_flash.a
                     esp-idf/nvs_flash/libnvs_flash.a
                     esp-idf/pthread/libpthread.a
                     esp-idf/esp_gdbstub/libesp_gdbstub.a
                     esp-idf/espcoredump/libespcoredump.a
                     esp-idf/esp_system/libesp_system.a
                     esp-idf/esp_rom/libesp_rom.a
                     esp-idf/hal/libhal.a
                     esp-idf/vfs/libvfs.a
                     esp-idf/esp_eth/libesp_eth.a
                     esp-idf/tcpip_adapter/libtcpip_adapter.a
                     esp-idf/esp_netif/libesp_netif.a
                     esp-idf/esp_event/libesp_event.a
                     esp-idf/wpa_supplicant/libwpa_supplicant.a
                     esp-idf/esp_wifi/libesp_wifi.a
                     esp-idf/lwip/liblwip.a
                     esp-idf/log/liblog.a
                     esp-idf/heap/libheap.a
                     esp-idf/soc/libsoc.a
                     esp-idf/esp_hw_support/libesp_hw_support.a
                     esp-idf/esp_pm/libesp_pm.a
                     esp-idf/esp_ringbuf/libesp_ringbuf.a
                     esp-idf/driver/libdriver.a
                     esp-idf/xtensa/libxtensa.a
                     esp-idf/perfmon/libperfmon.a
                     esp-idf/esp32/libesp32.a
                     esp-idf/esp_common/libesp_common.a
                     esp-idf/esp_timer/libesp_timer.a
                     esp-idf/freertos/libfreertos.a
                     esp-idf/newlib/libnewlib.a
                     esp-idf/cxx/libcxx.a
                     esp-idf/app_trace/libapp_trace.a
                     esp-idf/nghttp/libnghttp.a
                     esp-idf/esp-tls/libesp-tls.a
                     esp-idf/tcp_transport/libtcp_transport.a
                     esp-idf/esp_http_client/libesp_http_client.a
                     esp-idf/esp_http_server/libesp_http_server.a
                     esp-idf/esp_https_ota/libesp_https_ota.a
                     esp-idf/sdmmc/libsdmmc.a
                     esp-idf/esp_serial_slave_link/libesp_serial_slave_link.a
                     esp-idf/ulp/libulp.a
                     esp-idf/mbedtls/mbedtls/library/libmbedtls.a
                     esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
                     esp-idf/mbedtls/mbedtls/library/libmbedx509.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcoexist.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libcore.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libespnow.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libmesh.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libnet80211.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libpp.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libsmartconfig.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libwapi.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/libphy.a
                     /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32/librtc.a
                     -Wl,--wrap=mbedtls_mpi_exp_mod
                     -u esp_app_desc
                     -u pthread_include_pthread_impl
                     -u pthread_include_pthread_cond_impl
                     -u pthread_include_pthread_local_storage_impl
                     -u ld_include_panic_highint_hdl
                     -u start_app
                     -u start_app_other_cores
                     -L /home/kristof/esp/esp-idf/components/esp_rom/esp32/ld
                     -T esp32.rom.ld
                     -T esp32.rom.api.ld
                     -T esp32.rom.libgcc.ld
                     -T esp32.rom.newlib-data.ld
                     -T esp32.rom.syscalls.ld
                     -u vfs_include_syscalls_impl
                     -L /home/kristof/esp/esp-idf/components/esp_wifi/lib/esp32
                     /home/kristof/esp/esp-idf/components/xtensa/esp32/libxt_hal.a
                     -L /home/kristof/esp/hello_world/build/esp-idf/esp32
                     -T esp32_out.ld
                     -L /home/kristof/esp/hello_world/build/esp-idf/esp32/ld
                     -T esp32.project.ld
                     -L /home/kristof/esp/esp-idf/components/esp32/ld
                     -T esp32.peripherals.ld
                     -u call_user_start_cpu0
                     -mfix-esp32-psram-cache-issue
                     -mfix-esp32-psram-cache-strategy=memw
                     -Wl,--undefined=uxTopUsedPriority
                     -u app_main
                     -lm esp-idf/newlib/libnewlib.a
                     -u newlib_include_heap_impl
                     -u newlib_include_syscalls_impl
                     -u newlib_include_pthread_impl
                     -lgcc
                     -u __cxa_guard_dummy
                     -lstdc++ esp-idf/pthread/libpthread.a
                     esp-idf/app_trace/libapp_trace.a
                     -lgcov
                     esp-idf/app_trace/libapp_trace.a
                     -lgcov
                     -lc

The approach is a bit different. But the result should be the same: the archive files get passed to the linker, such that they are available to be linked against the application code.


However, what really troubles me is that some component libraries appear twice or more in the ESP-IDF linker call! Take for example the mbedtls library. The Arduino linker call passes this library to the linker just once:

Code: Select all

xtensa-esp32-elf-g++
    ...
    -lmbedtls
    ...

And if you look in the Arduino folder where the archive files are, you'll find just one libmbedtls.a file. Now consider the ESP-IDF linker call, passing the mbedtls library multiple times:

Code: Select all

xtensa-esp32-elf-g++
    ...
    esp-idf/mbedtls/libmbedtls.a
    esp-idf/mbedtls/mbedtls/library/libmbedtls.a
    esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
    esp-idf/mbedtls/mbedtls/library/libmbedx509.a
    ...
    esp-idf/mbedtls/libmbedtls.a
    esp-idf/mbedtls/mbedtls/library/libmbedtls.a
    esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a
    esp-idf/mbedtls/mbedtls/library/libmbedx509.a
    ...

You see, it not only gets passed multiple times to the linker, but there are even two different files with the name libmbedtls.a. Aside from that, there are also two other archive files that have something to do with this mbedtls library: libmbedcrypto.a and libmbedx509.a.

All of this confuses me. Why isn't there just a single libmbedtls.a file produced by the ESP-IDF tool?


Thank you very much for your help.

[ESP_Sprite]

Passing things multiple time to the linker is a trick that's used sometimes as the linker isn't intelligent enough to 'scan backwards' for dependencies. Arduino solves this using '-Wl,--start-group' magic, ESP-IDF solves it by linking the same library multiple times (which is allowed). Furthermore, libraries by themselves are just archives with .o files in them; the names don't matter so having a file called libmbedtls.a in two places will work fine (as long as they contain non-conflicting object files).

(This is not specific to ESP-IDF or ESP32 Arduino, btw; this is standard gnu ld behaviour / tricks.)

[kmulier]

Thank you very much @ESP_Sprite.

Thanks to your explanation and also the help of my colleague, I now understand why the same archive file is being passed multiple times to the linker.

Unfortunately, I'm still struggling to understand why there are several different archive files with the same name. For example, the ESP-IDF tool passes the 'libmbedtls.a' archive as two different files to the linker:

Code: Select all

esp-idf/mbedtls/libmbedtls.a
esp-idf/mbedtls/mbedtls/library/libmbedtls.a

This is in contrast with the Arduino IDE, which passes only one 'libmbedtls.a' archive file. How does the Arduino IDE 'libmbedtls.a' archive file differ from those in ESP-IDF?

Please also have a look at this forum post:
https://forum.embeetle.com/t/comparing- ... ystems/455
I've posted the issue there, on our own Embeetle IDE forum, because it's faster. There's no need to wait for admin moderation.

PS: My eventual goal is to understand how the 'libmbedtls.a' (and many other archive files) stored in the Arduino IDE are actually built.

[ESP_Sprite]

I don't know Arduino well enough to properly answer that question, but my guess would be that Arduino doesn't use the certificates stored in one of the mbedtls.a files, so it only needs the other one.

Also, you only need mod approval for the first 3 posts you make (it's an anti-spam measure). Seeing you have made more than that already, you should be able to post without approval.

[kmulier]

Thank you @ESP_Sprite

Your colleague Ivan Grokhotkov is the one who took care of the Arduino support for ESP microcontrollers. I'm trying to contact him, but was unsuccessful so far. Perhaps he is very busy at the moment, or maybe he's on holiday.

If he's unavailable, do you perhaps know other colleagues with knowledge about this topic? We'd really like to support the ESP microcontroller family in our IDE, but we're pretty much stuck here.

Kind regards,
Kristof

[ESP_Sprite]

...I'm not sure what makes you stuck? Easiest way to confirm is to take the lib from Arduino and esp-idf, unpack all of them using ar and comparing the contents.

[kmulier]

Hi @ESP_Sprite,

I had a reply on my forum post here:
https://forum.embeetle.com/t/comparing- ... tems/455/2

Comparing the Arduino-generated archive file with the ones from ESP-IDF, it looks like the Arduino archive file `libmbedtls.a` is essentially a combination of:

- `~/esp/hello_world/build/esp-idf/mbedtls/libmbedtls.a`
- `~/esp/hello_world/build/esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a`
- `~/esp/hello_world/build/esp-idf/mbedtls/mbedtls/library/libmbedx509.a`

Now I know how the Arduino-generated `libmbedtls.a` was built, so I'm no longer stuck

Thank you for the help,

Kind regards,
Kristof