Make causes antimalware executable to run at high cpu and memory

[zliudr]

I didn't notice this until now. Whenever I make, win 10 antimalware executable runs at a high CPU % and memory size. When make completes, this drops down. Any reason for this? Any way to make changes?

[zliudr]

I wish to bump this post since the issue is still there, release 3.3. I got a very fast computer and it compiles much faster but if I can make the antimalware executable not take over 100% of the cpu and memory, I bet even a moderate computer setup will do well.

[Sprite]

Compiling esp-idf opens lots and lots of (source) files, and the antimalware tries to scan all of them. I think you can add the esp-idf directory to the do-not-scan-on-access-list; if I recall we have a bit in the esp-idf documentation about that.

[zliudr]

Thank you! I made the change as you suggested but I didn't find where the esp-idf document is so if you could post a link that would be great! I added folder and process for exclusion. It's not specifically do-not-scan-on-access-list so maybe I can do better if you point me to the doc.

Here's some test result with make -j clean;time make -j app for one project:
Without changing windows defender setting:
Over 2 minutes.
With both folder and process exclusion:
1 min 39 sec. So it's 20% faster. I still see the antimalware program taking some CPU load but not like before with high percentages.
This is on a 2004 version of win 10, moderate 4-core i5

On my new machine, the exclusion doesn't seem to make enough difference, at least I still see some moderate percentage of CPU use by antimalware program. But it is a much faster machine so either with exclusion on or without, it takes about 30 seconds to compile. Although it is a new machine I just received a day ago, the windows version seems old, 1909, and windows being microsoft, I have to wait for THEM to decide when I can get this machine its 2004 version of win 10. This version difference could be the difference. I'll let you know once this new machine as got its 2004 version.

[chegewara]

Just a suggestion, if you have older PC with old hard drives, then dont use make -j, try with make -j 6 or something to match you core number. With NVM PCIE it may work, because this drive is capable to do many I/O operations per sec, but older disk will slow down whole process.

[zliudr]

Chegewara,

Thanks for the suggestion. I got rid of my old computers and have been using SSDs for a couple of years now. My data above were an i5-7400 with Samsung 850 PRO SATA SSD (~500MB/s read) vs. an i9-10900K with Samsung 970 EVO PCIe SSD (~3200MB/s read).

The i5 has 4 cores/8 processes with 3.0GHz speed. Good entry-level gaming computer on a good discount.
The i9 has 10 cores/20 processes with 3.7GHz speed. Pretty much the fastest I can get without having to build my own, no discounts.

Comparison after I turned on exception for both machines:

i9-10900K
no -j make clean; make app
4m 5s

with j make -j clean; make -j app
30s

-j10
43s

-j16
35s

-j20
32s

i5-7400
no -j make clean; make app
I really don't want to do this

with j make -j clean; make -j app
1m 39s

-j8
1m 41s

-j4
1m 48s
My conclusion is that just let make do as many jobs in parallel as it can and it works slightly better than telling it to match the CPU's total process count. If you go with total core count, you get worse outcome than going with total process count.

I hope this justifies my spending

[potatoBand]

Compiling esp-idf opens lots and lots of (source) files, and the antimalware tries to scan all of them. I think you can add the esp-idf directory to the do-not-scan-on-access-list; if I recall we have a bit in the esp-idf documentation about that.

Add ~\esp to exclusions in Settings -> Updates and Security -> Windows Security -> Virus and Threat Protection -> Virus and threat protection Settings, Manage Settings -> Exclusions, add or remove exclusions

[xutengl]

Slow ESP-IDF builds on Windows – Microsoft Defender process exclusions

I found that the ESP-IDF build process is significantly slower on Windows compared to Ubuntu due to Microsoft Defender. System Monitor shows high CPU usage from Microsoft Defender during builds.

On my system, folder exclusions alone were not sufficient; Defender continued scanning compiler processes during the build.

Measuring Microsoft Defender usage

Use Defender’s built-in performance tools in PowerShell (run as Administrator).

Code: Select all

New-MpPerformanceRecording -RecordTo "$env:TEMP\DefenderPerf.etl"

Run:

Code: Select all

idf.py build

Then stop recording and generate the report:

Code: Select all

Stop-MpPerformanceRecording
Get-MpPerformanceReport -Path "$env:TEMP\DefenderPerf.etl" -TopProcesses 10
Get-MpPerformanceReport -Path "$env:TEMP\DefenderPerf.etl" -TopFiles 10

This shows which executables and files Defender scans most. In my case, the main offenders were GCC backend and binutils tools (cc1.exe, as.exe, ld.exe, etc.).

What fixed it

Adding process (.exe) exclusions in Microsoft Defender.

Excluded paths (example)

• []<ESP-IDF project directory>
  []<ESP-IDF install directory>

Excluded processes (minimum)

• cc1.exe
• cc1plus.exe
• collect2.exe
• cmake.exe
• ninja.exe
• idf.py

Xtensa toolchain

• xtensa-esp-elf-gcc.exe
• xtensa-esp-elf-g++.exe
• xtensa-esp-elf-ar.exe
• xtensa-esp-elf-as.exe
• xtensa-esp-elf-ld.exe
• xtensa-esp-elf-objdump.exe
• xtensa-esp-elf-ranlib.exe
• xtensa-esp32s3-elf-ar.exe
• xtensa-esp32s3-elf-objdump.exe

ESP-IDF Python (path-specific)

Code: Select all

<ESP-IDF python venv>\python.exe

Result

• Defender peak CPU during build dropped from ~36% to ~5% (most CPU is now used by the compiler itself)
• Build times improved by ~3–6×

Hope this helps others building ESP-IDF on Windows.

[Roland]

I think what xutengl pointed out is exactly what https://github.com/espressif/idf-env/ does do. idf-env is used under the hood in both https://dl.espressif.com/dl/esp-idf/ and https://dl.espressif.com/dl/eim/ installers for adding Windows Defender exclusions.