Depends how you are powering it. If it's 5V then it's going through the linear requlator that is on the zero board.
If you directly power it with 3.3V you have to really remove the regulator as it will still leak current backward.
And yes, also remove the rgb led.

When you do that the system uses 8 ...