I have TP5000 as a LiFePo charger and it works perfectly just in a way you described. It can also be used as a solar charger provided you are using the panel with max voltage within 5~6V range.
The only thing you should care about is a low voltage protection circuit to cut off the ESP and ...