Fix: adding new ips with inventory builder (#7577) (#7583)

* Fix: adding new ips with inventory builder (#7577)

* moved conflig loading logic
to after checking whether the config
should be loaded, and added check for
whether the config should be loaded

* added check for removing nodes from config
if the user wants to remove a node, we
need to load the config

* Fix tox errors