A default gateway on the IP Pool servicing the NSX controllers had been mistyped during installtion. I needed to change this so my montioring system could route traffic to the 3 controllers.
NTP Server was on a different subnet so current controllers could not route to them due to an incorrect gateway.
It is not possible edit the default gateway via IP Pools.
NSX had recently been upgraded from 6.4.2 -> 6.4.3 – which (I believe) in the process added a new feature called Common Controller Attributes with blank settings.
Firstly I tried to delete a controller, add another IP Pool with correct default gateway then re-create the controller using this pool This failed as the NSX Manager pushes the NTP settings to the current controllers – which causesd a failure due to not being able to reach the NTP servers.
Unfortunately the resolution is to remove the all the controllers completely, fill out the common attributes settings, create a new IP pool then re-create the controllers
1. Ensure you have backups (NSX specifically but I like to belt and braces when doing changes like this)
2. Organize a maintenance window, where no new VMs will be deployed or powered on/off.
2. Set your clusters' DRS mode to manual.
3. Create the new IP pool with the correct default gateway.
4. Delete the existing controllers.
i) Note you will need to FORCE REMOVE the last controller
5. Deploy new controllers pointing to the new IP pool.