In my current role I have inherited a largish vSphere 4.1 environment, that has to put it nicely “evolved”. We have been having host disconnection issues in one of the clusters, coupled with HA Configuration errors. well today one host disconnected and absolutely refused to reconnect, bizarrely it kept telling us that the username/password combination was incorrect within vCenter.
So we jumped on the ILo and entered the SAME username and combination and low and behold we gained access to the promised land. Now I know what you are thinking “CapLocks”, but no checked that.
My next troubleshooting step was to restart the management agents on the errant host but again this did not fix the error.
The I had a “lightbulb” moment, as we checked the network settings on the host. I saw that the gateway address was incorrect, it was configured with a correct gateway address but for the wrong subnet. it was set to the vMotion network not the Management network. reset that and all of a sudden we could rejoin vCenter.
I then checked the rest of the Hosts in the Cluster and found the same basic configuration error, we reset the gateway across the cluster and low and behold now HA works as designed and no more configuration errors.
Moral of this story is “Check the Basics”