Today I was trying to add a NFS share from a NAS device to my vSphere lab servers. I’ve used these same NFS shares before, but I had a lab setup so did not care too much that the NFS shares were not on an isolated network. This weekend I decided to fix that sloppy setup and move traffic to isolated NIC’s in their own VLAN. Simple I thought, a quick job.
Somehow just moving the shares to a new network segment on another VLAN broke things. I was puzzled and should have asked Tom I suppose.. but as a good IT guy I had to dig until I find out why by myself.So what do you do when you have NFS problems? Right.. you follow the steps of basic troubleshooting.
Both my lab hosts (vSphere 4.1 -ESX- and vSphere 5.1 -ESXi-) behaved the same and would not let me connect.
Trying to connect to any NFS share would -after a while- result in the following error:
"Call "HostDatastoreSystem.CreateNasDatastore" for object "ha-datastoresystem" on ESX "192.168.0.7" failed. Operation failed, diagnostics report: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.
The vmkernel.log had the following helpful tidbit:
2012-12-09T14:25:05.746Z cpu2:198983)NFS: 157: Command: (mount) Server: (192.168.0.9) IP: (192.168.0.9) Path: (/nfs/MyShare) Label: (MyShare) Options: (ro) 2012-12-09T14:25:05.746Z cpu2:198983)StorageApdHandler: 692: APD Handle 271460f7-918a5edb Created with lock[StorageApd0x410016] 2012-12-09T14:25:35.979Z cpu4:198983)StorageApdHandler: 739: Freeing APD Handle [271460f7-918a5edb] 2012-12-09T14:25:35.979Z cpu4:198983)StorageApdHandler: 802: APD Handle freed! 2012-12-09T14:25:35.979Z cpu4:198983)NFS: 168: NFS mount 192.168.0.9:/nfs/MyShare failed: Unable to connect to NFS server.
Connecting to these exact same shares via an ubuntu guest on the same host was no problem at all, just connect the guest to the NFS vSwitch and boom, gotcha, it works.
Connecting from the vSphere 5.1 host to the iSCSI target the NAS device also exposes worked beautifully too. Click click.. done… easy.
So then one scratches behind the ear and starts to follow all the standard steps from the knowledge base:
also followed my own troubleshooting steps from here:
Everything looked just fine. No firewall problems, all services worked fine. No root squash problem as I used these same shares before, albeit with different IP’s and not via the same vmkernel interface.
Wondered about things like a gateway being set incorrectly, but as they are all in the same isolated subnet – with no connection to the outside world – there is no gateway required.
Now just before I was considering pulling out some hair, I came accross this post on vmtn: ESXi 4.1 can not add NFS Datastore on a Netgear READYnas 1100 ( Please Help)
Down there it mentioned that your DNS server has to be set correctly so that both host and NFS server can be found…
But I am connecting via IP, the vSphere hosts are not accessing the NFS shares via DNS resolution! Well.. that might be the case, but …
.. sure enough.. I simply added the IP’s of both vSphere servers to the /etc/hosts file on the NFS device and the first time I tried it worked. Immediately!
Happy that it works now, but a bit frustrated that there’s no better error description to be found anywhere.