On MPIO and In-Guest iSCSI

Introduction
I recently had a “stimulating discussion”* with Tom about MPIO initiated from a virtualized guest OS to a new SAN I am installing, and the pros and cons there of. This is one of those situations where the constraints of the technology make sense in a physical situation, but look a bit odd virtualized, so I wanted to ensure that I had the best possible solution for my needs, with the least extra effort, and importantly, some logic to back up the decision.

What is MPIO?

MPIO stands for MultiPath Input Output. It could be tagged as “For when one cable simply isn’t enough!”. This is the storage version of what at the network level we call “Link Aggregation”. The simple idea is to take multiple connections (be they Fiberchannel, SAS or iSCSI) between the SAN and what-ever is consuming it, and use them all. This can give you resiliency (Failover, failback) or increased bandwidth, through Round Robin use of Paths, or other options depending on what is available in the implementation.
My use case is iSCSI over standard Ethernet connections, so that is what I’m going to focus on.
The first thing to work out is what is a “Path” in order to work out how to get multiple paths. At the IP level a path is defined by two IP addresses, a source and destination. The network underlying that is irrelevant, it could be 100Mb/s Fast Ethernet (shudder) or 10 or 100Gb/s intercontinental Fibre. It could consist of 1 physical cable, multiple cables connected by switches, multiple cables between multiple switches etc etc. It really doesn’t matter. What matters is that the storage presents multiple IPs, in order to get Multiple Paths. Obviously, the same applies to the consumer. In order to get the full benefit of Multipathing, it must have multiple Paths, so multiple IPs.
Physical MPIO
 
So, you have a SAN with 4 IPs and 4 NICs dedicated to block traffic. You have one or more Physical Servers, which have 4 free NICs for iSCSI traffic. You now have two choices.
You could aggregate the links from the SAN to the Switch, and from the Servers to the Switch using LAGs. This would be dependant on your Switches supporting that (probably true) and on your guest supporting that (True from windows Server 2012 upwards, dependent on NIC drivers before that, but probably true).
Or you could leave the connections simple and use MPIO to distribute traffic over the links.
Do you want the Servers/SAN to handle the paths, or do you want the switch to do it?
There is a reasonable chance that MPIO will give better performance than a LAG, after all, if the SAN software can control the path end to end, it has more options and more information to order the flows with than a switch would. That said, I’ve done no testing, and would recommend that anyone who needs to make this decision to do that testing.
Physical Host
 
Host Level MPIO
 
Host Level MPIO works in exactly the same way as Physical outlined above. With the exception that once LUNs are mapped, they are formatted with VMFS and so passed up to the guests. This implies that there will be many guests using each LUN, and so a few connections.
 
Guest Level MPIO
In this situation, the Physical Host has no idea about the SAN what so ever. iSCSI traffic is just more VM traffic going out over vSwitches and into the network. It is arguable weather that is a sensible plan in the first place, but lets assume that for whatever reason, you want to do in Guest IO.
In our situation we would need to give the VM multiple vNICs to gain the advantage of MPIO, at least two vNICs, but preferably four. To each VM. We have a moderately small farm of 200 VMs, that’s nearly one thousand vNICs, which all need IPs on the storage network!
Lets look again at what MPIO gives us, failover and more bandwidth.
More Bandwidth in both the LAG and the MPIO situations, a single Path can only use the maximum bandwidth of that path, not the aggregate bandwidth, and one “conversation” will be limited to, in our case 1Gb/s despite the four NICs. If we are using multiple LUNs per VM (we are) then we have the possibility to utilise all the bandwidth, but that is true of both LAG and MPIO.
Resiliancy In both MPIO and LAG we get resiliency from having multiple connections. Again there is no clear winner.
The real test though is to consider the VM as a whole. In the physical server situation it is assumed that None SAN traffic is being aggregated in some way on “other” NICs. With our VM this is potentially true also, but in all likely hood that resiliency is being granted at the vSwitch level, by a LAG. If we already have a LAG in place, will the MPIO give us any advantages?
As stated above there is a chance that MPIO will give better utilisation, but this is less likely with “in guest”, as there are many more “conversations” than in the Physical or Host situation
Guest
Conclusion
 
We use in guest iSCSI for a few reasons, that mainly relate to recovery time objectives and the ability to snapshot subsets of our data. My conversation with Tom centered around the need to add, in our case 8 vNICs to each VM in our farm, which seemed excessive (bear in mind this is as well as any NLB, DAG, Clustering vNICs). No single VM is using 1Gb/s, in fact the Phsycial Hosts only have 2 1Gb/s NICs connected to the SAN switch anyway.
We could put two NICs in each VM, and then track which VMs see which Paths, but that seems like an awful lot of administrative effort. It *would* break at some point.
We could put a single NIC in each VM, and leave the resiliency to the Host. Again we’d have to track which VMs saw which Paths.
Or we could just utilise the LAG we have in place, and not confuse matters with MPIO. That is the route we have ultimately chosen.
 
* It wasn’t quite an argument, more a challenge to Tom to defend a position while I attacked it. Obviously, I lost. I always do ;)

Host Becomes disconnected

In my current role I have inherited a largish vSphere 4.1 environment, that has to put it nicely “evolved”.  We have been having host disconnection issues in one of the clusters, coupled with HA Configuration errors.  well today one host disconnected and absolutely refused to reconnect, bizarrely it kept telling us that the username/password combination was incorrect within vCenter.

So we jumped on the ILo and entered the SAME username and combination and low and behold we gained access to the promised land.  Now I know what you are thinking “CapLocks”, but no checked that.

My next troubleshooting step was to restart the management agents on the errant host but again this did not fix the error.

The I had a “lightbulb” moment, as we checked the network settings on the host. I saw that the gateway address was incorrect, it was configured with a correct gateway address but for the wrong subnet.  it was set to the vMotion network not the Management network. reset that and all of a sudden we could rejoin vCenter.

I then checked the rest of the Hosts in the Cluster and found the same basic configuration error, we reset the gateway across the cluster and low and behold now HA works as designed and no more configuration errors.

Moral of this story is “Check the Basics”

The evolution of VMware Tools for Linux

VMware Tools for Linux is getting better all the time. Actually it has been in movement for years and the way it has improved shows the strength of Open Source.

Back in the old days, we only had one package to care about: VMware Tools as bundled with the VMware product. Many times it stopped working after a kernel upgrade and we had to patch the source code by hand. Continue reading

VMware Security Advisory: VMSA-2014-0002

Updated Security Advisory, this one relates to some third party advisories. but please note that it does include a critical status!!, however that said, there is currently nothing you can do about the issue unless you are running vCenter 5.5, the update to the latest version “Update 1”.

The issue relates to a possible DDoS attack on the NTP Service and I advise you to read KB2070193 for full information on the vulnerability and remediation approaches.

Continue reading

News: VSAN and vSphere 5.5 Update 1 are available, finally pricing is released!

Today marks the release of vSphere 5.5 Update 1, you can find the release notes here and the links to the download are found here.  Why is this such a milestone? Well it also finally marks the release of VMware’s long awaited entry into the world of Software defined storage the VSAN, but before we move onto that what else is contained in Update 1? Continue reading

New: VSAN is now GA

Today finally marks the announcement of the general availability of VMware VSAN. This is VMware’s most public beta to date.

So, What Is VSAN?

VSAN is true integrated software-defined storage. It is embedded into the vKernel, it is a distributed object store, and as such, it is resource efficient. There is no requirement for an extra virtual machine on each host.

to read the full post click here