Virtualisation 101 – VMotion

What Is It?

VMotion is arguably VMware’s “killer app” – the feature that gave VMware’s hypervisor product a USP edge over its competition.    It enables an ESX host to transfer a running virtual machine over to a different ESX host without incurring downtime.  

When a VMware administrator initiates a VMotion migration the memory state of the chosen virtual machine is copied via a dedicated network link from the source host to target host;  when completed the target host registers the guest machine, attaches the virtual NICs to its own vSwitch(es) and takes control of the guest.

The handover happens so smoothly that network connections are maintained, rendering the process invisible to users, who at worst see the server pause for a second or two.

This feature effectively separates the physical hardware from the operating system, resulting in major benefits to business :-

  • A running virtual machine is no longer dependent on a single piece of hardware, reducing the risk of service outages should a hardware failure occur.
  • There is no need to perform planned hardware maintenance outside of working hours, reducing costs and improving responsiveness.
  • Workloads can be juggled across servers to best utilise the resources available: if an ESX host gets busy, guests can be moved off to a less busy hosts until balance is restored, improving efficiency.

What Do I Need to Deploy It?

Two ESXi/ESX hosts with compatible CPUs:    The target host must support the same processor features as the source host, otherwise the virtual guest could issue a command that the host cannot understand and result in the guest crashing.   For example you cannot migrate from an Intel server to an AMD server.  There are VMotion Compatibility Guides that group compatible server types together (see links below).   

New to vSphere, Enhanced VMotion Compatibility (EVC) can be enabled on a cluster to improve compatibility by checking with the hosts and calculating a CPU “mask” – a list of features supported by all hosts in the cluster.   As a last resort, and unsupported by VMware, a custom CPU mask can be defined manually (see the KB article linked to below for more details).

VMware Licensing:    Both hosts must be licensed with either Essentials+, Advanced, Enterprise or Enterprise+.

vCenter Server:   Source and target hosts must be managed by the same vCenter server (or linked vCenters), as migrations are initiated from the vCenter server, either from vSphere Client or the Move-VM cmdlet in Powershell.

VMotion Network:    A vmkernel interface (with an IP address separate from the Service Console or Management interface) must exist on both hosts for the express purpose of VMotion comms.

Because of the time-critical nature of the migration it must be a fast link (bandwidth >622Mbps, latency <5ms round trip) so Gigabit is required.  For resilience the vSwitch should have two or more NICs from different physical switches. 

Also note that VMotion data is NOT encrypted — and therefore insecure — so it is recommended that a dedicated VLAN and IP range be allocated for the VMotion interface.

Finally, if the VMotion network traverses a firewall then tcp port 8000 needs opening up.

Shared Storage:   The underlying files that make up the guest virtual machine must be accessible by both source and target host.  

Is VMotion Safe?

Generally VMotion is very safe, with any errors reported in the Task Pane and the migration aborted.  There are a few circumstances where VMotion isn’t possible :-

  • If the VM has a resource attached that is not available on the target – for example if a mapped CD ISO is stored on the source host’s local datastore.  (Storing ISOs on a shared LUN avoids this issue.)
  • If the VM has a physical SCSI controller attached, for example on virtual Microsoft Cluster nodes, or has a VMDirectPath device attached (which gives the guest direct access to a PCI device on the host).
  • Where the target host has insufficient resources to honour the guest’s requirements AND strict admission controls are in place for the cluster.

There are two priorities of VMotion available – High and Low.  This is about protecting performance of the guest, not of the migration.  A High Priority migration reserves sufficient CPU cycles on the target host to satisfy the guest’s requirements, otherwise it will abort the migration.  A Low Priority migration will go ahead regardless of target host CPU utilisation.

As VMotion transfers the memory state of the guest to the target host, a guest with 64Gb of RAM will take significantly longer to migrate than a guest with 1Gb of RAM. 

It is possible to run 4 concurrent VMotions on vSphere 4.1 hosts with Gigabit networking – if you’re lucky enough to have deployed 10Gb networking you can run up to 8 VMotions at the same time.

How Does VMotion Tie In With Other Features?

Putting a host into Maintenance Mode initiates the VMotioning of all running guests off that host.

DRS (Distributed Resource Scheduler) provides automated balancing of resources across all hosts in a cluster.   When enabled, vCenter analyses host utilisation every 5 minutes, and if a host is deemed significantly busier than the rest it will initiate VMotions of one or more guests off that host to lighten the load.

HA (High Availability) is a technology that monitors host availability, responding to failures to ensure a failed host’s virtual machines are brought online quickly on a working host.   It doesn’t use VMotion to achieve this.

So What Is Storage VMotion?

Storage VMotion is a separate feature for Enterprise or Enterprise+ hosts that provides the ability to move a running guest’s data files from one datastore to another.  This feature is fantastic for SAN migrations or maintenance work.   Migrations can also convert VMDK files between Thin and Thick formats.

One word of caution:  Storage VMotion of guests with RDM (Raw Disk Mapping) disks attached will by default convert the RDMs into VMDK files!   If RDMs are deployed, use the Advanced mode in the migration wizard.

Storage VMotion won’t work for guests with snapshots in place, or if any disks are non-persistent. 

One final tip:  If taking advantage of Storage VMotion it is worth checking whether your SAN is VAAI capable, in which case the vCenter can talk with the SAN to offload some disk actions such as this,  improving performance.

Where Do I Go From Here?

Introduction to VMotion:

Configuring VMotion Networking:

VMotion Compatibility Guide for Intel processors:

VMotion Compatibility Guide for AMD processors:

Modifying the CPU mask:

4 thoughts on “Virtualisation 101 – VMotion”

  1. Hi, thank you for your great post.

    Just about one thing :

    Are you sure of that : “Putting a host into Maintenance Mode initiates the VMotioning of all running guests off that host.” ?

    As far i as remember, and unfortunately i don’t have any VMware infra to test it again under my hands…, this is not automatic at least not in all cases. I remember havind to validate and to initiate the vmotion myself on a small infra with essentials+ pack and 2 ESXs. If i did not, the ESX won’t go in maintenance mode. Can someone confirm that please ? Perhaps the automation comes from DRS feature ?

    But again, don’t have a way to confirm this 🙁


  2. Good write-up. Two corrections:
    EVC was introduced in one of the updates of VI3.5.
    SvMotion of VM with RDMs won’t convert them to VMDKs, unless you chose to change format to thin or thick.

Comments are closed.