High availability and shared storage for a small XEN cluster

Why high availability needs shared storage

High availability (HA) for virtual machines means that after a host fails, the machine starts up on another host without manual intervention. That's only possible when the virtual machine's disk doesn't live on the local storage of the host that just went down – it has to be readable from elsewhere. That's why HA and shared storage practically always go together in a small infrastructure: without shared storage, “high availability” is just a restart from the last backup, not a live takeover.

Shared storage: SAN and NAS

For a small infrastructure, shared storage is typically handled with a combination of:

• ▸ SAN (Storage Area Network) – block storage that hosts access like a local disk; virtual machine disk images live here so that any host in the cluster can reach them;
• ▸ NAS (Network Attached Storage) – file storage attached over the network, mainly suited for backups and shared data that doesn't need the performance or latency of block storage.

Both layers stand or fall on load balancing between the hosts, the storage and the network – if the bottleneck sits in just one of them, HA alone doesn't fix the problem, it just moves it somewhere else.

What HA means for day-to-day operations

• ▸ when one host fails, virtual machines with HA automatically start up on the remaining host (or hosts);
• ▸ live migration of a virtual machine between hosts is possible thanks to shared storage even outside a failure scenario – for example for planned maintenance;
• ▸ backups of the shared storage are handled with one procedure for all virtual machines, instead of separately for each physical server.

Operational details a small cluster runs into too

A few things a small XEN cluster deals with before the first real outage arrives:

• ▸ the storage type XCP uses for a shared volume must be correctly registered on the system – a missing storage repository type (for example LVM) only shows up when the cluster tries to use it, not at install time;
• ▸ it's worth keeping host consoles (DOM0) reachable independently of the virtual machines running on that host – a serial or HVC console over a tunnel is simpler for diagnosing an outage than waiting on a network that happens to be served by the failed virtual machine;
• ▸ documenting the whole infrastructure – storage topology, network assignment, the backup procedure – is worth keeping separate from individual hosts, so the infrastructure can still be managed after the person who set it up is gone.

When it pays off even at small scale

HA and shared storage carry extra operational overhead – load balancing, storage management, a network sized for live migration. For a small infrastructure (on the order of a handful of hosts), it pays off once it's running services whose outage costs more than that overhead – typically mail, web, or a database that the whole company's operation depends on, not just a single internal service.