A handful of standalone servers carried the whole company: mailings, the mail server, web servers and a number of smaller services. Each machine was managed on its own, and an outage of one meant an outage of whatever happened to run on it.
Goal
Design a unified OS infrastructure concept on which all services run as virtual machines. They can be moved between hosts, backed up with one procedure and survive the loss of one physical machine.
Solution
- XEN XCP virtualization platform on two hosts. Highly available virtual machines start on the other host when one fails.
- Dom0 hosts and SAN storage set up as the base of the platform.
- Workload balancing across the Dom0 hosts, SAN storage and network so that no layer becomes the bottleneck.
- Shared storage via SAN and an attached NAS, including backups and storage high availability.
- Migration of existing physical and virtual servers (mail, databases, websites) to the new platform, with each service tuned after the move.
- Documentation of the whole infrastructure so that someone else can run it.
Everything is built on open-source software: XEN XCP and GNU/Linux Debian and CentOS.
Outcome
Services that used to depend on individual machines run as virtual machines on a shared platform with backups and failover to the second host. About a year after launch, load and availability monitoring was added on top of the infrastructure.