Kernel isolation: PV, HVM and OS-level virtualization in Linux

Paravirtualization (PV)

Paravirtualization doesn't use a kernel module but the Xen hypervisor; guests are managed by the privileged domain 0 (dom0) running on top of it. The guest operating system has to be aware that it is being paravirtualized (kernel 3.0+), and the hypervisor exposes an ABI that the guest calls.

Properties:

• ▸ practically no performance loss – direct access to resources;
• ▸ faster boot – the kernel can boot directly, without a bootloader;
• ▸ guests use their own kernel;
• ▸ isolation on top of the underlying OS can additionally be secured with AppArmor or SELinux.

Hardware-assisted virtualization (HVM)

A typical example is KVM, or Xen in HVM mode. Guests are completely isolated through the hypervisor; instructions that can't be virtualized directly are trapped by the processor's hardware support (Intel VT-x, AMD-V) and handed to the hypervisor. Binary translation was what x86 virtualization needed before that support existed.

Properties:

• ▸ the guest has its own bootloader;
• ▸ the guest has its own kernel;
• ▸ the guest OS doesn't need to be modified in any way;
• ▸ all resources are handled indirectly, through emulation;
• ▸ a supported OS (kernel 2.6.32+) can nowadays use PVHVM – a combination of both approaches;
• ▸ it requires CPU flags for virtualization (Intel vmx, AMD svm).

Operating-system-level virtualization (containers)

Containers can be pictured as a book on a shelf: the book has several chapters, each telling a different “story”, but they all belong to the same copy. Containers are sometimes called “jail on steroids”.

• ▸ they add another layer of security by isolating resources at the OS level;
• ▸ they can be combined with AppArmor or SELinux for even stronger security;
• ▸ they solve problems with shared libraries existing in multiple versions and help keep the OS clean;
• ▸ they are easy to destroy and re-create;
• ▸ they share the kernel with the host.

Differences between virtual machines and containers

Virtual machines:

• ▸ generally “heavier” to set up and start;
• ▸ OS boot can take up to minutes (depending on PV vs. HVM);
• ▸ isolation at the hardware level through the hypervisor (HVM, PV, PVHVM);
• ▸ a virtual machine is represented by a Qemu process, with a storage backend involved.

Containers:

• ▸ lightweight – typically single-digit to a few hundred MiB;
• ▸ can be focused on a single application;
• ▸ isolation at the OS level, through the process tree.

Overview of container technologies

• ▸ chroot – only partial filesystem isolation, nested virtualization.
• ▸ OpenVZ – filesystem isolation, disk quotas, IO, memory and CPU limiting, network isolation, partial nested virtualization, live migration, root isolation.
• ▸ lxc/lxd – filesystem isolation, partial disk quotas (lvm/btrfs), partial IO limiting (btrfs), memory limits, CPU quotas, network isolation, partial nested virtualization, root isolation.
• ▸ docker – filesystem isolation, IO limiting (since 1.10), memory limits, CPU quotas, network isolation, partial nested virtualization, root isolation (since 1.10).
• ▸ systemd-nspawn – filesystem isolation, disk quotas, partial IO limiting (systemd + cgroups), memory limits (systemd + cgroups), CPU quotas (systemd + cgroups), network isolation, nested virtualization, root isolation.

What containers use to isolate resources

Namespaces:

• ▸ PID – process identifiers; a container has its own tree with PID 1;
• ▸ UTS – host and domain name;
• ▸ MNT – filesystem access and structure;
• ▸ IPC – process communication over shared memory;
• ▸ NET – network access and structure;
• ▸ USER – user and group identifiers; root in a container need not be root on the host.

Alongside namespaces, containers rely on cgroups, which protect resources (CPU usage, memory usage, IO).

When to choose containers, and when virtual machines

Containers are a good fit for:

• ▸ testing a new application (from source, or from the internet);
• ▸ fast deployments – a container as an “iso” template for an application, or for a whole cycle.

Virtual machines are the better choice where you need:

• ▸ wider isolation – access to resources is filtered or emulated (HVM), or goes through an API and drivers (PV);
• ▸ separate “sandboxes” for customers.

Based on a talk “(Kernel) Isolation – PV, HVM, OS-level virtualization in Linux” (LinuxDays 2017), slides, video.