Postcopy Live Migration for Qemu/KVM - Yabusame

Yabusame is yet another live migration mechanism for Qemu/KVM, which implements the migration technique known as "postcopy" or "lazy" migration. Just after the "migrate" command is invoked, the execution host of a VM is instantaneously switched to a destination host.

The advantages of the postcopy approach are
  • The migration time of a live migration is shorter than precopy, because the postcopy approach does not need iterative memory copy involved in the precopy approach.
  • The migration time is always deterministic for any types of VMs.
  • The execution host of a VM is quickly switched to another host. This allows quick load balance of VMs in IaaS datacenters.
The key of Yabusame is an on-demand memory transfer mechanism, which traps the first access to a memory page at the destination and copies its content from a source host. Yabusame migration basically works as follows:
  1. Stop the VM at the source host. Copy the content of virtual CPU registers and the states of devices to the destination.
  2. Resume the VM at the destination without any memory pages. Now the VM is running on the destination.
  3. If the VM touches a not-yet-transferred memory page, pause the VM temporarily. Copy the content of the memory page from the source. Then, resume the VM.
The third step is repeated until all memory pages are transferred to the destination.In parallel with this on-demand transfer, a background transfer works to make bulk copies of not-yet-transferred pages. The background copy mechanism analyzes important memory areas with page fault statistics, and starts to deal with hot-spot memory pages for current VM workloads.

See the demo movie of an early prototype. A VM, updating memory intensively, is migrated from the left host to the right just in one second.



See our latest journal article for its design and implementation. Our technique greatly alleviated performance impact on running workloads during live migration.
  • Postcopy Live Migration with Guest-cooperative Page Faults,
    Takahiro Hirofuchi, Isaku Yamahata, Satoshi Itoh,
    IEICE Transactions on Information and Systems, pp.2159-2167, Vol.E98-D, No.12, IEICE, Dec 2015
    DOI: 10.1587/transinf.2015PAP0011
    PDF (Full Text)

Code

Update:
Finally, the recent Qemu/KVM (Qemu-2.5 or later) officially supports postcopy live migration. Congrats!
For production use, we recommend that you use the postcopy migration code merged in Qemu, not the code maintained here. For reproducible research, we keep our code here.

Our latest code is available from the below repository.

It includes Qemu with postcopy support and Linux kernel with a postcopy helper driver (uvmem.ko). The device driver is used on the host OS at the destination (i.e., -incoming) side. There is also the source of the first prototype used in the below papers.


Here is the brief introduction of usage.

At the destination host, load the device driver and start Qemu with the -incoming option. It supports both KVM and TCG modes.

# modprobe uvmem.ko

# qemu-system-x86_64 -incoming tcp:0:4444 ...(other options)...

At the source host, open the Qemu monitor console and use migrate_set_capability to enable postcopy.

(qemu) migrate_set_capability postcopy on

(qemu) migrate tcp:<dest ip address>:4444

There are other options and commands. See the code for detail. RDMA and hybrid migration are experimentally supported.


Documents

The below technical talks report the latest development of Yabusame.

Talks in Developers Conferences
  • Yabusame: Postcopy Live Migration for Qemu/KVM,
    Takahiro Hirofuchi, Isaku Yamahata,
    KVM Forum 2011, Aug 2011, slides
    This talk discusses the development plan of Yabusame.

  • Yabusame: Postcopy Live Migration for Qemu/KVM,
    Takahiro Hirofuchi, Isaku Yamahata,
    Linux Plumbers Conference 2011, Sep 2011, slides
    This talk discusses the development plan of Yabusame.

  • Yabusame: Postcopy Live migration for QEmu/KVM,
    Isaku Yamahata, Takahiro Hirofuchi
    LinuxCon Japan, 2012, June 2012, slides
    This talk presents the project update including preliminary performance evaluation. Postcopy live migration alleviates performance degradation in a live migration of a large-memory VM. Asynchronous page fault contributes to mitigating performance degradation of on-demand memory copy.

Selected Papers

When you mention our postcopy live migration work, we would appreciate it if you cite our CCGrid2010 paper and IEICE journal article. The latter article reports thorough performance evaluations using the latest code.
  • Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension,
    Takahiro Hirofuchi, Hidemoto Nakada, Satoshi Itoh, Satoshi Sekiguchi,
    The 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid2010), pp. 73--83, IEEE Computer Society, May 2010
    DOI: 10.1109/CCGRID.2010.42
    This paper presents the design and implementation of the first prototype of postcopy live migration for Qemu/KVM. In comparison with related work, the proposed mechanism supports guest operating systems without any modifications to them (i.e, no special device drivers and programs are needed in VMs). It is implemented as a lightweight extension to KVM (Kernel-based Virtual Machine Monitor). It is not required to modify critical parts of the VMM code.

  • Postcopy Live Migration with Guest-cooperative Page Faults,
    Takahiro Hirofuchi, Isaku Yamahata, Satoshi Itoh,
    IEICE Transactions on Information and Systems, pp.2159-2167, Vol.E98-D, No.12, IEICE, Dec 2015
    DOI: 10.1587/transinf.2015PAP0011
    PDF (Full Text)
    This journal article presents a performance improvement technique of postcopy migration, extending the para-virtualized page fault mechanism of a virtual machine monitor (a.k.a., Asynchronous Page Fault in Linux). It also describes the design and implementation of Yabusame (i.e., the production-level code of postcopy migration intended to be merged to the mainline of Qemu/KVM).

 On dynamic VM placement, please also look at and cite our VTDC2011 paper and journal paper.

  • Reactive consolidation of virtual machines enabled by postcopy live migration,
    Takahiro Hirofuchi, Hidemoto Nakada, Satoshi Itoh, Satoshi Sekiguchi,
    Proceedings of the 5th international workshop on Virtualization technologies in distributed computing (VTDC2011), pp. 11--18, ACM Publishing, Jun 2011
    DOI: 10.1145/1996121.1996125
    slides
    This talk presents a dynamic VM packing system exploiting postcopy live migration.
  • Reactive Cloud: Consolidating Virtual Machines with Postcopy Live Migration,
    Takahiro Hirofuchi, Hidemoto Nakada, Satoshi Itoh, Satoshi Sekiguchi,
    IPSJ Online Transactions, pp.34-46, Vol.5, Information Processing Society of Japan, Mar 2012
    DOI: 10.2197/ipsjtrans.5.34
    PDF (Full Text)
    This article presents a dynamic VM packing system exploiting postcopy live migration. Our experiments showed postcopy live migration greatly contributes to improving performance assurance and energy efficiency of IaaS datacenters.

Contact

Feel free to contact us.
  • Isaku Yamahata <yamahata _at_ valinux.co.jp>
  • Takahiro Hirofuchi <t.hirofuchi _at_ aist.go.jp>