How to transition from VirtualBox to KVM (using Virt-Manager)

Motivation

MDVL has long been enhanced with Oracle VirtualBox. Virtual machines are used to test new systems (like Debian Sid for a new MDVL), experiment with other OS (like Solaris) and run Windows software (mainly on an old WindowsXP VM).

While this worked well, the VirtualBox solution had a few disadvantages:

In order to solve some of these issues, multiple attempts have been made to transition to KVM with Virt-Manager. The attempts have often failed in early stages, due to some features being too difficult to use or not even available.

With the advent of Debian Jessie (Debian 8), the situation has finally improved towards a useful state. This article only discusses the KVM version included in kernel 3.16 / QEMU emulator version 2.1.2 and virt-manager 1.0.1. The situation may have improved with newer versions.

Why KVM and Virt-Manager?

One might wonder, why KVM was perferred over VMWare and XEN and why the front-end Virt-Manager was selected. While VMWare provides a few advantages over KVM and VirtualBox like Windows 3D acceleration and efficient memory management, it is also available under a non-free license. More annoyingly, this license also restricts commercial usage, which is not an option for MDVL.

The decision to prefer KVM over XEN mainly arose from the fact that XEN seemed more complex and previous experiments with KVM had already shown that KVM would be a good solution except for some points which could now be addressed as of the new version in Debian Jessie.

Virt-Manager was selected because it was reasonably simple to setup (it did not require setting up a whole data center ☺) and did not rely on a web-interface (current MDVL trends go toward eliminating most reasons to use a web-browser, thus a web-interface would have been counter-productive). Also, the user interface was similar to the previously used VirtualBox GUI and thus the transition would be simplified.

General Idea

In order to perform the transition, the following steps were performed:

Transition overview

On the left in the transition overview picture, you can see the list of VMs before the transition and on the right you can see the list of VMs after the transition. Successful migrations are connected with green lines, re-installations with yellow lines and not-migrated VMs are crossed with red lines.

Btw. if you are migrating Windows XP VMs, check http://slashsda.blogspot.de/2012/03/migrate-from-virtualbox-to-kvm.html. I did not know of that page when migrating my VMs, thus I re-installed all of the WindowsXP VMs, but you might save time, trying out the solutions suggested in the linked article.

Unlike VirtualBox, KVM offers a lot of configuration options. Many of them are exposed to the Virt-Manager GUI and thus have to be considered by the user. It can be very time-consuming to tune these settings which is why I am presenting a few options I have chosen for different OS here.

My recommended default configuration is not primarily targeted at performance, but compatibility instead. If you require the highest performance, you will probably need to tune the options even further.

I choose Virtio whenever possible.

I prefer to use the “Virtio”-type of virtual devices, i. e. “Virtio”-bus for virtual HDDs and “virtio” for the network-card type. Without any testing, I expect those devices with VM-optimized guest drivers to give the highest performance and support the most advanced features. In order to use virtio-devices with Windows, you will need to install the guest utilities for Windows (see “Windows Guest Utils” below).

Also, for virtual displays I always choose “Video QXL”. If you want to try some advanced (and highly experimental, in my tests: unsuccessful) graphics acceleration, you might also be interested to select “VMVGA” to use VMWare’s graphics driver. As you can see in the picture, I install a virtual tablet which is essential for all mouse-enhanced OS, because otherwise the pointer on the guest system can get out of sync with the host’s GUI.

Do not use this option. Instead power off the VM and configure and install it afterwards

I suggest you to always configure the VM as much as possible before doing the installation. In order to do this, you need to shut down the VM once it was started after the setup initiated after creating a new VM. Only then, you can see and modify all of the virtual hardware (there is also a customization in the setup which will not allow you to configure everything). Be sure to also add a virtual CD drive and make it bootable under “Boot Options” for OS installation. For Windows guests, I never choose to copy the host CPU configuration. This originates from the idea, that I want to be able to migrate the VM to a new (or just another) machine without having to re-activate Windows. But as Windows ties it’s activation to some of the processor’s properties, I prefer to go for a generic model. As with the “virtio” preference (see above) I generally go for the generic “kvm64” option if possible (works for Windows XP). Alternatively I choose old CPUs which I imagine many (and all newer) systems can “provide” like “Nehalem” (for Windows 8.1) or “core2duo” (for Windows 10). If you are using AMD processors or require enhanced performance, you might want to choose differently and consider the “Copy host CPU configuration” switch.

A sensible configuration for a memory-intensive VM which does not have to share it’s host with many other VMs.

I do not have recommendations on RAM and CPU amounts/counts but I generally recommend you to configure a lower “Current allocation” than “Maximum allocation”. I have yet to fully understand, what these mean, but I experienced the situation where a virtual Linux VM’s CPU and RAM count/amount could be increased while the VM was online up to the amount configured in “Maximum allocation”. If you have long-running or computationally intensive VMs, it might even be advisable to give the host’s maximum for “Maximum allocation” values.

For all options not mentioned here, I generally go for the defaults. See the specific sections below for recommendations for specific cases.

Virtual Networking

Virtual Networking is more difficult with Virt-Manager but also more powerful than with VirtualBox. While VirtualBox generally configures networking on a per-VM basis, Virt-Manager requires the creation of “Virtual Networks” which also need to be “Active” before they can be used by any of the VMs.

Virtual network which allows Internet access

Generally, the situation is similar to VirtualBox: For every VM we need to decide if we want it to be separated from the host’s network via a NAT or if we want to make it part of the host’s network (“bridge”-mode) or if we want to restrict communication to be between virtual machine guest and host only.

Virtual network which disallows Internet access

In order to separate VMs from the rest of the network, I have configured the virtual network “default” to be a NAT and in order to keep old operating systems (like Windows 2000 and Windows XP) away from the Internet, I have created a special virtual network “nointernet” as an “isolated network”.

Enabling VM saving

A feature I heavily use with any virtualization software is the ability to “save” a VM’s state and later restore it to the exact status it had when it was “saved”. This feature is similar to real computer’s “hibernation” (suspend to disk) feature but unlike that, “saving” a VM also works very well with linux guests and one can usually be very confident that even running/calculating applications can be paused and continued later this way. Finally, VM saving permits pausing a calculation, performing some other task and later resuming it. All in all, this feature is just essential.

Noo…. VM saving impossible?

I was very surprised, to find out that KVM and Virt-Manager often give an error message when attempting to use this feature (which is in the GUI’s menu).

Gladly, the message explains the reason and it can be fixed by directly editing a VMs configuration file and then restarting the VM.

In order to fix it, it is recommended to invoke virsh edit <VM-NAME> and search for invtsc. Delete the corresponding line and VM saving should work after restarting the VM.

Delete the line with invtsc… and “Error saving domain: Requested operation is not valid: domain has CPU feature: invtsc” is fixed!

If you want to do this manually instead, open /etc/libvirt/qemu/<VM-NAME>.xml and edit it as suggested before. Then you need to restart the libvirtd service (First stop the VM, then restart libvirtd, then start the VM). Either of these methods require root permissions. The first method is really preferable, because the service is not required to be restarted but the manual variant is probably useful if you want to automatically deploy VM configuration files…

Update 2016/07/23: There is another nasty error which can occur upon saving the VM. It yields a similar error dialog as above but has the message: Error saving domain: internal error: unable to execute QEMU command ‘migrate’: State blocked by non-migratable device ‘0000:00:06.0/ich9_ahci’. This one seems to be pretty unknown around the web, the solution can be found in a Korean E-Book. I gather you might not understand Korean (I can’t understand it either), so Google Translate is useful. The solution to this error is simple: Remove all SATA controllers and SATA devices from the virtual machine.

Remove SATA controllers to fix “Error saving domain: internal error: unable to execute QEMU command ‘migrate’: State blocked by non-migratable device 0000:00:06.0/ich9_ahci”

Enabling Nested Virtualization on Intel CPUs

Shortly after transitioning to Virt-Manager and KVM, I wanted to be able to employ nested virtualization similar to how it was possible with VirtualBox before. This feature is not all that important, but there are certain use cases for it. Luckily, I found a blog-post (as of today, it only remains only as an archive version) which explains how to do it:

When I tested this, these steps made it work but not very quickly… The nested VM’s performance is much slower than the “first-level” VM…

Improving Linux Guest Screen Resolution

Use “Video QXL” as recommended above. Generally, a few different screen resolutions can then be selected with xrandr (to list all) and xrandr --size 800x600 to select a specific size. While this is what can be expected of any OS, the situation can be improved by installing just two additional packages. The following are the Debian package names, probably different for other distributions:

Just installing them, will not immediately improve the situation. The spice-vdagent also has to automatically be started whenever the GUI is run (and the X-server has to be restarted for the new xserver-xorg-video-qxl driver to take effect).

For a Debian MDVL 64 VM, I have changed the ~/.xsession to contain these lines:

spice-vdagent
exec i3

This way, the spice-vdagent is executed upon login and just after that, the window manager is started. It is surely also possible to use any other means of automatically running this. spice-vdagent could also be launched manually if GUI access is only rarely necessary or the VM is never rebooted.

Windows Guest Utils

Generally, it is difficult to install Windows directly on a “virtio” drive. Therefore, I recommend installing Windows on a virtual IDE drive and later installing the necessary drivers.

The steps are as follows:

  1. Install Windows on a virtual IDE drive
  2. Install the guest utils (see below for details)
  3. Shutdown Windows
  4. Add a virtual Virtio-HDD to the VM.
  5. Start the Windows-VM.
  6. Make sure, the new drive is installed correctly under Windows (Check with diskmgmt.msc).
  7. Uninstall the IDE controller in the Windows device manager. This way, Windows will be able to start from a different drive type than before.
  8. Shutdown Windows.
  9. Change the drive type of the first virtual drive to “Virtio”.
  10. Remove the virtual drive added in step 4.
  11. Verify Windows starts correctly.
  12. If it does not work, switch the HDD back to IDE and re-add a virtual Virtio-HDD. Format it in Windows to see that it is really useable. Then continue with step 7.

If this was a bit too short of an explaination, see the pictures below for details on the different steps.

Installing the guest utils
This is pretty easy as long as you get the right installer. Go to http://www.spice-space.org/download.html and look for a spice-guest-tools-0.NNN.exe under “Windows binaries”. It should point you to an URL like http://www.spice-space.org/download/windows/spice-guest-tools/ where you can download the .exe-installer. Although the page suggests this is mainly for improved video (which it also does), the installer also installs network and HDD drivers for “virtio” devices.

Step 4: Adding a Virtio-HDD to a VM

Step 6: Windows has successfully recognized the additional virtual Virtio-HDD

Step 7: Uninstall the IDE controller in Windows

Windows XP: Shared folders

A feature I very much liked of VirtualBox was the “Shared Folders” option. This way, I could expose a directory on the host computer to the virtual machine without giving the VM network or Internet access. An important advantage of “Shared Folders” over virtual “external HDDs” was the ability to read and write to shared folders from both systems, the host and the guest, at the same time. This way, I could run a compiler inside the VM and the editor to enter the code outside the VM without any time difference from a local compiler.

Unfortunately, KVM + Virt-Manager do not support this for Windows guests. For Linux and probably other operating systems, there is a facility called “Filesystem passthrough” which does just about the same thing as “Shared Folders” in VirtualBox. I have not tested that feature, but I expect it to “just work” for Linux guests. For an example of how to use this, check http://www.linux-kvm.org/page/9p_virtio.

For Windows guest systems, we are left to “create our own” means of sharing files between host and guest. A very user-unfriendly means of achieving this is probably the following:

  1. Adding a second virtual HDD to the VM
  2. Writing files to the virtual HDD in the VM
  3. Unmounting the virtual HDD in the VM
  4. Accessing the virtual HDD from outside the VM (read/write)
  5. Mounting the virtual HDD in the VM
  6. Continue with Step 2.

I have actually experimented with this and the result is in two words: Not usable. Variations of this can be thought (like USB-passthrough of a real USB-Stick or such) but all of them share the same issue: There is no simple concurrent accessing the shared files.

Therefore we are left with another option often suggested when searching the web for keywords like “kvm share files windows guest”: Running a server on our host system and making the guest system access that server in order to exchange files.

The most commonly reported idea is setting up a Samba-Server, it seems there is a means of setting this up “on the fly” if configured correctly, see the discussion at ServerFault for details.

I do not like the idea of installing a server just to create Windows shares which can then be accessed by Windows guests. I did not like the idea of the guest being able to access a server on the host at all… but as there does not seem to be another solution yet, I want to use the SSH server which I already run on all host systems and which I know how to configure (unlike Samba).

Unlike the Samba-solution which works with many Windows versions, at least Windows XP to Windows 10, the SSH-filesystem solution probably only works with Windows XP (I have only checked it with Windows XP). If you are on a newer Windows version and want to use SSH and have checked the filesystem solution does not work, you might consider using a tool like WinSCP in order to transfer files between host and guest. Or see, if you can use a newer “Dokan” and “Win-SSHFS”: https://github.com/dokan-dev/dokany/releases

The solution I took, uses a set of programs similar to the Linux sshfs to connect a Windows drive letter to a SSH connection. In order to set this up, you need:

Newer versions, except for the dot-Net Framework are probably OK as well, if any of the links fail, notify me because (1) I still have the files, so you need not be stuck at this stage and (2) I may attempt to update the links if it is just that the download location has changed.

Compare the SHA256 checksums if you are not sure you have the correct setup files:

dokaninstall_0.6.0.exe a92ebb9f5baa83aa33a186d329eed2a2d063448432584f5bd88fd53bd44c0b3b
dotNetFx40_Full_x86_x64.exe 65e064258f2e418816b304f646ff9e87af101e4c9552ab064bb74d281c38659f
win-sshfs-0.0.1.5-setup.exe 2dee73dcfc449eea479014ee6c31839630618705f976e6bd12fd82c7e465adb3

Then, install these programs in the Windows guest. Create a key for the Windows guest to access your server via SSH. Encrypted keys (needing a passphrase) do not always work which is why I recommend you to create a special user for the virtual machine and create a separate key for that user so that it’s access permissions can be limited.

I have noted these points to setup the “Linux-side”

A key-pair can be created with ssh-keygen -b 16384 -t rsa (needs some time because 16384 is a large key length by current standards). You can not remember the key anyway, so why not go for a longer one ☺

Having installed all the programs on Windows, you should be presented with a small yellow icon in the task bar. Click on it and you will get a window where you can configure connections.

Win-SSHFS has been setup correctly.

As you can see, configuration is pretty straight-forward. Just enter the details every SSH client needs IP-Address, Port, Username and authentification. In order to configure the public key authentication, just select “PrivateKey” and give the path of your private key file. The field below the file is for a password – when I attempted to use an encrypted key, it gave me an error message saying that AES was not supported or such. Thus, I did not use an encrypted keyfile. The “Directory”-field refers to the host system and the “Drive Letter” is for the Windows guest. Finally, you can decide to automatically establish the connection or not.

Once setup, it almost feels like with VirtualBox’ “Shared Folders” before. However, there is a network connection. And once you “Save” the VM and reboot the host, then “Restore” the VM, it will have lost connection (Windows is not immediately realizing this) and you will have to take some effort to restart the DokanMounter-service and the win-sshfs.exe.

I did not yet manage to improve this procedure beyond clicking a few error messages and waiting for successful connection multiple times… it is probably faster to just shut down VMs which use Win-SSHFS instead of “saving” them.

After the transition

In summary, the transition provided a few benefits but also resulted in new problems. While there had been problems each time a transition had been attempted before, this time, they could be reduced to a usable minimum.

Still, this also means that as long as you are using VirtualBox and do not experience stability or licensing issues, there is not much of an incentive to switch.

Improvements archieved by transitioning to KVM + virt-manager

Unsolved issues and new problems

Although this has often been claimed, VM performance could not be verified to be largely improved or degenerated. KVM and VirtualBox (and VMWare) all perform just about equal in terms of CPU performance (sorry, no accurate measuring here).

For RAM usage it can definitely be said that VMware performs best, KVM performs OK and VirtualBox uses a lot of memory on the host system. (I often recall, although I have unfortunately not made any screenshot of this, the situation where I ran a Windows XP VM under VMWare and had a lower application-used memory size on the host-system than Windows XP memory usage as reported by the task-manager. I know that comparing these values is not “correct” but this can be interpreted as “guest uses more memory than host” and that is a very nice idea, even if it is not the whole truth ☺ ).

In terms of disk I/O I do not have any measures, which is why I can not tell. Virt-Manager offers a few options on I/O caching which sounded like they would not likely improve the situation over the default, but those might probably help in specific scenarios.

Virtual networking with KVM is often faster (by orders of magnitude like 40 MiB/s with KVM and 4 MiB/s with VirtualBox) than in VirtualBox, I do not have any comparison with VMWare here. Also, this highly depends on the type of virtual network card, virtual network configuration (like NAT) and where the actual target is located (on the same machine or remotely on the network).

All in all, I would summarize my performance impressions as: KVM is probably slightly better than VirtualBox, VMWare is probably better than both and it does not matter in the general case. In specific cases, however, it can not generally be said that one is faster than the other. It probably highly depends on the scenario and VirtualBox can probably not be recommended for networking-intensive scenarios. If RAM is important, VMWare is probably the best.

Information to be added

Convert to qcow2

http://www.agix.com.au/blog/?p=2696

qemu-img convert -f raw -O qcow2 /home/libvirt/images/vmguest1.img
                /home/libvirt/images/vmguest1.qcow2

In case Webcam Flickers in Zoom

Legacy notes about VirtualBox

Get USB working in VirtualBox

http://www.jeremychapman.info/cms/get-usb-working-in-virtualbox-under-debian-and-ubuntu

cat >> /etc/fstab <<EOF
none /proc/bus/usb usbfs devgid=INSERT_VBOXGID_HERE,devmode=664 0 0
EOF

Virtualbox 3D

bash-4.2$ ls -l /usr/lib/dri/swrast_dri.so
lrwxrwxrwx 1 root root 39 Nov 20 10:39 /usr/lib/dri/swrast_dri.so ->
                /usr/lib/xorg/modules/dri/swrast_dri.so

Virtualbox Compacting

http://superuser.com/questions/529149/how-to-compact-virtualboxs-vdi-file-size

Windows: sdelete -z
Linux:   Different means (zerofree or big4 -z)
Outside: VBoxManage modifyhd thedisk.vdi --compact

Ma_Sys.ma Website 5 (1.0.2) – no Flash, no JavaScript, no Webfont, no Copy Protection, no Mobile First. No bullshit. No GUI needed. Works with any browser.

Created: 2016/03/13 13:40:55 | Revised: 2022/09/18 20:48:27 | Tags: kvm, virtualbox, virtualization, mdvl, blog, migrate, transition, kb, virt-manager | Version: 1.0.4 | SRC (Pandoc MD) | GPL

Copyright (c) 2016, 2020 Ma_Sys.ma. For furhter info send an e-mail to Ma_Sys.ma@web.de.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.