QEMU, XEN, and Windows
Mozilla got me a new computer recently: one that can build Mozilla clobber in 7 minutes, or do a full XPCOMGC rewriting pass in 90 minutes. I wanted to be able to run Windows in a VM on this machine, but I’ve had a really unpleasant time setting up virtualization. This is a tale of what I’ve tried:
- I’m running Fedora Core 8.
- I told it to install the virtualization group on install, which means I got both the Xen and KVM stacks. Installing the Xen stack automatically sets up grub to boot into a Xen hypervisor.
- I tried creating a new VM in the Virtual Machine Manager GUI. I chose two virtual processors.
- I tried to install Windows XP by hooking that virtual machine up to /dev/cdrom directly. But the host automount factility kept causing conflicts between the guest and the host. So I copied the Windows CD to a local .iso file and told Xen to use that instead. So far so good, Windows XP tries to install.
- Every time Windows tries to restart (which is a lot, when you’re trying to install Windows, updates, and developer tools) the VM hangs. You can’t even shut down the VM from the host GUI: it’s stuck, and you have to reboot the host.
- Connecting to the VM from VNC is… difficult. If you use the default option of PS/2 mouse emulation, the guest mouse cursor doesn’t track the host cursor correctly. Configuring the USB tablet emulation fixes this problem. But frequently when typing quickly the keyboard will have doubled keys, missed keys, or the modifiers (shift and control) will simply stop functioning.
- Trying to connect from another machine (my mac laptop) is an exercise in disappointment. My old standby “Chicken of the VNC” won’t display anything other than black. A newer client, JollysFastVNC, connects, but I can’t make the “control” key work.
- I give up and disable Xen, starting the same process with KVM.
- I get a lot closer. The virtual machine can start and reboot successfully. But 2 CPUs run at 100% the entire time, even when the guest VM isn’t doing anything. I read bug reports and see that this is supposedly a problem with very old versions of KVM, but was fixed long before my current version (version 60). Virtual machine performance is terrible.
- I try a workaround I found via Google, installing the “Standard PC” driver instead of the “ACPI Multiprocessor PC” driver in Windows. This doesn’t help.
- I try allocating only one processor to the VM. This causes Windows to hang on bootup. Going back to 2 CPUs keeps the hang.
- I’m frustrated, so I re-install Windows from scratch with one CPU. This makes performance somewhat better.
- I discover that I’m not using a bridged network, but an emulated private network with NAT.
- I start to read docs about how to set up a bridged network with KVM/QEMU. They all involve launching KVM from a commandline, not from the GUI. And when I try to follow the directions, my local network goes down.
- I finally give up using the Virtual Machine Manager GUI. I switch to using the qemu-kvm command line, and get a virtual machine running using that. I can get a bridge network, maybe, but it doesn’t actually work in the guest VM.
- I discover that an SDL window in the host is a lot more reliable than VNC.
- I discover that the Windows CD I have is an OEM version and isn’t legal on this machine.
I’m still frustrated. I now have a Windows Vista license that is legal on this machine, but I haven’t installed it yet: I really want a bridged network, and preferably one that comes up automatically every time the host machine starts. I don’t really know much about Linux networking, or even what “a network” means in Linux… I just want a simple option like I used to have in VMWare: “Use bridged network” and it all works… but with better performance because KVM and Xen can both use fully virtualized I/O and multiple CPUs.
Does anyone know of a good guide to getting this working in a permanent sort of way?
March 10th, 2008 at 8:21 pm
One thing you can do is to install Vista natively on a partition, and then just point the VM bits at that partition. That should get you around the OEM install issues.
March 10th, 2008 at 8:26 pm
Whoops, hit enter too soon. Why not just use VMWare? There’s probably a slight performance loss, but I think it’s more stable with a windows VM.
March 10th, 2008 at 9:12 pm
Why not use vmware?
March 10th, 2008 at 9:40 pm
I would also echo Vlad regarding VMWare. The VMWare Server is even free, and installs easily on Linux. Despite the name, it works great on my laptop; haven’t had a need for the workstation in ages. I have had good experience running this on Ubuntu.
I don’t think bridged networking is supported in VMWare if you are using wireless networking from host; I just use NAT.
Occasionally I have had the Windows guest die quickly after startup. Most of the time this turned out to be a case where I had upgraded my host kernel and had forgotten to recompile the vmware tools. In any case, it is a good idea to take a snapshot of the guest once you have had everything installed so that if the guest becomes unusable for any reason you can return to a snapshot.
I tried to use a native Windows partition for the guest, but I never got it working properly.
Please note that I have only run Windows XP guest, not Vista.
March 10th, 2008 at 10:12 pm
Along those lines, have you looked into VirtualBox?
March 11th, 2008 at 12:35 am
When connecting to the Windows machine, you really want to be using RDP (rdesktop) instead of VNC. It’s tons better.
March 11th, 2008 at 2:09 am
I’ve had lots of fun with VirtualBox. Works well with XP (even things like USB passthrough), don’t know about Vista.
March 11th, 2008 at 4:21 am
Also, VMware Workstation has an embedded VNC server, bridged networking and supports two CPUs.
March 11th, 2008 at 4:41 am
Surely just use vmware server. Its will work just fine plus you get truly portable vm’s
March 11th, 2008 at 4:49 am
Benjamin, could you detail the specs of your new machine? I’m impressed by that 7 minutes clobber build :-)
March 11th, 2008 at 5:18 am
I’d be curious to learn your reasons for switching away from VMware. I’m all for open source, but your experience seems to confirm (mine and others’) that VMware is the most painless approach at least when you want to (or rather: have to) virtualise non-open OSs. Free VMware server (2) doesn’t seem to play nice with SELinux on FC8, but Workstation should be fine.
VMware uses VT extensions as well, can use multiple CPUs, and should have optimal I/O performance when using VMware tools (virtualised I/O). What would be the advantages of KVM or Xen, besides being open source?
March 11th, 2008 at 8:41 am
You could also try VirtualBox. I haven’t used in on a linux system yet, but I’ve used it on a Windows host running Linux, ReactOS, and WinXP. Very easy to get a new VM up and running quick. Performance seems to be pretty good as well.
March 11th, 2008 at 9:52 am
You’d better use VirtualBox or VMWare Server.
+ VirtualBox : Free, there’s an opensource version, easy GUI, easy to install, fast : http://www.virtualbox.org/
+ VMWare Server : Free, can be easy to install, can be managed remotely with VMWare Server Console Client : http://vmware.com/download/server/
http://www.linux-wizard.net/index.php?id_blog=188
March 11th, 2008 at 10:03 am
Please note that Xen and KVM are not faster than VMware …
Xen is faster than VMWare if the guest is using paravirtualized drivers. However whereas there are paravirtualized for Linux, paravirtualized drivers fro Windows are not free, and so Xen end up doing full virtualisation.
The same for KVM.
Last time I check, VMWare server support several CPU, or if the version 1 doesn’t support it, you can try the 2 beta. IIRC, VirtualBox also support multiple CPU.
To have permanent bridge network, you will have to use TUN/TAP and/or customised your init scripts when using Xen and KVM.
For VirtualBOx, have a look here :
http://www.virtualbox.org/wiki/Advanced_Networking_Linux
http://www.virtualbox.org/wiki/User_HOWTOS
March 11th, 2008 at 11:23 am
You could always try running the process in reverse by hosting Fedora in coLinux in Vista.
March 13th, 2008 at 1:38 pm
I am running CentOS 5, and have had trouble with XEN and with the VMWare server beta. However VMWare server 1.x seems to work well as long as the development tools are installed and VMWare can rebuild its kernel module.
March 16th, 2008 at 6:09 am
I’ve also experimented with the different virtualization packages around, and I prefer Vmware Server for it’s simplicity though it would be nice to have a more efficient solution. I will be paying close attention to KVM, especially since it’s now officially supported in Ubuntu Hardy Heron, which will be released in april. However, one thing you might not be aware of, is that Vmware Server always swaps when you run it on a Linux host and does so unless you’ve deactivated the pagefile on a Windows host. It’s possible to disable that function in Linux though, and you probably wouldn’t believe the performance boost that represents, particularly if you’re using a normal harddisk and not some super-fast storage. This is what you should do; 1) make sure your host Linux system uses tmpfs for /tmp, 2) disable mainMem.useNamedFile for the VMs to use /tmp (which now uses RAM instead of disk after step 1) instead of a file. 3) Deactivate memory sharing between virtual machines. You probably have more than enough RAM, and disabling this will give you a performance boost. Also notice that using a single virtual cpu in the guest is often faster than using two or more.
Lars Troen has written a very good document on this: http://vmfaq.com/?View=entry&EntryID=25
Following those advice, I got an extreme performance boost.
March 18th, 2008 at 5:56 am
Hello,
I’m running several virtual machines under production boxes (big servers) but also on my own laptop (A Sony Vaio where virtualisation was disabled by the bios (with no option to activate it, so you have to change a bit on the dump of the bios ‘by hand’)…
KVM, in theory, and in practice, is a lot faster than VMware as it’s para virtualisation and not full virtualisation, the guest os has acces to your processor quasi directly, and use the facility’s of the VT functionality (if you own an intel, i don’t remember the marketing name for AMD virtualisation tech). Devices are emulated (CG, network and sound) but you can have direct access to your USB Device.
If Windows is slow it’s in general because of an ACPI problem. The solution to change it after the installation process did not work for me, i had to do the full install with the no-acpi option. No-acpi means that you can’t have SMP enabled in the Guest OS, so the -smp 2 (or any number) can be removed from the command line (there is no SMP without ACPI).
So in a concise (in fact not so, but i explain in case you have problems with modules…) way :
1- Be sure to check that you have the KVM (and kvm-intel or kvm-amd) module compiled with your kernel. Maybe you’re just running qemu without the KVM module loaded, so don’t expect good performances. A simple lsmod in shell prompt should show you the kvm module loaded (modprobe kvm-intel or modprobe kvm-amd to load it in other case)
2- Prepare an image to install windows on it : qemu-img create WindowsXp.qcow -f qcow2 10G
This will prepare a qcow2 image wich will grow only when you put data (so the image will only be around 1GB after installation)
3- Install Windows with : kvm -hda /path/to/WindowsXp.qcow -cdrom /dev/cdrom -m 800 -no-acpi -std-vga -boot d (you can add more options see kvm –help) to install from your cdrom (change the path to /dev/cdrom to your device if it’s not this one)
4- Prepare your network, you should have TUN/TAP enabled in your kernel, and Bridged Network. You should also compile the Netfilter/Iptables modules to have access to the nat table. Lots of problem comes from the fact that default kernel does not include those kind of options by default.
5- Install the uml-utilities (usermode-utilities on gentoo) and bridge-utils for your distro (your distro *should* have the packages)
(usermode utilities webpage : http://user-mode-linux.sourceforge.net/)
5- here is an init script to automatically mount your bridged network :
———————————————————————-
#!/bin/bash
# id of the user running qemu (kvm)
USERID=1000
# number of TUN/TAP devices to setup
NUM_OF_DEVICES=1
case $1 in
start)
modprobe tun
echo -n “Setting up bridge device br0”
brctl addbr br0
ifconfig br0 192.168.100.254 netmask 255.255.255.0 up
echo “1” > /proc/sys/net/ipv4/ip_forward
for ((i=0; i < NUM_OF_DEVICES ; i++)); do
echo -n “Setting up ”
tunctl -b -u $USERID -t qtap$i
brctl addif br0 qtap$i
ifconfig qtap$i up 0.0.0.0 promisc
done
;;
stop)
for ((i=0; i /proc/sys/net/ipv4/ip_forward
;;
*)
echo “Usage: $(basename $0) (start|stop)”
;;
esac
———————————————————————-
The Gateway will be on 192.168.100.254 (it will be the Host OS)
So copy this file in /etc/init.d/kvm-network for example, and “chmod +x /etc/init.d/kvm-network”
Type “/etc/init.d/kvm-network start” in a shell, if all went fine it you should not have any error (hehe).
If something goes wrong, it should be a lack of the userland tools (bridge-utils / uml-utilities) AND/OR a lack of the TUN/TAP device module (so you’ll to recompile your kernel)
You’ll have to add the Masquerading in iptables with (in a shell prompt) :
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
(replace eth0 with the name of the device connected to the net (it could be ppp0 for example)
You could have an error like “can’t initialize iptables table `nat’ …
It means that you don’t have the iptable_nat module compiled with your kernel (If you don’t want to go in deep details, compile everything under NetFilter (Networking -> Networking Options->Network Packet Filtering framework : IP:Netfilter Configuration-> Full NAT + Masquerade / Bridge -> ebt: nat table support) as modules (maybe other modules are needed (do that by memory)), don’t forget to compile the 802.1d Ethernet Bridging (Networking -> Networking options->802.1d Ethernet Bridging too and the TUN/TAP (Device Driver->Network Device support->Universal TUN/TAP)
If everything went well you should start your window box with :
kvm -hda /path/to/WindowsXp.qcow -m 1024 -alt-grab -no-acpi -soundhw es1370 -usb -std-vga –cdrom /dev/cdrom -alt-grab -usbdevice tablet -net nic,model=rtl8139,macaddr=52:0B:02:97:BF:29 -net tap,ifname=qtap0,script=no
(this is my start command)but put what you want for the memory (-m), the sound etc… Just keep the -net nic and -net tap like that (change the macaddr for what you want, if you run multiple VM every should have it’s own Mac Address)
Under Windows, configure your network with : Ip address : 192.168.100.xxx , netmask : 255.255.255.0 gateway : 192.168.100.254
DNS : your DNS (grab them in your /etc/resolv.conf)
You could add the kvm-network script with your update-rc.d (or update-rc) script to your init to have the bridge automatically mounted at startup.
You could create a “Link to application” or something like that in your windows manager, with the full kvm command, to start you VM just by clicking a desktop icon. You should change the userid in the init script to your own user id on which the VM will be run.
Remeber to load the iptables IP masquerading before launching your VM or you won’t have access to the net. (Google iptables init script to find a script to load you rules at startup, or simply had the iptables command in the start section of the kvm-network start script
Sorry for my bad english, i’m french, and i’m very tired (didn’t slept at all this night :p )
PS: KVM could be a symlink to qemu-system-x86_64 binarie if you distro don’t have this one.
Hope it’ll help you. It seems a little complicated, but after some test you’ll see that you can add virtualisation to linux box very quickly (when you know what to do and have the scripts to help). It’s a lot more simplier than Xen (with Xenified kernel) and as i remeber, you could not have Windows guest on XEN (as you must have a xenified kernel)
March 18th, 2008 at 6:09 am
Oh i forgot,
i use the -std-vga because i have a wide screen on my laptop, so if you have a wide screen too or you want to use a resolution higher than 1024*768, use this option and under windows install this driver :
http://www.navozhdeniye.narod.ru/vbempxpg.zip when windows ask you for the video driver
Here is the page of the VBE driver project : http://www.geocities.com/bearwindows/vbemp.htm#2
It’s the driver included in ReactOs which work fine (i found that in the KVM documentation)
Bye and good luck :)
March 29th, 2008 at 7:37 pm
I found Xen to be surprisingly easy to set up on CentOS 5.1 Linux running full hardware virtualization with Windows 2003. We run VMWare ESX in production but licensing every little feature is a pain. I wanted to see how close Xen 3.x with full hardware virtualization was to ESX, and I ended up quite impressed with Xen.
On a Dell 2970 with two AMD 2212 dual-core processors with hardware virtualization, the process was something like this:
1. Install CentOS 5.1 with the virtualization bundle, configure, do a yum update, reboot to an updated kernel and confirm that the output of “uname -r” shows a xen kernel is running. I ended up with Xen 3.0.3-41 from the CentOS/Red Hat installation.
2. In /etc/xen/xend-config.sxp, uncomment “(network-script network-bridge)” and change “vnc-listen” to 0.0.0.0 to enable VNC “virtual console” connection from a remote workstation.
3. “/etc/init.d/xend start; chkconfig xend on” to start the xen daemon and add it to the init scripts to start after a reboot.
4. Make a large, blank file for the new VM’s disk image: “dd if=/dev/zero of=/srv/xen/domains/winserver/disk.img bs=1M count=8192” for an 8gb disk in /srv/xen/domains/winserver/.
5. Copy the prototype hardware VM file (HVM) “cp /etc/xen/xmexample.hvm /etc/xen/winserver.cfg” and change a few parameters in winserver.cfg:
memory=1024 # One 1gb RAM
# Boot off of an ISO you made with “dd if=/dev/dvd of=win2k3r2-32en.iso” or similar
disk = [ ‘file:/srv/xen/domains/winserver/disk.img,hda,w’,
‘file:/srv/iso/win2k3std-en.iso,hdc:cdrom,r’, ]
boot=”d”
usbdevice=tablet # Mouse works much better in the VNC console with this set
6. “xm create winserver.cfg” starts the virtual machine each time. If you shut down Windows, the VM shuts itself down (check with “xm list”).
7. From a Unix/Linux graphics workstation: “vncviewer xenserver.example.com:5900” to connect to the first VM (called a domain) on the server, or use your favorite VNC client. You should see the white-ASCII-on-blue Windows installer. Bridged ethernet works, the Windows can pick up a DHCP address and Microsoft updates right away.
You can run a Windows server on Xen using Terminal Server, and connect to it from your Linux/Unix/Mac clients with rdesktop. This way you don’t have to dedicate a server or special client workstations just to run some particular Windows app that most shops seem to need. VMWare ESX Virtual Infrastructure Client only runs on Windows, but VNC is an open spec and clients are available for most operating systems.
I’ve just started working with Xen but it’s been working so well for me that I’ll probably use it to run Windows on laptops occasionally, when spare Windows licenses are available. Next I’m going to try installing Windows Vista Ultimate and ReactOS with the full hardware virtualization.
April 21st, 2008 at 9:20 am
Hello,
I did have VMM and Qemu KVM works originally and somehow during updates it buggered up my system and my bridge network didn’t work anymore. The best way to fix it using the GUI instead playing up with shell script is:
1. you open VMM and go to “Host Detail” on your qemu connection.
2. open Virtual Networks tab and you should see one Virtual Network (by default)
3. just create a new one or replace your current one
4. follow all the steps until the part of “connecting to physical network”
5. select forward and select your eth0 or whatever network device you have on your comp
6. finish the setup
7. open your guest’s machine detail
8. just remove your current NIC and recreate it and choose the “virtual network” to the one you just finish in step 3-6
9. you should be able to have a bridge network between your guest and host.
About the performance issue it might be related to USB, i know from some posts that the USB can skyrocket your CPU usage. but I haven’t been there, so cannot give you any suggestion