This document contains the steps required for installing and configuring a KVM virtual host server for use as virtualized development lab environment. The main requirement was exposing virtual machines to the ambient local area network (bridged networking).
- Hardware: Lenovo ThinkServer TS140
- Host OS: Ubuntu Server 16.04 LTS
Hardware & BIOS Setup
- BIOS upgrade using a bootable DOS usb stick. Without the BIOS upgrade, system boot invariably failed if an external USB FDD was attached.
- Boot to BIOS. There’s also an option to use an EasySetup disk, but apparently it’s only useful when installing a windows OS or VMware ESXi.
- Enable virtualization extensions in CPU setup (VT)
- Enable VT-d as well if direct exclusive device assignment to guests is needed (see here).
Install Ubuntu Server 16.04 LTS, select the following features:
- Basic Ubuntu Server
- OpenSSH server
- Virtualization Host (installs the kvm host packages automatically. Leave unselected for manual installation)
Checking Hardware Virtualization Support
After installing the operating system, you can perform various tests to verify virtualization support.
egrep '(vmx|svm)' --color /proc/cpuinfo should display vmx or svm in red.
egrep -c '(vmx|svm)' /proc/cpuinfo should give non-zero result.
kvm-ok should give message saying:
INFO: /dev/kvm exists
KVM acceleration can be used
virt-host-validate should give a message like this:
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS …
lsmod | grep kvm should list either kvm_intel or kvm_amd
IOMMU (I/O Memory Management Unit) support
If the above command produces a warning
IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments then you can enable IOMMU like so:
Edit the file
/etc/default/grub and add
intel_iommu=on to the end of the existing GRUB_CMDLINE_LINUX option, separated by whitespace e.g.
Update grub and reboot
sudo update-grub sudo reboot
(Adapted from instructions for fedora)
Installing KVM and Command Line Management Tools
Install the following packages using apt-get, unless you selected “Virtualization Host” as one of the features during operating system installation:
ubuntu-virt-servermetapackage that contains:
bridge-utilsto manage bridged networking (brctl etc.)
The current user is added to libvirtd group automatically. Use
sudo usermod -a -G libvirtd <user> to add other users if necessary (log out and back in).
Check also that you belong to the kvm group and if not, run
sudo usermod -a -G kvm <user>.
The following packages are recommended:
ubuntu-vm-builder(one option for building virtual machines)
ubuntu-virt-mgmtmetapackage that contains:
virt-viewer(for viewing vms)
virt-manager(GUI for VM management)
virtinst(Programs to create and clone virtual machines)
libguestfs-tools(tools for accessing and modifying virtual machine disk images)
qemu-system(QEMU full system emulation binaries)
For libguestfs tools like
virt-sysprep to work (on Ubuntu), run the following commands after installing the package.
For more detailed instructions see here.
Setup Bridge on the Host
Setting up the bridge requires the bridge-utils package installed in the previous section. The bridge connects the virtual machines’ virtual network interfaces (taps) directly to the local area network through the host server’s primary network interface as if the virtual machines were physically present in the local area network along with the host.
Check the device name of the primary network interface using
nmcli dev status.
The device name may be something like “em1” or “eno1” (biosdevname naming for integrated network interfaces) instead of the familiar “eth0”
If network manager is in use, you may want to disable it before continuing (not necessary though).
On a local session, run
stop network-manager(don’t do this unless on a local session, kills the network connectivity entirely) and
echo "manual" | sudo tee /etc/init/network-manager.override
/etc/network/interfaces to disable primary interface (eth0/em1) unless controlled by network manager (no problem if it is), add bridge (with primary interface as bridged port).
ifconfig should show br0 with a valid ip address.
brctl show should show br0 connected to interface em1 and another bridge virbr0 (kvm private networking bridge).
nmcli dev status should show em1 as unmanaged (even without disabling network manager).
After starting a guest vm, a virtual interface should show on the bridge as well (e.g. vnetX or tapX).
Additional Bridge Setup Steps
These steps may improve the performance of the bridge (see Libvirt Networking).
Add the following to
sysctl -p /etc/sysctl.conf.
Add the following to
/etc/rc.local before the line with
Create a Virtual Network Using the Existing Bridge
This is an optional step and is not required for bridged networking. However a virtual network must exist for operations that rely on a named network, such as creating virtual machines with docker-machine.
Create the following network specification:
virsh net-create <file name> creates a transient virtual network based on the file, which is sufficient at least for the docker-machine case mentioned above
(the resulting VM configuration refers to the underlying bridge instead of the virtual network).
If you want to create a persistent virtual network, then run
virsh net-define <file name> followed by
virsh net-autostart <network name> and
virsh net-start <network name>.
Each guest needs a separate mac address. The
virt-install commands as well as VMM autogenerate a valid mac address if not explicitly specified.
The below script can be used to generate valid mac addresses (KVM Networking) if you want to specify them manually.
The manually created mac address can be specified in the virsh domain xml:
When installing the VM with
virt-install, the following parameters can be used:
--network bridge=br0,model=virtio,[mac=<predefined mac address>]
--network network=<network name>,model=virtio if you have created a libvirt virtual network using the bridge).
TODO network gets configured the same way even without the –network parameter if the network is persistent (eth0).
TODO Docker-machine created network docker-machines is assigned to interface called docker0 automatically on unrelated VMs. Why?
In VMM the bridge is configured during installation or later in the NIC section:
Debian based systems have the script
/usr/bin/kvmby default, that executes
qemu-system-x86_64 -enable-kvm <parameters>. Use the latter if the kvm command is not available.
About Guest Images
- Processor features: use
-cpu hostwith qemu to pass all host processor features to guest (don’t use if need to have portable image).
- Use virtio for networking if available (rtl8139 and e1000 have better guest support but virtio has better performance). For example, rtl18139 works with XP out of the box while e1000 and virtio do not.
- Use virtio for storage if available (best performance, but IDE has better support for guests). For example Windows OSes work with IDE or SATA out of the box but not with virtio.
- Easiest storage formats for images are qcow2 and raw. The latter gives better performance but reserves all allocated space when created. Disable cache when using raw.
- libvirtd logs from kvm commands are written into
More about tuning guest performance here.
Virsh can manage disk images in different kinds of storage pools.
The default pool is a directory pool, which refers to a directory on the local disk.
Manually created images can be added under libvirt management regardless of the location on disk.
However, VMM and
virt-install can create new image files only inside storage pools.
Therefore, if you want to store the images outside of the default pool,
you must either create a new pool or always create the image file first separately using
virsh pool-list to view existing pools (a single, default pool exists after installation).
Examine the xml returned by e.g.
virsh pool-dumpxml default to view e.g. the directory location of a pool.
To create a new directory pool, run the following commands:
After this, new images can be created with
virt-install or VMM in the new directory path.
A more advanced discussion on storage pools is outside the scope of this document.
Creating a Disk Image and Installing a Guest OS
Installation of guest OSes from cds, dvds or iso images usually requires that the user is able to interact with the operating system console (display, keyboard, mouse). The most straightforward way to accomplish this is to have an X session on the host (either physically or remotely, e.g. RDP or VNC). Guest installation on a headless server requires connecting to the guest machine over the network using VNC after starting the installation on the command line.
A lot of pre-installed images are available from various sources, for example:
The information in section Cloning Guests applies also for using pre-installed images.
TODO How to set up grub so that boot sequence never gets stuck in the boot menu.
Create a disk image with
qemu-img create -f qcow2 vdisk.img <disk size gigabytes>G.
Install the operating system from an installation media:
-boot d option instructs the OS to boot from the first CD-ROM drive.
-net nic,model=virtio -net tap part enables bridged networking.
Replace virtio with some other network hardware model (e.g. rtl8139) if the guest OS does not support virtio.
The kvm command with bridged networking must be run with root privileges unless the current user is granted permission to manipulate
/dev/net/tun (this is not a problem when using VMM or virt-install).
You can manually specify a mac address for the network interface, e.g.:
In a headless environment, you need to connect to the guest with a VNC client after starting the installation:
- Add the following parameters to the kvm command:
-vnc :X -no-rebootwhere X is the vnc display number (e.g. 2) which corresponds to a tcp port number in the range 59XX (e.g. 5902). The
-no-rebootswitch prevents the OS from rebooting automatically after installation.
- Connect to TCP port 59XX of the host machine (e.g. 5902) with a vnc viewer to complete the installation.
To password-protect the vnc session, add
,password to the
An additional option
-monitor stdio is required to get access to the qemu monitor (prompt
(qemu)) where the vnc password is set.
quit in the
(qemu) prompt to terminate the virtual machine and exit the monitor.
You can use the
-nographic option to open the guest console directly in the terminal if the guest OS supports it (exit terminal mode with Ctrl-A X).
-no-acpiuse this if guest is having problems with ACPI.
-localtimeset rtc to local time instead of utc. Required at least for DOS or Windows.
-usb -usbdevice tabletfor a better mouse experience in graphical mode (mouse control is better but may still be laggy).
After installation, the image can be started up with the same command line by removing the boot command or by changing it to boot from the first hard disk instead of the installation media (e.g.
The installation media option
--cdrom ... can also be removed.
-boot c option is the default and instructs the OS to boot from the first hard disk.
You can use the
-snapshot option to prevent any changes to be written on the disk image itself (writes to temp files instead).
In a headless environment, you can use the
-nographic option to run in terminal mode if the guest OS supports it (exit terminal mode with
If you need to Find the IP of a running VM and the mac address is known, run
arp -an | grep <mac address of the VM>
An example for installing and running a Windows XP guest:
The resulting image can be imported to be managed under libvirt using either
virt-install or VMM.
A third method is to create a domain xml file manually and use the virsh commands (see section Managing Guests) to define the virtual machine.
virt-install command can be used to create new or to import existing virtual machines.
The following command creates a new image and installs a system on it.
Note the size in the
--disk argument and the installation media specified using the
The disk location must point to a directory that belongs to a storage pool.
In a headless environment, you need to connect to the guest with a VNC client after starting the installation:
- Add the following parameters to the
--graphics vnc,listen=0.0.0.0,port=59XX,password=<pwd>where 59XX is the tcp port number for a vnc display (e.g. 5902 for display no. 2)
- Connect to TCP port 59XX of the host machine (e.g. 5902) with a vnc viewer and supply the specified password to complete the installation.
The following command imports an existing disk image.
--import switch skips installation phase and the
--noautoconsole switch prevents the command from automatically connecting to terminal or vnc display.
Note also that there is no disk size specified for the
--os-variant parameters can be used to set some useful default settings for the VM.
The supported OS Type values in VMM are Generic, Linux, Other, Solaris, UNIX and Windows. I haven’t seen these documented elsewhere.
The available OS Variants can be listed by running the command
virt-install --os-variant list. The
--os-type parameter is not required if
--os-variant is used.
For example, to install Windows Server 2012 R2:
For linux guests, the ttyconsole should be automatically created, check with
virsh ttyconsole <guest name>.
The newly created VM should show up in both
virsh list --all and VMM.
You can use the
--nographics option to open the guest console directly in the terminal if the guest OS supports it (exit terminal mode with Ctrl-A X).
With VMM (virt-manager)
virt-manager or Virtual Machine Manager from system menu.
The tool creates new images in the default storage pool
/var/lib/libvirt/images/ by default.
virsh list to see the image now running under virsh.
Curious people can look in
/var/log/libvirt/qemu/<guest name>.log to see the kvm command used for the installation.
To import an existing image, select the “Import existing disk image” option when creating a new VM.
TODO https://www.howtoforge.com/virtualization-with-kvm-on-ubuntu-12.10 for vmbuilder, LVM-based virtual machines
Console Access to Linux Guest From Host
virsh ttyconsole <name> should give e.g.
virt-install and VMM should create the following in the domain definition by default
echo $TERM output on host to determine which terminal to use in the command (
xterm is used here)
See KVM Access for more details.
/etc/init/ttyS0.conf with contents:
virsh connect <domain>. Press enter to get login prompt.
Ctrl + ] or
Ctrl + 5 or
Ctrl + [^¨~] to exit the console (depends on key map which escape
key works, see How to exit virsh.
If you’re experiencing problems while using the mouse on the guest system, try switching to a virtual tablet input device (absolute coordinates instead of relative).
--os-variant parameter with
virt-install should optimize the mouse along with other aspects of the guest.
There does not seem to be an explicit way of adding a tablet device with
When running manually with
kvm add parameters
-usb -usbdevice tablet.
In VMM add new input device:
In the guest definition XML the same is achieved with:
Share a Directory on Host to a Guest VM
Add the following under the devices section in domain xml. The label is used with the mount command later on the guest.
On guest, edit
/etc/modules and add the following lines if missing:
sudo service kmod start to load the modules or restart the VM.
sudo mount <label> <path on guest> -t 9p -o trans=virtio or add the following line to
/etc/fstab to mount automatically on boot.
You may need to experiment with different access modes and host directory permissions to get the read/write permissions right in your scenario. See these blog posts for more information:
A cloned virtual machine requires a new Name MAC address, UUID and a new host name (for DNS to work).
You can clone a virtual machine directly using either VMM UI or
Another option is to copy the disk image directly and import it with
In all cases the original virtual machine should be shut down before cloning / copying the image.
The last option is useful if you intend to keep template images ready for cloning new servers.
virt-sysprep utility in the
libguestfs-tools package that can be used to reset/unconfigure/customize/generalize a server image (e.g. add or remove static configuration like MAC address, host name, SSL certs) for this purpose.
The tool does not support windows images currently but probably will in a future version (in the mean time, there’s a windows tool
sysprep.exe that must be run on the guest).
For more information, see here: virt-sysprep.
All of the options above generate a new Name, MAC and UUID automatically. However, the hostname (and possibly other configuration) must still be set on the new guest itself.
There’s a guide for converting VirtualBox images to KVM images here: Convert VirtualBox Image to KVM Image
Cloning the Virtual Machine
Shut down the source guest VM before cloning, e.g.
virsh shutdown <source guest name>.
virt-clone -o <source guest name> -n <new guest name> -f <path-to-new-disk-image>.
Configuring the VM
The naïve way to configure the VM is to start it up, connect to it an make the changes there.
This can be done by starting the VM with virsh or VMM and connecting with
virsh console <new guest name>.
Alternatively, you can run the image directly with kvm and make the same changes there.
- edit both /etc/hostname and /etc/hosts on guest, set new host name in both files
- reboot guest or
sudo hostname <new hostname>
A better approach (for linux guests) is to use
virt-edit from the
virt-edit -d <new guest name> /etc/hostname
virt-edit -d <new guest name> /etc/hosts
Instead of the
-d option for guest name you can use the
-a option to supply a path to the disk image directly.
Unfortunately the EDITOR environment variable does not affect virt-edit so the command brings up the vi editor by default.
TODO maybe there’s another env variable… quick googling did not yield any results.
Libguestfs provides a lot more useful tools for vm image manipulation. See here: libguestfs.
If you do not have access to
libguestfs-tools, there’s another method described here: How to clone virtual machines in KVM - tutorial.
Resizing a VM Image
With the libguestfs tools, resizing cannot be done in place. Instead, a new image is created based on the old one. For more information, see the following documents:
Expand a single partition (linux)
Example of increasing the disk space on a single partition inside a basic linux VM. LVM must not be in use and the image must be in the qcow2 format. Shut down the virtual machine before executing the commands below.
virt-filesystems --long -h --all -a <old guest image> to list file systems.
Note the name of the partition to expand, e.g.
Run the following commands:
TODO kvm management tools http://www.linux-kvm.org/page/Management_Tools
virsh list --allshows all VMs managed by virsh
virsh define <filename>.xmldefine a new guest / update existing (guest name defined in xml file, changes effective after next boot)
virsh undefine <guest name>
virsh dumpxml <guest name> > <filename>.xmldumps the domain xml to given file
virsh edit <guest name>edit the domain xml for a guest, changes effective after next boot
virsh create <filename>.xmldefine and start a new guest
virsh start <guest name>starts the specified guest VM
virsh destroy <guest name>hard power off a guest
virsh shutdown <guest name>send shutdown signal
virsh suspend <guest name>send suspend signal
virsh resume <guest name>send resume signal
virsh reboot <guest name>send reboot signal
virsh console <guest name> [devname]connect to console on guest (optional device name)
virsh autostart <guest name> [--disable]sets guest to autostart on host start
virsh dominfo <guest name>displays basic domain info
- TODO virsh save/restore and other stuff https://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Managing_guests_with_virsh.html
--connect qemu:///system argument with these commands to explicitly target the local daemon’s system VMs (sometimes necessary)
virt-cat -d <guest name> <file path>
virt-cat -a <path-to-disk-image> <file path>
virt-edit -d <guest name> <file path>
virt-edit -a <path-to-disk-image> <file path>
For more commands, see libguestfs.
About Edit Commands
The default editor for edit commands like
virt-edit is vi.
For virsh commands, the editor can be changed using VISUAL or EDITOR environment variables.
For libguestfs commands, the editor can be changed using the EDITOR environment variable.
export EDITOR="nano -w", revert with
Libvirt changes the ownership of image files as part of it’s operations, regardless of original ownership.
- When the virtual machine is started, the ownership is changed to
- When the virtual machine is stopped, the ownership is changed to
Therefore you may need to change the image file ownership prior to executing any commands directly against the image file (such as the libguestfs commands) or run the commands as root.
Deleting a virtual machine
If the VM is running, execute
virsh destroy <guest name>.
virsh undefine <guest name>.
Updating the Guest Definition XML
virsh dumpxml <guest name> > <filename>.xml to dump the virsh definition xml.
Make changes to the domain xml and execute
virsh define <filename>.xml to update the definition (the name of the domain is included in the xml).
Or edit the domain xml directly using
virsh edit <guest name>.
In both cases, the changes will take effect on next proper boot (reboot has no effect):
virsh shutdown <guest name>
virsh start <guest name>
An example domain xml file:
Valid units for memory: b, KB, KiB, MB, MiB, GB, GiB etc.
From the documentation:
Valid units are “b” or “bytes” for bytes, “KB” for kilobytes (103 or 1,000 bytes), “k” or “KiB” for kibibytes (1024 bytes), “MB” for megabytes (106 or 1,000,000 bytes), “M” or “MiB” for mebibytes (220 or 1,048,576 bytes), “GB” for gigabytes (109 or 1,000,000,000 bytes), “G” or “GiB” for gibibytes (230 or 1,073,741,824 bytes), “TB” for terabytes (1012 or 1,000,000,000,000 bytes), or “T” or “TiB” for tebibytes (240 or 1,099,511,627,776 bytes). However, the value will be rounded up to the nearest kibibyte by libvirt, and may be further rounded to the granularity supported by the hypervisor. Some hypervisors also enforce a minimum, such as 4000KiB.
See Libvirt Domain XML Format for more information on the file format.
References & Resources
- KVM HOWTO
- KVM Installation
- KVM Command Line
- KVM Intro
- KVM Tuning
- How to Debug Virtualization problems
- KVM Networking (Ubuntu)
- KVM Networking
- KVM Network Bridging
- KVM Access
- KVM Management Tools
- KVM Virsh Help
- KVM VT-d
- Ubuntu libvirt guide
- Libvirt Domain XML Format
- Libvirt Networking
- How to exit virsh
- Convert VirtualBox Image to KVM Image
- Find the IP of a running VM
- How to get started with libvirt on Linux
- File System Pass-Through in KVM/Qemu/libvirt
- Sharing directories with virtual machines and libvirt
- How to clone virtual machines in KVM - tutorial