The introduction to this series, Getting Into VoIP, explained some of the requirements and testing that went into choosing a VoIP environment and application. One really-want (if not must-have) requirement is getting it to run in a virtual machine. There are so many advantages to virtualization:  easy backups and moving VMs to different hosts, to name two. However…

Danger Will Robinson!

“Don’t do it!” That was the most consistent response I got in the PBX in a Flash (PiaF) forum when I asked about running PiaF in a virtual machine. Why would you want to run a real-time application in a virtual machine? It’s fine for testing but don’t use it in production!

Well I’m a stubborn nut and when someone says it can’t be done, I feel more compelled to do it. Besides, we’re talking about my home office here, not some huge enterprise installation. How much power could the virtual machine need? How hard could it be?

Plenty Hard

It turns out that there are lots of variables to test when running a Linux guest in a virtual machine. PiaF uses CentOS 5.2, so all of my testing was done with that flavor of Linux. Even with that constant, there are a lot of variables, which made finding a workable solution quite a lot of work:

• VMWare or Microsoft?
• VMWare Player, VMWare Server, Virtual PC, or Virtual Server?
• With or without the Linux VM Additions offered by each vendor?
• What kernel options to use? clocksource? nosmp? noapic?
• CentOS “standard” with its 1000 Hz clock frequency, or the special “vm” kernel with its 100 Hz frequency?
• If CentOS “vm”, will the recompiled Zaptel drivers run?
• How to install CentOS when the installer uses 24-bit video and makes the screen illegible?

Choosing a Virtual Platform

Environment

I want to use an old PC to run my home-office PBX. It’s a Pentium 4, 2 GHz, 1 GB RAM, 80GB IDE hard drive, running Windows XP Professional. I’ll be installing the 32-bit version of PBX in a Flash 1.3, which installs CentOS 5.2 with the 2.6.18-92.1.6.el5 Linux kernel. During the install, I selected the Asterisk 1.4 over the newer 1.6. I read that Asterisk 1.4 is still the recommended version for production use.

It’s about Time

The biggest issue with Linux virtual machines, it turns out, is time. The clock runs too fast or too slow, or it requests so many interrupts that CPU usage on the host stays very high, or it doesn’t provide an accurate time source to Zaptel and throws errors. When the time is messed up, the audio usually sounds choppy. Finding the right virtual host has a lot to do with finding a host that can properly handle timing issues.

VMWare

You have to hand it to VMWare, they know about Linux. Just look through their documentation and you’ll see all the flavors of Linux they’ve tested.

Interestingly, running PiaF in the free VMWare Player with the standard CentOS kernel, pre-built Zaptel drivers, no kernel switches, and no VM additions, works pretty well! I actually followed the instructions on the PiaF web site for running in a Virtual Machine, but I didn’t realize that I needed to manually select the alternate kernel when booting, so I was using the standard kernel. However the clock did seem to drift in this scenario.

I wanted to run a server product so I can get the virtual machine to stop and start when the host machine needs to be rebooted. I downloaded and tested the enormous VMWare Server. I figured I needed to get the VM extensions installed to solve the clock drift issue, plus they would be required for shutdown and startup synchronized with the host. But the VM additions would not install when I told VMWare Server that the client was Red Hat Enterprise Linux 5 (which CentOS 5 is based on).

Since I got stuck here, and since it looked like the Microsoft platform would work, I did not test the “vm” kernel or recompiling Zaptel under VMWare.

Microsoft

On the Virtual PC/Server side, a “bare install” of PiaF leads to choppy audio and to host CPU usage consistently above 30%, even when the guest is idle. Installing the CentOS “vm” kernel drops CPU usage to under 15%, and the audio sounds good! A combination of VM Additions for Linux and some kernel parameters seems to keep the clock fairly accurate (still to test on a long-term basis).

The drawback is that after recompiling and running Zaptel, I was getting “rtc: lost some interrupts at 1024 Hz” errors, and I was back to choppy-sounding audio. The zttest command showed terrible results. Since I’m not using Zaptel hardware, the only Zaptel driver in question is ztdummy. ztdummy is only needed for “MeetMe” conferencing and IAX trunking, both of which I can do without for now. So I will simply disable Zaptel.

Like VMWare Server, Microsoft Virtual Server runs as a service so it can keep the guest OS running even when no one is logged on to the host. With the help of the Microsoft VM Additions for Linux, Virtual Server will let me reboot the guest OS when I reboot the host. So I’ll go with Virtual Server as the platform for now. The rest of this entry explains how to set that up.

Setting Up CentOS under Virtual Server

Follow these steps to set up CentOS under Virtual Server using the Piaf installation ISO:

1. Set up a new virtual machine with 512MB of memory, an 8GB virtual IDE hard disk, and a connected network card.
2. Download the PBX in a Flash 1.3 ISO file. Attach the ISO to the virtual machine’s CD-ROM drive and boot the machine.
3. At the first install prompt, type ksalt. This starts the install in text mode, which avoids the issue with the 24-bit video. It also does not use the LVM file system, which apparently adds unnecessary overhead (according to jroper). I deliberately chose a very simple root password during the setup so I could take this virtual machine to client sites one day if needed.
4. After the install completes, it will reboot. Quickly detach the ISO file or it will go back into the installation again. After the boot completes (you’ll see the “User” prompt), shut down the machine by sending a Ctrl-Alt-Del from Virtual Server and stopping the machine when it is at the BIOS startup stage. Make a copy of your virtual machine at this point, and/or turn on Undo Disks, so you can come back to this point if you encounter problems with the steps below. (WinZip is great for creating a compressed snapshot of the virtual machine. Name it something like “PiaF.1.Immediately after CentOS Install.zip”)
5. Start the virtual machine and log on to CentOS. Make sure that CentOS is online. At a command prompt:
   /sbin/ifconfig # check whether an IP address has been assigned; if so, go to the next step
   system-config-network # if you need to check the network setup
   nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 # view network setup in text form
   service network restart # restart with updated network setup if you made changes
   /sbin/ifconfig # check that an IP address has now been assigned
6. Add the kernel options. Regarding the clock source, see Microsoft KB 918461, but note that clocksource is now preferred to clock (see this post). nosmp disables Symmetric MultiProcessing. noapic and nolapic were recommended in other posts.
   nano -w /boot/grub/menu.lst 
   At the end of the “kernel” line, add this string:
   clocksource=pit nosmp noapic nolapic
7. Update the kernel with the CentOS Virtual Machine kernel. This is outlined on the PiaF site, but do not recompile Zaptel as instructed there. For some wonderfully arcane discussion of what the “vm” kernel does and why it is needed, see this CentOS bug report. The “vm” kernels are available here. Note:  choose the version that matches your current kernel.
   cd /root
uname -r # Check your Linux version. Modify the next two lines to match.
   wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-2.6.18-92.1.6.el5.i686.rpm
wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-devel-2.6.18-92.1.6.el5.i686.rpm
rpm -ivh kernel-vm*.rpm
8. Edit grub to make the new kernel load by default.
   nano -w /boot/grub/menu.lst In this file, set default=0 if the .e15vm kernel is listed first. After saving the file, reboot.
   shutdown -r now

9. After the reboot, check the Linux version to make sure it ends in “vm”:
   uname -r

10. You may also want to confirm that the clocksource is set to pit using the method suggested in this thread):
    cat /sys/devices/system/clocksource/clocksource0/current_clocksource

Installing Microsoft Virtual Machine Additions for Linux

You’ll need to download and install the Microsoft Virtual Machine Additions for Linux. Besides the Readme file that is installed with that package, I’d like to acknowledge two helpful posts that pointed me in the right direction:  Installing VM Additions on Linux by Erik Luppers, and a forum thread titled best “free” linux that has real additions support, in particular Alexander Kaufmann’s post.

1. Normally you would need to run yum update kernel, reboot, then run yum install kernel-devel gcc to get the kernel source. However, it seems that the PiaF already includes the kernel source, so this step is not needed with PiaF.
2. From Virtual Server, attach “VMAdditionsForLinux.iso” to the CD drive. Then inside the virtual machine, mount the drive and copy the files to a new directory as follows:
   mkdir /media/cdrom
mount /dev/cdrom /media/cdrom
mkdir /usr/src/vmadd
cp /media/cdrom/*.* /usr/src/vmadd
umount /dev/cdrom
   You can release the ISO image from the virtual machine at this point.
3. Install individual VM Additions modules, skipping X11 and SCSI (which we aren’t using):
   cd /usr/src/vmadd
rpm -ivh vmadd-kernel-module-RHEL-2.0-1.i386.rpm
tail -100 /var/log/vmadd-kernel-module.log | more # Make sure the main install/make worked
   rpm -ivh vmadd-heartbeat-2.0-1.i386.rpm
rpm -ivh vmadd-shutdown-2.0-1.i386.rpm
rpm -ivh vmadd-timesync-2.0-1.i386.rpm
   A note on SCSI:  although it’s a simple matter in Virtual Server to change the hard drive to SCSI, it looked like a fair amount of effort to get the (faster) SCSI drivers from VM Additions configured. Since some contend that IDE is just as fast in a Linux virtual machine, I’ll just stick with IDE.
4. You could start services manually as below, but I just rebooted:
   shutdown -r now

Here are the commands to start the VM Additions services manually:
/etc/init.d/vmadd start
/etc/init.d/vmadd-heartbeat start
/etc/init.d/vmadd-timesync start
/etc/init.d/vmadd-shutdown start

If you ever need to uninstall, uninstall in this order:
cd /usr/src/vmadd
rpm -e vmadd-heartbeat
rpm -e vmadd-shutdown
rpm -e vmadd-timesync
rpm -e vmadd-kernel-module-RHEL

Disabling NTP

According to the NTP support site, “NTP was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy. No known virtual machine is capable of meeting these requirements.” Their recommendation is to “Run NTP on the base OS of the machine, and then have your various guest OSes take advantage of the good clock that is created on the system.” In other words, don’t run NTP on the guest.

There does not seem to be universal agreement on this, however. Here is an article by Novell on how to configure NTP when running under VMWare ESX. One of the recommendations is to comment out the local clock source in /etc/ntp.conf:

#server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10

Since we have conflicting advice, and nothing definitive for Microsoft Virtual Server, I’ve decided to try the following:

1. In case NTP is ever used down the road, disable the local clock by commenting out the two lines in /etc/ntp.conf as shown above.
2. Disable ntpd for all runlevels as follows:
   chkconfig --list ntpd # show current ntpd runlevels (the default is On for 3, 4, and 5)
   chkconfig --level 0123456 ntpd off # turn off for all runlevels (applies at boot time)
   chkconfig --list ntpd # confirm that change worked
   service ntpd stop # shut down the server now
   service ntpd status # confirm that it stopped

Note:  When you first log on to a PiaF system, you will see a table of services and their statuses. You can re-create this table at any time by typing status at the command line. From now on, NTPD should show as OFFLINE.

At this point you should have a working CentOS installation that can more or less tell time. Check the time by typing date at the command prompt and comparing it to the clock on your host system. I’m still seeing some minor clock drift on my installation; I’ll have to monitor this for a while to determine whether it becomes a problem.

This is a good point at which to shut down the guest (shutdown -h now) and merge any Undo Disk changes. You’re welcome to make a copy of the virtual machine at this point, but you might want to wait a few minutes. We’ll start the next section,  Setting Up PBX in a Flash, Part 2:  Before Running FreePBX, with a couple simple steps to update PiaF. Once that “generic” part of our setup is complete, we’ll make a copy of the virtual machine before continuing with the installation-specific setup.