Back in November 2008, I blogged  about Setting Up PBX in a Flash, Part 1: Configuring a Virtual Machine. That article discussed running PBX in a Flash (PiaF) under Microsoft Virtual PC or Virtual Server 2005. Within a couple weeks, I abandoned the virtual machine approach and published Moving PBX in a Flash from a Virtual to a Physical Machine.

Recently I’ve been testing Microsoft’s free bare-metal hypervisor, Microsoft Hyper-V Server 2008 R2. Especially now that Hyper-V supports Red Hat Enterprise Linux, could Hyper-V be the answer to virtualizing PiaF on a Microsoft platform?

Let’s try it and see what happens.

Setting Up CentOS under Hyper-V Server 2008 R2

I’ll assume you have Hyper-V Server installed and running. Follow these steps to set up CentOS under Hyper-V Server using the PiaF installation ISO:

1. Set up a new virtual machine with 512MB of memory, a 10GB virtual IDE hard disk (for servers, I’m told that fixed disks perform better than dynamic disks), and a connected legacy network card. Note that the fixed disk and legacy card cannot be configured through the New Virtual Machine Wizard; you’ll have to modify the settings once the virtual machine exists.
2. Download the PBX in a Flash 1.4 ISO file. Attach the ISO to the virtual machine’s DVD-ROM drive (Media > DVD Drive) and start the machine.
3. At the first install prompt, type ksalt. This starts the install in text mode, which avoids an issue with 24-bit video. It also does not use the LVM file system, which apparently adds unnecessary overhead (according to jroper). I deliberately chose a very simple root password during the setup so I could take this virtual machine to client sites one day if needed.
4. Toward the end of the installation, you’ll see that CentOS is “Installing bootloader,” then the install reboots. As soon as the reboot starts, quickly select Media > DVD Drive > Eject. If you miss this, the install will start over.
5. After the boot completes, you’ll be prompted about which PiaF “payload” to use. Choose “A” for the latest stable release via the Internet. The download goes quickly but it does take awhile to compile Zaptel, Asterisk, etc. Once that completes, the system reboots again.
6. When this reboot completes (you’ll see the “pbx login” prompt), shut down the machine by sending a Ctrl-Alt-Del from Hyper-V Server and stopping the machine when it is at the BIOS startup stage. Use Hyper-V Manager to Export a copy of your virtual machine at this point, so you can come back to this point if you encounter problems with the steps below. (A snapshot would also work but an export can be imported at another location.) I’m testing the free 7-Zip for creating a compressed copy of the exported virtual machine. (To install the 64-bit version of 7-Zip 4.65 under Hyper-V Server 2008 R2, run the installer on the server by typing msiexec /i 7z465-x64.msi  at a command prompt.) Name the compressed file something like “PiaF.1.Immediately after CentOS Install.zip”.
7. Start the virtual machine and log on to CentOS. Make sure that CentOS is connected to the network. At a command prompt:
   /sbin/ifconfig # check whether an IP address has been assigned; if so, skip the rest of this step
   system-config-network # if you need to check the network setup
   nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 # view network setup in text form
   service network restart # restart with updated network setup if you made changes
   /sbin/ifconfig # check that an IP address has now been assigned
8. Add the kernel options. I tried using VMWare’s recommendations, since they have specific advice on CentOS 5.2. The CentOS bug report seems to confirm that the kernel options divider=10 clocksource=acpi_pm should take care of the clock cycle issue without the need for a “vm” kernel (compiled with a 100Hz clock instead of 1000Hz). However, after making that change, I was getting lots of messages reading, “rtc lost some interrupts at 1024hz”. So I’m going back to my original procedure posted in November 2008:
   nano -w /boot/grub/menu.lst 
   At the end of the “kernel” line, add this string:
   clocksource=pit nosmp noapic nolapic
9. Update the kernel with the CentOS Virtual Machine kernel. This is outlined on the PiaF site, but do not recompile Zaptel as instructed there. The “vm” kernels are available here. Note:  choose the version that matches your current kernel. Here are the download and install commands:
   cd /root
uname -r # Check your Linux version. Modify the next two lines to match.
   wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-2.6.18-92.1.22.el5.i686.rpm
wget http://dev.centos.org/~tru/kernel-vm/5/RPMS/i386/kernel-vm-devel-2.6.18-92.1.22.el5.i686.rpm
rpm -ivh kernel-vm*.rpm
10. Edit grub to make the new kernel load by default.
    nano -w /boot/grub/menu.lst
    In this file, set default=0 if the .e15vm kernel is listed first. After saving the file, reboot.
    shutdown -r now

11. After the reboot, check the Linux version to make sure it ends in “vm”:
    uname -r

12. You can confirm that the clocksource is set to pit using the method suggested in this thread:
    cat /sys/devices/system/clocksource/clocksource0/current_clocksource

After these changes, I no longer have any time errors appearing on the virtual machine console, even when left on overnight.

Installing Microsoft Linux Integration Components for Windows Server 2008 Hyper-V R2

You’ll need to download and install the Linux Integration Components for Windows Server 2008 Hyper-V R2. There are some significant differences here from the old VM Additions for Linux that worked with Virtual Server. Two notable changes are that the Integration Components for Hyper-V no longer include time synchronization or shutdown integration. However, we should benefit from the enhanced network and disk drivers.

Update 9/30/2010:  This article was written using V2 of the Linux Integration Components. See the comments for links to later version(s).

One “gotcha” right out of the gate:  to extract the ISO needed to install the Integration Components, you must run the downloaded Linux_IC_v2.EXE file. In spite of its claim to be a Win32 Cabinet Self-Extractor, when I tried to run it on my 32-bit XP machine, it tells me that is is not “not a valid Win32 application.” Windows 7 (32-bit) gives a similar error. Once I copied the file to my 64-bit Hyper-V Server 2008 R2 machine, I was able to extract the ISO.

The package includes a helpful readme in PDF format. Since CentOS is based on Red Hat, follow the section titled, “Installing the Linux Integration Components Version 2 on Red Hat Enterprise Linux 5.” Here’s the trimmed-down version:

1. Normally you would need to run yum update kernel, reboot, then run yum install kernel-devel gcc to get the kernel source. However, it seems that the PiaF already includes the kernel source, so this step is not needed with PiaF.
2. From Hyper-V Manager, start your PiaF virtual machine. Then select Media > DVD Drive > Insert Disk and load the extracted LinuxIC_v2.ISO file.
3. Log on to the virtual machine. Mount the drive and copy the files to a new directory as follows:
   mkdir /media/cdrom
mount /dev/cdrom /media/cdrom
mkdir /usr/src/linux_ic_v2
cp /media/cdrom/* /usr/src/linux_ic_v2 -R # -R to recursively copy directories
   umount /dev/cdrom
   You can eject the ISO image from the virtual machine at this point.
4. Install the synthetic drivers. No reboot is required here.
   cd /root  # install won’t run if you are in the source folder, so make sure we’re at root folder
   /usr/src/linux_ic_v2/setup.pl drivers

5. Verify that the installation succeeded.
   /sbin/lsmod | grep vsc
   The output should look something like this:
   netvsc                 45616  0
storvsc                37028 0
vmbus                  56520  2 netvsc,storvsc
scsi_mod              134605  3 storvsc,libata,sd_mod
6. Configure the synthetic network card (seth0).
   Note:  do this from the Hyper-V Manager connection directly, not through a remote connection like Putty.
   So far you probably have one Legacy Network Adapter configured for the VM. From Hyper-V Manager, set that adapter to “Not connected”, then set a non-legacy network adapter to an active connection. (You can do this while the VM is online, but you’ll have to wait until shutdown to actually remove the legacy adapter.) Finally, make the corresponding changes inside the virtual machine.
   nano -w /etc/sysconfig/network-scripts/ifcfg-eth0 #
   Disable the legacy card:  set
   ONBOOT=no
After saving the file:
   service network restart # restart with updated network setup
   /sbin/ifconfig # eth0 should be gone; check that an IP address has now been assigned to seth0
7. Check the Fastpath boot setup. When you run the following command, you should see that hda=noprobe and hdb=noprobe have been added to your kernel boot options. Copy these options to the non-vm kernel line in case you ever need to boot from it:
   nano -w /boot/grub/menu.lst 

As mentioned, the Linux Integration Components don’t support shutdown from the Hyper-V host. However Hyper-V is able to save a PiaF guest within a few seconds. I plan to try that as my Automatic Stop Action in Hyper-V Settings. Restarting a saved machine may cause problems until the clock is resynchronized (see NTP below), but it seems preferable to just “pulling the plug” if, for example, the UPS needs to shut down the host due to a power outage.

Tweaking NTP

With Virtual Server 2005, I disabled NTP and relied on the time to sync to the host. The Hyper-V extensions do not include time synchronization, so follow the instructions near the bottom of this VMWare document for doing so:

Since we have conflicting advice, and nothing definitive for Microsoft Virtual Server, I’ve decided to try the following:

1. Open the NTP configuration file:
   nano -w /etc/ntp.conf
2. Add the following as the first line in the file (“instructs NTP not to give up if it sees a large jump in time”):
   tinker panic 0
3. Comment out these two lines (prevent fallback to the “undisciplined local clock”):
   #server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10
4. Restart the NTP daemon:
   service ntpd restart

At this point you should have a working CentOS installation under Hyper-V that can more or less tell time. Check the time by typing date at the command prompt and comparing it to the clock on your host system.

On startup I do see some NTP messages reading, “ntpd[1970]: frequency error 512 PPM exceeds tolerance 500 PPM,” but this does not seem to keep NTP from syncing the time. I’ll have to check this occasionally to make sure the clock isn’t drifting too much. This command will show the last 100 ntpd messages:

tail -100 /var/log/messages | grep ntpd --color

Moving from Physical to Virtual

Most of the remaining setup steps have been covered in other posts, but there’s a huge shortcut if you are just moving from one PiaF machine to another. Here are the steps you’ll want to follow.

Finishing the CentOS Configuration

Follow the steps in Setting Up PBX in a Flash, Part 2:  Before Running FreePBX to complete the CentOS configuration. A few tips to keep in mind:

• You may want to temporarily give the new machine its own IP address in your router so you can have both the old and new machines online during the steps below. There should be no issue giving the new machine the same host name as the old machine.
• You can use the temporary IP address to access the new machine, so you don’t need to set up a special name in your DNS server.
• If you want to save your work as you go, shut down the virtual machine and user Hyper-V Manager to either take a snapshot and/or export the virtual machine.

Copying the tftp Setup

If you’re using tftp, review Setting Up a Polycom IP 430 Phone with FreePBX for details on the tftp setup. Since my system is not open to the network, I allow writing to the tftpboot folder. Follow these steps to copy the tftp setup:

1. Open access to tftpboot:
   chmod 777 /tftpboot
2. Use WinSCP to copy the contents of the tftp configuration file, /etc/xinetd.d/tftp, from your physical your Windows desktop, then paste it into the same file on the virtual machine.
3. Use WinSCP to copy all the files in /tftpboot from the physical machine to the virtual machine.

Updating the FreePBX Configurations

The shortcut will be to copy the configuration from the old, physical machine to the new, virtual machine. To make sure this goes smoothly, let’s first get the base configuration of the two machines to match.

Update the Physical Machine

On the old, physical machine, update all installed modules to the latest versions:

1. In a browser, open the IP address or DNS name of the old PBX.
2. From the PBX in a Flash menu, in the lower left corner, click on Admin. This adds a few buttons to the screen.
3. Click on the FreePBX Administration button. Type maint as the user name. Use the password that you assigned using passwd-master near the end of part 2. This loads the main FreePBX administration screen, which will be our starting point for the rest of the tasks this article. You might want to bookmark this page so you can return here directly without going through the PiaF button top-level screen.
4. The left column has two tabs, Setup and Tools. From the Setup tab, click on Module Admin. This shows the list of currently-installed modules.
5. At the top of the Module Administration page, click on the link Check for Updates Online. Now you will see all modules that are available.
6. Click on the Upgrade All link (not Download All). This simply selects all upgradable modules; it doesn’t start the download and upgrade process.
7. Scroll to the bottom of the page. Click on Process. Scroll to the bottom of that page. Click on Confirm. An orange popup window appears where you can monitor the progress of the installations. After a few minutes, when you scroll to the bottom of the orange popup, you’ll see a Return link. Click on Return.

Update the Virtual Machine

Log on to the new, virtual machine, and follow the steps in the first section of the Step 3 article, “Updating and Adding FreePBX modules.” Install the same modules on the new machine as are installed on the old machine.

Tip:  I found it simplest to install all the modules on the new machine, then go back and uninstall the ones that are not installed on the old machine.

Now that both machines have the same FreePBX modules installed, we are ready to copy the configuration.

Copy the Configuration from Old to New

Follow these steps to use the FreePBX Backup & Restore utility to copy your entire configuration from the old to the new machine:

1. On the old machine, from the FreePBX administration screen, go to the Tools tab, Backup & Restore. Add a Backup Schedule named OnDemand. Back up all sections (Voice Mail, System Recordings, System Configuration, CDR, and Oeprator Panel). Set the Run Backup time to Now and click on Submit Changes.
2. Using WinSCP, copy the new backup file from /var/lib/asterisk/backups/OnDemand to your Windows desktop. The file will have a timestamp as its name and end in .tar.gz.
   Note:  Do not rename the backup file! I thought I would be clever and give it a more meaningful name. This caused the restore to fail, probably because the restore expects the .tar file inside the .tar.gz file to have the same name as the .tar.gz file.
3. On the new machine, from the FreePBX administration screen, also create a backup called OnDemand. This is mostly done so FreePBX will create the OnDemand directory and will know to look for backups in that directory.
4. Use WinSCP to copy the old machine’s backup file from your Windows desktop to /var/lib/asterisk/backups/OnDemand on the new machine.
5. On the new machine, from the FreePBX administration screen, , go to the Tools tab, Backup & Restore. Click on Restore from Backup and navigate down to the OnDemand directory and to the old machine’s backup file (make sure the time stamp matches the backup from the old machine.) Choose Restore Entire Backup Set, and confirm that you want to overwrite all FreePBX and Asterisk files.
6. When the restore completes, click on the orange Apply Configuration Changes. Your new system should now match your old system exactly! In the Setup tab, check some known values (Extensions, Trunks, Inbound Routes) to be sure.
7. For some reason, the configuration backup/restore does not include the /etc/amportal.conf file. If you’ve customized this file (e.g. to specify the AMPWEBADDRESS or to send backups to an FTP server), copy the file from the old machine to the new machine using WinSCP.

Cut Over from Physical to Virtual

If you manage your IP addresses through your router, this last bit is easy:

1. Shut down both your old, physical machine and your new, virtual machine.
2. Optional:  in Hyper-V Manager, Export the fully-configured virtual machine. Back up this export so you can return to this state later.
3. In your router, in the list that maps MAC addresses to IP addresses, note down the IP address of the physical machine and then delete it from the (or assign it an IP that is not in use).
4. Also in the router’s address map, update the MAC address of the virtual machine to map to the IP address that had heretofore been used by the physical machine.
5. Use Hyper-V Manager to start the new, virtual machine.
6. After the virtual machine is running, reboot or cycle power on your other telephony devices (SPAs, IP phones) to force them to register with the new machine. Note that as far as these devices are concerned, the PBX server hasn’t moved, since it has the same IP address as the old, physical machine.

Now you are finally ready to make some test calls and see how they sound! In my initial tests, I hear no echo or jittery “cell phone” audio and can detect no difference from a physical machine. Maybe Hyper-V finally allows running Asterisk in a Window virtual environment!