What’s the best disk setup on a small business server?
I recently got to play around with some options as I set up a Hyper-V host on a Dell PowerEdge 2900 with a PERC 5/i RAID controller. Here are some things I learned along the way.
RAID 10 Beats RAID 5 – By a Lot
Somewhere I heard that it’s good to keep the Hyper-V host on a separate spindle from the VMs. So I originally set up the machine to put the host OS (Windows Server 2008 R2) on 2 mirrored disks (RAID 1), then use three disks in RAID 5 for the Hyper-V data. I installed SBS 2008 on a fixed 80GB virtual hard disk. It took 3 minutes 5 seconds to boot and 4 minutes 17 seconds to shut down.
Then I read here that, “RAID 1+0 [aka RAID 10] provides better write performance than any other RAID level providing data protection, including RAID 5.” Fast is good! So I dropped the idea of a separate spindle for the host OS (which wasn’t completely separate anyway since it was on the same RAID controller). I set up a RAID 10 using six disks, which means striping across three mirrored volumes. The same SBS 2008 machine booted in 1 minute 43 seconds and shut down in 4 minutes 10 seconds. That means it boots 80% faster! Not sure why the shutdown isn’t faster; maybe it involves more writing to disk, which RAID 10 doesn’t help as much as reading.
16KB Beats 4KB – By a Little
The benefits of adjusting cluster size, also called allocation unit size, are (so far) less impressive.
On my RAID volume, I set up partitions formatted with 4KB and 16KB clusters. I created an 80GB fixed virtual disk on the 16KB host partition, and installed SBS 2008 with a non-standard 16KB cluster size for the client partition (here’s how). That’s what I was testing with above.
I also created a separate SBS 2008 installation on the 4KB host partition, and a matching 4KB cluster size for the client partition. Its boot and shutdown times were just a little slower than the 16KB version.
So in summary we have this graph. The fastest in my test was RAID10, 16KB clusters on the host partition, and 16KB clusters on the client partition:
Cluster Size Considerations
Here a few notes and links on cluster sizes and why you might want to go larger than 4KB in some cases:
- If you want to defragment a volume containing shadow copies, it should be formatted with a 16KB or larger cluster size per MSKB 312067. It’s not clear whether that still applies to Server 2008, but since by default SBS 2008 turns on shadow copies on the system volume, and since 16KB is a little faster to boot and shut down, I decided to go with 16KB.
- Microsoft recommends that Exchange databases be hosted on volumes with a 64BK cluster size. See “Partition Allocation Unit Size” here: http://technet.microsoft.com/en-us/library/bb738145%28EXCHG.80%29.aspx. I doubt that this will matter much for my SBS 2008 installation with only a few mailboxes, but I created a 64KB partition on the host, so I may give it a try.
- Microsoft recommends that SQL databases be hosted on volumes with a 64KB cluster size. See “NTFS Allocation Unit Size” here: http://technet.microsoft.com/en-us/library/cc966412.aspx#EEAA. In addition to SBS 2008, I’ll also be hosting a Server 2003 server to run a legacy SQL 2000 application. I plan to try the 64KB cluster on that.
- Some say you should put the page file on a separate disk and that it should have a 64KB cluster size (no authoritative links here). I could create a separate virtual disk for each virtual machine’s page file, and place that virtual disk on the 64KB host partition. I doubt this would make much difference in my small environment, especially since the page files would still be on the same physical RAID array as their respective system volumes.
Beyond RAID and Cluster Size: Partition Alignment
The other major performance consideration that I came across while researching all of this is the need to align disk partitions to “stripe” boundaries, at least where SQL Server is involved. Fortunately Server 2008 does this automatically for the host partitions. How and whether that applies to client partitions (virtual disks) is not clear—how do you even know if a virtual hard disk starts on a stripe boundary? Some links for further consideration:
Good overview (blog post): Partition Alignment
Authoritative MSDN article: Disk Partition Alignment Best Practices for SQL Server
Enterprise-focused blog post: Disk Partition Alignment (Sector Alignment): Make the Case: Save Hundreds of Thousands of Dollars
3:05 down to 1:43 is about 44% faster, not 80% faster. The initial setup is 80% SLOWER than the new one, but the new one is not 80% faster than the original one.
Can you cite a reference for that approach? How do you calculate 44%? The only references I can find all give the same answer: if the first value was 185 seconds and the second value was 103, then there was an 80% decrease. But nothing defines “faster” and “slower”.
http://en.wikipedia.org/wiki/Percentage_change
http://www.csgnetwork.com/percentchangecalc.html
You take the original value, subtract the second value, to get the delta. In this case, 82.
Then you divide the original value by the delta. 185 / 82 = 1.44
so the change is 44%.
I can’t provide a reference, as I learned this in high school science class and it has served me ever since for HS, 5 years of electrical engineering in college (BS/MS), and throughout my career.
185 / 82 is 2.26. Maybe you meant you divide the delta by the original value: 82 / 185 gives 0.44.
That may well be correct. Still feels odd in terms of common speed comparisons. That means the guy with the fast car who gets there in one hour is only 50% faster than the guy with the slow car who gets there in two hours–even though the first guy would have to drive twice as fast, and the second guy took twice as long.
I know this thread is old, but what stripe size did you use when initializing your RAID array?
Is there a “best” RAID stripe size to use when creating the array with Hyper-V in mind for running SBS 2011 Standard? I have the following stripe sizes from which to choose: 8K-16K-32K-64K-128K-256K on my HP ML350 G6 server with RAID 1 SAS 15K 450GB drives.
BTW, I think TJ has it wrong. Using his method, decreasing from 100 to 75 would mean 100-75=25, then 100/25=4, whic makes no sense, because we know that 100 minus 25% equals 75. Your final statement is correct: divide 25 (delta) by 100 (original amount) = .25, or 25%.
Thank you!
Gregg Hill
Gregg – I used a 64K stripe size. Don’t recall if researched that at the time, but it kinda makes sense since I wanted to be able to use 64K cluster size on some volumes. In my understanding, that is then the minimum amount of data that is read at once. I don’t want the RAID controller to have to issue multiple reads to pull in 64K. But I also don’t want it reading more and throwing it away. With this setup, I’m running Server 2008 R2 as the Hyper-V host with SBS2008, Windows Server 2003 (for SQL 2000), and Windows 7 32-bit (PBX) as clients. That all fits in a total of 8GB of RAM, and it’s been pretty stable for nine or ten months. It is only a one-user office (how did my life get so complicated?), but I do think the RAID 10 helps.
There is an minor issues with Exchange 2007 when run on a Domain Controller (SBS 2008) that causes SBS 2008 shutodwn times to be long. I don’t think MS ever addressed this. But here is an article and script that speeds this up: http://blog.mpecsinc.ca/2010/01/sbs-2008-speed-up-that-reboot-script.html
Thanks Brian. I’ve also SBS 2008 blogged about slow startup and shutdown:
https://www.mcbsys.com/blog/2010/07/sbs-2008-slow-startup-and-shutdown/
Hi Mark. This was a great blog you wrote up. I am wondering when you setup this configuration, did you configure the Raid Controllers Virtual disk to have a 64K cluster size, and then when you created the partition on the HOST/Hyper-V did you also set this to 64K, along with the guest partition to be 64K cluster size? I have read a few other sites where they had configured the cluster sizes different from the controller to the host to the VM. It seems to be a bit conflicting to me to have different cluster sizes on the same raid 10 array, and I would think performance would go down. Any insight you have on this?
Lyle Epstein
Lyle,
– I think what you call RAID “cluster size” is what Dell calls “stripe size.” That is set to the default 64KB.
– The Win 2008 R2 Hyper-V host has three partitions:
— OS: 50GB, 4KB cluster size (the default when installing Windows)
— Data16: 500GB, 16KB cluster size
— Data64: 146GB, 64KB cluster size
– The SBS 2008 guest has OS and Data VHDs on the host’s Data16 partition. Both are configured as 16KB inside the guest.
– The Win 2003 SQL Server guest has its 4KB OS partition on the host’s Data16 partition, and its 64KB Data partition on the host’s Data64 partition.
So: for the SQL data, it’s 64KB at all three levels. For SBS, the first time it reads a file, it only requests 16KB, but the host VM will read 64KB or one RAID stripe. So if the next SBS reads are sequential in the same file, they should be in the host or RAID cache. However, I’m still able to store smaller files without the waste that I would have if I used 64KB in the guest.
My sense is that 16KB in the guest is a good balance of performance and not wasting too much space. Use 64KB in the guest for special applications like SQL or Exchange.
This is by far the best post on this subject. Mark, seriously, you rock! Thank you so much. You really nailed it down and have shed so much light and understanding. You truly “live-it” as I like to call folks that really care and pass the knowledge. :-)