10 Tips to Optimize Exchange 2003 Performance (Part

Introduction

Remember performance optimizer from Exchange 5.5 days? At the end of every Exchange installation you were prompted to run the optimizer that would configure some settings, based on the hardware, workload and specific use, contributing to the overall stability and performance of the server.
The 15 minutes of fame of this fine tool ended with the release of Exchange 2000, since that version was supposed to be self-tuning. But soon people discovered that there was still room for improvements, because tuning is such a complex process, so it was only natural that tech gurus and Microsoft itself started doing some recommendations.
When Exchange 2003 saw the daylight, some old Exchange 2000 recommendations turned obsolete, but new ones replaced them and some were adapted to cope with the then new Windows Server 2003.
There are some obvious things that you can do to improve performance, such as buying new processors (and use hyper-threading), adding more memory, and migrating to a faster storage system. You won’t see hardware related tips on the list I’m about to give you, but it’s always a best practice to monitor server performance with Windows Performance Monitor, in order to detect physical bottlenecks.
The following recommendations apply to Exchange Server 2003 installed on Windows Server 2003, because probably that’s the most common scenario nowadays. And Exchange Server 2003 combined with Windows Server 2003 make a perfect couple. When I started writing these lines, I even thought to recommend the use of Windows 2003 as my first tip, because it adds some enhancements and functionality to an Exchange 2003 infrastructure. You’ll only get the full user experience of Exchange 2003 if you use Windows 2003 (it’s the recommended OS by Microsoft, check http://www.microsoft.com/technet/prodtechnol/exchange/2003/bestconfig.mspx). So, now that it’s said, you can make that tip number 0.
You should also have present that, although these are general recommendations, depending on your Exchange implementation you may have to do some adjustments. That would be the case if you have a large implementation of Outlook Web Access (OWA), Outlook Mobile Access (OMA), ActiveSync or RPC over HTTP.

1.        Document yourself

Let’s face it, a knowledgeable IT staff makes a better Exchange environment. Although I’ll try to give you some clues on how to improve Exchange performance, there’s no way I could cover all possible scenarios of a complex infrastructure as any e-mail system can be.
There’s lots of information about Exchange on the Internet. There’s a whole community willing to help you with your most difficult tasks. I’m never tired of saying what a great job the Microsoft Exchange Team has done on documenting the product. I strongly advise you to visit Exchange Server 2003 Technical Documentation Library (http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/default.mspx) on a regular basis. And you’re always welcome on the Exchange Server Community (http://www.microsoft.com/exchange/community/default.mspx), where you can find lots of technical resources, related external sites, check out the latest webcasts or even discover the many blogs that exist and give you valuable information.

2.        Remove obsolete Exchange 2000 tuning parameters

You document every change you do on your production servers, right? No? Oh, oh… Now it’s the time to remove deprecated Exchange 2000 tuning parameters.
It’s a fact that some Microsoft recommendations change over time, either because they modify the software by launching service packs or new releases, or just because they come to a conclusion that their previous recommendation was wrong (or less right). Some tuning parameters for Exchange 2000 or Windows 2000 are no longer valid for the new 2003 versions, so make sure you undo those modifications.
You can find detailed information about the settings that must be removed, in "Removing Exchange 2000 Tuning Parameters" from the Exchange Server 2003 Deployment Guide (http://go.microsoft.com/fwlink/?LinkId=21768).

3.        Optimize Exchange memory utilization

You know Exchange loves memory. The store.exe process is mainly responsible for this behaviour, since it will grab as much memory as it can possibly get. This doesn’t represent any kind of problem or a memory leak, but actually it’s a normal and expected operation.
What you should also know is that if you have more than 1GB of RAM you can make Exchange’s use of memory more efficient. Yeah, that’s right, the famous /3GB switch. But this switch is not a one-size-fits-all. You should be aware that this setting is not recommended on front-end servers, dedicated bridgeheads or when you have Exchange installed on a Domain Controller (which is, by the way, not recommended).

• Add the switches /3GB and /USERVA=3030 to boot.ini. The /3GB switch modifies the way virtual address space is created so that 3 gigabytes are available for user mode applications;
• Configure the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\HeapDeCommitFreeBlockThreshold registry value to 0x00040000. The HeapDecommitFreeBlockThreshold registry key specifies the number of contiguous bytes above which the memory is decomitted rather than retained for reuse, thus avoiding virtual memory fragmentation.
• Verify that the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\SystemPages registry value is set to 0.
• If you have a server with more than 2 GB of memory, it may help to increase the size of the Store Database Cache (aka ESE buffer).

Figure 1: Windows virtual address space

4.        Implement an effective storage design

Storage design is very important, because disk subsystem bottlenecks cause more performance problems than processor or memory deficiencies. The most common error people do when planning an Exchange server is that they tend to design for capacity and not for performance.
Many of us know that we should use separate disk volumes for the OS, Exchange logs and Exchange store(s), with the following RAID levels:

OS: RAID 1
Logs: RAID 1
Database: RAID 0+1

But the thing most people forget is IOPS (I/O per second), which is the measure of throughput you should use. To implement an effective storage design you must calculate the necessary IOPS for your system.
The theoretical calculations require you should know in advance some numbers, such as user behaviour and disk specifications.
One can assume that an average user requires 0.5 IOPS or, being a little more aggressive, 0.75 IOPS. If you don’t want to estimate anything and you already have a live system, you can measure your true needs. There’s a good document, “Optimizing Storage For Exchange Server 2003” (http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/optimizestorage.mspx) that has detailed instructions on how to do that.
Next, for our calculations, we need to know the average performance of the disks. Typically a 10K RPM disk will do 130 IOPS, and a 15K RPM will achieve 180 IOPS.
Since we’re using RAID, there’s a penalty that depends on the RAID level. Assuming there are 3 reads for every write, the penalty factor for RAID 5 is 0.57 and 0.8 for RAID 1 (or 0+1).
Finally, here’s the formula we’ll use:

Total IOPS = #Disks x IOPS/Disk x RAID Penalty factor

So, for example, if you want to know how many 15K RPM disks you’ll need for 1000 users, assuming 0.5 IOPS/user and RAID 0+1:

0.5 x 1000 = #Disks x 180 x .8 <-> #Disks = 500 /(180 x .8) = 3.47

We must round up the result to the next even number (RAID 0+1 requires an even number of disks), which is 4. That’s how many disks we’ll need.
Although this entire math is true for every Exchange disk, this is particularly critical for database drives, since 90% of the IO on the system goes to the databases and only 10% goes to the logs.
The next thing we can do to improve storage performance is to work on disk geometry. Microsoft provides a tool, Diskpar, which allows aligning the disk tracks with sector tracks. For partitions created by Windows 2000 and Windows Server 2003, the default starting sector for disks that have more than 63 sectors per track is the 64th sector, causing one out of every eight blocks of data written to your disk to span two disk tracks. Diskpar can increase disk performance as much as 20 percent, but you should always consult your hardware vendor before using this tool. Some disk configurations will have no benefit from the tool.
The Diskpar utility can be found in the Windows 2000 Server Resource Kit. With the release of Windows Server 2003 SP1, diskpart will include this functionality, with the new switch /align.
To resume:

• Keep Exchange transaction logs and databases stored on separate disk volumes to provide both data protection and efficiency (separation of sequential writes and random read/write access, respectively);
• Calculate the number of spindles needed to provide the necessary IOPS;
• Use Diskpar if your hardware vendor recommends it;
• If your RAID controller has a mirrored, battery-backed, write-back cache, set the ratio to 100 percent write. Also configure the page size to be 4 KB.
• When you format the hard disks stay away from quick format. Configure the partition to use NTFS and to use an allocation unit size of 4096 (4 KB).

It’s impossible to cover all issues regarding the storage subsystem, so I strongly recommend further reading:
“Optimizing Storage for Exchange Server 2003”,
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/optimizestorage.mspx
“A few basic concepts in disk sizing”,
http://blogs.msdn.com/exchange/archive/2004/10/11/240868.aspx
“Some more thoughts on disk IO and calculations…”,
http://blogs.msdn.com/evand/archive/2004/10/14/242127.aspx

5.        Ensure that you have fast access to AD

When planning your Exchange deployment, it is crucial that you consider your Windows Server network topology, because Exchange requires Active Directory to store configuration settings and also to provide user authentication, permissions management and directory information.
From every kind of AD server, one has a particular relevance: the global catalog. A global catalog is required in each domain that contains Exchange servers. As a rule of thumb, one might say that every mailbox server needs a global catalog by its side, since, as I said before, this particular server is critical for some Exchange services (including log on, group membership, store services) and access to the global address list (GAL).
Consider the following when placing global catalog servers:

• All Exchange servers and users should have fast access to a global catalog server. Address lookups will become much faster if you contact a local global catalog, as opposed to a remote one which, besides increasing network traffic, will also impair the user experience. Verify that the DSAccess list only contains local DC/GC servers.
• There should generally be a 4:1 ratio of Exchange processors to global catalog server processors, assuming the processors are similar models and speeds. However, depending on your situation, higher global catalog server usage, a large Active Directory, or large distribution lists can necessitate more global catalog servers.
• In addition, using multiple domain controllers within domains distributes the lookup traffic and provides redundancy if a domain controller fails.
• Use the/3GB switch on global catalogs. It will increase the JET cache from 512MB to 1GB, so you’ll have more AD objects in memory.

Summary

These are the first 5 tips out of 10 I will cover and that will, hopefully, help you optimize your Exchange environment. You should keep in mind that performance optimization is not only about identifying bottlenecks and taking the appropriate measures to correct them, but it also has a lot to do with proactively preventing configuration problems before they arise.
What I’m trying to say is that, better than applying blindly the tips I’m giving you, you should take the time to understand all the variables that affect your system, including user profile, business needs, architecture and hardware.
Read the next 5 tips at 10 Tips to Optimize Exchange 2003 Performance (Part 2)