Exchange 2013 Messsage Tracking Tool - There are 2 best tools

1. As usual the PowerShell, using GridView
2. CodePlex - ExchangeReports -https://www.codeplex.com/site/search?query=exchangereports&ac=2

Exchange 2013 lacks the traditional Exchange Management Console that you are probably familiar with, and with it went the message tracking console that many administrators relied on to troubleshoot mail flow problems.  Sure there is now the Delivery Reports tool in the web Exchange Control Panel, but it’s just not quite the same, especially since it limits you to searching within a specific mailbox.

Luckily there is something similar to the Message Tracking Center in Powershell.

1	Get-MessageTrackingLog -ResultSize Unlimited -Start "January 1 2014" | Out-GridView

Run this from a Powershell prompt and this will launch a Powershell Gridview where you can quickly and easily filter through the data, including the stuff not shown in the Web ECP.

Exchange 2013 SP1 Sizing like RAM, HDD etc***

Reference:
http://blogs.technet.com/b/exchange/archive/2014/04/03/ask-the-perf-guy-sizing-guidance-updates-for-exchange-2013-sp1.aspx

With the release of Exchange 2013 SP1, it’s time to revise our sizing guidance given feedback from customers, observations from our own large-scale deployments, and requirements associated with new and changed components in SP1. In addition to this brief article, I’ve also updated the original blog post with updated formulas and tables to reflect the changes described here.
There are two specific changes that need to be highlighted:

CAS Processor Sizing

With the introduction of the MAPI/HTTP protocol, our existing sizing guidance for CAS processor utilization needs to be changed. Usage of MAPI/HTTP has a fairly dramatic increase in rate of requests handled by the CAS role, when compared to requests generated by clients using RPC/HTTP. As each connection has a measurable amount of processing overhead, this results in an overall increase to our CPU requirements on CAS, moving from a 1:4 ratio of CAS to Mailbox processor cores, to a 3:8 ratio (a 50% increase). It’s important to call out that MAPI/HTTP is disabled by default, as we expect that customers will want to carefully evaluate the deployment requirements and impact of MAPI/HTTP before enabling it. Because it is disabled by default, existing Exchange 2013 deployments do not need to immediately add more CPU resources at the CAS layer. Instead, we expect that additional capacity will be considered as part of the evaluation and deployment process for MAPI/HTTP. We do anticipate that over time it will become the standard method of connectivity for Outlook clients so it’s important to include these requirements in our sizing guidance as early as possible.
It’s also critical to deploy .NET Framework 4.5.1 if you intend to use MAPI/HTTP, as it contains an important fix that impacts the performance and scalability of MAPI/HTTP at the CAS layer.
Ross has updated the Exchange Server 2013 Server Role Requirements Calculator to take into account this guidance change in version 6.3.

Pagefile Sizing

As memory requirements have increased for Exchange, our historical guidance for sizing the pagefile has become more and more challenging from a deployment perspective. Previously, our guidance was to set a fixed pagefile size equal to the size of RAM + 10MB. On servers that are commonly deployed with 128GB of RAM or more, requiring a pagefile sized to RAM+10MB results in a large amount of space consumed (typically on the system drive) with questionable benefit. In our large-scale internal deployments, we have been running with a cap on pagefile size for quite some time, and as a result we are comfortable recommending that all of our on-premises customers follow that same guidance. Moving forward with Exchange 2013, we recommend a fixed pagefile of the smaller of RAM size + 10MB or 32,778 MB (which is 32GB + 10MB). Hopefully this will free up some needed space on your servers.
As we continue to learn more from customer feedback and monitoring of our own deployments, we will keep updating this guidance via posts to the blog.
Jeff Mealiffe
Principal Program Manager Lead
Exchange Customer Experience

Exchange 2013 back pressure event 15004 issue (Mail Stuck in Outbox or Drafts – Have to restart Transport Service to resume mail flow)

Reference:
http://danblee.com/exchange-2013-mail-stuck-in-outbox-or-drafts-have-to-restart-transport-service-to-resume-mail-flow/

Nice Article BANBLEE!!!

Exchange 2013 Mail Stuck in Outbox or Drafts – Have to restart Transport Service to resume mail flow

November 20, 2014 by Dan B. Lee Leave a Comment

Oh man, this was a tough one. A real doozy. First, here are my environment details where this was happening.

• MS Exchange 2013 CU6 (Issue was happening as far back as CU4)
• One Exchange Server managing all Exchange services and responsibilities
• Less than 50 mailboxes

Overview

Basically, every few days we’d get a call in the middle of the day from customers telling us that mail was being stuck in the Outbox in Outlook and mail would be put into the drafts folder in OWA. At first, we found that rebooting the server fixed the issue. Later, we learned that restarting the Topology Service (The service that will restart all of your Exchange Services) fixed the issue. Finally, we narrowed it down the restarting just the Transport Service. Once the Transport Service restarted, all mail would push from the Outboxes and mail would work again for a few days.

There were no logged errors on the Transport Service. The Transport Service was always On, never stopped.

Resolution

Adjust or disable Resource Pressure Monitor.

What is the Resource Pressure Monitor?

In perfect form, the resource pressure monitor collects information about your environment and will delay mail (called “tarpitting”) until things get back to normal. If the severity of resource usage gets too high, it will actually STOP mail flow of all kinds. This is what was happening to us. If your system is unhappy with the amount of resources available, you’ll have to restart the Transport Service to reduce the monitor’s severity level.

It wasn’t until we found this warning that we noticed that this was going on:

For my Exchange environment, Bucket Versions were getting to too high a level. So look at all the things that it stopped:

The resource pressure increased from Medium to High.

The following resources are under pressure:
Version buckets = 332 [High] [Normal=80 Medium=120 High=200]

The following components are disabled due to back pressure:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from Pickup directory
Mail submission from Replay directory
Mail submission from Mailbox server
Mail delivery to remote domains
Content aggregation
Mail resubmission from the Message Resubmission component.
Mail resubmission from the Shadow Redundancy Component

The following resources are in normal state:
Queue database and disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que”) = 71% [Normal] [Normal=95% Medium=97% High=99%]
Queue database logging disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\”) = 74% [Normal] [Normal=95% Medium=97% High=99%]
Private bytes = 3% [Normal] [Normal=71% Medium=73% High=75%]
Physical memory load = 61% [limit is 94% to start dehydrating messages.]
Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
Temporary Storage disk space (“C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp”) = 74% [Normal] [Normal=95% Medium=97% High=99%]

How to Disable the Back Pressure Resource Monitor

Luckily, there’s a handy config file that can be adjusted:

1. Open the config file for the executable here: %ExchangeInstallPath%Bin\EdgeTransport.exe.config
2. Find the entry called “EnableResourceMonitoring” and change the value to “false”
3. Restart the Transport Service during your normal maintenance window.\
4. Breathe

A small disclaimer here: You may want to actually look into why your Exchange environment thinks it’s having trouble with resources. An easier fix to all of this may be to give your machine more disk space or RAM, etc. Here’s a screenshot of the config file entry:

Some things that did NOT work

Here are some of the thing that I tried before finding this issue:

1. Disabling Malware Detection on the connector
2. Adding the servername and the domain controller name to the hosts file
3. Updating to CU6 entirely

Please let me know if there are other things that you have tried that aren’t working. Also, please let me know if this fixes the issue for you. The error you get will tell you exactly what needs to be adjusted, so as I said earlier, you may want to fix the issue instead of just disabling this feature. My environment is small, so this service is not needed.

Microsoft’s Explanation:

All from here: http://technet.microsoft.com/en-us/library/bb201658%28v=exchg.150%29.aspx

Back pressure is a system resource monitoring feature of the Microsoft Exchange Transport service that exists on Microsoft Exchange 2013 Mailbox servers and Edge Transport servers.

Exchange can detect when vital resources, such as available hard drive space and memory, are under pressure, and take action in an attempt to prevent service unavailability. Back pressure prevents the system resources from being completely overwhelmed, and the Exchange server tries to process the existing messages before accepting any new messages. When utilization of the system resource returns to a normal level, the Exchange server gradually resumes normal operation and starts accepting new messages again.

In Exchange 2013, when the Transport service on a Mailbox server or an Edge Transport server is under resource pressure, incoming connections are accepted, but incoming messages over those connections are either accepted at a slower rate or are rejected. When an SMTP host attempts to connect to an Exchange server that’s under resurce pressure, the connection will succeed. However, when the host issues the MAIL FROM command to submit a message, depending on the resource that’s under pressure, the Transport service either delays the acknowledgement of the MAIL FROM command or rejects the connection.

For my issue, the Bucket Version was building up:

A list of changes that are made to the message queue database is kept in memory until those changes can be committed to a transaction log. Then the list is committed to the message queue database itself. These outstanding message queue database transactions that are kept in memory are known as version buckets. The number of version buckets may increase to unacceptably high levels because of an unexpectedly high volume of incoming messages, spam attacks, problems with the message queue database integrity, or hard drive performance.

When Exchange starts receiving messages, these messages are grouped together in batches and then prepared as version buckets. If an incoming message has a large attachment, it can be separated into multiple batches. These batches that are being processed are known as batch points. The number of outstanding batch points can exceed the set thresholds, especially when there’s an unexpectedly high volume of incoming messages with large attachments.

When version buckets or batch points are under pressure, the Exchange server will start throttling incoming connections by delaying acknowledgement to incoming messages. Exchange will reduce the rate of inbound message flow by tarpitting, which introduces a delay to the MAIL FROM commands. If the resource pressure condition continues, Exchange will gradually increase the tarpitting delay. After the resource utilization returns to normal, Exchange will gradually start reducing the acknowledgement delay and ease into normal operation. By default, Exchange will start delaying message acknowledgements 10 seconds when under resource pressure. If the resources continue to be under pressure, the delay is increased in 5-second increments up to 55 seconds.

Exchange keeps a history of version bucket and batch point resource utilization. If the resource utilization doesn’t go down to normal level for a specific number of polling intervals, known as the history depth, Exchange will stop the tarpitting delay and start rejecting incoming messages until the resource utilization goes back to normal. By default, the history depths for version buckets and batch points are in 10 and 300 polling intervals respectively.

Mail will STOP flowing when the severity reaches high. No matter what the issue, the mail flowing stopping seems to be the same:

• Reject incoming messages from other Exchange servers
• Reject message submissions from mailbox databases by the Mailbox Transport Submission service on Mailbox servers
• Reject incoming messages from non-Exchange servers
• Reject message submissions from Pickup and Replay directories

So please, please let me know if you have experienced the same trauma. Hopefully Exchange will address this or at least make it more apparent to end users in the future.

Cheers!

- See more at: http://danblee.com/exchange-2013-mail-stuck-in-outbox-or-drafts-have-to-restart-transport-service-to-resume-mail-flow/#sthash.rn1pfG7i.dpuf

Outlook Web Access (OWA) and ActiveSync reporting using IIS logs

From: http://myriadofthings.com/outlook-web-access-owa-and-activesync-reporting-using-iis-logs/

http://www.msexchange.org/articles-tutorials/exchange-server-2003/tools/Using-Logparser-Utility-Analyze-ExchangeIIS-Logs.html

I was asked to report on how many people were still accessing a legacy Exchange server via Outlook Web Access for the purposes of retiring it permanently.  Here is a step by step walk through. Pasted commands may not work.  Please type directly into the cmd window!

1. Locate your IIS logs on your exchange server.  Mine were stored in C:\inetpub\logs\LogFiles\W3SVC1.  For more help refer to this article.
2. For this example we will be copying the logs we need locally to the C:\log directory.  This method could easily be adapted to use UNC paths but was not needed for my purposes.  IIS should create one log per day so copy the number of days you would like to report on to c:\log on your local machine.
3. Download Log Parser 2.2 from the Microsoft website and install it to the default directory.
4. Next we will use log parser to combine all of these logs into a single file.  Create a subdirectory under c:\log called mergedlog.  From the command line navigate to the log parser directory “C:\Program Files (x86)\Log Parser 2.2” and run the following command “logparser.exe -i:iisw3c “select * into c:\log\mergedlog\merge.log from c:\log\*” -o:csv”  This will create a single log file named merge.log and convert the data from iisw3c to csv format.
5. Next we will need to run a command that will pull the information we are looking for out of the log.  Here are three examples that list User Name, Date, Time, IP, page accessed, and user agent. Each will output the results into a file named output.csv in the c:\log directory.

The first command looks for OWA access

LogParser -i:csv “SELECT cs-username, date, time, c-ip, cs-uri-stem, cs(User-Agent) FROM C:\log\mergedlog\merge.log TO C:\log\Output.csv WHERE cs-method LIKE ‘%get%’ and cs-uri-stem LIKE ‘%owa%’

This next command lists ActiveSync users

LogParser -i:csv “SELECT cs-username, date, time, c-ip, cs-uri-stem, cs(User-Agent) FROM C:\log\mergedlog\merge.log TO C:\log\Output.csv WHERE cs-method LIKE ‘%post%’ and cs-uri-stem LIKE ‘%Microsoft-Server-ActiveSync%’

Finally as a Bonus This one looks for Mac Office Users

LogParser -i:csv “SELECT cs-username, date, time, c-ip, cs-uri-stem, cs(User-Agent) FROM C:\log\mergedlog\merge.log TO C:\log\Output.csv WHERE cs-method LIKE ‘%post%’ and cs(user-agent) LIKE ’%Macoutlook%’

There is a good book you can get on Amazon called Microsoft Log Parser Toolkit that has a goldmine of knowledge on how to use this tool.

Why exchange 2013 uses port 444 for communication?

Exchange 2013 Clients not able to connect using Outlook Anywhere

Mikhael Ortiz 

4 Aug 2014 11:37 PM 

• 4

I would like to share an issue occurring frequently with Exchange 2013 deployments. The behavior being reported by users is their Outlook clients cannot connect to any of the Exchange Web Services (OWA, EWS).  Prior to exhibiting the behavior, there were no reported configuration changes made to the web services by the Exchange Admins, so what changed? The first place that you need to look at is that Exchange back end web site certificate bindings on port 444.  Normally, if you check the IIS logs and the HTTP proxy logs you can see that you get Status code 500 when the connection proxy to the Exchange 2013 back end website. Another symptom that you see is the Exchange PowerShell console won't open a session to the server. The reason that this issue happens is due to the way connection flow happens in Exchange 2013. Below you can see the data flow when outlook client connects to exchange 2013 mailbox, and the explanation of the connection. I will cover the normal flow first and then we’ll go over why this breaks when missing or having a third party certificate bound to 444 on the Exchange Back End Web Site.

Exchange 2013 Mailbox

1. Outlook >>>>> 2013 CAS DWS:443
2. 2013 CAS >>>> 2013 MBX:444
3. 2013 CAS <<<< 2013 MBX:444
4. Outlook <<<<< 2013 CAS DWS:443  

Step by Step Connection Flow 

• IIS authenticates the request and Microsoft.Exchange.FrontEndHttpProxy.dll running in the MSExchangeRpcProxyAppPool.  HTTP Proxy determines the Mailbox Anchor, accessing Active Directory to look up the user, then maps to the user object that contains the mailbox GUID.  As soon as we obtain the GUID, HTTP Proxy refers to Active Manager to locate the database where the mailbox resides.  At this point, HTTP Proxy then builds the URL to the mailbox server that contains the user's mailbox GUID@Domain.com.
• HTTP Proxy takes the authenticated request and creates a serialized blob (that contains SID, user groups, and more. Then a new header is attached that contains the Client Access server name, in order to use CSC (serialized access token authentication, also known as server to server authentication) and sends this new http request URL to the Mailbox server over SSL on port 444.  The new URL contains the Mailbox server name so that the Mailbox server knows the request belongs to this server. 
• The request is sent to the Mailbox server's RPC Virtual Directory that is running in the MSExchangeRpcProxyAppPool by using by default, integrated authentication.
• RPCProxy.dll on the Mailbox server is responsible for extracting the RPC packet from the tunnel and forwarding it to the local RPC Client Access Service on Port 6001.  

Now that we went over the connection flow when a mailbox is located on an Exchange 2013 mailbox server, we can cover why this breaks and why is so important of not changing or having a third party certificate bound to port 444 on the Exchange Back End Web Site. When the client connects to the CAS, and HTTP Proxy determines where the mailbox is located, it builds an URL to the destination mailbox server on port 444. The URL is built using the hostname. Once we have the URL we try to connect on secure port 444 using SSL. In the process of the SSL handshake there is several validation task that have to take place before the connection is allowed. If the name being used on the URL is not in the certificate the request will fail and your client can’t connect.
But I see 200 status codes in the IIS logs…
The reason you ‘ll see the 200 status codes on IIS is due to the initial communication that is occurring on port 443 and on the Default website. Where the process breaks is during the second part of connecting to the mailbox server on port 444. If you look closely at the HTTP proxy logs, you will have two columns one for HTTP status and one for backend status. In that log you can see the 200 on the first hop but when passing it to the mailbox server on port 444 it fails with the 500 error. Another log that you can check is the IIS log for W3SVC2. In this folder you have the logs for all traffic going thru the Exchange Back End web site.

Screenshot 1 shows wrong binding configuration

Screen Shot 2 of Exchange Shell Error when certificate is misconfigured

 Screenshot 3 after editing (corrected) the Exchange Back End Certificate Binding

Screenshot 4

Screenshot 5 and 6 of OWA behavior when certificate binding incorrect on the Exchange Back End Web Site

 

After logging in you get a blank page

 

Conclusion

The main point of this article is that if you have incorrect certificate bindings on the Exchange Back End web site, it can cause the web services on Exchange 2013 server not to work properly. This incorrect certificate binding will break the connection flow, causing clients to have a bad experience. Always make sure the Exchange Back End certificate bindings for 444 always is configured to use the self-signed certificate.

How to Configure Public and Private Computer Settings in OWA 2013

This tip explains how to enable and configure the Public and Private computer settings in OWA 2013

The new Exchange 2013 Outlook Web App (OWA) logon page no longer allows users to select whether they are using a public or a private computer. By default, OWA 2013 assumes users are using a private computer the default timeout of 8 hours is used. This timeout specifies how long a user can be inactive before requiring him/her to sign in again.
The LogonPagePublicPrivateSelectionEnabled parameter in the Set-OWAVirtualDirectory cmdlet specifies whether the OWA sign-in page includes this private/public computer sign-in option. While by default this parameter is set to True in Exchange 2010, in 2013 it is set to False. To change this on server CAS1, simply run the following cmdlets:
Set-OwaVirtualDirectory “CAS1\owa (Default Web Site” -LogonPagePublicPrivateSelectionEnabled $True
IISreset /noforce
Similar to previous versions of Exchange, the default timeout for private computers is still 8 hours for public computers 15 minutes. You can change this by running the following cmdlets to create the necessary registry keys:
Set-ItemProperty “HKLM:\SYSTEM\CurrentControlSet\Services\MSExchange OWA” -Name PrivateTimeout -Value <timeout_minutes> -Type DWORD
Set-ItemProperty “HKLM:\SYSTEM\CurrentControlSet\Services\MSExchange OWA” -Name PublicTimeout -Value <timeout_minutes> -Type DWORD
IISreset /noforce






With Exchange 2013 administration moving to browser based, you can manage your Exchange 2013 environment from anywhere in the world, provided that they are published. How can we switch off access from outside the corporate network?

You can manage your 2013 environment with EAC internally by navigating to https://casserver/ecp or https://loadbalancedurl/ecp if you have an NLB or load balancer in place. Your external webmail url (https://mail.domain.com) can be used to access the EAC (with /ecp) from outside the corporate network.

You may have a valid reason to turn off access to EAC from outside the internal network and the good thing is that it can be turned off, without affecting your OWA or OWA Options. By default, the EAC access from outside the network is enabled.

You can turn it off by setting the AdminEnabled parameter of the ECP virtual directory to false.

It will take few minutes for Exchange to pickup the change. You can always run an iisreset to force the change. If you try accessing the EAC from outside now, you will receive a 404- website not found error, but your OWA and Options will continue to work.

Disable External Access of the Exchange Admin Center

With the introduction of Exchange 2013, Microsoft has removed the Exchange Management Console (EMC) and replaced it with the web-based Exchange Admin Center (EAC).  The EAC is a web based management console, which makes a lot of sense considering the large cloud implementations that Microsoft has with Office 365.  Moving from a thick client to a web client allows for more portability and management from anywhere.  The EAC is published through the ECP virtual directory on an Exchange 2013 CAS server.  The ECP also allows users to access their options in Outlook Web Access.  So if you've published OWA on the internet using Exchange 2013, then you probably have also published the EAC on the internet.  If that doesn't seem like a great idea to you, read past the break to see how it can be resolved.

Microsoft does provide a way to disable access to the EAC on a given CAS server.  The document for that is right here.  To summarize, you run a Set-EcpVirtualDirectory command and set the AdminEnabled property to False.  The ECP virtual directory will still allow users to access their options in OWA, but it will not allow access to the EAC.  Of course that means that you no longer have access to the EAC as well, so I hope you like PowerShell!  Microsoft recommends setting up an additional CAS server for EAC access only, if you don't have an internal only server already.  That seems a bit onerous and hamfisted to me.  There should be a more elegant solution, and lo and behold there is.  The solution is to add a second website to the CAS server and create an ECP and OWA virtual directory in that new website.

The steps for it are not necessarily obvious, so I have laid them out here:

1. Add a second IP address to the Exchange 2013 CAS server
2. Create the folder in this path %SystemDrive%\inetpub\wwwroot2
3. Create a second website in IIS pointed to the wwwroot2 folder and call it CustomEAC
4. Bind the website on ports 80 and 443 to the new IP address
5. Create a new ECP virtual directory using the command
   New-EcpVirtualDirectory -Server ServerName -WebSiteName "CustomEAC" -InternalUrl "https://eac.fqdn.com/ecp"
6. Create a new OWA virtual directory with this command
   New-OwaVirtualDirectory -Server ServerName -WebSiteName "CustomEAC" -InternalUrl "https://eac.fqdn.com/owa"
7. Set the original ECP virtual directory to restrict admin access with this command
   Set-ECPVirtualDirectory -Identity "ServerName\ecp (default web site)" -AdminEnabled $false
8. Edit the IP Address restrictions to only allow access to the EAC website from the internal subnets
9. Add a host record in DNS for the new EAC URL

If you are using a wildcard certificate then you can use the new eac.fqdn.com URL and not get a certificate error, otherwise you will get a certificate error when going to the website.  If you have an internal CA, you could always get an SSL certificate for that URL since it's for internal use only anyway.  The other option is to use non-standard ports for the second website, but continue to use the original IP Address.  For instance you could use 8080 and 8443 for the bindings, and still use the original URL.  That would avoid the certificate error issue as well.  The question becomes whether you want to remember the alternate ports you used, or the new URL.

I have tested this with Exchange 2013 SP1 and it seems to work without issue.  If you come across any problems, give me a holler in the comments!

Disaster Recovery for Exchange Server in 30 minutes!

Introduction The “Official Microsoft method” for Exchange disaster recovery The “fast way” for Exchange disaster recovery Exchange Server design Disk imaging methodology Backup methodology Recovery procedure Conclusion

Introduction 
What would be your reaction if I told you I intended to format the C partition on your production Microsoft Exchange server to run a quick test?  Your first reaction would probably be to grab the nearest blunt object for use on my head before I even got a chance to explain my self.  Unfortunately, this is no joke and the dire consequences of viruses, incompatible patches, or other malicious events on a companies Exchange server is all too often the worst living nightmare of any Exchange admin and their bosses.  In this article, I’ll show you how to protect your Exchange server with a method that recovers an Exchange server in 30 minutes!  Some of you at this point may wonder, “But I backup my exchange database, why do I even care about this?”  Unfortunately, an Exchange database is useless with out the original Exchange server in the original domain environment.  If anything bad happened to the OS or Exchange application where the server was not recoverable, the data would be worthless because you can’t just mount that database on any old exchange server.  Microsoft Exchange recovery is one trickiest things to execute, this article is meant to make that as painless and reliable as possible.
 
 
The “Official Microsoft method” for Exchange disaster recovery
If you’ve ever hired a Microsoft certified consultant or worked in a large corporation running Exchange, you’ve probably heard of or are using the “official Microsoft” method for Exchange disaster recovery.  Unfortunately, anyone thinking about or running this “official” method is a glutton for punishment.  The method involves maintaining a mirrored parallel universe of your Domain/AD and Exchange infrastructure.  It involves building a backup domain controller on your existing network, and then putting it in an isolated LAN and promoting it to the PDC.  Then you must carefully and meticulously build an Exchange server on that isolated LAN from scratch without making a single mistake in spelling using the identical settings of your production environment.  Only then can this parallel universe be used in the event of a catastrophic failure of your production Exchange environment.  This procedure is complex, prone to error, and expensive.  Needless to say and without getting further into the details of this clumsy formal methodology, you don’t want to go down this road because there is a better way.
 
Background note:  I’ve personally seen a high paid consultant spend 2 weeks implementing and documenting this disaster recovery plan for my company.  In the end, that same consultant could not replicate that environment using the documentation he wrote him self and finally gave up after a day. 
 
 
The “fast way” for Exchange disaster recovery
As a result of my personal experiences above, I refused to believe that this is what I must live with in order to have Exchange recoverability.  Believe me, I got plenty of flack for it from my collogues and the very same consultant who couldn’t read his own documentation and in turn had expected us to use in case of a disaster.  They insisted that this is the “official” way and is how all the big corporations do it.  At the time in year 2000, I was just beginning to use system imaging (a process of coping an entire hard drive or partition on to a single file called an image) to begin a large deployment of new workstations running Windows 2000 and all new applications.  I soon began to wonder, why not use system imaging for servers, and Exchange in particular.  It seemed a daunting challenge because these were high end servers running complex SCSI RAID 5 configurations and it seemed like imaging wouldn’t work.  As it turns out, since hardware RAIDs are completely transparent to the OS and applications, Norton Ghost or Power Quest Disk Image worked just as well on servers as they did on workstations.  Once I managed to get this working, I managed to build a test Domain and Exchange sever in which I took an image of the OS and Applications partition and managed to restore it in less than 30 minutes after I formatted the C drive (data resided else where).  I was confident that this would work to restore the basic Exchange server, but I wondered if the re-imaged Exchange server would recognize a more recent database if some additional emails were sent after the system image was taken.  I tried this in the lab and indeed it was possible, the image-restored Exchange server would even mount a newer database with more recent data.  What this meant was, even if I had made a month’s worth or any amount of updates to the database by just everyday usage, the Exchange server would have come up and brought up all the old and new data.  I realized had come across the ultimate disaster recovery procedure for servers and all that were needed was some refinement in the process.
 
The following refinements are what I came up with:

• Exchange Server design
• Disk imaging methodology
• Backup methodology
• Recovery procedure

Exchange Server architecture 
To make this recovery method feasible, a fundamental design must be followed on the Exchange Server.  Data must reside on a separate partition (physical preferred but logical is ok) from the OS and Application partition.  The last thing you want to image is 1 Gigabyte of OS/Apps plus 100 Gigabytes of data on a single partition.  You loose the granularity and convenience of being able to recover OS and Applications without affecting the Data.  For existing servers that already have data mixed in with the OS and Applications, you could add an additional storage device and move the data store and log files to separate partitions on the newly added device.  Ideally, one would go by the following guidelines for maximum safety, scalability, manageability and performance.

• Put the OS and Applications on the C partition.
• Log files (AKA transaction logs) on D partition.
• DO NOT put log files on same physical device as the Exchange Data Store (Database), if that device failed and you lost the database, you would also loose the ability to recover data made after the tape backup from the Exchange transaction logs.  I’ve seen bad designs that do this and they paid dearly for it loosing a whole day’s worth of data because the storage device they housed the data store and log files on failed.  Their tape backups managed to restore everything up to the previous night, but loosing a day’s worth of emails in a large corporation is potentially a “resume producing event”.
• Put the Data store on it’s own physical drive or block-level storage device such as a Fiber Channel or iSCSI (new IP based storage standard) SAN (Storage Area Network).  Better yet, use Exchange 2000 and break it up into multiple chunks along departmental lines and put them on separate physical RAID partitions.  The more separate physical partitions you use, the faster it performs.  This is due to the fact that a RAID partition can only seek one thing at a time without jumping around at a heavy cost to performance.  Two physical devices can seek two things at once without jumping back and forth between two tasks, and every physical partition you add gives you another simultaneous data seeker giving you a linear gain in performance.
• In practice, this could easily and cheaply be accomplished on typical midsize servers with hardware RAID and six drives.  Three pairs of drives can be configured for RAID 1 mirroring to create three physical partitions.  The first partition could be broken down in to logical C and D partitions for OS/Apps and Log files respectively, and the second two physical partitions can be set as the E and F partition for housing multiple data stores.  Higher end solutions can use external SAN based solutions using a similar approach in RAID setup, this opens the possibility of clustering your Exchange 2000 server.  Do not just lump all 6 drives into one massive RAID 5 array and chop it up into multiple logical partitions, this method is cheaper because only one drive is lost to redundancy but it is at a horrible expense to seek time (three fold loss in seek times to be exact because all 6 drives must act in unison).  Hardware and Drives are so cheap now that it isn’t worth the savings.  In general, database applications are best described as death by a thousand tiny requests, simultaneous seeks capability is heavily favored over the improved sequential transfer rates that RAID 5 offers.
• For Exchange 2000, put three data stores on the E partition and three data stores on the F partition.  Doing this makes recoverability and maintenance a snap.  If you put one large data store on one partition, you cannot safely use more than half the space on that partition.  If you need to compact a database during maintenance or if you needed to do a database repair in the event one of your data stores got corrupted, the minimum free space on that partition needed must be equal to the size of the data store you are compacting or repairing.  Three data stores on one partition means you would only need to reserve 25% of the partition for maintenance or repair jobs.  Database corruptions are common with any sort of database, having 6 data stores makes repairs 6 times faster and easier because the stores are 6 times smaller.  Additionally, the 5 remaining stores are not affected when the 6th data store is being compacted or repaired, allowing zero down time for most of the company.  I highly recommend Exchange 2000 or up because of this feature.

 
Disk imaging methodology
Before you start, be sure your Exchange server is in full operational order with all database store structure, anti-virus, backup agents, and anything else installed.  In order to image a system, you generally need a dump your image onto a separate physical partition from DOS mode, you cannot image a boot partition while that partition is loaded.  The easiest way to do this is to dump an image to a network file share. To avoid writing an entire 10 page chapter on how to create a TCP/IP network boot disk with SMB client capabilities and save you a ton of time, I’m going to say just one word; bootdisk.com.  www.bootdisk.com is the one stop place you can freely download pre-made bootable images that pretty much work with all common network adapters.  From there, you only need to make a few minor modifications to the drive mapping batch file to mount your network drives onto a drive letter.  I would then recommend that you make a bootable CDROM image of the modified floppy disk for vastly improved boot times.  Then you would simply boot up the CD with the network drivers and automatically map the network share.  That network share should also contain a recent copy of Norton Ghost or PQDI (Power Quest Drive Image).  From there, you simply run Ghost or PQDI and dump the C partition onto the network share.  Additionally, I would create an image backup of all the Log file and Data Partitions as well with just the database structure and no data.  Although imaging the data partitions is not mandatory, it will save you a lot of trouble by recreating the entire partition and database directory structures when doing a bare metal restore (starting from scratch with a new piece of identical or flushed hardware) of your Exchange Server.  Note:  The bare bone database structure partition images will be very small because they are almost all compressible.
 
With newer versions of Ghost or PQDI, they support a new hot backup feature where new changes to the OS and Application could be tracked while the system was operational.  Note that you must first create an initial image of the C partition in DOS mode.  Then you would track any changes to a production server even while the system is running by backing up all deltas to the OS and Applications with this new feature.  This is a valuable feature because it is not feasible to down your server just to do a cold image backup every other week.  Once all of this is in place, you can put your exchange server into production and start populating the Exchange data stores.
 
Backup methodology
Up to this point, everything has been focused on backing up the OS and Application partition using disk imaging software.  Backing up the data is equally important for an Exchange server.  For anyone serious about performing hot backups of an Exchange server, you must use a reliable 3rd party enterprise solution that hooks into Exchange with a backup and restore agent.  One of the better solutions I have seen is from Legato.  Some other solutions that I have worked with were utter nightmares and never worked consistently and couldn’t always successfully restore, which meant someone’s head was going to roll.  Since this article is not specifically about data backups, I will move on.  For the large enterprises that can’t even afford to loose an hour’s worth of data, I would recommend that you go the extra mile of continuously making copies of the log files onto a separate physical device with some automated hourly batch process.  Tape backup covers you up to the previous night and log files can cover you up to the last few minutes just before a database disaster.
 
Recovery procedure
There are several types of disasters that can happen to an Exchange Server such as database corruption and server corruption of the OS or Application.  In some cases you may be able to recover from a database corruption by running the repair operation on the affected data store or calling Microsoft and finding a way to fix a severe OS or Application error on the Exchange server.  However, in the even that the that a system completely dies where neither of the above options work or you simply loose an entire machine including all of the OS, Applications, Log files, and Data due to some catastrophic disaster, following the guidelines mentioned above is what is going to save you.  In the even of a catastrophe, you would run the following procedures.

• If Exchange server can’t boot, or the Exchange Services refuse to load normally, proceed to the following steps.
• Boot system with DOS disk with network support (same one you used to make image of C drive).
• Load the C image with the image of the good exchange server from the network share where you keep all server image backups onto the corrupted C partition and reboot when complete.
• If you were maintaining hot update backups with Power Quest Delta Deploy or Norton’s Ghost, apply those updates to your OS and Applications.  Reboot if needed.
• If you had not lost the data partition or corrupted the database, then all should be well at this point.  The Exchange server should recognize and mount the database automatically.  This whole process up to this point can be done in 30 minutes.
• If you lost the data as well, or you are doing a bare metal restore, you need to re-image the data partitions with the bare data structure images.  If you didn’t bother to do that, you will need to create the identical partitions and directory structure manually which may be very difficult.  (If you had put the data on a separate physical device, the odds of this happening simultaneously with an OS/App failure are slim).  Reboot if you re-imaged that log and database partitions.
• Once rebooted, your Exchange server will be fully operational with an empty database.
• Invoke the data recovery application and proceed to recover data from tape.
• If you were wise enough to have copied the log files to another physical server/device before the disaster, now would be the time to copy those log files back and apply those logs to recover data up to the last couple of minutes before the Exchange server crashed.

Note, it is rare that you would need to do a bare metal recovery of an Exchange server, but following the above procedure ensures you have maximum recoverability of your business critical Exchange servers in any event.  On a last note, you should keep an off site tape copy of your images and database, this goes for any other critical server or application. 


Conclusion
The above procedure is the easiest and most reliable of way of recovering from an Exchange disaster.  However, you should note that the concepts apply to any other server or application.  Some may ask “but Microsoft doesn’t support this do they?” or “Microsoft does not support disk imaging”.  The truth is, I use disk imaging on all my servers and I have never been turned down for support at Microsoft, nor do they even ask if you using disk imaging at all.  I even maintain a library of generic system images of every type of system configuration I have so that I can deploy or redeploy any new server rapidly and have never been turned down for support.  Microsoft support is one of the more reasonable solutions out there, where else can you get unlimited attention until an issue is resolved for $250.  Additionally, Microsoft has already announced their own image deployment strategy with a new API called ADS (Automated Deployment Services) in which major players have already pledged their support.  Bottom line, disk imaging makes eminent sense and can be applied to any type of server or workstation.