Exchange Server maintenance Scripts & Patch and reboot script for an entire Exchange environment

Articles in the "Reboot script for Exchange environment" series

1. Patch and reboot script for an entire Exchange environment, Part 1 [This article]
2. Patch and reboot script for an entire Exchange environment, Part 2
3. Patch and reboot script for an entire Exchange environment, Part 3
4. Patch and reboot script for an entire Exchange environment, Part 4
5. Patch and reboot script updated to version 2.1
6. Patch and reboot script updated to v2.4

Back in July, I posted a script that uses Microsoft Update to download and install hotfixes. That script is part of a bigger process that I use to patch all of my Exchange servers. I mentioned that the script for that process was forthcoming. It is a big script, and I have yet to post it because I am always tweaking it. But if I wait for it to be code complete, I will never post it. So this is the first post of what will be several to document what it does.  This post will only cover the environment in which I am working and what the script requirements are.

First is the environment.  It is wholly Exchange 2010 (as of this writing, all servers are SP1 RU6) on Windows 2008 SP2.  There are over 8000 mailboxes, of which 1500 or so are shared.  There are no public folders.  Server and database infrastructure:

• Two data centers, one for production and one effectively for DR
• Primary data center:
  • 2 HT/CAS servers behind a hardware load balancer (HLB)
  • 2 UM servers
  • 3 active MB servers
  • 1 lag MB server
• Secondary data center (DR):
  • 2 HT/CAS servers behind an HLB
  • 1 UM server
  • 2 MB servers
• DAG properties
  • Single DAG that spans both sites
  • 22 databases
  • 2 copies of each database across the 3 mailbox servers in the primary data center, e.g., DB1 is on server 1 and 2, DB2 is on server 2 and 3, etc.
  • All databases have a copy on the lag server
  • Each mailbox server in the secondary data center has a copy of every database
  • Each database has 5 copies total (including lag)

At my company, patch management is done by an application that has no awareness of Exchange, and reboots are simply staggered.  This is not an acceptable solution for patching an Exchange environment that has high availability, needs databases to be balanced after patching, and ideally best practices are followed for suspending DB replication before rebooting, etc.  As a result, these are the requirements for automated patching and rebooting of my Exchange environment:

• Process each server serially so that any problem that arises never affects more than one server, maintaining HA
• Move active copies of databases to other servers so there is zero impact to clients
• Suspend and resume replication of passive copies before and after reboots
• Balance active databases when all mailbox servers are complete
• Put servers in maintenance mode
• Send status notifications to mobile device
• Send detailed log when complete
• Gracefully exit if any timeouts are reached (patching, rebooting, service starting, etc.)
• Verify required services are started before moving on

My script, currently at 445 lines, meets all of these requirements.  There are other things I intend to add to it, but I will go into detail after I have deconstructed the script for you in future posts.  Part 2 of this series, which will cover the overall script structure and its first section (variables), will be published soon.

Download the complete script:

  Reboot-ExchangeServers.zip (7.5 KiB)

How to upgrade or install Exchange 2013 SP1

SP1

Exchange 2013 SP1 is available since few days (cf this news to see the news features). So now it is time to see how to install it.
Even if the installation/upgrade is not complexe before you start to install Exchange 2013 SP1, you should do a backup (baremetal backup or a snapshoot of your VM).

Most of time servers are multiroles but if you have mono roles (only CAS or Mailbox) you must update your Mailbox servers first then your CAS.

My Lab is just 2 Multiroles server, 1 CAS and 1 mailbox.

• 2013EXC01 = CAS + MBX
• 2013EXC02 = CAS + MBX
• 2013EXC03 = CAS
• 2013EXC04 = MBX

Upgrade and prepare Active Directory

The first thing to do is to upgrade the schema of your Active Directory and prepare your domain. We will use Powershell to do this. You need to have enought right on you AD and Schema to do it.

setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms
setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms

Prepare the Mailboxes Server

I start to upgrade the 2013EXC04. 2013EXC02 will receive the messages. In a second step I will do 2013EXC02 (2013EXC01 will receive the traffic), then I will update 2013EXC04 (mailbox server, no CAS role on this one). And I will finish by 2013EXC03 because it a CAS only.

Pass in maintenance mode :

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Draining -Requester Maintenance
Redirect-Message -Server $env:COMPUTERNAME -Target 2013EXC02.gsx.com

Stop replication if your mailbox server are in a DAG (skip this step if your servers are not DAG members)

Suspend-ClusterNode $env:COMPUTERNAME
Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $True
Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Blocked

Stops the server from acknowledging load balancer heartbeats and disables all proxy services.

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Inactive -Requester Maintenance

Upgrade or Install Exchange 2013 SP1

ok now it’s time to upgrade to Exchange 2013 SP1. If you still not have the SP1 you can get it here. You can start the GUI to upgrade or use the PowerShell :

Setup.exe /m:Upgrade /IAcceptExchangeServerLicenseTerms

Congratulaion you just install Exchange 2013 SP1.,,but it’s not over. At this point you have to reboot your server.

Finish your installation/Upgrade

Now you have to return in production (yes you’re still in “maintenance mode”). Pass those PowerShell command :

Set-ServerComponentState $env:COMPUTERNAME -Component ServerWideOffline -State Active -Requester Maintenance

if your server is a DAG member like me you have to restart the replication :

Resume-ClusterNode $env:COMPUTERNAME
Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyActivationDisabledAndMoveNow $False
Set-MailboxServer $env:COMPUTERNAME -DatabaseCopyAutoActivationPolicy Unrestricted

and you will finish with this command to restart the transport service on you mailbox server :

Set-ServerComponentState $env:COMPUTERNAME -Component HubTransport -State Active -Requester Maintenance

Check your installation

check all the components of your server. They should be “Active”

Get-ServerComponentState $env:COMPUTERNAME | Format-Table Component,State -Autosize

Check you replication. The state must be “Up”

Get-ClusterNode $env:COMPUTERNAME | Format-List

If you got it, check with GSX Monitor the status of the EWS, the status of the server in the main view should be OK. If it is the case that mean your DAG is running, your queues are running and you mail flow is OK. If you don’t have it…you should 

At this step you have to do others mailbox then your CAS.

Exchange Server 2013 SP1: A Mixture of New and Completed Features

Three months after they shipped Exchange 2013 CU3, Microsoft has announced the release of Exchange 2013 SP1, or cumulative update 4 (CU4) to its friends. The announcement will no doubt come as pleasant relief to those who insist that no Microsoft server product can ever be installed until the first service pack appears. Like waiting for the first cuckoo of spring to sing before planting, such a well-worn adage is challenged in an era when the demands of the cloud mandates that on-premises customers receive quarterly updates, but some people find it hard to shift old habits. In any case, build 847.32 aka Exchange 2013 SP1 is now available for download.

To make sure that those running older versions of Exchange are not left out, Microsoft has also released Rollup Update 13 for Exchange 2007 SP3 and Rollup Update 5 for Exchange 2010 SP3.

I won't bore you with the details of how to install Exchange 2013 SP1 because the upgrade from CU3 was easy (at least for me). A schema extension is required to accommodate new objects and cmdlets and the consequent updates to RBAC roles, so be sure to include this step in your planning. The normal caveats about preparing DAG member servers by putting them into maintenance mode before starting the upgrade and shutting down all Exchange components like EMS and EAC apply. My upgrades occurred without trauma, which was a nice surprise. The sole caveat is to check that all services come back online after the upgrade as the transport services can be picky about restarting.

Looking through the set of features and updates provided in Exchange 2013 SP1, we find a mixture of finishing off important components and extending new functionality. Adding S/MIME support back for Outlook Web App (OWA) is an example of the former; providing the ability to add custom sensitive data types through document fingerprinting for Data Loss Prevention (DLP) is an example of the latter. The full list of updated functionality in SP1 is shown below. Where appropriate, the features are also available to users of Exchange Online in Office 365. In fact, the nature of the development process is that new functionality is slip-streamed into production in the cloud some weeks before it is made available to on-premises customers in an update like SP1. It is therefore quite possible that you have been able to use upgraded functionality for some time, even if you never realized it. 

My personal selection of the most interesting changes in Exchange 2013 SP1 are:

• The introduction of MAPI over HTTP (aka “alchemy”) as the preferred interprocess connection mechanism for Outlook and Exchange. The RPC over HTTP mechanism is continued to serve older clients but it is likely that MAPI over HTTP will become the predominant and eventually the sole method to connect Outlook and Exchange in the future. RPC is an old mechanism that struggles with the kind of unreliable and often flaky networks (think public Wi-Fi) that we use so often today and it is hoped that the removal of a layer will make client connections more robust and easier to maintain. MAPI over HTTP used HTTPS, is based on the HTTP 1.1 specification, and clients use POST commands to communicate with Exchange. In effect, MAPI over HTTP makes Outlook connections behave much more like those from EWS, EAS, and OWA clients and should (hopefully) mean that Outlook is better able to cope with scenarios such as resuming from hibernation, network or adapter transitions, network failures, and the like. It might even mean that I never have to use OWA in offline mode again! It's important to understand that Outlook 2013 SP1 is the only client that can connect to Exchange via MAPI over HTTP so this transition will take a long time to become effective for many organizations. In addition to deploying supported clients, you have to make a one-time configuration change to enable MAPI over HTTP. In the meantime, RPCs will continue to flow as before and clients will remain connected. Microsoft plans to enable MAPI over HTTP for Office 365 soon (if they have not done so ny now) and new clients will begin to use MAPI over HTTP as they are deployed. No immediate plans exist for Microsoft to turn off RPC over HTTP within Office 365, but it wouldn't be a surprise if this happened in a couple of years. Stay tuned for an in-depth WindowsITPro magazine article covering the benefits and some of the downsides in this change, but in the meantime if you want to learn more, listen to this Channel 9 program about MAPI over HTTP.
• OWA and OWA for Devices can now display DLP policy prompts. This is important because until SP1, Outlook 2013 was the only DLP-aware client and this represented a deployment block for companies who considered the use of DLP to protect against the disclosure of sensitive data through email.
• Organizations can add custom documents to the set of DLP sensitive data types used by policies on clients (as messages are composed) or via transport rules (as messages pass through the transport pipeline). This is done by creating a document fingerprint of a sensitive data type such as a tax return form that makes it a known type for DLP checking. The set of DLP templates provided in Exchange 2013 SP1 is extended to accommodate the needs of more countries and regions. See the EHLO blog for more information on the DLP enhancements, which are now also available for Exchange Online tenants in Office 365.
• Cmdlet logging is restored for Exchange Administration Center (EAC). The EMC console used by Exchange 2007 and Exchange 2010 has three ways for administrators to see what EMS commands are executed to get work done. These are invaluable in terms of exposing people to cmdlet syntax and values. Exchange 2013 SP1 allows you to enable cmdlet logging and have a separate window where cmdlets are displayed as they are executed by EAC.
• Windows 2012 R2 joins the set of supported operating systems. You can now deploy Exchange 2013 SP1 on Windows 2012 R2 servers - and use Windows 2012 R2 domain controllers and global catalogs (and Windows 2012 R2 DFL/FFL). Windows 2012 R2 DC/GC support is also gained by Exchange 2010 SP3 RU5 and Exchange 2007 SP3 RU13. However, Exchange 2007 SP3 RU13 does not support Windows 2012 R2 DFL/FFL and you can't install Exchange 2010 or Exchange 2007 servers on a Windows 2012 R2 server.
• Simpler DAGs. If you deploy Database Availability Groups (DAGs) on Windows 2012 R2, you now have the chance to create an "IP-less-DAG" (or, as Microsoft call then, a DAG without a Cluster Administrative Access Point).  This is the continued evolution of the DAG model where Exchange takes on increased responsibility for all aspects of DAG management. With the new mode of DAG, all management is done through Exchange and the Failover Cluster Manager, CNO, network name resource, and DNS entry for the cluster are no longer required. This simplification is welcomed and I'll be commenting on it in more detail in the near future. Remember that the requirement that all of the member servers in a DAG must run the same operating system continues, which means that if you want to deploy Windows 2012 R2, you might have to rebuild DAGs.
• S/MIME support returns to OWA (for IE9 and above, but not for Chrome, Firefox, or Safari). This is functionality that was removed as part of the transition in OWA architectureto deal with multiple device display formats. With Exchange 2013 SP1, S/MIME is supported across Outlook, OWA, and Exchange ActiveSync clients. The Set-OWAVirtualDirectory cmdlet has been updated to allow S/MIME to be enabled or disabled on a server.
• Firefox browsers that support the HTML5 appcache mechanism can take advantage ofOWA’s offline mode.
• In another example of functionality being restored, OWA includes a rich text editorthat is really very usable (screen shot below). It's not quite Word, but the editor is more than acceptable when it comes to creating nice-looking memos.

• The Edge transport role reappears in a new guise. Up to Exchange 2013 SP1, deployments had to use Exchange 2010 Edge servers if they wanted to use Exchange as the point of contact for inbound mail from the Internet. If only because all of its administration has to be performed through PowerShell, the Exchange 2013 version of the Edge transport role is very different to its Exchange 2007 or Exchange 2010 equivalents, so some careful testing and assessment is required before the new servers should be deployed. Some will decry the lack of a GUI and the need to manage the server with PowerShell, but the tasks to set up an Edge server are relatively straightforward and anyway, if you can't handle a few PowerShell commands, maybe you should not be setting up a server that acts as the secure inbound email access point for the organization? 
• The Outlook app model is extended to accommodate “compose” apps. In other words, where you could use apps to process data as you read messages (such as extracting an address from a message and displaying it with Bing Maps), apps are now available when you compose messages (see the apps button in the screen shot of the message compose screen above).
• SSL offloading is supported, meaning that the SSL encryption and decryption workload for inbound connections to Client Access Servers can be moved to a load balancer.

Like any service pack, other changes exist that are less important to the general Exchange community but critical to specific groups. For example, authentication is now supported using U.S. Department of Defense smartcards. This definitely won't impact my working life, but it might for those who work with the U.S. military. No doubt many blog posts will fill in the detail of all the other changes in the coming weeks when the finer details of new cmdlets and functionality will be explored. For those who care about these details, Exchange 2013 SP1 includes 981 cmdlets whereas the RTM version has 958.

Naturally, because Exchange 2013 SP1 is also a cumulative update, it includes a large number of bug fixes in response to problems found in testing or reported by customers. For this reason and because a large amount of new functionality is present, you need to test Exchange 2013 SP1 thoroughly before introducing it into production.

I’ll be covering some of these features in detail in separate posts and articles over the next few weeks (or when I get around to it – other stuff happens that deserves immediate coverage and disrupts my writing plans). In the meantime, I am sure that the Interweb will abound with posts providing the opinion and views of those who care to comment whether Exchange 2013 SP1 is worth deploying. At first glance, it is.

Exchange 2013 Server Maintenance (Windows Patch Update etc.) - A Step by Step Guide

Submitted by Nathan O’Bryan on Jun 3, 2013.

So you’ve deployed Exchange 2013 in a highly available configuration. You have two (or more) Client Access servers running behind a hardware load balancer, and you have two mailbox servers running in a Database Availability Group. Now it’s a couple of days after “patch Tuesday,” and you’re ready to update your servers. You’ve applied the patches in question to a test server, and you’re confident that they will not have adverse effects on your Exchange servers.

Now what? Do you just apply the patches, and let the high availability features on Exchange keep your users connected? What is the proper order in which to patch your servers? Do all of your servers need to be running at the same patch level, or can you apply some patches to one server, but not another?

Exchange 2013 – Steps for Updating Servers

Today I am going to outline the recommended process for updating your Exchange 2013 servers. This process is intended to apply to Exchange 2013 servers configured with at least 2 CA servers behind a Hardware Load Balancer, and at least two MB servers in a DAG.

• Step 1 – Testing and Deploying Patch on High Availability Exchange
• Step 2 – Identifying Servers to Update, and Order of Update
• Step 3 – Put Servers into Maintenance Mode
• Step 4 – Scripting the process
• Step 5 – Wrapping it up: Returning to Previous Server Configuration

Step 1 – Testing and Deploying Patch on High Availability Exchange

Although it is a rare occurrence, every once in a while a patch is released that can have adverse effects on your Exchange organization. As you’ve put quite a lot of time and money into deploying a highly available Exchange organization, it would be a shame to waste that effort by bringing down your messaging services with a bad patch.
For that reason it is always recommended that you carefully review the release notes, and deploy patches into a test environment before moving them to production.

“Your test environment should be built to match your production environment as closely as possible.”

Step 2 – Identifying Servers to Update, and Order of Update

For previous versions of Exchange, one might recommend a complicated process to determine which servers should be patched in what order. With Exchange 2013, those decisions are much easier.

Exchange 2013 has simplified the process of patching in several ways. First there are now a reduced number of server roles. Exchange 2007 and 2010 each had 5 server roles, but Exchange 2013 now has only two. That means you don’t have to worry about Hub Transport, Unified Messaging, or Edge Transport (the Edge Transport role is expected to return to Exchange 2013, but as of this writing is not available). If you are running a previous version Edge Transport server, in can be patched at any point in the process as Edge Transport servers should not be a member of the same forest as your other Exchange servers, and thus have very little direct communication with those machines.

So in what order do I recommend patching your Exchange 2013 servers? I would recommend building all your Exchange 2013 servers as Multi-Role (CA and MB) servers making the order in which you patch your servers will be completely irrelevant. I’ll write more about the recommendation for Multi-Role servers in a future blog post about Exchange 2013 server sizing.

Step 3 – Put Servers into Maintenance Mode

Assuming you follow my above advice and build all your Exchange 2013 servers to be Multi-Role servers, the below instructions will put them into maintenance mode, and allow you to patch them without interrupting service for your users. If you have broken up your Exchange 2013 servers into CA and MB servers, follow the below instructions on your Mailbox servers. How do you put your Exchange 2013 Client Access servers into maintenance mode? There is no need. Exchange 2013 Client Access servers are completely stateless, which means their only job is to proxy (or in one case redirect) your connections to the appropriate Mailbox server.

A properly deployed collection of CA servers will not require any special attention should one of them be unavailable.

Enabling Multi-Role for Exchange 2013 or Maintenance Mode for Mailbox Server

The following 4 commands will put Exchange 2013 Multi-Role or Mailbox server into maintenance mode. If you are studying for the Exchange 2013 exam 70-341, you may be well served to remember these commands and the order in which they are run as that information could be relevant to you in the near future (subtle enough hint for you?)

Set-MailboxServer <servername> -DatabaseCopyActivationDisabledAndMoveNow $True
Set-ServerComponentState –Identity <servername> –Component HubTransport –State Draining –Requester Maintenance
Suspend-ClusterNode –Name <servername>
Set-MailboxServer –Identity <servername> –DatabaseCopyAutoActivationPolicy Blocked

Once you have completed patching your server, the following four commands will reverse the process.

Set-MailboxServer –Identity ExServer1 –DatabaseCopyAutoActivationPolicy Unrestricted
Resume-ClusterNode –Name ExServer1
Set-ServerComponentState –Identity ExServer1 –Component HubTransport –State Active –Requester Maintenance
Set-MailboxServer –Identity ExServer1 –DatabaseCopyActivationDisabledAndMoveNow $False

Step 4 – Scripting the Process

The easiest way to script this process would be to copy the above commands into notepad, then save that as a ps1 file. There are a lot of other options that PowerShell affords you to automate this process.

What about the StartDAGServerMaintenance.ps1 and StopDAGServerMaintenance.ps1 that were included with Exchange 2010? Those scripts are there in Exchange 2013, but they do not include the command to drain the transport queues. This command was not available in Exchange 2010 and the Exchange team did not update those scripts for Exchange 2013.

Step 5 – Wrapping It Up: Returning to Previous Server Configuration

Once you have patched all of your servers, you’ll need to reactivate your databases on the servers they were originally active on.

Exchange 2013 Server Maintenance: Simple and Safe

Overall the process of taking Exchange 2013 servers out of production for maintenance is much simpler and safer in Exchange 2013. There is much less chance of something going wrong, and if something does go wrong, it’s easier to recover.

- See more at: http://www.enpointe.com/blog/server-maintenance-with-exchange-2013-step-by-step-guide#sthash.gs05bOVN.dpuf