Using Veeam for Exchange 2013 snapshots

Does the thought of VMWare snapshots for Exchange 2013 make you cringe? If so, we are much alike and your concern probably stems from unpleasant experiences you’ve had in the past. Exchange 2010 SP1 began the “healing” process between Exchange and VMWare however much of the stigma remains implanted in the heads of Exchange administrators.

Fear not! As the technologies continue to bridge together they grow more compatible and can thrive together with a small amount of work and monitoring. First, lets talk about the problems Veeam can cause; you will likely see these errors:

FailoverClustering – Event ID: 1135 – Cluster node ‘SERVER’ was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster.

1

MSExchangeRepl – Event ID: 4087 – Failed to move active database ‘NAME’ from server ‘SERVER’. Move comment: None specified.

Error: An error occurred while attempting a cluster operation. Error: Cluster API failed: “ClusterRegSetValue() failed with 0x6be. Error: The remote procedure call failed”

2

The problem is the very way that Veeam operates since it must necessarily “freeze” the guest node to complete the snapshot. This isn’t to say that Veeam isn’t following Microsoft Best-Practices. Veeam DOES in fact initialize VSS to take an Exchange-Aware snapshot so all is well with the backups and logs are being correctly truncated. However, during the snapshot period, the Cluster will detect a short outage and attempt to fail the databases which could set off some other failures as shown above.

What we have to do is change the Cluster settings to be more forgiving of these short “freezes”. The end result is an error-free backup and failover detection that is a little more forgiving of slight network outages or slower server responses. From any server in the DAG, open the Command-Prompt and enter the following text:

cluster /cluster:<DAGNAME> /prop SameSubnetDelay=2000:DWORD
cluster /cluster:<DAGNAME> /prop CrossSubnetDelay=4000:DWORD
cluster /cluster:<DAGNAME> /prop CrossSubnetThreshold=10:DWORD
cluster /cluster:<DAGNAME> /prop SameSubnetThreshold=10:DWORD

These settings are recommended by Veeam to force the cluster to allow for twice the amount of delay for the cluster’s heartbeat and network delay. The numbers look high but they really aren’t. 4,000 milliseconds is 4 seconds and for most companies a 4-second heartbeat tolerance will probably be just fine. I personally think the default second of 1,000 milliseconds is probably too low anyway. The other settings …SubnetThreshold is the failed heartbeat tolerance. By increasing this you also increase the failure tolerances before a fail-over automatically occurs. The default setting is 5 so by doubling that, we decrease the potential for an unplanned failover due to “glitches” with the network or short “freezes” like those instigated with products like Veeam.

These problems should instantly quieten down the Event Logs on your server if Veeam or anther product forces your Exchange Servers to “pause” momentarily for whatever reason. Moreover, if your Exchange environment is not closely being monitored then you may want to make these changes in order to make your DAG more stable during short unplanned problems with the network or perhaps host machines.

Alive and Kicking!

 

Microsoft Exchange Server is alive and well and Office 365 is here to stay.

Roughly 70% of my customers and partners are EXCELLENT candidates for Office 365 and I am very vocal about my support for the move. How many of us have witnessed a crashed database due to excessive logs? How many times have you fought with the load-balancer to figure out why connections are being dropped? How many days do you spend a year debating, fighting, monitoring or discussing Exchange backups and disaster recovery?  With thinning IT departments and greater messaging loads, it is far more difficult and costly to maintain a healthy Exchange environment than ever before. BUT, if BPOS was re-branded to “Office 365” in 2011 and it was the 3rd iteration of Microsoft’s Hosted messaging then why are 80% of the mailboxes still On-Premises in 2015? Why is there such a gap between Hosted and Local mail populations?

Mind-the-Gap_2.14.14-600x398

 

The Radicati Group (www.radicati.com)  has indicated that in 2014, Office 365 accounted for less than 20% of Exchange mailboxes worldwide.  (“Microsoft Office 365, Exchange Server and Outlook Market Analysis, 2014 – 2018”). This paper goes on to identify that Microsoft’s On-Premises market share will increase from 64% to 76% in by 2018 “as it continues to gain market share away from its competitors”.

 

Confused yet? The explanation is pretty easy when you remember that we are human and not machines.

We are in a transitional gap right now with Microsoft Exchange. The industry wants us to be in the cloud, Microsoft wants us in the cloud and most of us want to be there but we all move at different paces. It took me three years to ramp up on Office 365 due to my subdued interest and the complete lack of interest by most of my larger customers. I have since made the transition but very few of my larger customers have made the same mental shift. THIS is the reason for the gap and the reason Microsoft will continue offering the On-Premises version of Exchange until 2018 or even later.

We are creatures of habit and most resist change. Fears about security, privacy and resilience have slowed the adoption of Office 365 but eventually we will all be there. When the On-Premises mailbox population decreases to a number Microsoft is willing to sacrifice then the Exchange Server product will be forever retired.  Until then, I will continue to close the gap with bridges or catapults depending on the need.

Exchange 2010 SP1: Under the Hood

Windows IT Pro has asked me to put together three technical classes for Exchange to be presented January 2011. 

“During this one-day, free online conference, five-time Exchange MVP Steve Bryant will teach you how to:

  • Master the Exchange control panel – In this session, you’ll learn the administrative differences between the Exchange Management Console (EMC) and the Exchange Control Panel (ECP), as well as the benefits and available features for end-users, group administrators, and enterprise administrators. Most importantly, we’ll cover the setup scenarios to help introduce you to Role Based Access Control (RBAC) and how it can help you help others to help themselves.
  • Improve Exchange archiving – In this session, we’ll dive deep into the Exchange 2010 SP1 archive functions with an end-to-end scenario for creating a separate archive store (with different HA), adding and managing user archives, managing the auto-archive feature, searching the archive, and directly importing PST files into the archive. Having your cake and eating it too is now possible with SP1.
  • Accomplish high availability databases with Exchange – In this session, we’ll discuss compatibility with Hyper-V, overall storage planning, WAN implications, and end-to-end scenarios for planning, creating, monitoring, and managing DAGs in your environment. We’ll focus on UI and PowerShell cmdlets available with SP1, so even those experienced with this new feature should learn something new”

EDITED: The PPT files are available should anyone like them.

 

The Exchange 2010 SP1 Archive Solution

For those who don’t know me, let me say that I have a terrible poker face. I am not much for suspense or grandeur so I will now spoil the ending; unless you already use an archiving program like Mimosa NearPoint, Symantec Enterprise Vault or Zantaz EAS then you should definitely read this article and seriously consider configuring the Exchange 2010 SP1 archiving options. In this article, I will show you how to control databases growth, eliminate PSTs and allow users to access both current and archived items from Outlook 2007, Outlook 2010 and Outlook Web Access.

Archiving Principals

Over the years, I have worked and partnered very closely with Mimosa Systems and I have helped to implement email archiving solutions for Fortune 500 companies. Moreover, I have worked directly to prepare for both Federal and State court litigation with email archiving tools. So, before I go any further let’s talk about what archiving means. Wikipedia defines an Archive as “…a collection of historical records, as well as the place they are located.[1] Archives contain primary source documents that have accumulated over the course of an individual or organization’s lifetime.

In general, archives consist of records that have been selected for permanent or long-term preservation on grounds of their enduring cultural, historical, or evidentiary value.“

Based on this definition alone, I would say that an Archive (as it relates to email) is a collection of emails that have been preserved for a set amount of time dictated by the entity that owns the records. The benefit of having an email archive is that it provides fault tolerance to the messages and the ability to globally search and export messages as needed for whatever purpose necessary. This is common for companies that are tightly regulated or under orders (either by the court or internal requirements) to preserve messages.

The Exchange 2010 archive functions do NOT provide tools that match this definition of “archive” and I want to make that perfectly clear. Exchange 2010 does provide Journaling tools to collect, protect and store emails but Microsoft does not label that as Archiving. Even though the tools I describe in this paper have the word Archive plastered all over them, they are in fact designed to manage mailbox and database sizes. Yes, there are some excellent global search tools and yes searches can be delegated and content exported but the user never loses the ability to delete items from their mailbox or archive. I absolutely adore these new features and strongly recommend every Exchange shop use them, but because the data is not protected against user deletion, I have a hard time labeling it as an archive solution.

Exchange 2010 Archive Components

Now that I have said my peace, let’s move on shall we! There are several components that I want to describe before we get into the meat of this. The components are:

 

  • Exchange 2010 Mailbox Server- Yes, this is obvious, but I want to make sure you understand that these features are only available on Exchange Server 2010 RTM and SP1.
  • Exchange 2010 Mailbox Databases – Yes, another obvious point but I wanted to emphasize the fact that Exchange 2010 RTM automatically places the user’s archive within the same Mailbox Store as the user’s mailbox. My original excitement about the archive features somewhat dissipated when I learned about this during the beta. In this scenario, the Archive is forced to participate in the same High-Availability (HA) plan as the live mailbox so if your Service Level Agreements (SLA) requires several copies of the Mailbox Stores then your Archive must follow along and chew up valuable drive space. Fortunately, SP1 allows you to specify a separate Mailbox Store for the user’s archive so you have the ability to tier your Recovery Point Objectives (RPO) and Recovery Time Objectives (RTP) separately.
  • Retention Policy Tags and Retention Policies allow you to control when things are moved to the user’s archive and to whom the policies should apply. While these features are available in RTM, the management of these features require the use of the Exchange Management Shell and the documentation around this is pretty thin. I would recommend considerable lab testing to perfect the management process with the RTM. Exchange Server 2010 SP1 changes how these (and all Mailbox Policies) features are managed and applied. In fact, SP1 makes the application of Retention Policy Tags exceptionally easy and intuitive.
  • Outlook Client – There are three ways to access the archives;
    • Outlook Web Access (OWA) provides direct access to both the user’s mailbox and the users server-based Archive. Unfortunately, it seems that the search tools do not span both repositories so if a user would like to search EVERYTHING for a specific email, they will need to perform two searches; one in the mailbox and one in the archive.
    • Outlook 2010 also provides direct access to the user’s mailbox and the user’s archive. It too currently suffers from the two-search problem. While this is not a show-stopper, it will certainly cause some user-confusion as they will need to know to search twice or they will need to know which repository contains the items they require. It is also important to note that the archive is not cached in Outlook’s Offline Store (OST) and so you can only access the archive when you are connected to the Exchange environment.
    • Outlook 2007 support is added with Exchange 2010 SP1. As of this writing, I have not had the pleasure to test this since it will most certainly require a patch to Outlook 2007 and I was unable to acquire those bits. The expectation is that it will function as the Outlook 2010 client does. I am hoping that Microsoft will figure out a way to provide a unified search, but I am not holding my breath since even OWA 2010 SP1 does not have that functionality.
  • Exchange Control Panel (ECP) is the web-based management interface that among other things allows those assigned the appropriate role to perform a Multi-Mailbox search. While this is not speficially an archive function, it will automatically search both user mailboxes and user archives simultaneously so I wanted to spend a little time on the subject.

 

Archive Management

Exchange 2010 archives are user-specific and so the attributes of an archive are maintained on the User Mailbox object and can easily be accessed by Powershell cmdlets. You can add an archive to an existing mailbox by using the Set-Mailbox cmdlet with an –Archive switch. Additional switches are provided to allow you to specify a different database (new with SP1) as well as quotas so the archive settings for the mailbox may look a little like this:

ArchiveDatabase              : NY Archives

ArchiveGuid                       : d2a0d37c-3a05-4a88-b196-3f71f291fde8

ArchiveName                    : {Online Archive – Kendall Bryant}

ArchiveQuota                    : 50 GB (53,687,091,200 bytes)

ArchiveWarningQuota     : 45 GB (48,318,382,080 bytes)

ArchiveDomain                 :

You can also use the Exchange Management Console (EMC) to enable or disable an archive for the selected user. To enable the archive for a user, simply right-click the mailbox name and choose Enable Archive.

image1

The Enable Archive option provides you the ability to select the specific database that should host the archive. With this feature, items that exist on your Tier1/High-Availability mailbox databases can be manually or automatically moved onto a database with lower availability. Those who are a Microsoft Online Business Suite tenant can enter their domain name to identify a remote hosted archive location.  These features became available with SP1 and represent a significant change of the archive architecture.

Also, the GUI does has a nice little icon it uses to denote who has an archive and who does not; a clever little folder-drawer icon! I am sometimes embaraased as to how easily I can be impressed or amused.

 

 

Automating  the Archive through Retention Polices

So if that was not enough, SP1 completely changed the policy tabs in the EMC. Gone are the tabs known as “Manage Custom Folders” and “Manage Default Folders.” Instead, we now see Retention Policy Tags and Retention Policies. This provides a much clearer definition and easier management for those new to Exchange Server administration.

image2

The first thing you will need to do is define your Policy Tags. The Default Archive Policy is now exposed to the EMC. Hooray! You will probably want to create a new one though if you want to do some granular configurations. Creating a new retention policy tag is just a right-click away or you can just click on the New Retention Policy Tag selection from the Action menu.

As first glance this wizard looks the same Mailbox Manager rules but there are two major differences with SP1. First, under the Action drop down under Age limit you can now select “Move to Archive.”  Secondly, when you want to see or modify the mailboxes that should receive the policy, you can edit the policy the click the Mailboxes tab. From there you can add or remove mailboxes as will.

image3

There are a few more changes that are little more subtle.  As I mentioned before, archive settings for the user are actually User-Mailbox attributes. Litigation Hold and Retention settings can be found under the MRM (Messaging Records Management) from the Mailbox Settings Tab.

image4

On this same Tab, you can select Archive Quote to set rules on the Archive size.

Accessing the Archive

There are three ways to access the user archive; Outlook 2007 (with SP1), Outlook 2010 and OWA 2010. Once the user archive is enabled using the EMC or EMS, clients will see it as another level in their Outlook. In fact, it is very similar to what you would expect if your Outlook was configured to open more than one mailbox.

image5

If you think of it as a separate mailbox, then the limitations I am about to mention make sense.

  • No offline Access – You must be connected to the Exchange environment to get to the user archive. In fact, the Outlook client even shows it as “Online Archive.”
  • Two searches – Neither Outlook or OWA can simultaneously search both the user archive and the Mailbox for items. NOTE: THIS IS NOT CORRRECT. ONE SEARCH WILL WORK FOR BOTH BUT IT RELIES ON THE MICROSOFT SEARCH SERVICE.

Multi-Mailbox Searches

Interesting enough, Exchange 2010 does provide the ability to search both the user archive and mailbox simultaneously, but not with Outlook clients and not with tools designed for the general population. Exchange 2010 now supports a robust Role Based Access Control permissions model. In this model the role-group named Discovery Management provides the assigned person the ability to perform Multi-Mailbox searches which have access to both mailboxes and archives.

Using the Exchange Control Panel, the Discovery Manager (Role Group) can select Reporting and New to Perform a New Multi-Mailbox Search.

image6

The searches can be fairly complex as you can select the search strings and the types of messages to search. You can also limit the search to specific senders/recipients, date ranges, the specific mailbox(s) you want to search. Lastly, you determine where you want the results stored.image7

The search runs on the server and when the job is complete the assigned Discovery Search Mailbox will receive an email that summarizes the search results. This message also contains an attachment that lists the items found in the search. If, on the New Mailbox Search page you selected to Copy Results to the Selected Mailbox the Discovery Search Mailbox will also contain a copy of all the items that met your search criteria. These items will be located in a folder names for the search itself.

Exporting the Search Items

It’s fairly safe to speculate that those who would require a global Multi-Mailbox (and Multi-Archive)   search would need to present the items to someone, right? Getting to the data takes a little more work. For starters, you need to find the Discovery Search Mailbox in the EMC and give yourself (or the auditor you have assigned) Full Permissions. Now you can simply open Outlook Web Access and see all the items that matched the search.

But what if you need to transport the items out of Exchange; perhaps for litigation? Well you really need an Outlook client for that so you have to jump through a few more hoops. With Exchange 2010 RTM, the Discovery Search Mailbox(s) could not be opened with Outlook. Fortunately that changed with SP1 so you can open it like any other (additional) mailbox by using the Microsoft Exchange account settings in Outlook.

Since we can see the folder from Outlook, we can now export it.

image8

 

The export feature in Outlook 2010 is a little more difficult to find however: First click File from the Outlook menu bar and then select Open in the left pane.image9

Now in order to export, we click Import (ironic huh?) Believe it or not, this is how we access both the import and export tools! Choose Export to File and then select Outlook Data File (.pst) and click Next again.

From this screen, you can select the parent folder you wish to export and make sure the “Include subfolders” option is chosen. Continue through the wizard to export the data to a PST file.

Summary

The Exchange 2010 Archiving tools (especially those that ship with SP1) have features that every Exchange 2010 shop can use. Tailored specifically to help control mailbox sizes, the Retention Policy Tags, Multi-Mailbox search and the separation of the Archive from the Mailbox database provide you the tools needed to better shape your databases and eliminate the need for PSTs. In fact, to make the transition easier, SP1 provides the means of importing PSTs directly into a person’s archive. One last thing I will point out is timing. You would be better served by waiting for SP1 before jumping head-first into Exchange 2010 archiving. Some things will need to be undone in order to do them right with SP1.

 

Debunking the top 5 Myths concerning Cross-Forest Exchange Migrations

Exchange Cross-Forest migrations are not as impossible, expensive or complex as you may think. If you are considering merging an Exchange organization into another organization, you should know that it can be done and you can do it.

Cross organizational moves are complex and on my last large cross-org project we had nearly 100 Exchange 2007/2010 servers and over thirty locations with multiple SMTP paths. Moreover, we were dealing with two separate AD forests with absolutely no automated directory synchronization. Even with these challenges plus WAN link migrations we established a process to successfully migrate roughly 600 people in a six hour window with minimal personnel and an exceptionally low failure rate. If you do the math you will see that we built capability to migrate 2400 a day or 16,800 people per week. Since we are not running four shifts a day, seven days a week I have a few moments to talk about how can do it too.

As you read this, you will notice that I have included code samples, a few tips and some overall ideas to enforce by conviction that this can be done without expensive tools and to illustrate my points. You should not take this article as a complete migration guide but as a confidence builder. There are far more technical strategies that are better described somewhere else such as sizing, migration throughput, error handling, WAN moves, server centralization, scheduling and the overall technical aspects of the scripting and process. I have tried however to give you enough information so you can understand how manageable this process really is.

So let’s set the stage. You are tasked with planning the migration of thousands of Exchange users from one company/organization to another. You have trusts in place and accounts in each Forest with rights and you have read very little documentation that would suggest you can accomplish this on your own. Moreover, you have a quote for $500,000 worth of migration software and have no idea how you will maintain your budget or if the software is even worth it.

Myth 1:

Migrating Exchange mailboxes from one org to another without 3rd Party tools is suicide

FALSE

In my last large cross-org migration project, we moved roughly 30,000 mailboxes using the standard Exchange 2007 “Move-Mailbox” PowerShell command. The syntax is described here: http://technet.microsoft.com/en-us/library/aa997599.aspx.

Having said that let me point out that you should augment that command with additional scripts that provide additional error-handling and account management. In the end, the Move-Mailbox command is the only tool I use to migrate terabytes of Exchange information from one organization to another.  I will show you the command in a moment, but first let’s talk about how we use it:

  •  For bulk moves, we script the command against a text file that contains the names we wish to migrate. I prefer this better than using an AD group to list the migration candidates since it allows us to “lock” the group and easily manipulate the names if so desired.
  • Perform your AD work ahead of schedule. Create Mail-enabled user objects in the target domain and instruct the user community in advance as to how to change passwords and logon. You should avoid using AD Contacts and focus on Mail-enabled users in order to maintain passwords, groups and other attributes before, during and after the moves. This part of the project is critical and deserves its own section as you must maintain all X500, SMTP attributes. Moreover, it is important to cross-pollinate the LegacyExchangeDN value in one directory as an X500 address in the opposite directory for each mailbox. This will dramatically reduce and possibly eliminate reply failures and meeting ownerships.

 

CrossImage1

 

  • Use the Move-Migration script to Mailbox-enable the target object and move the mail but use an outside process to handle all account changes in the source domain. This will give you more control and reliability of the source objects.  The Move-Mailbox script can perform these functions but there is little in the way of error-handling so if the AD is not responsive or there is a connection failure during object modifications, the Move-Mailbox command does not always recover. It is super reliable as a mail migration tool and semi-reliable with its AD changes so focus on its benefits and shore-up its weaknesses.
  • Execute a series of Post Scripts to perform any additional cleanup you may require for the accounts and mailboxes. There will be plenty. You will need a script to disconnect (do not delete in case you need to reconnect later) the mailbox on the source object and to turn it in to a Mail-enabled object with all the previous addresses and mail attributes. You will need another script to compare the object to make sure it is correct.

Just to make sure my point is perfectly clear, this is the exact code we use for every migration:

Param([string]$textfile,[string]$database)

$import = Get-Content $textfile

$SourceCredential = get-credential

$TargetCredential = get-credential

$targetGC = "DC2.targetcompany.com"

$sourceGC = "DC2.sourcecompany.com"

#move the migrated user's mailbox

$report = "g:\migrations\results\MailboxMove-$(Get-Date -format 'yyyy-MM-dd hh-mm-ss').xml"

Move-Mailbox -Identity $item -TargetDatabase $database -GlobalCatalog $targetGC -SourceForestGlobalCatalog $sourceGC -SourceForestCredential $SourceCredential -TargetForestCredential $TargetCredential -confirm:$False -RetryInterval 00:00:30 -BadItemLimit 50000 -IgnorePolicyMatch -AllowMerge -ReportFile $report

}

It is very, very simple. We create variables for the credentials and the Domain Controllers and allow the target database to be entered as a string so the execution of the migration looks something like this:

./migrationscript -textfile "C:\Group1.txt" -database "SERVERA\Storage Group 01\Database-01"

So let me explain a few of the details in this script. First, we force the retry interval to 30 seconds instead of the default 60. This is important since there is a delay between the time you write the object in AD and when the target Exchange server acknowledges the write. Also, there is a bug in the Move-Mailbox script that reports “Failed to set basic mailbox information, will retry in 60 seconds.”

You are more likely to see this message when performing cross-forest migrations:

“Failed to set basic mailbox information, will retry in 60 seconds”

Microsoft should rename this function to “Waiting” instead of “Failed” and you should just consider this 30 seconds part of the migration and move on!

Second, we set the BadIemLimit to a high number but we have NEVER seen a SINGLE item get dropped. Lastly, we added the IgnorePolicyMatch and -AllowMerge in order to meet our own goals.

TimeSaver-Make sure all of the target Exchange 2007 servers only have one Storage Group and one Mail Store. In bulk migrations, we found that roughly 5% of the migrations resulted in a (complete) disconnected mailbox in the designated target store and an empty connected mailbox in a completely different store. It seems that at some point during the end of a mailbox migration, the target server cannot enable the mailbox and Move-Mailbox creates a new empty mailbox on the same server on a different store. No error is flagged and the only way to detect this was to write a script:

get-mailboxserver | where {$_.name -like "SERVERNAME*"} |Get-mailboxStatistics | where {$_.DisconnectDate -notlike "" -and $_.Displayname -notlike "*test*"} | sort LastLogonTime | ft DisplayName, LastLogonTime, Database -wrap

This script is pretty simply as it is only looking to see if there are mailboxes that are in a disconnected state. This will be the case if the mailbox has been moved to another database or server or if the mailbox suffered the “split” problem as described.  However, by targeting a server with only a single database you eliminate this problem and have no need for my clever script.

Myth 2:

You must use 3rd party tools to automate the Outlook profile changes

FALSE

This is even more false than the first item since Outlook 2007 will correct itself automatically! Yes, you read that correctly. Outlook 2007 will sense the change and use the Autodiscover feature to find the target AD and automatically reset the Exchange server connection settings. For Outlook 2003 clients you can use Microsoft’s Exchange Server Profile Redirector Tool which for us has a 90-95% success rate.

The profile redirector can be easily deployed from a logon script. You can place the redirector files in the netlogon share and execute it from a logon script like this:

%logonserver%\netlogon\exprofre.exe /targetgc= DC2.targetcompany.com /n /v

You may notice that we are not using many of the switches here. That is by design.  By not adding the /F switch we are removing Outlook Favorites. By omitting the /A switch Outlook must download a new copy of the address book.  Since we omitted /O the OST file will be renamed instead of deleted. If you have strange problems with Outlook after test migrations you may want to add the /O switch in order to nuke OST files as they can be a problem. We left the/N switch in order to clear the nickname cache.

ExProfre is pretty sophisticated since it only makes changes when it detects the original mailbox is gone (converted to a Mail-Enabled object for example) and there is an entry in the target domain for the user.

Here is the link to the tool:

http://www.microsoft.com/downloads/details.aspx?FamilyId=56F45AC3-448F-4CCC-9BD5-B6B52C13B29C&displaylang=en

TimeSaver- Outlook problems will represent 5-10% of your help desk calls and the default fix for nearly all Outlook problems is to create a new fresh profile.

Myth 3:

Delegates and customized Mailbox Permissions are lost- FALSE

FALSE

This is false since the source rights on the mailbox will come over as part of the Move-Mailbox process. In cross-forest migrations the original AD accounts in ForestA can be used to access the mailbox in ForestB. This behavior is supported by default by Move-Mailbox but not always desired. If for example, you plan for the users to begin using accounts in ForestB to access mailboxes in ForestB then the old Access Control Entries for ForestA could create some problems. We found that the legacy credentials may work for accessing the mailbox in ForestB but other Exchange functions in the new Forest did not work with the old credentials. This problem can be overcome by nesting certain Forest groups into each other for a true Forest trust or you can simply write a script to remove the old ACLS from the new mailbox. Here is an example of that code:

Get-Mailbox -Server "TARGETSERVER" | Get-ADPermission | where { ($_.IsInherited -eq $false) -and ($_.User -like “LEGACYDOMAIN\*") } | Remove-ADPermission -confirm:$false

Get-Mailbox -Server "TARGETSERVER" | Get-MailboxPermission | where { ($_.IsInherited -eq $false) -and ($_.User -like "LEGACYDOMAIN\*") } | Remove-MailboxPermission -confirm:$false

The permissions are split between ADPermission and MailboxPermission so you must run two commands to remove them completely. Moreover, there is no -server option with Get-ADPermission or Get-Mailboxpermission so you have to first enumerate the object using Get-Mailbox. Once you have the users for a particular server you can pipe the results to get the permissions limited by the ACLS that contain the domain name you wish to remove. You can then pipe the results to remove the permissions.

It is also important to note that Delegates will also come over but remember that with Outlook, the X.500 address is used behind the scenes to link users with mailboxes. So for this to work, you need to copy the LegacyExchangeDN value from each mailbox in the source domain and populate the migrated target object with a matching X500 proxy address. This will ensure that the delegate remains linked with the appropriate user. Here is a Microsoft article that explains the process in a little more detail:

http://support.microsoft.com/kb/555197

The Move-Mailbox command should take care of this by itself, but it would be a good idea to write a script to collect the attribute then report on it after a group has been migrated just to make sure things are set as they should be. We don’t want end-user complaints to be our indicator that a directly entry is wrong!

Myth 4:

Cross-Forest Migrations are too complex and time-consuming-FALSE

FALSE

Well, I say False but let me clarify. Yes, they are complex but they are manageable. Yes they are time-consuming but you can spend most of the time upfront in preparation and keep the actual migrations to a minimum. Here are some of the things you can do to make the process easier.

1)      Try to minimize expectations for the migrations. I usually send an email to the migration team and management that sets the expectations a little lower than we can deliver. For the most part, the migration will go far smoother than this message suggests but it sets the expectations to something we know we can deliver:

  • For the first week please choose recipients from the Global Address List instead of typing their name or using reply.
  • PST should be identified before the migration as the Outlook profile may “forget” about them even though they have not been moved or deleted.
  • The migration cannot move corrupt or damaged Outlook items. Our target is to move 99.9% of the mailbox items and provide a report when a corrupt or otherwise unmovable item is found.
  • Outlook may take a long time after the migration to recreate its offline cache (OST)
  • Many customized settings in Outlook may be gone
  • Delegates will need to be setup again
  • ny customized Outlook rules will need to be setup again
  • If they have SmartPhones configured, they will no longer work
  • You may get notifications for meetings that have already passed or ones you have already dismissed.

2)      To make the transition smoother, I would highly recommend the installation of the Microsoft Exchange Server Inter-Organization Replication tool. This tool will provide Free/Busy information across the two organizations and it will set the environment up to replicate other Public Folders if necessary. This tool is probably the easiest tool to setup and will provide the most value with the least amount of overhead. I usually install the tool on a Public Folder server in the Target organization and the publisher on a Public Folder server in the source organization.  The link to this free tool is here: http://www.microsoft.com/downloads/thankyou.aspx?familyId=e7a951d7-1559-4f8f-b400-488b0c52430e&displayLang=en. Download the tool and expand to get the setup instructions. Once setup, this tool has never failed me.

3)      Move Workgroups at a time. Coexistence is by far the biggest point of confusion. “Have I been moved?” “Why does his/her email look differently than mine?” Moreover, when you move a workgroup together they become a support system for each other in the event that something does not go smoothly. When choosing who to move when, if you focus on business groups as the primary differentiator you will reduce helpdesk calls and overall confusion.

4)      Once you begin the migration, you should drive the migration to a conclusion. Every day you maintain a split organization you run an overly complex organization. Moreover, if your organization is not using automation to keep the directories synchronized every day that passes opens the door for more directly conflicts as people are added, removed or changed. You must minimize the amount of time you are coexisting on multiple platforms or in this case multiple Exchange organizations.

Myth 5:  

You must hire expensive Subject Matter Experts for your migration-False

FALSE

This is absolutely false if you have some bright folks on your team who understand Powershell and Directory updates.  Having been hired to do many of these types of projects, I can say that I am usually only involved in the first 20-30% of the moves.  So someone like me is often involved in the beginning phases to get the migrations teams quickly ramped up and the process defined and refined so it is easy to repeat.

Most organizations just don’t have the time to (self) ramp up and continue to perform their day to day operations so bringing in an outside person/group to kick things off is pretty common but certainly not a requirement.

For example, unless you have done a considerable number of Cross-forest migrations, you may not truly appreciate the negative impact WAN links and remote Domain Controllers have on the process. Intra-Org moves say between two sites in the same organization works perfectly and rather fast. In a Cross-Organizational move however, the performance of the migration can be painfully slow and AD replication will offer considerable delays and even potential problems with conflicts. Moreover, targeting Exchange servers in various sites and locations means you are never really sure how fast the process will be and your projections will likely be WAY off.

Understanding those potential obstacles up front means you can plan around it and put into place a very consistent, reliable and predictable process. Here is one example of a process we have refined with experience:

CrossImage2One thing  we have learned with this model is we do not have to change the migration scripts to target different DCs and servers and most importantly we know exactly what our migration capacity is and can hit our projected numbers every single time with no surprises. As I mentioned before, it is also important that the target Exchange servers have only one storage group and one mail store. This will eliminate a potential problem with mailboxes that may “split” across stores.

To move people to remote servers after this move, you would just use the normal Move-Mailbox command or even the GUI to distribute the mailboxes. This process is reliable and a simple “Fire and Forget” operation since you can just queue them up before you go to bed at night!

 

Move-Mailbox “Failed to set basic mailbox information, will retry in 60 seconds”

Having done cross-forest migrations since the Exchange 5.5 days, let me say that they are complex, there are many things that can go wrong and that it is almost ALWAYS cheaper to do it without expensive tools as long as you are careful and do extensive testing.

First, let me start off with one of the first thing you will likely see in your tests:

“Failed to set basic mailbox information, will retry in 60 seconds”

This error is recoverable. In fact, it is really just a warning that it took a little longer to stage the target Exchange Mailbox.

img6

So what is happening is Move-Mailbox is in the process of Mailbox-enabling the target AD account and it was unable to confirm the creation. The tool is designed to wait a while and then confirm its existence again. Now, here is where it gets strange….if you search enough you will find the Move-Mailbox command line reference on TechNet.

http://technet.microsoft.com/en-us/library/aa997599.aspx

This article includes the following details on Retry Interval:

RetryInterval Optional Microsoft.Exchange.Data.EnhancedTimeSpan The RetryInterval parameter specifies the interval for retrieving the move’s status from the server.

Unfortunately, Microsoft leaves out two very important pieces of information:

1)      The Syntax is 00:00:00 (Hours:Minutes:Seconds)

2)      No matter what you use as a valid, the status screen will continue to show a 60 second delay ““Failed to set basic mailbox information, will retry in 60 seconds”

For Syntax and testing, I would recommend you start with 00:00:30 which is essentially half the default period. Your Move-Mailbox Script should look something like this:

Move-Mailbox -identity $item -TargetDatabase $TargetExchange -GlobalCatalog $GC -SourceForestGlobalCatalog $SGC -NTAccountOU $OU -SourceForestCredential $SourceCredential -TargetForestCredential $TargetCredential -confirm:$False -BadItemLimit 0 -RetryInterval 00:00:45 -SourceMailboxCleanupOptions MailEnableSourceAccount -ReportFile $report

Now, here is where things get fun. After you add the RetryInterval 00:00:45 the status screen will still say “will retry for 60 seconds” but if you time it you will see that the value you entered is honored.

In the migrations I have done, I have found that 30-45 seconds seem to work best for me. Also, I have also found that while Move-Mailbox tends to do a pretty good job at moving messages, it does not seem to be as robust when working with the AD. The performance of the DC, the proximity to the script execution and other less-measurable factors can easily trip this up in a cross-forest migration. I recommend that you push as much of the AD tasks outside of Move-Mailbox in order to increase the effectiveness of your migration. In short, try to do the Account Setup and Account cleanup tasks OUTSIDE of Move-Mailbox.

I have some scripts, a project overview and other helpful suggestions for a Cross-Forest move of mailboxes. Stay tuned!

 

SmartPhones and the Global Address List

Microsoft has released a small piece of software that allows mobile devices to access the Global Address List within your Exchange environment. While ActiveSync is the agent most think of when SmartPhones are mentioned, this particular add-on actually leverages the Public Virtual Directory in IIS and not the ActiveSync agent. In this article, I will show you the program’s features and what you need to do to get it working in your environment. There are a few assumptions here; you are using ActiveSync to keep Pocket Outlook up to date with your Exchange Server 2003 mailbox and you have network connectivity to a Front End server or your Mailbox Server/Front End Server. Also, for this tool to work, the server you connect to must have the /Public virtual directory loaded in IIS. (It is there by default)

If you have read my blog, then you already know that I had a boating mishap during Spring Break. My MPX220, Garmin GPSV and my pride got dunked in salt water and damaged. The next week was spent discovering all the new features of Windows Mobile 5 on my brand new Cingular 8125 (HTC Wizard). Two irritating things I found right away was that my new $500 phone did not have the new AKU 2.0 up-to-date features and Windows Mobile 5 still has no access into the Global Address List. After days of searching, I found that Cingular may one day post an update for AKU 2.0 and Microsoft has an excellent add-on for mobile devices called Microsoft Global Contact Access.

Microsoft Global Contact Access

There are two flavors of this application. One is roughly 400K and is designed for the smaller SmartPhone screens and the other is 700K and is better suited for Pocket PCs. The Samsung i700, the Palm Treo 700W and Cingular 8125 devices are technically both, but you would be encouraged to use Pocket PC applications for the most part since those are formatted for the larger screens and usually comes with a few more features. The download locations for each are located on the downloads page of the Windows Mobile add-on site:

http://www.microsoft.com/windowsmobile/downloads/global/default.mspxnewtools

Installation is easy as you need only to run the setup on your machine and let ActiveSync install the application. For you propeller heads, you can still just copy the CAB file to the device and launch it to install the application.

Once installed, you should notice three additional applications in the Start Men
u; Find Contact Online, New Email and New Meeting. If you are running the SmartPhone version you will not get the New Emailapplication.

New Meeting

Since the names are all self-explanatory, let’s just go over the basics first.

newmeetingNew Meeting fires up a meeting request pane. If you know the SMTP address of the attendees you wish to invite then you need only to key their addresses into the Attendees box. Remember to separate the names with a semi-colon.

If you want to choose these users from the GAL, then use the Find Contacts Online option from the Options selection at the bottom of the screen.

The Find Contact application is then launched. findcontactKey in the name of the person you wish to find and click Find to begin the launch. After the lookup is complete, you should see the results. Scroll down to choose the correct contact and click Done when you found the right one.

Note: If you have not configured the logon credentials for these new tools, you should then get prompted to enter your password and potentially the domain name, user name, server name, etc. These settings should match what you have already entered for the ActiveSync components.

freebusyNow things start to get really interesting. Now that you have selected all the attendees, the meeting time, subject line, notes, etc you can check the group’s free-busy data. (Yeah, you heard me right)

 

 

 

How Find Contact Works

As I mentioned before, these tools to not currently leverage ActiveSync. In my larger SmartPhone deployments, I have created new Virtual Servers in IIS (on the Front Ends) to support Active-Sync and NAT’ed these IPs and Virtual servers from the outside using only port 443. Of course Network Load Balancing is much better, but in some situations I can’t use it.

SmartPhonelockdownWhat you have with this design is the minimum footprint required to support ActiveSync synchronization over the wire. Unfortunately, this configuration is so locked-down; it will not allow the Find Contact features to work! Here is why:

ActiveSync uses GETS, POSTS and OPENS to synchronize against the /Microsoft-Server-ActiveSync application that is loaded on the Exchange 2003 Servers:

POST, /Microsoft-Server-ActiveSync, User=STEVEBRYANT&DeviceId=200687CAB5517E14783A6C62D31D4DC1&DeviceType=PocketPC&Cmd=GetItemEstimate&Log=V4TNASNC:0A0C0D0FS:0A0C0D0SP:1C7I5801S74670R0S0L0H0P

So my locked down configuration works just peachy. Unfortunately, the Find Contact function must access the /Public virtual directory since that is where Free/Busy information is kept:

GET, /public/, Cmd=freebusy&start=2006-04-18T00:00:00-04:00&end=2006-04-19T00:00:00-04:00&interval=30&u=SMTP:Jason%2eSherry%40theproexchange%2ecom&u=SMTP:Steve%2eA%2eBryant%40theproexchange%2ecom

To ensure these online GAL-lookup features work, you will need to make sure the /Public virtual directory is loaded.

This does not mean that you need any of the OWA tools installed though. Mobile ActiveSync and the GAL Lookup tools will work just fine using minimal components in IIS. Loading the Public virtual directory will provide support for the necessary Cmd=freebusy and Cmd=galfind commands as the Find Contact application does not use the web controls needed for OWA.

New locked down design

What we have learned is that these new features are very important to Windows Mobile users so your design should allow for access.

newSmartPhonelockdown

As I mentioned before, this design would be far more sophisticated with Network Load Balancing on the Front End Servers and some type of reverse-proxy server such as ISA 2004 or Firepass F5 between the Internet clients and the Front End servers. It is also important to note that the ONLY port that should be opened is 443.