Debunking the top 5 Myths concerning Cross-Forest Exchange Migrations

Exchange Cross-Forest migrations are not as impossible, expensive or complex as you may think. If you are considering merging an Exchange organization into another organization, you should know that it can be done and you can do it.

Cross organizational moves are complex and on my last large cross-org project we had nearly 100 Exchange 2007/2010 servers and over thirty locations with multiple SMTP paths. Moreover, we were dealing with two separate AD forests with absolutely no automated directory synchronization. Even with these challenges plus WAN link migrations we established a process to successfully migrate roughly 600 people in a six hour window with minimal personnel and an exceptionally low failure rate. If you do the math you will see that we built capability to migrate 2400 a day or 16,800 people per week. Since we are not running four shifts a day, seven days a week I have a few moments to talk about how can do it too.

As you read this, you will notice that I have included code samples, a few tips and some overall ideas to enforce by conviction that this can be done without expensive tools and to illustrate my points. You should not take this article as a complete migration guide but as a confidence builder. There are far more technical strategies that are better described somewhere else such as sizing, migration throughput, error handling, WAN moves, server centralization, scheduling and the overall technical aspects of the scripting and process. I have tried however to give you enough information so you can understand how manageable this process really is.

So let’s set the stage. You are tasked with planning the migration of thousands of Exchange users from one company/organization to another. You have trusts in place and accounts in each Forest with rights and you have read very little documentation that would suggest you can accomplish this on your own. Moreover, you have a quote for $500,000 worth of migration software and have no idea how you will maintain your budget or if the software is even worth it.

Myth 1:

Migrating Exchange mailboxes from one org to another without 3rd Party tools is suicide

FALSE

In my last large cross-org migration project, we moved roughly 30,000 mailboxes using the standard Exchange 2007 “Move-Mailbox” PowerShell command. The syntax is described here: http://technet.microsoft.com/en-us/library/aa997599.aspx.

Having said that let me point out that you should augment that command with additional scripts that provide additional error-handling and account management. In the end, the Move-Mailbox command is the only tool I use to migrate terabytes of Exchange information from one organization to another.  I will show you the command in a moment, but first let’s talk about how we use it:

  •  For bulk moves, we script the command against a text file that contains the names we wish to migrate. I prefer this better than using an AD group to list the migration candidates since it allows us to “lock” the group and easily manipulate the names if so desired.
  • Perform your AD work ahead of schedule. Create Mail-enabled user objects in the target domain and instruct the user community in advance as to how to change passwords and logon. You should avoid using AD Contacts and focus on Mail-enabled users in order to maintain passwords, groups and other attributes before, during and after the moves. This part of the project is critical and deserves its own section as you must maintain all X500, SMTP attributes. Moreover, it is important to cross-pollinate the LegacyExchangeDN value in one directory as an X500 address in the opposite directory for each mailbox. This will dramatically reduce and possibly eliminate reply failures and meeting ownerships.

 

CrossImage1

 

  • Use the Move-Migration script to Mailbox-enable the target object and move the mail but use an outside process to handle all account changes in the source domain. This will give you more control and reliability of the source objects.  The Move-Mailbox script can perform these functions but there is little in the way of error-handling so if the AD is not responsive or there is a connection failure during object modifications, the Move-Mailbox command does not always recover. It is super reliable as a mail migration tool and semi-reliable with its AD changes so focus on its benefits and shore-up its weaknesses.
  • Execute a series of Post Scripts to perform any additional cleanup you may require for the accounts and mailboxes. There will be plenty. You will need a script to disconnect (do not delete in case you need to reconnect later) the mailbox on the source object and to turn it in to a Mail-enabled object with all the previous addresses and mail attributes. You will need another script to compare the object to make sure it is correct.

Just to make sure my point is perfectly clear, this is the exact code we use for every migration:

Param([string]$textfile,[string]$database)

$import = Get-Content $textfile

$SourceCredential = get-credential

$TargetCredential = get-credential

$targetGC = "DC2.targetcompany.com"

$sourceGC = "DC2.sourcecompany.com"

#move the migrated user's mailbox

$report = "g:\migrations\results\MailboxMove-$(Get-Date -format 'yyyy-MM-dd hh-mm-ss').xml"

Move-Mailbox -Identity $item -TargetDatabase $database -GlobalCatalog $targetGC -SourceForestGlobalCatalog $sourceGC -SourceForestCredential $SourceCredential -TargetForestCredential $TargetCredential -confirm:$False -RetryInterval 00:00:30 -BadItemLimit 50000 -IgnorePolicyMatch -AllowMerge -ReportFile $report

}

It is very, very simple. We create variables for the credentials and the Domain Controllers and allow the target database to be entered as a string so the execution of the migration looks something like this:

./migrationscript -textfile "C:\Group1.txt" -database "SERVERA\Storage Group 01\Database-01"

So let me explain a few of the details in this script. First, we force the retry interval to 30 seconds instead of the default 60. This is important since there is a delay between the time you write the object in AD and when the target Exchange server acknowledges the write. Also, there is a bug in the Move-Mailbox script that reports “Failed to set basic mailbox information, will retry in 60 seconds.”

You are more likely to see this message when performing cross-forest migrations:

“Failed to set basic mailbox information, will retry in 60 seconds”

Microsoft should rename this function to “Waiting” instead of “Failed” and you should just consider this 30 seconds part of the migration and move on!

Second, we set the BadIemLimit to a high number but we have NEVER seen a SINGLE item get dropped. Lastly, we added the IgnorePolicyMatch and -AllowMerge in order to meet our own goals.

TimeSaver-Make sure all of the target Exchange 2007 servers only have one Storage Group and one Mail Store. In bulk migrations, we found that roughly 5% of the migrations resulted in a (complete) disconnected mailbox in the designated target store and an empty connected mailbox in a completely different store. It seems that at some point during the end of a mailbox migration, the target server cannot enable the mailbox and Move-Mailbox creates a new empty mailbox on the same server on a different store. No error is flagged and the only way to detect this was to write a script:

get-mailboxserver | where {$_.name -like "SERVERNAME*"} |Get-mailboxStatistics | where {$_.DisconnectDate -notlike "" -and $_.Displayname -notlike "*test*"} | sort LastLogonTime | ft DisplayName, LastLogonTime, Database -wrap

This script is pretty simply as it is only looking to see if there are mailboxes that are in a disconnected state. This will be the case if the mailbox has been moved to another database or server or if the mailbox suffered the “split” problem as described.  However, by targeting a server with only a single database you eliminate this problem and have no need for my clever script.

Myth 2:

You must use 3rd party tools to automate the Outlook profile changes

FALSE

This is even more false than the first item since Outlook 2007 will correct itself automatically! Yes, you read that correctly. Outlook 2007 will sense the change and use the Autodiscover feature to find the target AD and automatically reset the Exchange server connection settings. For Outlook 2003 clients you can use Microsoft’s Exchange Server Profile Redirector Tool which for us has a 90-95% success rate.

The profile redirector can be easily deployed from a logon script. You can place the redirector files in the netlogon share and execute it from a logon script like this:

%logonserver%\netlogon\exprofre.exe /targetgc= DC2.targetcompany.com /n /v

You may notice that we are not using many of the switches here. That is by design.  By not adding the /F switch we are removing Outlook Favorites. By omitting the /A switch Outlook must download a new copy of the address book.  Since we omitted /O the OST file will be renamed instead of deleted. If you have strange problems with Outlook after test migrations you may want to add the /O switch in order to nuke OST files as they can be a problem. We left the/N switch in order to clear the nickname cache.

ExProfre is pretty sophisticated since it only makes changes when it detects the original mailbox is gone (converted to a Mail-Enabled object for example) and there is an entry in the target domain for the user.

Here is the link to the tool:

http://www.microsoft.com/downloads/details.aspx?FamilyId=56F45AC3-448F-4CCC-9BD5-B6B52C13B29C&displaylang=en

TimeSaver- Outlook problems will represent 5-10% of your help desk calls and the default fix for nearly all Outlook problems is to create a new fresh profile.

Myth 3:

Delegates and customized Mailbox Permissions are lost- FALSE

FALSE

This is false since the source rights on the mailbox will come over as part of the Move-Mailbox process. In cross-forest migrations the original AD accounts in ForestA can be used to access the mailbox in ForestB. This behavior is supported by default by Move-Mailbox but not always desired. If for example, you plan for the users to begin using accounts in ForestB to access mailboxes in ForestB then the old Access Control Entries for ForestA could create some problems. We found that the legacy credentials may work for accessing the mailbox in ForestB but other Exchange functions in the new Forest did not work with the old credentials. This problem can be overcome by nesting certain Forest groups into each other for a true Forest trust or you can simply write a script to remove the old ACLS from the new mailbox. Here is an example of that code:

Get-Mailbox -Server "TARGETSERVER" | Get-ADPermission | where { ($_.IsInherited -eq $false) -and ($_.User -like “LEGACYDOMAIN\*") } | Remove-ADPermission -confirm:$false

Get-Mailbox -Server "TARGETSERVER" | Get-MailboxPermission | where { ($_.IsInherited -eq $false) -and ($_.User -like "LEGACYDOMAIN\*") } | Remove-MailboxPermission -confirm:$false

The permissions are split between ADPermission and MailboxPermission so you must run two commands to remove them completely. Moreover, there is no -server option with Get-ADPermission or Get-Mailboxpermission so you have to first enumerate the object using Get-Mailbox. Once you have the users for a particular server you can pipe the results to get the permissions limited by the ACLS that contain the domain name you wish to remove. You can then pipe the results to remove the permissions.

It is also important to note that Delegates will also come over but remember that with Outlook, the X.500 address is used behind the scenes to link users with mailboxes. So for this to work, you need to copy the LegacyExchangeDN value from each mailbox in the source domain and populate the migrated target object with a matching X500 proxy address. This will ensure that the delegate remains linked with the appropriate user. Here is a Microsoft article that explains the process in a little more detail:

http://support.microsoft.com/kb/555197

The Move-Mailbox command should take care of this by itself, but it would be a good idea to write a script to collect the attribute then report on it after a group has been migrated just to make sure things are set as they should be. We don’t want end-user complaints to be our indicator that a directly entry is wrong!

Myth 4:

Cross-Forest Migrations are too complex and time-consuming-FALSE

FALSE

Well, I say False but let me clarify. Yes, they are complex but they are manageable. Yes they are time-consuming but you can spend most of the time upfront in preparation and keep the actual migrations to a minimum. Here are some of the things you can do to make the process easier.

1)      Try to minimize expectations for the migrations. I usually send an email to the migration team and management that sets the expectations a little lower than we can deliver. For the most part, the migration will go far smoother than this message suggests but it sets the expectations to something we know we can deliver:

  • For the first week please choose recipients from the Global Address List instead of typing their name or using reply.
  • PST should be identified before the migration as the Outlook profile may “forget” about them even though they have not been moved or deleted.
  • The migration cannot move corrupt or damaged Outlook items. Our target is to move 99.9% of the mailbox items and provide a report when a corrupt or otherwise unmovable item is found.
  • Outlook may take a long time after the migration to recreate its offline cache (OST)
  • Many customized settings in Outlook may be gone
  • Delegates will need to be setup again
  • ny customized Outlook rules will need to be setup again
  • If they have SmartPhones configured, they will no longer work
  • You may get notifications for meetings that have already passed or ones you have already dismissed.

2)      To make the transition smoother, I would highly recommend the installation of the Microsoft Exchange Server Inter-Organization Replication tool. This tool will provide Free/Busy information across the two organizations and it will set the environment up to replicate other Public Folders if necessary. This tool is probably the easiest tool to setup and will provide the most value with the least amount of overhead. I usually install the tool on a Public Folder server in the Target organization and the publisher on a Public Folder server in the source organization.  The link to this free tool is here: http://www.microsoft.com/downloads/thankyou.aspx?familyId=e7a951d7-1559-4f8f-b400-488b0c52430e&displayLang=en. Download the tool and expand to get the setup instructions. Once setup, this tool has never failed me.

3)      Move Workgroups at a time. Coexistence is by far the biggest point of confusion. “Have I been moved?” “Why does his/her email look differently than mine?” Moreover, when you move a workgroup together they become a support system for each other in the event that something does not go smoothly. When choosing who to move when, if you focus on business groups as the primary differentiator you will reduce helpdesk calls and overall confusion.

4)      Once you begin the migration, you should drive the migration to a conclusion. Every day you maintain a split organization you run an overly complex organization. Moreover, if your organization is not using automation to keep the directories synchronized every day that passes opens the door for more directly conflicts as people are added, removed or changed. You must minimize the amount of time you are coexisting on multiple platforms or in this case multiple Exchange organizations.

Myth 5:  

You must hire expensive Subject Matter Experts for your migration-False

FALSE

This is absolutely false if you have some bright folks on your team who understand Powershell and Directory updates.  Having been hired to do many of these types of projects, I can say that I am usually only involved in the first 20-30% of the moves.  So someone like me is often involved in the beginning phases to get the migrations teams quickly ramped up and the process defined and refined so it is easy to repeat.

Most organizations just don’t have the time to (self) ramp up and continue to perform their day to day operations so bringing in an outside person/group to kick things off is pretty common but certainly not a requirement.

For example, unless you have done a considerable number of Cross-forest migrations, you may not truly appreciate the negative impact WAN links and remote Domain Controllers have on the process. Intra-Org moves say between two sites in the same organization works perfectly and rather fast. In a Cross-Organizational move however, the performance of the migration can be painfully slow and AD replication will offer considerable delays and even potential problems with conflicts. Moreover, targeting Exchange servers in various sites and locations means you are never really sure how fast the process will be and your projections will likely be WAY off.

Understanding those potential obstacles up front means you can plan around it and put into place a very consistent, reliable and predictable process. Here is one example of a process we have refined with experience:

CrossImage2One thing  we have learned with this model is we do not have to change the migration scripts to target different DCs and servers and most importantly we know exactly what our migration capacity is and can hit our projected numbers every single time with no surprises. As I mentioned before, it is also important that the target Exchange servers have only one storage group and one mail store. This will eliminate a potential problem with mailboxes that may “split” across stores.

To move people to remote servers after this move, you would just use the normal Move-Mailbox command or even the GUI to distribute the mailboxes. This process is reliable and a simple “Fire and Forget” operation since you can just queue them up before you go to bed at night!

 

Move-Mailbox “Failed to set basic mailbox information, will retry in 60 seconds”

Having done cross-forest migrations since the Exchange 5.5 days, let me say that they are complex, there are many things that can go wrong and that it is almost ALWAYS cheaper to do it without expensive tools as long as you are careful and do extensive testing.

First, let me start off with one of the first thing you will likely see in your tests:

“Failed to set basic mailbox information, will retry in 60 seconds”

This error is recoverable. In fact, it is really just a warning that it took a little longer to stage the target Exchange Mailbox.

img6

So what is happening is Move-Mailbox is in the process of Mailbox-enabling the target AD account and it was unable to confirm the creation. The tool is designed to wait a while and then confirm its existence again. Now, here is where it gets strange….if you search enough you will find the Move-Mailbox command line reference on TechNet.

http://technet.microsoft.com/en-us/library/aa997599.aspx

This article includes the following details on Retry Interval:

RetryInterval Optional Microsoft.Exchange.Data.EnhancedTimeSpan The RetryInterval parameter specifies the interval for retrieving the move’s status from the server.

Unfortunately, Microsoft leaves out two very important pieces of information:

1)      The Syntax is 00:00:00 (Hours:Minutes:Seconds)

2)      No matter what you use as a valid, the status screen will continue to show a 60 second delay ““Failed to set basic mailbox information, will retry in 60 seconds”

For Syntax and testing, I would recommend you start with 00:00:30 which is essentially half the default period. Your Move-Mailbox Script should look something like this:

Move-Mailbox -identity $item -TargetDatabase $TargetExchange -GlobalCatalog $GC -SourceForestGlobalCatalog $SGC -NTAccountOU $OU -SourceForestCredential $SourceCredential -TargetForestCredential $TargetCredential -confirm:$False -BadItemLimit 0 -RetryInterval 00:00:45 -SourceMailboxCleanupOptions MailEnableSourceAccount -ReportFile $report

Now, here is where things get fun. After you add the RetryInterval 00:00:45 the status screen will still say “will retry for 60 seconds” but if you time it you will see that the value you entered is honored.

In the migrations I have done, I have found that 30-45 seconds seem to work best for me. Also, I have also found that while Move-Mailbox tends to do a pretty good job at moving messages, it does not seem to be as robust when working with the AD. The performance of the DC, the proximity to the script execution and other less-measurable factors can easily trip this up in a cross-forest migration. I recommend that you push as much of the AD tasks outside of Move-Mailbox in order to increase the effectiveness of your migration. In short, try to do the Account Setup and Account cleanup tasks OUTSIDE of Move-Mailbox.

I have some scripts, a project overview and other helpful suggestions for a Cross-Forest move of mailboxes. Stay tuned!