Using robocopy for file system migrations

The idea..

We have file system A (old) & file system B (new).  We wish to migrate the files & permissions of the OLD file system (on an old array) to the NEW file system (on a new array).

I've found that we want to copy as HIGH as we can get in the file system, so that robocopy can not only copy the directory and files..but ALSO the permissions!

In this particular implementation the old file system is called "fileserv1" and the new file system is called "fileserv2"

This particular customer's file system is only used for home drives that are managed via AD.  The customer's know this drive as their "M:/ drive."

Now, as you can imagine, the directory that holds all of these sub-user directories is quite massive; there is a folder for EVERY user.  In addition to each user's folder, the folder contains the security permissions of the individual + domain admin..we want robocopy to do all this work for us!

What is Robocopy?

Robocopy (the name is short for Robust File Copy) was introduced with the Windows Server 2003 Resource Kit and is included in all editions of Windows 7. Its many strengths include the ability to copy all NTFS file attributes and to mirror the contents of an entire folder hierarchy across local volumes or over a network. If you use the right combination of options, you can recover from interruptions such as network outages by resuming a copy operation from the point of failure after the connection is restored. 

The Robocopy syntax takes some getting used to. If you’re familiar with the standard Copy and Xcopy commands, you’ll have to unlearn their syntax and get used to Robocopy’s unconventional ways. The key difference is that Robocopy is designed to work with two directories (folders) at a time, and the file specification is a secondary parameter. In addition, there are dozens of options that can be specified as command-line switches.

Shamelessly stolen from Microsoft's TechNet...see here for the full article!

Preliminary

Once I've created the CIFS share per best practices (HIDE the ETC and the Lost & Found!), I'm going to navigate to the highest level that I can navigate to and mirror the permissions.

The reason that I've set the permissions on this to "Domain Admins" is so that when robocopy copy/creates the subdirectories..I want each directory to inherit the "Domain Admins" permission so that any user in the Domain Admins group can access/manage the files.  In addition, we do NOT want any non-Domain Admin user to have access to the "myhomefolder."  

Furthermore, where do we run the script?  It does not require much..so I'd simply pick a windows server that has access to both source/destination file systems.  Be sure to log into the server as a domain admin! 

NOTE: If you're copying in a windows environment, you're going to have to pass credentials BEFORE you can use robocopy

The syntax for that is..

net use \\server\ipc$ /user:username password

Example: "net use \\10.1.5.1\c$\Users\Bob\Desktop /user:john.doe Ilovepotatoes"

The script

:loop

robocopy C:\Users\kbarnes\Desktop\Source C:\Users\kbarnes\Desktop\Destination /MT /copyall /MIR /TEE /FFT /R:1 /W:5 /XD ~snapshot /NP /log+:C:\Users\kbarnes\Desktop\Robo.txt

echo complete

echo %date% %time%

goto loop

I feel as it is probably best to just present what I use and try to explain its functionality.  I'm using this script as an example, hense the source/destination of two folders on my computer's desktop.  In a REAL scenerio, it may look something like this:

robocopy \\<source FS>\myhomefolder \\<destination FS>\c$\cifs_home_fs\myhomefolder

Now...for the "weird" part: Robocopy switches.

Switches are the parameters that the administrator can modify to make Robocopy do what he/she wants.  It is honestly as simple as that.  My example is one that I've been using that works for me and I'll explain:

• /MT This is Robocopy's "multithread."  The default is 8 simultanous file copies.  We can use a value between 1-128 by using "/MT:32" with 32 threads as an example.
• /copyall This copies all.  Yea, no shit.  What I mean is that it also copies the security/last modified information of the file.
• /MIR This mirrors a directory tree.  Furthermore, it will copy subdirectories including empty ones and remove files that are not in the source directory.
• /TEE Writes the status output to the console window, as well as to the log file
• /FFT Assumes FAT file times (two-second precision)
• /R:1 Specifies the number of retries on failed copies. The default value of N is 1,000,000 (one million retries).  We used a read of "1" as we found that we'd rather to try and get the files in a another sweep..than to wait.
• /W:5 Specifies the wait time between retries, in seconds. The default value of N is 30 (wait time 30 seconds).  The same as the read...we'd rather not wait forever on a single file.
• /XD ~snapshot We found that there is NO reason to copy the snapshot directory.  Using /XD ~snapshot ignores the hidden snapshot directory.
• /NP This stops the progress information of the copy job.  We just found it not necessary in our uses.
• /log+ When using log+ followed by a location, we can allow the output to be sent to a text file.  The + sign says to prepend any output to the same file, versus creating a new one/replacing the old one every cycle.

Lastly, I've included some things in the script to make it loop and to output time, dates, and a string value of "complete." 

Example use of script

1. Log into windows server as a domain admin user.

2. Right-click on the .bat file and "Run as Administrator."

3. Assuming your script is set up correctly, you should see the log file pop up wherever you've indicated.

In my example, I have two folders: Source and Destination.  I've put in a couple files in the source and I want to see them pop over to the destination!

After we've run the script we can see a couple of things:  The text files have been copied and the "Date modified" has also been copied!  If we had permissions on the text files, they would have been copied over as well!

4. Review the output log

Here, we can see that the 8 files have been completed successfully and that there were no failures!

What if we run the script again...will it attempt to copy the files a second time?

Here we can see that it skipped ALL the files, as there was nothing new on the source directory!

What if we change one of the files?

Here, I simply put some random text into one of the source text files called "New Text Document - Copy.txt"

Lets run the script again...

NICE!  We can see that it skipped the 7 unmodified files but copied over the one file that was changed!  We can also see the file that was changed with an indicator that it is "Newer."

Lastly..what if we delete one of the files on the source?

 Here we can see that the file we deleted on the source is shown as "EXTRA File."  If we navigate to the destination directory "Destination" we can confirm that the deleted file on the source has been deleted on the destination.  This is made possible by the /MIR switch we used.

Yep!  It's gone!

How to perform the migration?

It is important to know that the first pass is quite time consuming.  It has to copy EVERY file/directory that we've indicated to the destination.  You can absolutely do this during business hours.  Because of the loop in the .bat file, its going to start over as soon as it finishes.  I typically like to kick it off as I'm leaving for the day so that I can come in and verify the progress.  

Here is a real world example of the first pass:

Note: This first pass took ~5 hours.  There were some skips because I did a couple tests before running the full on script.

Once we get the first pass done, I let robocopy continue to run until I'm ready to begin my migration.  The migration IS impacting.  We have to somehow quiesce writes to the old filesystem..or else we'll lose some files in the migration.  I've found that this can be achieved by making the source file system read-only (DO THIS AFTER HOURS TO SAVE YOU A HEADACHE!).  Once the file system is configured as read-only...verify.  Can you create a new file or does it bark at you?  If it barked at you..GOOD!

Once the source file system is read-only, run robocopy ONE FINAL TIME!  Since it is in read-only mode, there should be no new files after running this script.  If you're insane and want to be double-safe...run it one more time after this step.  We should see ALL skips and 0 copies.

Now the next part really depends on the environment.  I've had some environments that either use DNS or DFS (Distributed File System).  In this (and my last) file system migration it was via DNS.  If thee computers are using "FILESERVER" as the hostname to point to the source filesystem..simply update the DNS entry to make "FILESERVER" point to the IP address of the new CIFS server.

Lastly.....do NOT TURN OFF THE OLD FILE SYSTEM!  Should a user call post-migration that a file is missing/corrupt...have the old file system handy for recovery purposes (SHOULDN'T be needed..but I'm paranoid).

Good luck!