Transferring IMAP e-mail

One of the biggest issues in transferring web-host is often that all your e-mails (sometimes many years worth) are on the original host's servers. You should, of course, have back-ups if you want to make sure they don't get lost, but many folk do not.

imapsync is a superb piece of software to use to migrate your accounts. Assuming you wish to use the same user names and passwords, each account in turn can be migrated using a script similar to the following:

imapsync --host1 <old imap host> --user1 $1@<domain name> -ssl1 --passfile1 $1.pwd \
    --host2 <new imap host> --user2 $1@<domain name> --ssl2 --passfile2 $1.pwd 

If we call this script, we can then synchronise each user from the old account to the new using

syncuser <username>

Note that it is necessary to create a text file <username>.pwd containing the password of each user before we can run the script.

The users can be sync'd several times before the actual migration to make sure that data is being correctly transferred. DO NOT, however, attempt to use the new account for real until your have completed that transfer, as imapsync is not designed to be used when both accounts are being updated externally.


Now, if you want to move to Google Apps, or another provider that uses Google e-mail infrastructure, it is rather more complicated because Google use an unusual folder structure.

Moving standard folders over to Google Apps using imapsync is pretty straightforward and can be done without any special consideration:

 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN \
    --port2 993 --ssl2

Just running the command as above doesn't account for any naming differences between the old imap server and Gmail, this is easily taken care of with the --regextrans2 option which allows regular expressions to be applied to folder names as they are copied.

The old sever kept it's sent mail in a folder called "Sent Items", so a first attempt at renaming was:

 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN \
    --port2 993 --ssl2 --regextrans2 's/Inbox\\/Sent$/Sent Mail/'

This unfortunately led to a label being created in the Gmail web interface called "Imap/Sent Mail" - not the desired result. The mail should end up in the actual "Sent Mail" folder.

Sent Mail not Imap/Sent Mail

Having a look at the gmail account in Thunderbird, the standard gmail folders show up under a parent folder called [Google Mail]. Armed with this information a quick google search reveals a few posts about using the --prefix2 option to get the mail to sync into the correct place - it basically boils down to putting the existing sent mail into a folder called "[Google Mail]/Sent Mail". Because of needing to use the --prefix option the sync of the mail was split into two passes.

Two pass email syncing

  1. Sync over all the folders in your current imap account, but make sure you exclude your sent, draft, junk and deleted items
  2. Sync over the sent mail, drafts and if needed deleted items (you may have somebody who uses their deleted items folder as a filing system - beware!)

So the two pass solution becomes:

 #transfer the standard folders, skipping sent, drafts, deleted and spam
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN \
    --port2 993 --ssl2 --exclude "Sent Items|Deleted Items|Drafts|Spam" \
 #transfer the [Google Mail] folders
 imapsync --host1 imap.oldmail --user1 $user1 --passfile1 pass1.txt --authmech1 PLAIN \
    --host2 --user2 $user2 --passfile2 pass2.txt --authmech2 LOGIN \
    --port2 993 --ssl2 --folder "Inbox/Sent Items" --prefix2 '[Google Mail]/' \
    --regextrans2 's/Inbox\\/Sent\\ Items$/Sent Mail/' --folder "Inbox/Deleted Items" \
    --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\\/Deleted\\ Items$/Bin/' \
    --folder "Inbox/Drafts" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\\/Drafts$/Drafts/' \
 #add the following to the lines above if you want to keep your spam!
     --folder "Inbox/Spam" --prefix2 '[Google Mail]/' --regextrans2 's/Inbox\\/Spam$/Spam/' \\

A localisation gotcha

One word of caution - the "[Google Mail]" part as well as the actual folder names will change depending on the language set for the account. With the language set to US English (the default) rather than British English the folder prefix was [GMail] and some of the folder names were different e.g. Trash rather than Bin.

Incremental Syncing

As full migration of all accounts for a large domain may take a few weeks, it will be typically necessary to run the scripts multiple times. One by one users can be switched over to Google Apps. imapsync really comes into it's own at this point as it can be run multiple times and will not re-transfer email. The --skipsize option is needed if you re-run imapsync otherwise due to the way the Google imap works the messages will not be regarded as the same and will be re-transferred.

Thanks to Mark Seagrief for the article on which some of the above is based