Restore process creates duplicate messages and inflates mailbox size

[ylluminate]

When attempting to restore a backup to an empty mailbox on a new server, the process appears to be creating duplicate messages and significantly inflating the mailbox size. The restore operation is also generating numerous "UID already exists" errors, despite the destination mailbox being empty at the start of the process.

What's most alarming is the the original backup now is larger than it was (twice the size for one) and I have no way to obtain a backup of the original data again - so I have to work with what I have that could now be "corrupted" via a bunch of duplicate messages.

Here's what I roughly did:

1. Create a backup of an existing mailbox using imap-backup.
2. Set up a new, empty mailbox on a different server.
3. Configure imap-backup for restoration using the following configuration:

{
  "version": "2.2",
  "accounts": [
    {
      "username": "[email protected]",
      "password": "password",
      "local_path": "/path/to/local/backup",
      "server": "new.server.ip",
      "connection_options": {
        "port": 993,
        "ssl": {
          "verify_mode": 0
        }
      },
      "multi_fetch_size": 10
    }
  ],
  "download_strategy": "delay_metadata"
}

4. Run the restore command:

   imap-backup restore [email protected] --verbose --config=/path/to/restore_config.json
   

Obviously I was expecting the restore process should create an exact copy of the original mailbox on the new server, with the same folder structure and message count, and approximately the same total size.

What I've seen:

1. The restore process generates numerous "UID already exists" errors, such as:

   W, [2024-09-04T06:48:58.013306 #42005]  WARN -- : [INBOX.old-messages] uid: 31300 (25676/98252) - append error: UID 31300 already exists
   

2. The size of the restored mailbox is significantly larger than the original backup:
   • Original local backup size: 802M
   • Size of local backup after restore attempt: 2.3G
   • Size of mailbox on new server after restore: 1.08 GB

I had attempt to restore initially and realized that the server was running low on space so I resolved that and then started the restore again. I thought that the restore process would consider what was in place after the first upload round that was cancelled, but it seems as if while it is detecting the UIDs, it is still uploading them? Further, it is also... downloading the new ones somehow? And thus doubling the local backup as well?

Would you have any thoughts about this and how I can resolve this - and even deduplicate the original now so that it get's back into a pristine condition?

[deanezra]

Instead of using imap-backup 'restore' command, I have used the 'migrate' one following the steps shown in this link to migrate my emails from one provider to new one: