Duplicate E-mails: Prevention and clean up tips

Rules

Rules, and more specific “Move to folder” rules, is the most easiest one to get yourself some duplicates. This is because for each e-mail received, Outlook will fire all configured rules against it. The following example will show how this can lead to duplicates.

In this example there are two rules configured;

1. Apply this rule after the message arrives
   from Robert Sparnaaij
   move it to the Outlook folder
2. Apply this rule after the message arrives
   with Outlook in the subject
   move it to the Outlook folder

Now imagine that you receive an e-mail from me where I have the word Outlook in the subject; both rules with apply! So it will get moved out of the Inbox two times and thus create a duplicate.

The way to prevent this, is to add the action “stop processing more rules” to the rule. If we would add that action to the first rule and the rule applies to a received message, the action “stop processing more rules” will prevent that the second rule will be executed as well.

Keep in mind that if you have even more rules configured that should always trigger – for instance that assigns messages to a category or marks it for follow up – you’ll need to sort your rules in the correct order as well.

Also note that the “stop processing more rules” would only affect the current message that is being checked. When you’ve received multiple emails, all those emails will be verified starting with the first rule again.

Multiple Accounts

When you configure multiple accounts in Outlook, make sure that these are indeed separate accounts and not just aliases for the same mailbox.

In the case of an alias, you would basically have the same account configured twice as all emails from either alias is being received in the same mailbox.

When you also have the option configured to leave a copy on the server (POP3 account), both accounts will collect the same e-mail leaving you with duplicates. To read up on the difference between a separate mailbox and aliases see: About mailboxes, addresses and aliases.

To prevent duplicates from alias accounts (other than removing them), you must configure the alias account not to receive e-mails since they are already being received by the original account;

• Outlook 2007 and previous
  Tools-> Options-> tab: Mail Setup-> button: Send/Receive-> button: Edit…-> select the alias account-> uncheck: Receive mail items
• Outlook 2010, Outlook 2013 and Outlook 2016
  File-> Options->Advanced-> button: Send/Receive-> button: Edit…-> select the alias account-> uncheck: Receive mail items

You’d then still be able to select with which address you send out your emails.

Profile Recreation

When you recreate your mail profile – for instance when the original got corrupted or when you reinstalled your PC and configure Outlook for the first time again – and you originally had Outlook configured to leave a copy on the server (POP3 account), it will collect all the e-mails from the server again even when you have received them in a previous configuration.

This happens because the newly created profile doesn’t “know” that these messages have been received before by another profile.

For Outlook 2007 and previous, this means that when you recreate your mail profile and, during setup, you immediately configure it to use your old pst-file as the default delivery location, it will create duplicates for the messages that are already received and that are still on-line as well.

To prevent this from happening, make sure to start Outlook at least once with a clean pst-file before reconnecting the old one and setting it as the default delivery location. This way you’ll collect all the e-mails in a new pst-file.

When you are using Outlook 2010 or later, you can prevent this by reconnecting your original pst-file during the POP3 account configuration.

For Outlook 2002/XP and 2003

1. Connect to your original pst-file by File-> Open-> Outlook Data File… and once opened move all the newly received e-mails to that pst-file.
2. Close Outlook and go to Control Panel-> Mail-> button Data Files…-> button E-mail Accounts-> button Next-> at the bottom you can select the original pst-file as the default delivery location and press Finish.
3. Press the button Data Files… and remove the other pst-file. If you want to keep your computer clean press the Settings button first and write down the location of the pst-file so you can physically delete the file as well and not just the connection to it.

For Outlook 2007

1. Connect to your original pst-file by File-> Open-> Outlook Data File… and once opened move all the newly received e-mails to that pst-file.
2. Close Outlook and go to Control Panel-> Mail-> E-mail Accounts-> tab Data Files. Select the the original pst-file and press Set as Default
3. Select the other pst-file and press Remove. If you want to keep your computer clean press the Settings button first and write down the location of the pst-file so you can physically delete the file as well and not just the connection to it.
4. 
   
5. For Outlook 2010, Outlook 2013 and Outlook 2016
   Outlook 2010 and later do remember which emails have been downloaded before as it stores this information in the pst-file itself.
   Upon configuring your account, you can specify the delivery location to an already existing pst-file instead of letting Outlook create a new one for you. If you have retrieved your account settings via AutoDiscover, you can select the option to manually configure your account. This will get you to a screen where the account settings retrieved via AutoDiscover are shown and also has a section “Deliver new message to” to select your original pst-file.
   To change the delivery pst-file for your account after you have configured your account use;
   File-> Account Settings-> Account Settings-> select your e-mail account-> button Change Folder-> select New Outlook Data File… and browse to your original pst-file.

   Synchronization Tools

   Synchronization Tools are great to keep your contacts you have in Outlook and on your mobile device, like a tablet or smartphone, synchronized and up-to-date. Depending on the tool you are using, synchronizing for the first time might be tricky and could cause some duplicates or “similar” items.
   For instance, I used to store most of my mobile phone contacts by first name only and the ones in Outlook by their full name. So I had a contact named Edwin on my mobile phone and a contact named Edwin Sparnaaij (yep, trying to get my brother famous here :-D) in Outlook. When I would synchronize, I would end up with an item named Edwin and an item named Edwin Sparnaaij in both Outlook and my mobile phone since the synch tool doesn’t know these are one and the same person in real life.
   To prevent this from happening, make sure your contacts are in order before synchronizing for the first time or you’ll only duplicate the mess. Since my mobile phone was a big mess mainly because of the lack of a proper input device and contact storage properties on it (I had a Nokia 3210 at that time) getting things in order would be handier in Outlook. To make sure it doesn’t mix with your original Outlook contacts you can create a new mail profile first with a dummy account or no account configured at all (tip: call the profile Mobile Device so you can easily recognize it). Now you can safely synchronize with that mail profile and use Outlook to easily clean up your contacts on your mobile.
   If you are a control freak (I won’t be last to admit this) and want to make sure that the initial synchronization indeed doesn’t create any duplicates, move the cleaned up contacts from the Mobile Device mail profile to your original one;
   1. Start Outlook with your original mail profile.
   2. Connect to the pst-file from the Mobile Device profile by File-> Open-> Outlook Data File…
   3. Move the Contacts from the Mobile Device pst-file to your original Contacts folder.
   Now that you have created a “master copy” in Outlook make a back-up of your pst-file in case synchronization goes wrong after all. Also now that we have all contacts correctly in Outlook, we don’t want it to still end up creating duplicates because the synchronization tool isn’t “intelligent” enough to recognize the similar items (this depends on the synchronization tool you are using so I’m going for a fail safe here). As we have a master copy in Outlook we can safely delete all the contacts on the mobile device. Now reconfigure your synchronization tool to use your original mail profile and synchronize; there is just NO WAY you can end up with duplicates now! 

   Virus scanner / Firewall

   Another quite common evil which will end up creating duplicates is having a virus scanner which integrates itself with Outlook or a too strictly configured firewall. To understand this, I’ll explain a little bit how the e-mails are actually being received without a virus scanner or firewall;
   When Outlook does a Send/Receive it will make a connection with the mail server and will ask the server which e-mails it is currently storing. The mail server will respond with a list and Outlook will verify with its local list if it has collected all the e-mails already. If so; Outlook will close the connection to the mail server. If not; Outlook will request the new e-mails from the server and when it has received all the e-mails, it will close the connection with the mail server and update its own list of received e-mails.
   The above illustrates a perfect Send/Receive cycle. If for some reason the connection gets terminated unexpectedly so that Outlook cannot properly close the connection to the mail server, Outlook will consider it as a failed Send/Receive cycle and will not update its own list of received e-mails, even when some or all of them have been received already. This means that at the next Send/Receive interval, Outlook will check for new e-mails with an “outdated” list and collects the e-mails from the previous Send/Receive interval again and will not update its local list until the connection to the mail server closes properly.
   This is where the virus scanner and firewall come into the picture. The exact method depends a bit on the virus scanner solution chosen but basically when a virus scanner integrates with Outlook it will “intercept” the Send/Receive request from Outlook. So when Outlook does a Send/Receive and “thinks” it is talking to a mail server it is in fact talking to the virus scanner. This means that now the virus scanner is responsible for the connection. This means setting up the connection, requesting the list of mails stored on the server, managing the timeouts, etc… but also closing the connection properly and telling this to Outlook.
   Although this could go right of course many many times, keep in mind that you do have added a new point of failure to the Send/Receive process. This leaves you with another possibility that the connection to the mail server isn’t closed properly and already received e-mail being received again at the next Send/Receive interval.
   Therefore, and for various other reasons as well, it is another good reason to prevent your virus scan solution to integrate with Outlook. There is no direct need to integrate a virus scanner with Outlook as it is redundant anyway; it won’t add another level of security since it is one and the same tool as your on-access-scanner. Default Outlook security settings and an up-to-date virus scanner are enough to keep you safe! See the documentation of your virus scanner on how to disable its integration with Outlook. Also see: Disable virus scanner integration?
   Now that you know how the Send/Receive interval works, you can imagine that a too strictly configured firewall could terminate the connection prematurely as well. In this case, check/disable your firewall and wait two Send/Receive intervals. If the second one doesn’t cause any duplicates, you’ll know that your firewall isn’t properly configured. See the documentation of your firewall software on how to configure your firewall for e-mail retrieval with Outlook.