'Dedupe by email' not working

The 'Dedupe by email' setting when sending a mass mail isn't performing as described, and is instead allowing emails to go out multiple times to the same address.

To replicate: create multiple new contacts, each with the same email. Add them to a mailing list and send them a mail. Tick the 'Dedupe by email' box. The email address will receive X emails where X is the number of contacts.

Related issue which was marked as fixed: https://issues.civicrm.org/jira/browse/CRM-16465