CiviMail does not generate Message-ID header; result is mail getting flagged as spam

Mail sent by CiviMail does not include the Message-ID: header.

While RFC822 does not require Message-ID, in practice, not having Message-ID tends to trigger the "MSGID_FROM_MTA_ID" rule in SpamAssassin, which causes mail to be flagged as spam.

Somebody advise me about where to attack this, architecturally, and I'll proceed to fix.

There are two main ways that email is generated by CiviCRM,
(1) CiviMail - including "reaction" emails and
(2) ad-hoc email which should be generated by CRM_Utils_Mail::send.

In the case of CiviMail, there is already a basis for unique message IDs - the combination of job ID, event queue ID, and hash - which is used for verp routing of replies to the messages produced by a mailing run (actually the event queue record ID should be sufficient? - but apparently some anti-spam rules are suspicious of short message IDs). This is almost but not quite sufficient, as there are automatic responses to some incoming message types such as unsubscribes. So in all cases the mailing-related message IDs have a single character prefix and as far as possible these correspond to the verp prefixes of the original message.

m - message
a - autorespond
r - reply (as per verp)
e - resubscribe (as per verp)
s - subscribe
u - unsubscribe (as per verp)

For ad-hoc messages the Message ID is constructed from a civicrm_ prefix and a long UUID generated by PHP uniqid.

It turns out there are quite a few places where the PEAR mailer is called directly, so it is not enough to add the Message ID in CRM_Utils_Mail::send and CRM_Mailing_BAO_Mailing::getVerpAndUrlsAndHeaders.

The non-mailing related ad-hoc message sending has been changed to call CRM_Utils_Mail::send. In all of these cases error messages produced by the PEAR library are discarded. There will be a change in behaviour because CRM_Utils_Mail::send logs the error message and emits it for the user. Since a batch of calls to CRM_Utils_Mail::send might be made, it now uses CRM_Core_Session::setStatus in 'append' mode.

In the case of mailings, a new static method addMessageIdHeader is provided in CRM_Mailing_BAO_Mailing - this takes an array of headers by reference, and if there is no Message-ID header it will construct a mailing-style message ID based on prefix, job ID, queue ID and hash received as parameters, and add an appropriate entry to the array of headers.

This behaviour is controlled by a flag includeMessageIdHeader that can be set in the Mail Settings. UI changes are not included here, but CRM_Core_BAO_MailSettings has had a suitable property added, and this is initialised to true in the constructor CRM_Core_DAO_MailSettings (this is on the assumption that the underlying DB table will be modified when the UI changes are made).

Since this exercise started by making the setting part of the CiviMail settings, and Message ID should default to true, CRM_Admin_Form_Preferences was changed to allow the definition of a default value in the descendant classes.