Details
Description
Add the ability to spool CiviMail outbound emails to a table (instead of sending them through SMTP) by creating a virtual mailer that can be hooked up to the current infrastructure ($mailer->send() code).
Create a civicrm_mailing_spool table with the following columns: id (auto increment, PK), domain_id (FK to civicrm_domain), job_id (FK to civicrm_mailing_job), recipient_email (text), headers (text), body (text), added_at (datetime), removed_at (datetime, default NULL).
'Unfold' the headers into a string before putting them into the table (they're sent to the send() function as a table).
Extend the send() function so that it can receive the additional parameter (job_id) if its provided (use this parameter in the spooler mailer, while ignoring it in the SMTP mailer).
Add a CiviMail config option (and a UI for it) that will tell CiviMail whether to use the classic, SMTP mailer (mailings of size below the value) or the table spooler mailer (mailings of the given size and bigger). Make CiviMail honour this option and call the right mailer, depending on the mailing size.
Fire a Drupal hook event after successfully filling the table, so a third-party Drupal module can be notified that there are new emails to be taken care of (i.e., after each generation of BATCH_SIZE number of emails). This hook should also have a flag that says whether the given filling of the spool table is the last one for a given mailing (i.e., whether we marked the job as completed on this run or not).
Note: you have to make deliver() exit more cleanly for this; we now simply halt hard when reaching the limit, and we don't consider a job complete just because we never reach the end of the runJobs().
Note: runJobs() runs all the pending jobs in a loop, but if a BATCH_SIZE limit is set, we want it to consider all the currently ran jobs in total - i.e., we should count if we reached the limit, break out of deliver() if we do, but not try to run the next job with the counter reset - reaching the limit should make all the other pending jobs not run (hence it's based on a static variable).
Add the documentation that the third party modules can either delete the mails they took care of or fill in the removed_at datetime field instead (this could lead to this table growing fast, but I can see some uses for this in smaller mailings).
Consider enhancing the mailings report screen to handle mailings that are sent from the point of view of civicrm_mailing, but are actually still stuck in civicrm_mailing_spool (i.e., there are entries with the relevant job_id and removed_at == NULL). These emails are out of the system from CiviMail's point of view, but not really delivered yet, so ideally shouldn't be displayed as such.