CRM-15499 courier imap server responds "max atom size too small" if more than 5,000 bounce messages exist

    Details

    • Type: Bug
    • Status: Done/Fixed
    • Priority: Trivial
    • Resolution: Fixed/Completed
    • Affects Version/s: 4.5
    • Fix Version/s: 4.7.9
    • Component/s: CiviMail
    • Labels:
    • Documentation Required?:
      None
    • Funding Source:
      Core Team Funds

      Description

      According to the courier developer (http://markmail.org/message/63n2ukjznkcqsit7#query:+page:1+mid:j4x2z7j4ymytvjbf+state:results) this is the fault of the imap client.

      In particular, the ezc client is trying to fetch the size of all the messages in the inbox, which requires a query to the imap server that exceeds the 16K limit of courier.

      Although you normally won't get this many bounce messages - it is possible if:

      • It's your first mailing and you have a large list
      • Your bounce processor was broken for a while and you finally fixed it

      I've cobbled together the follow hackish fix:

      --- packages/ezc/Mail/src/transports/imap/imap_transport.php.orig	2014-10-20 13:26:13.000000000 -0400
      +++ packages/ezc/Mail/src/transports/imap/imap_transport.php	2014-10-20 13:18:52.000000000 -0400
      @@ -986,7 +986,8 @@
               {
                   // get the sizes of the messages
                   $tag = $this->getNextTag();
      -            $query = trim( implode( ',', $messageList ) );
      +            $truncatedMessageList = array_slice($messageList, 0, MAIL_BATCH_SIZE);
      +            $query = trim( implode( ',', $truncatedMessageList ) );
                   $this->connection->sendData( "{$tag} FETCH {$query} RFC822.SIZE" );
                   $response = $this->getResponse( 'FETCH (' );
                   $currentMessage = trim( reset( $messageList ) );
      

      However... it alters the ezc package to depend on a constant set in CiviCRRM.

      I'm also not sure if it has any other implications.

      I'm wondering if there are other suggestions for how to approach this, which could include exploring a different IMAP library or submitting a bug upstream (https://github.com/zetacomponents/Mail/).

        Attachments

          Activity

          [CRM-15499] courier imap server responds "max atom size too small" if more than 5,000 bounce messages exist
          Tim Otten added a comment -

          I think sending a bug report upstream would be good regardless of whether we merge it here – for starters, the Zetacomponents folks probably know a lot more about the nuances of IMAP implementations than anyone on Civi's team.

          Jamie McClelland added a comment -

          Yeah... I should have done this at the start. Issue submitted upstream:

          https://github.com/zetacomponents/Mail/issues/31

          Tim Otten added a comment -

          So I guess it's been a few months without feedback from upstream.

          What if we do this:

          • Setup a fork of upstream's repo
          • Apply a patch/hack like the one above. However, instead of strictly depending on the constant, include a default - so that it's at least interoperable.

          $mailBatchSize = defined('MAIL_BATCH_SIZE') ? MAIL_BATCH_SIZE : 50;
          $truncatedMessageList = array_slice($messageList, 0, $mailBatchSize);

          • Possibly port our patch in src/parser/interfaces/part_parser.php . (The only difference between our code and upstream's 1.7 is in that file.)
          • Update our composer.json to use the new fork.
          Tim Otten added a comment -

          I changed fixVersion to 4.7 because we've been blocked on upstream indefinitely. If we decide to move ahead without them (like above), then that change is still quite welcome for 4.6.

          Tim Otten added a comment -

          So it seems like upstream is dead in the water. I've forked and applied a variant of your patch. Jamie McClelland, could you try out these changes?

          https://github.com/civicrm/civicrm-core/pull/6494
          https://github.com/civicrm/civicrm-packages/pull/124

          Kurund Jalmi added a comment - - edited

          Jamie McClelland Did you get the chance to review Tim's PR and the fixes.

          David Greenberg added a comment -

          Moving this to Future version as we've had no feedback for several months and the existing PRs are no longer merge-able.

          Tim Otten added a comment -

          Reset FixVersion to 4.7 since the PR will get merged soon.

          For the record, the proper fork of zetacomponents-mail.git is at https://github.com/civicrm/zetacomponents-mail/commits/1.7-civi

          Tim Otten added a comment -

          Reset FixVersion to 4.7 since the PR will get merged soon.

          For the record, the proper fork of zetacomponents-mail.git is at https://github.com/civicrm/zetacomponents-mail/commits/1.7-civi

          Eileen McNaughton added a comment -

          This appears to be fixed & merged (a while back) so closing

            People

            • Assignee:
              Jamie McClelland
              Reporter:
              Jamie McClelland

              Dates

              • Created:
                Updated:
                Resolved: