CiviCRM
  1. CiviCRM
  2. CRM-4772

CiviMailProcessor silently fails when there are too many messages in an IMAP inbox

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.2.7
    • Fix Version/s: 4.3.0
    • Component/s: CiviMail
    • Labels:
      None

      Description

      When the CiviMailProcessor is run on an IMAP inbox with too many messages, it silently fails. I'm not sure where the cutoff is, but in my testing 16,567 was too many and 12,585 was OK. So it might be 16,384 only because that's the power of 2 in between those numbers. When it's too full, you get this error message back from the IMAP server (in the case of dovecot on RHEL 5.3, at least): A0007 BAD Error in IMAP command FETCH: Too long argument.

      So there are a couple of issues to address here:

      1. It shouldn't silently fail. The only reason I was able to grab that error message at all is because I spent hours tracing the breakdown and finally narrowed it down enough to insert a print statement into packages/ezc/Mail/src/transports/imap/imap_transport.php on line 980. The library is throwing an ezcMailTransportException when this occurs and putting that error message into the exception object. So ideally all that would propagate up to the stdout of the calling script so the caller sees it (and cron e-mails the error to the admin, etc.). But currently it doesn't, the CiviMailProcessor.php script just stops and no error is seen.

      2. It shouldn't fail at all. It seems like maybe this is a bug in the upstream ezc IMAP library we're using. So maybe a newer version of that would fix it? Or maybe we need to patch it for now and then submit that patch upstream? Seems like there should be a way to limit whatever argument it's sending so the size doesn't get so big the server rejects it. I don't know if dovecot is uniquely picky in that regard, but it's a pretty common IMAP server so probably it's best just to accommodate it either way.

        Activity

        Hide
        Donald A. Lobo added a comment -

        wanna investigate and submit a patch for this. seems like you probably have the right server and imap files to trigger this error :) If you can do it in the next week or so, we can release it with 3.0
        Show
        Donald A. Lobo added a comment - wanna investigate and submit a patch for this. seems like you probably have the right server and imap files to trigger this error :) If you can do it in the next week or so, we can release it with 3.0
        Hide
        Wes Morgan added a comment -
        I'm going to need some help figuring out why this happens and what the best fix is. But it does seem like it should be assigned to 3.1 with a fairly high priority. This is a pretty big problem.
        Show
        Wes Morgan added a comment - I'm going to need some help figuring out why this happens and what the best fix is. But it does seem like it should be assigned to 3.1 with a fairly high priority. This is a pretty big problem.
        Hide
        Donald A. Lobo added a comment -

        1. we modified the code in mid 2.2 to retrieve only 50 messages at a time. so surprised u r getting this error. can u check with a recent version of the codebase

        2. we will upgrade ezc for the 3.1 release

        3. since u have an imap server exhibiting the issue, would be great if you can take a look and potentially fix. If we need to catch and print the exception thrown, that should be quite easy. Can you investigate and provide a patch?

        lobo
        Show
        Donald A. Lobo added a comment - 1. we modified the code in mid 2.2 to retrieve only 50 messages at a time. so surprised u r getting this error. can u check with a recent version of the codebase 2. we will upgrade ezc for the 3.1 release 3. since u have an imap server exhibiting the issue, would be great if you can take a look and potentially fix. If we need to catch and print the exception thrown, that should be quite easy. Can you investigate and provide a patch? lobo
        Hide
        Donald A. Lobo added a comment -
        These 448 issues have not been worked on for the past 18 months.

        Doing a bulk close of old issues to make the issue queue more manageable. We should do this on a periodic basis.
        Show
        Donald A. Lobo added a comment - These 448 issues have not been worked on for the past 18 months. Doing a bulk close of old issues to make the issue queue more manageable. We should do this on a periodic basis.

          People

          • Assignee:
            Donald A. Lobo
            Reporter:
            Wes Morgan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development