CRM-14343 Import Error File Maxes out at 250

    Details

    • Type: Bug
    • Status: Open
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: 4.3.7, 4.4.2, 4.7.25
    • Fix Version/s: None
    • Component/s: None
    • Versioning Impact:
      Patch (backwards-compatible bug fixes)
    • Documentation Required?:
      None
    • Funding Source:
      Needs Funding

      Description

      Import error files max out at 250 rows. I am not able to see any errors beyond the 250th one. This appears to affect all the imports (its happened with contact and with contribution imports).

        Attachments

          Activity

          [CRM-14343] Import Error File Maxes out at 250
          Sean Madsen added a comment - - edited

          I can confirm this behavior.

          Steps to reproduce

          1. Begin with the attached CSV import file.
            • Or you can create your own. This file has 1 good row and 500 rows with invalid dates (2000-99-99) for the “Birth Date” field.
          2. Contacts > Import Contacts.
            • Choose the file.
            • Check “First row contains column headers”.
            • Select “No Duplicate Checking”.
            • Click “Continue”.
          3. Continue through the “Match fields” screens as well.
          4. In the preview screen, expect and observe “Rows with errors” to read “500” — good!
          5. Click “Download Errors” to download the CSV file of errors.
          6. Open this CSV file and go to the last line.
            • Expect the number of errors to be 500.
            • Observe the number of errors to be 250.

           

          Sean Madsen added a comment - - edited

          I also found this comment from Lobo:

          The error file maxes out at 250 lines. Not sure why we did this, but i suspect it was due to “is more than 250 errors reasonable to process” or some logic like that

           

          Sean Madsen added a comment -

          Here is the code location of this limit.

          Sean Madsen added a comment -

          Also, here's a StackExchange question with another person asking about this 250 limit.

          Sean Madsen added a comment -

          As far as I can tell, the motivation behind setting a limit here is to avoid timeout errors and poor performance. Based on my test below, I think we might be able to remove this MAX_ERRORS limit without any significant effect on performance or timeout.

          Performance test

          I tested an import with 10,000 invalid rows for different values of MAX_ERRORS and observed nearly identical performance.

          MAX_ERRORS Time to load “Preview” screen Time to download Import_Errors.csv
          250 32,070 ms 41 ms
          20,000 32,670 ms 47 ms

          Why are these so similar? I suspect that the logic to generate the “Preview” screen is already doing all the heavy lifting in order to display the total number of errors (which already displays correctly) and the MAX_ERRORS limit (somewhat ironically) only comes into effect afterwards.

          Noah Miller added a comment -

          I don't see any reason to keep the limit, especially if users aren't informed of it and performance isn't a factor. Lobo's comments speculate that the limit might have been based on the assumption that users wouldn't want to deal with more than 250 rows. But the number of errors is displayed right next to the "Download" link, so if users don't want to download a CSV with 700,000 error rows, they can just make an informed choice not to click the link.

          Sean Madsen added a comment -

          I raised a brief discussion about this issue in Mattermost and also got support from Jon Goldberg and Peter Davis for removing the limit.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jane Hanley

              Dates

              • Created:
                Updated: