Uploaded image for project: 'CiviCRM'
  1. CiviCRM
  2. CRM-6350

Decrease memory usage of export

    Details

    • Type: New Feature
    • Status: Done/Fixed
    • Priority: Trivial
    • Resolution: Fixed/Completed
    • Affects Version/s: 3.2
    • Fix Version/s: 3.2
    • Component/s: None
    • Labels:
      None

      Description

      export does not scale well for large number of contacts, a few issues:

      1. Using X contact IDs in an IN clause is not good, where X is a large number

      2. The contactIDQuery clause seems inefficient (returns the full clause). We should store the contactIDs in a table and do inner join instead of using the IN clause

      3. building a large array for ALL the rows the CSV file does not scale. We should do incremental CSV writes

      4. Currently for the NYSS cases, export leaks 5K per export line.

      5. mysql caches the results in some buffer. Doing a $dao->fetch( ) for 40K lines is a bad idea. better to split it into 40 loops, each processing 1K

        Attachments

          Activity

            People

            • Assignee:
              rahulbile Rahul Bile
              Reporter:
              lobo Donald A. Lobo
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: