Details
-
Type: New Feature
-
Status: Done/Fixed
-
Priority: Trivial
-
Resolution: Fixed/Completed
-
Affects Version/s: 3.2
-
Fix Version/s: 3.2
-
Component/s: None
-
Labels:None
Description
export does not scale well for large number of contacts, a few issues:
1. Using X contact IDs in an IN clause is not good, where X is a large number
2. The contactIDQuery clause seems inefficient (returns the full clause). We should store the contactIDs in a table and do inner join instead of using the IN clause
3. building a large array for ALL the rows the CSV file does not scale. We should do incremental CSV writes
4. Currently for the NYSS cases, export leaks 5K per export line.
5. mysql caches the results in some buffer. Doing a $dao->fetch( ) for 40K lines is a bad idea. better to split it into 40 loops, each processing 1K