Background

Generally, this problem manifests if you have an extension installed which defines custom CiviCRM settings. During an upgrade of said extension, the settings caches are rebuilt but the settings for the upgraded extension get left out, even if there are no changes to the extension's settings in the upgrade in question.

This issue was discovered while correcting the problems in VOL-310. The reports of problems relating to the location and campaign fields in the comments are a result of this issue, not that one.

Steps To Reproduce

These steps can be used for a developer to reliably reproduce the problem in a controlled environment.

• Spin up a drupal-clean buildkit instance using CiviCRM 4.7.22.
• Install Angular Profiles and the latest stable version of CiviVolunteer (4.6-2.2.1) through the UI.
• Go to the API explorer and do api.Setting.get with no params. Control-F for "vol" and you should find a few settings that match.
• Spin up an extdir buildkit instance to serve up CiviVolunteer 4.7.21-2.2.2-beta2 (you can check out the tag v4.7.21-2.2.2-beta2).
• Refresh the extension list. Our local extdir indicates an upgrade is available to CiviVolunteer. Run it.
• Go to the API explorer and do api.Setting.get with no params. Control-F for "vol" and you should find that the settings that matched a few minutes ago are gone.
• Run cv flush. (Note: this command always resolves the problem. Loading civicrm/clearcache has been successful zero or one times – I was thrashing the first time I encountered this and am not sure what the resolution was).
• Go to the API explorer and do api.Setting.get with no params. Control-F for "vol" and you should find a few settings that match.

The Workaround

After upgrading your extension, open up a terminal and run cv flush. This method of clearing and rebuilding caches seems to be more reliable than any other in solving this problem.

Technical Details

Thanks to Tim Otten for his debugging efforts, from which the details below were gleaned.

It's unclear whether this is the issue, but it's interesting to note that after extracting the code for the new version of the extension, CiviCRM does a cache-flush; however, it has already loaded the PHP code from the old version of the extension, so that's what is used during the flush.

These observations were made by adding some log statements into SettingsManager::getDefaults(), CRM_Extension_Downloader, CRM_Core_Invoke::rebuild as well as the old and new copies of volunteer.civix.php (for the alterSettingsFolders hook invocation). The following order of operations were revealed:

01: getDefaults(domain) => early
02: getDefaults(domain) => return
03: download fetch – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Extension/Downloader.php@113
04: download extract – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Extension/Downloader.php@118
05: download validate – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Extension/Downloader.php@124
06: download replace – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Extension/Downloader.php@129
07: rebuildMenuAndCaches – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Core/Invoke.php@366
08: getDefaults(domain) => fill cache
09: (OLD) _volunteer_civix_civicrm_alterSettingsFolders checking]
10: (OLD) _volunteer_civix_civicrm_alterSettingsFolders add /Users/totten/buildkit/build/dcmaster/sites/default/files/civicrm/ext/org.civicrm.volunteer/settings]
11: getDefaults(domain) => return
12: download done – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Extension/Downloader.php@132
13: rebuildMenuAndCaches – /Users/totten/buildkit/build/dcmaster/sites/all/modules/civicrm/CRM/Core/Invoke.php@366
14: getDefaults(domain) => fill cache
15: (OLD) _volunteer_civix_civicrm_alterSettingsFolders short circuited [Array]
16: getDefaults(domain) => return

Lines 9, 10, and 15 suggest that the rebuild is being done while the old code is active. This makes sense – you can't really unload a bunch of php functions in the same request. On the other hand, it doesn't make sense because the files have been moved/deleted, which means that anything checking those files is suspect.

Based on the above, Tim's hunches are that we should either:

• fix/rework the system.flush process so that it's strictly a cleanup operation (deleting caches); all new content-building should be deferred until another request
• the Downloader process should actually be launching subprocesses for each step (one subprocess to extract code; another subprocess to flush caches)