Partition WebTests to identify interactions

The WebTest suite has developed a reputation for producing inconsistent results – for example, when running a test locally, it may pass, but when running en masse under Jenkins, it fails. Consequently, we cannot rely on WebTests to promptly identify regressions, and we must expend significant manual effort to triage errors.

The aim of this project is to isolate variations which produce these inconsistencies.

== Batching ==

One category of inconsistency is the batching of the tests. For example, a developer tends to run tests in very small batches (one class or even one function at a time), but Jenkins tends to run very big batches (all WebTests in sequence in the same database).

The primary technique used in this ticket will be dividing the test-suite into different batches. Specifically, we will:

1. Execute all 140+ tests as a single batch.
2. Execute all 140+ tests as 140+ independent batches
3. Execute all 140+ tests as 4 batches of ~36 tests

In an ideal world, each would produce the same test results. We expect to see some variation in the results. Depending on what variations we find, we can investigate further by batching differently.

== Canonical Build ==

Another category is inconsistency in the CiviCRM build, such as:

• CMS version
• CMS single-site vs multi-site
• Third-party CMS modules
• DB structure (single DB vs split DB)
• Permissions of admin/demo users
• Hostname
• httpd version (Apache version, nginx version, etc)
• DB version (MySQL version, MariaDB version, etc)
• OS version
• Filesystem permissions

For purposes of this ticket, we'll define a single, canonical build based on https://github.com/civicrm/civicrm-buildkit. To wit, running tests would mean:

./bin/civi-download-tools
./bin/civibuild create drupal-clean --url http://localhost:8000
cd build/drupal-clean/sites/all/modules/civicrm/tools
./scripts/phpunit WebTest_AllTests

This will control almost all variables (except httpd, DB, OS, and maybe filesystem permissions).

Note: In our community's current adhoc approach to executing tests, each developer runs tests in a slightly different environment. It's good to test in diverse environments, but doing so in an unstructured fashion makes it difficult to analyze/reproduce/communicate issues that do arise from the environment. In a separate issue/project, we can systemically tackle the issue of testing different environments. For this ticket, we want a controlled environment.

== Task 1: FlatfileTests ==

We currently execute tests using the "WebTest_AllTests" suite, which scans for *Test classes and runs them all. To experiment with different batches, we need to take control over what tests run together. The task here is to add a new class, "FlatfileTests", which executes a specific list of tests.

Example:

$ export TESTLIST=/tmp/testlist.txt
$ echo WebTest_Mailing_SpoolTest >> $TESTLIST
$ echo WebTest_Member_EditMembershipTest >> $TESTLIST
$ echo WebTest_Pledge_AddCancelPaymentTest >> $TESTLIST
$ ./scripts/phpunit --tap --log-json=/tmp/results.json FlatfileTests
^^ Reads list from getenv('TESTLIST') ^^

== Task 2: Execute tests on test-ubu1204-2 ==

Create a list of all test classes (eg for a starting point, try "rgrep class tests/phpunit/WebTest" then use a text-editor/search/replace to clean it up)

For each of the batches (in "Batching" above):

• Install a copy of civicrm-buildkit
• Create text files suitable for use with the batches ("batches/1.txt", "batches/2.txt" etc)
• For each of the files, create a clean environment and run FlatfileTests, e.g.
  PRJDIR=/path/to/civicrm-buildkit
  for TESTLIST in $PRJDIR/batches/*.txt ; do
  export TESTLIST
  $PRJDIR/bin/civibuild restore drupal-demo
  pushd $PRJDIR/drupal-demo/sites/all/modules/civicrm/tools
  ./scripts/phpunit --log-json="$ {FILE}

  .json" FlatfileTests
  popd
  done

(Note: The test suite will take a long time. I suggest creating 3 builds ("civibuild create test-1 --type drupal-clean" , etc; then adjust the commands above to use "test-1") and letting them run in parallel overnight.

== Task 3: Combine results ==

Take the output of the various tests ($PRJDIR/batches/*.json) and combine them into a single CSV. You will probably need to write a PHP script. The table will look something like http://pastebin.com/3XBdFmz3