Uploaded image for project: 'CiviCRM'
  1. CiviCRM
  2. CRM-7922

Change test suite to reset database once every test case

    Details

    • Type: Improvement
    • Status: Done/Fixed
    • Priority: Critical
    • Resolution: Fixed/Completed
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.4.2
    • Component/s: Test suite
    • Labels:
      None

      Description

      Quick explanation first on what and why: previously, we had database
      reset before each test (reminder: which is each method in each class).
      To speed up things, we experimentally introduced db reset before each
      test case (which is class containing multiple methods - tests). It
      gives really huge speed improvement in running api and CRM tests (2
      minutes to run api_v2 tests on my machine, doesn't influence webtests
      in any way), but breaks significant amount of tests. That's because
      our tests expect clean database state, and with the change described
      above, tests leave data after they are executed, which intervenes with
      tests executed after them.

      First goal is to bring api_v2 tests to decent shape, see how it goes
      and than plan for fixing the rest of tests. I've set up an issue for
      that: http://issues.civicrm.org/jira/browse/CRM-7922
      Modified test suite is available in this branch:
      http://svn.civicrm.org/civicrm/branches/trunk.tests/ (I'm merging it
      with trunk on daily basis) - it's enough you do svn switch
      http://svn.civicrm.org/civicrm/branches/trunk.tests/ to have once per
      class db reset in your sandbox.

      Current results for api_v2 are available here:
      http://tests.dev.civicrm.org/trunk/results-New/
      Good news is that with this speed, it will be easy for any of you to
      run who api_v2 suite on your sandboxes - takes around 2 minutes on
      mine. I'll regen results in public site as well, so we can track
      progress.

      Some additional information on the new flow is (using
      api_v2_DomainTest as example), hope it will be helpful for you:

      • database is brought to clean state
      • api_v2_DomainTest::setUp is executed
      • first test from test case is executed
      • api_v2_DomainTest::tearDown is executed
      • above three steps are repeated for each method (setUp, test, tearDown)

      Fixing tests for new flow is mainly making sure, that:
      1. whatever is created in setUp needs to be removed in tearDown
      2. If specific test creates any objects, it should remove it before it
      finishes
      Above of course refer to single test suite (class) - no need to look
      beyond that.

      So if all above is clear, here's proposed split of test classes to fix
      between us.

      Yashi

      • api_v2_ActivityContactTest
      • api_v2_ActivityTest
      • api_v2_ActivityTypeTest
      • api_v2_ConstantTest
      • api_v2_ContactTest
      • api_v2_ContactTypeTest
      • api_v2_ContributionTest
      • api_v2_CustomGroupTest

      Lobo

      • api_v2_CustomValueContactTypeTest
      • api_v2_DomainTest
      • api_v2_EntityTagTest
      • api_v2_EventTest
      • api_v2_GroupContactTest
      • api_v2_GroupNestingTest
      • api_v2_GroupOrganizationTest
      • api_v2_GroupTest

      Rajan

      • api_v2_LocationTest
      • api_v2_MailerTest
      • api_v2_MembershipContactTest
      • api_v2_MembershipContributionLinkTest
      • api_v2_MembershipStatusTest
      • api_v2_MembershipTest
      • api_v2_MembershipTypeTest
      • api_v2_NoteTest

      Michal

      • api_v2_ParticipantTest
      • api_v2_PledgeTest
      • api_v2_RelationshipTest
      • api_v2_RelationshipTypeTest
      • api_v2_TagTest
      • api_v2_UFGroupTest
      • api_v2_UtilsTest

      If possible, let's try to get all above fixed before next chat on
      Wednesday - we're aiming for 3|4.0.1 to have it fixed.

        Attachments

          Activity

            People

            • Assignee:
              mover Michał Mach
              Reporter:
              mover Michał Mach
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: