Details
-
Type: Improvement
-
Status: Done/Fixed
-
Priority: Critical
-
Resolution: Fixed/Completed
-
Affects Version/s: 3.4.0
-
Fix Version/s: 3.4.2
-
Component/s: Test suite
-
Labels:None
Description
Quick explanation first on what and why: previously, we had database
reset before each test (reminder: which is each method in each class).
To speed up things, we experimentally introduced db reset before each
test case (which is class containing multiple methods - tests). It
gives really huge speed improvement in running api and CRM tests (2
minutes to run api_v2 tests on my machine, doesn't influence webtests
in any way), but breaks significant amount of tests. That's because
our tests expect clean database state, and with the change described
above, tests leave data after they are executed, which intervenes with
tests executed after them.
First goal is to bring api_v2 tests to decent shape, see how it goes
and than plan for fixing the rest of tests. I've set up an issue for
that: http://issues.civicrm.org/jira/browse/CRM-7922
Modified test suite is available in this branch:
http://svn.civicrm.org/civicrm/branches/trunk.tests/ (I'm merging it
with trunk on daily basis) - it's enough you do svn switch
http://svn.civicrm.org/civicrm/branches/trunk.tests/ to have once per
class db reset in your sandbox.
Current results for api_v2 are available here:
http://tests.dev.civicrm.org/trunk/results-New/
Good news is that with this speed, it will be easy for any of you to
run who api_v2 suite on your sandboxes - takes around 2 minutes on
mine. I'll regen results in public site as well, so we can track
progress.
Some additional information on the new flow is (using
api_v2_DomainTest as example), hope it will be helpful for you:
- database is brought to clean state
- api_v2_DomainTest::setUp is executed
- first test from test case is executed
- api_v2_DomainTest::tearDown is executed
- above three steps are repeated for each method (setUp, test, tearDown)
Fixing tests for new flow is mainly making sure, that:
1. whatever is created in setUp needs to be removed in tearDown
2. If specific test creates any objects, it should remove it before it
finishes
Above of course refer to single test suite (class) - no need to look
beyond that.
So if all above is clear, here's proposed split of test classes to fix
between us.
Yashi
- api_v2_ActivityContactTest
- api_v2_ActivityTest
- api_v2_ActivityTypeTest
- api_v2_ConstantTest
- api_v2_ContactTest
- api_v2_ContactTypeTest
- api_v2_ContributionTest
- api_v2_CustomGroupTest
Lobo
- api_v2_CustomValueContactTypeTest
- api_v2_DomainTest
- api_v2_EntityTagTest
- api_v2_EventTest
- api_v2_GroupContactTest
- api_v2_GroupNestingTest
- api_v2_GroupOrganizationTest
- api_v2_GroupTest
Rajan
- api_v2_LocationTest
- api_v2_MailerTest
- api_v2_MembershipContactTest
- api_v2_MembershipContributionLinkTest
- api_v2_MembershipStatusTest
- api_v2_MembershipTest
- api_v2_MembershipTypeTest
- api_v2_NoteTest
Michal
- api_v2_ParticipantTest
- api_v2_PledgeTest
- api_v2_RelationshipTest
- api_v2_RelationshipTypeTest
- api_v2_TagTest
- api_v2_UFGroupTest
- api_v2_UtilsTest
If possible, let's try to get all above fixed before next chat on
Wednesday - we're aiming for 3|4.0.1 to have it fixed.