Managing Software Quality Assurance Test Rounds – Part Four (Test Round Scenario)

Now that we understand the overall project risk and have our project matrix in place, we are ready to actually plan our tests and follow the critical numbers.

The purpose of this example is not to be entirely accurate in every detail. Exact test case and use case matching was not attempted. Such granularity was beyond the scope of this example. Likewise the amount and types of tests conducted at each level, may not reflect a perfectly balanced test round. Such discretion is the responsibility of the quality assurance manager based on the nature of the enhancements and the risk they pose to the system.

Here is the background to our sample test:

1. New functionality has been added to an existing application.
2. A new release is expected once every six weeks. For this reason no test cases or test scripts with a risk score of 1 or 2 are used for mission critical test types. Those with a risk score of three are used sparingly.
3. Each use case is tested by at least one method at some level. The same use case may be tested at different levels.
4. Test methods (white, gray or black box) are isolated by level. This may not apply to each test round. (At times a black box test may be required at the integration level, and some gray box testing at the unit test level etc.)
5. A/B testing is done at both the integration and acceptance levels. The acceptance testers may be aware of unusual execution paths based on their interaction with end users. The quality assurance team should become aware of these unusual interactions and insure new discoveries are documented in new or updated test cases.

This test assumes the following agreements have been made:

1. Use Cases from the business unit are available.
2. The quality assurance team has received some level of training from the business unit.
3. The quality assurance team has obtained from the development team the technical requirements.
4. Agreements have been struck with the development team for unit testing, some business analysts, customer support and end user stakeholders to assist with acceptance testing.

The critical path for this sample scenario is as follows:

1. Obtain agreement for participation. (see assumptions)
2. Begin unit testing.
3. Complete unit testing.
4. Begin integration alpha testing.
5. Complete integration alpha testing.
6. Begin acceptance beta testing.
7. Begin integration smoke testing.
8. Complete integration smoke testing.
9. Begin full integration testing and system testing.
10. Complete acceptance beta testing.
11. Complete full integration and system testing.
12. Begin acceptance smoke testing.
13. Complete acceptance smoke testing.
14. Begin full acceptance testing.
15. Successful completion of acceptance testing indicates the system is deployment ready and may move to the staging environment.

Unit Test

All tasks are assigned to the lead developer Mark. He will assign the test cases to individuals on his team.

Planing Matrix Example - Unit Test

Type	Method	Test ID	Risk	Case ID	Assigned
Installation	n/a
Compatibility	n/a
Smoke or Sanity	n/a
Regression	White Box	142	5	129	Mark
Acceptance	n/a
Alpha	n/a
Beta	n/a
Functional	White Box	111	4	271	Mark
Destructive	n/a
Performance	n/a
Usability	White Box	135	5	178	Mark
Accessibility	n/a
Security	n/a
Internationalization	n/a
Development	White Box	129	4.5	372	Mark
A/B Alternate Route	White Box	895	5	441	Mark

Integration Test

Acceptance testers asked to beta test Test Cases 875 – 878. For this reason after unit testing is complete, Charles begins his alpha tests.

Planing Matrix Example - Integration Test

Type	Method	Test Case ID	Test Case Risk Score	Tests Use Care ID	Assign to
Installation	Grey Box	121	4	976	Joyce
Installation	Grey Box	122	5	977	Joyce
Installation	Grey Box	123	3	978	Joyce
Compatibility	Grey Box	249	5	76	Reanna
Compatibility	Grey Box	250	5	77	Reanna
Compatibility	Grey Box	251	5	78	Reanna
Compatibility	Grey Box	252	4	79	Reanna
Smoke or Sanity	Grey Box	764	5	88	Daniel
Smoke or Sanity	Grey Box	765	5	89	Daniel
Smoke or Sanity	Grey Box	766	5	90	Daniel
Smoke or Sanity	Grey Box	767	4	91	Daniel
Regression	Grey Box	325	4	547	Donald
Regression	Grey Box	326	5	548	Donald
Regression	Grey Box	327	5	549	Donald
Regression	Grey Box	328	4	550	Donald
Regression	Grey Box	329	4	551	Donald
Regression	Grey Box	340	3	552	Donald
Regression	Grey Box	341	4	553	Donald
Acceptance	Grey Box	312	3	743	William
Acceptance	Grey Box	313	5	744	William
Acceptance	Grey Box	314	3	645	William
Acceptance	Grey Box	315	4	746	William
Acceptance	Grey Box	316	5	747	William
Acceptance	Grey Box	317	5	748	William
Acceptance	Grey Box	318	4	749	William
Alpha	Grey Box	875	5	658	Charles
Alpha	Grey Box	876	5	659	Charles
Alpha	Grey Box	877	5	670	Charles
Alpha	Grey Box	878	5	671	Charles
Beta	n/a			672	Charles
Functional	Grey Box	429	3	538	Daniel
Functional	Grey Box	430	5	539	Daniel
Functional	Grey Box	431	5	540	Daniel
Functional	Grey Box	432	5	541	Daniel
Functional	Grey Box	433	5	542	Daniel
Functional	Grey Box	434	5	543	Daniel
Functional	Grey Box	435	4	544	Daniel
Destructive	Grey Box	352	3	52	Joyce
Destructive	Grey Box	353	5	53	Joyce
Destructive	Grey Box	354	5	54	Joyce
Destructive	Grey Box	355	4	55	Joyce
Destructive	Grey Box	356	5	66	Joyce
Performance	n/a
Usability	Grey Box	443	5	442	Reanna
Usability	Grey Box	444	4	443	Reanna
Usability	Grey Box	445	4	444	Reanna
Usability	Grey Box	446	3	445	Reanna
Accessibility	Grey Box	104	2	304	William
Accessibility	Grey Box	105	1	305	William
Accessibility	Grey Box	106	3	306	William
Security	Grey Box	306	4	307	William
Security	Grey Box	307	3	308	William
Security	Grey Box	308	5	309	William
Security	Grey Box	309	5	310	William
Internationalization	Grey Box	920	3	400	Reanna
Internationalization	Grey Box	921	2	401	Reanna
Internationalization	Grey Box	922	2	402	Reanna
Development	n/a	923	1	403	Reanna
A/B Alternate Route	Grey Box	225	5	389	Daniel
A/B Alternate Route	Grey Box	226	5	390	Daniel
A/B Alternate Route	Grey Box	227	4	391	Daniel
A/B Alternate Route	Grey Box	228	5	392	Daniel

System Test

After Charles finishes his alpa testing, he may begin his system tests. System tests are run on an isolated system, having no impact on regression tests. A fully automated system is used. For this reason, one resource is sufficient to create new scripts and run the tests.

Planing Matrix Example - System Test

Type	Method	Test Case ID	Test Case Risk Score	Tests Use Care ID	Assign to	Additional Instructions
Installation	n/a
Compatibility	Grey Box	1249	5	76	Charles
Compatibility	Grey Box	1250	4	77	Charles
Compatibility	Grey Box	1251	4	78	Charles
Compatibility	Grey Box	1252	5	79	Charles
Compatibility	Grey Box	1253	5	80	Charles
Smoke or Sanity	n/a
Regression	Grey Box	1325	4	547	Charles
Regression	Grey Box	1326	5	548	Charles
Regression	Grey Box	1327	5	549	Charles
Regression	Grey Box	1328	4	550	Charles
Acceptance	n/a
Alpha	n/a
Beta	n/a
Functional	Grey Box	1429	3	538	Charles
Functional	Grey Box	1430	2	439	Charles
Functional	Grey Box	1431	5	540	Charles
Functional	Grey Box	1432	4	541	Charles
Functional	Grey Box	1433	3	542	Charles
Functional	Grey Box	1434	5	543	Charles
Destructive	Grey Box	1352	3	52	Charles
Destructive	Grey Box	1353	3	53	Charles
Destructive	Grey Box	1354	5	54	Charles
Performance	Grey Box	1900	5	n/a	Charles	load test
Performance	Grey Box	1901	5	n/a	Charles	scalability test
Performance	Grey Box	1902	5	n/a	Charles	endurance test
Usability	n/a
Accessibility	n/a
Security	n/a
Internationalization	n/a
Development	n/a
A/B Alternate Route	n/a

Acceptance Test

After alpha tests pass, Emma starts her bata testing. Full acceptance testing begins after acceptance smoke testing is finished.

Planing Matrix Example - Acceptance Test

Type	Method	Test Case ID	Test Case Risk Score	Tests Use Care ID	Assign to
Installation	n/a
Compatibility	n/a
Smoke or Sanity	Black Box	774	5	88	Jason
Smoke or Sanity	Black Box	775	5	89	Jason
Smoke or Sanity	Black Box	776	5	90	Jason
Smoke or Sanity	Black Box	777	4	91	Jason
Smoke or Sanity	Black Box	778	4	92	Jason
Smoke or Sanity	Black Box	779	5	93	Jason
Regression	Black Box	903	4	547	Brooke
Regression	Black Box	904	5	548	Brooke
Regression	Black Box	905	5	549	Brooke
Regression	Black Box	906	4	550	Brooke
Regression	Black Box	907	4	551	Brooke
Regression	Black Box	908	5	552	Brooke
Regression	Black Box	909	3	553	Brooke
Acceptance	Black Box	873	5	743	Hannah
Acceptance	Black Box	874	5	744	Hannah
Acceptance	Black Box	875	5	745	Hannah
Acceptance	Black Box	876	5	746	Hannah
Acceptance	Black Box	877	5	747	Hannah
Acceptance	Black Box	878	5	748	Hannah
Acceptance	Black Box	879	5	749	Hannah
Acceptance	Black Box	890	5	750	Hannah
Acceptance	Black Box	891	5	751	Emma
Acceptance	Black Box	892	5	752	Emma
Acceptance	Black Box	893	5	753	Emma
Acceptance	Black Box	894	5	754	Emma
Acceptance	Black Box	895	5	755	Emma
Alpha	n/a			756
Beta	Black Box	527	4	529	Emma
Beta	Black Box	528	5	530	Emma
Beta	Black Box	529	5	531	Emma
Beta	Black Box	530	5	532	Emma
Beta	Black Box	531	5	533	Emma
Functional	n/a
Destructive	Black Box	701	4	52	Alice
Destructive	Black Box	702	5	53	Alice
Destructive	Black Box	703	4	54	Alice
Destructive	Black Box	704	5	55	Alice
Destructive	Black Box	705	4	56	Alice
Destructive	Black Box	706	5	57	Alice
Destructive	Black Box	707	4	58	Alice
Destructive	Black Box	708	4	59	Alice
Destructive	Black Box	709	4	60	Alice
Destructive	Black Box	710	3	61	Alice
Destructive	Black Box	711	5	62	Alice
Performance	n/a
Usability	Black Box	693	4	442	Frank
Usability	Black Box	694	3	443	Frank
Usability	Black Box	695	4	444	Frank
Usability	Black Box	696	4	445	Frank
Usability	Black Box	697	3	446	Frank
Accessibility	Black Box	209	4	304	Laura
Accessibility	Black Box	210	3	305	Laura
Accessibility	Black Box	211	3	306	Laura
Security	n/a
Internationalization	Black Box	777	5	444	Jennifer
Internationalization	Black Box	778	4	445	Jennifer
Internationalization	Black Box	779	3	446	Jennifer
Internationalization	Black Box	780	3	447	Jennifer
Internationalization	Black Box	781	3	448	Jennifer
Development	n/a
A/B Alternate Route	Black Box	902	5	389	Jodie
A/B Alternate Route	Black Box	903	4	390	Jodie
A/B Alternate Route	Black Box	904	3	390	Jodie
A/B Alternate Route	Black Box	905	5	392	Jodie
A/B Alternate Route	Black Box	906	4	393	Jodie

Click her for Part Five – Critical Numbers (KPI)