My father-in-law was a well known Slovenian metal sculptor. He would heat, pound, weld, pound, re-heat and re-pound his scrap mettle until it became an amazing work of art. I see quality assurance in much the same light. The QA team must “pound away” at the application until a master piece arises out of
…until it became an amazing work of art.
the ashes of their efforts. To “forge” an application that helps the end user conduct business and one in which they can trust the quality assurance team must have sufficient tools. The most basic tool is a well designed quality assurance environment.
The classic rule is that the quality assurance environment must perfectly reflect the production environment. This belief however, is backward. The production environment must reflect the quality assurance environment. It is the QA server which is tested, tweaked, enhanced, adjusted and retested to insure the application performs as required. Only after the environment and the application that lives in that environment has been tested, can we establish benchmarks that define the production system requirements.
I have work in quality assurance for a number of years and “cheating” on the QA environment is common practice. The best tools and most dedicated team will not be able to do quality work in an underpowered environment with limited memory and disk space. The argument is simple. The application works in a specific environment we call “production”. To determine how that application will act and react ahead of time, the quality assurance and production environments must match.
The application must support the intended business needs in means of performance and reliability. Not only are business scenarios tested, but the environment must also undergo regular rigorous testing. This establishes benchmarks used to create and maintain the production environment. To establish these benchmarks, I recommend the following tests:
- Load testing – Determine how many concurrent users should typically use the system and reflect their various workflows through realistic test cases.
- Stress testing – After the typical load has been established and test cases for the workflows have been created and run during load testing, gradually introduce more users and test cases until the system fails to respond reliably. At this point you begin to establish the outer limits of the applications ability.
- Endurance testing – Once the typical load has been established, allow that load to run for a longer period of time. This will determine if and to what extent the performance degrades. Usually memory leaks, improper indexes and other hard to determine issues will be discovered.
- Spike testing – Now that the load has been determined and the outer limits have been established, suddenly introduce a large number of users while the system is under normal load. This spike will establish the benchmark for unexpected application activity.
- Configuration testing – Reduce the amount of physical memory. If using Tomcat adjust the heap-space and PermGen memory settings. Remove CPU capacity and other environmental assets. Carefully note the outer limits by running the previous tests in the changed environment. The more you test, the more configuration options you will discover.
Performance testing takes time but when it is initially finished, the system will have a solid set of benchmarks. How often performance testing is required depends on the amount of change introduced to the system at one time or over a set period of time. Closely monitoring the quality assurance and production system through such tools as HP’s LoadRunner Diagnostics Probes may offer an early warning that the system requires a closer performance review.
Click here for part five – “Staging Environment“