Testing organizations perform valuable functions, but due to the complex nature of software testing, it can be difficult to assess where a test organization is, either in test development or in regression test execution. By using a well-defined process, test organizations can provide timely status, predictable test execution schedules, and quality documentation.

Key words: inspections, process improvement, requirement management, traceability

INTRODUCTION

As a leader of a software test organization for products that combine complex electronics with several layers of software, the following questions are inevitably asked:

• "What is the status of testing the xyz product with the latest code? Is it stable? How many defects?"

• "We are scheduling a new project. Once you have developed the tests, how long will the regression cycle take?"

• "Are you testing feature x? When was it last tested? Did it pass the test? How are you testing it?"

• "You ran a test and it failed. Upon what requirement was this test based?

All of these are valid questions. How quickly and easily one can produce the answers depends upon the organization's test processes, how it maintains the test documentation, and the metrics that are generated by those processes.

This article describes the process and the metrics the authors' organization has continually refined over the last few years. Several commercial test process management software solutions were investigated, but none seemed to fit the needs of the organization. The process presented here uses a commercially available PC-based requirements tool and a popular PC spreadsheet (no add-ons are required).

TEST DEVELOPMENT PROCESS

Test organizations have two basic functions:

1. to develop and integrate new tests and maintain a full set of qualification tests, and

2. to choose a set of tests that sufficiently exercises the product functionality and run regression test cycles.

For the authors' purpose, a regression test is a set of selected tests that cover existing functionality, but does not include testing of new functionality.

The authors will first address the development/integration of new test, followed by how that process feeds the organization's test execution process. Figure 1 shows its test development process. Note the metrics that are captured.

Figure 1, Generic test development process and metrics

As the top box in Figure 1 shows, the first step is to capture and analyze software requirements. The obvious question is, "How do you capture and analyze software requirements?" The software team uniquely identifies its software requirements by including the key word "shall" in each requirement. The requirements management tool captures the individual requirements by scanning the document for this key word. Once the requirements are captured and enumerated, the document is reviewed, line-by-line, requirement-by-requirement, creating test requirements to cover the software requirements.

To state an obvious, but important, truth is that the software requirements must be the basis for test requirements. One metric that would be valuable is the relationship between test integration and quality (or absence of) of requirements. The results are obvious, but nonetheless, it would be nice to quantify. To date, this effort has been unsuccessful.

Once a draft of the test requirement document is complete, the organization's team subjects it to a Fagan-like inspection process. This process is part of the organization's culture and has been for more than four years (Fagan 1976).

This inspection team usually consists of five people and includes the producer, a software developer (preferably the one who wrote the software requirements), and at least one of the people responsible for actually running the test. Inspectors have at least a week to review the test requirements document and generate a list of defects in the document using standard forms and checklists. The results are captured in a master defect list during a defect-logging meeting. The producer is given the master list and fixes the issues. Depending upon the nature of the defects, a reinspection may be warranted; the inspection team is empowered to decide whether to reinspect the test requirements document.

Data captured during inspections includes:

• Number of defects
• Minutes of inspection time
• Minutes in the logging meeting
• Number of pages inspected
• Number of inspectors

From those measurements, the following metrics are computed:

• Inspection efficiency (minutes of inspection per defect found)
• Defects per inspected page
• Cost to find a defect (historical)

Using these metrics, and comparing them to the organization's regression process metrics, the inspection process is approximately 50 times more cost effective for finding defects than regression testing. For instance, based on three major product developments and 98 inspections over the last five years, it takes only 17 minutes to find a defect in an inspection, while it takes 14 to 30 hours of test to find an equivalent defect during test. This translates into $28 vs. $1,400 to $3,000 (assuming $100/hour). Another notable benefit of the inspection process is that it offers testers and developers the opportunity to interact during the inspection, so developers can help identify tests that may have missed and the testers get advance notice of upcoming new tests. It also allows the test team the opportunity to identify diagnostic requirements so the software requirements can be validated. It is a great opportunity to train test staff, share ideas, and reinforce excellent working relationships between developers and testers.

Once the test requirement document passes inspection, it is captured in the requirements management tool, the test requirements are tagged, and the test requirements are traced to the software requirements. The organization's requirement management tool has user definable attributes, which are used to support the regression process. Attributes include:

• Major test category
• Minor test category
• Test application and/or script to run for this test requirement
• Defect ID of defects found in the code under test
• Defect ID of defects found in the testing code/script

TEST REGRESSION PROCESS

At the start of each regression cycle, a spreadsheet is created to track the test progress. This technique is not novel, but how the spreadsheet is created and maintained is unique. The process used is closed-loop to ensure that the spreadsheet contains the latest test requirements. Figure 2 shows the regression process.

Figure 2, Generic test regression process The Test Spreadsheet

The requirement management tool generates a comma-separated value file for each test specification, using the attributes listed previously. This file contains the test requirement tags, requirement text, and the attributes. Simple macros in the spreadsheet program pull the files into a common spreadsheet that puts each major testing on a separate worksheet. A summary worksheet at the front of this file is also created to provide an executive summary with data automatically calculated from the subordinate worksheets. Finally, a worksheet for defects is created. This file is stored on a shared drive, where it serves as a one-stop shop for all the information about the regression cycle. It is available to test staff, developers, and managers for the project, and is also used by senior management to evaluate the project. It is up to date, online, and useful in many ways to different groups. It is a rational basis for communication between senior management, test management, development management, and the test team.

A separate spreadsheet is maintained for each software version tested, which creates a snapshot of the exact tests run and the status of those tests. As the test suite is enhanced over time, the history of the test evolution is saved.

The columns in Figure 3 warrant further explanation and are described in the subsequent paragraphs.

Figure 3, Summary status information from spreadsheet

The "testing planned" column identifies which major tests are planned for this test cycle. The senior manager and the test manager plan those tests at the beginning of the cycle. The "number of test requirements" column is computed by rolling up the number of test requirements on each detailed sheet. Likewise, most of the remaining columns are rolled up from the detailed sheets. The last two columns identify the tester who ran this test and how many hours he or she spent on this test.

The time column is used for a variety of purposes and worthy of separate discussion. Over the course of many test cycles, all testers get the opportunity to run each test suite, preferably several times. This serves two purposes: all the testers are familiar with the test suites and the average execution time can be computed from the empirical data. At the start of a test cycle, the exact suite of tests is identified. Since the average execution time for each suite is well known, an accurate schedule can be produced. This also allows for accurate schedule generation in the cases where resource levels fluctuate. Typical accuracy on execution time is now greater than 99 percent.

Although testers are familiar with the test suites, they do become more proficient in some areas. These data are also known, so in the event of a major crunch, the most efficient testers can be applied to their areas of expertise, thereby reducing the test cycle by 10 to 20 percent. This is the only purpose for knowing these data. They are not used for any type of performance evaluation.

The tester uses the summary spreadsheet to know what tests to run and how far they have gotten through those tests. The test manager looks at the summary sheet to note any significant deviations from the planned time and also keep an eye on the volume of tests not run ("can't test" and "not tested," which are explained later). Software management and senior test management use this summary spreadsheet to determine how far testing has progressed and how many defects have been found. Senior test management also keeps an eye on test efficiency (hours testing per defect).

Once the planning is complete, the testers print the detailed worksheet (see Figure 4) for the areas they are going to test. The detailed spreadsheet contains all of the information the tester needs to do his or her job. Each tester keeps a paper copy and jots updates on this copy as he or she runs tests. Once a day, each tester uses the data on the paper copy to update the electronic spreadsheet.

Following are definitions for the columns in Figure 3:

• The TR# is the requirement number generated by the requirement management tool.
• The test requirement is the textual portion of the document that was used to create the requirement (the text that was captured by the requirement management tool to create the TR#). If there is any ambiguity about the test requirement, it is easy to look up the test requirement in context for further clarification.
• The app/script is the name of the test application and/or test script that is used to exercise this requirement. The test engineer enters this when the test is created. Scripts and apps are under configuration management control and reside in a well-known location on a shared server, so there is no confusion about how to find them.
• The P/F/CT/NT/KI/KF/PV column requires a bit more explanation. It is used to define the status of the test requirement in the current release of code. The following definitions apply:
  1. Pass (P): Testing completed and expected results achieved
  2. Fail (F): Testing completed and expected results were not achieved. A new defect must be written. To facilitate testing of future code releases, the tester enters the defect number from the defect tracking system and the code version into the change request (code CR) column.
  3. Can't test (CT) (external problem): Lack of verification mechanism; external equipment missing/broken; test developed, software functionality not supported
  4. Not tested (NT) (test group internal problem): Test script/app under development; test script/app needs maintenance
  5. Known issue (KI): An existing defect within another area of code, does not allow further testing to be completed; test verification mechanism (that is, diagnostic output) requested but not implemented
  6. Known failures (KF): Test fails due to an existing defect, which was not fixed in this release
  7. Pass verified (PV): When retesting a known defect, and expected test results are achieved
• The code CR is used to enter the defect number and code version of any new defects found in this release.
• The app/script CR is used to enter the defect number against the test application or test script.
• The comments are used to explain the pass/fail column.

Looking at the example status sheet in Figure 4, the first line indicates a new defect found in this test cycle. A defect, 1234-5678, was logged against version 1.21. One might ask, "Why do you log the code version, since you already know this spreadsheet is being used to track test progress against version 1.21?" The reason is simple. At the end of the test cycle, new test failures and the defect number are copied to the requirements database. The next time a status spreadsheet is generated, the defect number and code version is clearly identified so that the tester, no matter who it is, knows that this test failed in a prior version. This failure status is not removed until the code is fixed and the defect is closed.

Figure 4, Detailed status worksheet

The second line indicated a test that failed in a prior version of code (1.19). The tester will look at the release notes or in the defect database to see if this defect has been fixed. If the defect has been identified as fixed, the test is rerun. If the test passes, the status is set to PV for Pass Verified. If the test fails, the status sheet remains as 'F' and the developer is informed of the failure. If the defect has not been identified as fixed, the status gets marked as KF and the test is skipped. There will be no wasted effort running a test that will fail and the tester will not produce duplicate defects. This is especially valuable when a different tester runs this major category in the next cycle.

Finally, if someone asks when feature "x" was last tested, one can look at the current spreadsheet to determine if it has been tested or is planned for testing. If it is not tested in the current release, one simply needs to look at the summary sheet for prior releases to determine when the test was last run. The answer is never more than a few clicks away.

Keeping Track of Defects

Another worksheet in the test spreadsheet is for identifying defects found against this product. A commercially available defect-tracking tool is used to track defects, so why would information be duplicated here? When a test fails, the tester can refer to this part of the spreadsheet to see if anyone from the group has already found this problem. This prevents the team from generating duplicate defect reports. The other benefit is that management can look quickly at how many defects were found in this release.

Closing the Loop

Once the test cycle is complete, the testers use the requirement management tool and log any new defect numbers against the test requirements. By doing this simple step, which is not very time consuming, the spreadsheet for the next release will contain the defect numbers listed next to the pass/fail column, as described previously.

Root-Cause Analysis on "Bad" Defects

All prudent test managers should be concerned when defect reports issued by their organization are assigned a status of "not a defect." It indicates a waste of both the tester's time and the developer's time. The organization's business rules for its tracking system were modified so that when a defect gets closed for this reason, the originator of the defect is notified via e-mail. The organization's informal process brings the tester together with his or her manager to understand why the defect was written and why it was rejected. If it is decided that the test was bad, the defect is reassigned to a test issue and the offending test is fixed. This helps the team continually improve the tests, and errors do not fall through the cracks. On the other hand, if the defect was due to a deficient or lacking requirement, the defect is reassigned as a documentation defect, which forces the requirements to be updated. Historical trends of both "good defects" and "bad defects" are reported to the staff periodically. Since the organization is aware that the rate of "bad defects" is a concern, they recognize the value and strive to keep the rate low.

CONCLUSIONS

As stated before, the benefits of this process include:

• A closed-loop process for status tracking helps one maintain schedules. Tests are not run unnecessarily and duplicate defects are minimized (tests that have already failed and have not yet been fixed do not get run again). Accurate schedules are developed and the team completes the cycle with a high degree of confidence.
• The test spreadsheet is a powerful management/communications tool. In an instant, one can see the pass/fail status of all objects under test, progress through the test cycle, see how many hours have been spent testing so far, see approximately how many hours are left, and see how many defects have been uncovered. Also, using this spreadsheet over time allows one to accurately predict how long a regression cycle will take. It also allows one to look at the major test sections and determine where the defects tend to appear. If a particular section passes 100 percent of its tests the last five times they have been run, perhaps it can be skipped next time. Alternately, perhaps the test needs to be analyzed to see if it should be improved or rewritten.
• Another useful aspect of this system is when a test fails and someone questions the test, the test requirements are easily traced back to the original test document. The test requirement number is taken from the spreadsheet and entered in the requirements management tool, which takes one directly to the test requirement in context. If necessary, the traced software requirement can be accessed to see how the test requirement relates to the software requirement. This whole process takes just a few moments and usually puts any questions about the validity of the tests to rest quickly.

An Evolving Process

This process has evolved over several years, several products, and many test cycles. It will continue to evolve as the organization progresses. The process does not impose an unnecessary burden on the test cycle and the overhead to keep the process going is minimal, approximately 1 to 3 percent of the total test effort.

Management has become accustomed to a high degree of predictability in the test processes. The authors' estimates are accurate and so is the list of defects produced. Their reports are always available online, are always up to date, and provide easy drill-down to details. Their processes ensure that the tests are correct - or when problems are found, that the tests are corrected. Management and developers understand that defect reports are taken seriously, and that rejected defects are always investigated. Lastly, the organization's staff takes pride in its test efficiency, thoroughness, and metrics-driven processes.

REFERENCES

Fagan, M. E. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal 15, no. 3.

BIOGRAPHIES

Wayne Woodruff is a Senior Manager for Motorola Broadband Communication Sector, where he is responsible for testing of Advanced Interactive Digital Cable Settop boxes. He has been working in embedded systems design, development, and testing for over 23 year. Mr. Woodruff has a Master of Electrical Engineering degree from Villanova University and a Bachelor of Electrical Engineering Technology degree from Spring Garden College. He has published numerous articles ranging from Embedded Systems Design to Integrated Development environments to Requirements Management, which can be viewed at http://www.jtan.com/~wayne/publications.html He is a member of ASQ and he can be reached at wwoodruff@motorola.com

Ronald Pisechko is a Managing Engineer for Motorola's Broadband Communication Sector, where he is responsible for product testing of Advanced Interactive Digital Cable Settop boxes. He has experience on both commercial and governmental software development and test programs. Mr. Pisechko has a Bachelor of Computer Information Science degree from Temple University