8.0  System Quality Assurance and Test Strategy

8.1  Quality Assurance Plan

8.1.1 Purpose

This document will provide a description of the testing activities to be performed on HummingBird during development. It will list out a general test approach, as well as individual test to be performed. This document will also describe the circumstances of the tests as well as a bug tracking system, and code control.

8.1.2 Overall Testing Approach

HummingBird will be modular and its development will be by module so its testings will be modular. Each of the major parts of the software rely on at least one other part, so modules will be tested in incremental steps. At each increment the module will be tested with cooked data and real data so that unusual and normal situations will both be tested.

For each implemented feature a list of tests will be run to try out the new module. Once determined that the feature runs well in normal use patterns it will be put through additional tests designed to break the module.

At the point where more than one module is integrated the compatibility of both of the modules will be tested. To determine how well the two function together as well as to determine if one module can break another. This will continue as more modules are added to the system.

At each stage of implementation and integration the system will be somewhat functional. The parts will not fill all requirements but will allow a user to interact with them on some level. When integrating this will allow the system to be used as a user might at each stage. Testing as a user will occur whenever the functionality allows.

Because each module of the system is important and depends almost equally on the rest of the system, testing will be about equal on each module. There may be a sightly larger amount of testing on the user interface because of the variety of possible input from a user.

Near the end of the project the system will be checked for portability. The system as a whole will be compiled and tested on the target platforms. This testing will also test the packaging and build scripts to see if they work properly.

HummingBird will be automatically tested as well, by using a large database of test data, that could be entered into configuration sources. But parts of HummingBird are interactive and can not be tested automatically.  These parts must be tested manually. The user interface tests will be this kind of test. They will require the tester to click on the buttons and enter the data to test the functionality.
 

8.1.3 Entrance and Exit Criteria

A section of code will be ready to test whenever functionality is present. As more and more functionality is added more and more tests will be performed. Once a module has tested through the pre-defined and automated test list it will be passed on to integration testing. At this point integration tests will occur until the modules pass all functional tests. Once the whole system is complete, every test will be run until all tests are passed.

8.1.4 Automated Regression Testing

Regression testing on HummingBird will occur when new modules in the system are added.  Each new module will be built on other modules so the interaction and dependences must be checked at every addition. It will not be necessary to test the whole system every time a source level change it made, only when a major module is integrated into the system whole.

An automated test suite will be created that will run as an option during the build process. The tests will be composed of Expect scripts, and C programs to test the functional requirements of code modules. The source will be compile into object code with the tests and be run, the expected output will be compared to the actual output as it occurs. Any differences in the expected output will be shown and the test name will be printed.

The automated test suite will also contain full system tests. Where multiple configurations of the entire HummingBird system are created. Each configuration is installed in a running Hummer and cooked data is sent to the system. The many pieces output from the Hummer will be compared to the expected output.

The the tests will be designed to cover most every module or function. The greatest effort will be to test the branches of every module.

8.1.5 Test Environment

The test environment will be the development machine and then target machines.

HummingBird will be developed on a Sun Ultra-1 Creator3D using SunOS 5.5 (or Solaris 2.5). The target machines will be:

8.2  Test Needs and Requirements

Since we are dealing with lots of code and this is a group project, testing and test tracking can become very complicated and confusing.  For these reasons, we need strong methods of code control and defect tracking in order to organize the tests and create applicable test cases.

8.2.1 Code Control

Concurrent Versions System (CVS) 1.9.6 will be used to maintain source code control. CVS is built upon Revision Control System (RCS) and keeps record and changes of the code. The system can report the version of the code per file and binary. Also groups of files can be ``tagged'' so that they can be retrieved as a package. By creating a tag for a group of files we can manage the test versions.

8.2.2 Defect Reporting and Tracking

GNATS, the GNU Problem Report Management System, version 3.2 will be used to keep a log of defects. GNATS keeps a search-able record of submitted bugs, as well as problem tracking. Each problem exists in one state: analyzed, closed, feedback, open, or suspended. The problem can be moved between these states, and records of what time, date who and why are kept. GNATS makes it easy to keep detailed problem logs by leading the user through the necessary information to log the full problem.  It is with this system that we can detect problem areas as a result of testing, and create new tests to prevent from bugs found.
 

Jamie Marconi

12/13/1997