8.0 System Quality Assurance and Test Strategy
8.1 Quality Assurance Plan
8.1.1 Purpose
This document will provide a description of the testing activities to be
performed on HummingBird during development. It will list out a general
test approach, as well as individual test to be performed. This document
will also describe the circumstances of the tests as well as a bug tracking
system, and code control.
8.1.2 Overall Testing Approach
HummingBird will be modular and its development will be by module so its
testings will be modular. Each of the major parts of the software rely
on at least one other part, so modules will be tested in incremental steps.
At each increment the module will be tested with cooked data and real data
so that unusual and normal situations will both be tested.
For each implemented feature a list of tests will be run to try out
the new module. Once determined that the feature runs well in normal use
patterns it will be put through additional tests designed to break the
module.
At the point where more than one module is integrated the compatibility
of both of the modules will be tested. To determine how well the two function
together as well as to determine if one module can break another. This
will continue as more modules are added to the system.
At each stage of implementation and integration the system will be somewhat
functional. The parts will not fill all requirements but will allow a user
to interact with them on some level. When integrating this will allow the
system to be used as a user might at each stage. Testing as a user will
occur whenever the functionality allows.
Because each module of the system is important and depends almost equally
on the rest of the system, testing will be about equal on each module.
There may be a sightly larger amount of testing on the user interface because
of the variety of possible input from a user.
Near the end of the project the system will be checked for portability.
The system as a whole will be compiled and tested on the target platforms.
This testing will also test the packaging and build scripts to see if they
work properly.
HummingBird will be automatically tested as well, by using a large database
of test data, that could be entered into configuration sources. But parts
of HummingBird are interactive and can not be tested automatically.
These parts must be tested manually. The user interface tests will be this
kind of test. They will require the tester to click on the buttons and
enter the data to test the functionality.
8.1.3 Entrance and Exit Criteria
A section of code will be ready to test whenever functionality is present.
As more and more functionality is added more and more tests will be performed.
Once a module has tested through the pre-defined and automated test list
it will be passed on to integration testing. At this point integration
tests will occur until the modules pass all functional tests. Once the
whole system is complete, every test will be run until all tests are passed.
8.1.4 Automated Regression Testing
Regression testing on HummingBird will occur when new modules in the system
are added. Each new module will be built on other modules so the
interaction and dependences must be checked at every addition. It will
not be necessary to test the whole system every time a source level change
it made, only when a major module is integrated into the system whole.
An automated test suite will be created that will run as an option during
the build process. The tests will be composed of Expect scripts, and C
programs to test the functional requirements of code modules. The source
will be compile into object code with the tests and be run, the expected
output will be compared to the actual output as it occurs. Any differences
in the expected output will be shown and the test name will be printed.
The automated test suite will also contain full system tests. Where
multiple configurations of the entire HummingBird system are created. Each
configuration is installed in a running Hummer and cooked data is sent
to the system. The many pieces output from the Hummer will be compared
to the expected output.
The the tests will be designed to cover most every module or function.
The greatest effort will be to test the branches of every module.
8.1.5 Test Environment
The test environment will be the development machine and then target machines.
HummingBird will be developed on a Sun Ultra-1 Creator3D using SunOS
5.5 (or Solaris 2.5). The target machines will be:
-
Sun Ultra-1 Creator3D using SunOS 5.5 (or Solaris 2.5)
-
HP 9000 series workstation using HP-UX B.10.xx
-
i386 based workstation using FreeBSD 2.2-STABLE
-
i386 based workstation using Linux 2.0.xx
8.2 Test Needs and Requirements
Since we are dealing with lots of code and this is a group project, testing
and test tracking can become very complicated and confusing. For
these reasons, we need strong methods of code control and defect tracking
in order to organize the tests and create applicable test cases.
8.2.1 Code Control
Concurrent Versions System (CVS) 1.9.6 will be used to maintain source
code control. CVS is built upon Revision Control System (RCS) and keeps
record and changes of the code. The system can report the version of the
code per file and binary. Also groups of files can be ``tagged'' so that
they can be retrieved as a package. By creating a tag for a group of files
we can manage the test versions.
8.2.2 Defect Reporting and Tracking
GNATS, the GNU Problem Report Management System, version 3.2 will be used
to keep a log of defects. GNATS keeps a search-able record of submitted
bugs, as well as problem tracking. Each problem exists in one state: analyzed,
closed, feedback, open, or suspended. The problem can be moved between
these states, and records of what time, date who and why are kept. GNATS
makes it easy to keep detailed problem logs by leading the user through
the necessary information to log the full problem. It is with this
system that we can detect problem areas as a result of testing, and create
new tests to prevent from bugs found.
Jamie Marconi
12/13/1997