Typical test issues

Test Participation - Project Office Staff. The project office should participate in testing as soon as possible. This may not be possible if the contractor performs some or all of the development at a remote location. At a minimum, the project office should participate in System Testing and all subsequent test phases. Where possible, the project office should participate in Functional and Integration testing. If the State will be maintaining the system, then the M&O staff should participate in unit testing, if possible. In some cases, the project office staff may request the IV&V vendor to participate in or execute some of the test phases to ensure an un-biased third-party opinion on the status of the system.

Test Participation - User and Sponsor. Often the sponsor and user elect to not participate until Acceptance Testing. However, SID recommends that the User participate or at least observe System Testing, and that the User and Sponsor participate as testers during Acceptance Testing. The User and Sponsor should participate in any test that formally verifies a business requirement to ensure their needs
have been addressed.
Test Environments. The Contractor usually provides the development and test environments, in addition to the production environment. How many environments, and which test environments can be co-located on the same hardware must be decided. If some of the development is being performed offsite, the RFP/ITP/contract should indicated which types of testing may be performed remotely and which must be performed on-site. Another consideration is whether the test environments are considered deliverables which will be retained by the State, or if the test environments remain the property of the Contractor.

Approach to Testing External Interfaces. Testing external interfaces is critical to ensuring a working system, and may help to identify performance issues before beginning production. Some external organizations have dedicated test environments, but most do not. Thus the Project Office must determine how to approach and coordinate testing of these interfaces. A tradeoff must be made about a reasonable level of confidence in the system/testing and the amount of risk the project is willing to accept, vs. the amount of work, coordination and ability of the external organization to participate in testing. The best approach is to include the external organization in the planning process early to determine what is and is not possible.

Approach to Testing COTS products. Although most COTS products are assumed to perform. correctly, there is some testing required to ensure that the COTS product correctly interfaces and supports the rest of the system. For any COTS other than the Operating System and DBMS, the outputs should be verified for typical and error cases. If data is being interchanged, then input and output formats should be verified for correctness. COTS testing should begin in parallel with Integration testing.

Scope of Acceptance Testing. The scope of acceptance testing may depend on what the Contractor is responsible for. Often testing needs to include the business processes, help desk functions (including knowledge base and procedures), backup and recovery, disaster recovery features, system administration tools, specialized hardware, M&O procedures, year-end and quarterly reports, and other user documentation.

Verification of Un-testable Requirements. In some cases, it may be difficult or impractical to test a given requirement. A method of verifying such requirements should be established. These un-testable requirements should be included in a test procedure/script(s) and verified during or just prior to Acceptance Test. The method of verification and appropriate witnesses and supporting documentation should be documented. Typical verification methods include code inspection, simulation using test tools, or, as a last resort, a certification letter from the contractor indicating they will be responsible for any damages resulting from failure of the requirement.

Criteria for Acceptance of the System. The criteria for acceptance is a critical decision that must be documented. Although not all criteria may be identified during the Planning phase, the majority should be documented as part of the RFP/ITP and/or contract. Acceptance criteria typically include (but are not limited to) satisfaction of all requirements (as stated in the RFP/ITP/contract and any associated change orders), approval of all deliverables, and satisfaction of all performance requirements. Some projects have required the system to be in production for a set period of time (to test system stability and its ability to satisfy the user's business needs) prior to conferring acceptance.

Pilot or Field Testing. The Project Office must decide if a pilot or field test (or several pilot/field tests) are warranted based on the type and complexity of the system being developed. The project should have an explicit, stated reason for conducting a pilot and a specific goal (e.g., verifying interfaces with other co-resident applications on the user's desktop). The type of user environment, volume of workload, types of work processed, location, and impact to day-to-day operations should be considered when choosing a pilot location. The outcomes of testing should also be considered: what happens if the pilot fails? what happens if it is successful? what constitutes "success" for the pilot? .

Performance and Capacity Requirements/Testing. Performance and capacity testing is critical for any system. The Project Office must work with the User and Sponsor to identify the performance and capacity requirements and then to determine how to verify the requirements have been satisfied. Large amounts of data will be required, and responsibility for gathering or generating this data must be determined. Specific methods formulas for measuring performance and capacity must be derived and reviewed to ensure that they are fair (often the Contractor does not have control over all of the network or the transmission lines; these should be factored out of the equation).