Establish a strong acceptance testing program, with independent code reviews, for ITS projects with software product deliveries.

A Washington State Department of Transportation implementation of a regional ATMS.

Date Posted
09/16/2005
TwitterLinkedInFacebook
Identifier
2005-L00111

North Seattle Advanced Traffic Management System (NSATMS) Project Evaluation

Summary Information

The North Seattle Advanced Traffic Management System (NSATMS) was envisioned as a multi-jurisdictional arterial traffic data collection and data sharing system that would offer agencies and local governments throughout the greater Seattle, Washington, area real-time access to timely regional information about traffic conditions and traffic device status. The goals of the system were to promote regional agency coordination and cooperation, manage area traffic more efficiently, and serve as a data source for future metropolitan transportation planning and management efforts. The system would feature a central database management system that would collect information about traffic conditions and traffic device status throughout the region, and a wide-area network of remote workstations that would give participating jurisdictions access to that database.

In 1994, the Washington State DOT initiated a project to test this concept by developing an arterial data sharing system for an urban region that includes Seattle, unincorporated King County, and selected jurisdictions within northern King County and southern Snohomish County; it would also provide a testbed and data source for state intelligent transportation systems (ITS) activities. However, because of technical and project management issues, the system as originally envisioned was not implemented.

This lesson is based on excerpted results from the project’s evaluation report.

Lessons Learned

Develop a strong acceptance testing program as part of large-scale software product deliveries, with independent code reviews performed to verify functionality. A testing program featuring software code reviews by a qualified independent contractor will provide reassurances that existing or newly developed code meets its functional design requirements, and will also minimize the possibility of unrealistic product expectations driven by untested claims. The testing program should include documentation sufficient to enable the customer to understand module functionality, inputs and outputs, and installation processes, as well as the ability to perform at least top-level maintenance. Specific software deliverables should be identified in the task plan.

The NSATMS project's testing process was hampered by several issues:

  1. There was a disconnect between the client’s and the contractor’s definitions of a delivered, tested product. While the contractor noted that software modules were delivered, installed, and tested for functionality, the client believed that the tests were not comprehensive and were not reinforced with support tools, documentation, or training sufficient to enable the client staff to perform its own tests and verification. This difference of viewpoint suggests that a mutually agreed-upon software testing and acceptance program, with accompanying user operations manuals and system support documentation, would have enhanced the likelihood of customer satisfaction and overall project success.
  2. The testing processes, combined with a misperception of the software development aspects of the project, contributed to a system that was not fully operational from the client perspective. As noted in a related lesson, there was a difference of perception regarding the extent to which this project was primarily a modified turnkey installation project, rather than a software development project. A clearer understanding of the software development tasks of this effort would likely have highlighted the importance of mutually agreed-upon testing and validation procedures.

The culmination of the project was what the client believed to be an unsuccessful delivery of the complete, working arterial traffic management system that it had expected to receive. The issue of whether, and to what extent, complete and functioning products were delivered under this contract became a source of discussion during the evaluation process and suggested a difference of opinion regarding the nature of the product that was delivered. If this difference had been resolved at the outset of the project, and a testing program had been put in place, the opportunity for greater customer satisfaction and project success would have been higher.