Establish a strong acceptance testing program, with independent code reviews, for ITS projects with software product deliveries.

A Washington State Department of Transportation implementation of a regional ATMS.

Seattle; Washington; United States

Background (Show)

Lesson Learned

Develop a strong acceptance testing program as part of large-scale software product deliveries, with independent code reviews performed to verify functionality. A testing program featuring software code reviews by a qualified independent contractor will provide reassurances that existing or newly developed code meets its functional design requirements, and will also minimize the possibility of unrealistic product expectations driven by untested claims. The testing program should include documentation sufficient to enable the customer to understand module functionality, inputs and outputs, and installation processes, as well as the ability to perform at least top-level maintenance. Specific software deliverables should be identified in the task plan.
The NSATMS project's testing process was hampered by several issues:
  1. There was a disconnect between the client’s and the contractor’s definitions of a delivered, tested product. While the contractor noted that software modules were delivered, installed, and tested for functionality, the client believed that the tests were not comprehensive and were not reinforced with support tools, documentation, or training sufficient to enable the client staff to perform its own tests and verification. This difference of viewpoint suggests that a mutually agreed-upon software testing and acceptance program, with accompanying user operations manuals and system support documentation, would have enhanced the likelihood of customer satisfaction and overall project success.
  2. The testing processes, combined with a misperception of the software development aspects of the project, contributed to a system that was not fully operational from the client perspective. As noted in a related lesson, there was a difference of perception regarding the extent to which this project was primarily a modified turnkey installation project, rather than a software development project. A clearer understanding of the software development tasks of this effort would likely have highlighted the importance of mutually agreed-upon testing and validation procedures.
The culmination of the project was what the client believed to be an unsuccessful delivery of the complete, working arterial traffic management system that it had expected to receive. The issue of whether, and to what extent, complete and functioning products were delivered under this contract became a source of discussion during the evaluation process and suggested a difference of opinion regarding the nature of the product that was delivered. If this difference had been resolved at the outset of the project, and a testing program had been put in place, the opportunity for greater customer satisfaction and project success would have been higher.

Lesson Comments

No comments posted to date

Comment on this Lesson

To comment on this lesson, fill in the information below and click on submit. An asterisk (*) indicates a required field. Your name and email address, if provided, will not be posted, but are to contact you, if needed to clarify your comments.


North Seattle Advanced Traffic Management System (NSATMS) Project Evaluation

Author: John M. Ishimaru and Mark E. Hallenbeck

Published By: Washington State Transportation Center (TRAC)

Source Date: 12/1/2002

EDL Number: 13818

URL: https://rosap.ntl.bts.gov/view/dot/4097

Other Lessons From this Source

Lesson Contacts

Lesson Contact(s):

John M. Ishimaru
Washington State Transportation Center (TRAC)

Agency Contact(s):

John M. Ishimaru
Washington State Transportation Center (TRAC)

Lesson Analyst:

Firoz Kabir


Average User Rating

0 ( ratings)

Rate this Lesson

(click stars to rate)

Lesson ID: 2005-00111