Commit to testing the new systems thoroughly, develop an acceptance matrix to document status of testing, and perform verification and validation before introducing them to support agency’s transportation operations.

Chattanooga Area Regional Transportation Authority's experience in deploying transit ITS

November 2009
Chattanooga,Tennessee,United States

Background (Show)

Lesson Learned

The Chattanooga Area Regional Transportation Authority's (CARTA) SmartBus ITS program offers valuable guidance on performing the testing, verification, and validation of new technological systems when implementing ITS at a mid-size transit agency. Key lessons learned include:
  • Develop an acceptance matrix to document the status and amendments resulting from testing, verification, and validation of requirements for ITS projects. A feature of CARTA's systems engineering approach was a strong commitment to thoroughly testing the ITS technologies before introducing them to support operations. CARTA's general approach worked as follows. All the requirements for a system were documented in an acceptance matrix. This matrix included: an identification number (ID) for each requirement, a description of and amendments to each requirement, and remarks from verification and validation exercises. For example, the acceptance matrix for the computer aided dispatch / automatic vehicle location (CAD/AVL) deployment was maintained in an Excel® worksheet that included nearly 800 lines of requirements information. The deployments themselves were divided up into a series of incremental demonstrations, with each demonstration exhibiting more of the final functionality required. Before each demonstration, the contractor was required to submit for CARTA approval an acceptance test procedure that identified the requirements to be met and the method used to verify that the requirements were met. CARTA and the contractor would follow the test procedures during the demonstration, and CARTA would update the requirements matrix with the result and status of each requirement tested. This approach provided a mechanism for (a) ensuring that all requirements were satisfactorily met and (b) monitoring progress towards meeting the requirements.
  • Commit to thoroughly testing systems before introducing them to operations. Having a mechanism for testing is not sufficient without a commitment to conducting thorough testing, and there were numerous examples that demonstrated CARTA's commitment. First was CARTA's recognition of the importance of testing. During discussions with the evaluators, CARTA identified testing as an important part of their system engineering process. Second was CARTA's willingness, if necessary, to change plans based on testing results. For example, during testing of the new operations software for its flex-route operations, CARTA determined that the planned new software would not provide the expected benefits. So, CARTA stopped the introduction of that software, selected an alternative that would better meet operational needs, and rescheduled the deployment to take place after the CAD/AVL system was in place. An example of the thoroughness of CARTA's testing is evident in the rollout of the next-stop announcement system. In late 2008, CARTA completed the onboard database and was capable of activating onboard audio and dynamic message sign (DMS) announcements. Rather than proceeding with system-wide operations after verifying the general functionality of the system, CARTA elected to conduct a comprehensive test of the system's functionality for each route. CARTA, using a test bus set up with the test database, drove each route variation in the system with an observer onboard to note any errors in the messages displayed and announcements made. During this testing, CARTA discovered and corrected several errors in the system, including incorrect announcements and announcements triggered at incorrect locations.
  • During thorough testing, identify enhancements to the system that would improve its operations. The test-bus end-use testing narrated above also allowed CARTA to identify enhancements to the system that would improve its operation. The system was originally configured to display a stop announcement first, then display the date/time, on the interior DMS prior to displaying information about the next stop. During testing, CARTA discovered that the time spent displaying the date/time sometimes delayed the next-stop information display, particularly when stops were closely spaced. Thus, performing end-use testing of the system prevented exposing the public to these errors, which could have reduced public confidence. Another example of CARTA's commitment to testing was the 2009 roll out of the Automated Vehicle Monitoring (AVM) system. The core infrastructure needed to support AVM—the onboard equipment, the Wireless Local Area Networks (WLAN) for daily data uploads, and the AVM software—was in place early in 2008. However, CARTA decided to focus on implementing systems that would deliver the most direct and visible benefits to riders, such as the next-stop announcements and the next-arrival-time predictions. After these systems came online in December 2008, CARTA shifted focus to rolling out AVM. By January 2009, the AVM system elements were integrated enabling the system database to receive daily uploads of data from the buses. CARTA then conducted internal testing of the AVM system to confirm it was operating correctly before releasing it for use by the mechanics in March 2009.
  • Beware of contractor’s resistance to extra time required for thorough testing. Initially, some contractors were resistant to the extra time CARTA required for testing. This resistance was typically overcome during the initial subsystem implementations, when CARTA first applied its thorough testing processes with that contractor. CARTA required that test procedures fully demonstrate all the specified requirements for the systems. CARTA reinforced its commitment to testing by refusing to accept systems that did not pass the documented test procedures. After working with CARTA's testing process, most contractors recognized that this approach, while it might delay final acceptance, helped prevent errors from being found after the system was in use when corrections would have been costlier and more difficult to make.
Developing an acceptance matrix for testing the requirements is a useful exercise. CARTA's strong commitment to thoroughly testing the requirements allowed the agency to identify errors early on and fix them in advance of introducing the systems to support operational efficiency and mobility in its transit services.

Lesson Comments

No comments posted to date

Comment on this Lesson

To comment on this lesson, fill in the information below and click on submit. An asterisk (*) indicates a required field. Your name and email address, if provided, will not be posted, but are to contact you, if needed to clarify your comments.


A Case Study on Applying the Systems Engineering Approach: Best Practices and Lessons Learned from the Chattanooga SmartBus Project

Author: Haas, R.; E. Perry; J. Rephlo

Published By: U.S. DOT Federal Highway Administration

Source Date: November 2009

URL: http://ntl.bts.gov/lib/32000/32600/32672/index.htm

Other Lessons From this Source

Lesson Contacts

Lesson Analyst:

Firoz Kabir


Average User Rating

4.5 (2 ratings)

Rate this Lesson

(click stars to rate)


Lesson of the Month for February, 2011 !

Lesson ID: 2010-00559