Lesson

Provide for large sample sizes when conducting before/after data collection efforts, to avoid impacting the ability to reveal statistically significant differences during the evaluation's statistical analysis.

Orlando, Florida's experience with a Field Operational Test (FOT) on using a single smart card for transportation payments at facilities operated by multiple regional agencies.


8/1/2004
Orlando,Florida,United States


Background (Show)

Lesson Learned

When designing a field study or experiment to investigate the effects of different conditions in a before and after design, there are a number of the issues that the study designers must be aware of to properly design the study. Proper study design will help to ensure that both the before and after data collection efforts yield sufficient data to detect significant differences in the conditions. Several key issues include using sufficient large sample sizes, and ensuring that data collection equipment is functioning correctly. The ORANGES experience revealed that the before and after data analysis found that the 95% confidence intervals for the before and after samples of the performance measures overlapped (i.e., not providing evidence to support a statistically significant change between the before and after periods). Larger sample sizes would likely have decreased the size of the confidence intervals, increasing the opportunities to reveal evidence of a statistically significant change in the performance measures. Therefore, the following guidance is provided.
  • Avoid delays in collecting data, so that data collection problems can be detected early. Collect any data being gathered with unattended equipment frequently, to avoid long periods of data loss in the event of equipment problems. LYNX attempted to gather accumulated Automatic Passenger Counter (APC) data from the equipped buses near the end of the after data collection period (APC data was used to measure before and after average boarding times). Unfortunately, only at that point did LYNX become aware that door sensor malfunctions had prevented APC data collection throughout much of the after data collection period. If the data had been retrieved more frequently (e.g., weekly) the door sensor problems could have been identified and resolved quickly, providing much more data.

The ORANGES evaluation suggests the following guidance for similar future initiatives:
  • Plan to collect a substantial amount of before and after data so that the statistical evaluation of the data is more likely to reveal statistically significant differences. The number of observations will depend on the experimental design, the predicted magnitude of the effect, and the expected variance of the observations.
  • Plan to include someone with expertise in experimental design and statistics to help design the experiment, create the data collection and analysis plan, and analyze and interpret the data.
  • Include contingency plans that allow for supplementary data collection, if deemed warranted based on the initial statistical assessment of the initially collected data.

Developing a regional smart card payment system is related to the ITS Goal of improving customer satisfaction, through making payments for multimodal travel easier by establishing a regional payment method. The limitations in the ability to draw conclusions on statistically significant differences did not affect customer satisfaction.

The overall project was completed behind schedule. If the project had not been well behind schedule for other reasons, there would have been more flexibility to consider taking time for supplementary data collection.

The participating cardholders generally expressed a positive opinion about the technology, with concerns focusing primarily on the limited scale of deployment. Data collection issues were not a factor that limited the scale of deployment. In fact, a more extensive deployment would have inherently created additional data collection opportunities.

The ORANGES experience has demonstrated that there are a number of important issues in experimental design and data collection to maximize the chances of detecting significant differences associated with the before and after conditions study conditions. These issues include avoiding delays in collecting data, collecting a substantial amount of before and after data, and planning to include someone with expertise in experimental design and statistics. It is also important to include contingency plans that allow for supplementary data collection, if there are problems identified with the primary data collection effort.


Lesson Comments

No comments posted to date

Comment on this Lesson

To comment on this lesson, fill in the information below and click on submit. An asterisk (*) indicates a required field. Your name and email address, if provided, will not be posted, but are to contact you, if needed to clarify your comments.



Source

Orlando Regional Alliance for Next Generation Electronic Payment Systems (ORANGES) Evaluation Final Report: Electronic Payment Systems Field Operational Test

Author: Moniz, Leisa M. (Volpe) et al.

Published By: U.S. DOT Federal Transit Administration

Source Date: 8/1/2004

EDL Number: 14268

URL: http://ntl.bts.gov/lib//jpodocs/repts_te/14268.htm

Other Lessons From this Source

Lesson Contacts

Lesson Contact(s):

Leisa Moniz
U.S.DOT/RITA/John A. Volpe National Transportation Systems Center
(617) 494-3793
moniz@volpe.dot.gov


Agency Contact(s):

Doug Jamison
Central Florida Regional Transportation Authority (LYNX)
(407) 254-6071
djamison@golynx.com

Lesson Analyst:

Jane Lappin
Volpe National Transportation Systems Center
617-494-3692
jane.lappin@volpe.dot.gov


Rating

Average User Rating

0 ( ratings)

Rate this Lesson

(click stars to rate)


Benefits From This Source

In Central Florida, focus group participants involved in a smart card study using a single card for multiple payment applications indicated that the card provided convenience and improved their transportation experience.

Lessons From This Source

Develop a detailed cardholder recruitment plan in the planning phase of the project, to provide guidance on processes to set pricing, and to ensure high user involvement and loyalty.

Establish a clear understanding among all partners on the level of technical support to be provided by suppliers and integrators, as equipment provided in-kind or at a reduced cost is often provided with minimal technical support.

Establish a forum for decision-makers and project managers to come together to receive project updates, work through critical project issues, make decisions, and support successful institutional collaboration in a project involving multiple agencies.

Include significant planning and development time in the overall project schedule to accommodate identifying and addressing the various compatibility issues, to integrate existing legacy system equipment across multiple agencies.

Provide for large sample sizes when conducting before/after data collection efforts, to avoid impacting the ability to reveal statistically significant differences during the evaluation's statistical analysis.

Seek assurances from your suppliers and sub-contractors, that their production and manufacturing schedules will meet your project schedule and inventory requirements throughout the lifecycle of the project.

Lesson ID: 2006-00192