Knowledge Resources

When designing a field study or experiment to investigate the effects of different conditions in a before and after design, there are a number of the issues that the study designers must be aware of to properly design the study. Proper study design will help to ensure that both the before and after data collection efforts yield sufficient data to detect significant differences in the conditions. Several key issues include using sufficient large sample sizes, and ensuring that data collection equipment is functioning correctly. The ORANGES experience revealed that the before and after data analysis found that the 95% confidence intervals for the before and after samples of the performance measures overlapped (i.e., not providing evidence to support a statistically significant change between the before and after periods). Larger sample sizes would likely have decreased the size of the confidence intervals, increasing the opportunities to reveal evidence of a statistically significant change in the performance measures. Therefore, the following guidance is provided.

• Avoid delays in collecting data, so that data collection problems can be detected early. Collect any data being gathered with unattended equipment frequently, to avoid long periods of data loss in the event of equipment problems. LYNX attempted to gather accumulated Automatic Passenger Counter (APC) data from the equipped buses near the end of the after data collection period (APC data was used to measure before and after average boarding times). Unfortunately, only at that point did LYNX become aware that door sensor malfunctions had prevented APC data collection throughout much of the after data collection period. If the data had been retrieved more frequently (e.g., weekly) the door sensor problems could have been identified and resolved quickly, providing much more data.

The ORANGES evaluation suggests the following guidance for similar future initiatives:

• Plan to collect a substantial amount of before and after data so that the statistical evaluation of the data is more likely to reveal statistically significant differences. The number of observations will depend on the experimental design, the predicted magnitude of the effect, and the expected variance of the observations.

• Plan to include someone with expertise in experimental design and statistics to help design the experiment, create the data collection and analysis plan, and analyze and interpret the data.

• Include contingency plans that allow for supplementary data collection, if deemed warranted based on the initial statistical assessment of the initially collected data.

Developing a regional smart card payment system is related to the ITS Goal of improving customer satisfaction, through making payments for multimodal travel easier by establishing a regional payment method. The limitations in the ability to draw conclusions on statistically significant differences did not affect customer satisfaction.

The overall project was completed behind schedule. If the project had not been well behind schedule for other reasons, there would have been more flexibility to consider taking time for supplementary data collection.

The participating cardholders generally expressed a positive opinion about the technology, with concerns focusing primarily on the limited scale of deployment. Data collection issues were not a factor that limited the scale of deployment. In fact, a more extensive deployment would have inherently created additional data collection opportunities.

The ORANGES experience has demonstrated that there are a number of important issues in experimental design and data collection to maximize the chances of detecting significant differences associated with the before and after conditions study conditions. These issues include avoiding delays in collecting data, collecting a substantial amount of before and after data, and planning to include someone with expertise in experimental design and statistics. It is also important to include contingency plans that allow for supplementary data collection, if there are problems identified with the primary data collection effort.