Perform a thorough and thoughtful analysis when scoping, sizing, and identifying the geographic location of connected vehicle projects, and ensure that the strategies used to recruit test subjects or drivers are consistent with these assumptions and those of the experimental plan.

Experience with the Safety Pilot Model Deployment in Ann Arbor, Michigan.

September 2015
Michigan; Ann Arbor; Michigan; United States

Background (Show)

Lesson Learned

Setting up the experiment, including selection of the test area and coordinating, preparing, and managing the vehicle fleets and drivers was of critical importance to the Safety Pilot Model Deployment. These elements were key components of the Experimental Plan, which was developed by the Test Conductor in conjunction with the Independent Evaluator. The Experimental Plan also included the Test Conductor’s plans for collecting all data and ensuring data quality as required by the Independent Evaluator to perform its independent evaluation in support of the 2013 NHTSA agency decision.

Prior to developing the statement of work for the SPMD Test Conductor, the USDOT needed to determine what size test was necessary to generate sufficient data for the evaluation of the V2V safety applications. The test size depended on numerous parameters, including the number test participants, number of V2V-equipped vehicles, and the test duration. All could have major impacts on the Test Conductor scope of work and budget required. Therefore it was critical to get an accurate size estimate prior to starting the experiment.

During the Model Deployment, there were two instances where the risk response plans were initiated as a result of lower than expected volumes of interactions. First, due to a delay in the vehicle awareness devices (VAD) deployment as discussed above, the interactions at the start of Model Deployment were less than projected. The USDOT authorized the Test Conductor to utilize staff to drive VAD equipped vehicles throughout the Model Deployment area to increase potential exposure of integrated vehicles to the VAD equipped vehicles. Second, several months into the test, the 3 integrated heavy vehicles were not generating as many interactions as were needed for evaluation purposes. This was due to the vehicles not being driven within the SPMD area as often as anticipated. As a result, the USDOT authorized the Test Conductor to utilize existing staff members with valid commercial driver’s licenses (CDLs) to drive the vehicles within the Model Deployment Geographic Area on predefined routes to generate more data.

Related recommendations made in the source report include:
  • Conduct an analysis using prior field test data to properly scope the size of a new test to meet the data collection and evaluation objectives. Do this early in the planning process as test size can have major impacts on scope and budget, and can be difficult to modify once the experiment has started.
  • Consider adding more vehicles or devices to act as ‘spares’ to ensure that the minimum number of devices required will be operational during the entire test.
  • Be prepared to conduct an iterative process to select the test site. Analyze the characteristics of the test area, i.e., historical crash and traffic volume data, against the performance requirements of the test. Ensure that the test site characteristics match the needs of all aspects of the pilot, particularly if involving commercial vehicles or transit. These vehicle types may be constrained by geographic area, operations and scheduling, or commercial and union restrictions.
  • Closely coordinate the test area selection process with the driver recruitment strategy to ensure that selected participants will be driving primarily in the test area. This approach is likely to produce better results when examining interactions between equipped vehicles and interactions between equipped vehicles and infrastructure.
  • Develop quantitative performance measures with intermediate targets for the environment that can be utilized throughout the test to evaluate progress and success towards project goals.
  • Develop risk response plans in the case that intermediate performance targets are not being met. Determine trigger points to determine when each response plan should be executed.
  • When identifying volunteer drivers, utilize metrics that focus on the volume of trips or time spent within the test area boundaries. Using metrics that focus exclusively on the total number of miles driven may not always be representative of the amount of time a driver spends within the test area. It may also be difficult to identify particular drivers within certain age or gender groups if there is a minimum miles driven requirement.
  • Replicate the community-based driver recruitment strategy used in the SPMD. Understand what motivational factors are important to the community (e.g., education, jobs, safety) and incorporate these factors in the recruitment process. Utilize in-person recruitment activities whenever possible.
  • Recruit participants iteratively to align with the planned device deployment schedule. This will reduce significant delays between recruitment and initial contact. Lags between recruitment and participation could cause participants to reconsider their willingness to participate for a variety of reasons or volunteers could move away from the test area making them non-viable candidates.
  • Utilize actual driving data (i.e. GPS, surveys) over self-reported driving data, if possible when selecting and prioritizing volunteer drivers from the recruitment pool. SPMD demonstrated that there was a significant advantage to using the BSM data logged from VADs to identify participants for future phases.
  • Take into account the geographic area where these vehicles will be operated prior to procurement of specific types of vehicles. There may be constraints that impact the types or configuration of the vehicles purchased, particularly in smaller cities.
  • Ensure that contingency plans are in place in the event that an insufficient amount of data is being generated when conducting naturalistic testing with heavy vehicles. This is especially important with a small sample size and when the geographic area limits the exposure of the heavy vehicles to other V2V equipped vehicles.
  • Ensure the vehicles will be exposed to the target populations addressed by the safety applications when deploying transit-specific safety applications.

Lesson Comments

No comments posted to date

Comment on this Lesson

To comment on this lesson, fill in the information below and click on submit. An asterisk (*) indicates a required field. Your name and email address, if provided, will not be posted, but are to contact you, if needed to clarify your comments.


Safety Pilot Model Deployment: Lessons Learned and Recommendations for Future Connected Vehicle Activities

Author: Kevin Gay, Valarie Kniss (Volpe)

Published By: U.S. Department of Transportation

Source Date: September 2015

EDL Number: FHWA-JPO-16-363

URL: https://rosap.ntl.bts.gov/view/dot/4361

Other Lessons From this Source

Lesson Contacts

Lesson Contact(s):

Kevin Gay

Lesson Analyst:

Gregory Hatcher


Average User Rating

0 ( ratings)

Rate this Lesson

(click stars to rate)

Lesson ID: 2016-00755