Lesson

Incorporate mechanisms for capturing user feedback for system evaluation, including the ability to intercept incoming 511 calls for survey or focus group recruitment.

An Arizona Department of Transportation experience in 511 implementation.


30 September 2005
Arizona,United States


Background (Show)

Lesson Learned

User feedback is critical to the operation of an effective 511 system and in documenting the benefits of the system, many of which are qualitative. Consequently, deployers of 511 should build into their systems methods for soliciting user input. In the Arizona 511 Model Deployment, two user input mechanisms were included. The first was a caller comment option on the 511 menu: callers could leave messages on a voice mailbox. The second was a call intercept capability: the capability to automatically reroute every nth call (a variable, user defined value) to a survey recruiter. In this evaluation, the intercepted calls were routed to a live survey recruiter who asked the caller if they would be willing to participate in a survey, and if so, scheduled the survey for a time convenient to the caller. The survey was then administered by a live interviewer in a separate, follow-up phone call initiated by the interviewer. Deployers of 511 should be aware of these suggestions from the experience of Arizona DOT:
  • Build user input tools into the system instead of adding them later. Incorporating tools to capture user input from the start is usually more cost-effective than adding them when the system is up and running. Moreover, having the input mechanisms built into the system early allows the operator to easily engage and disengage data collection, including the option of continuously collecting input.
  • Use caller comments to help debug and refine the system. The caller comment feature has proven to be a very useful source of user input. It was especially useful in the first few months following the roll-out of the enhanced system when intensive debugging and refinement of the user interface was occurring. Caller comments were very useful -- both in demonstrating callers sensitivity to what many of them considered to be an inadequately performing user interface and in identifying and correcting specific malfunctions. Utilization of the caller comment featured varied over the first six months of post-enhanced operation from about 0.2% to 0.5% of all 511 calls, with the number of comments ranging from 37 to 373 per month.
  • Use a live intercept survey method to get a higher response rate and a less biased sample of users. Initially ADOT objected to intercepting 511 callers for survey recruitment. They felt that callers would view a brief recorded message inviting them to leave their contact information, or an option to complete an automated survey on the phone, as less disruptive than a live operator intercept of their call. The evaluation team argued that respondents are more likely to agree when invited to participate by a human recruiter. A higher overall response rate in turn reduces sample bias, that is, it mitigates the tendency for only strongly positive and negative callers to participate. ADOT agreed and the live intercept approach was successfully implemented. Approximately 31% of intercepted callers agreed to be surveyed and approximately 71% followed through, yielding 411 surveys and an overall response rate of 22% (that is, 22% of those intercepted completed a survey—a very respectable participation rate.) No complaints were received from intercepted callers regarding the method of recruitment of survey completion.

User feedback is essential in developing a 511 system that is responsive to user needs and preferences. Effectively capturing user feedback is important both for debugging new or enhanced systems and for on-going monitoring of system effectiveness and marketing efforts. Building mechanisms to collect user input into 511 systems as they are being developed is more cost-effective than adding mechanisms later. Two key mechanisms that should be included are a caller comment feature allowing callers to leave voicemail comments and the ability for operators to intercept incoming 511 calls for live recruitment of user survey participants.


Lesson Comments

No comments posted to date

Comment on this Lesson

To comment on this lesson, fill in the information below and click on submit. An asterisk (*) indicates a required field. Your name and email address, if provided, will not be posted, but are to contact you, if needed to clarify your comments.



Source

Model Deployment of a Regional, Multi-Modal 511 Traveler Information System: Final Report

Author: Battelle and the University of Arizona

Published By: Federal Highway Administration, U.S. DOT

Prepared by Battelle and the University of Arizona for the U.S. DOT

Source Date: 30 September 2005

EDL Number: 14248

Other Reference Number: Report No. FHWA-JPO-06-013

URL: http://ntl.bts.gov/lib//jpodocs/repts_te//14248.htm

Other Lessons From this Source

Lesson Contacts

Lesson Contact(s):

Matt Burt
Battelle
480-753-1511
burtm@battelle.org

Lesson Analyst:

Firoz Kabir
Noblis
202-863-2987
firoz.kabir@noblis.org


Rating

Average User Rating

0 ( ratings)

Rate this Lesson

(click stars to rate)


Lesson ID: 2006-00242