'Garbage in, garbage out' holds true when it comes to the data used to assess crash risk and track changes in crash rates over time. Unreliable driver licensure data published by the Federal Highway Administration (FHWA) are being used by researchers, reporters, and others. The resulting conclusions follow the 'garbage in, garbage out' rule. They're erroneous.
For example, last year USA Today ranked states by crashes per licensed driver 16-19 years old. North Carolina came out worst. When researchers in this state disputed the ranking, the Institute began an inquiry that not only sided with the state researchers but also uncovered pervasive problems with FHWA's licensure data.
'Numerous other reports have been published and findings assumed credible because, after all, the underlying data came from an official government agency. Researchers have been using these data for years because there's no other good source. But even FHWA acknowledges that its own data aren't reliable, so now the key is to do something about it,' says Anne McCartt, Institute research vice president.
What's wrong: FHWA collects and publishes vehicle registration and driver licensure data provided by state officials. The Institute has found big year-to-year fluctuations in the licensure data that aren't explained by population changes, law changes, or any other logical factor. The fluctuations are worst for the youngest drivers.
For example, the reported number of 16- year-old drivers in Illinois was 79,391 in 1998, 8,159 in 1999, and 88,872 in 2000. Louisiana's reported numbers were 25,675 in 2001 and then 2 in 2002 and 3 in 2003. Other anomalies were identified but they weren't as frequent or dramatic.
These discrepancies prompted Institute researchers to compare FHWA's licensure data with data obtained directly from six states. The results confirm the problems because data from only two of the states closely resembled what FHWA was reporting. In the other states there were numerous and sometimes large differences. FHWA generally undercounted licensed drivers.
'We've stopped using these data in our research, at least for now. It's a hardship because it isn't practical to go state by state for data to conduct each individual study. We need the federal database, but we urge other researchers to beware until FHWA's reporting problems are fixed,' McCartt says.
What to do: Factors contributing to data quality are the sources in the states, clarity and appropriateness of FHWA instructions to state officials, extent to which instructions are followed, and quality controls. Problems in all of these areas may be contributing to the unreliability of FHWA data.
Responding to Institute concerns, FHWA has begun to improve teen licensure data, one of the biggest problems, and requested comments about other data issues. More needs to be done, so the Institute has advised the U.S. Department of Transportation to 'consider moving the responsibility for collecting and maintaining these data from FHWA to another agency such as the Bureau of Transportation Statistics or the National Highway Traffic Safety Administration . . . . The importance of the data to the highway safety community and the lack of other sources for these data point to the need to move expeditiously.'
