No One Knows How Safe New Driver-Assistance Systems Really Are

This week, a US Department of Transportation report detailed the crashes that advanced driver-assistance systems have been involved in over the past year or so. Tesla’s advanced features, including Autopilot and Full Self-Driving, accounted for 70 percent of the nearly 400 incidents—many more than previously known. But the report may raise more questions about this safety tech than it answers, researchers say, because of blind spots in the data.

The report examined systems that promise to take some of the tedious or dangerous bits out of driving by automatically changing lanes, staying within lane lines, braking before collisions, slowing down before big curves in the road, and, in some cases, operating on highways without driver intervention. The systems include Autopilot, Ford’s BlueCruise, General Motors’ Super Cruise, and Nissan’s ProPilot Assist. While it does show that these systems aren’t perfect, there’s still plenty to learn about how a new breed of safety features actually work on the road.

That’s largely because automakers have wildly different ways of submitting their crash data to the federal government. Some, like Tesla, BMW, and GM, can pull detailed data from their cars wirelessly after a crash has occurred. That allows them to quickly comply with the government’s 24-hour reporting requirement. But others, like Toyota and Honda, don’t have these capabilities. Chris Martin, a spokesperson for American Honda, said in a statement that the carmaker’s reports to the DOT are based on “unverified customer statements” about whether their advanced driver-assistance systems were on when the crash occurred. The carmaker can later pull “black box” data from its vehicles, but only with customer permission or at law enforcement request, and only with specialized wired equipment.

Of the 426 crash reports detailed in the government report’s data, just 60 percent came through cars’ telematics systems. The other 40 percent were through customer reports and claims—sometimes trickled up through diffuse dealership networks—media reports, and law enforcement. As a result, the report doesn’t allow anyone to make “apples-to-apples” comparisons between safety features, says Bryan Reimer, who studies automation and vehicle safety at MIT’s AgeLab.

Even the data the government does collect isn’t placed in full context. The government, for example, doesn’t know how often a car using an advanced assistance feature crashes per miles it drives. The National Highway Traffic Safety Administration, which released the report, warned that some incidents could appear more than once in the data set. And automakers with high market share and good reporting systems in place—especially Tesla—are likely overrepresented in crash reports simply because they have more cars on the road.

It’s important that the NHTSA report doesn’t disincentivize automakers from providing more comprehensive data, says Jennifer Homendy, chair of the federal watchdog National Transportation Safety Board. “The last thing we want is to penalize manufacturers that collect robust safety data,” she said in a statement. “What we do want is data that tells us what safety improvements need to be made.”

Without that transparency, it can be hard for drivers to make sense of, compare, and even use the features that come with their car—and for regulators to keep track of who’s doing what. “As we gather more data, NHTSA will be able to better identify any emerging risks or trends and learn more about how these technologies are performing in the real world,” Steven Cliff, the agency’s administrator, said in a statement.