What Went Wrong at Toyota?

Design engineers speculate on how the Japanese automaker got itself into the ugly mess it’s in

Renowned for its high quality and reliability, it is peculiar to see Toyota caught up in a swirl of controversy sparked by the stuck-gas-pedal problem that surfaced publicly in January.

The uncertainty over a technical fix underscores the fiendish challenge of designing complex systems. Yet engineers do get it right the majority of the time. If not, we’d have cars driving off the road and planes falling out of the sky left and right.

And when they don’t get it right, there’s a recall. Most recalls, however, don’t wind up as the subject of a Congressional hearing.

So we’re left with the question of what went wrong this time? Design News asked 150 engineers, many with first-hand experience in working with complex systems, to weigh in. (See chart below for complete survey results.)

Some 40% of the engineers surveyed point to the likelihood that Toyota underestimated the potential magnitude of the problem early on, blaming quick and easy-to-fix problems like floor mats and corroded hinges, when in fact problems with complex systems are almost never that straightforward.

Events spiraled from there until the situation was as out of control as, well, a runaway car with a stuck accelerator pedal.

William Ketel, an EE who worked for a company that did testing on GM’s first electronic throttle in the early 1990s, says engineers at the time did a rigorous Failure Mode Effects Analysis (FMEA) to determine what would happen when a part failed in a particular way. He wonders what sort of analysis Toyota did on its electronic throttle.

“In GM’s case, they basically assumed the throttle would fail at some time, and when it did they wanted it to result in a safe, but inconvenient fault condition. I think the top speed was about 20 mph in the failed mode,” he says.

Although he would like to know what Toyota engineers discovered when they did the FMEA on the throttle system, he worries that given the fact that today’s feature-laden electronic systems are orders of magnitude more complex, it’s getting harder and more costly to verify every bit of code and there is always the pressure on engineers to get products out the door quickly.

“A thorough FMEA of a system as complex as Toyota’s electronic throttle is an expensive proposition, although cheaper than losing even one big lawsuit or a recall campaign,” he says, noting that GM’s early electronic throttle probably added a kilobuck to the cost of the car.

The other possibility, he says, is that the software was indeed just too complex to be simply verified in a timely manner. “Feature creep almost always results in that,” he says.

Even with rigorous testing, some design engineers say that one big challenge is that you never know exactly how your safety system works until something goes wrong in the field. If you’re lucky the vehicle stalls. If you’re not so lucky, the vehicle goes 100 miles per hour.

“People are faulting Toyota for not having done enough testing to find this sticking accelerator problem,” says Charles Glorioso director of special projects at Davis Instruments. “But the reality is that some problems just don’t show up in accelerated testing. No matter how long you test, you don’t know until you get something in the field. It’s been that way forever, and the fact is that with electronics in general you can no longer see what’s going on.”

Glorioso contrasts how different things were when he took his first job in engineering at the Teletype Corp. “The teletype machines were electromechanical, and you could actually unhook the power from the motor and leave the rest of the machine functional. You could turn a crank by hand and watch the keyboard data being encrypted and sent up and turned back into parallel. Now, you can’t do anything without highly specialized tools.”

Nonetheless, other automakers like GM have not had problems with unexplainable acceleration, and many engineers question whether Toyota is to blame for not following industry best practices.

“I think it would be useful here with this changing technology to have some kind of organization that looks for best practices across the industry for safety-related technology. Right now NHTSA has no people with software capabilities,” says Glorioso. “They should have, and they should be looking at the way various manufacturers are handling these safety related issues and be mandating the adoption of best practices.”

Jim Baker, a former embedded engineer who works in the SCADA industry in Australia, raises the question of whether consumers themselves should consider their own culpability in the matter.

“As with all things these days, the consumer is pushing for cheaper, safer, better performing, more efficient cars with features that are more appropriate for the house: TVs, sound systems, refrigerators, etc. The manufacturers are struggling to keep up,” he says. “We have to accept some responsibility for the complexity and cut the manufacturers a bit of slack when there is a problem.”

Nearly one-third of engineers say that Toyota’s problems have been exacerbated by the media piling on. “With respect to Toyota’s handling of the whole thing I would say that no matter what they did it would not be enough or too slow,” says Baker, noting that the Australian press has less of the pack-of-wolves tendencies. “Having said that… if they are aware of the problem and don’t do anything about it they should be held accountable.”

Results of the Design News survey of 150 design engineers: