Simulation in Autonomous Driving – Why Societal Change Is as Necessary as Technical Innovation

A decade ago, the concept of a fully driverless vehicle remained largely the preserve of academic research institutes. Now, commercial activities, such as Waymo’s operations in Phoenix, have shown that not only are driverless vehicles commercially possible, they have the potential to completely alter our socio-economic perception of personal transportation.

However, according to the findings of a report issued by the Rand Corporation, driverless vehicle prototypes should be driven hundreds of millions of miles, and in some cases hundreds of billions of miles, over the course of several decades to demonstrate their reliability in terms of fatalities and injuries – an outcome the report’s authors deemed inconsistent with the near-term commercial viability of self-driving cars. [1]. Put into perspective, that is about half the distance that the Voyager 2 space probe has travelled in the last 40 years. In order to meet this challenge, the researchers pointed to innovative testing methods such as advanced simulation technologies.

Virtual automotive testing and development software has quickly emerged as one of the truly pivotal technologies for enabling the near-term viability of driverless transportation systems. Such toolsets range from full physical simulators, which model comprehensive scenarios with the capability to generate simulated raw sensor data, through to stochastic models which generate object level data. The commercial models for such tools are extremely diverse, ranging from open-source simulation implementations and hobbyist platforms based on gaming engines, to professional-grade commercial modelling tools and extreme-performance simulation technologies, such as those in the Siemens Simcenter PreScan platform.

In the context of these different toolsets and their distinct technical capabilities, it is difficult to define which tool is “state-of-the-art”, as this can only be defined within the scope of the user’s requirements. For example, the development of low technology readiness-level (TRL) sensor fusion algorithms is often most efficiently achieved with commercial modeling tools. However, raising the TRL level from the concept demonstration phase (TRL 5/6) to a verified commercial product (TRL 8/9) requires testing against a full physical model that simulates the complete processing chain -- from sensor physics through to the end-product. That is, such models will allow the algorithmic methodologies to be probed in both a physically and statistically meaningful way. But what does statistically meaningful mean in the context of assisted, automated and autonomous driving?

Waymo recently announced that the company’s cars had driven over 10 million miles on the road, and 10 million miles each day in virtual environments[1]. To put this in perspective, that is about a third of the way to Mars (at the minimum distance). To understand why these values are so high and whether they are statistically meaningful, consider the number of accidents as a function of the number of miles driven. Using Germany as an example, this equates to about one accident every 300,000 miles. If we consider that these accidents are caused by events which are statistical outliers -- that is, events which are associated with some form of error in the driver function -- then over the course of 10 million miles of driving, we can imply that only around 33 accident-causing events are likely to be observed (see [2] and [3] for more detailed analysis).

For the research engineers who are implementing autonomous driving functionality, including development of the algorithmic techniques and methodologies necessary to develop these capabilities, 10 million miles of driving fails to generate anywhere near a sufficient amount of data. Remember that traditional automotive environments and infrastructure have been built around human drivers who have a fantastic ability to both sense and perceive semantic content. The human driver is able to interpret information which is otherwise incomplete, and extrapolate events which have not yet been experienced. That is to say, the human driver is better than machine-learning algorithms at predicting (or rather extrapolating) events which have not yet been observed in the course of training. Therefore, in order to train an artificially intelligent system, it must be exposed to all likely variations across a vast array of scenarios. There remains an extant challenge in collecting sufficient real-world data to effectively develop, implement and -- most importantly -- verify such algorithms.

RELATED ARTICLES:

As a result, synthetic environments become critical across the entire design chain process, because full integration into the development and testing pipeline has distinct benefits. In the first instance, it will allow the system to be trained and developed using the same stochastic models and physical processes which form part of the full simulation. Furthermore, it allows stochastic simulations to be injected with disruptive, low likelihood-of-occurrence events. The effective use of a physical simulation model allows the autonomous vehicle designer to probe how these events will affect the entire end-to-end system -- from sensor data capture through to actuator input processing. As increasing scrutiny from both the general public and standards bodies alike seek to understand the safety implications of autonomous vehicles, simulation provides an opportunity to explore different driving situations, ranging from the mundane cases which account for most driving situations, through to those exceptional cases that most drivers are unlikely to ever experience (Russian dash-cam videos anyone?).

In the final analysis, the direction that future simulation software development will take is central to the near-term viability of autonomous vehicles. Level 5 autonomy will only be realized with a societal change in the way that transportation is perceived. As future transportation offerings transition increasingly to a service-based proposition (as opposed to the current product-based approach), it is likely that the supporting software will become increasingly open source. The key component will be the effective development of the core physics engine and supporting elements of both open source software wherever possible, and proprietary software where necessary.  These supporting components will include different physical sensors, sensor modelling algorithms, sensor configurations, map generation technologies and environmental conditions – all of those aspects which allow the autonomous vehicle design engineer to probe the full breadth of scenarios that an autonomous car will experience.

Daniel Clarke is principal engineer, Automotive Business Unit, for Mentor, A Siemens Business

References

¹Driving to Safety - How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? Nidhi Kalra, Susan M. Paddock

²https://medium.com/waymo/where-the-next-10-million-miles-will-take-us-de51bebb67d3

³ Press release No. 289 of 22 August 2017 of the German Statistisches Bundesamt

 ⁴https://www.itf-oecd.org/sites/default/files/docs/irtad-road-safety-annual-report-2018_2.pdf