Sensor quality will always be a key component to autonomous vehicle safety—even if manufacturers disagree on the type of sensors that work best. Whenever there is a crash or fatality around a self-driving car, it always puts the technology sector on notice.
|FLIR manufactures thermal cameras that can be mounted onto autonomous vehicles in conjuction with radar, LiDAR, and other sensors. (Image source: FLIR)|
Oregon-based FLIR Systems has a clear dog in the fight, being the largest commercial manufacturer specializing in thermal imaging sensors, components, and cameras. But the company is taking a unique approach to the issue by adopting open source.
In July, the company released a machine learning dataset of over 10,000 thermal images for researchers, engineers, and manufacturers working on self-driving vehicles and Advanced Driver Assistance Systems (ADAS). Those images can be leveraged in training the neural network artificial intelligence behind autonomous and connected vehicles.
The dataset, which includes both day and nighttime scenarios, contains annotations for cars and other vehicles as well as people, bicycles, and dogs. As a result, the images can be used to train convolutional neural networks (CNNs)—neural networks commonly used for image recognition tasks—to identify these objects. FLIR says that combining thermal imagery along with LiDAR, radar, and visible light cameras can create more robust algorithms capable of more easily identifying objects—particularly people and animals.
According to Mike Walters, vice president of micro-camera product development at FLIR, a big advantage that thermal sensors have is their ability to “see” in many potentially dangerous conditions. “Thermal cameras are great at reliably classifying people and animals—the things you most don’t want to hit—as [people and animals] have a strong thermal signal that stands out from the cluttered surroundings in both rural and urban driving,” Walters told Design News. “Thermal cameras can see in absolute darkness and are also able to see in challenging lighting conditions, like driving directly into sun glare that 'blinds' human drivers and visible cameras. Thermal cameras also perform well in most types of fog, where visible cameras and LIDAR can struggle.”
|This demonstration highlights the ability of FLIR's thermal camera system to spot a pedestrian crossing the street in low light conditions. Thermal cameras are capable of seeing humans at distances that headlights and even visible cameras cannot. (Image source: FLIR)|
In rural driving conditions, Walters said thermal cameras can see and reliably classify humans at 200 meters—four times farther than headlights can illuminate and farther than automotive radar typically sees. In an urban environment, thermal cameras can provide additional, complementary information about the environment 360 degrees around the vehicle, where visible cameras may struggle due to challenging lighting and background clutter.
The idea of an open, freely available dataset could be an attractive one—particularly to startups and smaller entities, which might not have access or resources to this type of data otherwise. Yet open source is often criticized as a double-edged sword. While open source enjoys many evangelists, it also boasts just as many wary skeptics. Having a completely open dataset opens up the possibility of low quality or junk data being introduced into the set. There's also the possibility that bad actors could attempt to manipulate the dataset for malicious reasons. Imagine, for example, if a neural network was trained to think that helpful inanimate objects on the road, such as road signs and traffic lights, were hazards to be avoided.
At the end of the day, the quality of the data will depend on how much oversight and ownership FLIR is willing to take over what goes into the dataset. Walters said the company is actively working with development partners to expand the dataset. He reiterated that even with the potential risks, the company still views open source as the best way forward to developing smarter AI and better sensors for self-driving vehicles. “Free datasets, such as MS COCO, were key to jumpstarting the advancement in machine learning using visible cameras,” Walters said. “Providing a free, annotated thermal image starter dataset for ADAS and autonomous vehicle developers will help speed up testing of thermal cameras and ultimately improve safety and reliability within the ADAS sensor suite.”
Anyone interested in accessing FLIR's thermal imaging dataset for algorithm training can do so by registering on the FLIR website. The company also offers an Automotive Development Kit (ADK) that provides thermal sensor hardware to be used in conjunction with the dataset.
|Today's Insights. Tomorrow's Technologies.
ESC returns to Minneapolis, Oct. 31-Nov. 1, 2018, with a fresh, in-depth, two-day educational program designed specifically for the needs of today's embedded systems professionals. With four comprehensive tracks, new technical tutorials, and a host of top engineering talent on stage, you'll get the specialized training you need to create competitive embedded products. Get hands-on in the classroom and speak directly to the engineers and developers who can help you work faster, cheaper, and smarter. Click here to submit your registration inquiry today
Chris Wiltz is a senior editor at Design News covering emerging technologies, including VR/AR, AI, and robotics.