“We're making the camera the first augmented reality platform,” Facebook CEO Mark Zuckerberg said. Delivering a keynote on April 18 at F8 2017, Facebook's annual developer conference, Zuckerberg announced that the social media company is finally ready to start revealing why it has been investing so much into the augmented reality (AR) and virtual reality (VR) space.

Think about how many things we have in our lives don't need to be physical,” he said, explaining Facebook's vision of “extending the physical world online” through three main use cases: displaying information (directions, messages, notifications, ect.); adding digital objects into scenes; and enhancing existing physical objects.

“I used to think glasses would be the first mainstream AR [augmented reality] platform,” Zuckerberg said. “But over last couple of years we've started to see primitive versions of these use cases on our cameras and phones. ... We see the beginning of a new platform. We're using primitive tools because we're on a journey to create better ones.”

For Facebook that better tool starts with a camera. Any of the billions of Facebook users out there has probably already noticed Facebook has rolled camera functionality out across all of its apps. That was phase one. Phase two, according to Zuckerberg, is creating a platform that will allow developers to build for AR, an SDK called AR Studio.

Think of some of the basic camera effects seen now, such as face masks, as a simple example. Facebook is aiming to let developers create their own AR camera effects as well and roll them out into the social network. Facebook is aiming for developers to create games, provide contextual information in images and videos, and more using AR.

To do this Facebook using artificial intelligence (AI) to allow 3D objects to be accurately positioned within a video by employing a type of AI called Simultaneous Localization and Mapping (SLAM). If you're unfamiliar with SLAM, it's the same AI technology behind the navigation for many autonomous car systems. In essence SLAM uses algorithms to construct a map of a space while also keeping track of an object's position in that space. In self-driving cars SLAM uses GPS mapping to let cars keep track of where they are while moving through city streets. For Facebook this technology allows the company to transform 2D images and video into 3D data that developers can then use to more easily code for AR applications.

Mike Schroepfer, CTO of Facebook, further explained to the audience that the AI technology for Facebook's AR platform is a “revolution in computer vision” and the culmination of research that can be traced back to the 1990s. To recognize people and objects Facebook's AI leverages a neural network technology called Mask R-CNN, which mimics the way that biological visual processing is done in animals.

Doing this, Facebook's platform can detect not only where a person or object is inside of a scene but also how that person or object is oriented. “Using AI we can identify surfaces in a scene and turn them into structured data using real time plane detection,” Schroepfer said, explaining that the platform has been trained in viewing both people and objects and can recognize people, floors, walls, counters, and other key aspects of scene. Facebook researchers have published a research paper on Mask R-CNN, as well. 

The challenge for Facebook, Schroepfer said, is taking these AI algorithms that normally require a lot of server and processing power and getting them to work in a form factor that fits in your pocket (I.e you phone). Schroepfer pointed to Santa Cruz, the protoype standalone headset from Oculus VR as an example. The next-generation VR headset promises to deliver a completely all-in-one, mobile VR experience without the need for an external PC and with a full six degrees of freedom. “What makes [Santa Cruz] work is SLAM,” Schroepfer said. “The device has four cameras that track real world motion and translate it to movement in the virtual world.”

In 2013 Facebook formed Facebook AI Research (FAIR), a group within the company tasked with developing AI and machine learning technologies. Then in 2014 Facebook spent $3 billion to acquire Oculus VR, makers of the popular Oculus Rift headset. However the company has remained relatively quiet about its full ambitions in either of these areas until this recent F8 announcement. 

Despite its large investment in VR, Facebook's new emphasis on AR may be a smart move in the short term. According to Michael Inouye, principal analyst at ABI Research,  AR is foreceast to surpass VR in total revenue opportunity by 2019.  In short VR is a long-term proposition dependent on harware and content development, while AR offers more immediate use cases and return on investment.  

“AR has tremendous viral potential because it leverages mobile devices already in most end users’ pockets, with service availability in the home, public spaces, and workplaces,” Inouye said in a statement released to Design News. “ This creates nearer term synergies with mobile device manufacturers (such as Apple) and social networks (such as Facebook).”

That is not to say Facebook is putting VR on the backburner. The company also announced that it is launching its first social VR product this year. Facebook Spaces, which is currently available for beta testing, will allow users to create virtual avatars and share their Facebook experience in immersive 3D environments instead of a 2D computer screen. Coinciding with this is the release of Facebook's first camera products the x24 and smaller x6 - both of which will allow users to record video for VR with full six degrees of freedom and depth.

“I want to set expectations and say this is gong to take a while to develop,” Zuckerberg said of the company's new AR platform. However, if it can get developers to embrace its new approach, Facebook could see itself positioned as the go-to destination for AR and VR experiences.

Chris Wiltz was formerly the managing editor of Design News.