To explore accessibility in AR technology and better understand how it can benefit people with varying physical abilities in the future, we started a pilot program in 2020 with Carnegie Mellon University’s NavCog project to build 3D maps of museums and airports. Developed by CMU, NavCog is an audio wayfinding app designed to help people with visual impairments better navigate their surroundings indoors, where GPS signals often don’t reach. CMU has been working on the NavCog project since 2014.

The open source project has many collaborators around the world. Prior to partnering with Meta, CMU relied on bluetooth beacons placed around an indoor space to accurately determine the location of aNavCog user within that space. Using the Project Aria device, CMU researchers built a 3D map of the Pittsburgh International Airport and other locations. They could then use that map to train AI localization models running on a mobile phone. This could reduce NavCog’s dependency on the external bluetooth beacons, inching us closer toward the not-so-distant future where NavCog can be deployed at scale.

In 2022, FAIR brought together an international consortium of 15 universities to collect and release the world’s largest public dataset of first-person or “egocentric” video of daily-life activities.
Teaching AI to perceive the world as people do requires more than video data.

Select Ego4D consortium universities are expanding their work on egocentric video understanding to include data captured using Project Aria’s sensors, which include stereo cameras, dual inertial measurement units, spatialized microphones, eye tracking cameras, and more. With this effort, we’re excited to bring Project Aria to more researchers around the world, helping the Ego4D project unlock deeper insights into the human experience.