Research
Introducing the Aria Training and Evaluation toolkit (ATEK)
Sept 26, 2024 · 5 min read
Key Takeaways
- Today, we’re introducing ATEK, an e2e framework for training and evaluating deep learning models on Aria data, for both 3D egocentric-specific and general machine perception tasks.
- AI and ML researchers can learn more, by reading the documentation here.
Project Aria has released various public datasets for academic researchers over the past year, such as Aria Everyday Activities, Aria Digital Twin, Aria Everyday Activities, Aria Synthetic Environments, Nymeria, and HOT3D. It is exciting to observe the growing usage of Aria public datasets from the community to accelerate egocentric machine perception tasks. Today, the Project Aria team is happy to release a new toolkit - Aria Training and Evaluation Kit (ATEK), designed to accelerate Deep-Learning (DL) related tasks that use Aria data.
Addressing Data Format Conversion
Sequences recorded by Project Aria glasses are captured in a format known as 'VRS'. Developed and open-sourced by Reality Labs Research, this format is optimized for capturing rich sensor modalities in real-time. However, VRS is relatively new to the community and lacks extensive support from popular deep learning frameworks like PyTorch. ATEK offers an efficient solution to convert VRS to PyTorch-compatible formats, simplifying integration and use within deep learning projects.
Accelerating Data Processing
ATEK Data pre-processing library
Handling raw sensor data from Project Aria can be challenging due to the need for detailed knowledge of various Aria specifications, such as camera calibration, sensor behavior, and data synchronization.
ATEK simplifies this process by offering robust processing functionalities for all types of Aria data. This approach replaces complex data processing pipelines with just a few API calls, using simple configuration JSON files, making it more accessible and efficient for developers to get started.
Direct Accessible Data Store
ATEK Data Store
Even with new tooling, preprocessing large datasets can still be time-consuming and computationally expensive.
To facilitate research acceleration, ATEK offers a Data Store where researchers can download preprocessed datasets with popular configurations. These datasets are designed to be seamlessly integrated into PyTorch-based deep learning pipelines without requiring additional adjustments, thereby streamlining the process and minimizing the risk of errors.
Standardized Evaluation Libraries
ATEK Evaluation library
Aria open datasets continue to enable new use cases, and we aim to accelerate breakthroughs in each area by providing standardized and unified evaluation libraries.
The ATEK evaluation library offers a comprehensive set of evaluation tools for various Aria-related machine perception tasks. This enables researchers to measure and compare the performance of their models with confidence, fostering innovation and advancement in the field.
New Hugging Face Space
Project Aria Hugging Face Space
Finally, we introduce the Project Aria Hugging Face space, which allows researchers to track their model performance, helping measure, compare, and accelerate AI and ML research within the open community.
AI models and datasets, released in the Project Aria Hugging Face space, will be supported by ATEK. Where possible, AI models trained using Project Aria data will also be available with full weights and scripts, making it easy for researchers to discover and benchmark existing models against their own methods.
EFM3D, the first project to harness ATEK for training and evaluation
Earlier this week, the Project Aria Team announced EFM3D, the first effort to establish a benchmark targeted to a novel class of foundation models rooted in 3D space.
As part of the EFM project, two datasets were released in ATEK format, including updates to Aria Synthetic Environments and Aria Digital Twin.
Learn more about the EFM project here
By providing a standardized platform for deep learning research with Project Aria, we hope ATEK will enable more researchers to focus on developing novel solutions, rather than spending time on data formatting and compatibility issues. In time, we believe this increased efficiency will translate to faster progress across the wide field of 3D egocentric perception, helping drive advancements across multiple areas including augmented reality, robotics, and computer vision.
We are excited to see the innovative solutions that will emerge from the use of ATEK and look forward to continuing to support the research community!
