Apple has introduced Matrix3D, a cutting-edge AI model capable of generating detailed 3D scenes and objects from as few as two or three 2D images, signaling a major advancement in photogrammetry. The model was developed in collaboration with researchers from Nanjing University and The Hong Kong University of Science and Technology.

Traditionally, photogrammetry requires numerous images taken from multiple angles and a complex pipeline involving separate models for different reconstruction stages. Matrix3D streamlines this by unifying the process into a single model that estimates camera positions, creates depth maps and generates novel views—all from minimal visual input.

Powered by diffusion transformers, similar to those used in models like DALL·E and ChatGPT, Matrix3D uses masked learning techniques to fill in missing data and handle sparse imagery, allowing for robust and efficient 3D reconstruction.

The research has been published on arXiv, with source code available on GitHub and interactive demos hosted on a companion website. While Apple hasn’t officially announced commercial plans, the technology could potentially be integrated into future Vision Pro experiences, transforming static images into immersive 3D environments.