Researchers at Cornell Tech have developed a groundbreaking approach to visualize lifelike 3D environments from everyday photos, revolutionizing industries like gaming, virtual tourism, and cultural preservation. Led by Assistant Professor Hadar Averbuch-Elor, the team has created 'WildCAT3D', a framework that significantly enhances novel view synthesis (NVS) capabilities. NVS is a technique that generates realistic angles of a scene from a single photo, but it has traditionally struggled with inconsistent, real-world images. WildCAT3D addresses this challenge by training computers on large collections of freely available photos, including tourist snapshots, images taken in various weather and lighting conditions, and partially obscured scenes. This approach enables the system to focus on the stable structure of a place, ignoring transient visual differences like lighting and weather changes. As a result, WildCAT3D can generate multiple realistic views of the same place from a single photo, opening up new possibilities for virtual tourism, immersive gaming, and accurate digital reconstructions of real-world locations. Moreover, it allows creators and researchers to easily explore how a scene might appear under different conditions, making it an invaluable tool for cultural preservation and planning. Averbuch-Elor believes this work is a step towards making high-quality 3D scene creation more accessible, enabling anyone with ordinary photos to build realistic digital worlds. The research, presented at the Conference and Workshop on Neural Information Processing, was carried out in collaboration with Meta AI and Tel Aviv University.
