At SIGGRAPH 2025, 3D generative AI research is pushing the boundaries of creativity, computation, and control.
VANCOUVER, BC, June 13, 2025 /PRNewswire/ — As 3D generative AI matures, it’s reshaping creativity across multiple disciplines.
Unlike traditional artificial intelligence, which focuses on recognizing patterns and decision-making, generative AI can create entirely new content — from images and animation to realistic 3D models — opening up new frontiers in creative and technical applications.
SIGGRAPH 2025, which will be held 10–14 August in Vancouver, B.C., highlights exciting, groundbreaking research in computer graphics and interactive techniques. This year, ever-expanding work in the 3D generative AI space will be explored as part of the SIGGRAPH Technical Papers program, including these three novel methods — each offering a unique take on how 3D generative AI is being applied.
A Bold Leap in 3D Scene Creation
While generative AI tools have made progress in individual 3D object reconstruction, recreating realistic scenes with multiple objects and accurately capturing their physical interaction remains a challenge. A research team from ShanghaiTech University and Deemos is addressing this limitation with an innovative method for reconstructing high-quality 3D scenes from a single photo. Motivated by bridging the gap between single-object 3D generation and holistic scene reconstruction, this new work levels up the accuracy and efficiency in creating realistic 3D scenes.
“This leap from generating single objects, like a 3D model of a chair, to entire scenes is part of the generative AI revolution, where AI is becoming production-ready,” says Qixuan (QX) Zhang, a researcher at ShanghaiTech University and Deemos and the project lead behind “CAST: Component-Aligned 3D Scene Reconstruction From an RGB Image“. “This is transforming how we create immersive digital experiences and train smart machines.”
Building on earlier work, “CLAY”, which earned the researchers a Best Paper nomination at SIGGRAPH 2024 for its high-quality single-object 3D geometry generation, they developed a new approach called CAST to take things a step further. CAST addresses some of CLAY’s previous limitations around training data and accuracy. By combining powerful AI models that can recognize a wide range of objects, account for hidden parts of a scene, and precisely match camera angles, CAST can reconstruct full 3D scenes from a single, regular photo. The result, say the researchers, is realistic and produces physically accurate environments.
“Imagine taking a picture of your living room and turning it into a fully interactive 3D scene for a video game, movie, or robot training — all with objects perfectly placed and behaving as they would in the real world, like cups resting on tables or chairs tucked against the walls,” explains QX. “CAST brings us closer to a future where anyone can snap a single photo and instantly create such a realistic 3D virtual world that feels alive and true to reality.”
In future work, the CAST team will tackle existing limitations like capturing object details of difficult materials such as textiles and glass, as well as incorporating lighting and background modeling — which all contribute to more realistic and accurate 3D scene generation.
Generative AI’s impact on computer graphics, adds QX, has been profound. “Looking ahead, I believe GenAI will make 3D reconstruction more accessible and production ready. Future enhancements will likely integrate real-time dynamic scene generation, allowing for interactive, evolving environments.”
Working with QX on the CAST research team are collaborators from Deemos and ShanghaiTech, including Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Wei Yang, Lan Xu, Jiayuan Gu, and Jingyi Yu. Visit here for the team’s project page.
Rewriting the Storyboard With 3D GenAI
A new generative AI method is turning 2D storyboard sketches into accurate 3D animations. This underexplored area in computer animation — transforming traditional storyboarding to a 3D process — is being addressed with a new sophisticated tool called Sketch2Anim. This novel method successfully transfers storyboards into high-quality 3D motions and inherently supports direct 3D animation editing.
In this work, an international team of researchers from University of Edinburgh, Northeastern University, and Snap, Inc., leveraged advancements in motion diffusion models — a type of AI that helps generate natural-looking motions frame by frame — to develop a new approach that can automatically translate 2D drawings into 3D animations.
The Sketch2Anim team devised what they call a conditional generation approach, a method that can generate realistic 3D motion, like walking, jumping, or dancing based on a combination of key body poses, movement paths, and action words like “run”, “jump”, or “wave” from a storyboard. Part of the method involves a novel mapping system that can match a user’s simple 2D sketches with the correct and accurate 3D movements. The researchers say this is the first approach to adapt motion diffusion models for generating 3D animation directly from 2D storyboards.
“This task has significant practical importance and has been a longstanding challenge in the field,” says Changjian Li, lead author of the work and an assistant professor in the School of Informatics at University of Edinburgh. “Our method makes a substantial contribution by addressing this complex problem effectively.”
The team conducted comprehensive experiments and evaluations, as well as a user perceptual study to demonstrate the effectiveness of their approach.
The introduction of the Sketch2Anim method marks a promising step forward, and it’s just the beginning. Currently, Sketch2Anim handles simplified storyboards with keyposes and movement paths, such as a character jumping over one object in a scene like a table or chair or bending at the knees. However, real-world storyboarding is often much more complex.
In future work, the team plans to address current limitations and extend the system’s capabilities to handle more detailed 2D sketches and translate them into accurate 3D animations. Additionally, the researchers are exploring ways to enhance control features such as incorporating speed control by interpreting speed lines in sketches. Improving the tool’s controllability, they note, remains a key area for future development.
“With this method and future developments, the 3D animation industry is poised for a transformative shift, moving away from traditional workflows,” adds Li. “We envision generative AI technologies contributing to AI-assisted design tools that lower entry barriers for non-experts, enhance efficiency for professionals, and accelerate innovation across creative industries.”
The “Sketch2Anim: Towards Transferring Sketch Storyboards Into 3D Animation” team will present at SIGGRAPH 2025. In addition to Li, the team includes Lei Zhong, Chuan Guo, Yiming Xie, and Jiawei Wang. Visit their project page for more information.
What Are You Wearing — But Make It 3D
The integration of generative AI tools in fashion tech is rapidly evolving, and it has the ability to take a user’s experience of virtually “trying-before-buying” to the next level. While these innovative tools are becoming more popular, challenges remain to make animated 3D fashion tech truly realistic and easily accessible.
A team of computer scientists at University of California, Los Angeles, and University of Utah are proposing a solution with a new generative AI tool called Dress-1-to-3. This novel end‑to‑end system transforms a single photograph into fully separable, simulation-ready 3D garments — complete with sewing patterns, textures, material parameters, and a posed human body.
“What sets it apart is that the resulting assets are not just visually realistic, but physically accurate and immediately usable in downstream applications such as cloth simulation, virtual try-ons, and digital fashion workflows — all from just one casual image,” says Xuan Li, lead author of SIGGRAPH 2025 Technical Paper “Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit With Diffusion Prior and Differentiable Physics“. Li earned his PhD at UCLA, conducting research at the Chenfanfu Jiang lab and is now a research scientist at NVIDIA.
Dress-1-to-3 takes a single photo of a person and generates realistic, animated 3D clothing that looks and moves like real fabric. Unlike other similar computational frameworks that create stiff or fused models, this tool creates separate garments — like shirts, pants, or dresses — that can automatically be used in digital simulations like a fashion brand’s virtual try-on option or in video games.
The system works by combining a model that predicts sewing patterns from the photo with another that imagines the clothing from multiple angles. Then it fine tunes the result using a virtual fabric simulator to ensure the clothes behave realistically.
“No special equipment is needed — just your smartphone,” says Li. “This is a major step toward making digital fashion and virtual dressing rooms accessible to everyone.”
In this work, Li and the team addressed several obstacles faced by earlier research. A single photo generally lacks depth and often hides parts of the garment due to occlusion, making full 3D reconstruction highly ambiguous. The new tool overcomes this problem by using a multiview diffusion model that imagines how the garment looks from different angles. These synthetic views, say the researchers, help generate a coherent 3D shape. Dress-1-to-3 also successfully reconstructions the complex geometry taken from photographs that can result in better, optimized re-creation of single garments.
“We make a state-of-the-art cloth simulation algorithm — Codimensional Incremental Potential Contact (C-IPC) — differentiable and perform optimization with simulation in the loop,” explains Li. “This physics-based optimization allows us to jointly determine an optimal sewing pattern and physical parameters such that the deformed garments under realistic conditions match the input images.”
The team sees an opportunity to extend this system to handle videos, which are just as easy to capture on smartphones. It may also provide richer information about how garments deform and may improve the estimation of physical properties like stiffness and, in turn, help the sewing pattern optimization.
Li will be showcasing this work at SIGGRAPH 2025 with his collaborators, Chenfanfu Jiang, Chang Yu, Wenxin Du, Ying Jiang, Tianyi Xie, Yunuo Chen, and Yin Yang. Visit their project page for more information.
The research previewed here is a small sampling of the vast Technical Papers research on the schedule for SIGGRAPH 2025. Visit the Technical Papers listing on the full schedule to discover more generative AI content, and register now to experience it among a global community of computer graphics professionals.
About ACM, ACM SIGGRAPH, and SIGGRAPH 2025
The Association for Computing Machinery (ACM), is the world’s largest educational and scientific computing society, uniting educators, researchers, and professionals to inspire dialogue, share resources, and address the field’s challenges. SIGGRAPH is a special interest group within ACM that serves as an interdisciplinary community for members in research, technology, and applications in computer graphics and interactive techniques. The SIGGRAPH conference is the world’s leading annual interdisciplinary educational experience showcasing the latest in computer graphics and interactive techniques. SIGGRAPH 2025, the 52nd annual conference hosted by ACM SIGGRAPH, will take place live 10–14 August at the Vancouver Convention Centre, along with a Virtual Access option.
SOURCE SIGGRAPH 2025