“What you see is what you get” is a guiding principle for many software engineers — create programs where the content you’re editing looks the same as the final product. But when you’re using generative AI (genAI) systems to 3D print, say, a mug, you’ll likely get a cup that can’t hold your coffee. Why is that?The issue is that AI models understand how an object should look, but not how it works, leading to impractical designs that undermine an item’s intended use. Even if you want to fix these errors, the models are typically hard to edit, especially for users new to 3D design.A new approach called “InstructMesh” makes it much easier to design and print household items, accessories, and robots that work in the real world. The design software, which was developed by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University, can be prompted to generate a 3D design for a pair of glasses, for instance, and users can then highlight specific parts of the blueprint they’d like refined before 3D printing. It’s an AI-driven interface designed to understand how these designs should look and which edits experts and novices alike want to make, helping them create the objects they actually want to see.CSAIL researchers used InstructMesh to put a personalized, creative spin on otherwise regular items. For example, the tool produced a mug that appears to be enveloped by a dragon, with its tail being the handle. It also fabricated a shiny blue whistle resembling a shell and a pair of glasses with butterfly wings spreading out just above the lens. Getting even more creative, it made an octopus-like dispenser, with liquids flowing out of each tentacle to distribute drinks into several cups at once. So what makes InstructMesh so adept at following such unique prompts? It pairs Microsoft’s TRELLIS system, which creates 3D models from text and image prompts, with the large language model (LLM) GPT-4, which supports ChatGPT. In other words, visual and textual knowledge combined.“We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want,” says MIT graduate Faraz Faruqi PhD ‘26, lead author on a paper presenting the project and CSAIL alumnus. “Language models are great at text and images, while TRELLIS’ talent lies in its ability to create 3D models, since it’s seen so many.”InstructMesh’s strengths come in handy in other, more surprising areas. MIT scientists used the program to fabricate a knee brace that looks like denim to match a patient’s jeans. InstructMesh can even help create robots — that is, clever enclosures that house wireless components. The researchers made a “bristle bot” that resembles a colorful shrimp to demonstrate this. It has a motor hidden inside, and when switched on, it can slide across surfaces, sort of like a wind-up toy.New to 3D modeling? No problemFaruqi and his colleagues found that InstructMesh could easily make their desired items. But what would someone who’s never 3D modeled anything think of their program? And could they really detect design flaws before fabrication?The team has TRELLIS recreate popular 3D models found on Thingiverse, a platform home to millions of 3D printable models, to help them find out. Nearly 80% of the models it generated were structurally flawed in some way. CSAIL researchers then asked novices to identify and fix these issues in InstructMesh — and they were able to do both around 90% of the time, as reviewed by an expert. What these newcomers lacked in expertise, they made up for in intuition.InstructMesh scaffolds the actual modeling process, which previously required domain expertise in 3D modeling tools. “With manipulation happening in the latent space of the generative model, InstructMesh supports natural language description of issues, and creates interpretive changes in the geometry for the user to evaluate and approve,” says Faruqi.Users then created items resembling things like phone stands and vases, noting that InstructMesh was easy to use. They also found that InstructMesh enabled them to express a wide range of ideas, while the sliders gave them more precision to make certain tweaks, such as enlarging or extruding a particular part of the model.“The users got what they prompted for and easily tweaked designs where needed,” adds Faruqi. “What they saw is what they got, and the items worked as advertised, so to speak.”While users enjoyed using the InstructMesh, Faruqi has an even grander vision for the project. He now works at Google, where he may soon incorporate InstructMesh into an augmented reality (AR) platform. The idea: Prompt the system by explaining what you need using the context of your surroundings, then it’ll rapidly 3D print it (like making a phone case that matches your wallet).InstructMesh may also begin to incorporate physics simulations to model how your design may react to specific uses, such as whether a bowl breaks when dropped, and which materials would work best. The software might also integrate the more recent TRELLIS.2 to refine even smaller features in 3D models.Stefanie Mueller, an associate professor of electrical engineering and computer science (EECS) and Mechanical Engineering at MIT, and a member of CSAIL, is a senior author on the paper. Faruqi and Mueller wrote the paper with Google researchers Ahmed Katary ‘23; Fabian Manhardt; Vrushank Phadnis MEng ‘13, PhD ‘20; Ruofei Du; Federico Tombari; Northeastern assistant professor Megan Hofmann; and several CSAIL colleagues: PhD students Demircan Tas SM and SMArchS ‘24; former visiting researcher Theresa Hradilak; undergraduate student Ning Zhang; postdoc Jiaji Li; and alum Martin Nisser SM ‘19, PhD ‘24.The researchers’ work was supported, in part, by Google and the MIT-HPI Collaborative Research Program. They will present it at the ACM Symposium on User Interface Software and Technology (UIST) in November.
New tool lets users repair AI-generated 3D models, then fabricate them just the way they want
“What you see is what you get” is a guiding principle for many software engineers — create programs where the content you’re editing looks the same as the final product. But when you’re using generative AI (genAI) systems to 3D print, say, a mug, you’ll likely get a cup that can’t hold your coffee. Why is that?The issue is that AI models understand how an object should look, but not how it works, leading to impractical designs that undermine an item’s intended use. Even if you want to fix these errors, the models are typically hard to edit, especially for users new to 3D design.A new approach called “InstructMesh” makes it much easier to design and print household items, accessories, and robots that work in the real world. The design software, which was developed by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University, can be prompted to generate a 3D design for a pair of glasses, for instance, and users can then highlight specific parts of the blueprint they’d like refined before 3D printing. It’s an AI-driven interface designed to understand how these designs should look and which edits experts and novices alike want to make, helping them create the objects they actually want to see.CSAIL researchers used InstructMesh to put a personalized, creative spin on otherwise regular items. For example, the tool produced a mug that appears to be enveloped by a dragon, with its tail being the handle. It also fabricated a shiny blue whistle resembling a shell and a pair of glasses with butterfly wings spreading out just above the lens. Getting even more creative, it made an octopus-like dispenser, with liquids flowing out of each tentacle to distribute drinks into several cups at once. So what makes InstructMesh so adept at following such unique prompts? It pairs Microsoft’s TRELLIS system, which creates 3D models from text and image prompts, with the large language model (LLM) GPT-4, which supports ChatGPT. In other words, visual and textual knowledge combined.“We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want,” says MIT graduate Faraz Faruqi PhD ‘26, lead author on a paper presenting the project and CSAIL alumnus. “Language models are great at text and images, while TRELLIS’ talent lies in its ability to create 3D models, since it’s seen so many.”InstructMesh’s strengths come in handy in other, more surprising areas. MIT scientists used the program to fabricate a knee brace that looks like denim to match a patient’s jeans. InstructMesh can even help create robots — that is, clever enclosures that house wireless components. The researchers made a “bristle bot” that resembles a colorful shrimp to demonstrate this. It has a motor hidden inside, and when switched on, it can slide across surfaces, sort of like a wind-up toy.New to 3D modeling? No problemFaruqi and his colleagues found that InstructMesh could easily make their desired items. But what would someone who’s never 3D modeled anything think of their program? And could they really detect design flaws before fabrication?The team has TRELLIS recreate popular 3D models found on Thingiverse, a platform home to millions of 3D printable models, to help them find out. Nearly 80% of the models it generated were structurally flawed in some way. CSAIL researchers then asked novices to identify and fix these issues in InstructMesh — and they were able to do both around 90% of the time, as reviewed by an expert. What these newcomers lacked in expertise, they made up for in intuition.InstructMesh scaffolds the actual modeling process, which previously required domain expertise in 3D modeling tools. “With manipulation happening in the latent space of the generative model, InstructMesh supports natural language description of issues, and creates interpretive changes in the geometry for the user to evaluate and approve,” says Faruqi.Users then created items resembling things like phone stands and vases, noting that InstructMesh was easy to use. They also found that InstructMesh enabled them to express a wide range of ideas, while the sliders gave them more precision to make certain tweaks, such as enlarging or extruding a particular part of the model.“The users got what they prompted for and easily tweaked designs where needed,” adds Faruqi. “What they saw is what they got, and the items worked as advertised, so to speak.”While users enjoyed using the InstructMesh, Faruqi has an even grander vision for the project. He now works at Google, where he may soon incorporate InstructMesh into an augmented reality (AR) platform. The idea: Prompt the system by explaining what you need using the context of your surroundings, then it’ll rapidly 3D print it (like making a phone case that matches your wallet).InstructMesh may also begin to incorporate physics simulations to model how your design may react to specific uses, such as whether a bowl breaks when dropped, and which materials would work best. The software might also integrate the more recent TRELLIS.2 to refine even smaller features in 3D models.Stefanie Mueller, an associate professor of electrical engineering and computer science (EECS) and Mechanical Engineering at MIT, and a member of CSAIL, is a senior author on the paper. Faruqi and Mueller wrote the paper with Google researchers Ahmed Katary ‘23; Fabian Manhardt; Vrushank Phadnis MEng ‘13, PhD ‘20; Ruofei Du; Federico Tombari; Northeastern assistant professor Megan Hofmann; and several CSAIL colleagues: PhD students Demircan Tas SM and SMArchS ‘24; former visiting researcher Theresa Hradilak; undergraduate student Ning Zhang; postdoc Jiaji Li; and alum Martin Nisser SM ‘19, PhD ‘24.The researchers’ work was supported, in part, by Google and the MIT-HPI Collaborative Research Program. They will present it at the ACM Symposium on User Interface Software and Technology (UIST) in November.
MIT CSAIL, Google, Northeastern lanciano InstructMesh combinando TRELLIS e GPT-4 per riparare modelli 3D; l'80% aveva difetti, novizi li hanno corretti al 90%. Democratizza il 3D design riducendo il gap tra AI e prodotti funzionanti; impatta skill team manufacturing.
