MIT tool visualizes and edits “physically impossible” objects

M.C. Escher’s artwork is a gateway into a world of depth-defying optical illusions, featuring “impossible objects” that break the laws of physics with convoluted geometries. What you perceive his illustrations to be depends on your point of view — for example, a person seemingly walking upstairs may be heading down the steps if you tilt your head sideways. Computer graphics scientists and designers can recreate these illusions in 3D, but only by bending or cutting a real shape and positioning it at a particular angle. This workaround has downsides, though: Changing the smoothness or lighting of the structure will expose that it isn’t actually an optical illusion, which also means you can’t accurately solve geometry problems on it.Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a unique approach to represent “impossible” objects in a more versatile way. Their “Meschers” tool converts images and 3D models into 2.5-dimensional structures, creating Escher-like depictions of things like windows, buildings, and even donuts. The approach helps users relight, smooth out, and study unique geometries while preserving their optical illusion.This tool could assist geometry researchers with calculating the distance between two points on a curved impossible surface (“geodesics”) and simulating how heat dissipates over it (“heat diffusion”). It could also help artists and computer graphics scientists create physics-breaking designs in multiple dimensions.Lead author and MIT PhD student Ana Dodik aims to design computer graphics tools that aren’t limited to replicating reality, enabling artists to express their intent independently of whether a shape can be realized in the physical world. “Using Meschers, we’ve unlocked a new class of shapes for artists to work with on the computer,” she says. “They could also help perception scientists understand the point at which an object truly becomes impossible.”Dodik and her colleagues will present their paper at the SIGGRAPH conference in August.Making impossible objects possibleImpossible objects can’t be fully replicated in 3D. Their constituent parts often look plausible, but these parts don’t glue together properly when assembled in 3D. But what can be computationally imitated, as the CSAIL researchers found out, is the process of how we perceive these shapes.Take the Penrose Triangle, for instance. The object as a whole is physically impossible because the depths don’t “add up,” but we can recognize real-world 3D shapes (like its three L-shaped corners) within it. These smaller regions can be realized in 3D — a property called “local consistency” — but when we try to assemble them together, they don’t form a globally consistent shape.The Meschers approach models’ locally consistent regions without forcing them to be globally consistent, piecing together an Escher-esque structure. Behind the scenes, Meschers represents impossible objects as if we know their x and y coordinates in the image, as well as differences in z coordinates (depth) between neighboring pixels; the tool uses these differences in depth to reason about impossible objects indirectly.The many uses of MeschersIn addition to rendering impossible objects, Meschers can subdivide their structures into smaller shapes for more precise geometry calculations and smoothing operations. This process enabled the researchers to reduce visual imperfections of impossible shapes, such as a red heart outline they thinned out.The researchers also tested their tool on an “impossibagel,” where a bagel is shaded in a physically impossible way. Meschers helped Dodik and her colleagues simulate heat diffusion and calculate geodesic distances between different points of the model.“Imagine you’re an ant traversing this bagel, and you want to know how long it’ll take you to get across, for example,” says Dodik. “In the same way, our tool could help mathematicians analyze the underlying geometry of impossible shapes up close, much like how we study real-world ones.”Much like a magician, the tool can create optical illusions out of otherwise practical objects, making it easier for computer graphics artists to create impossible objects. It can also use “inverse rendering” tools to convert drawings and images of impossible objects into high-dimensional designs. “Meschers demonstrates how computer graphics tools don’t have to be constrained by the rules of physical reality,” says senior author Justin Solomon, associate professor of electrical engineering and computer science and leader of the CSAIL Geometric Data Processing Group. “Incredibly, artists using Meschers can reason about shapes that we will never find in the real world.”Meschers can also aid computer graphics artists with tweaking the shading of their creations, while still preserving an optical illusion. This versatility would allow creatives to change the lighting of their art to depict a wider variety of scenes (like a sunrise or sunset) — as Meschers demonstrated by relighting a model of a dog on a skateboard.Despite its versatility, Meschers is just the start for Dodik and her colleagues. The team is considering designing an interface to make the tool easier to use while building more elaborate scenes. They’re also working with perception scientists to see how the computer graphics tool can be used more broadly.Dodik and Solomon wrote the paper with CSAIL affiliates Isabella Yu ’24, SM ’25; PhD student Kartik Chandra SM ’23; MIT professors Jonathan Ragan-Kelley and Joshua Tenenbaum; and MIT Assistant Professor Vincent Sitzmann. Their work was supported, in part, by the MIT Presidential Fellowship, the Mathworks Fellowship, the Hertz Foundation, the U.S. National Science Foundation, the Schmidt Sciences AI2050 fellowship, MIT Quest for Intelligence, the U.S. Army Research Office, U.S. Air Force Office of Scientific Research, SystemsThatLearn@CSAIL initiative, Google, the MIT–IBM Watson AI Laboratory, from the Toyota–CSAIL Joint Research Center, Adobe Systems, the Singapore Defence Science and Technology Agency, and the U.S. Intelligence Advanced Research Projects Activity.