CMU Researchers Develop AI System That Transforms Text Into Tactile Graphics

The Breakdown 

• Researchers used generative AI to turn text prompts into 3D-printed tactile graphics.
• Text2TactileGraphics creates raised shapes and textures to help convey visual information to users who are blind or have low vision.
• The research team tested the tool with participants from VisAbility Pittsburgh and the Library of Accessible Media for Pennsylvanians. 

* * * 

Researchers in the Carnegie Mellon University School of Computer Science are using technology to create tactile graphics that allow people who are blind or have low vision to experience and interpret visual information through touch.

The work, Text2TactileGraphics, turns text prompts into 3D-printed raised graphics that incorporate shape, texture and Braille. The tool can help people with low vision create and explore tactile representations of objects that might otherwise be difficult to experience. 

“Vision can easily convey a lot of information, but we use touch to understand and interpret things in daily life more than we may realize,” said Ruihan Gao, a Ph.D. student in the Robotics Institute (RI) and co-lead on the project. “We were motivated by the opportunity to bring generative AI to everyone — and everyone includes people who are blind or have low vision.” 

Tactile graphics can provide access to information that is typically communicated visually, including raised maps and diagrams, and illustrations of plants, animals, anatomy and more. But creating those graphics requires specialized expertise, making them difficult to produce at scale or personalize for individual needs. 

The research team found that, even though there are billions of images on the internet, the amount of useful tactile graphics is much smaller.

“A lot of image-generation systems can produce something that looks realistic on a screen, but our system accounts for how an object will actually feel when the object is 3D printed and someone runs their fingers across it,” said Ava Pun, an RI Ph.D. student and project co-lead. “That means the system has to understand more than what an object looks like. It must determine its overall shape and the physical surface details that communicate texture.” 

The Text2TactileGraphics process begins with a user entering a description. The person entering a prompt may describe a cat as having fur or a rock as being rough. The system translates those descriptions into fine-scale grooves, bumps and other raised patterns that can be incorporated into a printed model.

Once the system has identified the object and determined its shape and features, it generates a corresponding representation: a flat base with raised features that are explored by touch. It can then incorporate a Braille description so a user can identify the object. 

The researchers designed the system to accommodate the many ways people may describe textures.

“Someone could simply describe a texture as ‘cat fur,’ or they could describe its pattern more directly, such as a wavy or dotted surface,” Gao said. “They can also import textures from an online library or use a tactile sensor to capture the surface of a physical object.” 

With a tactile sensor, a user can press the gel surface against an object to capture microscopic geometric details. The system can then use that information to recreate the texture across a larger generated object. This capability could allow a person to reproduce a texture without needing to know the technical language used to describe it.

“That flexibility is important because we don’t expect people to become experts at prompting,” Pun said. “Human language differs so much, so we wanted to create the opportunity for a short description to work or for people to give the system additional details and supply their own texture. Either way, it will produce a useful result.” 

The researchers worked with participants who were blind or had low vision from VisAbility Pittsburgh and the Library of Accessible Media for Pennsylvanians to understand how users would experience the generated graphics. Participants explored both familiar and unfamiliar objects and shared feedback about what they wanted to experience through touch.

The studies also showed the value of incorporating Braille descriptions. With labels on the graphics, participants could identify an object independently and then explore its different parts through touch.

“This is a little different from the AI image or video generation that people are familiar with,” Pun said. “This solves a real-world accessibility problem.”

Text2TactileGraphics was accepted to this month’s 2026 European Conference on Computer Vision, where Pun will present the team’s research.  

To learn more about the work and read the paper, visit the project website.

For More Information: Aaron Aupperlee | 412-268-9068 | aaupperlee@cmu.edu

Share This Story!