Researchers in the Carnegie Mellon University School of Computer Science have developed a tool that uses AI to analyze medical images, reducing the time healthcare professionals spend finding and labeling organs in scans.

Labeling medical images can be time-consuming, requiring experts to spend hours manually identifying specific organs or areas of the body. And it needs to be done repeatedly, depending on the types of scans or body parts involved.

AutoMiSeg simplifies that analysis and labeling by using text instructions to identify parts of a medical scan. For example, a physician could ask to locate the optical disc in an image of the eye fundus, which is the back part of the eyeball. Unlike existing tools, AutoMiSeg does not require intensive task-specific training to recognize a new organ or adapt to a different hospital scanner.

"AutoMiSeg leverages automation and large foundation models of medical images to help improve throughput in medical imaging analysis," said Min Xu, an associate professor in the Ray and Stephanie Lane Computational Biology Department. "We're reducing the number of required human interactions, which could accelerate the analysis of more biomedical images."

Researchers created AutoMiSeg by combining multiple foundation models into one automated system. It all starts with a text input, and then the system manually identifies an initial area in the medical image — for example, by tracing the optic disc in a retinal scan. From this initial rough image, the system sharpens it and makes it easier for an AI tool to understand what it's looking at and identify areas for analysis.

Once a more precise area is identified, the AutoMiSeg method adapts for differences in machines. Medical images can vary significantly between machines or areas of the body. To make the image more standard for better identification, AutoMiSeg automatically adjusts color and brightness or improves edge clarity without human supervision.

Finally, the system checks its work. The module estimates if the predicted area aligns with the requested target, ensuring that the AI hasn't accidentally selected the wrong organ before the final analysis is complete. For example, it will ensure that the optic nerve is selected on the eyeball scan.

"Typically, it's very time-consuming for a doctor or a medical expert to read the image and annotate the various areas required," said Xingjian Li, a computational biology post-doctoral associate. "And sometimes it's even inaccurate because different experts may have different opinions, meaning an area of interest's location isn't consistent among humans. Our work is designed to reduce this effort."

Xu noted this work is a building block for future research into automation and medical image analysis. Along with expanding this area of work, the researchers also plan to improve AutoMiSeg's accuracy.

In addition to Xu and Li, CMU team members included Qifeng Wu, a research intern; Runmin Jiang, a research associate; Emma Li, a research intern; and Colleen Que, an alumna of the Robotics Institute's Master of Science in Computer Vision program. The team also included Adithya Ubaradka from the National Institute of Technology Karnataka, Yiran Ding from Brown University, Jianhua Xing from the University of Pittsburgh, and Tianyang Wang from the University of Alabama at Birmingham. This research received support from the National Science Foundation.

To learn more about AutoMiSeg, read the team's paper.