In biomedicine, segmentation involves annotating pixels from a crucial structure in a medical image, like an organ or cell. Artificial intelligence models may help clinicians by highlighting pixels that will show signs of a certain disease or anomaly.
Nevertheless, these models typically only provide one answer, while the issue of medical image segmentation is usually removed from black and white. Five expert human annotators might provide five different segmentations, perhaps disagreeing on the existence or extent of the borders of a nodule in a lung CT image.
“Having options may help in decision-making. Even just seeing that there may be uncertainty in a medical image can influence someone’s decisions, so it’s important to take this uncertainty into consideration,” says Marianne Rakic, an MIT computer science PhD candidate.
Rakic is lead writer of a paper with others at MIT, the Broad Institute of MIT and Harvard, and Massachusetts General Hospital that introduces a brand new AI tool that may capture the uncertainty in a medical image.
Often known as Tyche (named for the Greek divinity of likelihood), the system provides multiple plausible segmentations that every highlight barely different areas of a medical image. A user can specify what number of options Tyche outputs and choose probably the most appropriate one for his or her purpose.
Importantly, Tyche can tackle latest segmentation tasks without having to be retrained. Training is a data-intensive process that involves showing a model many examples and requires extensive machine-learning experience.
Since it doesn’t need retraining, Tyche may very well be easier for clinicians and biomedical researchers to make use of than another methods. It may very well be applied “out of the box” for a wide range of tasks, from identifying lesions in a lung X-ray to pinpointing anomalies in a brain MRI.
Ultimately, this method could improve diagnoses or aid in biomedical research by calling attention to potentially crucial information that other AI tools might miss.
“Ambiguity has been understudied. In case your model completely misses a nodule that three experts say is there and two experts say just isn’t, that might be something it is best to concentrate to,” adds senior writer Adrian Dalca, an assistant professor at Harvard Medical School and MGH, and a research scientist within the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).
Their co-authors include Hallee Wong, a graduate student in electrical engineering and computer science; Jose Javier Gonzalez Ortiz PhD ’23; Beth Cimini, associate director for bioimage evaluation on the Broad Institute; and John Guttag, the Dugald C. Jackson Professor of Computer Science and Electrical Engineering. Rakic will present Tyche on the IEEE Conference on Computer Vision and Pattern Recognition, where Tyche has been chosen as a highlight.
Addressing ambiguity
AI systems for medical image segmentation typically use neural networks. Loosely based on the human brain, neural networks are machine-learning models comprising many interconnected layers of nodes, or neurons, that process data.
After speaking with collaborators on the Broad Institute and MGH who use these systems, the researchers realized two major issues limit their effectiveness. The models cannot capture uncertainty and they need to be retrained for even a rather different segmentation task.
Some methods try to beat one pitfall, but tackling each problems with a single solution has proven especially tricky, Rakic says.
“If you ought to take ambiguity into consideration, you regularly need to use an especially complicated model. With the strategy we propose, our goal is to make it easy to make use of with a comparatively small model in order that it might probably make predictions quickly,” she says.
The researchers built Tyche by modifying an easy neural network architecture.
A user first feeds Tyche a number of examples that show the segmentation task. For example, examples could include several images of lesions in a heart MRI which were segmented by different human experts so the model can learn the duty and see that there may be ambiguity.
The researchers found that just 16 example images, called a “context set,” is enough for the model to make good predictions, but there isn’t any limit to the variety of examples one can use. The context set enables Tyche to resolve latest tasks without retraining.
For Tyche to capture uncertainty, the researchers modified the neural network so it outputs multiple predictions based on one medical image input and the context set. They adjusted the network’s layers in order that, as data move from layer to layer, the candidate segmentations produced at each step can “talk” to one another and the examples within the context set.
In this fashion, the model can be certain that candidate segmentations are all a bit different, but still solve the duty.
“It’s like rolling dice. In case your model can roll a two, three, or 4, but doesn’t know you will have a two and a 4 already, then either one might appear again,” she says.
In addition they modified the training process so it’s rewarded by maximizing the standard of its best prediction.
If the user asked for five predictions, at the top they will see all five medical image segmentations Tyche produced, regardless that one is likely to be higher than the others.
The researchers also developed a version of Tyche that may be used with an existing, pretrained model for medical image segmentation. On this case, Tyche enables the model to output multiple candidates by making slight transformations to pictures.
Higher, faster predictions
When the researchers tested Tyche with datasets of annotated medical images, they found that its predictions captured the range of human annotators, and that its best predictions were higher than any from the baseline models. Tyche also performed faster than most models.
“Outputting multiple candidates and ensuring they’re different from each other really gives you an edge,” Rakic says.
The researchers also saw that Tyche could outperform more complex models which were trained using a big, specialized dataset.
For future work, they plan to try using a more flexible context set, perhaps including text or multiple kinds of images. As well as, they need to explore methods that would improve Tyche’s worst predictions and enhance the system so it might probably recommend the perfect segmentation candidates.
This research is funded, partially, by the National Institutes of Health, the Eric and Wendy Schmidt Center on the Broad Institute of MIT and Harvard, and Quanta Computer.