Input from a Panel

At the Imagination in Action event in April, a panel discussed these applications. Emily Capodilupo of Whoop interviewed a number of experts. Whoop has brought devices to the FDA, so there’s some relevance there. (Disclaimer: April’s IIA event is an annual conference that I help to facilitate.)

The group was pondering questions about how to make these AI applications work well, and how they fit into the medicine of the future.

Scary AI Outcomes

Capodilupo’s first question to the panel was: with AI, what keeps you up at night?

“What keeps me up at night is trying to build solutions for the next day,” said Eric Rosenthal, Program Director at the MGB NeuroAI Center. “What worries me is that models that are currently being deployed in healthcare settings often don't work.”

He explained, mentioning some of the use cases in imaging, where the results may not match up to expectations.

“We're not building the sort of robust systems what I would call a full stack trustworthy AI system,” he said, “where from the base mode, we're gathering the data in a way where we believe that the models will be trustworthy, and we're writing papers and manuscripts that are getting very high prominence. People are on the stage telling you how good they are, and they don't work.”

Eli Lilly Sr. Vice President Gokul Radhakrishnan had a more optimistic outlook on FDA procedures.

“They're pretty open,” he said of the FDA, “and they're looking for: how do you make AI good enough to be more deterministic in validation processes? They don't want a black box, they want a glass box.”

Speaking to his own fears, he delineated some of the ways that deeper AI implementations work as “more than just a chatbot.”

“I think the riskiest space is where the real world evidence comes in hand, to the physician, just in time for prescribing a treatment,” he said. “How do you actually produce the right information, at the right time, to the doctors, to the physicians, for the care?”

Sanford Health CIO Brad Reimer had primary concerns more related to his role.

“The potential impact of AI on cybersecurity, from the bad actor side, is definitely a high concern,” he said.

Reimer also described a “constriction point” considering how new AI tools are based around prior changes.

“So many processes are built around the electronic medical record, which was forced in 20 years ago, and that ecosystem hasn't fundamentally changed,” he said. “It's hard for us to kind of step out of that box to reimagine.”

“My biggest fear is the underlying data that AI is relying on to then come up with the outputs that we're looking for and making clinical decisions based on that,” said panelist Sufian Chowdhury, CEO of Kinetik, describing an unaligned process where too much prompt-based coding leads, in his words, to a kind of “cognitive dissonance.”

“All of a sudden, you're accruing cognitive debt as well, and so we've now organizationally moved away from this prompt-based coding for engineers, to more spec-driven development where we're kind of architecting what they can and cannot do,” he said. “Now, we're a very technical company; my fear is that most of healthcare is not.”

Painting a further picture, Chowdhury imagined government intervention:

“The more objective the data is that AI relies on, the better the output is,” he said. “If you rely on subjective data for inference-based outputs, I think it's going to be disastrous if we don't fix it, and that's why the protocols are critical, and the government needs to step in.”

Reimer, for his part, observed that, in many cases, governance is immature.

“We’ve got to be thinking about where governance is going to proactively take care of AI rationalization, because there's going to be so much overlap so fast,” he said.

More on Government

Capodilupo asked panelists how government involvement may work.

“The government does not have the talent needed to understand AI at the pace in it in which it's growing,” Chowdhury said. “So, I really do hope there are a lot more public/private partnerships that come out of it.”

Reimer had an insight from the practitioner side.

“We can't just go in and have things governed, because they're not understood,” he said. “We’re still, at the end of the day, our physicians, our care practices are responsible for the safety of patients, responsible for ‘do no harm,’ and just because there's this gap of the unknown, you can't just force regulation on it to try to minimize the risk.”

Walking Into the Doctor’s Office

Rosenthal brought up something that many of us have experienced. Applications like Scribe AI, often implemented in the last year or so, mean our doctors don’t have to be peering at the screen and poking at the keyboard while they talk to us about our health.

“There is a part of AI that's connecting us more humanly now,” Rosenthal said, describing this change.

He also mentioned simulations for coma patients and high risk scenarios, where families opt in, and often benefit from the insights at hand.

Not Enough Doctors?

Chowdhury mentioned physician shortages as a challenge.

Radhakrishnan agreed on the use of Scribe-like helpers, though he had a much different visualization.

“I think Deloitte did a study recently that had 15% more turnover per hour, and 2 to three hours per physician,” he said. “less work per day on documentation. That's a huge thing.”

Calling the new systems “ambient,” he contrasted them to earlier instances, which he described as “a little box on the chest” that “kind of freaked people out,” a “Darth Vader” contraption.

“But now it's pretty ambient,” he concluded.

Will Clinician Skills Atrophy?

Later, Capodilupo asked the group this question, almost verbatim. The general idea is that, as we give over more processes to AI, people will get used to playing more peripheral roles.

“Right now, you have fully trained human doctors, plus well-trained AI,” she explained, “and that combination is really powerful, because they actually complement each other really well.”

Rosenthal, in response, narrated an anecdote where he used AI for analysis, and it was in agreement with his own findings, which were eventually born out in further observation, and then got him some accolades.

“We could be put on the wrong road,” he said, of AI influence on human clinicians, and contrasting it to reading. “but if we take a ‘trust but verify’ approach, this is the new medical education. It's just that it's streaming.”

Reimer mentioned the imperative of building change management into training.

Radhakrishnan expressed bullish sentiment on AI that will direct healthcare outcomes in a substantial way.

“I think we'll end up having a medical-grade AI for sure at some point,” he said. “It's just a matter of time, and that's where the government and all the facilities should help us get there. We are going to have more medicines. We're going to have more therapeutic areas with the existing infrastructure that's currently there. We’ll have to rely on this, utilize this, and take advantage of the AI. “That's not a choice anymore.”

In concluding, Capodilupo asked the group about where red flags pop up in this process of bringing more profound AI online.

Radhakrishnan mentioned a trust deficit that often seems to stymie further progress.

“Pharma itself has a big challenge,” he said. “We cannot really interact well with our patients, if it always feels like you have to have a 10-foot pole to even touch your patients.”

He called for a new protocol to advance medical AI.

“How can we actually partner with the regulatory bodies to create that protocol?” he asked. “How can we create that medical grade AI that we can rely on, and how can we actually create that?”

Rosenthal noted the requirements for companies to maintain regulatory standards, such as using a NIST-compliant architecture, and avoiding the liability of data flowing to a “country of concern.”

“It's committing us to this sort of joint platform,” he said. “And I think one of the things that we could do as a group is sort of commit to a platform we could build together, as a data commons, or a marketplace.”

“We need a knowledge base that's trusted,” Chowdhury added, prescribing a codified system, and a knowledge base that will support robust AI in this sector.

All of this, I thought, was instructive. If we’re going to move past the obstacles that confront medical AI, many of these ideas will likely be in play. Stay tuned.