What do early results show?

According to newly released data, Doctronic's AI recommended prescription renewal in 72% of cases, which then went to a human physician for review. In the remaining 28% of cases, AI escalated the case to a human physician due to case characteristics, including the need for new lab work or complications.

Among the cases where the AI suggested a prescription renewal, the reviewing physician agreed that a renewal was appropriate based on the AI's gathered information in 91% of cases. In the other 9% of cases, the reviewing physician wanted additional information, such as confirmatory lab results.

In 69% cases where AI escalated to a physician, the physician agreed that the escalation was appropriate. For the remaining 31% of cases, the physician determined the escalation was overly cautious.

"For a new system like this, overcaution is appropriate and welcome," the state's Office of Artificial Intelligence Policy noted in the data sheet. "In the long term, reducing overcaution without compromising safety would improve patient access to care, but we aren't rushing to see that happen."

Pilot raises safety concerns, courts controversy

The AI prescribing pilot has come under fire in the months since its launch. In April, the state's Medical Licensing Board sent a letter to the Office of Artificial Intelligence Policy stating that it was not made aware of the agreement launching the pilot until after the fact. The board asked that the program be suspended, pointing out that prescription renewals require "reassessment and clinical decision-making to safely adjust doses, monitor for side effects, contraindications, or new drug interactions, and ensure the medication remains effective."

In response, the Office of Artificial Intelligence Policy sent a letter to the board, highlighting that the pilot is in phase one. During that time, every AI recommendation will be sent to human physicians for review. Additionally, the letter noted that the AI is "strictly prohibited from handling controlled substances, modifying treatment plans or initiating new prescriptions."

However, independent researchers have also expressed concerns regarding the pilot's design. In a commentary published in JAMA Health Forum, Michelle Mello, Ph.D., of the Stanford Health Policy Institute, stated that the contract is murky regarding Doctronic's accountability if AI-based prescribing errors injure patients.

She also underscored the risk of "scope creep," that is, when an AI system has been accepted and implemented, vendors may be able to make significant changes to the tool without undergoing the same degree of scrutiny as the initial adoption.

"If Utah's pilot program is the proverbial camel's nose in the tent, it will be important for someone to be firmly holding the reins when the camel emerges," she wrote.

In an article for Health Affairs, Eric Bressman, MD, an assistant professor at the University of Pennsylvania's medical school, noted the need for a licensure framework for AI. He wrote that as AI tools increasingly function like human clinicians, the healthcare industry will need to develop a common set of standards to define its readiness for practice.

"Standardization emerged not to stifle innovation but to protect patients from uneven training and unproven practitioners," he wrote. "AI will undoubtedly shape the future of medicine, as it will every industry. The difference is that when we move fast and break things, the stakes are human lives."

Anuja Vaidya has covered the healthcare industry since 2012. She currently covers healthcare IT and innovation, including artificial intelligence, digital healthcare, EHRs and interoperability.