Doctors in the state of Utah are challenging an artificial intelligence-powered prescription medication refill system they say puts patient safety at risk.
In January, the state inked a partnership with AI doctor startup Doctronic to test out an AI-powered system to “automate routine, guideline-based prescription renewals” for Utah residents. It marked the first test of AI as an autonomous clinical decision-maker under a regulatory suspension paradigm, as Fierce Healthcare reported in January.
Rather than wait days or weeks to get an appointment with their doctor to get a routine medication refilled, Utahns can chat with an AI agent online to renew their prescriptions nearly instantly.
The AI system allows 30-, 60-, or 90-day renewals for medications already prescribed by a licensed provider, according to a mitigation agreement (PDF) between the Office of Artificial Intelligence Policy for the state of Utah and Doctronic. The company has limited the types of medications it will refill to 191 commonly prescribed drugs. The formulary was reviewed by independent pharmacists and the state.
The Utah Medical Licensing Board said it was made aware of this agreement only after its implementation, once the system was already live and available for use, it wrote in a letter (PDF) to the Utah Department of Commerce.
"Overseeing prescription refills is a task reserved for properly licensed medical practitioners for critical safety and clinical reasons. Each refill requires reassessment and clinical decision-making to safely adjust doses, monitor for side effects, contraindications, or new drug interactions, and ensure the medication remains effective. Patients who continue refilling medications without assessment may remain on outdated or suboptimal therapy for months or years. There is a reason prescription refills require physician authorization," the medical board wrote.
Proceeding with the AI-powered pilot project without consulting the medical board "potentially places Utah citizens at risk and remains a major concern of the board," according to the letter, dated April 20 and signed by 11 of the 14 physicians on the medical board.
"It is imperative that professionals with medical backgrounds review all proposals prior to implementation to ensure these programs do not compromise patient safety. We must not allow AI or other financial motivations to override this obligation, yet that is precisely what occurred here," the medical board wrote.
The medical board wants the program "immediately suspended" pending further discussion.
In a statement to Fierce Healthcare, a Doctronic spokesperson said, "We are participating in the process as designed, with defined safeguards, physician oversight of every prescription in the first phase of the program, and continued physician involvement throughout. We remain focused on demonstrating safe, evidence-based expansion of routine care access."
In a response letter dated April 21, Utah officials sought to address the medical board's concerns, noting that companies requesting a regulatory mitigation agreement submit a "detailed deployment and safety plan, which is thoroughly vetted."
"For any agreement involving a healthcare application, the OAIP (Office of Artificial Intelligence Policy) consults with practicing medical specialists in that specific field, as well as public health experts and regulators, to ensure the technology contains the requisite safety guardrails," state officials said in the letter, which was written jointly by OAIP and the Division of Professional Licensing (DOPL).
The pilot involving Doctronic was "rigorously reviewed by several medical professionals prior to launch," state officials asserted. "This evaluation process generated a large number of suggested substantive adjustments and guardrails, many of which were integrated into the pilot. As we communicated in a recent email thread, we look forward to involving the Board more in our vetting and oversight process as we evaluate this pilot and consider future pilots."
The pilot is currently in phase one, and every AI-generated prescription renewal is reviewed and approved by a licensed human physician before transmission to a pharmacy, according to state officials in the letter.
"Any cases of disagreement between the automated system and the reviewing clinician are further reviewed, analyzed, and assessed for risk using an established academic scoring procedure," according to the letter.
These consistent safety standards must be demonstrated before the program moves to phase two, where clinician review is required soon after the prescription is issued, state officials said. If the AI system is deemed safe in phases one and two, the pilot then moves to phase three, which allows the AI to "renew prescriptions semi-automously (sic) with continued clinical oversight in the form of random sampling of the AI outputs to ensure continued adherence to standards," state officials said.
State officials assert that in all phases, the AI system is required to conduct a comprehensive medical assessment that "mirrors human clinical decision-making, including verifying the medication, screening for new side effects, analyzing drug interactions, confirming allergy status, and evaluating continued effectiveness."
The AI system is "strictly prohibited" from handling controlled substances, modifying treatment plans, or initiating new prescriptions. Patients also must be evaluated by a physician in-person or via telehealth at regular intervals.
"If a case falls outside established guidelines, presents clinical complexity, or involves conflicting information, the AI is required to automatically escalate the encounter to a physician. Patients and pharmacists also retain a persistent option to request human physician review," state officials wrote.
The state plans to continue the pilot, noting that OAIP "retains the absolute authority to modify or cancel the pilot if safety benchmarks are not met.
The Department of Commerce and DOPL said it wants to work collaboratively with the Utah Medical Licensing Board, which works in an advisory capacity to the department, to review the pilot's progress, evaluate the system's performance and evaluate future healthcare proposals.
In a recent blog post, Doctronic co-founders and co-CEOs Matt Pavelle and Adam Oskowitz wrote that "the status quo is the biggest risk."
"The question is not whether AI is perfect. Neither AI nor human doctors will ever be perfect. The question is whether it's safer than what we have now—which is patients going without medication because they can't get a timely renewal. 125,000 people die each year from medication non-adherence. That's not a statistic about AI. That's a statistic about the system we inherited," Pavelle and Oskowitz wrote.
"In Utah, we're still in Phase 1: where every renewal is reviewed by one of our physicians before it reaches a pharmacy. We'll earn the right to operate fully autonomously by proving that we should. This is how trust gets built," the co-founders wrote.
Pavelle and Oskowitz contend that healthcare is facing a human capital problem with a shortage of primary care doctors and rising demand for care that results in patients waiting weeks to see a clinician.
"The greatest risk in healthcare is continuing to rely on a system that cannot meet current or future demand. Expanding access requires rethinking how care is delivered, with AI playing a central role in managing routine tasks while physicians focus on complex decision-making. With careful implementation, measurable outcomes, and ongoing validation, this approach can improve access, reduce costs, and better align healthcare delivery with patient needs," the Doctronic co-CEOs wrote.