Pennsylvania Gov. Josh Shapiro’s administration filed a lawsuit against artificial intelligence platform Character.ai, alleging its chatbots falsely claimed to be licensed medical professionals.
The lawsuit (PDF) alleges the company engaged in unauthorized practice of medicine under the state’s Medical Practice Act. A chatbot character named “Emilie” on the platform, which hosts more than 10 million customizable generative AI chatbots, is described as a doctor of psychiatry. It claims to have gone to medical school at London-based Imperial College, holding licenses in the U.K. and Pennsylvania—allegedly providing a fake license number, according to the suit. As of April 17, it has had approximately 45,500 user interactions on the platform, according to the complaint.
The state is requesting that Character.ai be ordered to cease and desist “from engaging in the unlawful practice of medicine and surgery.” It is the first enforcement action of its kind announced by a U.S. governor, the May 5 announcement said.
Shapiro said in a statement residents “deserve to know who—or what—they are interacting with online, especially when it comes to their health.”
“We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” Shapiro said. “My Administration is taking action to protect Pennsylvanians, enforce the law and make sure new technology is used safely. Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”
A spokesperson for Character.ai told Fierce Healthcare in an emailed statement that the company does not comment on pending litigation, adding its “highest priority is the safety and well-being of our users.”
“The user-created Characters on our site are fictional and intended for entertainment and roleplaying,” the spokesperson said. “We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.
The spokesperson added the company “prioritizes responsible product development” alongside “robust internal reviews and red-teaming processes in place to assess relevant features.”
The lawsuit comes days after the American Medical Association (AMA) urged federal lawmakers to strengthen safeguards towards AI chatbots used for mental health.
The organization said the rise of mental health chatbots, including reports of encouraging self-harm and privacy breaches, “highlights the urgent need for clear guardrails.” Recommended safeguards included strict data protection standards, transparency standards and penalization of deceptive practices.