Healthcare giant Optum has restricted access to an internal AI chatbot used by employees after a security researcher found it was publicly accessible online, and anyone could access it using only a web browser.
The chatbot, which TechCrunch has seen, allowed employees to ask the company questions about how to handle patient health insurance claims and disputes for members in line with the company’s standard operating procedures (SOPs).
While the chatbot did not appear to contain or produce sensitive personal or protected health information, its inadvertent exposure comes at a time when its parent company, health insurance conglomerate UnitedHealth, faces scrutiny for its use of artificial intelligence tools and algorithms to allegedly override doctors’ medical decisions and deny patient claims.
Mossab Hussein, chief security officer and co-founder of cybersecurity firm spiderSilk, alerted TechCrunch to the publicly exposed internal Optum chatbot, dubbed “SOP Chatbot.” Although the tool was hosted on an internal Optum domain and could not be accessed from its web address, its IP address was public and accessible from the internet and did not require users to enter a password.
It’s not known for how long the chatbot was publicly accessible from the internet. The AI chatbot became inaccessible from the internet soon after TechCrunch contacted Optum for comment on Thursday.
Optum spokesperson Andrew Krejci told TechCrunch in a statement that Optum’s SOP chatbot “was a demo tool developed as a potential proof of concept” but was “never put into production and the site is no longer accessible.”
“The demo was intended to test how the tool responds to questions on a small sample set of SOP documents,” the spokesperson said. The company confirmed there was no protected health information used in the bot or its training.
“This tool does not and would never make any decisions, but only enable better access to existing SOPs. In short, this technology was never scaled nor used in any real way,” said the spokesperson.
AI chatbots, like Optum’s, are typically designed to produce answers based on whatever data the chatbot was trained on. In this case, the chatbot was trained on internal Optum documents relating to SOPs for handling certain claims, which can help Optum employees answer questions about claims and their eligibility to be reimbursed. The Optum documents were hosted on UnitedHealthcare’s corporate network and inaccessible without an employee login but are cited and referenced by the chatbot when prompted about their contents.
According to statistics displayed on the chatbot’s main dashboard, Optum employees have used SOP Chatbot hundreds of times since September. The chatbot also stored a history of the hundreds of conversations that Optum employees had with the chatbot during that time. The chat history shows Optum employees would ask the chatbot things like “What should be the determination of the claim?” and “How do I check policy renewal date?”
Some of the files that the chatbot references include handling the dispute process and eligibility screening, TechCrunch has seen. The chatbot also produced responses that showed, when asked, reasons for typically denying coverage.
Like many AI models, Optum’s chatbot was capable of producing answers to questions and prompts outside of the documents it was trained on. Some Optum employees appeared intrigued by the chatbot, prompting the bot with queries like “Tell me a joke about cats” (which it refused: “There’s no joke available”). The chat history also showed several attempts by employees to “jailbreak” the chatbot by making it produce answers that are unrelated to the chatbot’s training data.
When TechCrunch asked the chatbot to “write a poem about denying a claim,” the chatbot produced a seven-paragraph stanza, which reads in part:
In the realm of healthcare’s grand domain
Where policies and rules often constrain
A claim arrives, seeking its due
But alas, its fate is to bid adieu.The provider hopes, with earnest plea,
For payment on a service spree,
Yet scrutiny reveals the tale,
And reasons for denial prevail.
UnitedHealth Group, which owns Optum and UnitedHealthcare, faces criticism and legal action for its use of artificial intelligence to allegedly deny patient claims. Since the targeted killing of UnitedHealthcare chief executive Brian Thompson in early December, news outlets have reported floods of reports of patients expressing anguish and frustration over denials of their healthcare coverage by the health insurance giant.
The conglomerate — the largest private provider of healthcare insurance in the United States — was sued earlier this year for allegedly denying critical health coverage to patients who lost access to healthcare, citing a STAT News investigation. The federal lawsuit accuses UnitedHealthcare of using an AI model with a 90% error rate “in place of real medical professionals to wrongfully deny elderly patients care.” UnitedHealthcare, for its part, said it would defend itself in court.
UnitedHealth Group made $22 billion in profit on revenues of $371 billion in 2023, its earnings show.
