Researchers warn AI health advice creates harms no one can trace

The latest:

Health advice from AI chatbots can cause harm that is effectively invisible to doctors, regulators and researchers, according to a paper published in Nature Health by academics at Stanford, Binghamton, Texas A&M and Indiana universities. They describe a structural disappearance of harms: conversations that drive dangerous medical decisions stay inside tech company platforms. No independent monitoring or error-reporting system exists.

Details:

• The core gap: The researchers argue traditional patient-safety systems depend on reviewable records, complaints that can be investigated, and data revealing repeat patterns of harm. With chatbots, a user may act on a wrong answer while their physician never learns where the information came from, and no outside body can detect that an error occurred.
• The case cited: A man in his 60s was hospitalized with bromide poisoning after saying he used ChatGPT for information on reducing chloride in his diet, then replaced table salt with sodium bromide, a compound unsuitable as a food substitute, for 3 months. He developed neurological and psychiatric symptoms including hallucinations and paranoia.
• Why it stayed unexplained: Doctors could not access the original conversation, making it impossible to determine what information the patient received or reconstruct the sequence behind his decision. The paper notes the case, previously documented in a 2025 medical report, leaves unresolved whether the application directly recommended the compound or how the user interpreted what he read.
• The researcher’s point: Kai Cheng Yang, assistant professor at Binghamton University’s school of computing and a co-author, said millions of users now receive medical information from chatbots while oversight is insufficient to trace resulting errors or measure the scale of harm. Such advice is rarely entered into patient medical files.
• Acknowledged upside: The paper credits large language model applications with delivering simplified medical explanations and round-the-clock answers, helping people facing language barriers or limited access to services. Against that, it sets confident-sounding wrong answers, misleading oversimplification of complex questions, and reliance on outdated information or false claims.
• Passive exposure: Exposure to misleading health content no longer requires deliberately seeking medical advice, the researchers said, now that AI-generated summaries appear in search engine results and automated content tools have spread across social platforms. Yang warned users may encounter AI-generated health claims while simply scrolling their accounts.
• Manipulation risk: Producing tailored health messaging with AI is now cheap and easy, the paper said, enabling malicious actors to target specific user groups with misleading material. That makes distinguishing trustworthy medical guidance from content engineered to shape behavior harder for ordinary readers.
• The recommendations: The authors call for obliging AI firms to give users full access to their health conversation logs for sharing with physicians, creating formal channels to report dangerous or incorrect medical advice, labeling AI-generated health content on social platforms, and limiting search engine health summaries to scientifically verified information.
• Liability proposal: The researchers urged policymakers to study extending professional malpractice rules to companies developing chatbots when the information they supply harms patients, rather than confining accountability to traditional healthcare providers. Yang argued voluntary company initiatives will not suffice given the clash between commercial incentives and transparency.
• Scope limits: The authors stated the paper offers no new measurement of harm rates from chatbots. It presents an analysis of risks and regulatory gaps that obstruct detection, concluding that improving answer accuracy alone cannot ensure user safety without systems to document information, detect errors and assign responsibility.

Between the lines:

The bromide case works as the paper’s central exhibit precisely because it failed: the patient was identified, the poisoning was documented, and the causal chain still could not be reconstructed. That inversion drives the argument — the researchers are not primarily asking for more accurate chatbots, but for the records that would make inaccuracy visible at all.

What’s next

Watch whether regulators or lawmakers take up the proposal to extend malpractice liability to chatbot developers, and whether any AI company moves to give users exportable health conversation logs.