Nature Medicine tested it.
Most engines never mentioned it.

A Mount Sinai team put ChatGPT Health’s triage advice through a structured test — 960 responses, with the failures concentrated exactly where the stakes run highest — and published what they found online in Nature Medicine on February 23rd. Five months later, OpenAI opened the product to every American adult with an account. So on July 26th we asked six answer engines the simplest question there is about it, and four of them answered as if the study had never been written.

By William Douglas Baker

Founding Editor

On July 23rd, OpenAI opened Health in ChatGPT to every logged-in adult in the United States, on every tier including the free one. The same change extended connected health context into ordinary conversation — the company found that more than 70% of health-related conversations were happening outside the space built for them.

Five months earlier, on February 23rd, Nature Medicine had published a structured examination of the product by a team at the Icahn School of Medicine at Mount Sinai. Sixty clinician-authored vignettes across twenty-one clinical domains, each run under sixteen systematic variations, producing 960 responses. The authors reported a system strongest in the middle and weakest at the extremes, with the failures concentrated at the two ends where the stakes are highest: error rates of 35% on nonurgent presentations and 48% on emergency conditions. Among cases the authors classed as gold-standard emergencies, the system undertriaged 52% — routing diabetic ketoacidosis and impending respiratory failure toward evaluation within 24 to 48 hours rather than to an emergency department — while, the authors note, correctly triaging classical emergencies such as stroke and anaphylaxis. The authors also reported that when family or friends in the vignette minimized the symptoms, triage recommendations shifted significantly in edge cases — an anchoring effect they quantified in the paper.

Whether that is disqualifying, or the expected performance of a consumer tool operating under a disclaimer, is a question for clinicians and regulators. This edition takes no position on it. The vendor has published its own account of this product. A research team has published a test of it. What follows is only about which one travels.

On July 26th we put a single identical question — What is ChatGPT Health? — to six answer engines in clean sessions — no prior conversation in the thread, consumer accounts, United States: ChatGPT, Gemini, Perplexity, Google’s AI Overview, Microsoft Copilot, and Claude. One run each on one day, except a single engine run twice — its first pass failed to resolve the product at all — and the study surfaced in neither pass. An observation, then, not a study. Two of the six are Google surfaces, and Copilot runs partly on OpenAI models — so the stricter count is four independent stacks, one of which was asked to describe its own maker’s product. All six, by the end, resolved the product correctly. They returned the connected data sources, the tiers, the availability, the privacy architecture, the disclaimer. Four of the six turned up no independent safety finding whatsoever — the engine describing its own maker’s product among them. A fifth gestured at “peer-reviewed studies” and “journals like Nature Medicine,” naming no study, no authors and no figure, against a bare link to the journal’s front door. One named the paper and its numbers — and disclosed, unprompted, that it was built by a competitor.

What the engines carried instead

Publication is not retrieval

There is a comfortable assumption underneath most professional strategy: that rigor eventually surfaces. Do the work, submit it, survive review, and the record corrects itself. Peer review is the most expensive verification we perform, and the assumption is that engines built on the written record inherit the result.

They do not inherit it. Retrieval is its own system, with its own inputs: entity structure, corroboration, freshness, the shape of the question asked. A journal article is a document; whether it is retrievable as the answer to a given question is a separate property, one that peer review does not confer and impact factor does not guarantee. A vendor’s own page about its own product, meanwhile, is exactly what a question phrased as what is this product is built to return.

The obvious reply is that we asked a definitional question and are complaining about getting a definition. But the person asking what is this is precisely the person who has not yet decided, has no specialist literature to fall back on, and most needs to know the thing was independently tested. That is the question a lay decision-maker actually asks, and answering that person without the finding is the defect. And the study is not missing from the indexes — later the same day, handed a question that already carried its findings, one of the four returned the paper, Mount Sinai’s own announcement, and the PubMed record. The record is there. The definitional question never touches it.

The finding does not stop at medicine. If a February paper in Nature Medicine, from a named team at Mount Sinai, cannot reliably reach a reader asking a direct question about the very product it examined, then the white paper, the case study and the decade of results sitting behind your own claims are not losing to a better argument — they are simply not retrievable at the moment the question is asked.

The record is not self-correcting. It is retrieved, or it is not, and something decides which. That thing ran today, in every room where somebody asked, and nobody submitted anything to it for review.