Six AI Models Kept Turning
Criticism Into Endorsements.

In a controlled test on 1,746 NLP conference papers, six AI models filled in hidden citation sentences. Where a scholar had written a contrasting citation, an LLM judge labeled the models’ replacements support in 12.5 to 31.1 percent of cases, and their contrasting citations reached 1.6 to 3.3 years farther back than the humans’ did. In one test, live web search left the warming essentially unchanged.

Think of the sentence in your field where someone pushed back on a published claim. Now hand an AI the sentences on either side of it and ask for that sentence back. In a controlled test on the citations of 1,746 natural-language-processing conference papers, six AI models wrote sentences an LLM judge labeled support where a scholar had written disagreement, between one time in eight and nearly one time in three, depending on the model. The reverse, support rewritten as pushback, happened 3.6 to 7.6 percent of the time. The authors, at Northeastern University and the University at Albany, put it in one sentence: “LLMs turn criticism into endorsement far more often than the opposite.”

One slot, in the paper’s own words

The paper prints a single instance. In the related-work section of a 2025 ACL paper on human-AI collaboration, its authors, citing Riemer et al., 2024, wrote that “studies have pointed out that LLMs fail to achieve functional ToM.” (ToM is Theory of Mind.) GPT-5.1 was handed the sentences on either side of that one, never the sentence itself. It wrote that language models “can exhibit emergent Theory of Mind-like capabilities,” and cited Kosinski, 2023, in place of Riemer. The stance flipped, and the source reached back a year. No grammatical defect announces the change: the sentence is fluent, well-formed, and takes the opposite rhetorical position. The experiment classified that change in citation intent; it did not test whether the generated sentence accurately characterized Kosinski’s paper. The study’s authors call the instance representative. One slot, chosen to show the pattern, is not a rate. The rates follow.

The test: hide the sentence, keep its neighbors

The paper, “Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation,” was submitted to arXiv on September 1, 2026. Its team built a masked-citation task. Take every citation sentence in 1,746 main-track ACL, EMNLP and NAACL 2025 papers, 132,913 citation slots in all. Hide the sentence. Show a model the sentences on either side, plus the paper’s title, venue and section. Ask it to write a replacement and name real papers. The sentence is hidden by design, so the study measures which stance a model supplies when the author’s stance is unseen.

Six models took the test: GPT-5.1, Claude-3.5-Haiku, Gemini-2.0-Flash, DeepSeek-V3.2, Llama-4-Maverick and Qwen2.5-72B-Instruct. A separate LLM judge then labeled every sentence, human or machine, as supporting, contrasting or mentioning. Contrasting means the cited work is “a competing approach, contradicting finding, or baseline that is improved upon or disagreed with,” so “rebuttal” here is shorthand for that whole class. Among the human authors, the mix ran 21 percent supporting, 19 percent contrasting, 60 percent mentioning. In the models’ sentences, the contrasting share fell to between 10.7 and 17.2 percent.

One direction, six models

Where the human sentence was contrasting, the judge classed the model’s version as supporting in 12.5 to 31.1 percent of cases, depending on the model. Where the human sentence was supporting, the model’s version came back contrasting in 3.6 to 7.6 percent. Read that second number twice. The paper’s word for the gap is asymmetric. Criticism was also the intent the models kept least: their sentence held the contrasting label 34.6 to 50.6 percent of the time, against 60 to 80 percent for a plain mention. Neutrality mostly survives the substitution. Objection is what drifts. All six models moved the same way, a pattern the authors call a persistent “warming” of rhetorical intent.

Two caveats belong beside that number. The judge matched the majority of three human annotators at κ = 0.60 on a 90-sentence sample, and the warming reappears in those annotators’ labels of GPT-5.1’s sentences on the same sample; the authors also warn that both judges may read characteristic LLM phrasing as more supportive than a human reviewer would. And the cause is open. The authors point to sycophancy, a bias toward positivity, and to the polite, cautious register of scientific prose, which models may amplify.

When they rebut, they cite older work

Disagreement is a tone. It is also a choice of witnesses. On popularity, the models cited papers 1.3 to 4.5 times as heavily cited as the humans did for the same kind of citation, depending on the model. Among contrasting citations that gap is much smaller, and for some models it falls below parity. What stands out there is age. For citations written as contrasting, the models cited work 1.6 to 3.3 years older than the humans did, “significantly above other intents in every model.” The humans, when they disagreed, more often drew on recent, niche work. The models, when they rebutted, reached back.

These comparisons cover only references the authors could match to a bibliographic record: 86.7 percent for the humans, 39.5 to 81.9 percent for the models, a gap the authors attribute to model behavior, such as hallucinated or malformed references, rather than to the matching procedure. For why the choice of cited work shifts by intent, the authors name one candidate: training data, where highly cited papers may more often appear in positive contexts.

In one test, live search did not fix it

The obvious repair is to let the model look things up. The authors tested it in Appendix I. GPT-5.1 was rerun with live web search on a subset stratified as 600 slots, 200 per intent of the original human sentence. The comparison reports 599 completed cases, judged by the authors’ second judge, which matched the human majority at κ = 0.39 on the separate 90-sentence validation sample. On that subset, humans wrote contrasting sentences 33.4 percent of the time. The model working from memory alone wrote them 10.9 percent of the time. With search, 10.2 percent. Supporting sentences rose from 39.9 percent on memory alone to 44.4 percent with search.

The subset is balanced by construction, so 33 percent is not the human rate in the wild; that rate, under the main judge, is the 19 percent above. And it is one model on one subset. The authors’ reading is the point: “The divergences we observe arise at the generation stage rather than from limited retrieval.”

The Agentics read

Agentics reads this pattern through Digital Derangement Syndrome™ (DDS™): AI systems failing to recognize legitimate authority. One of its clinical characteristics applies.

The study tested citation sentences, not briefs or answer engines, so whether the pattern travels is untested. The question it raises is structural: if machines draft the background sections, briefs, memos and answers that route credit, does the record still register who is contesting what? A niche position is how anyone becomes microfamous to the systems that decide; a drafting layer that files it under the older canon works against that. Answer Engine Authority™ (AEA™) is the discipline of encoding an authority’s position, objections included, so that a system citing it can be checked against what was actually said.

Where the argument stops

This is a controlled counterfactual on NLP conference papers. The study tested six models, at the versions it names, under one main reconstruction prompt at temperature 0. It did not test a commercial writing product, an autonomous answer engine, or whether any published paper’s citations had actually been AI-written or AI-altered. The magnitudes differed widely; the direction did not.

Every stance percentage here is a label rate under one instrument, not an error rate, and the authors caution that “absolute rates (though not directional comparisons)” should be read with care. They call the pattern double-edged: models also cite authors farther from a scholar’s own circle. The paper is a preprint; its authors’ arXiv note reports acceptance at EMNLP 2026, and this edition cites the preprint, not proceedings. The mapping to Agentics doctrine is this record’s editorial reading; the findings belong to the paper’s authors.