Studies of model updates show how changes can affect routine tasks. Stanford and UC Berkeley researchers Lingjiao Chen, Matei Zaharia and James Zou reported that GPT-4’s accuracy on a prime-number task fell from 84.0% in March 2023 to 51.1% in June; its chain-of-thought accuracy boost dropped from 24.4% to -0.1%. On a happy-number task, accuracy declined from 83.6% to 35.2%. In another test, directly executable code fell from 52.0% to 10.0%, while answer-extraction performance dropped from 99.5% to 0.5% under the same model name and API.
A 2024 study by researchers from Carnegie Mellon and HKUST found that 58.8% of prompt-and-model combinations lost accuracy after an API update. Of those declines, 70.2% exceeded five percentage points. The article also cites a 2025 paper by researchers at Oxford and the Alan Turing Institute, which warns that providers can retire models, change serving systems and collect prompts under commercial licenses—even when a customer’s pinned model ID remains unchanged.
The article points to complaints about Claude Code as an example of changes beyond the underlying model. In April 2026, Anthropic said it had lowered default reasoning effort from high to medium, introduced a bug that discarded earlier reasoning after idle sessions and added a brevity instruction that hurt coding output. It also cites Anthropic’s acknowledgment of infrastructure bugs in September 2025, including a routing error that grew from 0.8% of requests to 16% of Sonnet 4 traffic and affected roughly 30% of active Claude Code users. Separately, Anthropic is defending against a California federal class action alleging degradation of paid Claude tiers between March 4 and May 6, 2026; the allegations have not been established in court.
For businesses, the proposed alternative is to run open-weight models on hardware they control, particularly for tasks such as drafting, classification, summarization, contract review and coding assistance. The article recommends recording outputs and error rates for a ten-prompt baseline, reviewing results monthly, comparing 90 days of API and repair costs with local hardware, and seeking contractual notice of changes and data-sovereignty protections. Its central warning: without control or clear disclosure, a provider’s update can turn a working workflow into an unplanned expense.

Comments (0)
No comments yet. Be the first!