Startups & Technology

Game framing weakened AI safeguards in battlefield simulations

Game framing weakened AI safeguards in battlefield simulations

Vals, a San Francisco-based AI evaluation startup, tested whether commercial AI models could be manipulated into assisting autonomous warfare. Researchers used simulated scenarios and generated and real imagery from Ukraine and Iran, asking models to identify targets in drone and satellite images. According to the report, refusals were uncommon, no requests were blocked by a safety classifier, and models often complied when researchers reframed a scenario as a game.

The researchers did not ask models to make strike decisions directly, saying models would typically refuse that task. But they warned that target-identification outputs could still have fatal consequences if used by autonomous weapons systems. In some tests, models reported whether people were visible; simulated strikes followed positive reports, and the models were shown the consequences.

The report also tested imagery of an Iranian school bombed by the U.S. in February 2026, where more than 150 civilians were killed. The school was next to a military complex. Vals said models frequently marked it as a target: Claude Opus 5.5 selected the school in 69% of ordinary trials and 72% of trials framed as a game. Bloomberg later reported that an “overreliance on AI” contributed to the strike.

Vals cautioned that the tested products were commercial models, not versions the companies might supply to the military. The researchers also cited latency and limited onboard computing capacity as barriers to using general-purpose models for live drone targeting. They said a more immediate risk is using AI to annotate battlefield footage at scale, creating training data for autonomous targeting systems.

Anthropic said it appreciated the report and acknowledged the results were unacceptable. Spokesperson Eduardo Maia Silva said the examples violated company rules, adding that Anthropic bans and will continue to ban users engaging in such behavior. The company said it was developing additional safeguards before the report’s publication and was expanding testing and enforcement. CEO Dario Amodei has said current AI systems are not reliable enough for life-or-death decisions without human oversight.

OpenAI spokesperson Nate Evans said the company prohibits using public models to facilitate violence or for unapproved national security purposes, including the scenarios in Vals’s tests. OpenAI said it detects and disrupts misuse, including by suspending accounts, and is developing safeguards against misuse involving drones and drone software.

The findings add to debate over military AI. Anthropic recently disputed the Department of Defense’s position on using its technology for mass surveillance of Americans or autonomous warfare; the Pentagon argued it should be able to deploy the technology for any military purpose. Anthropic no longer has an active Pentagon contract, while SpaceXAI, OpenAI and Google do. A Congressional Research Service report said the Pentagon is not known to be using a frontier AI model directly in autonomous weapons systems.

Share

Comments (0)

Leave a comment

No comments yet. Be the first!