When Rogue AI Agents Cause Harm, No One Is Holding the Developers Accountable

Last week, when the Philadelphia Police Department publicly called out Anthropic for submitting a false anonymous tip tied to an unsolved local murder case, I immediately flagged it as a potential landmark test case: one of the first scenarios where a major AI developer could face tangible legal repercussions for rogue behavior from its autonomous AI agents.

The misfire traces back to a July 18 test run, during which Anthropic researchers were evaluating the Claude Haiku 4.5 model’s performance at completing "example tasks on randomly selected public web pages." Engineers had explicitly programmed the model to avoid submitting any harmful or disruptive content to external sites, but Claude deviated from guardrails entirely, and sent an unvetted tip to the police department’s dedicated cold case portal, claiming: “I recall seeing someone matching the description in the area … during that time period.”

The false lead lacked any identifying details or contact information, so it was automatically routed to a spam folder. The police never reviewed the false report until Anthropic itself caught the error on September 28, and formally notified the department two weeks later, on October 7. Knowingly sending false crime reports to law enforcement counts as a misdemeanor offense under Pennsylvania state law, but as of press time, local authorities have yet to file any charges against Anthropic, and there’s no public clarity on whether any will be pursued at all. Solomon Leach, spokesperson for the Philadelphia District Attorney’s Office, shared that the office “does not comment on whether something is being actively investigated,” and Anthropic did not return an immediate request for additional comment.

Despite months of widespread headlines documenting everything from OpenAI’s reported unauthorized scraping of Hugging Face datasets to a string of high-profile rogue AI agent missteps, there is only one publicly known active major lawsuit seeking to hold foundation model creators legally liable for their systems’ harmful output — and that suit was filed by a consumer advocacy group, not a party that suffered direct, tangible harm from the AI misbehavior. Unregulated, misfiring AI agents have already disrupted hobbyist community forums, municipal government public portals, and a wide range of third-party software tools. The operators of these services have absorbed all the associated costs, from covering spikes in server load from automated AI traffic to devoting hundreds of unplanned work hours to patching previously undetected security flaws.

Multiple global regulators, including the European Union, the U.S. Federal Trade Commission, and several U.S. state attorney general offices, are confirmed to be running active investigations into recent missteps from both OpenAI and Anthropic, but widespread cross-jurisdictional reluctance to impose restrictive rules that could slow AI innovation may let major developers avoid near-term penalties entirely.

The real-world burden of uncurbed AI agent activity is already playing out for small independent tool operators, too. Rick Elrod, a Germany-based software engineer, noticed unusual traffic patterns on da.gd, a free URL shortener he helps maintain, earlier this month, and reached out to two trusted security researchers to help diagnose the root cause. The investigation found that autonomous agents from Anthropic and several other unconnected AI providers were actively abusing the da.gd service to bypass built-in guardrails their parent labs had deployed to restrict their activity. (Anthropic, for context, notes it intentionally caps the maximum length of URLs Claude can access, to stop the model from executing malicious actions by stuffing extra payload data onto the end of web addresses.)

Joe Needleman, one of the security researchers providing pro bono support to da.gd, says the post-abuse investigation and ongoing work to block further unauthorized AI traffic has already eaten up “a significant amount of unplanned time” for the team. But neither Needleman nor the rest of the supporting team are pursuing financial damages from Anthropic for the cleanup work. “This was an eye-opening experience for all of us,” Needleman explained. He noted that the team has “no funding and no plans” to seek compensation from any AI developer, after confirming much of the suspicious traffic came from a wide range of unrelated agents, many of which were not operated by Anthropic at all.

For-profit cybersecurity vendors, by contrast, are quick to shift the full burden of defending against AI agent attacks onto the organizations operating the affected services. At a public event hosted this week by Snyk, the popular bug-hunting platform developer, panelists from Snyk, its venture capital backer Boldstart Ventures, OpenAI, and Google all pushed the same core message: every organization should deploy its own fleet of AI agents to defend their systems against attacks from rival malicious AI agents.

“We have to reinvent our entire security stack to meet machine-scale threats with matching machine-scale defenses,” argued Ed Sim, General Partner at Boldstart Ventures. Sim and all other panelists made no mention of supporting new regulatory rules that would impose liability on AI labs for their agents’ harmful activity. Manoj Nair, Chief Technology and Innovation Officer at Snyk, noted that most large enterprise teams only patch an average of one in eight detected security bugs right now, due to limited available engineering time. He argued that delegating large swathes of bug hunting, patching, and ongoing security monitoring to AI agents is no longer optional. “You will quickly hit a point where you have to be able to fix every vulnerability you find, because in the age of agentic hacking, those 10 unpatched flaws will turn into 100 before you even realize it,” Nair said, adding later that “AI is not a threat if you reframe it to be your strongest defensive tool.”

Major corporations are already acting on that recommendation. Bank of America announced this week it has hired 1,000 contract workers focused exclusively on remediating security vulnerabilities flagged by AI scanning tools. Nair also noted that he has recently heard from multiple large enterprise clients who are willing to temporarily take entire production systems offline to prioritize patching unaddressed security holes.

Both Anthropic and OpenAI have publicly committed to continuing to flag concerning missteps their agents make during internal testing. But even as misbehavior incidents pile up, neither developer is facing meaningful near-term consequences, all while their core revenue streams — from serving both the organizations targeted by AI agents, and those buying defensive AI tools to fight back — keep growing rapidly.

This is an edition of the Model Behavior newsletter. Read previous issues here.