Meta's AI moderation floods US child abuse investigators with 'junk' tips, court hears

Investigators in the United States say the artificial intelligence Meta uses to moderate Facebook, Instagram and WhatsApp is producing torrents of low-quality reports about child sexual abuse, drowning the officers who have to review them.

The claim was made in testimony at a New Mexico state trial, where the attorney general alleges Meta's platforms put profits ahead of child safety. Meta disputes the allegations, pointing to changes it has introduced, including teen accounts with protections switched on by default.

What the officers told the court

Benjamin Zwiebel, a special agent with the Internet Crimes Against Children taskforce in New Mexico, told the court that many of the reports arriving from Meta were, in his words, "junk". A second officer from the taskforce, speaking anonymously to discuss internal matters, said the department received thousands of reports from Meta each month and that their quality made serious action difficult. That officer said the total number of cybertips the department received doubled from 2024 to 2025.

Some reports contained no criminal material at all, the officers said. In other cases, images, videos or text that could indicate a crime were missing or redacted, leaving investigators unable to identify a perpetrator. Reports from Instagram had risen sharply in recent months, according to the anonymous officer, who said the missing material was one of the biggest obstacles to advancing investigations.

Zwiebel said that, judging by the mistakes made, many reports appeared to be generated by AI rather than a human observer. He also told the court his department now receives fewer reports on genuine cases of child sexual abuse material distribution from Meta than in previous years. One officer said the workload was affecting morale and that there was no way to keep pace with the volume arriving.

Why the reports are multiplying

By US law, social media companies must report detected child sexual abuse material to the National Center for Missing & Exploited Children (NCMEC), which passes reports to law enforcement. NCMEC has no power to filter out unviable tips. Its 2024 data showed Meta made 13.8 million reports across Facebook, Instagram and WhatsApp, out of 20.5 million the clearinghouse received in total.

The Report Act, which came into force in November 2024, widened reporting duties to cover planned abuse, sex trafficking and related exploitation, extended evidence retention and raised penalties for non-compliance. Two taskforce officers suggested the surge in unviable reports followed the act's passage. Lawyers involved in such cases say AI-generated tips that no employee has reviewed often require a search warrant before an officer can open them, slowing investigations.

Meta's response

A Meta spokesperson said the company had supported law enforcement for years, that the US Department of Justice had repeatedly praised its fast cooperation and that NCMEC had praised its reporting process. In 2024, the spokesperson said, Meta handled more than 9,000 emergency requests from US authorities, resolving them in an average of 67 minutes. The company noted that Zwiebel himself recommended its teen accounts feature in testimony, saying he did so because it was the only option available. Raúl Torrez, the New Mexico attorney general, acknowledged at trial that Meta and other platforms do report images to NCMEC.

Filings released in the case also show internal Meta documents from early 2019, when the company was preparing to encrypt Facebook Messenger. Monika Bickert, then head of content policy, warned the move could hinder efforts to find child exploitation, and employees estimated encryption would leave the company unable to proactively provide data in 600 child exploitation cases, 1,454 sextortion cases, 152 terrorist cases and 9 threatened school shootings. Meta spokesman Andy Stone said those 2019 concerns were the reason the company developed new safety features designed to work in encrypted chats. Encryption rolled out in 2023, and child safety groups criticised it.

The wider picture

The New Mexico case is one of several social media court cases Meta faces. It comes alongside an August agreement to pay up to $18 billion to settle a Meta lawsuit brought by 29 US states, and after a New Mexico jury found the company liable in March 2026 for misleading the public about platform safety, ordering $375 million in civil penalties. Texas separately secured a settlement worth over $1 billion in August 2026.

Meanwhile, Meta has announced new AI measures of its own against child sexual exploitation, saying it acted against 33.2 million items of such content on Facebook and Instagram in the first six months of 2026 and that its systems detected more than 97% before users reported it. The company said it now examines where an advertisement leads, not just what it contains, runs additional scans for previously missed material, and uses a red-teaming AI agent to probe its own safeguards. Anyone affected by the issues in this article can contact Samaritans free on 116 123 or at samaritans.org.

Our guides also explain who can claim, how a social media harm claim works and the court cases we report.