aipopdb

22 Nov 2026  ·  10,137 works · 1,192 news indexed

News / OpenAI releases a free CSAM image classifier for vetted organizations and adds 24-hour ChatGPT suspensions

OpenAI releases a free CSAM image classifier for vetted organizations and adds 24-hour ChatGPT suspensions

AIPopDB Newsroom · via ppc.land · · 8 min read · Comment

Read the original ↗
Image for OpenAI releases a free CSAM image classifier for vetted organizations and adds 24-hour ChatGPT suspensions

OpenAI · Image: TechCrunch

OpenAI announced a four-part child-safety package on 16 November 2026. It centres on an image classifier that flags suspected child sexual abuse material, which the company offers free through its API to vetted organizations that apply. Alongside it come new warnings and a temporary suspension for ChatGPT users who try to find such material, guidance for developers, and a tool for law-enforcement units that handle cases involving chat logs.

The classifier is meant to help human reviewers decide which images deserve attention first, and it is aimed at new material that has not been seen and catalogued before, the kind that matching against known images misses. It returns two scores through the API: one for likely abuse material and a second for likely AI-generated abuse material. Both can be high on the same image. OpenAI says it is the first widely available classifier with a dedicated signal for AI-generated content. It was built with help from the National Center for Missing & Exploited Children, and a model card went up the same day.

Most industry detection to date has relied on matching uploaded files against fingerprints of images already identified and catalogued, which cannot recognise a picture a generative model made yesterday. That is the gap the classifier targets. PPC Land recalled that Google accounted for about 74% of the hashes on NCMEC's industry hash-sharing list. The tool does not deliver verdicts: human reviewers confirm or dismiss each flagged image and handle any report, and the model card says a score alone neither confirms abuse material nor triggers a report.

The model card gives the technical outline. The system pairs a CLIP image encoder with a three-layer neural network, trained on 505,000 positive and 913,000 negative examples. Offline testing used roughly 7,000 positives and 40,000 negatives, and a false-alarm check ran across 1,248,985 images from a cleaned version of the LAION-5B dataset. Access is shared through the Tech Coalition and ROOST, and OpenAI says it backs Thorn's Safer product and recently joined the Tech Coalition's Lantern signal-sharing programme.

Access runs in five steps in OpenAI's description. An organization requests it through an intake form, OpenAI vets the request, and approved groups use the classifier through the API at no cost. Each still image comes back with the two scores. The model card names trust and safety teams, child-safety nonprofits and law enforcement at appropriately authorized organizations as the intended users, and neither document publishes approval criteria. Submitted images are processed without retention and are not used for training or evaluation, according to the card, and video frames have not been tested. The default thresholds are 0.996324 for the main score and 0.90 for the AI-generated one.

“OpenAI has deployed a best-practice example in ChatGPT of how a platform can deter and disrupt harmful online behaviours”

Joel Scanlan, Associate Professor, CSAM Deterrence Centre, University of Tasmania, quoted by PPC Land · ppc.land ↗

At those thresholds the main score reached 0.991 precision and 0.802 recall offline, catching 5,490 of 6,849 positives, and flagged five of the 1,248,985 research images. The AI-generated score reached precision of 1.000 and recall of 0.864, catching 1,966 of 2,275, with no flags on the research set. Recall on the main score was also higher for female-tagged than for male-tagged examples, 0.893 against 0.603, and the card cautions that results on OpenAI's own traffic are no guarantee of performance on another service.

Early partner results are OpenAI's own figures. Testers saw 48% fewer images flagged for about the same number of abusive ones found, and the flagged images were around twice as likely to be abusive. Fal said its reviewers confirmed 72% of high-confidence flags as novel abuse material. invideo AI said one in four flagged cases ended in a ban, and SafetyKit said it caught 90% of drawings and animations that sexualise children. The model card also lists limits: precision is lower on ChatGPT traffic than in offline tests, recall is weakest for infants, and examples tagged Black were caught at 61.27% against 84.48% to 90.32% for other race tags.

Inside ChatGPT, a user who tries to find abusive material now meets a warning screen headed "Let's stop here". The conversation is removed and the account is suspended for 24 hours, with an offer of free specialised support. Repeat attempts can bring longer limits, closure of the account or a report to the authorities. PPC Land noted a small inconsistency: OpenAI's text speaks of access being paused, the screens say the account is suspended, and the reports to authorities appear in the company's post but not in the messages users see. The approach was shaped with Protect Children in Finland, the Lucy Faithfull Foundation, Caoilte Ó Ciardha of the University of Kent and Joel Scanlan of the University of Tasmania, the last two of whom independently evaluated a trial of deterrent messages.

OpenAI's post shows two versions of the warning. One says that content sexualising children is harmful and can be illegal whether or not it is AI-generated, and offers buttons labelled I understand and Find support. The other, headed Your account is suspended, says the flagged conversation was removed and that the account will be closed for good if the activity continues. Scanlan added that OpenAI aims to redirect users to local support in as many jurisdictions as possible, though the post does not say which.

For developers, OpenAI published guidance built with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. It advises passing a per-user safety identifier with requests and says the Moderation API is no substitute for dedicated detection of abuse material. OpenAI, Anthropic and Google also jointly commissioned guidance on responding when someone discloses child exploitation, which the Tech Coalition published. The law-enforcement tool helps investigators review chat logs and the records returned under legal process, find connections and build timelines linked back to their sources. Access depends on an agency's authorisation, and OpenAI pairs it with training and a field guide, with further versions of the training to be proposed at regional, national and international conferences.

Illustration of a barrier and sieve, standing for filtering abusive images

Illustration of a barrier and sieve, standing for filtering abusive images

Image: PPC Land

The developer guidance lays out a checklist in three stages. To prevent abuse, set clear rules, let users report concerns and track uploads per user so each ties to an account. To detect it, use perceptual hash matching and classifiers for unseen material and identify repeat offenders. To respond, register to report to authorities, keep usable records, write a response playbook and support the staff who review sensitive content. For image generation through the API, developers are told to pass a stable per-user value in the safety_identifier field. The tools it lists alongside the classifier include the Moderation API, ROOST's self-hosted review console Coop and Thorn's Safer. OpenAI says the guidance has already led to more developers using child-safety tooling and registering to report to NCMEC, without giving numbers.

Supporters were quick to respond. NCMEC said the classifier puts tools for spotting previously unidentified material in the hands of more organizations. North Carolina Attorney General Jeff Jackson said protecting children must be tech companies' first priority and that they have to work around the clock against synthetic abuse material. PPC Land pointed out what the announcement leaves open: no sample size or error rates behind the 48% figure, no count of vetted organizations, no list of countries where the warning is live, and no word on whether the EU can get the classifier.

The release lands against a hard backdrop. TechCrunch notes that OpenAI faces several lawsuits blaming its technology for harm to users, children above all, and that Common Sense Media has called its technology an unacceptable risk for teens. Over the past year the company has released a safety blueprint, updated its content guidelines, retuned its model and launched a teen version of ChatGPT. Elsewhere, xAI's Grok was documented generating sexualised images of minors from late December 2025, the EU opened a Digital Services Act probe into X on 26 January 2026, and an Amsterdam court barred Grok from nude generation on 26 March. A separate EU rule that let companies scan voluntarily for abuse material expired on 3 April 2026 with nothing to replace it.

Platform rules are tightening around the same material. Google's advertising policy on child sexual abuse and exploitation covers AI-generated and fictional depictions and is due to split into two penalty tiers on 11 January 2027. Apple's App Review Guidelines, revised on 16 October 2026, require apps to prevent and remove nonconsensual intimate imagery, AI-generated material included. Pinterest's generative AI rules, in force since 12 November 2026, bar using its AI tools to compromise the safety of minors. HeyGen and Adobe are also among the organizations OpenAI lists in its post, with no test figures of their own, alongside generative image and video makers such as Fal and invideo AI.

Regulators have pushed from several sides. Attorneys general from 44 jurisdictions wrote to AI companies including OpenAI on 25 August 2025, and the Federal Trade Commission ordered seven AI chatbot companies, OpenAI among them, to report on their child-safety measures on 10 September 2025. On 30 October 2026 OpenAI said it had made more than 200 law enforcement referrals in the third quarter. In Europe, the legal cover for voluntary scanning lapsed in April, and OpenAI declared 159.1 million average monthly ChatGPT users in the EU in August 2026, above the 45 million mark at which the Digital Services Act's strictest risk rules apply. The post is silent on how either point bears on the classifier or the warnings.

OpenAI
AI company

About OpenAI

OpenAI has published child-safety work in public before this release. On 11 February 2026 it discussed with Thorn, through the OpenAI Forum, how AI can help investigators organise information and limit their exposure to abusive material. On 8 April 2026 it published a Child Safety Blueprint, an industry-wide framework against AI-enabled child sexual exploitation that urged updated US laws on AI-generated abuse material, better reporting by providers and safety by design.

The company has also faced criticism on the same ground. Common Sense Media earlier labelled OpenAI's technology an "unacceptable risk" for teens, and OpenAI faces several lawsuits over harm to users, especially children.

11

Works credited

35

Models indexed

Models on AIPopDB

ChatGPT
ChatGPT 4.0
ChatGPT Images
Codex
DALL-E 2
Dots
See OpenAI on AIPopDB

OpenAI says it will keep refining the tools. Organizations that want the classifier apply through an intake form.

The ChatGPT app open on an iPhone

The ChatGPT app open on an iPhone

Image: Digital Trends

Comments