Key Points
- OpenAI released an image classifier that detects child sexual abuse material, including AI-generated images.
- Testing with partners cut flagged images 48% while finding the same number of abusive images.
- The rollout follows lawsuits and safety criticism over how OpenAI's tools affect minors.
The latest:
A new image classifier built to detect child sexual abuse material, including AI-generated images, is being offered free to select companies through OpenAI’s API, the company announced Monday. The system scores previously unseen images for visual signs of abuse to guide human reviewers. OpenAI said it developed the tool with the National Center for Missing and Exploited Children.
Details:
- The results: In early testing with partners, OpenAI said the classifier flagged 48% fewer images while identifying the same number of abusive ones, cutting reviewer workload. Images the classifier does flag are twice as likely to be abusive, according to the company, which did not name the partners involved in the trial.
- The distribution: Access runs through OpenAI’s API at no cost to selected companies. The company said it is working with Tech Coalition and ROOST to widen access to the tool. OpenAI did not disclose how many organizations currently have it or set a timeline for broader availability.
- ChatGPT enforcement: OpenAI is adding warning screens for users who attempt prompts seeking material that sexualizes children. The app will show an alert, point users toward specialized support aimed at stopping harmful behavior, and suspend the account for 24 hours. Repeated offenses can lead to permanent removal, the company said.
- Already blocked: ChatGPT already blocks CSAM content and requests related to it, according to OpenAI. The new layer is deterrence rather than detection: the warning screen and temporary suspension sit on top of existing refusals, targeting users who repeatedly attempt such prompts rather than those who do so once.
- Who shaped it: OpenAI said it developed the deterrent messaging and guidelines with outside organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of the deterrent messages.
- Developer rules: The company is publishing guidelines for developers building on its tools covering how to prevent, detect, and help users report CSAM. OpenAI said early testing pushed a number of developers to adopt child safety tooling and to register to report material to NCMEC.
- Law enforcement tools: OpenAI is giving investigators working on child safety cases AI-powered features including dedicated plugins, investigative tools, and complex data analysis, with access controlled by program administrators. The company said it is also running training for investigators and distributing a field guide on using AI in such cases.
- The pressure: OpenAI faces several lawsuits alleging its technology harmed users, particularly children. Safety-focused nonprofit Common Sense Media has labeled the company’s technology an “unacceptable risk” for teens. Other chatbots, specifically Grok, have been used to generate nonconsensual sexualized imagery, including of children.
- Prior moves: Over the past 12 months, OpenAI released a safety blueprint, updated content guidelines to bar inappropriate material, retuned its models to limit harmful outputs, and shipped a version of ChatGPT built for teenagers.
Between the lines:
The 48% reduction figure is framed around reviewer workload, not detection breadth — OpenAI’s own account is that the classifier finds the same number of abusive images while surfacing fewer false positives. That positions the tool as an efficiency gain for moderation teams rather than a claim of catching more abuse than existing systems.
What’s next
Watch whether Tech Coalition and ROOST expand classifier access beyond the initial selected companies, how courts handle the pending lawsuits against OpenAI, and whether Common Sense Media revises its teen risk rating.