AI TrackerGet the app
← All news
OpenAIProducts

OpenAI releases a CSAM image classifier to vetted organizations and adds ChatGPT warnings

OpenAI says it is releasing an image classifier that detects potential child sexual abuse material (CSAM), including AI-generated material, and returns scores to help reviewers prioritize. It was developed with help from the National Center for Missing and Exploited Children (NCMEC). Vetted organizations can request access through the OpenAI API at no cost; OpenAI worked with the Tech Coalition and ROOST to share it. In the API it is the model omni-moderation-csam-latest on the /v1/moderations endpoint, with separate signals for potential CSAM and potential AI-generated CSAM. OpenAI describes it as the first broadly available CSAM classifier with a specific signal for whether material may be AI-generated, and has published a model card.

In OpenAI's own evaluation on user-uploaded images, the classifier would flag 48% fewer images than its existing tool while finding roughly the same number of abusive images, and flagged images were about twice as likely to contain abuse. Early partners report their own results: fal says reviewers confirmed 72% of high-confidence flags as novel CSAM, invideo AI says one in four flagged cases led to a ban, and SafetyKit says it detected 90% of cases involving drawings or animations.

Other parts of the announcement:

  • ChatGPT warnings: attempts to find abusive material now trigger a warning that points people to help. OpenAI says it removes the flagged conversation and pauses access to ChatGPT for 24 hours, with longer restrictions, account closure or reports to authorities for repeated violations.
  • Developer guidance: OpenAI published CSAM guidance for developers on prevention, detection and reporting, and says it worked with Anthropic and Google to commission Tech Coalition guidance on responding to disclosures of abuse in AI systems.
  • Law enforcement: child-safety investigative units get new tools and training. The first tool helps investigators review chat logs and legal process returns, find connections and build timelines, with access managed by program administrators.

Get news like this as a push.

AI Tracker alerts you when something changes for the tools you use.

Get the app