Skip to content
worth noting Security

OpenAI deploys classifier for detecting CSAM and new safety warnings in the ChatGPT product

only one source so far

OpenAI has released a classifier for detecting CSAM images (including AI-generated ones), available free of charge via API to selected entities, and new warning screens in the ChatGPT product with a 24-hour account block upon violation.

OpenAI has released new tools to combat child sexual abuse material (CSAM), including AI-generated content. At the core is a new image classifier that detects CSAM based on visual signs of abuse even in previously unseen images, assigning them a score for moderators. The classifier was developed in collaboration with the National Center for Missing and Exploited Children (NCMEC), and the company is making it available free of charge to selected companies via API, further distributing access with the Tech Coalition and ROOST organizations.

According to OpenAI, in early testing with partners the classifier flags 48% fewer images as suspicious, while still catching the same number of genuinely abusive images, thereby reducing the burden on human reviewers. Images flagged by the classifier are, according to the company, twice as likely to be genuinely abusive as before.

In the ChatGPT product, new warning screens have been added: when a prompt attempting to sexualize children is detected, the system displays a warning, directs the user to specialized help, and temporarily blocks the account for 24 hours; repeated violations can lead to permanent account cancellation. The application already blocked CSAM content and related requests before. OpenAI created the messages and guidelines for this behavior together with Protect Children, the Lucy Faithfull Foundation, and researcher Joel Scanlan from the University of Tasmania, who independently evaluated the pilot testing of the deterrent messages.

OpenAI has also published guidelines for developers using its tools on how to prevent, detect, and report CSAM content, and according to the company, these guidelines helped a number of developers in early testing deploy tools for child protection and register for reporting to NCMEC. For law enforcement agencies, OpenAI offers AI tools — specialized plugins, investigative tools, and data analysis — with access controlled by program administrators, as well as training and a practical handbook for investigators.

What changed

Why it matters

For operators of image-based platforms, this is a concrete tool that can be deployed free of charge to reduce the number of false positives in moderation while maintaining detection of genuinely abusive content, according to OpenAI with the same detection sensitivity. For developers building applications on OpenAI technologies, specific guidelines have been added on how to detect, prevent, and report CSAM content to NCMEC, which makes it easier to meet legal and compliance obligations in this area.

Relevant practical impact

What this means

01

For a business

Companies operating platforms with image generation or uploading gain the ability to deploy a free CSAM classifier and new developer guidelines, which reduces the risk of hosting illegal content and the workload on moderation teams.

Risks and compliance
What to decide If a company develops or integrates tools for image generation/processing, check whether it can use free access to the CSAM classifier API for content moderation.
More business impacts →
content safety ChatGPT compliance CSAM child protection OpenAI

Check the original

Event sources

only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.

1
TechCrunch AI independent context · first detected OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM