OpenAI deploys classifier for detecting CSAM and new safety warnings in the ChatGPT product
OpenAI has released a classifier for detecting CSAM images (including AI-generated ones), available free of charge via API to selected entities, and new warning screens in the ChatGPT product with a 24-hour account block upon violation.
OpenAI has released new tools to combat child sexual abuse material (CSAM), including AI-generated content. At the core is a new image classifier that detects CSAM based on visual signs of abuse even in previously unseen images, assigning them a score for moderators. The classifier was developed in collaboration with the National Center for Missing and Exploited Children (NCMEC), and the company is making it available free of charge to selected companies via API, further distributing access with the Tech Coalition and ROOST organizations.
According to OpenAI, in early testing with partners the classifier flags 48% fewer images as suspicious, while still catching the same number of genuinely abusive images, thereby reducing the burden on human reviewers. Images flagged by the classifier are, according to the company, twice as likely to be genuinely abusive as before.
In the ChatGPT product, new warning screens have been added: when a prompt attempting to sexualize children is detected, the system displays a warning, directs the user to specialized help, and temporarily blocks the account for 24 hours; repeated violations can lead to permanent account cancellation. The application already blocked CSAM content and related requests before. OpenAI created the messages and guidelines for this behavior together with Protect Children, the Lucy Faithfull Foundation, and researcher Joel Scanlan from the University of Tasmania, who independently evaluated the pilot testing of the deterrent messages.
OpenAI has also published guidelines for developers using its tools on how to prevent, detect, and report CSAM content, and according to the company, these guidelines helped a number of developers in early testing deploy tools for child protection and register for reporting to NCMEC. For law enforcement agencies, OpenAI offers AI tools — specialized plugins, investigative tools, and data analysis — with access controlled by program administrators, as well as training and a practical handbook for investigators.
Why it matters
For operators of image-based platforms, this is a concrete tool that can be deployed free of charge to reduce the number of false positives in moderation while maintaining detection of genuinely abusive content, according to OpenAI with the same detection sensitivity. For developers building applications on OpenAI technologies, specific guidelines have been added on how to detect, prevent, and report CSAM content to NCMEC, which makes it easier to meet legal and compliance obligations in this area.
Relevant practical impact
What this means
For a business
Companies operating platforms with image generation or uploading gain the ability to deploy a free CSAM classifier and new developer guidelines, which reduces the risk of hosting illegal content and the workload on moderation teams.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.