Mistral launches a moderation API

Date:

Share post:


AI startup Mistral has launched a new API for content moderation.

The API, which is the same API that powers moderation in Mistral’s Le Chat chatbot platform, can be tailored to specific applications and safety standards, Mistral says. It’s powered by a fine-tuned model (Ministral 8B) trained to classify text in a range of languages, including English, French, and German, into one of nine categories: sexual, hate and discrimination, violence and threats, dangerous and criminal content, self-harm, health, financial, law, and personally identifiable information.

The moderation API can be applied to either raw or conversational text, Mistral says.

“Over the past few months, we’ve seen growing enthusiasm across the industry and research community for new AI-based moderation systems, which can help make moderation more scalable and robust across applications,” Mistral wrote in a blog post. “Our content moderation classifier leverages the most relevant policy categories for effective guardrails and introduces a pragmatic approach to model safety by addressing model-generated harms such as unqualified advice and PII.”

AI-powered moderation systems are useful in theory. But they’re also susceptible to the same biases and technical flaws that plague other AI systems.

For example, some models trained to detect toxicity see phrases in African American Vernacular English (AAVE), the informal grammar used by some Black Americans, as disproportionately “toxic.” Posts on social media about people with disabilities are also often flagged as more negative or toxic by commonly used public sentiment and toxicity detection models, studies have found.

Mistral claims that its moderation model is highly accurate — but also admits it’s a work in progress. Notably, the company didn’t compare its API’s performance to other popular moderation APIs, like Jigsaw’s Perspective API and OpenAI’s moderation API.

“We’re working with our customers to build and share scalable, lightweight, and customizable moderation tooling,” the company said, “and will continue to engage with the research community to contribute safety advancements to the broader field.”

Mistral also announced a batch API today. The company says it can reduce the cost of models served through its API by 25% by processing high-volume requests asynchronously. Anthropic, OpenAI, Google, and others also offer batching options for their AI APIs.



Source link

Lisa Holden
Lisa Holden
Lisa Holden is a news writer for LinkDaddy News. She writes health, sport, tech, and more. Some of her favorite topics include the latest trends in fitness and wellness, the best ways to use technology to improve your life, and the latest developments in medical research.

Recent posts

Related articles

Meta, X approved ads containing violent anti-Muslim, antisemitic hate speech ahead of German election, study finds

Social media giants Meta and X (formerly Twitter) approved ads targeting users in Germany with violent anti-Muslim...

Court filings show Meta staffers discussed using copyrighted content for AI training

For years, Meta employees have internally discussed using copyrighted works obtained through legally questionable means to train...

Brian Armstrong says Coinbase spent $50M fighting SEC lawsuit – and beat it

Coinbase on Friday said the SEC has agreed to drop the lawsuit against the company with prejudice,...

iOS 18.4 will bring Apple Intelligence-powered ‘Priority Notifications’

Apple on Friday released its first developer beta for iOS 18.4, which adds a new “Priority Notifications”...

Nvidia CEO Jensen Huang says market got it wrong about DeepSeek’s impact

Nvidia founder and CEO Jensen Huang said the market got it wrong when it comes to DeepSeek’s...

Report: OpenAI plans to shift compute needs from Microsoft to SoftBank

OpenAI is forecasting a major shift in the next five years around who it gets most of...

Norway’s 1X is building a humanoid robot for the home

Norwegian robotics firm 1X unveiled its latest home robot, Neo Gamma, on Friday. The humanoid system will...

Sakana walks back claims that its AI can dramatically speed up model training

This week, Sakana AI, an Nvidia-backed startup that’s raised hundreds of millions of dollars from VC firms,...