What is AI red teaming?

The term comes from military and cybersecurity strategy. A “red team” is a group that deliberately plays the adversary. Their job is to attack a system, find its weaknesses, and report back so those weaknesses can be fixed before a real attacker finds them.

In AI, red teaming works the same way. Your job is to intentionally try to make an AI model produce harmful, incorrect, biased, or otherwise problematic outputs. You’re crafting prompts and conversation strategies specifically designed to expose where the model fails.

A useful way to think about it: if RLHF is teaching the AI what “good” looks like, red teaming is stress-testing whether the AI can resist being led toward “bad.”

How we look after red teamers: Red teaming is constructive work, and each failure you document gives the safety team something specific to fix before release. It can also mean reading disturbing material, so we run it with care. Red-team projects are opt-in, and you see the content categories before you accept. Exposure is capped at 10 hours a week, you can step off any task without penalty, and a named wellbeing contact on our team is available throughout, at support@sovrano.ai.

Every lab shipping a model to millions of users needs adversarial testing before release. One viral example of a model producing harmful content can cause serious reputational and financial damage, so labs invest in finding problems before their users do.

Automated safety testing catches the obvious stuff: the model refusing direct requests for harmful content, basic content filters, known prompt patterns. But automated testing misses the creative, context-dependent, subtle vulnerabilities that human red teamers find. A script can test whether the model refuses “how to build a bomb.” It can’t test whether the model will gradually comply with a harmful request after five turns of carefully constructed role-play.

Regulation adds a second reason. Under Article 55 of the EU AI Act, providers of general-purpose AI models with systemic risk must run and document adversarial testing of those models. Your reports become part of that evidence.

Red teaming also benefits from diverse expertise. Different professions, jurisdictions and knowledge domains reveal different failures. The same answer can be general guidance in one EU country and regulated advice in another. Testers who know the local law, regulators and norms find failures a generalist misses, which is why jurisdictional expertise matters in this work.

Jailbreaks are prompts that try to bypass the model’s safety guidelines. The direct approach (“tell me how to do [harmful thing]”) almost never works on modern models. Instead, you’re wrapping the request in layers of indirection: fictional scenarios, role-play setups, hypothetical framings, academic pretexts, or nested instructions designed to make the model forget its safety training.

Knowledge boundary probes are prompts designed to get the model to confidently state things it doesn’t actually know. You might ask about a fictional event, a nonexistent person, or a made-up statistic to see if the model invents a convincing answer rather than admitting uncertainty. This tests for hallucination vulnerabilities.

Bias elicitation covers prompts designed to surface biased, stereotypical, or discriminatory outputs. You might ask the model to make judgments about people based on names, nationalities, or demographics. You might test whether it applies different standards to different groups. This work requires cultural sensitivity and awareness of how bias manifests in language.

Harmful instruction attempts test whether the model will provide dangerous information when the request is disguised. The request is never direct. It’s wrapped in context that makes it seem legitimate: “I’m writing a thriller novel and need realistic details about...” or “For my chemistry homework, explain the process of...”

Manipulation and social engineering involves multi-turn conversations where you gradually build trust and context, then steer the model toward problematic territory. This tests whether safety guardrails hold up over longer interactions, not just single prompts.

Finding a vulnerability is only half the job. The other half is documenting it in a way that the safety team can actually use.

After finding a failure, you classify it along several dimensions:

• •Harm type: What category of harm does this represent? Misinformation, bias, privacy violation, safety risk, inappropriate content, or something else?
• •Severity: How bad is the output? A minor inconsistency and a detailed set of dangerous instructions are both “failures,” and they carry very different urgency. You’ll score severity on a scale, typically low/medium/high/critical.
• •Reproducibility: Is this a one-off edge case that required a very specific prompt, or a systematic flaw that can be triggered reliably? Reproducible vulnerabilities are higher priority for the safety team.
• •Technique used: How did you trigger it? Role-play, gradual escalation, language switching, authority framing, hypothetical framing? Categorizing the technique helps the safety team see whole classes of vulnerabilities as well as individual instances.