AI guardrails and red teaming: a simple guide for students

AI guardrails are the checks and limits around an AI system that keep it safe and on its intended job. They check what goes into the model, check what comes out, and limit what the model is allowed to do. AI red teaming is how you test those guardrails. People try to make the system fail on purpose, in a safe setting, so the flaws get fixed early.

We use two free, official sources: the OWASP Top 10 for LLM Applications 2026 and the NIST AI Risk Management Framework. It is about defence, so it has no attack prompts.

What are AI guardrails?

AI guardrails are controls placed around a large language model (LLM), so that the whole system stays safe even when the model makes a mistake. The model is one part. The guardrails are the code, rules and permissions around it.

A simple example: a college chatbot answers questions from the course handbook. Guardrails can stop it from showing one student's marks to another. They can also stop it from inventing rules or sending emails on its own.

The leads of the OWASP list put the main idea well in their 2026 letter: "Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks."

What are the types of AI guardrails?

The main types of AI guardrails are input checks, output checks, policies, tool limits, human approval and cost limits. Real systems use several at once, because no single check is enough.

• Input checks: look at everything that goes into the model. Web pages, PDFs, emails and tool results can carry hidden instructions, just like a user's message.
• Output checks: check the answer before a person sees it or a program uses it. For example, check the format, remove personal data and block unsafe links.
• Policies: written rules for what the system may and may not do. Enforce them in normal code as well as in the system prompt.
• Tool limits: give the model only the tools and permissions it needs. A tool that summarises emails needs to read emails. It does not need to send or delete them.
• Human approval: a person must approve high-impact actions, such as paying money, posting in public or deleting files.
• Cost limits: caps on tokens and spending, so one user or agent cannot run up a huge bill.

What is AI red teaming?

AI red teaming is a planned test in which people try to make an AI system fail, so its flaws can be found and fixed. NIST calls it "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs."

The name comes from security practice, where a "red team" plays the attacker and a "blue team" defends. Guardrails and red teaming are two halves of one job. Guardrails are the defence. Red teaming checks if the defence works.

NIST names four kinds of AI red teams: the general public, domain experts, a mix of both, and humans working with AI tools.

What is prompt injection?

Prompt injection is when some input changes what an AI model does in a way its developer did not intend. Prompt injection is number one on the OWASP Top 10 for LLM Applications 2026.

The root cause: an LLM reads instructions and data as one stream of text. It cannot reliably tell "this is my instruction" apart from "this is a document I was asked to read".

• Direct injection: the user types input that overrides the system's rules.
• Indirect injection: the model reads outside content, such as a web page, an email or a shared file, that carries hidden instructions. The user may never see them.

Jailbreaking is one kind of prompt injection. OWASP defines it as prompt injection where the goal is to make the model break its safety rules.

Here is the hard part. OWASP says no reliable way to prevent prompt injection exists today. The UK's National Cyber Security Centre agrees: "The best we can hope for is reducing the likelihood or impact of attacks." So the defence is to limit the damage: assume the model will be fooled, and make sure a fooled model cannot do much harm.

A useful check, which the 2026 OWASP list repeats, is developer Simon Willison's "lethal trifecta". An AI agent is at high risk when it has all three of these:

• access to your private data
• exposure to untrusted content
• the ability to send data out

Remove any one, and the conditions for the most serious attacks are gone. This check matters most for AI agents.

What is the OWASP Top 10 for LLM Applications 2026?

The OWASP Top 10 for LLM Applications 2026 is a free list of the ten biggest security risks in apps built on large language models. OWASP is a non-profit security community. The 2026 version came out in August 2026 and replaces the 2025 list.

This edition is the first to check expert opinion against real data. The team collected 7,714 real incidents from public databases and sorted the 6,639 that had enough detail. The community vote carries three-quarters of the weight, and the incident data carries the other quarter.

Here is the 2026 list in plain words. The numbers match the official IDs, LLM01 to LLM10.

1. 1Prompt Injection: input from users, web pages or files changes what the model does. The 2026 entry also covers instructions hidden in images or audio.
2. 2Sensitive data: the system shows private data, such as personal details, passwords or API keys, to the wrong person. Official name: Sensitive Information Disclosure.
3. 3Excessive Agency: the model has more tools, permissions or freedom than it needs, so one wrong answer can cause real damage. OWASP calls its rise to number three the most important change in 2026.
4. 4Supply Chain: a model, dataset or package you download has been tampered with, or is not what it claims to be.
5. 5Data poisoning: someone changes training data or retrieved documents, so the model learns bad or hidden behaviour. Official name: Data and Model Poisoning.
6. 6Unbounded use: the app allows too much computing, so attackers can crash it, run up huge bills or copy the model. Official name: Unbounded Consumption.
7. 7Misinformation: the model gives false answers that look believable, and people or programs act on them. It ranked low in the community vote, but the incident data placed it near the top.
8. 8Hidden context: hidden instructions, such as the system prompt, get exposed. Assume they can be found, and keep passwords and keys out of them. Official name: Hidden Context Exposure, which replaces System Prompt Leakage.
9. 9Vector weaknesses: flaws in the search step that picks which documents the model sees, as in retrieval-augmented generation (RAG). Official name: Vector and Embedding Weaknesses.
10. 10Output handling: model output goes to other programs without checks, which can lead to attacks such as cross-site scripting. Official name: Improper Output Handling. It fell from fifth to tenth.

If your model acts on its own, with tools and memory, OWASP says to also read its separate Top 10 for Agentic Applications.

What is the NIST AI Risk Management Framework?

The NIST AI Risk Management Framework (AI RMF) is a free, voluntary guide for managing the risks of AI systems. It comes from NIST, the US National Institute of Standards and Technology. Version 1.0 (document NIST AI 100-1) came out on 26 January 2023.

The core of the AI RMF has four functions:

• Govern: set up the rules, roles and culture for handling AI risk. NIST says Govern runs through the other three functions.
• Map: understand the context. Who uses the system, for what, and what could go wrong?
• Measure: test and track the risks with numbers or careful checks. NIST says AI systems should be tested before release and regularly while in use.
• Manage: decide which risks to treat first, fix them, and plan how to respond when something goes wrong.

On 26 July 2024, NIST added a Generative AI Profile (NIST AI 600-1). The profile lists 12 risks that generative AI creates or makes worse, such as confabulation (confident but false answers), data privacy and information security. It also explains red teaming. As of November 2026, NIST says the AI RMF 1.0 is being revised as part of the White House AI Action Plan.

OWASP tells you what can go wrong in an LLM app. NIST tells you how to organise the work of fixing it. In a student project, use OWASP as your list of risks and the four NIST functions as your steps.

How can students start AI guardrails and red teaming research?

Students can start with a small, defensive project. Build your own simple app, test it with careful test cases, and measure how much each guardrail helps. You do not need a big lab.

1. 1Read the two sources: the OWASP 2026 PDF and the red teaming part of NIST AI 600-1. Pick one or two risks, not all ten.
2. 2Build a small app: for example, a chatbot that answers from your own notes and has one tool, such as saving a note. Use fake data only.
3. 3Map the risks: for each risk you picked, write down how it could happen in your app and what harm it would cause.
4. 4Build a test set: write 50 to 100 test cases. Each case has an input, the safe behaviour you expect, and a label such as LLM01. Include normal, harmless requests too.
5. 5Measure two numbers: how many unsafe cases got through, and how many safe requests were wrongly blocked. A guardrail that blocks everything is not a good guardrail.
6. 6Change one thing: add one guardrail at a time and run the full test set again, so you know which change helped.
7. 7Grade carefully: code rules can grade simple cases. For harder ones, use people, or an AI grader you check by hand. See our guide to LLM-as-a-judge.
8. 8Write it up: share your code, test categories and results, but not working attack prompts.

A test set is an experiment, so plan it like one. Our free course on designing experiments shows how to choose what to measure before you start.

Free, open-source tools can help. NVIDIA's NeMo Guardrails and the Guardrails AI library add guardrails to LLM apps. NVIDIA's garak is a scanner that probes LLMs for weak spots. Use them on your own apps only.

What are the rules for ethical AI red teaming?

Ethical AI red teaming means you test with permission, protect real people and stop once a flaw is proven.

• Get permission: test only systems you own, or systems where you have written permission. A company's bug bounty page usually says what you may test.
• Use fake data: never put real people's personal data in your tests.
• Stop at proof: once you show that a flaw exists, stop. Do not use it to take data or cause harm.

How do you report an AI flaw responsibly?

Responsible disclosure means you report a flaw privately to the owner first and give them time to fix it. Only then do you talk about it in public.

1. 1Find the contact: look for a security contact, a security.txt file or a bug bounty page on the company's website.
2. 2Write a clear report: the product and version, what you did, what happened, and why it matters.
3. 3Wait for the fix: do not post details in public until the owner has fixed the flaw or agreed to publish.

In India, you can also report security flaws in any product to CERT-In, India's Computer Emergency Response Team. Its Responsible Vulnerability Disclosure and Coordination Policy explains how. CERT-In says it acknowledges a report within 72 working hours, and it aims to resolve an issue within 120 days of first contacting the vendor.

What do students often get wrong?

In our view, three mistakes are common in beginner projects on AI safety:

• Trusting the prompt: a rule in the system prompt is a request, not a lock. OWASP says to assume the system prompt can be found and worked around.
• Collecting tricks: a long list of jailbreaks copied from the internet is not research. A small, labelled test set is.
• Testing only chat: many real attacks arrive through documents, web pages and tool results, not the chat box.

At APJ-RAISE, students study topics like this in small groups and turn them into careful research projects. Joining is free.