.svg){kind=small}
AI for red teams
The course links offense to operations. AI risk is not just another web app risk. Models can generalize and hallucinate, retrieval chains blend internal and external data, and tool use lets models perform actions, multiplying impact. Red teams must model this system boundary, then pressure test across prompts, retrieval, tools, and supply chains.
Course Content
AI risk is not just another web app risk. Inputs are untrusted language, models can generalize and hallucinate, and retrieval chains blend internal and external data. Tool use lets models perform actions, multiplying impact. Red teams must model this system boundary, then pressure test across prompts, retrieval, tools, and model supply chain.
You will use MITRE ATLAS to structure threat modeling and attack hypotheses for AI-specific behaviors. OWASP Top 10 for LLMs provides concrete failure modes and controls to test. Findings roll up to NIST AI RMF functions for governance reporting, while CISA Secure by Design guides how teams fix root causes in code, configuration, and defaults.
The course links offense to operations. You will measure bypass rates, grounding errors, and detection quality, then hand off artifacts that SOC and platform teams can automate, monitor, and tune. Engineering receives actionable repros, guardrail configs, and remediation briefs tied to enterprise controls.
Course Objectives
By the end of this course, you will be able to:
- Map AI attack surfaces across LLMs, agents, tools, and RAG so you can target tests that matter to the business. Why it matters: Focused testing reduces cost and increases risk coverage on critical workflows.
- Apply MITRE ATLAS and OWASP LLM risks to plan adversarial campaigns with shared language and repeatable scope. Why it matters: Common taxonomies enable consistent testing and defensible reporting.
- Execute AI-specific TTPs such as prompt injection, tool abuse, data poisoning, and model extraction. Why it matters: Offense validates whether existing controls prevent real loss scenarios.
- Instrument and measure evaluations including safety, grounding, and jailbreak success. Why it matters: Quantified results drive risk decisions and model or policy changes.
- Integrate platform guardrails from Microsoft and Google into test harnesses. Why it matters: Real platform constraints shape what is exploitable and what can be remediated quickly.
- Report and remediate using NIST AI RMF and CISA Secure by Design alignment. Why it matters: Executives act on findings framed in accepted governance and assurance models.
