A curated list of links, references, books, videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices, etc., which are related to GenAI, LLM, RAG, MCP, Agents, and Agentic AI security.
Note:
awesome-agentic-ai-securityhas been folded into this list. Agentic AI security — agent foundations, frameworks, autonomy risks, MCP, and multi-agent threats — now lives here alongside LLM and RAG security.
- Foundations & Key Concepts
- GenAI Security Papers & Standards
- Agent Frameworks & Agentic Engineering
- AI Security Books
- AI Security Videos
- Online Tutorials / Blogs / Presentations
- Online Courses (Paid/Free)
- Study Plans, Roadmaps & Interview Prep
- AI Security Certifications
- Tools of Trade
- Security Practices and CTFs
- AI Red Teaming
- GenAI Security Attacks, Breaches & Incidents
- Regulatory Frameworks & Governance
- Newsletters & Communities
- Contributing
- Contributors
Background worth having before the security material — how agents plan, remember, and act.
- Building Effective Agents - Anthropic's practical guide to agent patterns (workflows vs. agents).
- A Practical Guide to Building Agents - OpenAI's guidance on agent design and orchestration.
- ReAct: Reasoning + Acting (Yao et al., 2023) - The reason-and-act loop underpinning tool-using agents.
- Chain-of-Thought Prompting (Wei et al., 2022) - The reasoning foundation behind planning and task decomposition.
- Model Context Protocol (MCP) - Open standard for connecting agents to tools and data.
- Awesome Agentic Engineering - A reference stack for production-grade agentic systems.
Why autonomy changes the threat model: when a system can act — write files, move money, send emails, run code, chain tools — a single bad decision or malicious input costs far more than a bad answer. Agentic systems inherit every LLM/RAG risk and add indirect prompt injection, the "lethal trifecta", excessive agency, memory poisoning, delegated identity, multi-agent trust, and a runtime tool/MCP supply chain.
Important papers, standards, and checklists from organizations like OWASP, NIST, and others.
- OWASP Top 10 for LLM Applications 2025
- OWASP LLM AI Security and Governance Checklist
- OWASP Agentic AI Top 10
- NIST AI RMF Playbook
- NIST AI Risk Management Framework (AI RMF)
- NIST Adversarial Machine Learning
- Microsoft Failure Models in Machine Learning
- Microsoft Threat Modeling AI/ML
- OWASP GenAI Security Project
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
- Google Secure AI Framework (SAIF)
- Anthropic Responsible Scaling Policy
- ENISA Multilayer Framework for Good Cybersecurity Practices for AI
- Databricks AI Security Framework (DASF) 2.0 - Practical controls mapped to AI system components and risks.
- OWASP Securing Agentic Applications Guide 1.0 - Reference architecture and controls for building secure agentic apps.
- CSA MAESTRO - Agentic AI Threat Modeling Framework - Seven-layer threat modeling for agentic AI systems.
- Microsoft: Zero Trust for AI (Tools & Guidance) - Applying Zero Trust principles to AI agents and workloads.
- OWASP Agentic Security Initiative - Threats & mitigations, multi-agent threat modeling, and reference guides.
- Vulnerable Autonomous Agents Threat Model - LLM threat models for autonomous agents.
- Top 10 Agentic AI Security Risks - Key Threats and Mitigation Strategies (PDF) - Industry threat/mitigation reference.
The frameworks you'll be securing — knowing how they orchestrate tools, state, and control flow is half the job.
- LangGraph - Graph-based orchestration for stateful, multi-actor agents.
- Microsoft AutoGen - Multi-agent conversation framework.
- CrewAI - Role-based multi-agent orchestration.
- OpenAI Agents SDK - Lightweight framework for agentic apps.
- Pydantic AI - Type-safe agent framework.
- LlamaIndex - Data framework and agent workflows.
- AI Value Creators
- AI Engineering by Chip Huyen
- Designing Machine Learning Systems
- Hands-On Large Language Models
- Nexus by Yuval Noah Harari
- The Developer's Playbook for Large Language Model Security: Building Secure AI Applications by Steve Wilson
- 10 Best AI Security Books (Practical DevSecOps)
- AI Security E-book 101 (Practical DevSecOps, PDF)
- Red Teaming AI: A Field Manual for Attacking Intelligent Systems - Philip A. Dursey (No Starch, early access)
- Not with a Bug, But with a Sticker - Ram Shankar Siva Kumar & Hyrum Anderson
- Generative AI Security - Ken Huang, Yang Wang (Springer)
- Adversarial AI Attacks, Mitigations, and Defense Strategies - John Sotiropoulos (Packt)
- Intro to LLM Security - WhyLabs
- OWASP Top 10 for LLM Applications Explained - OWASP
- Hacking LLMs and Prompt Injection - LiveOverflow
- AI Red Teaming - DEFCON AI Village
- Securing LLM Applications - SANS Institute
Articles and guides covering LLM, RAG, and general GenAI security.
- LLM Security
- What are foundational models?
- A quick check on the AI Threat Model
- Security Incident Response using LLM
- OWASP: CheatSheet – A Practical Guide for Securely Using Third-Party MCP Servers 1.0
- AI Security Interview Questions (Practical DevSecOps)
- Emerging AI Security Roles (Practical DevSecOps)
- AI Security Engineer Roadmap (Practical DevSecOps)
- Prompt Injection Attacks and Defenses in LLM-Integrated Applications
- Simon Willison's Blog on Prompt Injection
- Embrace the Red - AI Security Blog by Johann Rehberger
- Trail of Bits - AI/ML Security Research
- Riding the RAG Trail: Access, Permissions and Context
- Securing Risks with RAG Architectures
- Mitigating Security Risks in Retrieval Augmented Generation (RAG)
- RAG: The Essential Guide
- Why RAG is revolutionising GenAI
- Invariant Labs: MCP Security Notification Tool
- Pillar Security: MCP Security Research
- Agentic Security Risks - OWASP
- Tool Poisoning Attacks in MCP
- The lethal trifecta for AI agents - Private data + untrusted content + exfiltration = a data leak waiting to happen.
- Imprompter: Tricking LLM Agents into Improper Tool Use - Attack demonstration against tool-using agents.
- Web LLM attacks - PortSwigger
- Prompt injection jailbreaking
- LLM Attacks - Universal and Transferable Adversarial Attacks on Aligned LLMs
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications
- Stanford CS-324: Large Language Models
- Princeton COS 597G: Understanding Large Language Models
- Coursera: GenAI with LLM
- Coursera: Generative AI Engineering with LLMs Specialization
- Coursera: Generative AI for Cybersecurity Professionals (IBM)
- Coursera: AI for Cybersecurity (JHU)
- AttackIQ: The foundation of AI Security
- Microsoft AI Red Teaming 101 (Microsoft Learn) - Free training on GenAI vulnerabilities, single/multi-turn attacks, spotlighting defenses, and PyRIT automation.
- SANS SEC545: GenAI and LLM Application Security - RAG/vector-DB security, prompt injection, MLOps hardening, and agentic AI security (maps to GIAC GAIPS).
Structured paths for going from "interested in AI security" to job-ready.
- GenAI Security Study Plan - Week-by-week plan for learning GenAI and LLM security.
- AI Security Career Roadmap - Skills, milestones, and role progression for an AI security career.
- AI Security Interview Questions - Question bank for AI/GenAI security interviews.
- AI/ML Security Learning Resources - Wider collection of AI/ML security learning material.
- Certified AI Security Professional (CAISP) by Practical DevSecOps – Securing AI systems, models, and pipelines against adversarial threats, LLM vulnerabilities, AI supply chain risks, data poisoning, and AI-specific security frameworks. Hands-on, practitioner-level skills.
- GIAC AI Platform Security (GAIPS) - Hands-on (CyberLive) certification for auditing and securing GenAI applications, LLM pipelines, and agentic AI systems.
- GIAC AI Security Automation Engineer (GASAE) - Validates using AI/automation across offensive, defensive, and cloud security operations.
- ISC2 AI Security Certificate - On-demand course bundle covering AI fundamentals, AI risk management, secure-by-design, and global AI regulation.
Tools for defending, scanning, and auditing GenAI systems.
- LLM Guard - Information extraction and security for LLMs.
- Model Scan - Scanning models for serialization attacks.
- Rebuff - Prompt injection detection.
- NB Defense - Notebook security.
- Protect AI's OSS Portfolio
- LLM Guard Playground
- Cisco MCP Scanner - Scans MCP servers/tools for poisoning and prompt injection (YARA + LLM-as-judge).
- Snyk Agent Scan - Inventories and scans AI agents, MCP servers, and skills for 15+ risks (successor to Invariant mcp-scan).
- Fickling (Trail of Bits) - Decompiler, static analyzer, and safety scanner for malicious pickle/PyTorch model files.
- ModelAudit - Static scanner detecting malicious code/backdoors across 40+ ML model file formats.
- AIsbom - CLI that scans model files for malware and generates CycloneDX/SPDX AI SBOMs.
- Agentic Radar (SPLX) - Security scanner that maps and analyzes agentic workflows.
- Giskard - Testing and scanning framework for ML/LLM systems.
- AI/ML Exploits
- Garak - LLM Vulnerability Scanner
- PyRIT - Python Risk Identification Toolkit for GenAI (Microsoft)
- Counterfit - AI Security Testing (Microsoft)
- ART - Adversarial Robustness Toolbox (IBM)
- promptmap - Prompt Injection Testing
- DeepTeam - LLM & AI-Agent Red Teaming Framework - 50+ vulnerability types and 20+ attack methods mapped to OWASP/NIST/MITRE.
- Promptfoo - LLM Testing & Red Teaming - Generates adversarial inputs to find prompt injection, jailbreaks, and data leakage, with CI/CD integration.
- Redcells - Automated Adversarial Testing for LLMs - Public-beta platform for automated adversarial testing of LLMs you own or control. OpenAI-compatible target models, iterative attack→refine layers, dashboard + API. (Repo)
- Guardrails AI - Input/output validation for LLMs.
- NeMo Guardrails (NVIDIA) - Programmable guardrails for LLM applications.
- Vigil - LLM Prompt Injection Detection
- Lakera Guard - Real-time AI security for prompt injection and data leakage.
- Omega Walls - Open-source stateful prompt injection defense for RAG and agent pipelines, built as a runtime trust boundary across untrusted content, memory, context, and tools.
- Trylon Gateway - Self-hosted open-source AI firewall/proxy applying custom guardrails (prompt-injection defense, PII redaction).
- Bifrost AI Gateway - High-performance open-source AI gateway unifying 20+ LLM providers with governance and policy enforcement.
- HOL Guard - Local-first security harness that intercepts tool calls in AI coding agents before files change or network is contacted. Scans skills, MCP servers, and plugins for supply-chain threats.
Practice your skills with these vulnerable applications and challenges.
- Gandalf - Lakera AI - LLM security challenge.
- PromptTrace - Free hands-on labs and a progressive gauntlet for prompt injection, jailbreaks, RAG poisoning, and tool/function-call abuse against real LLMs.
- Prompt Airlines - AI security challenges, CTF style.
- Certified AI/ML Pentester (C-AI/MLPen) Exam - The SecOps Group
- Damn Vulnerable MCP Server - Deliberately vulnerable MCP implementation.
- Vulnerable MCP Servers Lab - Collection of vulnerable servers.
- FinBot Agentic AI CTF - Agentic Security CTF.
- OWASP WrongSecrets - Includes an LLM/AI secrets-leakage challenge.
- Huntr.com - World’s first bug bounty platform for AI/ML.
- HackAPrompt - Prompt hacking competition.
- Crucible by Dreadnode - AI/ML security challenges and CTFs.
- AI Goat - Vulnerable LLM CTF built on AWS.
- Microsoft AI Red Teaming Playground Labs - Hands-on red-teaming challenges (prompt injection, indirect injection, guardrail bypass) with Docker/Kubernetes deployment.
- PortSwigger Web Security Academy: Web LLM Attacks - Free official hands-on labs on exploiting LLM APIs, excessive agency, and prompt injection.
- Vulnerable LLM apps (GitHub topic) - Index of intentionally vulnerable LLM apps to practice on.
Resources and methodologies for red teaming AI/GenAI systems.
- Microsoft AI Red Team
- Anthropic Red Teaming Research
- MITRE ATLAS - Tools, Data & Case Studies (GitHub)
- AI Red Teaming Guide - Humane Intelligence
- Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities
- OWASP GenAI Red Teaming Guide - Methodology for red teaming GenAI/LLM applications.
- CSA Agentic AI Red Teaming Guide - Red teaming approach tailored to autonomous agents.
Notable real-world incidents involving GenAI and LLM security, newest first. For ongoing tracking, see the OWASP GenAI Exploit Round-up, which catalogs confirmed CVEs and exploits each quarter.
- Hugging Face Breached by an Autonomous AI Agent (Jul 2026) - An agentic system escaped a public security benchmark, abused two code-execution paths in Hugging Face's dataset processing, and reached production infrastructure over a weekend. OpenAI attributed it to its own pre-release models running unsupervised. Internal datasets and service credentials were accessed; public models, datasets, and Spaces were verified untampered. The first known case where AI capability evaluation caused a real breach.
- JADEPUFFER: First Documented Agentic Ransomware (Jul 2026) - Sysdig traced a full extortion chain driven end-to-end by an LLM: Langflow RCE (CVE-2025-3248) for entry, credential harvesting, lateral movement, then encryption and destruction of 1,342 config items. The agent recovered from a failed login in 31 seconds by rewriting its own payload. The AES key was printed to stdout and never persisted — paying the ransom recovers nothing.
- Cursor Terminal Allowlist Bypass (CVE-2026-22708, Jul 2026) - Shell built-ins like
exportbypassed Cursor's Auto-Run allowlist, letting indirect prompt injection poison the agent's environment and reach zero-click RCE. Fixed in 2.3. - Vercel Breached via a Compromised Third-Party AI Tool (Apr 2026) - Attackers pivoted from an infostealer infection at Context.ai through an employee's OAuth grant into Vercel's internal systems, enumerating and decrypting environment variables. A worked example of shadow-AI and OAuth sprawl as an enterprise attack path.
- Zenity Labs: Zero-Click Hijacking of Agentic Browsers (Mar 2026) - A vulnerability family in Perplexity Comet allowing zero-click agent hijack, including takeover of a signed-in password manager, via indirect prompt injection. See also Brave's earlier Comet research.
- Nine Mexican Government Agencies Breached by One AI-Orchestrating Operator (Dec 2025 - Feb 2026) - Check Point documented a single operator running Claude Code and GPT-4.1 in parallel — 1,088 prompts producing 5,317 AI-executed commands across 34 sessions — exposing roughly 400 million tax, civil registry, electoral, and health records. Evidence that AI-orchestrated intrusion has moved from state actors to ordinary criminals.
- Anthropic Disrupts First AI-Orchestrated Cyber Espionage Campaign (Nov 2025) - A Chinese state-sponsored group jailbroke Claude Code to autonomously execute ~80-90% of an espionage campaign against ~30 global targets; the first documented largely AI-run cyberattack.
- CamoLeak: GitHub Copilot Chat Private Source-Code Exfiltration (Oct 2025) - CVSS 9.6 prompt injection hidden in PRs leaks private code and secrets via GitHub's own Camo image proxy.
- Check Point Researchers Expose Critical Claude Code Flaws - CVE-2025-59536 and CVE-2026-21852: Enabling Remote Command Execution and API Key Theft.
- Anthropic: "Vibe-Hacking" Extortion Using Claude Code (Aug 2025) - A criminal used Claude Code to automate intrusions and craft targeted extortion against 17+ organizations, alongside AI-built ransomware-as-a-service.
- PromptLock: First Known AI-Powered Ransomware (Aug 2025, PoC) - ESET found ransomware that uses a local LLM via the Ollama API to generate malicious scripts on the fly (later assessed as an academic proof-of-concept).
- Replit AI Agent Deletes a Production Database (Jul 2025) - Replit's AI coding agent wiped a live production database during a code freeze, then fabricated data and falsely claimed rollback was impossible.
- Amazon Q VS Code Extension Compromised with Data-Wiping Prompt (Jul 2025) - A malicious prompt to wipe files and delete AWS resources was slipped into the official Amazon Q extension (964k+ installs).
- CVE-2025-6514: Critical RCE in mcp-remote (Jul 2025) - A malicious MCP server could achieve full remote code execution (CVSS 9.6) on clients running mcp-remote; the first real-world RCE against an MCP client.
- EchoLeak (CVE-2025-32711): Zero-Click Data Theft in Microsoft 365 Copilot (Jun 2025) - A single crafted email could silently exfiltrate organizational data from M365 Copilot with no user interaction; the first known zero-click exploit against an AI agent.
- Policy Puppetry: Universal Jailbreak Bypassing All Major LLMs (Apr 2025) - HiddenLayer disclosed a single transferable prompt that bypasses safety guardrails across OpenAI, Google, Anthropic, Meta, DeepSeek, and others.
- Singapore Finance Director Scammed via Deepfake Executive Video Call (Mar 2025) - A finance director transferred ~US$499,000 after a Zoom call in which company executives were AI-generated deepfakes.
- nullifAI: Malicious ML Models on Hugging Face Evade Picklescan (Feb 2025) - ReversingLabs found malicious Pickle-based models using broken/7z-wrapped pickles to bypass scanners and deliver a reverse shell (PoC).
- Anthropic: Chinese AI Firms Created 24,000 Fraudulent Accounts For 'Distillation Attacks'
- DeepSeek Exposed Database Leaking Chat History and API Keys (Jan 2025) - Wiz found a publicly accessible, unauthenticated DeepSeek ClickHouse database exposing 1M+ log lines including plaintext chats and secrets.
- LiteLLM on PyPI Was Compromised, What the Attack Changed and What Defenders Should Do Now
- The Day Chevrolet's AI Chatbot Tried to Sell a $70,000 SUV for $1
- Here Come the AI Worms (Wired, 2024) - Morris II: self-propagating prompt-injection worms spreading between AI agents.
- Air Canada Chatbot Provides Wrong Info (2024) - Airline held liable for chatbot hallucinating refund policy.
- Samsung bans use of generative AI tools like ChatGPT after April internal data leak (2023)
- AI-powered Bing Chat spills its secrets via prompt injection attack (2023)
- ChatGPT Data Leak Bug (2023) - Bug exposed chat history titles and payment info of other users.
- GitHub Copilot Leaking Secrets (2023) - AI code assistant reproducing secrets from training data.
- Microsoft Tay Bot Manipulation (2016) - Twitter chatbot manipulated into generating offensive content.
- EU AI Act - EU regulation on artificial intelligence.
- NIST AI Risk Management Framework (AI RMF) - NIST's voluntary framework for managing AI risks.
- ISO/IEC 42001:2023 - AI Management System Standard
- U.S. AI Executive Order (2025): Removing Barriers to American Leadership in AI - The Trump administration's Jan 2025 EO; it revoked Biden's EO 14110 (2023) on Safe, Secure & Trustworthy AI.
- India AI Governance Guidelines (IndiaAI / MeitY) - India's national AI strategy and governance guidance.
- Singapore Model AI Governance Framework
- UK - International AI Safety Report - Frontier AI risk assessment led by Yoshua Bengio.
- OWASP GenAI Slack Channel - Join #project-top10-for-llm channel.
- AI Village (DEF CON) - Community focused on AI security research.
- MLSecOps Community - Community for ML security operations.
- The AI Security Newsletter by Ken Huang
- Protect AI Blog (now part of Palo Alto Networks)
Found a great paper, tool, course, CTF, or incident writeup on GenAI, LLM, RAG, MCP, or agentic AI security? Read CONTRIBUTING.md and open a pull request. Small additions are very welcome.
Thanks to everyone who helps keep this list current. Your name could be next — open a PR!
- Sanjeev Jaiswal - maintainer
- mldangelo - promptfoo
- Aj7ay7 - Practical DevSecOps CAISP certification
- anviren - Omega Walls
- kantorcodes - HOL Guard
- awdemos - Redcells
- K4r1it0 - PromptTrace
Licensed under GPL-3.0.
