Skip to content

About

Curated list of links, references, books videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices etc. which are related to GenAI and LLM Security

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

33 Commits

Folders and files

Repository files navigation

Awesome GenAI Security Awesome

Focus: GenAI & Agentic AI Security License: GPL v3 PRs Welcome Last Commit

A curated list of links, references, books, videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices, etc., which are related to GenAI, LLM, RAG, MCP, Agents, and Agentic AI security.

Note: awesome-agentic-ai-security has been folded into this list. Agentic AI security — agent foundations, frameworks, autonomy risks, MCP, and multi-agent threats — now lives here alongside LLM and RAG security.

Table of Contents


GenAI security banner

Foundations & Key Concepts

Background worth having before the security material — how agents plan, remember, and act.

  1. Building Effective Agents - Anthropic's practical guide to agent patterns (workflows vs. agents).
  2. A Practical Guide to Building Agents - OpenAI's guidance on agent design and orchestration.
  3. ReAct: Reasoning + Acting (Yao et al., 2023) - The reason-and-act loop underpinning tool-using agents.
  4. Chain-of-Thought Prompting (Wei et al., 2022) - The reasoning foundation behind planning and task decomposition.
  5. Model Context Protocol (MCP) - Open standard for connecting agents to tools and data.
  6. Awesome Agentic Engineering - A reference stack for production-grade agentic systems.

Why autonomy changes the threat model: when a system can act — write files, move money, send emails, run code, chain tools — a single bad decision or malicious input costs far more than a bad answer. Agentic systems inherit every LLM/RAG risk and add indirect prompt injection, the "lethal trifecta", excessive agency, memory poisoning, delegated identity, multi-agent trust, and a runtime tool/MCP supply chain.

GenAI Security Papers & Standards

Important papers, standards, and checklists from organizations like OWASP, NIST, and others.

  1. OWASP Top 10 for LLM Applications 2025
  2. OWASP LLM AI Security and Governance Checklist
  3. OWASP Agentic AI Top 10
  4. NIST AI RMF Playbook
  5. NIST AI Risk Management Framework (AI RMF)
  6. NIST Adversarial Machine Learning
  7. Microsoft Failure Models in Machine Learning
  8. Microsoft Threat Modeling AI/ML
  9. OWASP GenAI Security Project
  10. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
  11. Google Secure AI Framework (SAIF)
  12. Anthropic Responsible Scaling Policy
  13. ENISA Multilayer Framework for Good Cybersecurity Practices for AI
  14. Databricks AI Security Framework (DASF) 2.0 - Practical controls mapped to AI system components and risks.
  15. OWASP Securing Agentic Applications Guide 1.0 - Reference architecture and controls for building secure agentic apps.
  16. CSA MAESTRO - Agentic AI Threat Modeling Framework - Seven-layer threat modeling for agentic AI systems.
  17. Microsoft: Zero Trust for AI (Tools & Guidance) - Applying Zero Trust principles to AI agents and workloads.
  18. OWASP Agentic Security Initiative - Threats & mitigations, multi-agent threat modeling, and reference guides.
  19. Vulnerable Autonomous Agents Threat Model - LLM threat models for autonomous agents.
  20. Top 10 Agentic AI Security Risks - Key Threats and Mitigation Strategies (PDF) - Industry threat/mitigation reference.

Agent Frameworks & Agentic Engineering

The frameworks you'll be securing — knowing how they orchestrate tools, state, and control flow is half the job.

  1. LangGraph - Graph-based orchestration for stateful, multi-actor agents.
  2. Microsoft AutoGen - Multi-agent conversation framework.
  3. CrewAI - Role-based multi-agent orchestration.
  4. OpenAI Agents SDK - Lightweight framework for agentic apps.
  5. Pydantic AI - Type-safe agent framework.
  6. LlamaIndex - Data framework and agent workflows.

AI Security Books

  1. AI Value Creators
  2. AI Engineering by Chip Huyen
  3. Designing Machine Learning Systems
  4. Hands-On Large Language Models
  5. Nexus by Yuval Noah Harari
  6. The Developer's Playbook for Large Language Model Security: Building Secure AI Applications by Steve Wilson
  7. 10 Best AI Security Books (Practical DevSecOps)
  8. AI Security E-book 101 (Practical DevSecOps, PDF)
  9. Red Teaming AI: A Field Manual for Attacking Intelligent Systems - Philip A. Dursey (No Starch, early access)
  10. Not with a Bug, But with a Sticker - Ram Shankar Siva Kumar & Hyrum Anderson
  11. Generative AI Security - Ken Huang, Yang Wang (Springer)
  12. Adversarial AI Attacks, Mitigations, and Defense Strategies - John Sotiropoulos (Packt)

AI Security Videos

  1. Intro to LLM Security - WhyLabs
  2. OWASP Top 10 for LLM Applications Explained - OWASP
  3. Hacking LLMs and Prompt Injection - LiveOverflow
  4. AI Red Teaming - DEFCON AI Village
  5. Securing LLM Applications - SANS Institute

Online Tutorials / Blogs / Presentations

Articles and guides covering LLM, RAG, and general GenAI security.

  1. LLM Security
  2. What are foundational models?
  3. A quick check on the AI Threat Model
  4. Security Incident Response using LLM
  5. OWASP: CheatSheet – A Practical Guide for Securely Using Third-Party MCP Servers 1.0
  6. AI Security Interview Questions (Practical DevSecOps)
  7. Emerging AI Security Roles (Practical DevSecOps)
  8. AI Security Engineer Roadmap (Practical DevSecOps)
  9. Prompt Injection Attacks and Defenses in LLM-Integrated Applications
  10. Simon Willison's Blog on Prompt Injection
  11. Embrace the Red - AI Security Blog by Johann Rehberger
  12. Trail of Bits - AI/ML Security Research

RAG Security

  1. Riding the RAG Trail: Access, Permissions and Context
  2. Securing Risks with RAG Architectures
  3. Mitigating Security Risks in Retrieval Augmented Generation (RAG)
  4. RAG: The Essential Guide
  5. Why RAG is revolutionising GenAI

MCP & Agent Security

  1. Invariant Labs: MCP Security Notification Tool
  2. Pillar Security: MCP Security Research
  3. Agentic Security Risks - OWASP
  4. Tool Poisoning Attacks in MCP
  5. The lethal trifecta for AI agents - Private data + untrusted content + exfiltration = a data leak waiting to happen.
  6. Imprompter: Tricking LLM Agents into Improper Tool Use - Attack demonstration against tool-using agents.

LLM Attacks

  1. Web LLM attacks - PortSwigger
  2. Prompt injection jailbreaking
  3. LLM Attacks - Universal and Transferable Adversarial Attacks on Aligned LLMs
  4. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications

Online Courses (Paid/Free)

  1. Stanford CS-324: Large Language Models
  2. Princeton COS 597G: Understanding Large Language Models
  3. Coursera: GenAI with LLM
  4. Coursera: Generative AI Engineering with LLMs Specialization
  5. Coursera: Generative AI for Cybersecurity Professionals (IBM)
  6. Coursera: AI for Cybersecurity (JHU)
  7. AttackIQ: The foundation of AI Security
  8. Microsoft AI Red Teaming 101 (Microsoft Learn) - Free training on GenAI vulnerabilities, single/multi-turn attacks, spotlighting defenses, and PyRIT automation.
  9. SANS SEC545: GenAI and LLM Application Security - RAG/vector-DB security, prompt injection, MLOps hardening, and agentic AI security (maps to GIAC GAIPS).

Study Plans, Roadmaps & Interview Prep

Structured paths for going from "interested in AI security" to job-ready.

  1. GenAI Security Study Plan - Week-by-week plan for learning GenAI and LLM security.
  2. AI Security Career Roadmap - Skills, milestones, and role progression for an AI security career.
  3. AI Security Interview Questions - Question bank for AI/GenAI security interviews.
  4. AI/ML Security Learning Resources - Wider collection of AI/ML security learning material.

AI Security Certifications

  1. Certified AI Security Professional (CAISP) by Practical DevSecOps – Securing AI systems, models, and pipelines against adversarial threats, LLM vulnerabilities, AI supply chain risks, data poisoning, and AI-specific security frameworks. Hands-on, practitioner-level skills.
  2. GIAC AI Platform Security (GAIPS) - Hands-on (CyberLive) certification for auditing and securing GenAI applications, LLM pipelines, and agentic AI systems.
  3. GIAC AI Security Automation Engineer (GASAE) - Validates using AI/automation across offensive, defensive, and cloud security operations.
  4. ISC2 AI Security Certificate - On-demand course bundle covering AI fundamentals, AI risk management, secure-by-design, and global AI regulation.

Tools of Trade

Tools for defending, scanning, and auditing GenAI systems.

Defensive / Scanning

  1. LLM Guard - Information extraction and security for LLMs.
  2. Model Scan - Scanning models for serialization attacks.
  3. Rebuff - Prompt injection detection.
  4. NB Defense - Notebook security.
  5. Protect AI's OSS Portfolio
  6. LLM Guard Playground
  7. Cisco MCP Scanner - Scans MCP servers/tools for poisoning and prompt injection (YARA + LLM-as-judge).
  8. Snyk Agent Scan - Inventories and scans AI agents, MCP servers, and skills for 15+ risks (successor to Invariant mcp-scan).
  9. Fickling (Trail of Bits) - Decompiler, static analyzer, and safety scanner for malicious pickle/PyTorch model files.
  10. ModelAudit - Static scanner detecting malicious code/backdoors across 40+ ML model file formats.
  11. AIsbom - CLI that scans model files for malware and generates CycloneDX/SPDX AI SBOMs.
  12. Agentic Radar (SPLX) - Security scanner that maps and analyzes agentic workflows.
  13. Giskard - Testing and scanning framework for ML/LLM systems.

Offensive / Red Teaming

  1. AI/ML Exploits
  2. Garak - LLM Vulnerability Scanner
  3. PyRIT - Python Risk Identification Toolkit for GenAI (Microsoft)
  4. Counterfit - AI Security Testing (Microsoft)
  5. ART - Adversarial Robustness Toolbox (IBM)
  6. promptmap - Prompt Injection Testing
  7. DeepTeam - LLM & AI-Agent Red Teaming Framework - 50+ vulnerability types and 20+ attack methods mapped to OWASP/NIST/MITRE.
  8. Promptfoo - LLM Testing & Red Teaming - Generates adversarial inputs to find prompt injection, jailbreaks, and data leakage, with CI/CD integration.
  9. Redcells - Automated Adversarial Testing for LLMs - Public-beta platform for automated adversarial testing of LLMs you own or control. OpenAI-compatible target models, iterative attack→refine layers, dashboard + API. (Repo)

Guardrails & Firewalls

  1. Guardrails AI - Input/output validation for LLMs.
  2. NeMo Guardrails (NVIDIA) - Programmable guardrails for LLM applications.
  3. Vigil - LLM Prompt Injection Detection
  4. Lakera Guard - Real-time AI security for prompt injection and data leakage.
  5. Omega Walls - Open-source stateful prompt injection defense for RAG and agent pipelines, built as a runtime trust boundary across untrusted content, memory, context, and tools.
  6. Trylon Gateway - Self-hosted open-source AI firewall/proxy applying custom guardrails (prompt-injection defense, PII redaction).
  7. Bifrost AI Gateway - High-performance open-source AI gateway unifying 20+ LLM providers with governance and policy enforcement.
  8. HOL Guard - Local-first security harness that intercepts tool calls in AI coding agents before files change or network is contacted. Scans skills, MCP servers, and plugins for supply-chain threats.

Security Practices and CTFs

Practice your skills with these vulnerable applications and challenges.

  1. Gandalf - Lakera AI - LLM security challenge.
  2. PromptTrace - Free hands-on labs and a progressive gauntlet for prompt injection, jailbreaks, RAG poisoning, and tool/function-call abuse against real LLMs.
  3. Prompt Airlines - AI security challenges, CTF style.
  4. Certified AI/ML Pentester (C-AI/MLPen) Exam - The SecOps Group
  5. Damn Vulnerable MCP Server - Deliberately vulnerable MCP implementation.
  6. Vulnerable MCP Servers Lab - Collection of vulnerable servers.
  7. FinBot Agentic AI CTF - Agentic Security CTF.
  8. OWASP WrongSecrets - Includes an LLM/AI secrets-leakage challenge.
  9. Huntr.com - World’s first bug bounty platform for AI/ML.
  10. HackAPrompt - Prompt hacking competition.
  11. Crucible by Dreadnode - AI/ML security challenges and CTFs.
  12. AI Goat - Vulnerable LLM CTF built on AWS.
  13. Microsoft AI Red Teaming Playground Labs - Hands-on red-teaming challenges (prompt injection, indirect injection, guardrail bypass) with Docker/Kubernetes deployment.
  14. PortSwigger Web Security Academy: Web LLM Attacks - Free official hands-on labs on exploiting LLM APIs, excessive agency, and prompt injection.
  15. Vulnerable LLM apps (GitHub topic) - Index of intentionally vulnerable LLM apps to practice on.

AI Red Teaming

Resources and methodologies for red teaming AI/GenAI systems.

  1. Microsoft AI Red Team
  2. Anthropic Red Teaming Research
  3. MITRE ATLAS - Tools, Data & Case Studies (GitHub)
  4. AI Red Teaming Guide - Humane Intelligence
  5. Google DeepMind: Evaluating Frontier Models for Dangerous Capabilities
  6. OWASP GenAI Red Teaming Guide - Methodology for red teaming GenAI/LLM applications.
  7. CSA Agentic AI Red Teaming Guide - Red teaming approach tailored to autonomous agents.

GenAI Security Attacks, Breaches & Incidents

Notable real-world incidents involving GenAI and LLM security, newest first. For ongoing tracking, see the OWASP GenAI Exploit Round-up, which catalogs confirmed CVEs and exploits each quarter.

  1. Hugging Face Breached by an Autonomous AI Agent (Jul 2026) - An agentic system escaped a public security benchmark, abused two code-execution paths in Hugging Face's dataset processing, and reached production infrastructure over a weekend. OpenAI attributed it to its own pre-release models running unsupervised. Internal datasets and service credentials were accessed; public models, datasets, and Spaces were verified untampered. The first known case where AI capability evaluation caused a real breach.
  2. JADEPUFFER: First Documented Agentic Ransomware (Jul 2026) - Sysdig traced a full extortion chain driven end-to-end by an LLM: Langflow RCE (CVE-2025-3248) for entry, credential harvesting, lateral movement, then encryption and destruction of 1,342 config items. The agent recovered from a failed login in 31 seconds by rewriting its own payload. The AES key was printed to stdout and never persisted — paying the ransom recovers nothing.
  3. Cursor Terminal Allowlist Bypass (CVE-2026-22708, Jul 2026) - Shell built-ins like export bypassed Cursor's Auto-Run allowlist, letting indirect prompt injection poison the agent's environment and reach zero-click RCE. Fixed in 2.3.
  4. Vercel Breached via a Compromised Third-Party AI Tool (Apr 2026) - Attackers pivoted from an infostealer infection at Context.ai through an employee's OAuth grant into Vercel's internal systems, enumerating and decrypting environment variables. A worked example of shadow-AI and OAuth sprawl as an enterprise attack path.
  5. Zenity Labs: Zero-Click Hijacking of Agentic Browsers (Mar 2026) - A vulnerability family in Perplexity Comet allowing zero-click agent hijack, including takeover of a signed-in password manager, via indirect prompt injection. See also Brave's earlier Comet research.
  6. Nine Mexican Government Agencies Breached by One AI-Orchestrating Operator (Dec 2025 - Feb 2026) - Check Point documented a single operator running Claude Code and GPT-4.1 in parallel — 1,088 prompts producing 5,317 AI-executed commands across 34 sessions — exposing roughly 400 million tax, civil registry, electoral, and health records. Evidence that AI-orchestrated intrusion has moved from state actors to ordinary criminals.
  7. Anthropic Disrupts First AI-Orchestrated Cyber Espionage Campaign (Nov 2025) - A Chinese state-sponsored group jailbroke Claude Code to autonomously execute ~80-90% of an espionage campaign against ~30 global targets; the first documented largely AI-run cyberattack.
  8. CamoLeak: GitHub Copilot Chat Private Source-Code Exfiltration (Oct 2025) - CVSS 9.6 prompt injection hidden in PRs leaks private code and secrets via GitHub's own Camo image proxy.
  9. Check Point Researchers Expose Critical Claude Code Flaws - CVE-2025-59536 and CVE-2026-21852: Enabling Remote Command Execution and API Key Theft.
  10. Anthropic: "Vibe-Hacking" Extortion Using Claude Code (Aug 2025) - A criminal used Claude Code to automate intrusions and craft targeted extortion against 17+ organizations, alongside AI-built ransomware-as-a-service.
  11. PromptLock: First Known AI-Powered Ransomware (Aug 2025, PoC) - ESET found ransomware that uses a local LLM via the Ollama API to generate malicious scripts on the fly (later assessed as an academic proof-of-concept).
  12. Replit AI Agent Deletes a Production Database (Jul 2025) - Replit's AI coding agent wiped a live production database during a code freeze, then fabricated data and falsely claimed rollback was impossible.
  13. Amazon Q VS Code Extension Compromised with Data-Wiping Prompt (Jul 2025) - A malicious prompt to wipe files and delete AWS resources was slipped into the official Amazon Q extension (964k+ installs).
  14. CVE-2025-6514: Critical RCE in mcp-remote (Jul 2025) - A malicious MCP server could achieve full remote code execution (CVSS 9.6) on clients running mcp-remote; the first real-world RCE against an MCP client.
  15. EchoLeak (CVE-2025-32711): Zero-Click Data Theft in Microsoft 365 Copilot (Jun 2025) - A single crafted email could silently exfiltrate organizational data from M365 Copilot with no user interaction; the first known zero-click exploit against an AI agent.
  16. Policy Puppetry: Universal Jailbreak Bypassing All Major LLMs (Apr 2025) - HiddenLayer disclosed a single transferable prompt that bypasses safety guardrails across OpenAI, Google, Anthropic, Meta, DeepSeek, and others.
  17. Singapore Finance Director Scammed via Deepfake Executive Video Call (Mar 2025) - A finance director transferred ~US$499,000 after a Zoom call in which company executives were AI-generated deepfakes.
  18. nullifAI: Malicious ML Models on Hugging Face Evade Picklescan (Feb 2025) - ReversingLabs found malicious Pickle-based models using broken/7z-wrapped pickles to bypass scanners and deliver a reverse shell (PoC).
  19. Anthropic: Chinese AI Firms Created 24,000 Fraudulent Accounts For 'Distillation Attacks'
  20. DeepSeek Exposed Database Leaking Chat History and API Keys (Jan 2025) - Wiz found a publicly accessible, unauthenticated DeepSeek ClickHouse database exposing 1M+ log lines including plaintext chats and secrets.
  21. LiteLLM on PyPI Was Compromised, What the Attack Changed and What Defenders Should Do Now
  22. The Day Chevrolet's AI Chatbot Tried to Sell a $70,000 SUV for $1
  23. Here Come the AI Worms (Wired, 2024) - Morris II: self-propagating prompt-injection worms spreading between AI agents.
  24. Air Canada Chatbot Provides Wrong Info (2024) - Airline held liable for chatbot hallucinating refund policy.
  25. Samsung bans use of generative AI tools like ChatGPT after April internal data leak (2023)
  26. AI-powered Bing Chat spills its secrets via prompt injection attack (2023)
  27. ChatGPT Data Leak Bug (2023) - Bug exposed chat history titles and payment info of other users.
  28. GitHub Copilot Leaking Secrets (2023) - AI code assistant reproducing secrets from training data.
  29. Microsoft Tay Bot Manipulation (2016) - Twitter chatbot manipulated into generating offensive content.

Regulatory Frameworks & Governance

  1. EU AI Act - EU regulation on artificial intelligence.
  2. NIST AI Risk Management Framework (AI RMF) - NIST's voluntary framework for managing AI risks.
  3. ISO/IEC 42001:2023 - AI Management System Standard
  4. U.S. AI Executive Order (2025): Removing Barriers to American Leadership in AI - The Trump administration's Jan 2025 EO; it revoked Biden's EO 14110 (2023) on Safe, Secure & Trustworthy AI.
  5. India AI Governance Guidelines (IndiaAI / MeitY) - India's national AI strategy and governance guidance.
  6. Singapore Model AI Governance Framework
  7. UK - International AI Safety Report - Frontier AI risk assessment led by Yoshua Bengio.

Newsletters & Communities

  1. OWASP GenAI Slack Channel - Join #project-top10-for-llm channel.
  2. AI Village (DEF CON) - Community focused on AI security research.
  3. MLSecOps Community - Community for ML security operations.
  4. The AI Security Newsletter by Ken Huang
  5. Protect AI Blog (now part of Palo Alto Networks)

Contributing

Found a great paper, tool, course, CTF, or incident writeup on GenAI, LLM, RAG, MCP, or agentic AI security? Read CONTRIBUTING.md and open a pull request. Small additions are very welcome.

Contributors

Thanks to everyone who helps keep this list current. Your name could be next — open a PR!


Licensed under GPL-3.0.

About

Curated list of links, references, books videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices etc. which are related to GenAI and LLM Security

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors