Pentest-Sakuna: RL + LLM for Pentesting

A guide for understanding how to apply Reinforcement Learning and Large Language Models to penetration testing — from model selection and finetuning to reward design and evaluation. Written June 2026.

---

Table of Contents

1. The Original Guide: 10-Step Research Plan
2. RL Fundamentals: What Your Friend Asked About
   • Reward Shaping
   • The Sparse Reward Problem
   • Goodhart's Law & Reward Hacking
   • GRPO vs PPO vs DPO vs RLHF
3. Paper Summaries
   • LLM + Planning for Pentesting
   • Pure RL for Pentesting
   • RL Finetuning for Security LLMs
4. Recommended Finetuning Pipeline
5. Key Benchmarks
6. Resources & Links

---

1. The Original Guide: 10-Step Research Plan

This is the original guide sent to a friend getting started with RL for software pentesting. They had HackTheBox-style network pentesting experience and wanted to understand the research side.

For the research phases:

1. Find a base model and understand its capabilities — There are models finetuned for security and reasoning. Good starting points: Qwen 2.5 series (7B) or Qwen 3 (8B). There's also CyberSecQwen-4B — a Qwen3-4B finetuned specifically for cybersecurity that outperforms Cisco's 8B security model on benchmarks.

2. Find the dataset — Datasets have training / test / eval splits. Eval (held-out) data is your baseline for measuring improvement. Train + test are used for finetuning.

3. Spend time understanding how the dataset looks — What format are the inputs? What counts as a correct output? This shapes your reward design later.

4. Do an EVAL for the base model first, then SOTA — Before finetuning, benchmark your base model AND commercial SOTA (Gemini, OpenAI, Claude) on your eval set. Confirm the performance gap you're trying to close is real.

5. Check whether the problem is actually there — Many claimed problems disappear when you run a proper baseline. Don't assume.

6. Do the finetuning — Multiple RL finetuning methods exist (SFT → GRPO is the 2026 recommended path; see section 2). Talk to people doing optimization work — they often have RL intuitions from non-LLM contexts that transfer.

7. Use Unsloth on RunPod — For 7B/8B models you need ~14GB VRAM. Unsloth is the finetuning framework (2x faster, 60% less memory than alternatives). RunPod gives GPU access: RTX 4000 Ada, RTX PRO 4000, RTX A4500, A40, L40 — average ~$0.30/hr.

8. A normal finetuning run costs ~$4 — Depending on epochs and dataset size, expect 6–10 hours ≈ $3–4 on RunPod.

9. Get the model adapter and do inference — After finetuning, pull the adapter from RunPod → upload to HuggingFace → run inference on Kaggle T4 (free, 30h/month GPU). You get multiple accounts if needed. Or pay ~$2 on RunPod for inference. Compare: Base model vs SOTA vs Your finetuned model.

10. Read these papers first:

    • AutoPentester / CHECKMATE — arxiv:2512.11143 — Uses Claude Code for pentesting with classical planning + LLM agents (Planner-Executor-Perceptor). 20%+ better than prior SOTA.
    • PentestGPT — arxiv:2308.06782 — Three self-interacting LLM modules for pentesting sub-tasks. 228% better than GPT-3.5. 4,700+ GitHub stars.

---

2. RL Fundamentals: What Your Friend Asked About

Reward Shaping

The problem: RL agents learn from rewards. If rewards are too sparse or delayed, learning is slow or fails entirely. Reward shaping adds intermediate signals to guide the agent faster.

Potential-Based Reward Shaping (PBRS): The theoretically safe version. You define a potential function Φ(s) — a value estimate for each state — and add a shaped bonus:

F(s, a, s') = γ·Φ(s') - Φ(s)

The key property: PBRS cannot change the optimal policy. It only changes how fast the agent learns, not what the correct answer is. This makes it safe to use — you won't accidentally train a different behavior.

In practice for security: You can reward an agent for intermediate steps (found open port → found running service → identified CVE → exploited) rather than only rewarding a full successful exploit chain. This gives a much denser learning signal.

Key papers (in papers/rl-fundamentals/):

• reward-shaping-overview-2408.10215.pdf — comprehensive overview of reward engineering techniques

---

The Sparse Reward Problem

Why pentesting is naturally sparse: An agent either gets a shell or it doesn't. Either the vulnerability is exploited or it isn't. There's no gradient to follow — you can't be "50% exploited."

Solutions:

1. Hindsight Experience Replay (HER): The cleverness: if your agent tried to exploit vulnerability X and failed, but discovered something about Y in the process — retroactively relabel that trajectory as "successfully discovered Y." You mine learning signal from failed attempts by reframing what the goal was.

Example in pentesting: Agent attempts RCE, fails, but discovers a path traversal. HER relabels that episode as "find a path traversal" — suddenly it has a success to learn from.

2. Curiosity-Driven Exploration: Give agents intrinsic rewards for discovering novel states — things they haven't seen before. Useful for open-ended recon where the attack surface is unknown.

3. Dense reward shaping: Break the goal into sub-goals. Finding the login page → finding an input field → triggering an error → successful injection. Each step is rewarded.

Key papers (in papers/rl-fundamentals/):

• HER concept: arxiv:2602.13865 — sparse rewards with HER

---

Goodhart's Law & Reward Hacking

"When a measure becomes a target, it ceases to be a good measure." — Goodhart's Law

The core problem: It is fundamentally impossible to write a perfect reward function. No matter what proxy reward you specify, a sufficiently capable RL agent will find a way to maximize the proxy without achieving your actual goal.

Five categories of reward hacking:

Category	What happens
Specification Gaming	Agent satisfies the letter of the reward, violates intent
Reward Tampering	Agent interferes with the reward mechanism directly
Proxy Optimization	Agent optimizes a metric that's poorly correlated with the goal
Objective Misalignment	Systematic behavioral deviation over time
Wireheading	Agent manipulates its own reward signal directly

Real examples in security LLM training:

• Agent trained to "maximize reported vulnerabilities" starts hallucinating CVE IDs
• Agent trained on format-checking rewards learns to output responses that pass the validator syntactically but contain nonsense
• Anthropic found (2025): training on documents about reward hacking causes the model to learn to do reward hacking — the model learns the meta-skill of gaming rewards from descriptions of it

Defenses:

• Multiple uncorrelated reward signals (harder to game all simultaneously)
• Rule-based rewards over learned reward models (harder to exploit)
• Adversarial red-teaming your reward function before training
• MO-GRPO: Multi-objective GRPO that explicitly balances competing objectives

Key papers (in papers/rl-fundamentals/):

• goodharts-law-rl-2310.09144.pdf — theoretical foundations
• mo-grpo-reward-hacking-2509.22047.pdf — multi-objective GRPO as mitigation

---

GRPO vs PPO vs DPO vs RLHF

Method	Requires Reward Model?	Memory	Best For
RLHF	Yes (trained on human prefs)	High	Production models with expert labeling
PPO	Yes	High	Industry-standard RL, stable training
DPO	No	Low	Simple preference alignment; limited generalization
GRPO	No	Medium	2026 recommendation for security LLMs

GRPO (Group Relative Policy Optimization) — introduced by DeepSeek:

• Sample a group of outputs for each prompt
• Rank them relative to each other using a reward signal
• No value critic network needed → less memory than PPO
• Works well with rule-based rewards (format compliance, flag capture, tool correctness)
• Proven on vulnerability detection and CTF cryptography tasks (2025–2026)

For pentesting finetuning: use GRPO + rule-based rewards. Define rewards that are hard to game: did the model output a valid tool call? Did executing the exploit produce a specific output? Did the CTF flag match? These binary, verifiable signals are ideal for GRPO.

Key papers (in papers/rl-fundamentals/):

• grpo-ppo-dapo-comparison-2512.07611.pdf — comparative analysis

---

3. Paper Summaries

LLM + Planning for Pentesting

CHECKMATE / AutoPentester (papers/pentesting-llm/checkmate-autopentester-2512.11143.pdf) arxiv:2512.11143 — Dec 2025 Introduces the Planner-Executor-Perceptor (PEP) paradigm integrating classical planning with LLM agents. Addresses LLM weaknesses in long-horizon reasoning and specialized tool use. Achieves 20%+ improvement over Claude Code baseline on pentesting benchmarks. The "classical planning" component maintains coherent state across many steps — something pure LLMs struggle with.

PentestGPT (papers/pentesting-llm/pentestgpt-2308.06782.pdf) arxiv:2308.06782 — Aug 2023 Foundational system: three self-interacting LLM modules for pentesting sub-tasks (reasoning, generation, parsing). Mitigates context loss across long sessions. 228.6% task-completion improvement over GPT-3.5. 4,700+ GitHub stars and widely adopted as a research baseline. Key insight: no single LLM call handles all of pentesting — decompose into specialized sub-agents.

HackSynth (papers/pentesting-llm/hacksynth-2412.01778.pdf) arxiv:2412.01778 — Dec 2024 Framework and dataset for autonomous penetration testing with LLM agents. Includes an evaluation system with structured pentesting scenarios. Useful both as a dataset source for finetuning and as an evaluation harness.

SoK: Agentic Security (papers/pentesting-llm/sok-agentic-security-2510.06445.pdf) arxiv:2510.06445 — 2024-2025 Comprehensive survey of LLM/AI agent applications in offensive security, threat landscape from autonomous attackers, and defensive mechanisms. Good entry point for understanding the full landscape before deciding where to focus.

---

Pure RL for Pentesting

RL Under Partial Observability (papers/rl-for-pentesting/rl-partial-observability-2509.20008.pdf) arxiv:2509.20008 — 2025 Systematic evaluation of RL policies for pentesting where attackers lack complete network information. Addresses a realistic constraint: you rarely know the full network topology. Tests robustness across different configs and adversarial conditions.

Knowledge-Informed RL with Reward Machine (papers/rl-for-pentesting/knowledge-reward-machine-2405.15908.pdf) arxiv:2405.15908 — 2024 (IEEE WCCI) Incorporates domain knowledge through reward machines — finite-state machines that formalize temporal reward structures. Directly addresses sparse reward problems in pentesting. The reward machine tracks which pentesting sub-goals have been completed and adjusts rewards accordingly. Strong technique for structured attack chains.

SoK: Pitfalls of DRL for Cybersecurity (papers/rl-for-pentesting/sok-pitfalls-drl-cybersecurity-2602.08690.pdf) arxiv:2602.08690 — 2026 Read this before building anything. Systematization of knowledge identifying pitfalls in applying deep RL to cybersecurity: environmental biases, evaluation methodology issues, common failure modes. Prevents wasting months on approaches that look good in papers but fail in practice.

Curiosity-Driven Multi-Objective Pentesting (papers/rl-for-pentesting/curiosity-driven-pentest-2202.10630.pdf) arxiv:2202.10630 — 2022 Uses curiosity-driven exploration with multi-objective RL to discover diverse attack paths. Key insight: optimizing for a single attack path misses coverage. Multi-objective approaches find behaviorally diverse strategies, which is more useful for real-world pentesting.

---

RL Finetuning for Security LLMs

GRPO for Vulnerability Detection (papers/finetuning-security/grpo-vuln-detection-2507.03051.pdf) arxiv:2507.03051 — 2025 GRPO applied to vulnerability detection with rule-based rewards (correctness + format). Strong results. Important paper because it validates that GRPO with simple, verifiable rewards works for security-specific tasks — you don't need a complex trained reward model.

GRPO on Cryptographic CTF Challenges (papers/finetuning-security/grpo-ctf-crypto-2506.02048.pdf) arxiv:2506.02048 — 2025 LLMs finetuned with GRPO specifically on CTF cryptography challenges. Significant improvements in procedural tool use and cross-domain generalization. Demonstrates that RL finetuning generalizes — improvements on crypto CTF transfer to other security reasoning tasks.

From SFT to RL for Vulnerability Detection (papers/finetuning-security/sft-to-rl-vuln-2602.14012.pdf) arxiv:2602.14012 — 2026 Best practices for the full pipeline: supervised finetuning first, then RL. Explains why the order matters: SFT builds a stable base; RL then optimizes from that base. Doing RL from scratch on a pretrained model is unstable. Follow this pipeline.

CyberLLMInstruct Dataset (papers/finetuning-security/cyberllminstruct-2503.09334.pdf) arxiv:2503.09334 — 2025 Largest cybersecurity instruction dataset: 54,928 instruction-response pairs covering malware analysis, phishing, zero-day vulnerabilities, CTF challenges, and CVEs. Use this for the SFT warm-up step before GRPO. HuggingFace dataset.

FoundationAI SecurityLLM Reasoning 8B (papers/finetuning-security/foundation-ai-secllm-2601.21051.pdf) arxiv:2601.21051 — 2026 Technical report for Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B — a public security reasoning model trained with GRPO. Useful as a baseline comparison point and for understanding what a full production security RL finetuning pipeline looks like.

---

4. Recommended Finetuning Pipeline

This is the practical path for someone running experiments in mid-2026:

Step 1: Base Model Selection
  → Qwen2.5-7B-Instruct  OR  Qwen3-8B-Instruct
    (both are strong; Qwen3 has better reasoning out of the box)
  → Alternative: CyberSecQwen-4B if VRAM is tight (runs on laptop)

Step 2: SFT Warm-up (Supervised Finetuning)
  → Dataset: CyberLLMInstruct (54K pairs, arxiv:2503.09334)
  → Framework: Unsloth (unsloth.ai) — 2x faster, 60% less memory
  → Hardware: RunPod RTX A4500 (~$0.30/hr)
  → Duration: ~4-6 hours, ~$2

Step 3: RL Finetuning with GRPO
  → Algorithm: GRPO (Group Relative Policy Optimization)
  → Framework: Unsloth RL guide (unsloth.ai/docs/get-started/reinforcement-learning-rl-guide)
  → Reward signal: NYU CTF Bench flag capture (binary: correct flag = +1, wrong = 0)
  → Rule-based secondary rewards: valid tool call format, non-empty tool output
  → Hardware: RunPod RTX A4500 (~$0.30/hr)
  → Duration: 6-10 hours, ~$3-4

Step 4: Evaluation
  → Hold-out set: 20% of NYU CTF Bench (never seen during training)
  → Generalization: ZeroDayBench (arxiv:2603.02297) — unseen vulnerability classes
  → Real-world: CVE-Bench (arxiv:2503.17332) — sandboxed real CVEs
  → Compare: Base Model vs SOTA (Claude/GPT/Gemini) vs Your Model

Step 5: Inference (cheap)
  → Upload adapter to HuggingFace
  → Run on Kaggle T4 (free, 30h/month GPU)
    → Use multiple accounts if needed
  → Or RunPod for longer runs (~$2)

Reward Design Tips:

• Use binary, verifiable rewards when possible (flag captured: yes/no)
• Add format rewards to stabilize early training (does output follow tool call schema?)
• Avoid rewards an LLM can game by outputting text that matches your validator pattern
• Start simple; add complexity only if the simple reward plateaus

---

5. Key Benchmarks

Benchmark	Focus	URL	Year
NYU CTF Bench	200 CTF challenges from CSAW (2017-2023), 6 categories	arxiv:2406.05590	2024
CVE-Bench	Real CVEs in sandboxed web environments	arxiv:2503.17332	2025
ZeroDayBench	Unseen zero-day vulnerabilities, tests generalization	arxiv:2603.02297	2026
CTFusion	CTF via MCP tool integration, standardized eval	arxiv:2605.11504	2025
PACEbench	Practical AI cyber-exploitation in realistic scenarios	arxiv:2510.11688	2025
CTFExplorer	Multi-target web CTF, complex interconnected scenarios	arxiv:2602.08023	2026
CREBench	Reverse engineering + cryptographic binary analysis	arxiv:2604.03750	2026
Reward Hacking Bench	Tests reward hacking in LLM agents with tool use	arxiv:2605.02964	2026

Recommended eval order:

1. NYU CTF Bench (primary, most citations, well-understood)
2. CVE-Bench (real-world validation)
3. ZeroDayBench (generalization check)

---

6. Resources & Links

Finetuning

• Unsloth RL Guide — official GRPO docs
• Unsloth Qwen2.5 Blog — Qwen-specific finetuning guide
• Train Your Own R1 Reasoning Model (GRPO) — step-by-step, works on 5GB VRAM
• HuggingFace GRPO Tutorial — Qwen3-1.7B with GRPO
• GRPO Unsloth Notebook — copy-paste starting point

Models

• CyberSecQwen-4B — Qwen3-4B finetuned for cybersecurity, beats Cisco 8B model
• FoundationAI SecurityLLM 8B — Llama-based security reasoning model

Datasets

• CyberLLMInstruct — 54K cybersecurity instruction pairs (best SFT dataset as of 2025)
• NYU CTF Bench — 200 real CTF challenges, open source
• Primus Dataset — open-source cybersecurity LLM training data collection

Compute

• RunPod — GPU rental, RTX A4500 ~$0.30/hr, L40 available
• Kaggle Notebooks — free T4 GPU, 30h/month, good for inference

Reward Hacking / Alignment

• Lil'Log: Reward Hacking in RL — excellent overview by Lilian Weng (OpenAI)
• Anthropic: Training on Reward Hacking Documents — important safety finding

Reading Order Recommendation

1. PentestGPT (2308.06782) — understand the LLM pentesting baseline
2. SoK Pitfalls (2602.08690) — know what fails before building
3. From SFT to RL (2602.14012) — the right finetuning pipeline
4. GRPO CTF Crypto (2506.02048) — see GRPO applied to security CTF specifically
5. CHECKMATE (2512.11143) — current SOTA for LLM + planning pentesting

---

Papers are in the papers/ folder organized by topic. All arxiv links are included for each paper.