The AI Cybersecurity Playbook

Practical controls, agentic testing, and incident workflows built on open-weight models

Version 1.5, July 2026 A practical guide for builders, operators, and security professionals preparing for AI-assisted offensive and defensive cybersecurity.

Who this is for. If you build or run software without a security background, this playbook gives you baseline controls, checklists, and AI-augmented workflows you can adopt without waiting for a full security team. If you already work in security, Parts IV–VI focus on tooling: agentic pentesters, path-aware audit loops, verification, and rollout patterns.

The premise. Frontier AI models can now find vulnerabilities, chain attack steps, and operate over long sessions. Attackers can use this capability. Defenders need reliable access, deployment choice, and portable workflows. This playbook uses Kimi K3 on Fireworks AI as the reference stack while keeping prompts, schemas, and scripts portable across providers.

Safety and legality. Every offensive technique in this playbook exists so you can test your own systems before someone else does. Only run scanners, pentest agents, or exploit validation against assets you own or have explicit written authorization to test. All exercises use local, disposable targets.

Repository. The interactive playbook is published through GitHub Pages. The public repository includes the exercise pack under exercises/.

Companion pieces. The Trilogy AI Center of Excellence article explains the applied case for the playbook. The Fireworks article explains the hosted K3 endpoint and the deployment-choice argument. This document remains the practical resource: controls, tools, loops, labs, and rollout.

Contents

• Part I: Orientation
• Part II: The Baseline: Ten Controls Before Any AI Tooling
• Part III: The Open-Model Advantage
• Part IV: The AI Security Toolchain: Strix, Shannon, and the Classics
• Part V: Build Your Own AI Security Harness: Loops, Graphs, and Subagents
• Part VI: Live POCs and Exercises
• Part VII: Operational Playbooks: Hardening, IR, BCP/DR
• Part VIII: The 30/60/90-Day Rollout
• Appendices: Tool Matrix, Resources, Prompt Library, Glossary, References

Part I: Orientation

1.1 Why this playbook exists

In mid-July 2026, Hugging Face, the world's largest AI model repository, disclosed a breach of its production infrastructure, detected and contained over a single weekend. OpenAI later disclosed that the intruder was a set of frontier models, including GPT-5.6 Sol, running inside an internal cyber-capability evaluation with refusals reduced: the models found a real zero-day, achieved RCE on Hugging Face servers, and logged more than 17,000 autonomous actions. When a commercial frontier model's guardrails refused parts of the forensic analysis afterward, an open-weight model finished the job.¹, ², ³, ⁴²

The operating lesson is direct. Offense can now move at machine speed: the fastest observed access-to-exfiltration is 72 minutes, and the fastest eCrime breakout is 27 seconds.⁴ Attackers can use frontier models. Many defenders depend on access programs, provider policies, or commercial endpoints they do not control.⁹ Open weights change the defensive architecture by giving teams a portable model option, hosted access today, and a path to dedicated or self-hosted deployments when the workload justifies it.

For the full incident anatomy, the capability benchmarks, and the industry context, including the July 24 open letter on open weights and American AI leadership,⁴⁹ read the companion article. This document stays practical from here on.

1.2 How to read this playbook

• New to security? Read Part II carefully, run Lab 0 and Lab 5, then work the 30/60/90 plan in Part VIII. Skim Parts IV–V for vocabulary; come back when the baseline is in place.
• Engineering leader at a small company? Parts II, VI, and VII are your operating manual. Assign the checklists; run the exercises as a team workshop (they fit in a day).
• Security professional? Jump to Parts III–VI. The comparison matrices, harness architectures, and the divide-and-conquer audit loop are built for you, and the exercises double as evaluation harnesses for K3 against your own tooling.
• Everyone: Part II is non-negotiable. AI tooling amplifies whatever posture you already have. It does not substitute for one.

Part II: The Baseline: Ten Controls Before Any AI Tooling

AI codegen closed the feature gap between proof of concept and production without closing the hardening gap. Production changes three things at once: exposure (the app is reachable), value (real users, real data), and velocity (AI-written code ships daily). Your controls must match all three.

This part is the baseline doc you can hand to any team. Each control has a what, a why now, and a first hour: the cheapest meaningful version. Aim for coverage, not perfection; Part VIII sequences it.

2.1 Identity first: every app behind real auth

What. Internal tools and administrative surfaces use company-managed identity: SSO via your IdP (Google Workspace, Okta, Entra ID) or the org tiers of your platform providers. Public applications define their unauthenticated surface explicitly and enforce production-grade authentication and authorization everywhere else. No shared logins.

Why now. Identity weaknesses were implicated in ~90% of the major incidents Unit 42 analyzed this year.⁴ AI-generated phishing and credential-stuffing at machine speed make "we'll add auth later" a breach plan, not a roadmap item.

First hour. Move your Vercel/Supabase/Netlify/etc. projects out of personal accounts into a company org tier with SSO and audit logs. This single move gives you: login through company identity, offboarding that actually removes access, and visibility into what apps exist in the first place. Shadow-IT apps on personal accounts are invisible until they are breached.

2.2 Secrets out of code, into a manager

What. No API keys, tokens, or passwords in repositories, CI logs, or chat. Secrets live in a manager (Doppler, 1Password, Vault, cloud-provider secret stores) and reach code through environment injection.

Why now. The Hugging Face attackers' primary prize was harvested credentials, the skeleton keys to everything else,³ and AI coding agents will cheerfully hardcode a key into source if you let them (see the Lab 3 sample app, which plants exactly this).

First hour. Run a secret scan across all repos (gitleaks detect, or GitHub's built-in scanning, which is free). Rotate anything found; a committed secret is compromised by definition; deletion is not remediation. Then add the scan to CI so it stays clean.

2.3 Decide what data is allowed in AI tools

What. A one-page data classification: what may never leave your systems (customer PII, credentials, PHI/PCI), what may go to contracted APIs with BAAs or DPAs, and what is fair game for any tool.

Why now. Your team is already pasting data into AI tools. The question is whether they paste the customer table. Regulated data (healthcare: HIPAA; payments: PCI) has legal routing requirements: this is where provider agreements, data-processing terms, and deployment boundaries stop being paperwork and start being architecture.

First hour. Write the three buckets on one page, name an owner for questions, and post it where your team actually reads. Perfect taxonomy later; a clear "never paste these five things" today.

2.4 Supply-chain hygiene: your dependencies are your attack surface

What. Pin dependencies, review lockfile changes, scan for known-vulnerable packages, and treat build/CI plugins as privileged code.

Why now. Two receipts from the past year. React2Shell (CVE-2025-55182, CVSS 10.0) gave unauthenticated RCE under default React 19/Next.js configurations, with public exploits within days and an exposed base in the millions: companies that knew their dependency inventory patched in hours, the rest found out later.¹⁵, ¹⁶, ¹⁷ And the LiteLLM poisoning backdoored a popular AI gateway through a compromised security-scanning package: the tools you add to improve security are also attack surface.¹⁸, ¹⁹

First hour. Enable Dependabot/Renovate (free) on every repo; turn on npm audit/pip-audit/Trivy in CI; pin exact versions in lockfiles and require review for lockfile diffs.

2.5 Secure defaults for AI-generated code

What. A short, enforced checklist that every AI-generated feature passes before merge: auth on by default, parameterized queries, output encoding, input validation at the boundary, no debug=True, no disabled TLS verification.

Why now. AI coding agents optimize for "works," not "safe": they reproduce the internet's average code, including its average vulnerabilities, at 10x velocity. Put a gate between generation and production, then aim the same class of model back at the code as a reviewer (Part V). The drop-in AGENTS.md security section in the exercise pack (prompts/AGENTS-security-template.md) encodes these rules where the agent itself will read them.

First hour. Adopt the AGENTS.md template in your two most-active repos and add the five checks above to your PR template.

2.6 Logging you will actually use

What. Centralized auth logs (who logged in, from where, when), application error logs, and an alert on the two or three patterns that matter (brute force, impossible travel, off-hours admin access).

Why now. Hugging Face caught a 17,000-action autonomous intrusion with log anomaly detection.¹ The defender-side corollary of machine-speed attacks is that human-speed log review is no longer detection (it is archaeology). You do not need a SOC; you need logs that exist, centrally, plus one automated analyst (Lab 5 builds one with K3).

First hour. Ship auth and app logs to one queryable place (a Wazuh instance, your cloud's native logging, even a single rsyslog box with lnav), and verify you can answer: "list all successful logins for user X in the last 30 days."

2.7 Backups that restore

What. 3-2-1: three copies, two media, one offline/off-site, with a tested restore.

Why now. Ransomware is the canonical business-ending event for small companies, and agentic ransomware operations are now documented in the wild (Sysdig's JADEPUFFER tracking is the current reference case).²⁵ Backups are the difference between an incident and an obituary.

First hour. Verify one full restore of your most important database. Not "the backup job is green": actually restore it, somewhere, and open the data.

2.8 A breach runbook people can find at 2 a.m.

What. One page: who is on point, who gets called, what gets preserved, what gets isolated, what gets said (and by whom). Part VII §7.2 has the template.

Why now. At machine-speed attack tempos, the first hour decides the outcome; Unit 42's 72-minute median-to-exfiltration cases leave no room for figuring out who is in charge.⁴ Most people who end up running a breach response have never done it before: the runbook is what makes that survivable.

First hour. Copy the Part VII template, fill in three names and three phone numbers, and pin it in your ops channel.

2.9 Offboarding that actually removes access

What. A checklist that fires on every departure: IdP account disabled, org-tier platform access revoked, shared credentials rotated, tokens and SSH keys inventoried and pulled.

Why now. Orphaned access is the quietest breach vector there is, and with personal-account shadow apps (see §2.1), you cannot revoke what you cannot see.

First hour. Audit last year's departures against current access. Anything still open gets closed today; then wire the checklist into HR's process.

2.10 Guardrails for your own AI agents

What. Rules for the agents you operate: least-agency tool permissions, separate scoped credentials per agent, human approval for high side-effect actions (deletion, deployment, purchases, external messages), and sandboxed execution for anything that runs generated code.

Why now. The OWASP Top 10 for Agentic Applications was built from real incidents, not projections: EchoLeak (CVE-2025-32711), a zero-click indirect prompt injection that exfiltrated data through Microsoft 365 Copilot; the Amazon Q compromise, where a coding assistant with 950,000+ installs was weaponized; and Replit's agent deleting a production database during a code freeze.¹³, ¹⁴ An agent's exposure equals every credential, tool, and API it can reach, and multi-step autonomy compounds damage across a whole plan rather than a single response.

First hour. For each agent your team runs: give it its own credentials (not a developer's), revoke what it doesn't need, and require confirmation for irreversible actions. The OWASP Agentic Top 10 table below is your design review checklist.

2.11 Frameworks: a shared language, not homework

Frameworks exist so a finding, a control, and a report all mean the same thing to everyone. You need four maps. Learn to read them, not memorize them:

OWASP Web Top 10 (2025). The baseline vocabulary for application risk: Broken Access Control, Security Misconfiguration, Injection, and friends.¹¹ Every finding in Part VI's labs is labeled with it. When Shannon or Strix reports "A03: Injection," this is the dictionary.

OWASP Top 10 for LLM Applications. Model-level risks (prompt injection, sensitive information disclosure, supply chain, excessive agency) for anything that puts an LLM in front of users.¹²

OWASP Top 10 for Agentic Applications (2026). The newest and, for teams deploying AI agents, the most important. Ten categories, ASI01–ASI10:¹³, ¹⁴

MITRE ATT&CK and the Cyber Kill Chain. ATT&CK is the taxonomy of attacker behavior (recon, initial access, persistence, lateral movement, exfiltration); use it to label what a finding would enable, and to check your detection coverage against real techniques.²⁰, ²² The kill chain's operational insight: you do not have to stop everything (you have to break the chain once, early). Each control in this part is a chain-breaker at a different link.

Zero Trust (NIST SP 800-207). The organizing principle behind §2.1–§2.3: never trust by network location, always verify identity, least privilege per request.²¹ For a small company, this becomes an operating habit: ask “what identity is making this request, and is it allowed exactly this?”

How to use the maps without drowning

Run every surface through three questions, the same loop the workshop class uses: as a checklist (what can fail here?), as a backlog (what needs an owner, a fix, and a verification?), as a report (where does evidence and residual risk live?). A risk that ends in evidence and verifiable work is managed. A risk that ends in a slide is not.

Part III: The Open-Model Advantage

The industry case for open weights, including the July 24 open letter and its signatories, is told in the Fireworks article. This part keeps the operating consequences.

3.1 Over-refusal is now a security regression

When a safety system cannot distinguish defense from offense, it defaults to disarming the defender, because the defender is the one who follows the rules. The receipts: during the Hugging Face response, a commercial frontier model refused to analyze real attack commands, exploit payloads, and C2 artifacts, and an open-weight model completed the forensics instead.¹, ² Anthropic's Fable 5 shipped cyber guardrails researchers described as keyword-triggered, intercepting even requests to write secure code.⁸, ¹⁰ Attackers route around refusals (jailbreaks, open models, stolen access); defenders queue for access programs.

Operating guidance. Refusal is not the enemy; indiscriminate refusal is. A model with context you control (your system prompt, your harness) beats a vendor-side keyword filter for professional defensive work, and that is what the architecture in Part V provides. Test your IR copilot against real attack artifacts before you need it.

3.2 Self-sufficiency: own your defense capability

Open weights reduce provider risk by creating an exit path: a portable client today, a tested second provider next, and a dedicated or self-hosted deployment when the workload justifies it. A hosted open-model endpoint can still be unavailable, rate-limited, or contractually unsuitable during an incident. A self-hosted or dedicated deployment removes more provider dependency after the team has validated the license, serving stack, hardware, and operational ownership.

The practical claims are:

1. Availability. A model used in security operations is part of the attack surface. On June 12, 2026, every Fable 5 and Mythos 5 customer lost access on a day's notice through no action of their own.⁹ Open weights create a migration path around a single endpoint.
2. Data control. Incident response involves attack logs with internal hostnames and usernames, source code, credential-adjacent artifacts, and customer data. Provider retention settings, DPAs, BAAs, regional processing, dedicated deployments, and self-hosting determine which material can leave your systems.
3. Economics. Defenders audit every repo, triage every scan, and review every PR; an attacker needs one hole. K3 on Fireworks costs $3.00 per million input tokens, $0.30 per million cached input tokens, and $15.00 per million output tokens. The exercise scripts keep these values in an overrideable table. For calibration, the UK AISI measured a 100M-token autonomous cyber-range run at roughly $85 on Opus 4.8, $46 on GLM-5.2, and $1.19 on DeepSeek V4-Pro.⁴³
4. Diversity and routing. Models specialize by task. A routing layer can send code-audit fan-out to the cheapest capable model, deep reasoning passes to the strongest available model, and critical findings to a second model family. Fireworks' ~1,030-task agentic study found that its oracle router selected K3 for most tasks, K3 led on terminal and security work, and K3 plus Fable reached 93% accuracy.⁴⁵ A separate Kimi K3 and Qwen 3.8 Max architecture benchmark produced its strongest design by combining Kimi's lifecycle model with Qwen's system boundary and replay record.

Routing in practice.

• Effort follows blast radius. Use lower effort for first-pass triage, log reduction, and summarization. Use higher effort for path-reachability analysis, verification passes, and incident-response copilot sessions.
• Stable prefixes improve cache reuse. Put the system prompt and schema first, followed by the file or log chunk that changes per call.
• Route by task. Use the cheapest capable model for fan-out, the strongest available model for verification, and a second family for critical cross-checks. K3_PROVIDER and K3_MODEL are the routing seam, so the loops stay unchanged when the endpoint changes.
• Measure each default. Every script records token usage and estimated cost in its manifest when the provider returns usage fields. Lab 3 provides the calibration run. Rate cards rank cost; your evaluation runs rank models on your repositories.

3.3 The reference stack: Kimi K3 on Fireworks

Parts V and VI use a frontier open-weight model behind an OpenAI-compatible endpoint, called through a provider-neutral client. The reference stack is Kimi K3 on Fireworks AI.

• Model: Kimi K3, a 2.8T-parameter Stable LatentMoE model with a 1,048,576-token context window and native multimodal capability.²³
• Serving: Fireworks exposes OpenAI-compatible inference at https://api.fireworks.ai/inference/v1 with model ID accounts/fireworks/models/kimi-k3. Pricing is $3.00 per million input tokens, $0.30 per million cached input tokens, and $15.00 per million output tokens.²⁴
• Zero Data Retention: Fireworks applies Zero Data Retention by default to open-model inference. Prompt and generation data stay in volatile memory for the request and are excluded from persistent logs unless the user opts in. Fireworks retains service metadata such as token counts. The Responses API stores conversations for 30 days when store=true, its default, so sensitive audit workflows must send store=false.

Why this pairing works for security:

• Large-context headroom. A 1M-token window holds long incident timelines, tool transcripts, and audit state. The playbook still shards source code by attack path because bounded analysis improves precision and makes coverage auditable.
• Long-horizon agentic behavior. K3 was built for multi-hour, tool-using tasks: the same shape as a pentest agent run, a source-audit loop, or an incident copilot session.
• Standard access for authorized defensive work. Fireworks provides hosted access under its terms, acceptable-use policies, data-handling settings, rate limits, and safety controls. Public weights add dedicated and self-hosted deployment paths.

Operational limits:²³

3.4 Where restricted models still fit

Restricted cyber models add useful routes when your organization has access. Mythos 5 via Glasswing and GPT-5.5-Cyber via Trusted Access for Cyber support authorized red-team work through their respective programs.³⁴, ³⁵ Google's new Gemini 3.5 Flash Cyber is a lightweight CodeMender model fine-tuned to find, validate, and patch vulnerabilities. Google is introducing it through a limited-access pilot for governments and trusted partners. Use each restricted model for the tasks its program covers, keep K3 available for standard defensive workflows, and preserve prompts, schemas, and run state outside any one provider.

3.5 Self-sufficiency is a level, not a label

One correction this playbook owes to its reviewers: "open weights" is not synonymous with uninterrupted access. A hosted open-model endpoint can also be unavailable, restricted, degraded, or contractually unsuitable. Self-sufficiency is an operating capability, measured in levels:

Set a target level per workload, not per company: the nightly audit loop and the IR copilot do not need the same answer. Practical guidance:

• The exercise pack is S1-ready and S2-shaped: the client is provider-neutral, and the prompts, schemas, and artifacts are files you own. Automated failover remains implementation work before you can claim full S2.
• The public weight release makes S3 possible. A dedicated deployment, through your own GPUs or a reserved-capacity host, can remove hosted-provider availability from the threat model. Budget for real hardware: K3's MXFP4 weights are roughly 1.4 TB, so this is a multi-accelerator deployment, not a laptop. Validate the license and serving stack before promising this to leadership.
• S4 is a kit, not a platform: a cached smaller open model, the deterministic halves of this pack's scripts, cached copies of your runbooks and this playbook, and an annual drill where the team triages an incident without external services.

This framing makes the public argument stronger: the claim is that defense-critical capability should reach at least S2, and the path to S3 should exist before you need it.

Part IV: The AI Security Toolchain

Three categories of tool matter now: agentic pentesters (AI agents that probe and exploit your apps the way an attacker would), fix loops (agents that close findings), and classic primitives (nmap, semgrep, Trivy), now with an AI analyst on top. This part is a field guide, with setup for the two tools we recommend starting with: Shannon for white-box and Strix for black-box.

A word on posture before the catalog: these are active tools. Their agents create users, submit forms, mutate data, and execute real exploits to prove findings. That is their value (no exploit, no report) and their risk. Point them only at local, staging, or disposable environments, never production, and follow each tool's authorization gates.

4.1 Shannon: white-box agentic pentesting

What it is. Shannon (KeygraphHQ, open source, AGPL-3.0) is an autonomous white-box AI pentester for web apps and APIs: it reads your source code, maps the attack surface, and executes real exploits against the running application to prove vulnerabilities. Its rule is if it can’t exploit it, it doesn’t report it, which keeps speculative findings out of the report.²⁶ On the XBOW security benchmark it posts a 96.15% exploit success rate (100/104).²⁷

Architecture. Five phases, with parallel analysis and exploitation:²⁶, ²⁷

Key properties for planning:²⁶, ²⁷

• Runtime and cost: target size, test scope, provider, and retry count determine both. Resumable workspaces preserve completed work after an interruption.
• Coverage: actively exploitable Injection, XSS, SSRF, Broken Authentication, and Broken Authorization. Broader static coverage, including dependencies and misconfiguration, lives in the commercial Keygraph platform.
• Authenticated testing: a YAML config describes login flows (form or SSO), test credentials, TOTP secrets, focus paths, and avoid rules such as /logout and destructive admin actions. Use these every run.
• Model support: Shannon 2.0 runs on the model-agnostic Pi agent harness. Configure K3 through Pi with the Fireworks endpoint and model ID shown below.⁴⁷ Calibrate the setup on a known-vulnerable local target before using it on an application. Do not scan untrusted codebases with it: source text can prompt-inject a white-box agent.²⁶

Setup for K3 on Fireworks:

export FIREWORKS_API_KEY="fw_..."

npx @keygraph/shannon@latest setup
# In the Pi custom-provider configuration:
# API base: https://api.fireworks.ai/inference/v1
# Model:    accounts/fireworks/models/kimi-k3

npx @keygraph/shannon@latest start \
  -u https://your-staging-app.com \
  -r /path/to/your/repo

4.2 Strix: black-box (and grey-box) agentic pentesting

What it is. Strix (open source) fields teams of autonomous AI "hackers" that plan attacks, drive a real pentest toolkit, interpret results, chain vulnerabilities, and produce working PoCs: a think-plan-act-observe loop rather than a script.²⁹, ³⁰

The toolkit inside the sandbox: an HTTP interception proxy (Caido) for request/response manipulation, an automated browser for XSS/CSRF/auth flows, an interactive terminal, a Python sandbox for custom exploit validation, recon/OSINT for attack-surface mapping, and static+dynamic code analysis. Findings land in a structured knowledge base with CVSS scoring and OWASP classification.²⁹

Multi-agent orchestration. Strix arranges agents in a graph: specialized agents (recon, exploitation, post-exploitation) run in parallel across targets, share discoveries, and redirect each other as new attack paths open.²⁹, ³⁰

Why Strix is the natural K3 vehicle. Strix is LLM-agnostic (built on LiteLLM): you select the brain with environment variables, including any OpenAI-compatible endpoint, which is exactly what Fireworks exposes:²⁹

# Supply-chain hygiene (§2.4 applies to security tools too): pin a release,
# inspect the installer or package, and verify checksums where the project
# publishes them. Do not run curl | bash in a security workflow.
# Example after installing from a reviewed, pinned release:
export STRIX_LLM="openai/accounts/fireworks/models/kimi-k3"
export LLM_API_KEY="$FIREWORKS_API_KEY"
export LLM_API_BASE="https://api.fireworks.ai/inference/v1"

strix --target https://your-staging-app.com

Modes that matter:

strix --target ./app-directory                     # white-box: local source
strix --target https://github.com/org/repo         # source review of a repo
strix --target https://staging.example.com         # black-box web assessment
strix --target api.example.com --instruction "Focus on IDOR and business logic"
strix -n -t ./ --scan-mode quick                   # headless, CI-friendly;
                                                   # exits non-zero on findings
strix -n --target ./ --scan-mode quick --scope-mode diff --diff-base origin/main
                                                   # PR diff-scoped review in CI

The last two are the sleeper features: headless mode plus diff-scoping turns Strix into a per-PR pentest gate that only burns tokens on what changed.²⁹

4.3 Pensar Apex: terminal-native targeted testing

Apex is a terminal/TUI pentest agent with clean black-box/white-box separation: pensar pentest --target <url> for external probing, --cwd ./repo to add source-code analysis, and targeted-pentest --objective "Test authentication bypass" for hypothesis-driven work.³¹ Its white-box mode deterministically maps frameworks, endpoints, and auth requirements from source before testing them against the live target, emitting structured JSON attack-surface maps that also work as inventory.³² Provider support includes Anthropic, OpenAI, Bedrock, OpenRouter, and vLLM (local models); through OpenRouter you can route to Kimi K3 (moonshotai/kimi-k3).²³, ³¹

Install: npm install -g @pensar/apex, or use the curl/Brew options documented in the repository.

4.4 Which agent when: comparison matrix

Recommended default: Shannon for the deep pre-production white-box pass on your flagship app; Strix (on K3 via Fireworks) for external surface and CI diff-scope; Apex when someone says "I just want to hammer on this one auth flow."

4.5 The fix loop: findings are the easy half

Discovery is now cheap; remediation is the bottleneck. OpenAI's Daybreak framing is explicit that "reports alone do not make systems safer."³⁴ Two tools close the loop:

• OpenHands Vulnerability Fixer (open source, MIT): point it at a GitHub repository for a Trivy scan or upload SARIF 2.1.0 reports from other scanners. Agents generate, test, and open pull requests for each finding. Strix writes findings.sarif natively. Shannon Lite produces its final report in Markdown, so convert its findings to SARIF before upload. The repository also provides a reference implementation for custom fix agents on the OpenHands Cloud API.³³
• Codex Security / Daybreak ecosystem (restricted): if you are in OpenAI's program, the plugin scans commits, triages scanner findings, and generates codebase-specific patches: the 30M-commits/70k-fixes numbers show what this looks like at industrial scale.⁷, ³⁴, ³⁶

The pattern to internalize (and the one Lab 6 drills) is find → fix → verify: no finding is "done" until the patch exists and the same tool that found it can no longer reproduce it. Shannon's "no exploit, no report" has a remediation corollary: no retest, no fix.

4.6 Classic primitives, AI analyst

Do not throw out the old tools. The durable pattern for 2026: deterministic tools produce facts; the model reasons about them. The model never runs the scan (slow, flaky) and never counts log lines (it is bad at counting); scripts do both, and the model does the correlating and explaining it is actually good at.

Why this division of labor survives contact with reality: LLM accuracy degrades in very long contexts and counting tasks, while its reasoning over pre-digested facts is excellent.³⁷ Compute statistics in code; let K3 do judgment. (Labs 4–5 are built exactly this way: the scripts' --dry-run mode shows you the deterministic half with zero API spend.)

4.7 Where Metasploit fits now

Metasploit (and related tools such as Burp Suite and sqlmap) is not obsolete (it is specialized). The agentic tools above internally orchestrate the same classes of actions; where the classic frameworks still earn their place:

• Structured validation and retesting of specific known CVEs with stable, reviewed exploit modules (e.g., confirming a React2Shell patch actually closed the hole on your staging instance).
• Training and calibration: Metasploitable/DVWA-style ranges remain the best way to build the mental model of what the agents are doing under the hood.
• Regulated engagements that specify tooling and evidence formats.

For most small-company defenders, the agentic tools cover routine needs; reach for Metasploit when you need a named, reviewable, repeatable module rather than an autonomous operator. If you operate critical infrastructure, check your eligibility for the restricted programs and run both stacks when access is granted.

4.8 The evidence ladder: what "verified" actually means

"Verified" is the most overloaded word in AI-assisted security. A second LLM reviewing a finding (our Lab 3), a pentest agent's working exploit (Shannon), and a human re-firing the original attack path after a patch (Lab 6) are three different kinds of evidence, and treating them as equivalent is how false positives ship to backlogs and false closures ship to production. Label every finding with its rung on the ladder:

How the tools in this part map:

• code_audit.py --dry-run heuristics produce E1; the model fan-out with evidence and attack paths produces E2 candidates; the --verify pass is a fresh-context review that confirms or rejects the E2 reasoning. It does not create E3, because nothing has touched a running system.
• Shannon and Strix earn their keep precisely at E3: "no exploit, no report" means every finding arrives with reproduced impact (§4.1). That is why their reports are short and their false-positive rate is low.
• Closure requires E4. A patch plus the same instrument that found the issue now failing (Lab 6) plus regression tests plus an independent check. "The model says it's fixed" is E0 with extra steps.

Two operating rules fall out of the ladder. First, reports must label the rung: a backlog that mixes E0 hypotheses with E3 proofs is not a backlog, it's noise; triage by level. Second, escalate evidence before escalating severity. Before an E2 finding gets a critical label and an engineer's weekend, spend the thirty minutes to reach E3. The ladder is also the honest answer to the benchmark question: when Semgrep measures precision on scanner output,⁴⁴ they are measuring E1/E2 quality: your harness's job is to push findings up the ladder before they cost human time.

Part V: Build Your Own AI Security Harness

Off-the-shelf agents cover the common cases. The durable skill (the one that transfers when tools change) is wiring a frontier model into your own security loops. This part covers the two patterns practitioners converge on (divide-and-conquer analysis and tool-plus-analyst), the three orchestration strategies available in 2026 (native subagents, prompt-directed fan-out, deterministic scripts), and how to choose between them.

5.1 Why the one-shot whole-codebase prompt fails

The naive move (paste the repo into a giant prompt and ask "find the vulnerabilities") fails three ways:

1. Accuracy decays with context fill. Independent harness testing keeps reproducing the finding that model accuracy drops meaningfully once conversations push into the high tens of thousands of tokens; more context makes models dumber past a threshold, not smarter.³⁷ K3's own launch data echoes it: on BrowseComp, selective context compaction at 300K (91.2) beat dumping the full 1M window (90.4).²³ A bigger window is headroom, not a strategy.
2. You get a sermon, not a schema. Whole-repo prompts produce prose summaries with uneven coverage: some files audited three times, some never touched, no dedupe, no verification, no way to CI it.
3. Cost and latency scale the wrong way. You pay full price to re-read the entire codebase for every question, instead of paying small prices to read the right files deeply.

Independent measurements back both halves of this. Semgrep's 2026 benchmark ran the same models through two different scanning harnesses and watched rankings reshuffle. Kimi K3's F1 moved from 0.340 with simple guided prompts to 0.400 in a richer multimodal harness, and every model degraded sharply on the largest repository.⁴⁴ Aikido's CVE-reproduction benchmark adds the economic half: several runs of a cheap model approximated one run of a flagship, so repeated well-sharded passes beat single heroic prompts.⁴⁸ Divide-and-conquer is how strong models are used correctly.

The fix is the pattern that has shown up in practitioner write-ups all year: divide and conquer with function-to-file granularity: plan first, fan out small, verify hard, aggregate deterministically.

5.2 The divide-and-conquer audit loop

This is the reference architecture implemented by code_audit.py in the exercise pack (Lab 3), and it generalizes to nearly every security-analysis task:

Design principles worth stealing:

• Small, stateless calls. Each fan-out call sees one file plus a system prompt (nothing else). No conversation history to manage, no cross-contamination between files, and K3's multi-turn thinking-history sensitivity never enters the picture.²³
• The model proposes, code disposes. Anything countable (dedupe, sorting, statistics, the repo map itself) is done in Python, not by the model. LLMs are bad at counting and excellent at judgment; allocate accordingly (§4.6).
• Verification is a separate context. The same conversation that proposed a finding will defend it (anchoring). A fresh, skeptical pass (the "prove it or drop it" prompt in the exercise pack) is what turns a raw LLM audit into a trustworthy backlog. Shannon's whole product philosophy ("no exploit, no report") is this principle, productized.²⁶
• Schemas or it didn't happen. JSON-only outputs with title/category/severity/line/evidence/attack_path/fix make findings diffable across runs, which is what turns an audit into a regression suite.

Granularity guidance: file-level chunks (≤ ~12k chars) are the sweet spot for most audits; drop to function-level splitting for large files (code_audit.py splits on top-level def/class boundaries) and for targeted re-review of hot spots. Going finer than function-level usually just re-pays context costs (imports, types) without accuracy gains.

5.3 Loops vs. graphs

The trending vocabulary, made concrete:

• A loop is linear and re-entrant: map → plan → fan-out → fan-in → verify → (optionally) re-plan on what was found. Simple, predictable, CI-able. Our audit loop is one.
• A graph lets branches route conditionally and agents message each other: Strix's agent graph (recon agent's discovery redirects exploitation agents mid-run) is the packaged example; LangGraph-style orchestration is the framework version.³⁰, ⁴¹

Rule of thumb: start with a loop; graduate to a graph when the plan itself must change mid-run. A nightly code audit is a loop. An autonomous pentest (where an SSRF discovery should immediately spawn internal-service probes) is a graph. The failure mode to avoid is graph-shaped complexity for loop-shaped problems: conditional routing multiplies debugging surface and token spend, and most defensive workflows are genuinely linear with a verification branch.

5.4 Three ways to orchestrate (and when to use each)

Option A: Native subagents in an existing harness

The major agent harnesses now ship sub-agent primitives you can lean on directly:

• Codex (OpenAI): subagents went GA in March 2026: built-in explorer/worker/default agents plus custom agents defined as TOML files in ~/.codex/agents/, each with its own instructions and optional model assignment, invoked by name in prompts ("Have code_mapper trace the path, then ui_fixer implement the fix"). Under the hood the parent gets spawn_agent / wait_agent / send_message / close_agent primitives.³⁸, ³⁹
• Claude Code: Agent Teams for parallel sub-agents, project memory via CLAUDE.md, skills. Note the cyber-guardrail caveat from Part III for security-flavored work.
• OpenCode: the open-source, MIT-licensed, most-starred harness (180k+ stars), with model-agnostic BYOK across 75+ providers, MCP + LSP support, and sub-agents. It is a low-friction path for running a full agent harness on Kimi K3 via Fireworks after Fireworks provider settings and the final model ID are configured.⁴⁰
• OpenHands: the Cloud API/SDK route when you want programmatic multi-conversation orchestration (the Vulnerability Fixer app is the reference implementation).³³
• Pi: primitives-first with an RPC mode: the right substrate if a coding/security agent is one component of a larger system you are building.³⁷

Use when: a human is in the loop, the task is exploratory (triage this alert, investigate this repo), or you want the harness's sandboxing, permissions, and tool ecosystem for free.

Option B: Prompt-directed fan-out

Ask the orchestrating model itself to plan and delegate: "You are the lead auditor. List the files worth auditing, then audit them one at a time, tracking findings in JSON. Verify each critical finding before reporting." Modern long-horizon models (K3 included) will spontaneously decompose, spawn subagent calls where the harness allows, and self-verify. Moonshot's K3 is explicitly trained for this, which is also why it needs approval boundaries stated up front.²³

Use when: prototyping a workflow before hardening it, one-off investigations, or teaching (it makes the model's decomposition visible and discussable). Weakness: non-deterministic (same input, different decomposition), so don't put it in CI, and always pair it with a human or scripted verification gate.

Option C: Deterministic harness (scripts)

Code drives; the model fills judgment-shaped holes. This is code_audit.py / nmap_triage.py / log_triage.py: the walking, chunking, concurrency, dedupe, statistics, and report rendering are Python; K3 appears only at "plan," "analyze this unit," and "verify this finding."

Use when: the workflow runs unattended (nightly, per-PR), must be reproducible and auditable, must enforce budgets, or feeds other systems (ticketing, dashboards). This is where compliance-friendly evidence trails come from.

Decision matrix

The pragmatic path most teams take: prototype with B, operationalize with C, and use A for the messy interactive work in between. And remember the harness-effect research: the same model swings 5–40 percentage points on agentic benchmarks depending on the harness around it: the engineering outside the model is not packaging, it is capability.³⁷

The exercise pack implements one representative of each pattern, so the trade-offs are runnable rather than rhetorical. Here they are answering the same question (the planted SQLi in the sample app):

None is "best." The first two are production loops with different units of analysis (§5.7 is why paths usually win); the third is how you investigate when you cannot enumerate the space in advance, and it is the reference skeleton for building your own agent (Lab 9).

5.5 Context engineering for security work

• 1M context is for holding the case, not for skipping design. Whole-incident log sets, a full pentest transcript, the complete audit state: K3's window gives an IR copilot session room to preserve context (Part VI, Lab 7). Per-unit analysis still works best small and focused (§5.1).
• Cache-friendly prompt structure. K3's cached-input price is a tenth of fresh input, and Moonshot reports >90% cache hit rates on programming traffic: put your stable system prompt and schemas first, vary only the tail (the file/log chunk), and fan-out costs collapse.²³
• Reasoning budget is real budget. K3 always reasons (effort configurable, max by default); a max_tokens ceiling that's too small returns empty completions. The scripts default to 8–16k for analysis calls for this reason, and the pack's client exposes reasoning_effort for cheap bulk passes.²³, ⁴⁶
• Measure your spend, don't vibe it. The pack's client accumulates token usage per run (cached vs. fresh input, output) and estimates cost from a price table you can override; run-manifest.json records both. Budgets for fan-out loops become a measurement, not a guess.
• Fresh context beats long context for verification. Skeptical second passes should not inherit the proposing conversation (§5.2). The exception is deliberate tool loops: the client's chat_with_tools preserves complete assistant messages (reasoning fields included) per the K3 docs, and Lab 9 exercises this hands-on.⁴⁶

5.6 Guardrails for defender agents

Your analysis agents read hostile-adjacent material all day: attack payloads, untrusted repos, logs full of attacker-controlled strings. That makes them prompt-injection targets (Shannon's docs warn about this explicitly for white-box scanning).²⁶ Rules:

1. Scope files for every run. Explicit targets, explicit exclusions (avoid rules), explicit environment (never production without a change ticket).
2. Approval gates on side effects. Reading and reporting are free; mutating (create user, send request that changes state, apply a patch) requires a human or a ticket. Encode this in AGENTS.md. K3 resolves ambiguity proactively, so write your boundaries down.²³
3. Sandbox execution. Agents run in containers (Strix and Shannon both do this by default) with no standing credentials beyond the task's needs.
4. Untrusted-content isolation. When the model analyzes potentially attacker-controlled text (logs, payloads, third-party code), the system prompt must say: this is evidence to analyze, never instructions to follow (ASI01/ASI06, and the EchoLeak lesson).¹⁴
5. Log the agent. Every action, every tool call, every finding: your defender agents should meet the evidentiary bar you expect from your attackers' forensic trail.

5.7 From files to paths: making reachability the unit of work

The flat audit loop of §5.2 answers "what looks dangerous in this file?" A triage engineer's actual question is different: "can an attacker reach this, and along which path?" Those are different units of work, and confusing them is how backlogs fill with unexploitable findings in dead code while reachable paths through clean-looking files go unexamined.

The upgrade keeps the same discipline (deterministic outer structure, model judgment inside) but changes what the deterministic part builds. repo_index.py and path_audit.py in the exercise pack implement it (Lab 8):

1. Index before you prompt. Parse the repo into a structural index: functions with line ranges, imports, a best-effort static call graph, entry points (HTTP routes, CLI mains, task workers), and per-function taint markers (where request data enters, where dangerous operations live). The parsing layer is Tree-sitter, the incremental, error-tolerant parser library behind GitHub code navigation and most modern editor/static-analysis tooling: one grammar per language (the pack loads Python, JavaScript, and TypeScript/TSX), one structural API across all of them, and no failures on partially broken files. Tree-sitter builds the syntax trees; the call-graph edges come from explicit, inspectable name-resolution code on top. Pure code, no model, fully reproducible.
2. Build the risk queue from structure, not vibes. Entry points with reachable sinks first; then orphan sinks (dangerous functions no entry point reaches statically: dead code or a missed path, either way worth a look); then module-level code and unparsed files.
3. Bundle paths, not files. The analysis unit sent to the model is one entry point plus the functions reachable from it in the call graph: the same bounded, small-context discipline as §5.2, but a bundle now contains an attack path's worth of context instead of an arbitrary file slice. A sink three calls deep in a boring database module gets reviewed with the route that reaches it.
4. Audit the path. The model's job changes accordingly: not "find issues in this file" but "is this sink reachable from this entry point, along which hops, with what impact?", with the same strict-JSON discipline and a sharper question.
5. Hand confirmed paths to dynamic verification. Static reachability tops out at E2 on the evidence ladder (§4.8). Confirmed critical/high paths are written to dynamic-verification-targets.md: a work order for Strix/Shannon (Labs 1–2). Dynamic reproduction can raise a finding to E3. E4 requires a fix, regression test, failed original exploit, and independent closure check. The audit harness never fires exploits itself.
6. Cluster by root cause before ticketing. Nine findings with one missing input-validation helper are one fix. Deterministic clustering (same sink site, same defect class) turns a findings list into a fix list.
7. Re-verify on the path, not the repo. After a fix, re-run the affected bundle with a fresh-context verifier (--verify-model), add a regression test, and replay the original exploit path. Fix → regression → retest is the loop that keeps E4 honest (Lab 8, step 4).

Two consequences worth internalizing:

• Coverage becomes checkable. The index knows every entry point and every function; the bundle list records what was actually analyzed. "Which routes has no one looked at?" becomes a query, not a worry. path-bundles.json answers it per run.
• "No path found" is not "safe." Static call graphs miss dynamic dispatch, framework magic, and string-based routing, whatever the parser. An orphan sink means "no path found statically", which is why dynamic verification (step 5) is a stage of the pipeline, not an optional extra. Languages without a loaded Tree-sitter grammar are indexed by path and audited as whole-file bundles, so coverage degrades visibly rather than silently.

The sample app, seen both ways

On the one-file training app the difference is modest (everything is in app.py). On a real repo, where the route lives in routes.py and the sink sits three imports deep in db.py, the flat audit flags the sink without knowing the route reaches it, and cannot separate "exploitable tonight" from "theoretical." That sentence, reachable, from this route, along this path, is what prioritization actually is.

Part VI: Live POCs and Exercises

Ten labs, ordered to build on each other. Lab 0 is setup; Labs 1–2 are the packaged agentic pentesters; Labs 3–5 are the custom K3 loops from Part V; Lab 6 closes the fix loop; Lab 7 is the team tabletop; Lab 8 upgrades the audit from files to attack paths; Lab 9 is an optional capstone for readers building their own agents. Run them by track: the full set end-to-end is well over a day, so don't try:

The Foundation track is the workshop to give a whole portfolio company; AppSec is the engineering team's on-ramp; Frontier is for the people who will run the program: its extension work (model/harness bake-offs on your own repos, using the Semgrep methodology of measuring on your code before trusting anyone's benchmark)⁴⁴ is where the tool decisions get made with evidence instead of marketing. Everything targets local, disposable systems: the included training app, the included sample data, or OWASP Juice Shop.

The exercise pack lives alongside this document in exercises/ (scripts, sample targets, prompt library). Every script runs offline in --dry-run mode so you can validate the pipeline before spending a token.

Lab 0: Workbench (30 min)

Objective. A working K3-on-Fireworks environment and the practice targets running.

# 1. API access
export FIREWORKS_API_KEY="fw_..."        # fireworks.ai → Settings → API Keys
cd exercises && pip install -r requirements.txt
python3 scripts/k3_client.py --check     # expect: "Connection OK"

# 2. Practice target: OWASP Juice Shop (Docker)
#    NOTE the binding: 127.0.0.1 only. A deliberately vulnerable app on
#    0.0.0.0 is reachable by everything on your network; don't.
docker run --rm -d --name juice-shop -p 127.0.0.1:3000:3000 bkimminich/juice-shop
# browse http://localhost:3000 (the deliberately insecure training app)
# Even better: run it in a dedicated Docker network with no outbound access.

# 3. Optional alternates: DVWA (docker run --rm -d -p 127.0.0.1:8080:80 vulnerables/web-dvwa)
#    and the pack's sample-vuln-app (static target, no server needed)

Checkpoint. k3_client.py --check prints provider, model, latency, and a round-trip response. If the model ID 404s, check the Fireworks models page for the exact K3 listing and set K3_MODEL accordingly.²³, ²⁴

Discussion. Why do we verify connectivity before anything else? (Every downstream failure looks like a model failure until proven otherwise.) What does the --dry-run mode prove, and what can it never prove?

Lab 1: White-box agentic pentest with Shannon (2–3 h, mostly unattended)

Objective. Watch a white-box agent turn source access into proven exploits, and learn to scope it safely.

Steps.

1. Set up Shannon per Part IV §4.1 and point it at Juice Shop or your own app’s staging branch.²⁸
2. Write the engagement config before starting: test credentials, focus: paths, and avoid: rules (/logout, anything destructive). This rules-of-engagement habit transfers to every agentic tool.
3. Launch the run and record elapsed time, Fireworks token use, and cost. The workspace is resumable if interrupted.
4. Read the report against the PoCs: for each finding, replay the proof of concept yourself against the local target. This is the human verification habit. Agent output is evidence, not a verdict.

K3 calibration: Run Shannon against the known-vulnerable local target with the Fireworks configuration from §4.1. Record findings, PoC quality, run stability, token use, and cost. Replay each reported PoC before moving the setup to an application.

Discussion. Why does “no exploit, no report” beat a longer findings list? Which findings map to which OWASP categories (§2.11)? What guardrails limited the agent’s blast radius, and which would you tighten for a staging run with production-like data?

Lab 2: Black-box agentic pentest with Strix on K3 (2 h)

Objective. Run the external-attacker simulation (no source access, K3 as the brain) and wire it toward CI.

Steps.

# Install Strix from a pinned, reviewed release according to its current docs.
# Do not pipe remote shell scripts directly into bash.
export STRIX_LLM="openai/accounts/fireworks/models/kimi-k3"
export LLM_API_KEY="$FIREWORKS_API_KEY"
export LLM_API_BASE="https://api.fireworks.ai/inference/v1"

strix --target http://localhost:3000          # Juice Shop from Lab 0

Watch the agent graph work: recon maps the surface, exploitation agents fan out, findings stream in with PoCs and CVSS/OWASP labels. Then repeat headless, the way CI will see it:²⁹

strix -n --target http://localhost:3000 --scan-mode quick   # non-interactive
echo $?                                                     # non-zero = findings

CI extension (diff-scoped): on a PR, strix -n -t ./ --scan-mode quick --scope-mode diff --diff-base origin/main audits only what changed: the per-PR pentest gate from Part IV §4.2.²⁹

Discussion. How did black-box findings differ from Lab 1's white-box results, and what does that teach about coverage? (White-box sees unreachable-from-outside flaws; black-box sees what an attacker actually reaches. You need both.) What did K3’s reasoning look like inside the agent traces: tool discipline, recovery from dead ends? Record the run’s token use and cost, then identify which stage consumed the most.

Looking ahead. Lab 8 generates exactly the input a dynamic agent wants: dynamic-verification-targets.md, with statically confirmed attack paths, entry points, and safe validation steps. The static → dynamic hand-off you will run there is the production version of this lab's "point Strix at it and see."

Lab 3: Divide-and-conquer code audit with K3 (60–90 min)

Objective. Run the Part V loop yourself, on the planted training app, and measure why it beats a one-shot prompt.

cd exercises

# Offline pipeline check first (heuristics stand in for the model):
python3 scripts/code_audit.py sample-vuln-app --out ./out-dry --dry-run

# The real audit: map → plan → fan-out → fan-in → verify → report
# (--allow-cloud-upload is the data-policy gate: source leaves your machine
#  for the hosted endpoint; see the README and §2.3 before enabling.)
python3 scripts/code_audit.py sample-vuln-app --out ./out --verify --allow-cloud-upload

What a run leaves behind. Beyond audit-report.md and audit-findings.json, every run persists audit-plan.json (which files were selected and why), coverage.json (in-scope vs. audited vs. skipped: with reasons), and run-manifest.json (mode, provider/model, structured-output usage, budgets). Audits you can't evidence are just opinions; this is the difference.

What to observe.

1. The plan. audit-plan.json records which files K3 prioritized and why: for the sample app it should home in on app.py's endpoints (and it now sees CI configs, Dockerfiles, and .env-style files that naive walkers skip). On a real repo this is the step that saves 80% of spend.
2. The findings. The sample app plants SQLi, command injection, path traversal, SSTI, insecure deserialization, hardcoded secrets, and debug mode (each marked with a # VULN comment so you can grade the run). A good run finds all seven classes with evidence and attack paths.
3. The verification pass. --verify re-interrogates each high finding in a fresh context. Watch it downgrade or reject anything it cannot construct an attack path for: and compare against the dry-run heuristic output, which flags everything that pattern-matches (false positives included, by design). On the evidence ladder (§4.8): heuristics produce E1, the fan-out produces E2 candidates, and this pass confirms or rejects the E2 reasoning (it is still static analysis, not runtime proof). That distinction is the whole point of Lab 6.
4. The contrast experiment. Now paste the entire app.py into one K3 chat with "find all vulnerabilities." Compare: coverage, evidence quality, attack-path specificity. On small files the gap is modest; repeat on a 50-file repo and the one-shot approach misses whole files while the loop's coverage is enumerable (this is the §5.1 lesson, felt instead of told).

Discussion. Where would you slot this loop in CI (nightly full-repo vs. per-PR diff)? What is the right granularity for your largest repo (§5.2)? What did the run cost, and how much of the prompt was cache-stable?

Looking ahead. This lab's unit of analysis is the file. Lab 8 upgrades it to the attack path: same loop discipline, but the model answers "reachable, from this route, along these hops" instead of "this file contains a sink." If you only have time for one more lab after this, make it Lab 8.

Lab 4: Network exposure triage: nmap + K3 (45 min)

Objective. The "AI as analyst on classic primitives" pattern (§4.6): deterministic scan, deterministic rules, model judgment.

# On the included synthetic scan:
python3 scripts/nmap_triage.py sample-data/nmap-sample.xml --out ./out --dry-run
python3 scripts/nmap_triage.py sample-data/nmap-sample.xml --out ./out --allow-cloud-upload   # K3 pass

# Then on infrastructure you own:
nmap -sV -oX my-scan.xml <your-host-or-subnet>
python3 scripts/nmap_triage.py my-scan.xml --out ./out --allow-cloud-upload

The sample scan plants a greatest-hits of small-company exposure: MongoDB and Redis on open ports, Telnet, vsftpd 2.3.4, Samba 3.x, RDP, and an exposed Docker socket. The rules pass flags them deterministically; the K3 pass is where the value shows: correlation ("Redis and MongoDB exposed on the same host suggests a missing bind-address default"), version-specific risk, and a prioritized, safe next-step list a non-specialist can execute.

Discussion. Why do we forbid the model from running nmap itself (§4.6)? Which findings were rule-caught vs. model-correlated? What would you add to RULES for your own stack?

Looking ahead. Note the shape: deterministic primitive, deterministic rules, model judgment: the same "code computes, model assesses" pattern Lab 8 applies to source code, where the deterministic part is a call graph instead of a port scan.

Lab 5: Auth-log anomaly triage with K3 (45 min)

Objective. Same pattern on detection: code computes, the model assesses.

python3 scripts/log_triage.py sample-data/auth-sample.log --out ./out --dry-run
python3 scripts/log_triage.py sample-data/auth-sample.log --out ./out --allow-cloud-upload    # K3 pass

The sample log plants a classic mini-incident: a 47-failure brute-force burst against admin/root from one IP, a successful login from that same IP 26 minutes later (at 00:07), an impossible-travel pair for sandra (office network, then a distant network 18 minutes later), and off-hours access (all buried in ordinary noise). The deterministic pass counts what matters; K3's assessment turns it into severity, timeline, 30-minute containment, follow-up checks, and prevention: the exact shape of a first-responder ticket.

Extension. Pipe a week of real auth logs from a host you own through the same script. Then wire it to cron and you've built a $0.05-a-night anomaly detection layer (§2.6) with an analyst attached.

Discussion. Where does the model add the most value: detection, or communication? (Notice: the statistics caught everything; the model made it actionable.) How would you raise/lower the burst thresholds for your environment's baseline?

Lab 6: Find → fix → verify: closing the loop (60 min)

Objective. The full remediation cycle on one real finding: the muscle this whole playbook exists to build.

1. Pick a Lab 3 finding: the /user SQLi is ideal.
2. Fix it with K3 as pair-programmer. In OpenCode (or any harness pointed at Fireworks), scope the task tightly: "In sample-vuln-app/app.py, the /user endpoint builds SQL with string formatting. Convert to parameterized queries. Change nothing else. List what you changed and why it's safe." Review the diff like any junior's PR.
3. Verify with the same instrument that found it. Re-run code_audit.py: the finding must be gone from the report. Then (because no retest, no fix) fire the original attack path at the running app and confirm it now fails: curl 'http://127.0.0.1:5050/user?username=%27%20OR%201%3D1--' should no longer dump the table. Add a regression test that asserts the endpoint returns only the requested user and cannot be widened by SQL syntax, then run it in CI. This is the full climb to E4 on the evidence ladder (§4.8): fix applied, original test failing safely, regression test in place, independent confirmation. E4 is the only level at which a finding is allowed to leave the backlog.
4. Optionally, automate the whole arc: run OpenHands Vulnerability Fixer on a repository with known dependency CVEs. Trivy finds the issues, agents patch them, pull requests open, and you review.³³

Discussion. What belongs in the fix prompt's scope limits? (Unscoped "improve security" edits wander; scoped edits land.) Who approves agent-written security patches in your org, and what's the evidence bar?

Lab 7: Incident-response tabletop with a K3 copilot (90 min, team)

Objective. Rehearse a machine-speed incident at human speed, with K3 as the copilot it would be in a real event.

Setup. One facilitator, 2–6 responders, one shared K3 session initialized with prompts/ir-copilot.md. The facilitator holds the injects; the room gets evidence in stages.

Scenario (60 minutes of injects).

Variant: run the real July 2026 chain. For an advanced group, replay the actual Hugging Face incident anatomy: initial access is a zero-day in a third-party package-registry caching proxy; the operator is fully autonomous (no human typing); its objective is data in a production database; and it pivots through stolen credentials toward internet-facing nodes.⁴² The room's job goes past containment. The real work is deciding, in advance, the controls that would have prevented it: egress denial from build/research environments, per-environment credential scoping, behavioral monitoring on agent workloads, and named kill-switch authority. That is the OWASP Agentic Top 10 (§2.10) as an agenda, not a poster.

Why K3 shines here: the whole incident (every inject, every log excerpt, every decision) fits in one 1M-token session, so the copilot never needs re-briefing; and per the IR-copilot prompt, it keeps a running timeline and will not let the room skip "evidence to preserve."²³ Also note what does not happen: no guardrail refuses to discuss the attack artifacts, the exact wall Hugging Face's responders hit with a gated model.²

Debrief. Where did the room hesitate? (That's the gap in your runbook: fix it in Part VII §7.2 this week.) What did the copilot get wrong, and how did the room catch it? (If nobody can answer the second half, that's the finding.)

Lab 8: Path-centric audit: reachability, clustering, and the fix loop (75 min)

Objective. Upgrade from "findings in files" to "verified attack paths": the workflow difference between a scanner and an auditor (§5.7). Prerequisite: Lab 3.

cd exercises
# 1. See what the analyzer sees (no model, fully offline):
python3 scripts/repo_index.py sample-vuln-app --out ./path-out/repo-index.json

# 2. Offline pipeline check (heuristic path candidates, NOT an audit):
python3 scripts/path_audit.py sample-vuln-app --out ./path-dry --dry-run

# 3. The real path audit, with fresh-context verification:
#    (same data-policy gate as Lab 3: source leaves your machine)
python3 scripts/path_audit.py sample-vuln-app --out ./path-out --verify-model --allow-cloud-upload

Second target, second language. The pack ships a twin training app, sample-vuln-js/ (Express, five planted classes). Run the same three commands against it: the Tree-sitter JavaScript extractor picks up the Express routes (inline and named handlers alike), and the /users finding traces the injection across files, from the route in server.js into query construction in db.js. If your own estate is polyglot, this is the cheapest way to see what the index does with it.

What to observe.

1. Read the index first. repo-index.json is what the analyzer can see, parsed by Tree-sitter (Python, JavaScript, TypeScript/TSX grammars; the manifest names the parser used): 8 entry points (7 routes + the __main__ startup), 9 functions, the call-graph edge from user_lookup to get_db. On your own repo this is the moment you discover entry points you forgot existed: the index is inventory, and inventory is a control. If your stack mixes Python services with JS/TS frontends, point it at the whole repo and check that both halves show up.
2. Compare bundle thinking to file thinking. path-bundles.json lists each analysis unit: an entry point plus its reachable callees, sinks noted, size-bounded, with entry-point coverage stated (8/8 here). On a real repo, anything under 100% is a named, auditable gap instead of an unknown.
3. Grade the findings against Lab 3's. Same seven planted classes, but each finding now carries an entry point, an explicit hop-by-hop path, a root-cause cluster, and (with --verify-model) a verdict from a fresh-context skeptic. Check /user: the path should read user_lookup → get_db with the sink at line 51.
4. Close the loop: the point of the lab. Fix the planted SQLi as in Lab 6 (parameterized query), then re-run with --verify-model. The finding must close, and the second run's manifest is your regression evidence. This is §5.7 step 7 as muscle memory: the same instrument that found it confirms it is gone.
5. Hand off what's left. Open dynamic-verification-targets.md: confirmed high-severity paths formatted as a work order for a dynamic agent. Feed one to your Lab 1/2 setup (Strix or Shannon against the local app) and watch a static E2 finding become a dynamic E3 finding (§4.8). After remediation, the regression test, failed original exploit, and independent closure check establish E4.
6. Compare against the expected-output notes. The training apps contain known planted classes so you can tell whether your harness is producing the right shape of result. Treat the examples as calibration material, not as evidence about your own repo.

Discussion. Where does static reachability err in both directions: paths it cannot see (framework magic, dynamic dispatch) and paths it sees that are not real (dead branches)? How would you wire --verify-model into CI: every PR, or nightly on changed-path bundles? What is the entry-point coverage number on your largest service, and who owns closing it?

Lab 9: Under the hood: a native tool-calling loop (45 min, optional capstone)

Objective. Experience the third orchestration pattern from §5.4 (the model drives, tools answer) by running and reading a minimal one.

Who this is for: read before running. This lab is a reference implementation for readers building their own agentic tooling on K3, or evaluating whether to. Self-select honestly:

• Relevant if you want native function-calling done correctly for K3: tool registration, a bounded loop, complete-assistant-message preservation (the multi-turn contract from §3.3), sandboxed read-only tools.
• Interesting if you want to feel the determinism and cost trade-off between "code drives" (Labs 3, 8) and "model drives."
• Skippable if you want security answers on your repo. It asks one question, non-deterministically, and finds nothing Lab 8 doesn't. Skipping it costs you nothing operational.

python3 scripts/path_agent.py sample-vuln-app \
  --question "Is the SQL injection in app.py reachable from an unauthenticated route?" \
  --allow-cloud-upload

What to observe.

1. Watch the loop, not the answer. agent-transcript.json records every round: which tool K3 chose, with what arguments, what came back. A typical run reads the route, searches for auth decorators, then verdicts. The tools are read-only and path-sandboxed to the target directory; the loop is bounded (default 8 rounds); the upload gate is the same as every lab's.
2. Ask the same question twice. Different investigation order, different round count, different cost (usually the same verdict on this tiny app). Now imagine it in CI. That is "least agency" (§2.10) felt instead of told, and why this playbook's production loops are deterministic-outer.
3. Read the skeleton. path_agent.py is ~150 lines of plumbing over chat_with_tools (§5.5): tool specs as JSON Schema, a handler map, and the preserved-message loop. If you are building your own agent, this file plus k3_client.py is the correct minimal starting point. Swap the four read-only tools for your own, keeping the sandbox and the round bound.
4. Contrast with the deterministic path. Run Lab 8 on the same territory and compare: rounds vs. bundles, variable cost vs. manifest cost, exploration vs. coverage guarantee. §5.4's comparison table, now with receipts.

Discussion. Where would a tool-calling loop beat the deterministic harness in your environment? (Open-ended incident investigation is the classic answer: you cannot enumerate a call graph for a question you haven't formed yet.) What tools would you add, and what is the approval gate before any of them can write, execute, or send traffic (§5.6)?

Part VII: Operational Playbooks

Checklists and runbooks that turn Parts I–VI into operations. Copy them into your wiki; assign owners; revisit quarterly.

7.1 Pre-production hardening checklist

The vibe-code-to-production gate. Every item maps to a Part II control; nothing ships until every box has an owner.

Identity & access

• [ ] App sits behind authentication tied to company identity (SSO/org tier). §2.1
• [ ] No shared logins; service accounts are scoped and inventoried. §2.1, §2.10
• [ ] Admin surfaces are not internet-exposed (VPN/allowlist). §2.1

Secrets & config

• [ ] gitleaks/equivalent clean on the repo; any past hit rotated. §2.2
• [ ] No secrets in env files committed anywhere; manager injection in place. §2.2
• [ ] Debug mode off, TLS verification on, security headers set. §2.5

Code & supply chain

• [ ] Dependencies pinned; lockfile-diff review habit; Trivy/npm audit in CI. §2.4
• [ ] Agentic pentest run (Shannon white-box and/or Strix black-box) with findings triaged. Part IV
• [ ] Divide-and-conquer audit (Lab 3 loop) on the security-critical modules. Part V
• [ ] AGENTS.md security section in place if coding agents work this repo. §2.5

Data & logging

• [ ] Data classification written; PII/PHI/PCI flows documented. §2.3
• [ ] Auth + app logs centralized; brute-force and impossible-travel alerts live. §2.6
• [ ] Backups configured and one restore tested. §2.7

Response readiness

• [ ] Breach runbook location known to on-call. §2.8, §7.2
• [ ] Offboarding checklist exists and was used for the last departure. §2.9

7.2 Incident-response lite: the first 60 minutes

For the responder who has never done this before. Print it.

0. You suspect something. Now what? You are the incident coordinator until relieved. Start a written timeline now: every fact with a timestamp. (Your K3 copilot session from Lab 7 keeps one too; yours is the record of record.)

First 10 minutes. Confirm and contain (reversibly).

1. Verify the signal is real (one log check, one human check). False alarms are fine.
2. Contain with reversible actions first: disable the suspect account, block the suspect IP, and isolate the host from the network. Preserve volatile evidence when feasible; if active destructive behavior cannot be contained any other way, prioritize stopping damage and document what you did.
3. Preserve: copy the relevant logs somewhere the attacker can't reach. Assume the attacker can see the compromised box. Coordinate from a clean device/channel.

Minutes 10–30. Scope and call for help.

1. Scope the blast radius: what did the account/host touch? Which credentials lived there?
2. Escalate per your runbook: leadership, your security contact, external IR counsel if it looks big (active ransomware, confirmed data theft, regulated data). Saying "this is beyond us" early is competence, not failure.
3. Rotate credentials in dependency order: identity provider first, then cloud, then the affected services, then user-facing.

Minutes 30–60. Stabilize and communicate.

1. Short, honest internal note: what we know, what we're doing, what's next, when the next update comes.
2. External comms only from the named spokesperson; drafts reviewed by leadership (and counsel if data was taken).
3. Open the recovery track: rebuild-from-known-good beats clean-in-place for anything the attacker touched (Hugging Face rebuilt compromised nodes from scratch).¹

Never in the first hour: wipe evidence, negotiate with attackers, announce attribution you don't have, or log into compromised systems with privileged credentials.

7.3 BCP/DR lite

Business continuity for teams without a continuity team.

• Know your two numbers. RTO (how long can you be down?) and RPO (how much data can you lose?). Write them per system; they drive every other decision. A honest "4 hours / 15 minutes" beats an aspirational zero you never engineered for.
• 3-2-1 backups, restore-tested quarterly (§2.7). An untested backup is a rumor.
• Degrade deliberately. Decide now what "minimum viable service" looks like during an incident (read-only mode? status page + manual intake?) and write the three steps to get there.
• Vendor failure is in scope. Your model provider, your hosting provider, your IdP: each gets a "what if it's down or cut off tomorrow?" answer (Part III §3.2 is the AI-specific version; the June 2026 Fable/Mythos suspension is why it's not hypothetical).⁹
• Tabletop twice a year. Lab 7's scenario, your org's cast. Ninety minutes, twice a year, is the cheapest insurance you will ever buy.

7.4 Supply-chain and AI-agent hygiene (recurring)

• Weekly: review dependency-update PRs (not blind-merge); check gitleaks CI is green; glance at agent-action logs for anything outside scope.
• Monthly: eliminate long-lived credentials where possible, review the remaining ones, and rotate on exposure, role change, or defined risk-based schedules; review new tools/plugins added to CI and agents (the LiteLLM lesson: security tools are supply chain too);¹⁸, ¹⁹ run the Strix diff-scope report across merged PRs.
• Quarterly: full agentic pentest (Lab 1/2) on the flagship app; review agent permissions against least-agency (ASI02/ASI03);¹⁴ re-run the Lab 3 audit on new modules; test one restore.

7.5 Metrics that matter

Track five, ignore the vanity fifty:

Part VIII: The 30/60/90-Day Rollout

For a portfolio company (or any small team) starting from "we ship fast and hope."

Days 1–30. Stop the bleeding. Controls §2.1–§2.3 (SSO org tiers, secret scan + rotation, data classification page), Lab 0 workbench, Lab 5 log triage wired to cron, breach runbook drafted and pinned. Exit criteria: no app without auth, no unrotated known-leaked secret, logs queryable, one page that says who to call.

Days 31–60. See yourself. Controls §2.4–§2.7, first Shannon white-box run (Lab 1) and Strix external run (Lab 2) on the flagship app, Lab 3 audit loop in nightly CI, findings backlog triaged with owners. Exit criteria: a living findings backlog with severity, owner, and verification per item; nightly automated audits; one tested restore.

Days 61–90. Close the loop and rehearse. Controls §2.8–§2.10, fix-loop automation (Lab 6 pattern, optionally OpenHands Vulnerability Fixer), per-PR diff-scoped agentic testing in CI, Lab 7 tabletop with the full team, metrics dashboard (§7.5). Exit criteria: findings close with retest evidence; CI blocks on new high-severity findings; the team has run one full IR rehearsal and patched the runbook gaps it exposed.

Then it becomes cadence (§7.4): weekly, monthly, quarterly. Security is a rhythm, not a project.

Appendices

A. Tool matrix

B. Resource directory

Frameworks and references: OWASP Web Top 10 (2025),¹¹ OWASP LLM Top 10,¹² OWASP Agentic Top 10 (2026),¹³ MITRE ATT&CK,²⁰ NIST SP 800-207 Zero Trust,²¹ Lockheed Martin Cyber Kill Chain,²² and CVE.org.

Learning targets for authorized practice: OWASP Juice Shop, DVWA, and Metasploitable are local, disposable targets.

Threat intelligence: Check Point Live Cyber Threat Map, Talos Intelligence, and the CISA Known Exploited Vulnerabilities Catalog. Prioritize vulnerabilities with documented exploitation in the wild. Palo Alto Networks’ Defender Guide to the Frontier AI Impact on Cybersecurity surveys current attacker and defender capabilities.⁶

Incident anatomy: the Hugging Face July 2026 disclosure and coverage,¹, ², ³ the React2Shell analyses,¹⁵, ¹⁶, ¹⁷ and the LiteLLM supply-chain postmortem.¹⁸, ¹⁹

Companion document: the Spanish-language workshop deck Estación 10: Ciberseguridad: de POC funcional a pentesting asistido covers the POC-to-production risk shift, OWASP taxonomies, and a Shannon demonstration for team training.

C. Prompt library

Full text in exercises/prompts/:

• code-audit-system.md. Per-file audit system prompt (JSON schema). Lab 3
• verification-pass.md. "prove it or drop it" finding validation. Labs 3, 6
• ir-copilot.md. Incident-response copilot with timeline discipline. Lab 7
• AGENTS-security-template.md. Security invariants + approval boundaries for coding agents. §2.5, §5.6

D. Glossary (for the non-infosec reader)

• Attack surface: everything reachable that could be abused: endpoints, ports, dependencies, people.
• Black-box / white-box: testing without / with source-code access.
• PoC (proof of concept): a demonstration that a vulnerability is real and reproducible: the currency of serious findings.
• Fan-out / fan-in: splitting work into many small model calls, then merging results deterministically.
• Guardrails: limits on what a model or agent will do. Vendor-side (refusals) or yours (scope, approvals).
• Least privilege / least agency: grant exactly the permissions the task needs: to users (privilege) and to AI agents (agency).
• RTO / RPO: how long you can be down / how much data you can lose.
• SIEM: central log and alert platform (Wazuh is the open-source starting point).
• SSRF / SSTI / IDOR: server-side request forgery / server-side template injection / insecure direct object reference. Three web vuln classes the labs make concrete.
• Zero trust: never trust by network location; verify identity and authorize every request.

E. References

The AI Cybersecurity Playbook: v1.5, July 2026. Prepared by Trilogy AI Center of Excellence as a public practical resource for defensive cybersecurity with open-weight models. Public repository: trilogy-group/ai-cybersecurity-playbook. Exercise pack: repository source.