BB Arena Labs
Black-box security labs for training and evaluating AI models
Most security training data hands the model the source code, or is public and may already be memorised. Ours is black-box and private: the model gets a target, a scope and a shell, like a real attacker, and every step is confirmed by the lab's own verifier.
Built and checked the same way as the BB Arena benchmark, where the best AI model solves 62 of 100. There's plenty left to learn.
Request the evaluation pack →10 labs under NDA, each in freshly seeded copies, with reference solutions.
1,000 labsEach one through independent audit
46 vulnerability classesIn 10 groups, 110+ CWEs
7 languages21 frameworks
4 rungs per labEach confirmed by the lab's own verifier
Never the leaderboard's labs. The labs behind the BB Arena leaderboard are never sold, licensed or shared. The labs we license are a separate set, built the same way. That keeps leaderboard scores honest, and it makes our labs safe to train on: you can't train on the test.
How a lab works
Prefer to read? The nine steps
- 01A black-box targetYour model gets a target, a scope and a shell. No source code, like a real attacker.
- 02A real flaw in an ordinary productBuilt from our own security research: a fully synthetic, isolated product that recreates the attack path behind a real-world vulnerability.
- 03Proof, not claimsFour rungs, each counted only when the lab's own verifier confirms it. A claim alone earns nothing.
- 04Nothing to memoriseEvery lab ships in many seeded copies, with fresh proof values each run, so nothing can be memorised.
- 05A secure twinThe same product with the flaw fixed. Claiming the flaw there scores zero, so bluffing shows up.
- 06A reward you can trustPartial progress earns partial reward: reward per rung for training, full solves for evaluation. Infrastructure failures are marked invalid, not zero.
- 07Measured difficultyEvery lab is measured against a top frontier model and ships at its measured tier: medium or hard.
- 08Plenty of headroomThe best model on our leaderboard solves 62 of 100. The catalogue has 1,000 labs in 46 vulnerability classes, never the leaderboard's labs.
- 09A scorecard on your own machinesRun the evaluation pack, 10 labs under NDA, on your own machines and get a scorecard: results per lab, by tier and class.
Why the reward can be trusted
A lab only ships after it passes all of these.
- Independent auditsSeparate reviewers audit every lab before release. Every lab family needs 2 to 4 audit rounds before it passes.
- Reward-hacking reviewAn adversarial review looks for shortcuts that earn reward without doing the exploit.
- Random and idle agents earn nothingAn agent that does nothing, or acts at random, scores zero.
- No free stepsAn ordinary user going about ordinary tasks earns no rung.
- Nothing to memoriseEvery lab ships as many seeded copies, each with its own secrets, IDs and data, so a score can't come from a memorised answer. Contamination canaries show if lab content turns up where it shouldn't, such as in training data.
- Measured difficultyEvery lab is measured against a top frontier model and ships at its measured tier: medium or hard. Not a guess.
Coverage
The full catalogue: 1,000 labs across 46 vulnerability classes, 7 languages and 21 frameworks. It uses a finer taxonomy than the benchmark's 12 classes.
- Access control and authorisation156Object-level authorisation (IDOR, BOLA) · Function-level authorisation and privilege escalation · Property-level authorisation and mass assignment · Parser and normalisation differentials
- Authentication and identity184Authentication bypass · MFA and step-up bypass · Session management · OAuth, OIDC, SAML and SSO · JWT and token validation · Account takeover flows · Brute force and rate limiting
- Injection165SQL injection · ORM and filter injection · NoSQL injection · OS command injection · Argument injection · Code injection and sandbox escape · Server-side template injection · XXE · Insecure deserialisation · LDAP, XPath, CRLF and header injection · Server-side prototype pollution
- Server-side requests and files100SSRF · Path traversal and file read · File upload and file write · Unsafe consumption of external data
- Client-side90Cross-site scripting · CSRF · CORS misconfiguration · Open redirect and cross-window flaws · WebSocket flaws
- Protocol, caching and infrastructure57Web cache poisoning and deception · Host header attacks · Misconfiguration and exposed services · Cloud and container control planes
- Business logic and concurrency82Business logic and workflow abuse · Race conditions
- Exposure and cryptography81Secret and key exposure · Sensitive information disclosure · Weak cryptography and predictable tokens
- Software supply chain20Dependency confusion and build artefacts
- AI, LLM and agent applications65Prompt injection and excessive agency · Improper output handling · Hidden context and RAG leakage · MCP and agent tool-server flaws · Agent sandbox and allowlist bypass
Languages and frameworks
- Node.js
- 250
- Express, Next.js, NestJS, Fastify
- Python
- 200
- FastAPI, Django, Flask
- PHP
- 150
- Laravel, Symfony, plain PHP
- Java
- 150
- Spring Boot, Quarkus
- Go
- 100
- net/http, Gin, Echo, a real open-source app
- .NET
- 100
- ASP.NET Core minimal API and MVC
- Ruby
- 50
- Rails, Sinatra
Measured difficulty
40% Medium60% Hard
Every lab is measured against a top frontier model and ships at the tier it measured.
Surfaces
REST and server-rendered apps, GraphQL, WebSocket, gRPC and MCP tool servers.
Data stores
PostgreSQL, MySQL and MariaDB, SQLite, a MongoDB-compatible document store, Valkey caches, S3-style object storage and search.
See them on your own stack10 labs under NDA, with reference solutions, datasheets and the grader.
Request the evaluation pack →What you get
- Self-hosted with Docker Compose, on your own machines.
- Packaged for common agent evaluation and RL training stacks: Harbor, Inspect and verl-style setups.
- A grader that reports reward per rung, and marks infrastructure failures as invalid instead of scoring them zero.
- A local scorecard after every run: results per lab, by tier and class.
- Many seeded copies of every lab, each with its own secrets, IDs and data: more variants to train on, and fresh proof values on every run.
- A datasheet and a reference solution for every lab.
- Fix tasks for the same labs, with their own grader.
- A 25-minute default time limit per attempt. Change it to suit your setup.
Bonus: fix tasks
The same labs also come the other way round. The agent gets the vulnerable code and a ticket, and has to repair the flaw without breaking the app.
- Three ticket levels: a reproduction to follow, a weakness type and area, or a blank-slate audit.
- Graded by behaviour: every exploit variant blocked, normal use still working, the feature still there, nothing leaking.
- Hard to game: a small set of visible smoke tests, with the full test suite kept hidden. Deleting the feature or refusing every request scores zero.
- Attack and defend on the same products, so one catalogue trains both skills.
Get started
- For buyersEvaluation pack10 labs under NDA, with reference solutions, datasheets and the grader. Run them on your own stack before you decide.Request the evaluation pack →
- Licensing and pricingTalk to usTell us what you're building and how many labs you need. We don't publish prices; we'll quote.Talk to us about labs →
Questions
- Can we train on these labs?
- Yes, that's what they're for. They're a separate set from the leaderboard's labs, so training on them can't contaminate a BB Arena result.
- Does licensing labs help a model on the leaderboard?
- Only through real skill. The leaderboard's labs are never sold, and every model runs under the same protocol and scoring, whoever licenses labs. See the methodology.
- Do you share how the labs are built?
- No. You get the labs, the grader, datasheets and reference solutions. How we build and check them stays with us.
- Can we run them offline?
- Yes. Labs are self-hosted with Docker Compose. We deliver images from our private registry, or as encrypted archives for offline sites.
- Which tools do they work with?
- They come packaged for Harbor, Inspect and verl-style RL training, with a catalogue manifest. Any model and harness that can reach a web address and use a shell can run them.
- Are labs licensed exclusively?
- No. Every licensed lab is available to every buyer.
- How do you stop labs leaking?
- Every licensed buyer's instances carry their own watermark, and every lab has contamination canaries, so lab content that turns up where it shouldn't can be traced.
- How much does it cost?
- It depends on what you need. Talk to us.
Training AI models for security?Start with the evaluation pack and run it on your own stack.
Request the evaluation pack →