Methodology
How a number earns your trust
It is easy to produce an evaluation number once, with a pipeline nobody probed, on infrastructure the agent could see into. RL Gym treats every one of those shortcuts as a defect. This page explains the four pillars in plain language.
Statistical verdicts, not single runs
Agents are not deterministic. Run the same agent on the same suite twice and the scores differ. A single-run delta of 2 to 3 points can be pure evaluation noise, which means a team can celebrate an improvement, or roll back a release, over nothing.
RL Gym never issues a verdict from one run. Every evaluation is N independent runs, and the verdict reports the variance across them alongside a noise band derived from power analysis. If the difference between two versions sits inside that band, the verdict says so, instead of pretending the difference is real.
Suites must survive adversarial probes
A benchmark can be broken in ways that hand out high scores to agents that solved nothing. Before any suite’s numbers count on RL Gym, the suite itself is put on trial against four adversaries:
- A null agent that does nothing. If it scores, the suite leaks credit.
- A random agent that acts arbitrarily. Its score calibrates the floor.
- Prompt injection, probing whether task text can manipulate the grader.
- State tampering, probing whether the environment’s state can be edited into a win.
A suite that fails a probe does not produce countable numbers until it is fixed. This is the gate that keeps a broken benchmark from quietly inflating every score built on it.
VM-level isolation between agent and evaluator
Agent code executes in VM-isolated sandboxes of the Firecracker class: a separate microVM with a separate kernel, not a shared container. The sandbox is fully isolated from the evaluator, and reference answers are never readable by the agent.
This closes the family of exploits where an agent scores by inspecting the machinery that grades it, reading the answer key off disk, or interfering with the evaluation process itself. Contamination resistance is an architectural property here, not a policy request.
Verdicts attested to what really ran
A score is only meaningful if you know exactly what produced it. Every RL Gym verdict is attested to the exact suite, backend, and image that really ran. Two verdicts are comparable when their attestations match, and the platform can prove it.
This is what makes regression tracking honest: when v1.4.2 is judged against v1.4.1, both numbers are bound to the same trial conditions, so the delta measures the agent and not a drifting pipeline.