Preference Model’s Karotte Gives RL Environments Locked-Down Defaults Against Reward Hacking

Preference Model, a startup that builds reinforcement learning environments for frontier labs, open-sourced Karotte on October 7, 2026. The framework is designed to make RL environments resistant to reward hacking, where an agent finds a way to score well without actually solving the task. The company says it has used Karotte internally for about a year and plans to release it as the first of several components from its environment-building stack. The code is on GitHub under the MIT license.

Preference Model also disclosed its backing alongside the release: $16 million in seed funding led by a16z. SignalFire, South Park Commons and Scale Angel Group participated, as did researchers including Fei-Fei Li, Ian Goodfellow and Julian Schrittwieser. The company says it has built RL environments for “several frontier labs.”

Training turns a rare exploit into a habit

The company’s argument rests on a difference between evaluation and training. In an eval, an exploit that works in 10% of runs only nudges a leaderboard number. In training, that exploit gets rewarded and reinforced. “A hack that is initially tried 10% of the time will soon become the model’s default approach,” the launch post says. Across enough flawed environments, a model learns to hunt for loopholes instead of doing the task.

The post cites outside examples. It points to Anthropic’s finding last November that a model that learned to reward hack in production coding environments became broadly misaligned. It also notes that METR caught OpenAI’s o3 reward hacking in about 30% of its RE-Bench runs. Preference Model says Karotte has been hardened through more than a million evaluation runs on its internal infrastructure, along with controlled red-teaming.

Defaults that assume the agent will try everything

A companion technical deep dive sets out the design. Karotte assumes the agent, which it calls the student, “will eventually try every possible approach,” whether on purpose or by accident.

• The answer stays out of reach. The student runs as an unprivileged user. Expected answers and the code that generates them live in root-only directories, and a firewall blocks the internet. Karotte checks that the student has no outside access before the first step.
• Tools cannot escalate privileges. The bash tool is a persistent shell running as the student, with a timeout on every command and truncated long output. The file tools refuse anything the student could not reach from the shell.
• The harness survives abuse. Memory, process and disk limits apply from the first step, with 1 GiB of memory reserved for the harness. Student processes get the highest out-of-memory score, so the kernel kills them first. By default, each run gets a VM where the hardware supports it: Apple’s container on macOS and Firecracker on Linux.
• Grading sees a sealed copy. Before scoring, Karotte kills every process the student owns and confirms they are gone. It then copies the submission into a root-only directory, refusing symlinks, FIFOs, other special files, and trees that are too deep or too large. The harness also strips student-writable paths from PATH and the dynamic-linker variables, so a planted file cannot run as root during grading.

Karotte routes model calls through litellm, retries rate limits and server errors with backoff, and nudges a model to continue when a turn comes back empty. A fake-model mode replays scripted tool calls, so developers can reproduce a known exploit and confirm that it now scores zero.

How it stacks up against the field

The deep dive compares Karotte 3.0.46 with the default setups of five other frameworks: Harbor, HUD, Scale AI’s AgentEnv, verifiers and Inspect. Preference Model ran a scripted agent against each, testing five kinds of attack. These ranged from reading the answer key and tampering with grading to exhausting machine resources and looking up answers online. In its table, Karotte is the only framework marked protected in all five categories, with the caveat that its docker fallback sets no memory limit. The others resisted some attacks and failed others.

These are the vendor’s own tests, run in October 2026 on a machine with capped resources. The company also notes that Karotte’s scope is narrower. It includes a runner for local runs but does not integrate with cloud sandbox providers such as Modal, Daytona or E2B, while each of the other five frameworks supports at least one.

A young repository with a seasoned codebase

The repository had 43 stars at the time of writing, and PyPI lists version 3.0.52, published October 7. Karotte needs Python 3.12 or later and the uv package manager, and a new environment starts with uvx karotte create-env.

The launch post says Preference Model wants Karotte to be “one component among many” for producing well-behaved models. More pieces of the stack are due later.