Harbor

Run Harbor evaluations and RL rollouts on Tensorlake Sandboxes: fresh isolation per trial, pre-warmed snapshots for expensive environments, and independent test verification.

Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. With Harbor you can evaluate arbitrary agents (Claude Code, OpenHands, Codex CLI, and others) against curated datasets like Terminal-Bench, SWE-Bench, and Aider Polyglot, build and share your own benchmarks, run thousands of trials in parallel across cloud providers, and generate rollouts for RL optimization.

Harbor abstracts the execution backend behind an --env flag. Tensorlake plugs in as one of those providers (alongside other sandboxes and local Docker), so the same Harbor commands run on Tensorlake sandboxes without changing your tasks, agents, or evaluators.

This guide focuses on running CLI-agent evaluations against benchmarks like Terminal-Bench. Harbor also supports generating rollouts for RL optimization. We’ll cover those workflows in follow-up guides.

New to Tensorlake? Sign up at the dashboard. New accounts include free credits, enough to run a full Terminal-Bench sweep before you pay for anything.

Quick start

1. Get a Tensorlake API key

   Grab one from the Tensorlake Dashboard. You’ll also need an API key for whichever agent provider you want to evaluate (e.g., Anthropic).

2. Install Harbor with the Tensorlake provider

   The harbor[tensorlake] extra installs the TensorLakeEnvironment provider alongside Harbor.

   uv

   uv pip install "harbor[tensorlake]"
   

   pip

   pip install "harbor[tensorlake]"
   
3. Set your environment variables

   export TENSORLAKE_API_KEY="tl_..."
   export ANTHROPIC_API_KEY="sk-ant-..."   # or another agent provider
   
4. Run a Terminal-Bench task

   Run a single Terminal-Bench task on Tensorlake with Claude Code as the agent:

   harbor run --env tensorlake \
     --include-task-name terminal-bench/pytorch-model-cli \
     --dataset terminal-bench/terminal-bench-2-1 \
     --agent claude-code \
     --model anthropic/claude-sonnet-4-6 \
     --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
   

   Drop --include-task-name to run the full Terminal-Bench 2.1 suite. --ae KEY=VALUE forwards an environment variable from your shell into the sandbox where the agent runs. Add more --ae flags for any other secrets the agent needs.

Why Tensorlake for Harbor

Harbor’s value comes from running large fleets of environments in parallel and trusting the results. Tensorlake’s runtime is designed for exactly that workload:

• Per-trial sandboxes: each task starts on a clean machine and is destroyed at the end. No shared kernel state between trials, which matters for both eval reproducibility and RL reward integrity.
• Full task-environment support: Tensorlake imports a task’s real Docker image and converts it into a sandbox image that boots directly, so every trial runs the exact environment the benchmark defines rather than one approximated by replaying a Dockerfile. That closes the environment gap that otherwise quietly skews results.
• Pre-warmed snapshots: environments with heavy apt/pip installs (PyTorch, CUDA toolchains, full Linux desktops) can be built once, snapshotted, and restored under a second for every subsequent trial or rollout.
• Independent verification: Harbor’s test script runs inside the sandbox and writes 1.0 or 0.0 to reward.txt. The agent never sees or touches the verifier, so “the agent said it worked” is never confused with “the tests pass.”
• Parallel scale: Tensorlake schedules thousands of sandboxes concurrently, which is what RL rollout generation and full benchmark sweeps need.

Anatomy of a Harbor task

Harbor expects each task to be laid out like this - take gcode-to-text as an example:

gcode-to-text
├── environment
│   ├── Dockerfile
│   └── text.gcode.gz
├── instruction.md
├── solution
│   └── solve.sh
├── task.toml
└── tests
    ├── test_outputs.py
    └── test.sh

• environment/Dockerfile defines the base image and any setup steps.
• instruction.md is the prompt the agent receives.
• solution/ is an oracle reference used to validate the environment itself.
• tests/test.sh runs after the agent finishes and produces reward.txt.

Tune sandbox resources

Each task’s task.toml controls the sandbox Harbor provisions on Tensorlake. Set resources in the [environment] block:

task.toml

[environment]
cpus = 2
memory_mb = 4096
storage_mb = 20480
network_mode = "public"

Field	Default	Forwarded to Tensorlake
cpus	1	cpus
memory_mb	2048	memory_mb
storage_mb	10240	ephemeral_disk_mb
network_mode	public	allow_internet_access / allow_out

Tensorlake requires memory_mb to be between 1024 and 8192 MB per CPU core.

allow_internet is deprecated in favor of network_mode. Harbor still accepts it and maps true/false to network_mode = "public"/"no-network", but new tasks should set network_mode directly.

A few rules of thumb:

• Large or heavy images: if your environment/Dockerfile pulls in big toolchains (PyTorch, CUDA, full Linux desktops, large datasets), bump cpus and memory_mb so the build and runtime have headroom, and raise storage_mb past the image size plus working-set room. Underprovisioned sandboxes show up as build timeouts or OOMs mid-trial.
• Lock down network_mode: set network_mode = "no-network" to stop the agent from searching the web for answers, or "allowlist" with allowed_hosts to permit only specific destinations.

Dynamic network policy

Different phases of a trial often need different network access: setup may need the open internet to install dependencies, while the agent and verifier need tighter limits so the agent can’t search the web for answers and the verifier’s result can’t be tampered with. Tensorlake sandboxes support switching the network policy on a running sandbox, so Harbor can scope network_mode per phase in task.toml:

task.toml

[environment]
network_mode = "public"

[agent] # applies during agent.run() -- but *not* during agent.setup()!
network_mode = "allowlist"
allowed_hosts = ["pypi.org", "api.openai.com"]

[verifier] # applies during verifier.verify()
network_mode = "no-network"

Tensorlake applies each phase override to the running sandbox’s firewall as a single atomic swap, so there’s no window where the sandbox is unprotected and no restart between phases. allowed_hosts accepts exact hostnames, leading-wildcard hostnames (e.g. *.example.com), IPv4 literals, and IPv4 CIDR ranges.

allowlist and per-phase overrides apply to single-container tasks only. Docker Compose tasks support public and no-network alone; see that section for why.

See Dynamic Network Policies in Harbor for more on why phase-scoped policies matter, and Harbor’s Network Policy docs for the full field reference.

Image build & caching

Each trial boots from an image. Harbor uses a prebuilt image when the task declares one, and otherwise builds the task’s Dockerfile:

Source	How to set it	When to use
Prebuilt image	docker_image in task.toml	Fastest start: boot directly from an image with the environment already baked in. Terminal-Bench ships these.
Dockerfile	environment/Dockerfile	No prebuilt image declared. Built once, cached, and reused across trials.

Either way the image is built or imported once and reused. You only pay the cost on the first trial, then every later trial boots directly from the cached image. If a task sets both, the prebuilt image wins over the Dockerfile if it’s exist.

Reusing one heavy environment across many runs (RL rollouts or repeated eval of the same task) can restore in under a second from a pre-warmed snapshot instead of rebuilding. See Snapshots.

Prebuilt image

If a task declares a docker_image in task.toml, Harbor boots directly from that image and skips the Dockerfile entirely:

task.toml

[environment]
docker_image = "myorg/my-task-env:2025-06"

Harbor looks the image up in Tensorlake by name and boots from it; if it isn’t registered yet, it imports the image once and reuses it on every later trial. The registered name is derived from the reference string you put here, so the first import of a given reference is what every later run boots.

Always publish with an immutable tag, never latest. Because the registered name comes from the reference string (not the image contents), a tag like latest is captured at its first import and then frozen: if you push new content to latest, Harbor keeps booting the old image and never re-pulls. Use an immutable tag or digest (e.g. myorg/my-task-env:2025-06 or ...@sha256:...) so a new build means a new reference, which is what triggers a fresh import. This is the convention the published Terminal-Bench images follow.

To refresh an image that’s already registered, point docker_image at a new immutable tag/digest, or delete the registered Tensorlake image so the next run re-imports it. (--force-build does not re-import: it builds from the Dockerfile instead.)

Terminal-Bench 2.1 images are already published. We’ve registered every Terminal-Bench 2.1 task image publicly, so anyone with Tensorlake access boots straight from them: no build, no import. Just run the dataset as usual and each task picks up its published image.

Harbor uses TENSORLAKE_API_KEY for image lookup and import. The API key selects the project, so no separate organization or project environment variables are needed. If Harbor reports that image lookup requires organization and project context, upgrade its Tensorlake SDK integration; that message comes from an older SDK contract.

Dockerfile

If a task has no docker_image, Harbor builds its environment/Dockerfile once via Tensorlake’s image builder, caches it, and boots every later trial directly from the cached image: no per-trial apt/pip work. The cache is keyed on the Dockerfile and every file in the build context, so editing a requirements.txt pin or any COPY’d file automatically triggers a rebuild.

harbor run --env tensorlake \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6

If a build ever fails, Harbor automatically falls back to replaying the Dockerfile’s RUN/COPY steps on each trial, so a trial is never blocked. It just runs a little slower. The fallback is also available as an explicit escape hatch while you iterate on a Dockerfile:

harbor run --env tensorlake ... --ek use_oci_image_build=false

Dockerfile requirements

The image builder is stricter than a local docker build, so a few Docker conventions need small adjustments:

• COPY does not auto-create parent directories: COPY x /a/b/c fails if /a/b doesn’t exist yet. Add RUN mkdir -p /a/b before the COPY.
• Don’t pin exact apt versions (apt-get install curl=8.5.0-2ubuntu10.6): drop the pin or pick a version that exists in the target distro.
• Use a FROM image that ships the Python you need (e.g. python:3.10-bookworm) rather than relying on a non-native version being fetched at build time.

To force a fresh rebuild even when a valid cached image exists, add --force-build. This applies to that run only and doesn’t disturb the cache used by subsequent normal runs.

harbor run --env tensorlake \
  --force-build \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6

Sharing images publicly

By default, images Harbor builds or imports are private to your organization: only your org can boot from them. Add --ek is_public=true to register a freshly built or imported image as public, so any organization with Tensorlake access can reuse it:

harbor run --env tensorlake \
  --ek is_public=true \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6

The flag applies to both Dockerfile-built images and prebuilt docker_image imports. Automatic boot-from-public by another organization is wired through the prebuilt docker_image path (that’s how the published Terminal-Bench images are reused), so if your goal is to publish an environment others boot directly, prefer a docker_image reference.

Publishing public images is gated to an allow list. If your account isn’t on it, the flag is ignored and the image stays private. Reach out to Tensorlake to be added.

is_public only takes effect when the image is newly registered. If an image with the same name already exists (a private copy from an earlier run, or an existing public one), Harbor boots it as-is and won’t republish it. To turn an already-private image public, delete it first (or change the build context so it gets a new name), then rerun with --ek is_public=true.

Ad-hoc native dependencies

If a task just needs a couple of extra apt packages and you don’t want to edit the Dockerfile or maintain a snapshot, use preinstall_packages:

harbor run --env tensorlake \
  --ek 'preinstall_packages=["build-essential","rustc","cargo"]' \
  --dataset terminal-bench/terminal-bench-2-1 \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6

The packages are installed at the start of each trial. Prefer snapshots when the package set is large or reused across many runs so you pay the install cost once.

Docker Compose (multi-container) tasks

If a task needs more than one container (a database or sidecar service alongside the agent’s environment), add an environment/docker-compose.yaml. Harbor detects it automatically and boots a docker-capable sandbox instead of a single container. No extra flag is needed:

harbor run --env tensorlake \
  --include-task-name my-org/multi-service-task \
  --dataset my-org/my-dataset \
  --agent claude-code \
  --model anthropic/claude-sonnet-4-6

Under the hood, Harbor boots a systemd-managed Docker-in-Docker (DinD) host sandbox and runs your compose project inside it: dockerd and docker compose run on the sandbox, with your main service (where the agent and verifier operate) alongside any other services you declare.

We’ve published Harbor’s default DinD host image (the outer sandbox: systemd, dockerd, docker compose) publicly, so Harbor never has to build or import it. Your compose services still build or pull as usual once docker compose runs inside that host.

A few differences from single-container tasks:

• No snapshots: snapshot_id isn’t supported for compose tasks. The compose project builds inside a fresh DinD host on every run.
• preinstall_packages is ignored: put packages in the compose service images instead.
• Network policy is coarser: only public and no-network are available (see the Dynamic network policy note above). allowlist and per-phase switching require host-level firewall control that a nested DinD host doesn’t have, so no-network isolation is enforced inside the VM instead.
• Size storage generously: compose pulls and builds layers on top of the DinD host’s own rootfs, so storage_mb needs headroom beyond what a single-container task would need for the same images.

To use a custom DinD host image instead of Harbor’s default (for example, a smaller or Alpine-based one), pass --ek dind_image=.... It must ship dockerd, docker compose, and a systemd-managed docker.service.

Interactive debugging

When a trial fails and you want to poke around the live environment, attach to the session:

harbor env attach <session_id>

Drop directly into the running sandbox to inspect state, rerun tests by hand, and confirm whether the failure was the agent or the environment.

Structured logs

Each trial produces structured artifacts, e.g.:

gcode-to-text__UFALMLv
├── agent/
├── verifier/
├── result.json
└── trial.log

So you can trace:

• The agent’s actions and outputs
• What the verifier checked
• Why the trial passed or failed

What to build next

Snapshots

Build an environment once, snapshot it, and restore in seconds for every trial.

Reproducible RL Environments

Use sandboxes as a deterministic reward oracle for RL training loops.