Inspect Robots

Inspect Robots is an evaluation framework for robot policies: you define a task, an embodiment, and a policy, and it runs trials and records an EvalLog. Dreamscale plugs in as a policy, so a Dreamscale-hosted model can be evaluated by the same harness, on the same task and embodiment, as any policy you already run. The adapter is inspect-robots-dreamscale (source, Apache 2.0). It changes one thing in your evaluation: which policy is selected. Your registered task and embodiment stay exactly as they are. Runnable reference code for everything below lives in examples/native-30hz/: a complete evaluation script and a YAM embodiment skeleton, adapted from a run that completed against production. Copy from there rather than retyping this guide.

This guide covers the software path. It does not authorize an unattended physical run. For any physical robot test you own the validated limits, an attended operator gate, a working e-stop, and a rehearsed termination procedure. Serving evidence is not safety evidence.

Before you start

You need an Inspect Robots evaluation that already runs on your hardware with some other policy — a scripted or teleop baseline is ideal. Bring up the robot, the cameras, and the task first, and confirm trials complete end to end. Adding Dreamscale to an evaluation that is not yet working makes the first failure much harder to attribute. You also need a Dreamscale API key and an entitlement for the model you intend to call. Some models, including DreamZero-YAM, are granted per organization.

1. Install

uv add inspect-robots-dreamscale

The adapter pins the exact SDK release it was qualified against, so it installs a matching dreamscale[dreamzero] for you, including the dreamscale command used below. Adapter 0.1.18 pins dreamscale[dreamzero]==0.1.0a26; confirm that is the SDK your environment resolved:

uv run python -c 'import dreamscale; assert dreamscale.__version__ == "0.1.0a26"'

Confirm the policy registered without opening a cloud session:

inspect-robots list policies

The output must contain dreamscale.

2. Authenticate

uv run dreamscale login

A robot controller usually has no browser, so pass the key directly there instead:

uv run dreamscale login --api-key "<your key>"

Both paths write the same credential file, and both are verified the same way:

uv run dreamscale status --model dreamzero-yam

status confirms the credential is accepted and reports whether the model has capacity, before you spend a session finding out.

3. Select the policy

Change only the policy selection in your existing evaluation command:

inspect-robots run \
  --task your-task \
  --embodiment your-yam \
  --policy dreamscale -P model=dreamzero-yam \
  --log-dir logs

Substitute your own registered --task and --embodiment; --policy dreamscale -P model=dreamzero-yam is the entire edit. Everything else — epochs, scorer, logging — is unchanged.

Parameters

Parameter	Default	Meaning
`model`	required	Dreamscale model id, for example `dreamzero-yam`
`sampling`	`async_latest`	Action sampling mode, see below
`control_hz`	`30`	Rate the action chunk is executed at. DreamZero-YAM accepts only `30`; any other value fails when the policy is constructed, before a session opens
`keep_warm_s`	`0`	Hold the session warm after close so the next run skips the cold start, 0–3600 s. Billed while held
`startup_timeout_s`	`1800`	Budget for the first connection while the model loads and warms up
`timeout_s`	`60`	Per-step action deadline

async_latest is the qualified default: inference is single-flight and latest-only, so the robot always acts on the freshest available chunk. Use async_8 only as an explicit compatibility or rollback path, and upstream_eval only for an agreed open-loop dataset evaluation — it blocks at each replan and is not intended for hardware. The first connection is slow on purpose. A cold model loads weights and warms up before it serves, which is why startup_timeout_s defaults to 30 minutes. Later trials in the same run reuse the open connection.

Skipping the cold start while you iterate

A cold start is roughly two minutes, paid once per connection — every trial in one evaluation shares it. It only hurts when you are iterating: fixing a camera mapping, re-running, fixing the next thing, and paying two minutes each time. keep_warm_s holds the session after close so the next run reclaims it instead of starting cold:

--policy dreamscale -P model=dreamzero-yam -P keep_warm_s=300

Measured against production, back to back:

	cold	reclaimed
Time to first action	147 s	23 s

The reclaim is matched on your organization, user, model, region mode, and optimization settings, so a later inspect-robots run with the same policy arguments picks up the session you left warm.

A warm hold is billed at the full rate, and close() no longer stops the meter. With keep_warm_s set, closing parks the session rather than terminating it: the GPU stays reserved for you, and billing continues until the window expires or you reclaim it. In a verified run, a session that was active for well under a minute billed 141.9 s across its park.So this is a cost you are choosing, not a cache. At DreamZero-YAM’s rate a five-minute hold is worth it only if you actually re-run inside it — otherwise you have paid five minutes to save two. Leave it at 0 for unattended or production runs, and reach for it only during hands-on iteration.

Control rate, and who owns the clock

DreamZero-YAM emits 24 actions per chunk on a 30 Hz action timebase, so one chunk covers 0.8 s. The model is qualified at exactly 30 Hz. You can pass the rate explicitly:

--policy dreamscale -P model=dreamzero-yam -P control_hz=30

Any other value fails when the policy is constructed, before a paid session opens. The 30 Hz timebase is the rate your embodiment executes actions at; it is not the inference call rate or your robot driver’s internal servo loop.

control_hz does not make anything run at that rate. Inspect’s rollout imposes no wall-clock pacing, so the real rate is however fast your embodiment’s step() returns — hardware paces itself on its actuators, and anything simulated or stubbed must pace itself deliberately. Your embodiment is the clock.What control_hz changes is what everything else believes: the action scheduler sizes its replan threshold from it (a 400 ms round trip is 12 steps of a 24-step chunk at 30 Hz), the compatibility check reconciles it against your embodiment’s declared rate, and it is stamped on every action in the EvalLog. Pace your embodiment at 30 Hz — a loop that runs at any other rate degrades replanning while the run still looks healthy.A rate disagreement between policy and embodiment is only a warning, never an error, so it will not stop a run. The adapter measures the real gap between policy steps, records it as step_interval_ms in the sidecar, and warns once if the measured rate is more than 25% from the commanded one.

4. Run the evaluation

What your embodiment must provide

On every policy step, the task and embodiment must supply:

top_cam, left_cam, and right_cam uint8 images, each with its own real capture time in Unix-epoch seconds (not a process-monotonic clock, and not one synthetic time shared across cameras);
finite packed joint_pos state with shape (14,), ordered left arm, left gripper, right arm, right gripper; and
Inspect’s integer extra["env_step"], starting at zero and advancing once per delivered action.

The adapter declares a 14-dimensional raw absolute-joint action at the commanded rate and returns exactly one action per act() call, while Dreamscale owns the action buffering behind it. See Model contracts for the full DreamZero-YAM contract, including gripper conventions. Capture times must be real. The model reasons over a short history of frames, so a fabricated or stale clock silently degrades its output rather than raising.

The whole script

The shape of a full run, with the hardware halves stubbed. The complete versions are run_eval.py and yam_embodiment.py.

import collections, json, pathlib, time

import numpy as np
from inspect_robots import Observation, eval as run_evaluation
from inspect_robots.embodiment import EmbodimentBase, EmbodimentInfo, StepResult
from inspect_robots.registry import resolve

LOG_DIR = "logs"


class YamEmbodiment(EmbodimentBase):
    """Your arm. Replace the driver calls; the rest is the contract above."""

    @property
    def info(self) -> EmbodimentInfo:
        ...  # 14-D joint_pos action, three 360x640 cameras, control_hz=30.0

    def _observation(self, instruction: str) -> Observation:
        top, left, right, times = self._robot.frames()  # your driver
        return Observation(
            images={"top_cam": top, "left_cam": left, "right_cam": right},
            # Real per-camera capture times, in Unix-epoch seconds.
            image_times=dict(zip(("top_cam", "left_cam", "right_cam"), times)),
            state={"joint_pos": self._robot.state.astype(np.float64)},
            state_time=time.monotonic_ns() / 1e9,
            instruction=instruction,
            extra={"env_step": self._env_step},  # advances once per action
        )

    def step(self, action) -> StepResult:
        payload = getattr(action, "data", action)  # an Action, not a bare array
        self._robot.execute(np.asarray(payload).reshape(-1))  # your driver
        self._env_step += 1
        return StepResult(
            observation=self._observation(""),
            reward=0.0,
            terminated=False,
            truncated=False,
            info={},
        )


# `resolve` returns a constructed object, not a factory, and takes the same
# parameters you would pass with `-P`.
policy = resolve("policy", "dreamscale", model="dreamzero-yam")
task = resolve("task", "your-task")
embodiment = resolve("embodiment", "your-yam")

try:
    logs = run_evaluation(task=task, policy=policy, embodiment=embodiment, log_dir=LOG_DIR)
finally:
    # Sessions bill while open and hold an exclusive lease, so this is not
    # optional — including when the run raised.
    policy.close()

# Then check where the actions actually came from. See the warning below.
sidecar = sorted(pathlib.Path(LOG_DIR).glob("dreamscale/*/*.jsonl"))[-1]
rows = [json.loads(line) for line in sidecar.read_text().splitlines() if line.strip()]
print(collections.Counter(row["action_source"] for row in rows))

Closing

Confirm nothing is still running:

uv run dreamscale sessions list

The adapter registers a bounded process-exit fallback, but it is not a substitute for the explicit close(), and it cannot help if the machine loses power. uv run dreamscale sessions stop --all clears a zombie lease.

If you construct the policy object yourself rather than naming it in the registry, you own its lifecycle. Inspect closes what it resolved from a registry name; it does not close objects you passed in. The script above resolves the policy itself, so the explicit close() is required.

5. Diagnostics

The adapter writes one atomic sidecar per trial and records its relative path at TrialRecord.metadata["dreamscale_telemetry"]:

dreamscale/<run_id>/<sanitized-scene-id>-e<epoch>.jsonl

Each row carries serving identity, the source control tick, camera capture-to-execution age, timing, chunk and merge disposition, and the same Inspect environment step. Join to the EvalLog, the action JSONL, or a Rerun timeline on env_step; use join_key for Dreamscale chunk-level diagnostics. Sidecars never contain action vectors, images, credentials, or endpoints, so they are safe to attach to a bug report. The two numbers worth reading first are the observation-to-action round trip and action_source, which records for every step whether the robot executed a model action or held its previous position.

A green evaluation can contain no model actions at all. Inspect reports success when trials complete without raising, and episode_length counts steps, not inference. A run where every step held position looks identical in the EvalLog to one the model drove throughout.Count action_source before trusting a result — the last few lines of the script above. Seeing only hold means the model never contributed, and no amount of green in the EvalLog says otherwise. The usual cause is a control loop that runs faster than one round trip, so no chunk can arrive before the episode ends — most often an embodiment that returns instantly instead of pacing itself at the control rate the way hardware does.

A high but non-zero hold fraction means the action buffer drained, usually because the round trip came close to the 0.8 s chunk horizon. Treat it as a diagnostic to investigate, not a reason to change the rate. A production run from Sydney shows the size of the effect:

	30 Hz from Sydney
Chunk horizon	0.8 s
Holds	81 / 120 (67.5%)
GPU inference p50	333 ms
Observation-to-action p50	644 ms

Most of that round trip is distance and GPU. obs_to_action_ms minus server_inference_ms is network, encode, decode, and scheduling combined, and it is dominated by distance to the serving region, us-west-2. Running the client closer to the region reduces it far more than any client-side tuning. Once the round trip exceeds the chunk horizon, the hold fraction does not degrade gracefully — it goes to 100%. No chunk can ever land before the buffer it was meant to refill has drained, so every step holds and the model contributes nothing. Re-running the same 30 Hz configuration from a tethered mobile connection instead of fixed broadband, with nothing else changed, the round trip rose from 644 ms to 945 ms — past the 800 ms horizon — and model-sourced actions went from 39/120 to 0/120. The run still reported success. That makes the horizon a threshold worth knowing: the observation-to-action round trip must stay under about 800 ms. The margin is thin if you are far from the serving region: a few hundred milliseconds of extra round trip is the difference between a working run and an empty one. Anything shared, metered, or wireless — a hotspot, a busy lab network, a VPN — can cost you that much on its own. Check action_source rather than trusting the status, and prefer a fixed, wired connection close to the region.

Where the boundaries are

Inspect Robots remains canonical for the EvalLog, scores, commanded-action logs, stored frames, Rerun recordings, and trial termination state. Dreamscale is responsible only for turning an observation into an action chunk. The adapter calls neither your embodiment nor your hardware directly. Compatibility with this contract means the cameras, state, action, clock, and rate line up. It does not by itself establish physics parity, task success, or physical-robot safety — those remain yours to demonstrate on your own hardware.

Getting started

Build with Python

Robot guides

Reference

Help

Before you start

1. Install

2. Authenticate

3. Select the policy

Parameters

Skipping the cold start while you iterate

Control rate, and who owns the clock

4. Run the evaluation

What your embodiment must provide

The whole script

Closing

5. Diagnostics

Where the boundaries are

​Before you start

​1. Install

​2. Authenticate

​3. Select the policy

​Parameters

​Skipping the cold start while you iterate

​Control rate, and who owns the clock

​4. Run the evaluation

​What your embodiment must provide

​The whole script

​Closing

​5. Diagnostics

​Where the boundaries are

Before you start

1. Install

2. Authenticate

3. Select the policy

Parameters

Skipping the cold start while you iterate

Control rate, and who owns the clock

4. Run the evaluation

What your embodiment must provide

The whole script

Closing

5. Diagnostics

Where the boundaries are