Self-Hosted Bare Metal Runners

A Self-Hosted Runner is a Bare Metal Runner that you run and manage yourself, instead of using a Bencher managed On-Demand or Dedicated Runner. You point the runner binary at a Bencher API server, and it claims and executes Jobs on your own hardware.

Self-hosting a Runner is a good fit when you want to:

• Run benchmarks on specific hardware that Bencher Cloud does not offer
• Keep benchmark workloads inside your own network or an air-gapped environment
• Run Bencher Self-Hosted end-to-end

This guide walks through registering a Runner, describing its hardware with a Spec, linking the two together, and starting the runner binary. For a conceptual overview of how Runners, Specs, Sandboxes, and Jobs fit together, see the Bare Metal Overview.

🐰 Firecracker sandboxing requires Linux 5.14 or later with KVM enabled and a writable cgroup v2 hierarchy at /sys/fs/cgroup, and the Runner must run as root. Each sandboxed Job runs in its own cgroup under /sys/fs/cgroup/bencher, whatever the number of CPU cores, with the cpuset, memory, and pids controllers enabled for it. The Runner kills a Job’s processes through that cgroup’s cgroup.kill, which older kernels lack: there a Runner exits at startup or, started with --danger-allow-no-sandbox, fails each sandboxed Job. The Firecracker process behind each sandboxed Job is confined to a chroot as an unprivileged user, and building that confinement needs privileges that enabling KVM alone does not grant. A Runner that only serves non-sandboxed Specs can run on any supported host.

Create a Runner

A Runner is the resource that authenticates the runner binary to your Bencher API server. Create one with the bencher CLI, the Runners REST API, or the Bencher Console.

bencher runner create \
  --host https://api.bencher.example.com \
  --name "My Runner"

Bencher returns the new Runner along with its key:

{
  "uuid": "...",
  "key": "bencher_runner_..."
}

The key is only shown once, so store it securely. You will pass it to the runner binary when you start the Runner. If a key is ever lost or leaked, rotate it with the POST /v0/runners/{runner}/key API, which immediately invalidates the previous key.

Create a Spec

A Spec describes the hardware that a Runner provides: operating system, CPU architecture, Sandbox type, CPU count, memory, disk, and network access. A Job declares the Spec it needs, and a Runner only claims Jobs for Specs it supports.

Create a Spec with the bencher CLI, the Specs REST API, or the Bencher Console:

bencher spec create \
  --host https://api.bencher.example.com \
  --name "Intel v1" \
  --os linux \
  --architecture x86_64 \
  --sandbox firecracker \
  --cpu 4 \
  --memory 51539607552 \
  --disk 137438953472

Omit --sandbox to define a non-sandboxed Spec, and pass --network if Jobs on this Spec are allowed network access. Memory and disk are specified in bytes. For more on Specs and how a Testbed records the Spec used for each Report, see Testbeds & Specs.

Assign a Spec to a Runner

A Runner Spec links a Spec to a Runner. A Runner can support multiple Specs, and a Spec can be supported by multiple Runners. A Runner only claims Jobs whose Spec it has been assigned.

Add a Spec to a Runner with the bencher CLI or the Runner Specs REST API:

bencher runner spec add my-runner \
  --host https://api.bencher.example.com \
  --spec intel-v1

Remove the link later with the DELETE /v0/runners/{runner}/specs/{spec} API.

Start the Runner

With a Runner created and at least one Spec assigned, start the runner binary on your hardware with runner up. Provide the API server host, the Runner UUID or slug, and the Runner key, either as flags or environment variables:

runner up \
  --host https://api.bencher.example.com \
  --runner my-runner \
  --key bencher_runner_...

export BENCHER_HOST=https://api.bencher.example.com
export BENCHER_RUNNER=my-runner
export BENCHER_RUNNER_KEY=bencher_runner_...
runner up

The runner binary opens a single WebSocket connection to the API server, polls for Jobs matching its Specs, executes each one, and reports the results back. The connection stays open across Jobs, and the binary updates itself between Jobs when a new version is available. By default the Runner follows the stable update channel, updating only to versioned releases. The canary channel instead tracks the rolling canary build deployed to Bencher Cloud.

A Runner that executes sandboxed Jobs must run as root, so the commands above are shown as they would be run by the root user or under a service manager that starts the Runner as root.

By default a Runner rejects any Job whose Spec has no Sandbox. To let a Runner execute non-sandboxed Jobs directly on the host, start it with the --danger-allow-no-sandbox flag.

• runner up

  
  

  runner up

  Start the runner, polling for and executing benchmark Jobs. This is the long-running command used to operate a Self-Hosted Runner. The runner opens a single WebSocket connection to the API server, claims Jobs that match its Specs, executes them, and reports the results back. The runner writes its log to stderr as JSON, one object per line, and logs a Job’s output only as byte counts.

  runner up [OPTIONS]

  Options

  --host <HOST>

  The Bencher API server to connect to. By default, https://api.bencher.dev is used. Can also be set with the BENCHER_HOST environment variable.

  --runner <RUNNER>

  The UUID or slug of the Runner to operate as. Can also be set with the BENCHER_RUNNER environment variable.

  --key <KEY>

  The Runner authentication key (bencher_runner_...) returned when the Runner was created. Can also be set with the BENCHER_RUNNER_KEY environment variable.

  --poll-timeout <POLL_TIMEOUT>

  The long-poll timeout in seconds while waiting for a Job, between 1 and 900. By default, 55 is used.

  --state-dir <STATE_DIR>

  The persistent state directory for the Runner. It must be an absolute path. The jail that confines the sandbox is built under this directory, and jails left behind by an unclean exit are swept from it before each sandboxed Job runs. Each jail holds a full guest root filesystem and a copy of the microVM binary, and a Runner runs its Jobs one at a time, so this directory needs real capacity for one jail. With --image-cache-size it also holds the Image cache, up to that size. A host runs one Runner at a time, whatever its state directory: a Runner started as root holds the lock /run/bencher/runner.lock for as long as it runs, and a second one started as root exits at once with an error naming that lock; a Runner that is not root cannot open the lock and runs without it. When a Runner started as root comes up, and again before each sandboxed Job, it kills any process still running in a cgroup under /sys/fs/cgroup/bencher, such as the sandbox of a Runner killed mid-Job, whatever that Runner’s state directory. A dedicated filesystem or a tmpfs mount is supported and recommended: it keeps that write traffic off the system disk, and a freshly created filesystem is accepted even though it arrives with a lost+found directory. The filesystem must not be mounted nodev or noexec, as /tmp, /dev/shm, and /run often are, so mount a tmpfs for it without those options. The state directory itself may be a root-owned symbolic link to such a filesystem, such as /var/lib/bencher-runner pointing at a dedicated disk. The link is followed only when the directory holding it is writable only by root, it points directly at its final target, and that target already exists on a path writable only by root; a link whose target does not exist yet is refused, so create the target first. By default, /var/lib/bencher-runner is used. Can also be set with the BENCHER_STATE_DIR environment variable.

  --jail-uid <JAIL_UID>

  The unprivileged user ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. A process on the host that owns this user ID can signal the sandbox, so set it to an ID the host does not allocate to anything else. Can also be set with the BENCHER_JAIL_UID environment variable.

  --jail-gid <JAIL_GID>

  The unprivileged group ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. Can also be set with the BENCHER_JAIL_GID environment variable.

  --danger-allow-no-sandbox

  Allow executing Jobs without a Sandbox. Without this flag, a Job whose Spec has no Sandbox is rejected at runtime. Non-sandboxed Jobs run directly on the host, so only enable this for trusted workloads. Can also be set with the BENCHER_DANGER_ALLOW_NO_SANDBOX environment variable.

  --sandbox-log-level <SANDBOX_LOG_LEVEL>

  The log level for the sandbox process. By default, warning is used.

  --no-raid-pause

  Keep taking Jobs while an md RAID array on the host resyncs, recovers, checks, repairs, or reshapes. By default, the Runner checks its md arrays before it asks for each Job, and while any of them runs one of these it pauses: it takes no new Job until the sync is done, and it never stops a Job that is already running. A paused Runner stays connected and still updates itself.

  --no-auto-update

  Disable automatic updates from the server. By default, the runner updates itself between Jobs when the server offers a new version. Can also be set with the BENCHER_NO_AUTO_UPDATE environment variable.

  --update-channel <UPDATE_CHANNEL>

  The update channel for automatic updates, either stable or canary. By default, stable is used. The stable channel updates to versioned releases. The canary channel tracks the rolling canary build, which is rebuilt whenever the runner changes on a Bencher Cloud deploy. Can also be set with the BENCHER_UPDATE_CHANNEL environment variable. Conflicts with --no-auto-update.

  --max-download-size <MAX_DOWNLOAD_SIZE>

  The maximum download size in bytes for self-update binaries. By default, 500 MiB is used. Conflicts with --no-auto-update.

  --image-cache-size <IMAGE_CACHE_SIZE>

  The maximum size in bytes of the Image blobs the Runner keeps across Jobs. The Runner keeps the config and layer blobs of each Image it pulls in blobs under its --state-dir, so a Job downloads only the blobs the Runner does not already hold, though it still fetches its Image manifest from the registry every time. Between Jobs, the Runner removes the least recently used blobs until the rest fit within this size. When it starts, the Runner creates the store and makes it its own, and if it cannot, it exits at once with an error naming the directory. By default, or set to 0, the Runner keeps no blobs across Jobs and removes any it kept before. Can also be set with the BENCHER_IMAGE_CACHE_SIZE environment variable.

  --max-output-size <MAX_OUTPUT_SIZE>

  The maximum size in bytes for collected stdout and stderr. By default, 25 MiB is used.

  --max-file-count <MAX_FILE_COUNT>

  The maximum number of output files to decode. By default, 255 is used.

  --max-symlinks <MAX_SYMLINKS>

  The maximum number of symlinks to follow during path resolution, matching the Linux kernel MAXSYMLINKS limit. By default, 40 is used. Only used in non-sandboxed mode.

  Host Tuning

  Before each benchmark, the runner applies host tuning to reduce measurement noise. By default it disables ASLR, the NMI watchdog, SMT / hyper-threading, and turbo boost; sets the CPU scaling governor to performance, swappiness to 10, and perf_event_paranoid to -1. On a host the runner has not tuned since it last booted, the flags below keep individual optimizations at their host defaults instead.

  Host tuning persists until the host reboots: every setting is a runtime-only write that the runner does not revert when it exits, except the C-state hold, which ends when the runner exits. After a tuned run, a flag that skips any other setting takes effect from the next reboot: until then, the setting keeps its tuned value, and the runner logs that value at startup. Reboot a shared host after removing the runner.

  --no-tuning

  Disable all host tuning optimizations.

  --aslr

  Do not disable ASLR (default: disabled for benchmarks).

  --nmi-watchdog

  Do not disable the NMI watchdog (default: disabled for benchmarks).

  --smt

  Do not disable SMT / hyper-threading (default: disabled for benchmarks).

  --turbo

  Do not disable turbo boost (default: disabled for benchmarks).

  --numa-balancing

  Do not disable automatic NUMA balancing (default: disabled for benchmarks).

  --timer-migration

  Do not disable timer migration (default: disabled for benchmarks).

  --soft-watchdog

  Do not disable the soft lockup watchdog (default: disabled for benchmarks).

  --ksm

  Do not disable KSM, kernel samepage merging (default: disabled for benchmarks).

  --cstates

  Do not disable deep C-states (default: disabled for benchmarks).

  --no-irq-steering

  Do not steer device IRQs and kernel workqueues to housekeeping cores.

  --no-cpuset-partition

  Do not detach benchmark cores from the scheduler via an isolated cpuset partition.

  --swappiness <SWAPPINESS>

  Set the swappiness value. By default, 10 is used.

  --governor <GOVERNOR>

  Set the CPU scaling governor. By default, performance is used.

  --perf-event-paranoid <PERF_EVENT_PARANOID>

  Set the perf_event_paranoid value. By default, -1 is used.

  --thp <THP>

  Set the host transparent hugepage mode: never, madvise, always, or leave. Use leave to keep the host’s current mode. By default, never is used.

  --help

  Print help.

See the runner CLI reference for every option, and the Runner Protocol reference for the messages exchanged with the server.

Host Maintenance

A Runner and its host’s maintenance, such as package upgrades or a RAID scrub, take turns, so maintenance never runs during a Job. They take turns through /run/bencher, a directory that a Runner started as root creates for root alone:

• The job lock, /run/bencher/job.lock. The Runner takes it before it asks for a Job and holds it until that poll ends with no Job or the Job it brings ends. Maintenance waits for the lock, so it starts only between Jobs, and a Runner that finds the lock held takes no Job until it is free.
• Maintenance markers, /run/bencher/maintenance or /run/bencher/maintenance.<name>. Maintenance creates its own marker before it waits for the lock and removes it when it is done. While any marker exists, the Runner asks for no Job and leaves the lock free, so waiting maintenance starts once the Runner’s current poll, or the Job that poll brings, ends.

While it waits for maintenance, a Runner is paused: it stays connected and still updates itself. runner run waits for the job lock as well, and warns but runs while a marker exists or an md array syncs. A Runner that is not root can neither open the job lock nor see the markers, so it takes Jobs without them.

With systemd, a drop-in has a maintenance unit take its turn. For example, /etc/systemd/system/fstrim.service.d/zz-bencher-quiet.conf wraps the unit’s own command in flock -F. The empty ExecStart= clears that command, so the next line must repeat it exactly:

[Service]
ExecStartPre=+/usr/bin/install -d -m 0700 /run/bencher
ExecStartPre=+/usr/bin/touch /run/bencher/maintenance.%n
OpenFile=/dev/null
ExecStart=
ExecStart=/usr/bin/flock -F /run/bencher/job.lock /sbin/fstrim --listed-in /etc/fstab:/proc/self/mountinfo --verbose --quiet-unsupported
ExecStopPost=+/bin/sh -c '/usr/bin/busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager PreparingForShutdown | /usr/bin/grep -q true || /usr/bin/rm -f /run/bencher/maintenance.%n'
TimeoutStartSec=infinity

OpenFile=/dev/null fills file descriptor 3, so the lock stays held even if the command reopens descriptor 3, as apt.systemd.daily does. While the host prepares to shut down, ExecStopPost= leaves the marker, so the Runner takes no Job before the reboot empties /run.

Drain a Runner

To drain a Runner, create the plain marker as root:

touch /run/bencher/maintenance

The Runner finishes its current poll, at most --poll-timeout seconds, and any Job running or brought by that poll, then takes no other Job until the marker is gone:

rm /run/bencher/maintenance

/run is emptied at boot, so a reboot removes the marker too.

Self-Hosted Runner Workflow

Once the Runner is connected, you submit benchmarks the same way as with any Bare Metal Runner: build and push an Image, then run bencher run --image with a matching Spec. The diagram below shows the full path, from registering the Runner through executing a Job.

Keep Going: runner CLI Reference ➡