Bencher runner CLI

runner up

Start the runner, polling for and executing benchmark Jobs. This is the long-running command used to operate a Self-Hosted Runner. The runner opens a single WebSocket connection to the API server, claims Jobs that match its Specs, executes them, and reports the results back. The runner writes its log to stderr as JSON, one object per line, and logs a Job’s output only as byte counts.

runner up [OPTIONS]

Options

--host <HOST>

The Bencher API server to connect to. By default, https://api.bencher.dev is used. Can also be set with the BENCHER_HOST environment variable.

--runner <RUNNER>

The UUID or slug of the Runner to operate as. Can also be set with the BENCHER_RUNNER environment variable.

--key <KEY>

The Runner authentication key (bencher_runner_...) returned when the Runner was created. Can also be set with the BENCHER_RUNNER_KEY environment variable.

--poll-timeout <POLL_TIMEOUT>

The long-poll timeout in seconds while waiting for a Job, between 1 and 900. By default, 55 is used.

--state-dir <STATE_DIR>

The persistent state directory for the Runner. It must be an absolute path. The jail that confines the sandbox is built under this directory, and jails left behind by an unclean exit are swept from it before each sandboxed Job runs. Each jail holds a full guest root filesystem and a copy of the microVM binary, and a Runner runs its Jobs one at a time, so this directory needs real capacity for one jail. With --image-cache-size it also holds the Image cache, up to that size. A host runs one Runner at a time, whatever its state directory: a Runner started as root holds the lock /run/bencher/runner.lock for as long as it runs, and a second one started as root exits at once with an error naming that lock; a Runner that is not root cannot open the lock and runs without it. When a Runner started as root comes up, and again before each sandboxed Job, it kills any process still running in a cgroup under /sys/fs/cgroup/bencher, such as the sandbox of a Runner killed mid-Job, whatever that Runner’s state directory. A dedicated filesystem or a tmpfs mount is supported and recommended: it keeps that write traffic off the system disk, and a freshly created filesystem is accepted even though it arrives with a lost+found directory. The filesystem must not be mounted nodev or noexec, as /tmp, /dev/shm, and /run often are, so mount a tmpfs for it without those options. The state directory itself may be a root-owned symbolic link to such a filesystem, such as /var/lib/bencher-runner pointing at a dedicated disk. The link is followed only when the directory holding it is writable only by root, it points directly at its final target, and that target already exists on a path writable only by root; a link whose target does not exist yet is refused, so create the target first. By default, /var/lib/bencher-runner is used. Can also be set with the BENCHER_STATE_DIR environment variable.

--jail-uid <JAIL_UID>

The unprivileged user ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. A process on the host that owns this user ID can signal the sandbox, so set it to an ID the host does not allocate to anything else. Can also be set with the BENCHER_JAIL_UID environment variable.

--jail-gid <JAIL_GID>

The unprivileged group ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. Can also be set with the BENCHER_JAIL_GID environment variable.

--danger-allow-no-sandbox

Allow executing Jobs without a Sandbox. Without this flag, a Job whose Spec has no Sandbox is rejected at runtime. Non-sandboxed Jobs run directly on the host, so only enable this for trusted workloads. Can also be set with the BENCHER_DANGER_ALLOW_NO_SANDBOX environment variable.

--sandbox-log-level <SANDBOX_LOG_LEVEL>

The log level for the sandbox process. By default, warning is used.

--no-raid-pause

Keep taking Jobs while an md RAID array on the host resyncs, recovers, checks, repairs, or reshapes. By default, the Runner checks its md arrays before it asks for each Job, and while any of them runs one of these it pauses: it takes no new Job until the sync is done, and it never stops a Job that is already running. A paused Runner stays connected and still updates itself.

--no-auto-update

Disable automatic updates from the server. By default, the runner updates itself between Jobs when the server offers a new version. Can also be set with the BENCHER_NO_AUTO_UPDATE environment variable.

--update-channel <UPDATE_CHANNEL>

The update channel for automatic updates, either stable or canary. By default, stable is used. The stable channel updates to versioned releases. The canary channel tracks the rolling canary build, which is rebuilt whenever the runner changes on a Bencher Cloud deploy. Can also be set with the BENCHER_UPDATE_CHANNEL environment variable. Conflicts with --no-auto-update.

--max-download-size <MAX_DOWNLOAD_SIZE>

The maximum download size in bytes for self-update binaries. By default, 500 MiB is used. Conflicts with --no-auto-update.

--image-cache-size <IMAGE_CACHE_SIZE>

The maximum size in bytes of the Image blobs the Runner keeps across Jobs. The Runner keeps the config and layer blobs of each Image it pulls in blobs under its --state-dir, so a Job downloads only the blobs the Runner does not already hold, though it still fetches its Image manifest from the registry every time. Between Jobs, the Runner removes the least recently used blobs until the rest fit within this size. When it starts, the Runner creates the store and makes it its own, and if it cannot, it exits at once with an error naming the directory. By default, or set to 0, the Runner keeps no blobs across Jobs and removes any it kept before. Can also be set with the BENCHER_IMAGE_CACHE_SIZE environment variable.

--max-output-size <MAX_OUTPUT_SIZE>

The maximum size in bytes for collected stdout and stderr. By default, 25 MiB is used.

--max-file-count <MAX_FILE_COUNT>

The maximum number of output files to decode. By default, 255 is used.

--max-symlinks <MAX_SYMLINKS>

The maximum number of symlinks to follow during path resolution, matching the Linux kernel MAXSYMLINKS limit. By default, 40 is used. Only used in non-sandboxed mode.

Host Tuning

Before each benchmark, the runner applies host tuning to reduce measurement noise. By default it disables ASLR, the NMI watchdog, SMT / hyper-threading, and turbo boost; sets the CPU scaling governor to performance, swappiness to 10, and perf_event_paranoid to -1. On a host the runner has not tuned since it last booted, the flags below keep individual optimizations at their host defaults instead.

Host tuning persists until the host reboots: every setting is a runtime-only write that the runner does not revert when it exits, except the C-state hold, which ends when the runner exits. After a tuned run, a flag that skips any other setting takes effect from the next reboot: until then, the setting keeps its tuned value, and the runner logs that value at startup. Reboot a shared host after removing the runner.

--no-tuning

Disable all host tuning optimizations.

--aslr

Do not disable ASLR (default: disabled for benchmarks).

--nmi-watchdog

Do not disable the NMI watchdog (default: disabled for benchmarks).

--smt

Do not disable SMT / hyper-threading (default: disabled for benchmarks).

--turbo

Do not disable turbo boost (default: disabled for benchmarks).

--numa-balancing

Do not disable automatic NUMA balancing (default: disabled for benchmarks).

--timer-migration

Do not disable timer migration (default: disabled for benchmarks).

--soft-watchdog

Do not disable the soft lockup watchdog (default: disabled for benchmarks).

--ksm

Do not disable KSM, kernel samepage merging (default: disabled for benchmarks).

--cstates

Do not disable deep C-states (default: disabled for benchmarks).

--no-irq-steering

Do not steer device IRQs and kernel workqueues to housekeeping cores.

--no-cpuset-partition

Do not detach benchmark cores from the scheduler via an isolated cpuset partition.

--swappiness <SWAPPINESS>

Set the swappiness value. By default, 10 is used.

--governor <GOVERNOR>

Set the CPU scaling governor. By default, performance is used.

--perf-event-paranoid <PERF_EVENT_PARANOID>

Set the perf_event_paranoid value. By default, -1 is used.

--thp <THP>

Set the host transparent hugepage mode: never, madvise, always, or leave. Use leave to keep the host’s current mode. By default, never is used.

--help

Print help.

runner run

Pull an Image, create a rootfs, and execute it once on the local host. Unlike runner up, this does not connect to an API server or claim a Job. It runs a single Image directly, which is useful for testing and debugging a Runner host or an Image before submitting real Jobs. The Image’s output goes to stdout and stderr as it is, and the runner’s own log goes to stderr as JSON, one object per line.

runner run --image <IMAGE> [OPTIONS]

Options

--image <IMAGE>

The OCI Image to run, as a local path or a registry reference.

--token <TOKEN>

A JWT token for registry authentication when pulling the Image.

--sandbox <SANDBOX>

The sandbox mode for benchmark execution. Use firecracker for a Firecracker microVM (Linux only). Omit for non-sandboxed host execution.

--timeout <TIMEOUT>

The execution timeout in seconds, shared by every iteration from the start of the run. By default, 300 is used.

--state-dir <STATE_DIR>

The persistent state directory for the Runner. It must be an absolute path. The jail that confines the sandbox is built under this directory, and jails left behind by an unclean exit are swept from it before each sandboxed Job runs. Each jail holds a full guest root filesystem and a copy of the microVM binary, and a Runner runs its Jobs one at a time, so this directory needs real capacity for one jail. A host runs one Runner at a time, whatever its state directory: a Runner started as root holds the lock /run/bencher/runner.lock for as long as it runs, and a second one started as root exits at once with an error naming that lock; a Runner that is not root cannot open the lock and runs without it. When a Runner started as root comes up, and again before each sandboxed Job, it kills any process still running in a cgroup under /sys/fs/cgroup/bencher, such as the sandbox of a Runner killed mid-Job, whatever that Runner’s state directory. A dedicated filesystem or a tmpfs mount is supported and recommended: it keeps that write traffic off the system disk, and a freshly created filesystem is accepted even though it arrives with a lost+found directory. The filesystem must not be mounted nodev or noexec, as /tmp, /dev/shm, and /run often are, so mount a tmpfs for it without those options. The state directory itself may be a root-owned symbolic link to such a filesystem, such as /var/lib/bencher-runner pointing at a dedicated disk. The link is followed only when the directory holding it is writable only by root, it points directly at its final target, and that target already exists on a path writable only by root; a link whose target does not exist yet is refused, so create the target first. By default, /var/lib/bencher-runner is used. Can also be set with the BENCHER_STATE_DIR environment variable.

--jail-uid <JAIL_UID>

The unprivileged user ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. A process on the host that owns this user ID can signal the sandbox, so set it to an ID the host does not allocate to anything else. Can also be set with the BENCHER_JAIL_UID environment variable.

--jail-gid <JAIL_GID>

The unprivileged group ID the sandbox process drops to. By default, 61016 is used. It cannot be 0. Can also be set with the BENCHER_JAIL_GID environment variable.

--iter <ITER>

The number of benchmark iterations to execute. By default, 1 is used.

--allow-failure

Allow a benchmark failure without short-circuiting the remaining iterations. Running out of time still ends the run as timed out. An Image that cannot be pulled or prepared also ends the run.

--entrypoint <ENTRYPOINT>

Override the container entrypoint. May be specified multiple times.

--cmd <CMD>

Override the container command. May be specified multiple times.

--env <ENV>

Set an environment variable in KEY=VALUE format. May be specified multiple times.

--network

Enable network access inside the VM.

--output <OUTPUT>

An output file path inside the guest to collect. May be specified multiple times.

--vcpus <VCPUS>

Override the number of vCPUs (for testing).

--memory <MEMORY>

Override the memory in MiB (for testing).

--disk <DISK>

Override the disk size in MiB (for testing).

--max-output-size <MAX_OUTPUT_SIZE>

The maximum size in bytes for collected stdout and stderr. By default, 25 MiB is used.

--max-file-count <MAX_FILE_COUNT>

The maximum number of output files to decode. By default, 255 is used.

--max-symlinks <MAX_SYMLINKS>

The maximum number of symlinks to follow during path resolution, matching the Linux kernel MAXSYMLINKS limit. By default, 40 is used. Only used in non-sandboxed mode, so it conflicts with --sandbox.

--sandbox-log-level <SANDBOX_LOG_LEVEL>

The log level for the sandbox process. Requires --sandbox. By default, warning is used.

Host Tuning

Before each benchmark, the runner applies host tuning to reduce measurement noise. By default it disables ASLR, the NMI watchdog, SMT / hyper-threading, and turbo boost; sets the CPU scaling governor to performance, swappiness to 10, and perf_event_paranoid to -1. On a host the runner has not tuned since it last booted, the flags below keep individual optimizations at their host defaults instead.

Host tuning persists until the host reboots: every setting is a runtime-only write that the runner does not revert when it exits, except the C-state hold, which ends when the runner exits. After a tuned run, a flag that skips any other setting takes effect from the next reboot: until then, the setting keeps its tuned value, and the runner logs that value at startup. Reboot a shared host after removing the runner.

--no-tuning

Disable all host tuning optimizations.

--aslr

Do not disable ASLR (default: disabled for benchmarks).

--nmi-watchdog

Do not disable the NMI watchdog (default: disabled for benchmarks).

--smt

Do not disable SMT / hyper-threading (default: disabled for benchmarks).

--turbo

Do not disable turbo boost (default: disabled for benchmarks).

--numa-balancing

Do not disable automatic NUMA balancing (default: disabled for benchmarks).

--timer-migration

Do not disable timer migration (default: disabled for benchmarks).

--soft-watchdog

Do not disable the soft lockup watchdog (default: disabled for benchmarks).

--ksm

Do not disable KSM, kernel samepage merging (default: disabled for benchmarks).

--cstates

Do not disable deep C-states (default: disabled for benchmarks).

--no-irq-steering

Do not steer device IRQs and kernel workqueues to housekeeping cores.

--no-cpuset-partition

Do not detach benchmark cores from the scheduler via an isolated cpuset partition.

--swappiness <SWAPPINESS>

Set the swappiness value. By default, 10 is used.

--governor <GOVERNOR>

Set the CPU scaling governor. By default, performance is used.

--perf-event-paranoid <PERF_EVENT_PARANOID>

Set the perf_event_paranoid value. By default, -1 is used.

--thp <THP>

Set the host transparent hugepage mode: never, madvise, always, or leave. Use leave to keep the host’s current mode. By default, never is used.

--help

Print help.