Testbeds & Specs with bencher run

A Testbed is the name of the testing environment used when running a Report. When using the bencher run CLI subcommand, select a Testbed with the --testbed option or the BENCHER_TESTBED environment variable. If neither is specified, Linux, macOS, or Windows is used as the default based on the host operating system. If the bencher CLI has been compiled for a different operating system, then localhost is used.

A Testbed will have a Spec assigned when it is used for Bare Metal benchmarking. If neither the --testbed option nor the BENCHER_TESTBED environment variable are set during Bare Metal benchmarking, then the Spec’s name is used. When a Report is created, the Testbed’s current Spec at that point in time is recorded with the Report. Only results from the same Testbed and Spec are used for Thresholds. This lets you track how benchmark results relate to the underlying hardware, even if the Testbed’s Spec changes over time. Use the --spec option to set or update a Testbed’s Spec, or the --spec-reset flag to clear it.

--testbed <TESTBED>


Either the --testbed option or the BENCHER_TESTBED environment variable may be set to the name, slug, or UUID for a Testbed. If the value specified is a name or slug and the Testbed does not already exist, it will be created for you. However, if the value specified is a UUID then the Testbed must already exist. If both are specified, the --testbed option takes precedence over the BENCHER_TESTBED environment variable. If neither are specified then Linux, macOS, or Windows is used, based on the host operating system. If the bencher CLI has been compiled for a different operating system, then localhost is used as the default Testbed.

--spec <SPEC>


Specify the slug or UUID for a Spec. A Spec is a hardware specification that describes the resources available to a Runner:

  1. Operating System (OS)
    1. Linux
    2. macOS
    3. Windows
  2. CPU Architecture
    1. x86_64
    2. ARM64
  3. Sandbox
    1. Firecracker microVM
    2. No Sandbox
  4. CPU Count
  5. Memory Size (RAM)
  6. Disk Size
  7. Network Access

Specs are either server-scoped or organization-scoped, so the same Spec can be shared across multiple Testbeds on the same server or within the same organization respectively.

The following Specs are currently available for Shared Runners on Bencher Cloud:

NameSlugOSArchitectureSandboxCPUMemoryDiskNetwork
Intel v1intel-v1Linuxx86_64Firecracker448.0 GiB128.0 GiBNo

The order of precedence for selecting the Spec when using bencher run is as follows:

  1. The --spec option
  2. The Spec assigned to the Testbed
  3. The default organization-scoped Spec
  4. The default server-scoped Spec

The selected Spec will then be assigned to the current Testbed. This option requires the --image option to be set.

--spec-reset


Reset the Testbed Spec, removing its hardware specification. This clears the spec associated with the Testbed when the run is submitted. Requires the --testbed option. Conflicts with the --image option.


🐰 Congrats! You have learned all about Testbeds & Specs! 🎉


Keep Going: Thresholds & Alerts ➡


Published: Fri, April 10, 2026 at 6:00:00 AM UTC