Bare Metal Overview
Most benchmark tracking tools only let you run benchmarks from a single environment. Either you run your benchmarks locally or you run them in CI. Even if they support both local and CI execution, you’re unable to compare results between the two.
Bencher Bare Metal allows you to have the best of both worlds. You can run benchmarks from your local machine and have them executed on dedicated hardware, and you can also run benchmarks from CI and have them executed on the exact same dedicated hardware. This allows you to have a consistent benchmark execution environment regardless of where you run your benchmarks.
Bencher Bare Metal runs on bare metal hardware to eliminate variability and provide more accurate and reliable benchmark results.
Runner
A Runner is a remote benchmark executor that runs on dedicated, bare metal hardware. The Runner polls the Bencher API server for pending Jobs based on its associated Specs. Once a Job is claimed, the Runner pulls the specified Image for the Job, unpacks the Image, executes the benchmark command, and sends the results back to the Bencher API server. Only one Job is executed at a time to ensure that the benchmarks are not affected by other workloads.
| Runner | Platform | Scope | Managed By | Sandbox | Network Access | Tier |
|---|---|---|---|---|---|---|
| Shared | Bencher Cloud | Server | Bencher | Required | No | Free, Team, Enterprise |
| Dedicated | Bencher Cloud | Organization | Bencher | Optional | No | Enterprise |
| Custom | Bencher Cloud | Organization | User | Optional | Optional | Enterprise |
| Self-Hosted Server | Bencher Self-Hosted | Server | User | Optional | Optional | Free, Team, Enterprise |
| Self-Hosted Organization | Bencher Self-Hosted | Organization | User | Optional | Optional | Enterprise |
Spec
A Spec is a hardware specification that describes the resources available to a Runner. A Job must specify which Spec it wants to use. The Job is then assigned to a Runner that supports the specified Spec. See Testbeds and Specs for a full overview.
Sandbox
A Sandbox is an isolation mechanism that allows for safe multi-tenancy on a Runner.
Bencher Bare Metal uses Firecracker microVMs as the default Sandbox implementation.
Running without a Sandbox requires a Dedicated or Custom Runner on Bencher Cloud (contact Bencher Plus support for more information) or Bencher Self-Hosted.
When a Runner is configured to use a Sandbox, each Job is executed within that Sandbox.
See the runner documentation for a full overview.
Job
A Job tracks the lifecycle of a remote benchmark execution request. The Job must specify the benchmark command and which Spec it wants to use. The Job is then assigned to a Runner that supports the specified Spec. See Bare Metal images for a full overview.
Image
An Image is an OCI container image that contains your benchmark code and its dependencies. When a Runner executes a Job, it pulls the Image, unpacks the Image, and executes the benchmark command specified in the Job. See Bare Metal images for a full overview.