Bencher Bare Metal Quick Start

What is Bencher?

Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they merge.

• Run: Run your benchmarks locally or in CI using the exact same bare metal runners and your favorite benchmarking tools. The bencher CLI orchestrates running your benchmarks on bare metal and stores its results.
• Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
• Catch: Catch performance regressions locally or in CI using the exact same bare metal hardware. Bencher uses state of the art, customizable analytics to detect performance regressions before they merge.

For the same reasons that unit tests are run to prevent feature regressions, benchmarks should be run with Bencher to prevent performance regressions. Performance bugs are bugs!

Bencher Bare Metal

Bencher Bare Metal runs your benchmarks on bare metal hardware to eliminate variability and provide more accurate and reliable benchmark results. If you are interested in using Bencher Cloud without bare metal hardware, check out the Bencher Cloud Quick Start tutorial. This tutorial will get you setup using Bencher Bare Metal.

🐰 Once you feel comfortable using Bencher Bare Metal, consider checking out the following resources:

• Bare Metal Overview
• Bare Metal Images
• Testbeds & Specs

Install bencher CLI

Select your operating system and run the provided command to install the bencher CLI. For more details, see the bencher CLI install documentation.

⠀

Now, lets check that you have the bencher CLI installed. Run:

bencher --version

You should see:

bencher 0.6.2

Create a Project

Before you can push a benchmark image, you need a Bencher project. The easiest way to create one is to run the bencher run CLI subcommand with the bencher mock subcommand. Run:

bencher run "bencher mock --count 0"

This creates a new project on Bencher Cloud. Look at the View report URL in the output. Your project slug is the path segment after /perf/ in the URL. For example, if the URL is:

https://bencher.dev/perf/my-project-abc1234/reports/709d3476-51a4-4939-9584-75d9a2c04c54

then your project slug is my-project-abc1234. You will need this project slug in the next steps.

Push your Benchmark Image

Bencher Bare Metal runs benchmarks packaged as OCI container images on dedicated hardware. For this quick start, we will use the bencher CLI Docker image as our benchmark image. No custom image is needed.

First, pull the bencher CLI image from GitHub Container Registry. Run:

docker pull --platform linux/amd64 ghcr.io/bencherdev/bencher:latest

Next, retag the image for the Bencher OCI registry. Replace YOUR_PROJECT_SLUG with the project slug from the previous step:

docker tag ghcr.io/bencherdev/bencher:latest registry.bencher.dev/YOUR_PROJECT_SLUG:latest

Then, push the image to the Bencher OCI registry:

docker push registry.bencher.dev/YOUR_PROJECT_SLUG:latest

For more details on creating your own custom image, see the Image documentation.

Track your Bare Metal Benchmarks

You are now ready to track your benchmark results on bare metal hardware! To do so, you will use the bencher run CLI subcommand to run your benchmarks and collect the results. Replace YOUR_PROJECT_SLUG with the project slug from the previous steps and run:

bencher run --project YOUR_PROJECT_SLUG --image YOUR_PROJECT_SLUG:latest noise --format json

We are using the bencher noise subcommand as our benchmark command to generate some sample data. It measures how noisy your benchmark environment is. When we use the --format json option the results are output in Bencher Metric Format (BMF). If everything works as expected, the end of the output should look something like this:

View report: https://bencher.dev/perf/project-abc4567-wxyz123456789/reports/29d61b7c-5d80-4ac2-a118-f80bc5de431b

View results:
- bencher::noise::cache_jitter (Cache Jitter): https://bencher.dev/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=6bed122a-b0d1-42e6-93c1-8736a910ec9c&measures=fdee350f-f53a-44bc-af7c-2f4c51415ea8&start_time=1741390156000&end_time=1743982156000&report=29d61b7c-5d80-4ac2-a118-f80bc5de431b
- bencher::noise::composite (Noise Score): https://bencher.dev/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=a5d0aea8-666f-4554-958a-9ad8acf1a20d&measures=995e8a26-58ee-46c4-8ce7-d0162b7d9d86&start_time=1741390156000&end_time=1743982156000&report=29d61b7c-5d80-4ac2-a118-f80bc5de431b
- bencher::noise::compute_jitter (Compute Jitter): https://bencher.dev/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=489b1d60-5dbd-43cd-bb96-ba50bcba648a&measures=8b715d47-ed42-4ac5-a380-4fe4143aa159&start_time=1741390156000&end_time=1743982156000&report=29d61b7c-5d80-4ac2-a118-f80bc5de431b
- bencher::noise::io_jitter (I/O Jitter): https://bencher.dev/perf/project-abc4567-wxyz123456789?branches=88d5192d-5cd1-47c6-a817-056e5968737c&heads=657a8ee9-1f30-49d4-bd9b-ceed02576d7e&testbeds=f3a5db46-a57e-4caf-b96e-f0c1111eaa67&benchmarks=0a3da391-2fe3-40ec-8a10-983e81231af8&measures=b34f6fd4-60ff-4866-a7a4-60396aab976a&start_time=1741390156000&end_time=1743982156000&report=29d61b7c-5d80-4ac2-a118-f80bc5de431b

Claim this project: https://bencher.dev/auth/signup?claim=d4b0cd5a-8422-40af-9872-8e18d5d062c4

You can now view the results for each of your benchmarks in the browser. Click or copy and paste the links from View results. To claim these results, click or copy and paste the Claim this project link into your browser.

🐰 Congrats! You tracked your first Bare Metal benchmark results! 🎉

Keep Going: How to Claim Benchmark Results ➡