CONTINUOUS BENCHMARKING

Catch performance regressions before they merge

Bencher runs on bare metal to eliminate noise and deliver benchmark results you can actually trust.

TRUSTED BY ENGINEERING TEAMS AT

THE PROBLEM

CI benchmarks are too noisy to trust

If you can't tell a real regression from noise, the data is worthless. So teams stop looking, and performance regressions silently ship.

CI runners are shared and unpredictable

Shared CPU, memory contention, and scheduler jitter mean two identical benchmark runs can return wildly different results.

Noisy results train you to ignore alerts

When benchmarks cry wolf on every PR, engineers stop looking. By the time a real performance regression lands, nobody catches it.

Performance regressions silently ship

Without trustworthy benchmark results in the PR workflow, performance regressions reach production. You find out when users do.

WHY BARE METAL

All signal. No noise.

CI runners introduce enough noise to mask real performance regressions. Bencher Bare Metal isolates your benchmarks on dedicated bare metal hardware, so when a number changes, it means something.

TYPICAL CI RUNNER VARIANCE

>30%

Noisy shared CI runners hide real performance regressions.

BENCHER BARE METAL VARIANCE

<2%

Dedicated bare metal hardware. When a number moves, it's real.

HOW IT WORKS

Run locally. Run in CI. Same bare metal every time.

Run your benchmarks

Run your benchmarks locally or in CI using the exact same bare metal runners and your favorite benchmarking tools. The bencher CLI orchestrates running your benchmarks on bare metal and stores the results.

Track your benchmarks

Track the results of your benchmarks over time. Monitor, query, and graph the results using the bencher CLI and the Bencher web console based on the source branch and testbed.

Catch performance regressions

Catch performance regressions locally or in CI using the exact same bare metal hardware. Bencher uses state of the art, customizable analytics to detect performance regressions before they merge.

IN DEVELOPMENT

Catch performance regressions before you even push

The bencher CLI runs anywhere your code does. Catch regressions at the source, before you even open a PR. The CLI and API are agent-ready, your coding agents can benchmark on bare metal without you in the loop.

~/code/bencher

$ bencher run \
    --project bencher \
    --image bench:latest \
    --branch main \
    --testbed intel-v1 \
    --spec intel-v1 \
    --adapter rust_criterion \
    --error-on-alert

Waiting for remote job e2215f33-824c-4aa9-925b-8056526b1a07 to complete...
Note: If you interrupt (Ctrl+C), the remote job will continue running.
Job status: Running
Job status: Processed

Job stdout:
Adapter::Magic (JSON)  time:   [5.02 µs 5.03 µs 5.04 µs]
Adapter::Json          time:   [21.82 µs 21.83 µs 21.84 µs]
Adapter::Magic (Rust)  time:   [4.97 µs 4.98 µs 4.99 µs]
Adapter::Rust          time:   [5.01 µs 5.02 µs 5.03 µs]
Adapter::RustBench     time:   [4.98 µs 4.99 µs 5.00 µs]

View report: https://bencher.dev/perf/bencher/reports/36a1eeff-57f5-4b99-b058-8c9c240a9f2c

View results:
- Adapter::Json (Latency): https://bencher.dev/perf/bencher/benchmarks/e93b3d71-8499-4fae-bb7c-4e540b775714
- Adapter::Magic (JSON) (Latency): https://bencher.dev/perf/bencher/benchmarks/3bfd5887-83ec-4e62-8690-02855a38fbc9
- Adapter::Magic (Rust) (Latency): https://bencher.dev/perf/bencher/benchmarks/3525f177-fc8f-4a92-bd2f-dda7c4e15699
- Adapter::Rust (Latency): https://bencher.dev/perf/bencher/benchmarks/5655ed2a-3e45-4622-bdbd-39cdd9837af8
- Adapter::RustBench (Latency): https://bencher.dev/perf/bencher/benchmarks/1db23e93-f909-40aa-bf42-838cc7ae05f5

View alerts:
- Adapter::Json (Latency): https://bencher.dev/perf/bencher/alerts/91ee27a7-2aee-41fe-b037-80b786f26cd5

github-actions bot commented 1 day ago · edited ▾

Bencher Report

Branch	254/merge
Testbed	intel-v1

🚨 1 ALERT: Threshold Boundary Limit exceeded!

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
Adapter::Json	Latency microseconds (µs)	📈 plot 🚨 alert 🚷 threshold	21.83 (+336.60%)	15.00 (145.53%)

Click to view all benchmark results

View full continuous benchmarking report in Bencher

IN REVIEW

Catch performance regressions in code review, not in production

No dashboards to remember to check. No manual benchmark runs. Every benchmark run lands as a PR comment. Regressions fail the build.

WHAT ENGINEERS SAY

Performance coverage for your PRs

Bencher is like CodeCov for performance metrics.

Jonathan Woollett-Light @JonathanWoollett-Light

Now that I'm starting to see graphs of performance over time automatically from tests I'm running in CI. It's like this whole branch of errors can be caught and noticed sooner.

Price Clark @gpwclark

95% of the time I don't want to think about my benchmarks. But when I need to, Bencher ensures that I have the detailed historical record waiting there for me. It's fire-and-forget.

Joe Neeman @jneem

BENCHMARK HARNESSES

Bring your own benchmark harness

Rust

Criterion
libtest bench
Iai
Gungraun

C++

Google Benchmark
Catch2

Python

pytest-benchmark
airspeed velocity

Java

BenchmarkDotNet

JavaScript

Benchmark.js

go test -bench

Ruby

Benchmark

Dart

benchmark_harness

Shell

Hyperfine

Don't see your harness? Open an issue →

DEPLOYMENT

Your infrastructure, or ours.

Open Source

Self-Hosted

Deploy Bencher on your own infrastructure. Bare metal, Docker, or Kubernetes. Full control, no data leaving your environment.

Deploy in 60 seconds

Recommended

Bencher Cloud

Zero infrastructure to manage. On-demand bare metal runners, billed by the minute. Pay for your benchmark runs, not idle servers.

Benchmark for free

Your next performance regression won't announce itself

Catch it in review, or pay for it in production.

Catch the next regression