Continuous benchmarking on bare metal

A benchmark result (called a Metric) is a single point-in-time measurement. Each benchmark run produces one Metric, so 5 benchmarks × 10 runs = 50 Metrics. Even if your harness samples a benchmark 1,000 times internally for accuracy, that still counts as 1 Metric per run. A benchmark with multiple measures (like latency and throughput) creates one Metric per measure per run, so 5 benchmarks × 2 measures × 10 runs = 100 Metrics.

A Public Project is visible to anyone who can reach your Bencher instance. On Bencher Cloud, all Public Projects are listed here. A Private Project is only visible to members of your organization and requires an active Bencher Plus plan (Team or Enterprise).

On Bencher Cloud, benchmark results (called Metrics) are billed monthly based on metered usage. If you create 5,280 Metrics in a month, you're billed for 5,280 Metrics. Bencher Self-Hosted Metrics are billed annually based on a licensed quantity.

Yes. On Team or Enterprise, benchmark results (called Metrics) from both Public and Private Projects count toward your plan. The Free plan isn't billed for Metrics, but daily rate limits apply.

• Unclaimed Project: 255 Metrics per day
• Free plan: 65,535 Metrics per day
• Bencher Plus plan: No rate limit

Community support (Free): GitHub issues and our Discord community.

Customer support (Team): Email support with responses within 1 business day.

Priority support (Enterprise): Dedicated Slack channel with direct access to Bencher engineering, priority on bug fixes and feature requests, and dedicated onboarding.

Yes. You can upgrade, downgrade, or cancel your plan at any time. On Bencher Cloud, metered billing stops at the end of the current period. Bencher Self-Hosted licenses aren't refunded mid-term, but you can downgrade or cancel before your next renewal.

Your Projects and benchmark results (called Metrics) remain accessible after cancellation. Public Projects stay publicly visible unless you delete them, and Private Projects stay private. You can export your data via the Bencher API at any time.

The Bencher License is heavily inspired by the GitLab License. You can use Bencher Self-Hosted in development, testing, and production without a Bencher Plus license, equivalent to Bencher Cloud on the Free plan.

For Bencher Plus features (Team or Enterprise), you need to purchase a Bencher Plus license.

On Bencher Self-Hosted, new Metrics stop being accepted once you hit your licensed quantity limit. You can increase your licensed quantity at any time. We recommend setting your quantity with an extra margin to avoid interruption.

Bencher is the first Continuous Benchmarking tool that runs your benchmarks on the exact same bare metal hardware both locally and in CI. Developers and agents can compare work in progress against any commit in your project's history.

Locally, keep multitasking. No stopping what you're doing, pulling an old branch, and waiting on a comparison.

In CI, trust the numbers. No noisy neighbors, throttling, or host migrations.

No. You can run benchmarks right away without creating an account, with a 1 minute timeout per run. A Free account raises that timeout to 5 minutes and increases your daily rate limits. No credit card required either way. Check out the Bencher Bare Metal Quickstart to run your first benchmark in minutes.

Bencher Cloud On-Demand runners use the intel-v1 Spec: Linux on x86_64 with 4 CPUs, 48 GiB of memory, and 128 GiB of disk. Code runs in a Firecracker microVM sandbox for isolation. See the full Spec reference for details. Bencher Cloud Enterprise customers can request Dedicated and Custom runners. Contact us for more details.

On Bencher Self-Hosted, Free and Team users can create server-scoped Bare Metal runners, and Enterprise users can also create organization-scoped runners.

Bencher Cloud On-Demand runners support Linux x86_64 and (coming soon) ARM64, sandboxed in Firecracker. Dedicated and Custom runners can run without the Firecracker sandbox on Linux (x86_64 or ARM64), macOS, and Windows. Contact us for more details.

Bencher Self-Hosted runners can run on Linux (x86_64 or ARM64), macOS, and Windows.

See the bare metal runner reference for details.

Yes. Metrics are billed per result as described above. On Bencher Cloud, Bare Metal runner time is billed separately at $1.00/hr ($0.01666/min) and covers the compute used to execute your benchmarks. You're only billed for the minutes your benchmarks are actively running.

Bencher Self-Hosted users do not pay for runner minutes.

No minimums, no monthly fees.

• On-Demand (Team, Enterprise): billed for the minutes your benchmarks are actively running
• Dedicated (Enterprise): flat monthly rate
• Custom (Enterprise): usage-based

Bencher Self-Hosted users do not pay for runner minutes. Contact us for more details.

Free users get one concurrent Bare Metal job with a 5 minute timeout and Standard queue priority. This is the same for both Bencher Cloud and Bencher Self-Hosted.

For Bencher Cloud On-Demand runners, you're running on the same Bare Metal machines as Team and Enterprise customers, just with lower queue position during busy periods.

Bencher Bare Metal runners usually execute your already-compiled benchmark binaries. Compilation happens locally or in your CI pipeline and is typically the longest step, so most benchmark suites fit well under the 5 minute timeout.

Team and Enterprise jobs are prioritized ahead of Free jobs in the runner queue and share equal priority with each other. All Bencher Cloud On-Demand jobs run on the same Bare Metal hardware. During idle periods Free jobs run immediately. During busy periods paid jobs are scheduled first, and Free jobs wait for the next available slot.