Bencher Bare Metal delivers bare metal continuous benchmarking servers as a service (CBaaS). These servers are tuned by performance experts to eliminate noise and get wall-clock time variance to less than 1%! Take your performance engineering to the next level and start catching even subtler performance regressions in CI.
A benchmark result is referred to as a Metric.
A Metric is a single, point-in-time benchmark result. Each time a benchmark is run, it creates at least one new Metric.
For example, if you have 5 benchmarks that measure latency
then they would create 5 Metrics each time they run.
If you ran your benchmarks 10 times,
you would then have 50 Metrics:
5 benchmarks x 10 runs = 50 Metrics
Even if your benchmark harness runs your benchmarks 1,000 times to get a more accurate result,
you would still only have 50 Metrics.
Some benchmarks may create more than one new Metric per run.
For example, if you have 5 benchmarks that each have 2 measures, latency and throughput,
then they would create 10 Metrics each time they run.
If you ran your benchmarks 10 times,
you would then have 100 Metrics:
5 benchmarks x 2 measures x 10 runs = 100 Metrics
On Bencher Cloud, Metrics are billed either:
For example, if you create 5,280 Metrics in a particular calendar
month with metered usage then you would be billed for 5,280 Metrics that month.
Bencher Self-Hosted Metrics are always billed annually based on a licensed quantity.
For example, if you create 5,280 Metrics within a given year,
then you would need to have a licensed quantity of say 6,000 Metrics.
A Public Project is visible to anyone who has access to your Bencher instance.
On Bencher Cloud, all Public Projects are available here . If you are using Bencher Self-Hosted, then anyone with access to your network will likewise be able to see all Public Projects.
The Bencher License is heavily inspired by the GitLab License. You can use Bencher Self-Hosted in development, testing, and production without a Bencher Plus license. This is equivalent to using Bencher Cloud on the Free plan. If you would like access to any of the Bencher Plus features (Team or Enterprise), then you will need to purchase a Bencher Plus license.
Our stewardship promise applies only to the Bencher open source codebase, not to services such as Bencher Cloud. For Bencher Cloud, we may add limits around usage (e.g., storage, compute, traffic) or proxies for usage (e.g., number of users in a project) as we seek to optimize costs and revenues.
Have you ever had a performance regression impact your users? Bencher could have prevented that from happening with continuous benchmarking.