如何使用 Bencher 跟踪自定义基准

Bencher supports the most popular benchmarking harnesses out-of-the-box, and we are always open to suggestions for new adapters. However there can be situations where an off-the-shelf benchmarking harness doesn’t fit your needs, necessitating the creation of a custom benchmarking harness. Lucky for you, Bencher also supports using a custom benchmarking harness. The easiest way to integrate your custom benchmark harness with Bencher is to output Bencher Metric Format (BMF) JSON.

This is an example of BMF JSON:

{
  "benchmark_name": {
    "latency": {
      "value": 88.0,
      "lower_value": 87.42,
      "upper_value": 88.88
    }
  }
}

In this example, the key benchmark_name would be the name of a Benchmark. Benchmark names can be any non-empty string up to 1024 characters. The benchmark_name object can contain multiple Measure names, slugs, or UUIDs as keys. If the value specified is a name or slug and the Measure does not already exist, it will be created for you. However, if the value specified is a UUID then the Measure must already exist. In this example, latency is the slug for the built-in Latency Measure. Each Project by default has a Latency (ie latency) and Throughput (ie throughput) Measure, which are measured in nanosecond (ns) and operations / second (ops/s) respectively. The Measure object contains a Metric with up to three values: value, lower_value, and upper_value. The lower_value and upper_value values are optional.

In this example, the latency Measure object contains the following values:

• A value of 88.0
• A lower_value of 87.42
• An upper_value of 88.88

You can use the bencher mock CLI subcommand to generate mock BMF data. We will use it as a placeholder for your own custom benchmark runner. Using bencher run and the json adapter we can track our benchmarks with the following command:

bencher run --adapter json "bencher mock"

If your results were instead stored in a file named results.json, then you could use the --file option to specify the file path. This works both with a benchmark command and without one.

With a benchmark command:

bencher run --file results.json --adapter json "bencher mock > results.json"

Without a benchmark command:

bencher mock > results.json && bencher run --file results.json --adapter json

多个度量

在 Bencher Metric Format (BMF) JSON 中，基准测试 对象可以包含多个 度量 名称、缩略名或 UUID 作为键。 如果指定的值是名称或缩略名且该度量尚不存在，它将会为你创建。 然而，如果指定的值是 UUID，则该度量必须已经存在。 每个度量对象必须包含一个 度量指标，最多可包含三个值： value，lower_value 和 upper_value。 lower_value 和 upper_value 是可选的。

这是具有多个度量的 BMF JSON 示例：

{
  "benchmark_name": {
    "latency": {
      "value": 88.0,
      "lower_value": 87.42,
      "upper_value": 88.88
    },
    "throughput" {
      "value": 5.55,
      "lower_value": 3.14,
      "upper_value": 6.30
    }
  }
}

在此示例中，latency 度量对象包含以下值：

• value 为 88.0
• lower_value 为 87.42
• upper_value 为 88.88

而 throughput 度量对象包含以下值：

• value 为 5.55
• lower_value 为 3.14
• upper_value 为 6.30

你可以使用 bencher mock CLI 子命令 以及 --measure 选项 来生成具有多个度量的模拟 BMF 数据。 我们将其用作你自己的自定义基准测试运行器的占位符。 使用 bencher run 和 json 适配器 我们可以使用以下命令跟踪具有多个度量的基准测试：

bencher run --adapter json "bencher mock --measure latency --measure throughput"

🐰 恭喜！你已经学会了如何跟踪自定义基准！🎉

继续学习：如何在 CI 中跟踪基准 ➡