How to use Bencher to Track Benchmarks

---

The easiest way to track your benchmarks is the bencher run CLI subcommand. See the benchmarking overview for a more in-depth explanation. This is an example of a bencher run CLI subcommand to track benchmarks on a feature branch aptly named feature-branch:

bencher run \
--project save-walter-white \
--token eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhdWQiOiJjbGllbnQiLCJleHAiOjE2NzQwNjA3NTAsImlhdCI6MTY3MTQ2ODc1MCwiaXNzIjoiYmVuY2hlci5kZXYiLCJzdWIiOiJzYXVsQGJldHRlcmNhbGxzYXVsLmNvbSIsIm9yZyI6bnVsbH0.CABcvWlPobAHs7wsdR6wX5p0R2jaCd7RmpsnMp5pwEc \
--adapter json \
--if-branch feature-branch \
--else-if-branch main \
--else-branch \
--testbed phoenix \
--err \
"bencher mock"`

1. The Project must already exist. Set the --project flag or the BENCHER_PROJECT environment variable to the Project slug or UUID. (ex: --project save-walter-white)
2. The API token must already exist. Set the --token flag or the BENCHER_API_TOKEN environment variable to the API token. (ex: --token ...)
3. Optional: Set the --adapter flag or the BENCHER_ADAPTER environment variable to the desired adapter name. If this is not set, then the magic Adapter will be used. See benchmark harness adapters for a full overview. (ex: --adapter json)
4. There are several options for setting the project branch. See branch selection for a full overview.
   1. Use the current branch if it already exists. (ex: --if-branch feature-branch)
   2. Create a clone of target branch data if it already exists. (ex: --else-if-branch main)
   3. Otherwise, create a new branch with the name provided to --if-branch, which would be feature-branch. (ex: --else-branch)
5. Optional: Set the --testbed flag or the BENCHER_TESTBED environment variable to the Testbed slug or UUID. The Testbed must already exist. If this is not set, then the default localhost Testbed will be used. (ex: --testbed phoenix)
6. Set the command to fail if an Alert is generated. In order for an Alert to be generated, a Threshold must already exist. (ex: --err)
7. Run your benchmarks and generate a Report from the results. (ex: "bencher mock")

Relative Benchmarking

Relative benchmarking runs a side-by-side comparison of two commits. This can be useful when dealing with noisy CI/CD environments, where the resource available can highly variable between runs. This is an example of a bencher run CLI subcommand to perform relative benchmarking on a feature branch aptly named feature-branch:

git checkout feature-branch

export FEATURE_BRANCH=feature-branch-$(git rev-parse --short HEAD)

git checkout main

bencher run \
--if-branch "$FEATURE_BRANCH" \
--else-branch \
--iter 3 \
"bencher mock"

git checkout feature-branch

bencher threshold create \
--metric-kind latency \
--branch "$FEATURE_BRANCH" \
--testbed localhost \
--test t \
--right-side 0.95

bencher run \
--if-branch "$FEATURE_BRANCH" \
--iter 3 \
--fold min \
--err \
"bencher mock"

1. Checkout the feature branch. (ex: feature-branch)
2. Create an environment variable that is the name of the feature branch concatenated with the short git commit ID. This is important! It guarantees that for each run a new branch is created.
3. Checkout the target branch. (ex: main)
4. Run bencher run for the target branch:
   1. The given branch will not exist yet. (ex: --if-branch "$FEATURE_BRANCH")
   2. So it will be create. (ex: --else-branch)
   3. Run the benchmarks three times. (ex: --iter 3)
5. Checkout the feature branch. (ex: feature-branch)
6. Create a Threshold for the feature branch:
   1. The Metric Kind for the benchmarks is Latency. (ex: --metric-kind latency)
   2. The Branch is the feature branch with the appended git commit ID. (ex: --branch "$FEATURE_BRANCH")
   3. The Testbed is running locally. (ex: --testbed localhost)
   4. There are less than 30 metrics, use a Student’s t-test. (ex: --test t)
   5. Set a right side boundary of 95.0% because a larger Latency indicates a performance regression. (ex: --right-side 0.95)
7. Run bencher run for the feature branch:
   1. The Branch will exist since it was just created. (ex: --if-branch "$FEATURE_BRANCH")
   2. Run the tests three times. (ex: --iter 3)
   3. Fold all three Metrics into the minimum value. (ex: --fold min)
   4. Set the command to fail if an Alert is generated by the Threshold. (ex: --err)

🐰 Congrats! You have learned how to use Bencher to track benchmarks! 🎉

Add Bencher to GitHub Actions ➡

Add Bencher to GitLab CI/CD ➡