How to use Bencher to Track Benchmarks


The easiest way to track your benchmarks is the bencher run CLI subcommand. See the benchmarking overview for a more in-depth explanation. This is an example of a bencher run CLI subcommand to track benchmarks on a feature branch aptly named feature-branch:

bencher run \
--project save-walter-white \
--token eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhdWQiOiJjbGllbnQiLCJleHAiOjE2NzQwNjA3NTAsImlhdCI6MTY3MTQ2ODc1MCwiaXNzIjoiYmVuY2hlci5kZXYiLCJzdWIiOiJzYXVsQGJldHRlcmNhbGxzYXVsLmNvbSIsIm9yZyI6bnVsbH0.CABcvWlPobAHs7wsdR6wX5p0R2jaCd7RmpsnMp5pwEc \
--adapter json \
--if-branch feature-branch \
--else-if-branch main \
--else-branch \
--testbed phoenix \
--err \
"bencher mock"`
  1. The Project must already exist. Set the --project flag or the BENCHER_PROJECT environment variable to the Project slug or UUID. (ex: --project save-walter-white)
  2. The API token must already exist. Set the --token flag or the BENCHER_API_TOKEN environment variable to the API token. (ex: --token ...)
  3. Optional: Set the --adapter flag or the BENCHER_ADAPTER environment variable to the desired adapter name. If this is not set, then the magic Adapter will be used. See benchmark harness adapters for a full overview. (ex: --adapter json)
  4. There are several options for setting the project branch. See branch selection for a full overview.
    1. Use the current branch if it already exists. (ex: --if-branch feature-branch)
    2. Create a clone of target branch data if it already exists. (ex: --else-if-branch main)
    3. Otherwise, create a new branch with the name provided to --if-branch, which would be feature-branch. (ex: --else-branch)
  5. Optional: Set the --testbed flag or the BENCHER_TESTBED environment variable to the Testbed slug or UUID. The Testbed must already exist. If this is not set, then the default localhost Testbed will be used. (ex: --testbed phoenix)
  6. Set the command to fail if an Alert is generated. In order for an Alert to be generated, a Threshold must already exist. (ex: --err)
  7. Run your benchmarks and generate a Report from the results. (ex: "bencher mock")

Relative Benchmarking

Relative benchmarking runs a side-by-side comparison of two commits. This can be useful when dealing with noisy CI/CD environments, where the resource available can highly variable between runs. This is an example of a bencher run CLI subcommand to perform relative benchmarking on a feature branch aptly named feature-branch:

git checkout feature-branch

export FEATURE_BRANCH=feature-branch-$(git rev-parse --short HEAD)

git checkout main

bencher run \
--if-branch "$FEATURE_BRANCH" \
--else-branch \
--iter 3 \
"bencher mock"

git checkout feature-branch

bencher threshold create \
--metric-kind latency \
--branch "$FEATURE_BRANCH" \
--testbed localhost \
--test t \
--right-side 0.95

bencher run \
--if-branch "$FEATURE_BRANCH" \
--iter 3 \
--fold min \
--err \
"bencher mock"
  1. Checkout the feature branch. (ex: feature-branch)
  2. Create an environment variable that is the name of the feature branch concatenated with the short git commit ID. This is important! It guarantees that for each run a new branch is created.
  3. Checkout the target branch. (ex: main)
  4. Run bencher run for the target branch:
    1. The given branch will not exist yet. (ex: --if-branch "$FEATURE_BRANCH")
    2. So it will be create. (ex: --else-branch)
    3. Run the benchmarks three times. (ex: --iter 3)
  5. Checkout the feature branch. (ex: feature-branch)
  6. Create a Threshold for the feature branch:
    1. The Metric Kind for the benchmarks is Latency. (ex: --metric-kind latency)
    2. The Branch is the feature branch with the appended git commit ID. (ex: --branch "$FEATURE_BRANCH")
    3. The Testbed is running locally. (ex: --testbed localhost)
    4. There are less than 30 metrics, use a Student’s t-test. (ex: --test t)
    5. Set a right side boundary of 95.0% because a larger Latency indicates a performance regression. (ex: --right-side 0.95)
  7. Run bencher run for the feature branch:
    1. The Branch will exist since it was just created. (ex: --if-branch "$FEATURE_BRANCH")
    2. Run the tests three times. (ex: --iter 3)
    3. Fold all three Metrics into the minimum value. (ex: --fold min)
    4. Set the command to fail if an Alert is generated by the Threshold. (ex: --err)


🐰 Congrats! You have learned how to use Bencher to track benchmarks! πŸŽ‰


Add Bencher to GitHub Actions ➑

Add Bencher to GitLab CI/CD ➑