How to use Bencher to Track Benchmarks
The easiest way to track your benchmarks is the bencher run
CLI subcommand.
See the benchmarking overview for a more in-depth explanation.
This is an example of a bencher run
CLI subcommand to track benchmarks on a feature branch aptly named feature-branch
:
bencher run \
--project save-walter-white \
--token eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhdWQiOiJjbGllbnQiLCJleHAiOjE2NzQwNjA3NTAsImlhdCI6MTY3MTQ2ODc1MCwiaXNzIjoiYmVuY2hlci5kZXYiLCJzdWIiOiJzYXVsQGJldHRlcmNhbGxzYXVsLmNvbSIsIm9yZyI6bnVsbH0.CABcvWlPobAHs7wsdR6wX5p0R2jaCd7RmpsnMp5pwEc \
--adapter json \
--if-branch feature-branch \
--else-if-branch main \
--else-branch \
--testbed phoenix \
--err \
"bencher mock"`
- The Project must already exist. Set the
--project
flag or theBENCHER_PROJECT
environment variable to the Project slug or UUID. (ex:--project save-walter-white
) - The API token must already exist. Set the
--token
flag or theBENCHER_API_TOKEN
environment variable to the API token. (ex:--token ...
) - Optional: Set the
--adapter
flag or theBENCHER_ADAPTER
environment variable to the desired adapter name. If this is not set, then themagic
Adapter will be used. See benchmark harness adapters for a full overview. (ex:--adapter json
) - There are several options for setting the project branch. See branch selection for a full overview.
- Use the current branch if it already exists. (ex:
--if-branch feature-branch
) - Create a clone of target branch data if it already exists. (ex:
--else-if-branch main
) - Otherwise, create a new branch with the name provided to
--if-branch
, which would befeature-branch
. (ex:--else-branch
)
- Use the current branch if it already exists. (ex:
- Optional: Set the
--testbed
flag or theBENCHER_TESTBED
environment variable to the Testbed slug or UUID. The Testbed must already exist. If this is not set, then the defaultlocalhost
Testbed will be used. (ex:--testbed phoenix
) - Set the command to fail if an Alert is generated. In order for an Alert to be generated, a Threshold must already exist. (ex:
--err
) - Run your benchmarks and generate a Report from the results. (ex:
"bencher mock"
)
Relative Benchmarking
Relative benchmarking runs a side-by-side comparison of two commits.
This can be useful when dealing with noisy CI/CD environments,
where the resource available can highly variable between runs.
This is an example of a bencher run
CLI subcommand to perform relative benchmarking on a feature branch aptly named feature-branch
:
git checkout feature-branch
export FEATURE_BRANCH=feature-branch-$(git rev-parse --short HEAD)
git checkout main
bencher run \
--if-branch "$FEATURE_BRANCH" \
--else-branch \
--iter 3 \
"bencher mock"
git checkout feature-branch
bencher threshold create \
--metric-kind latency \
--branch "$FEATURE_BRANCH" \
--testbed localhost \
--test t \
--right-side 0.95
bencher run \
--if-branch "$FEATURE_BRANCH" \
--iter 3 \
--fold min \
--err \
"bencher mock"
- Checkout the feature branch. (ex:
feature-branch
) - Create an environment variable that is the name of the feature branch concatenated with the short git commit ID. This is important! It guarantees that for each run a new branch is created.
- Checkout the target branch. (ex:
main
) - Run
bencher run
for the target branch:- The given branch will not exist yet. (ex:
--if-branch "$FEATURE_BRANCH"
) - So it will be create. (ex:
--else-branch
) - Run the benchmarks three times. (ex:
--iter 3
)
- The given branch will not exist yet. (ex:
- Checkout the feature branch. (ex:
feature-branch
) - Create a Threshold for the feature branch:
- The Metric Kind for the benchmarks is Latency. (ex:
--metric-kind latency
) - The Branch is the feature branch with the appended git commit ID. (ex:
--branch "$FEATURE_BRANCH"
) - The Testbed is running locally. (ex:
--testbed localhost
) - There are less than 30 metrics, use a Studentβs t-test. (ex:
--test t
) - Set a right side boundary of 95.0% because a larger Latency indicates a performance regression. (ex:
--right-side 0.95
)
- The Metric Kind for the benchmarks is Latency. (ex:
- Run
bencher run
for the feature branch:- The Branch will exist since it was just created. (ex:
--if-branch "$FEATURE_BRANCH"
) - Run the tests three times. (ex:
--iter 3
) - Fold all three Metrics into the minimum value. (ex:
--fold min
) - Set the command to fail if an Alert is generated by the Threshold. (ex:
--err
)
- The Branch will exist since it was just created. (ex:
π° Congrats! You have learned how to use Bencher to track benchmarks! π