How to track Python pytest-benchmark benchmarks in CI

Everett Pompeii

Now that you have learned how to benchmark Python code with pytest-benchmark, let’s see how to track those benchmarks in CI. Continuous Benchmarking is the practice of running benchmarks on every changeset to ensure the changes do not introduce a performance regressions. The easiest way to implement Continuous Benchmarking with pytest-benchmark benchmarks is to use Bencher.

What is Bencher?

Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they make it to production.

• Run: Run your benchmarks locally or in CI using your favorite benchmarking tools. The bencher CLI simply wraps your existing benchmark harness and stores its results.
• Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
• Catch: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!

Steps for Bencher Cloud

1. Create a Bencher Cloud account.
2. Create an API token and add it to your CI as a secret.
3. Create a workflow for your CI, like GitHub Actions or GitLab CI/CD.
4. Install the Bencher CLI in your CI workflow.
5. Run your benchmarks with the bencher run subcommand in your CI workflow using the python_pytest adapter.

Steps for Bencher Self-Hosted

1. Create a Bencher Self-Hosted instance.
2. Create an account on your Bencher Self-Hosted instance.
3. Create an API token and add it to your CI as a secret.
4. Create a workflow for your CI, like GitHub Actions or GitLab CI/CD.
5. Install the Bencher CLI in your CI workflow. Make sure the CLI version matches the version of your Bencher Self-Hosted instance.
6. Run your benchmarks with the bencher run subcommand in your CI workflow using the python_pytest adapter and setting the --host option to your Bencher Self-Hosted instance URL.

🐍 Python Pytest

The Python Pytest Adapter (python_pytest) expects pytest-benchmark output in JSON format (ie --benchmark-json results.json). This JSON output is saved to a file, so you must use the bencher run CLI --file option to specify that file path. The latency Measure (ie nanoseconds (ns)) is gathered.

bencher run --adapter python_pytest --file results.json "pytest --benchmark-json results.json benchmarks.py"

There are two options for the Metric:

• mean (default): The lower_value and upper_value are one standard deviation below and above the mean (ie value) respectively.
• median: The lower_value and upper_value are one interquartile range below and above the median (ie value) respectively.

This can be specified in the bencher run CLI subcommand with the --average option.