How to track Rust binary sizes in CI

Everett Pompeii

Binary sizes can get bloated over time, especially in Rust. Continuous Benchmarking is the practice of running benchmarks on every changeset to ensure the changes do not introduce a performance regressions. You can treat binary sizes as a benchmark and track them in CI to catch large binary sizes. The easiest way to implement Continuous Benchmarking for Rust binary sizes is to use Bencher.

What is Bencher?

Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they make it to production.

• Run: Run your benchmarks locally or in CI using your favorite benchmarking tools. The bencher CLI simply wraps your existing benchmark harness and stores its results.
• Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
• Catch: Catch performance regressions in CI. Bencher uses state of the art, customizable analytics to detect performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should be run in CI with Bencher to prevent performance regressions. Performance bugs are bugs!

Steps for Bencher Cloud

1. Create a Bencher Cloud account.
2. Create an API token and add it to your CI as a secret.
3. Create a workflow for your CI, like GitHub Actions or GitLab CI/CD.
4. Install the Bencher CLI in your CI workflow.
5. Compile your Rust code with the bencher run subcommand in your CI workflow using the --file-size option set to the output path for your binary along with the json adapter.

Steps for Bencher Self-Hosted

1. Create a Bencher Self-Hosted instance.
2. Create an account on your Bencher Self-Hosted instance.
3. Create an API token and add it to your CI as a secret.
4. Create a workflow for your CI, like GitHub Actions or GitLab CI/CD.
5. Install the Bencher CLI in your CI workflow. Make sure the CLI version matches the version of your Bencher Self-Hosted instance.
6. Compile your Rust code with the bencher run subcommand in your CI workflow using the --file-size option set to the output path for your binary along with the json adapter and setting the --host option to your Bencher Self-Hosted instance URL.

⚖️ File Size

The bencher run CLI subcommand can be used to track the file size (ie binary size) of your deliverables with the --file-size option. The --file-size option expects a file path to the file who’s size will be measured. Under the hood, bencher run outputs the results as Bencher Metric Format (BMF) JSON. It is therefore good practice to explicitly use the json adapter. For more details see how to track file size.

The file-size Measure (ie bytes (B)) is gathered. Only the file size value (ie value) is available. Neither lower_value nor upper_value are collected. The file-size Measure is not created by default for all Projects. However, when you use the --file-size option, this Measure will be automatically created for your Project. The --file-size option can be used multiple times to track multiple file sizes.

bencher run --file-size ./target/release/example --adapter json "cargo build --release"