bencher run CLI Subcommand

bencher run is the most popular CLI subcommand. It is used to run benchmarks and report the results. As such, it is one of the most complicated subcommands. This page will explain the options, flags, and arguments that can be passed to bencher run.

bencher run [OPTIONS] [COMMAND] [ARGUMENTS...]

Benchmark Command

The first argument to bencher run is the optional benchmark command. This is the command that will be executed, invoking your benchmark harness. It can also be set using the BENCHER_CMD environment variable. By default this command is executed in a shell, which can be configured with the --shell and --flag options. Its output is parsed by a benchmark harness adapter, which can be set using the --adapter option. However, if the benchmark harness outputs to a file then the --file option must also be used to specify the output file path. Alternatively to track the size of the output file (ie binary size) instead of its contents, use the --file-size option to specify the output file path.

If you would prefer to not have the command executed in a shell, you can use the --exec flag or simply provide additional arguments to your command as additional arguments to bencher run.

Shell Form:

bencher run "bencher mock"

Exec Form:

bencher run bencher mock

The benchmark command can be run multiple times using the --iter option, and those results can be folded into a single result using the --fold option. If any of the iterations fail, then the entire command is considered to have failed unless the --allow-failure flag is set.

If the benchmark command is not specified but the --file option is, then bencher run will just read from output file path instead. Similarly, if the the benchmark command is not specified but the --file-size option is, then bencher run will just read the size of the file at the given file path instead. Both --file and --file-size options can be specified multiple times to read from multiple files.

🐰 Tips for using the -⁠-file option

The -⁠-file option can be used to read benchmark results from a file. You can specify the -⁠-file option multiple times to read from multiple files. If you do specify the -⁠-file option multiple times, then each file will be considered a separate iteration of the benchmark command. When displayed, these results will be shown in separate tables.

If you wish to combined these results into a single iteration, then you can set the -⁠-iter option to 1 and set the -⁠-fold option to either the min, max, or median aggregate functions. This will fold the results into a single iteration and display them in a single table. Do not set the -⁠-fold option to mean.

If neither the benchmark command, --file option, nor --file-size option are specified, then bencher run will read from stdin instead. This allows you to save the output of another command to a file or pipe it into bencher run.

Options

--project <PROJECT>

Optional: Either the --project option or the BENCHER_PROJECT environment variable may be set to the slug or UUID for a Project. A project API key cannot be used to create a new Project or claim an existing Project, so it requires --project/BENCHER_PROJECT to be set to an existing Project. If the value specified is a slug and the Project does not already exist, it will be created for you when a user API key (the --key option), the deprecated --token option, or no credentials at all are used. However, if the value specified is a UUID then the Project must already exist. If both are specified, the --project option takes precedence over the BENCHER_PROJECT environment variable. If neither are specified then a Project slug will be generated for you based on:

1. The name of the parent directory for the git repository, if it is available.
2. The 7 digit hexadecimal short hash for the initial commit of the git repository, if it is available.
3. A 13 digit alphanumeric fingerprint of the local machine, for supported operating systems.

For example, a generated Project slug might look like: project-abc4567-wxyz123456789

When a new Project is created for you, whether the slug is specified or generated, the Project has to belong to an Organization. If the user is authenticated with a user API key (the --key option) or the deprecated --token option, then the Project is added under that user’s personal Organization. If the user is not authenticated, the Project will be created under a new unclaimed Organization. This Organization can then be claimed from the Claim this project link in the bencher run output or by using the Project slug while authenticated in a subsequent invocation of bencher run.

🐰 IMPORTANT: If the CI environment variable is set to true, then you either need to specify the Project or set the --ci-on-the-fly flag.

--ci-on-the-fly

Optional: Allow On-the-Fly Project creation in CI environments. Required if the CI environment variable is set to true. Both GitHub Actions and GitLab CI/CD set this variable by default.

Conflicts with the --project option.

--key <KEY>

Optional: Either the --key option or the BENCHER_API_KEY environment variable may be set to a valid API key. If both are specified, the --key option takes precedence over the BENCHER_API_KEY environment variable. There are two kinds of API key:

• A user API key (bencher_user_*) is scoped to a user and authenticates as that user. Click here to create a user API key.
• A project API key (bencher_run_*) is scoped to a single project and can only be used for bencher run and other non-destructive commands. A project API key cannot be used to create a new Project or claim an existing Project, so it requires the --project option to be set to an existing Project.

If neither the --key option nor the deprecated --token option are specified then the Project must be unclaimed. Conflicts with the deprecated --token option.

--token <TOKEN>

🐰 DEPRECATED: API tokens are deprecated in favor of API keys. New API tokens can no longer be created; existing API tokens continue to work. Use the --key option instead.

Optional: Either the --token option or the BENCHER_API_TOKEN environment variable may be set to a valid API token. If both are specified, the --token option takes precedence over the BENCHER_API_TOKEN environment variable. If neither the --token option nor the --key option are specified then the Project must be unclaimed. That is, if you have already claimed a Project you must provide a valid API key or API token. Conflicts with the --key option.

--image <IMAGE>

--entrypoint <ENTRYPOINT>

--env <KEY=VALUE>

--job-timeout <SECONDS>

--job-poll-interval <SECONDS>

--detach

--callback-url <URL>

--callback-header <NAME: VALUE>

--callback-body <JSON>

--job <UUID>

See Bare Metal images for a full overview.

--branch <BRANCH>

--hash <HASH>

--start-point <BRANCH>

--start-point-hash <HASH>

--start-point-max-versions <COUNT>

--start-point-clone-thresholds

--start-point-reset

--start-point-idempotency-key <KEY>

See branches for a full overview.

--testbed <TESTBED>

--spec <SPEC>

--spec-reset

See testbeds and specs for a full overview.

--threshold-measure <MEASURE>

--threshold-test <TEST>

--threshold-min-sample-size <SAMPLE_SIZE>

--threshold-max-sample-size <SAMPLE_SIZE>

--threshold-window <WINDOW>

--threshold-lower-boundary <BOUNDARY>

--threshold-upper-boundary <BOUNDARY>

--thresholds-reset

--error-on-alert

See thresholds and alerts for a full overview.

--adapter <ADAPTER>

--average <AVERAGE>

--file <FILE>

--build-time

--file-size <FILE>

See benchmark harness adapters for a full overview.

--iter <COUNT>

Optional: Number of run iterations. The default is 1.

--fold <AGGREGATE_FUNCTION>

Optional: Fold multiple results into a single result using an aggregate function.
Requires: --iter to be set.
Possible values:

• min: Minimum value
• max: Maximum value
• mean: Mean of values
• median: Median of values

--backdate <SECONDS>

Optional: Backdate the report (seconds since epoch). NOTE: This will not effect the ordering of past reports! This is useful when initially seeding historical data into a project in chronological order.

--allow-failure

Optional: Allow benchmark test failure.

--format <FORMAT>

Optional: Format for the final Report. The default is human.

Possible values:

• human: Human-readable format
• json: JSON format
• html: HTML format

--quiet

Optional: Quite mode, only output the final Report. Use the --format option to change the output format.

--github-actions <GITHUB_TOKEN>

Optional: Set the GitHub API authentication token. The most convenient way to do this is the GitHub Actions GITHUB_TOKEN environment variable (ie --github-actions ${{ secrets.GITHUB_TOKEN }}).

Results will be added as a GitHub Check to the head commit, named:

1. Bencher Report (<ID>) if the --ci-id option is set
2. Bencher Report (<PROJECT NAME>) for existing projects
3. Bencher Report for --ci-on-the-fly projects (while in-progress)

This requires the token to have the checks scope with write permissions. If the GitHub Check cannot be created, then bencher run will log a warning and continue.

When bencher run is used as a part of a pull request, then the results will also be added to the pull request as a comment. Every job of a Project on the pull request shares this one comment, with one Report row for each job. This requires the token to have the pull-requests scope with write permissions. The results will always be added to the job summary as well. With the --detach flag, the detached run posts no comment and writes no job summary, and the attach completes the GitHub Check it started (see the --ci-callback-token option).

With two or more rows, the comment shows a summary table of each Report with each of its Branch, Testbed, and Adapter that differs between the Reports, and its status (Pending, the number of Alerts, No Alerts, or Failed). The rows are in the order the Reports started, numbered from 1, as the Console lists the Reports of a run. Each row’s label is its number, followed by the value of the --ci-id option in parentheses if it is set (ex: 2 (microbench)). The label links to the Report, a Testbed cell to its page in the Console, and an Adapter cell to its part of the Adapters docs. A Branch, Testbed, or Adapter that every Report shares moves out of the summary table and into the table at the top of the comment, which links it the same way. Next, one table holds the Alerts of every Report, with a Report column. Then the results of each Report follow in a collapsible block (ex: Report 2 (microbench): 14 benchmarks, 2 alerts), which is open when the Report has Alerts. Each job keeps its own GitHub Check and job summary, so GitHub Check names and required status checks do not change. A job whose commit is no longer the head of the pull request leaves the comment as it is. If the token cannot read the pull request, then the job will log a warning and write the comment without checking its head commit.

A GitHub comment has a size limit, so the comment uses the first of these forms that fits:

1. Every part above
2. Without the collapsible blocks, with a note that the comment is truncated
3. Every summary row and the total number of Alerts, without the Alert rows
4. As a last resort, the first summary rows, then and N more reports for the rest

With one row, the comment is the same as before, and if it is too large, it links to the full Report.

To find the other jobs, bencher run lists the GitHub Checks on the head commit, which the checks scope with write permissions above already allows. If a job cannot create or complete its GitHub Check, cannot tell which GitHub App owns it, or cannot list the GitHub Checks, then it will post a comment with only its own results. Comments for the Project from an earlier version of bencher run are replaced: the first job to write the shared comment updates the oldest of them and deletes the rest. When writing the shared comment, a job only updates or deletes comments posted by the GitHub App that owns its GitHub Check, which with the GITHUB_TOKEN is the github-actions bot.

🐰 If you are running inside of a Docker container within GitHub Action, you will need to pass in the following environment variables and mount the path specified by GITHUB_EVENT_PATH:

• GITHUB_ACTIONS
• GITHUB_EVENT_NAME
• GITHUB_EVENT_PATH
• GITHUB_SHA
• GITHUB_API_URL (GitHub Enterprise Server only)

--ci-only-thresholds

Optional: Only post results to CI if a Threshold exists for the Branch, Testbed, and Measure. If no Thresholds exist, then no pull request comment will be posted. Every job of a Project on a pull request shares one comment, so the comment is posted if a Threshold exists for any of its Reports. This only applies to pull request comments. The GitHub Check is always created. Requires: --github-actions

--ci-only-on-alert

Optional: Only start posting results to CI if an Alert is generated. If an Alert is generated, then all follow up results will also be posted even if they don’t contain any Alerts. Every job of a Project on a pull request shares one comment, so the comment is created once any of its Reports has an Alert, and then every job updates it. This only applies to pull request comments. The GitHub Check is always created. Requires: --github-actions

--ci-id <ID>

Optional: Custom ID for posting results to CI. Every job of a Project on a pull request shares one comment, with one Report row for each job. With an ID, the row’s label is its number and the ID (ex: 2 (microbench)). Until the job’s Report exists, its row shows only the ID and no number (ex: (microbench)), as Pending while the job runs, and comes after the numbered rows. Without an ID, the label is the number only, and the row appears once the job’s Report exists. By default, jobs share one row when they have the same combination of: Branch, Testbed, and Adapter, along with --build-time and --file-size if set. Setting a custom ID is useful when Bencher is being run multiple times on the same commit for the same Project, Branch, Testbed, and Adapter combination, so that each run gets its own row. For example, a push run and a pull_request run on the same branch share the head commit, so give each run its own ID, or do not use --github-actions for the push run. Otherwise, the runs share one row, which shows only one of them, and an older run writes nothing once a newer run holds the row. The ID is also used in place of the Project name in the GitHub Check name (ie Bencher Report (<ID>)), so each invocation gets a stable name that can be used as a separate required status check. Requires: --github-actions

--ci-number <NUMBER>

Optional: Issue number for posting results to CI. Bencher will try its best to detect the CI issue number needed to post results. However, this isn’t always available in complex setups, like using workflow_run in GitHub Actions. Requires: --github-actions

--ci-public-links

Optional: All links should be to public URLs that do not require a login. Requires: --github-actions

--ci-callback-token <TOKEN>

Optional: Set the authentication token used by a detached Bare Metal Job callback.

When using the --detach flag and the --github-actions option, the Bencher API server triggers a repository_dispatch workflow when the Job finishes. For a Project with a linked repository, Bencher creates the token itself, so this option is not needed. To use your own token instead, set this option to a fine-grained personal access token for the repository with at least the contents: write permission. Bencher then sends the repository_dispatch event with this token, whether or not the Project has a linked repository. The CLI requires this option on GitHub Enterprise Server and with a Bencher API server older than v0.6.16. Without it, a detached run of a claimed Project is refused if the server has no GitHub App, the Project has no linked repository, or its linked repository is disconnected. See Pull Requests on Bare Metal in GitHub Actions for a full overview.

🐰 You cannot just use the GITHUB_TOKEN here as it expires as soon as the current workflow finishes!

--shell <SHELL>

Optional: Shell command path. Defaults to /bin/sh on Unix-like environments and cmd on Windows.

--flag <FLAG>

Optional: Shell command flag. Defaults to -c on Unix-like environments and /C on Windows.

--exec

Optional: Run command as an executable not a shell command. Default if the number of arguments to bencher run is greater than one.

--host <URL>

Optional: Backend host URL. Defaults to Bench Cloud: https://api.bencher.dev

--insecure-host

Optional: Allow insecure connection to Bencher API server.

This flag is useful when you’re connecting to a Bencher Self-Hosted instance with a self-signed certificate or a certificate that’s not included in the system’s certificate store. It is only applicable to HTTPS connections, as HTTP connections are inherently insecure.

WARNING: Only use --insecure-host in a secure network with verified sources, as it bypasses SSL verification and could expose you to man-in-the-middle attacks.

--native-tls

Optional: Load TLS certificates from the platform’s native certificate store.

By default, bencher loads certificates from the bundled webpki-roots crate. The webpki-roots are a reliable set of trust roots from Mozilla, and including them in bencher improves portability and performance. This is especially true on macOS, where reading the system trust store incurs a significant delay.

However, in some cases, you may want to use the platform’s native certificate store, especially if you’re relying on a corporate trust root that’s included in your system’s certificate store for a mandatory proxy or self-signed Bencher Self-Hosted connections.

--timeout <SECONDS>

Optional: Request timeout in seconds. Defaults to 15 seconds.

--attempts <COUNT>

Optional: Max request retry attempts. Defaults to 35 attempts.

--retry-after <SECONDS>

Optional: Initial seconds to wait between attempts (exponential backoff). Defaults to 1 second.

--max-retry-after <SECONDS>

Optional: Max seconds to wait between attempts (caps exponential backoff). Defaults to 30 seconds.

--dry-run

Optional: Perform a dry run. This will not store any data to the backend. Neither a Report, Branch (as detailed in branches), nor Testbed will be created.

--help

Optional: Print help.

🐰 Congrats! You have learned the basics of bencher run! 🎉

Keep Going: Bare Metal Images ➡