Branches & Start Points with bencher run

A Branch is the git ref (ie branch name or tag). When using the bencher run CLI subcommand, select a Branch with the --branch option or the BENCHER_BRANCH environment variable. The current git hash can also be specified with the --hash option. If a Branch is not specified then the current git branch name is used if it is available. Otherwise, main is used as the default Branch. Similarly, if the git hash is not specified then the current git hash is used if it is available.

When Continuous Benchmarking, that is benchmarking in CI, you will often want to create or rebase a branch on an already existing Branch. This is called a Branch Start Point. A Start Point is another Branch at a specific version (and git hash, if available). When using the bencher run CLI subcommand, select a Start Point with the --start-point option. Additionally, use the --start-point-hash option to specify a specific git hash for the Start Point. The amount of data that is copied over from the Start Point is controlled with the --start-point-max-versions option. Similarly, Thresholds can be copied over from the Start Point with the --start-point-clone-thresholds flag. To prevent drift, use the --start-point-reset flag to always reset the Branch to the Start Point.

--branch <BRANCH>


To select a Branch, set the --branch option or the BENCHER_BRANCH environment variable to the name, slug, or UUID for the Branch. If the value specified is a name or slug and the Branch does not already exist, it will be created for you. However, if the value specified is a UUID then the Branch must already exist. If both are specified, the --branch option takes precedence over the BENCHER_BRANCH environment variable.

If neither are specified, the Bencher CLI tries to find the current git branch. It starts by looking for a git repository in the current working directory. If unsuccessful, it continues to its parent directory and retries all the way up to the root directory. If a git repository is found, then the current short branch name is used. However, if a git repository is not found then main is used as the default Branch.

--hash <HASH>


A 40-character SHA-1 git commit hash. If two reports have the same branch and hash, they will be considered to be from the same commit. Therefore, they will have the same branch version number.

If not provided, the Bencher CLI tries to find the current git hash. It starts by looking for a git repository in the current working directory. If unsuccessful, it continues to its parent directory and retries all the way up to the root directory. If a git repository is found, then the current branch’s HEAD git hash is used.

--start-point <BRANCH>


Use the specified Branch name, slug, or UUID as the Start Point for the branch option. For GitHub this is usually the base branch for a Pull Request, and for GitLab this is usually the target branch for a Merge Request. The primary use case for --start-point is Statistical Continuous Benchmarking.

  • If the specified Start Point is an empty string, then this option is ignored.
  • If branch does not exist, it will be created using this Start Point.
  • If branch does exist and the Start Point is the same, then this option is ignored.
  • If branch does exist and the Start Point is different, a new branch Head will be created using this updated Start Point.
  • If branch does not exist and the Start Point does not exist, then the branch will be created without any historical data.
  • If branch does exist and the Start Point does not exist, then this option is ignored.

If a new branch is created with a Start Point, then the most recent data and optionally Thresholds from the specified Start Point are copied over. See the --start-point-max-versions option and the --start-point-clone-thresholds option for more details.

For example, if feature_branch does not yet exist, but main does exist and has historical data. Then using --branch feature_branch --start-point main would:

  1. Create a new Branch named feature_branch
  2. Copy the most recent data from main to feature_branch
  3. bencher run would proceed using feature_branch

🐰 Note: Whenever using environment variables they should be wrapped in double quotes (ie --start-point "$MY_BRANCH_ENV_VAR"), if you would like bencher run to not error when MY_BRANCH_ENV_VAR is undefined.

--start-point-hash <HASH>


Use the specified full Git hash as the Start Point for the branch option. This option requires the --start-point option to be set.

  • If branch does not exist, it will be created from start-point at the specified hash.
  • If branch does exist and both start-point and the specified hash are the same, then this option is ignored.
  • If branch does exist and either the start-point or the specified hash are different, a new branch Head will be created from start-point at the specified hash.
  • If branch does not exist and start-point does not exist at the specified hash, then the branch will be created from the latest start-point version. If start-point does not exist, then the branch will be created without any historical data.
  • If branch does exist and start-point does not exist at the specified hash, a new branch Head will be created from the latest start-point version. If start-point does not exist, then this option is ignored.

If a new branch is created with a Start Point, then all the most recent data and optionally Thresholds from the start-point at the specified hash are copied over. See the --start-point-max-versions option and the --start-point-clone-thresholds option for more details.

For example, if feature_branch exists with its Start Point being the main Branch at hash 32aea434d751648726097ed3ac760b57107edd8b then using --branch feature_branch --start-point main --start-point-hash 68b12758509ca5e2b1e3983525625e532eb5f562 would:

  1. Remove the old feature_branch Head
  2. Create a new feature_branch Head
  3. Copy of all the most recent data from main at hash 68b12758509ca5e2b1e3983525625e532eb5f562 to the new feature_branch Head
  4. bencher run would then proceed using the new feature_branch Head

--start-point-max-versions <COUNT>


Use the specified maximum number of historical branch versions to include from the Start Point for the branch option. Any versions beyond this number will be omitted. The default is 64 versions. This option requires the --start-point option to be set.

For example, if feature_branch exists with its Start Point being the main Branch and main has 300 versions, then using --branch feature_branch --start-point main --start-point-max-versions 100 would only include the most recent 100 versions of main in feature_branch.

--start-point-clone-thresholds


Clone all of the Thresholds from the Start Point for the branch option. This flag requires the --start-point option to be set.

  • If a Threshold does not exist, it will be created from the start-point Threshold.
  • If a Threshold does exist and it is the same as the start-point Threshold, then that Threshold is effectively ignored.
  • If a Threshold does exist and its model is different from the start-point Threshold, then the old model will be replaced with the start-point Threshold model.
  • If a Threshold does exists and the start-point Threshold does not exist, then the Threshold model is removed.

For example, if feature_branch exists with its Start Point being the main Branch and main has a Threshold for the built-in localhost Testbed and latency Measure, then using --branch feature_branch --start-point main --start-point-clone-thresholds would create a new Threshold for feature_branch with the localhost Testbed and latency Measure.

--start-point-reset


Reset the specified branch. When the --start-point option is set, the branch will be reset to the specified Start Point. This is useful to prevent drift in Statistical Continuous Benchmarking data. The primary use case for --start-point-reset without specifying a Start Point is Relative Continuous Benchmarking.

  • If branch does not exit and no Start Point is specified, the branch will be created without any historical data.
  • If branch does exist and no Start Point is specified, a new branch Head will be created without any historical data.
  • If branch does not exist and a Start Point is specified, the branch will be created using the Start Point.
  • If branch does exist and a Start Point is specified, a new branch Head will be created using the Start Point.

If a new branch is created with a Start Point, then all the most recent data and optionally Thresholds from the start-point at the specified hash are copied over. See the --start-point-max-versions option and the --start-point-clone-thresholds option for more details.

Resets of the same branch with the same idempotency key share one branch Head, so the parallel jobs of a CI run do not reset the branch over each other. When running in GitHub Actions, the key is set for you to the current workflow run attempt. Later steps of the same workflow run that reset the branch also use that Head, so set a different key for a step that needs to reset the branch again.

For example, if feature_branch exists without a Start Point then using --branch feature_branch --start-point-reset would:

  1. Remove the old feature_branch Head
  2. Create a new feature_branch Head with no historical data
  3. bencher run would then proceed using the new feature_branch Head

--start-point-idempotency-key <KEY>


Use the specified key as the idempotency key for the --start-point-reset flag. The key can be any text. Set either the --start-point-idempotency-key option or the BENCHER_START_POINT_IDEMPOTENCY_KEY environment variable. If --start-point-reset is not set or the key is empty, then this option is ignored.

  • If no branch Head has the key, then the reset will create a new branch Head with the key.
  • If a branch Head already has the key, then that Head will be used and the branch will not be reset again, even if the Start Point has changed or that Head has since been replaced.

When running in GitHub Actions, the key defaults to the current workflow run attempt. Every job in a workflow run shares one branch Head, and a re-run of the workflow creates a new Head. Later steps of the same workflow run that reset the branch also use that Head, so set a different key for a step that needs to reset the branch again. For other CI providers, set the key to a value that is the same for every job in a pipeline run (ex: --start-point-idempotency-key "$CI_PIPELINE_ID" in GitLab CI/CD).

For example, if a workflow runs a matrix of three jobs that each use --branch feature_branch --start-point main --start-point-reset --start-point-idempotency-key run-abc123, then:

  1. The first job to report creates a new feature_branch Head from main with the idempotency key run-abc123
  2. The other two jobs use that same feature_branch Head, since it has the idempotency key run-abc123
  3. The results from all three jobs appear together on feature_branch

🐰 Congrats! You have learned all about Branches & Start Points! 🎉


Keep Going: Testbeds & Specs ➡


Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Wed, October 21, 2026 at 10:47:36 PM UTC