How to use Bencher in GitLab CI/CD

Depending on your use case, you can set up Continuous Benchmarking in GitLab CI/CD for your:

• Target Branch
• Merge Requests

🐰 Make sure you have created an API token and set it as a masked variable named BENCHER_API_TOKEN before continuing on! Navigate to Your Repo -> Settings -> CI/CD -> Variables -> Expand -> Add variable. The variable key should be BENCHER_API_TOKEN and the variable value should be your API token. Check both the Protect variable and Mask variable boxes.

Target Branch

A cornerstone of Statistical Continuous Benchmarking is having a historical baseline for your target branch. This historical baseline can then be used to detect performance regressions in Merge Requests.

benchmark_target_branch:
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      when: always
  image: debian:bullseye
  before_script:
    - curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh
  script:
    - |
      bencher run \
      --project save-walter-white-1234abcd \
      --token "$BENCHER_API_TOKEN" \
      --branch main \
      --testbed debian:bullseye \
      --adapter json \
      --err \
      bencher mock

1. Create a GitLab CI/CD file. (ex: .gitlab-ci.yml)
2. Create a GitLab CI/CD job. (ex: benchmark_target_branch)
3. Run if the pipeline was triggered by a push to the main branch. See the GitLab CI/CD rules documentation and GitLab CI/CD predefined variables documentation for a full overview. (ex: rules: - if: ...)
4. Set the image the job will run in. See the GitLab CI/CD image documentation for a full overview. (ex: image: debian:bullseye)
5. Install the Bencher CLI using the convenience script. (ex: before_script: ...)
6. Use the bencher run CLI subcommand to run your main branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
7. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
8. Set the --token option to the masked BENCHER_API_TOKEN environment variable. See the --token docs for more details. (ex: --token "$BENCHER_API_TOKEN")
9. Set the --branch option to the Branch name. See branch selection for a full overview. (ex: --branch main)
10. Set the --testbed option to the Testbed name. This should likely match the machine selected in image. See the --tested docs for more details. (ex: --testbed debian:bullseye)
11. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)
12. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)
13. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)

Merge Requests

In order to catch performance regression in Merge Requests, you will need to run your benchmarks on MRs. The below example should only be used for branches within the same repository.

benchmark_mr_branch:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always
  image: debian:bullseye
  before_script:
    - curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh
  script:
    - |
      bencher run \
      --project save-walter-white-1234abcd \
      --token "$BENCHER_API_TOKEN" \
      --branch "$CI_COMMIT_REF_NAME" \
      --branch-start-point "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" \
      --branch-start-point-hash "$CI_MERGE_REQUEST_TARGET_BRANCH_SHA" \
      --branch-reset \
      --testbed debian:bullseye \
      --adapter json \
      --err \
      bencher mock

1. Create or update a GitLab CI/CD file. (ex: .gitlab-ci.yml)
2. Create a GitLab CI/CD job. (ex: benchmark_mr_branch)
3. Run if the pipeline was triggered by a merge_request_event. See the GitLab CI/CD rules documentation and GitLab CI/CD predefined variables documentation for a full overview. (ex: rules: - if: ...)
4. Set the image the job will run in. See the GitLab CI/CD image documentation for a full overview. (ex: image: debian:bullseye)
5. Install the Bencher CLI using the convenience script. (ex: before_script: ...)
6. Use the bencher run CLI subcommand to run your merge request branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
7. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
8. Set the --token option to the masked BENCHER_API_TOKEN environment variable. See the --token docs for more details. (ex: --token "$BENCHER_API_TOKEN")
9. Set the --branch option to the MR branch name using a GitLab CI/CD predefined variable. See branch selection for a full overview. (ex: --branch "$CI_COMMIT_REF_NAME")
10. Set the --branch-start-point option to the MR target Branch start point using a GitLab CI/CD predefined variable. See branch selection for a full overview. (ex: --branch-start-point "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME")
11. Set the --branch-start-point-hash option to the MR target Branch start point hash using a GitLab CI/CD predefined variable. See branch selection for a full overview. (ex: --branch-start-point-hash "$CI_MERGE_REQUEST_TARGET_BRANCH_SHA")
12. Set the --branch-reset flag to always reset the Branch to the start point. This will prevent benchmark data drift. See branch selection for a full overview. (ex: --branch-reset)
13. Set the --testbed option to the Testbed name. This should likely match the machine selected in image. See the --tested docs for more details. (ex: --testbed debian:bullseye)
14. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)
15. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)
16. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)

🐰 Congrats! You have learned how to use Bencher in GitLab CI/CD! 🎉

Keep Going: Benchmarking Overview ➡