How to use Bencher in GitLab CI/CD

Depending on your use case, you can set up Continuous Benchmarking in GitLab CI/CD for your:

Make sure you have created an API token and set it as a masked variable named BENCHER_API_TOKEN before continuing on! Navigate to Your Repo -> Settings -> CI/CD -> Variables -> Expand -> Add variable. The variable key should be BENCHER_API_TOKEN and the variable value should be your API token. Check both the Protect variable and Mask variable boxes.

Target Branch

A cornerstone of Statistical Continuous Benchmarking is having a historical baseline for your target branch. This historical baseline can then be used to detect performance regressions in Merge Requests.

.gitlab-ci.yml
benchmark_target_branch:
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
      when: always
  image: debian:bullseye
  before_script:
    - curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh
  script:
    - |
      bencher run \
      --project save-walter-white-1234abcd \
      --token "$BENCHER_API_TOKEN" \
      --branch main \
      --testbed debian:bullseye \
      --threshold-measure latency \
      --threshold-test t_test \
      --threshold-max-sample-size 64 \
      --threshold-upper-boundary 0.99 \
      --thresholds-reset \
      --err \
      --adapter json \
      bencher mock
  1. Create a GitLab CI/CD file. (ex: .gitlab-ci.yml)
  2. Create a GitLab CI/CD job. (ex: benchmark_target_branch)
  3. Run if the pipeline was triggered by a push to the main branch. See the GitLab CI/CD rules documentation and GitLab CI/CD predefined variables documentation for a full overview. (ex: rules: if: ...)
  4. Set the image the job will run in. See the GitLab CI/CD image documentation for a full overview. (ex: image: debian:bullseye)
  5. Install the Bencher CLI using the convenience script. (ex: before_script: ...)
  6. Use the bencher run CLI subcommand to run your main branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
  7. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
  8. Set the --token option to the masked BENCHER_API_TOKEN environment variable. See the --token docs for more details. (ex: --token "$BENCHER_API_TOKEN")
  9. Set the --branch option to the Branch name. See the --branch docs for a full overview. (ex: --branch main)
  10. Set the --testbed option to the Testbed name. This should likely match the machine selected in image. See the --tested docs for more details. (ex: --testbed debian:bullseye)
  11. Set the Threshold for the main Branch, debian:bullseye Testbed, and latency Measure:
    1. Set the --threshold-measure option to the built-in latency Measure that is generated by bencher mock. See the --threshold-measure docs for more details. (ex: --threshold-measure latency)
    2. Set the --threshold-test option to a Student’s t-test (t_test). See the --threshold-test docs for a full overview. (ex: --threshold-test t_test)
    3. Set the --threshold-max-sample-size option to the maximum sample size of 64. See the --threshold-max-sample-size docs for more details. (ex: --threshold-max-sample-size 64)
    4. Set the --threshold-upper-boundary option to the Upper Boundary of 0.99. See the --threshold-upper-boundary docs for more details. (ex: --threshold-upper-boundary 0.99)
    5. Set the --thresholds-reset flag so that only the specified Threshold is active. See the --thresholds-reset docs for a full overview. (ex: --thresholds-reset)
  12. Set the --err flag to fail the command if an Alert is generated. See the --err docs for a full overview. (ex: --err)
  13. Set the --adapter option to Bencher Metric Format JSON (json) that is generated by bencher mock. See benchmark harness adapters for a full overview. (ex: --adapter json)
  14. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)

Merge Requests

In order to catch performance regression in Merge Requests, you will need to run your benchmarks on MRs. The below example should only be used for branches within the same repository.

.gitlab-ci.yml
benchmark_mr_branch:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always
  image: debian:bullseye
  before_script:
    - curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh
  script:
    - |
      bencher run \
      --project save-walter-white-1234abcd \
      --token "$BENCHER_API_TOKEN" \
      --branch "$CI_COMMIT_REF_NAME" \
      --start-point "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" \
      --start-point-hash "$CI_MERGE_REQUEST_TARGET_BRANCH_SHA" \
      --start-point-clone-thresholds \
      --start-point-reset \
      --testbed debian:bullseye \
      --err \
      --adapter json \
      bencher mock
  1. Update the GitLab CI/CD file. (ex: .gitlab-ci.yml)
  2. Create a GitLab CI/CD job. (ex: benchmark_mr_branch)
  3. Run if the pipeline was triggered by a merge_request_event. See the GitLab CI/CD rules documentation and GitLab CI/CD predefined variables documentation for a full overview. (ex: rules: if: ...)
  4. Set the image the job will run in. See the GitLab CI/CD image documentation for a full overview. (ex: image: debian:bullseye)
  5. Install the Bencher CLI using the convenience script. (ex: before_script: ...)
  6. Use the bencher run CLI subcommand to run your merge request branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
  7. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
  8. Set the --token option to the masked BENCHER_API_TOKEN environment variable. See the --token docs for more details. (ex: --token "$BENCHER_API_TOKEN")
  9. Set the --branch option to the MR branch name using a GitLab CI/CD predefined variable. See the --branch docs for a full overview. (ex: --branch "$CI_COMMIT_REF_NAME")
  10. Set the Start Point for the MR Branch:
    1. Set the --start-point option to the MR Branch start point using a GitLab CI/CD predefined variable. See the --start-point docs for a full overview. (ex: --start-point "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME")
    2. Set the --start-point-hash option to the MR Branch start point git hash using a GitLab CI/CD predefined variable. See the --start-point-hash docs for a full overview. (ex: --start-point-hash "$CI_MERGE_REQUEST_TARGET_BRANCH_SHA")
    3. Set the --start-point-clone-thresholds flag to clone the Thresholds from the start point. See the --start-point-clone-thresholds docs for a full overview. (ex: --start-point-clone-thresholds)
    4. Set the --start-point-reset flag to always reset the MR Branch to the start point. This will prevent benchmark data drift. See the --start-point-reset docs for a full overview. (ex: --start-point-reset)
  11. Set the --testbed option to the Testbed name. This should likely match the machine selected in image. See the --tested docs for more details. (ex: --testbed debian:bullseye)
  12. Set the --err flag to fail the command if an Alert is generated. See the --err docs for a full overview. (ex: --err)
  13. Set the --adapter option to Bencher Metric Format JSON (json) that is generated by bencher mock. See benchmark harness adapters for a full overview. (ex: --adapter json)
  14. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)

To clean up the MR branch after its MR is closed, you can create a separate job that queries for the MR state using the GitLab API. If the state is closed, this job will archive the MR branch using the bencher archive command.

.gitlab-ci.yml
archive_mr_branch:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always
  image: debian:bullseye
  before_script:
    - curl --proto '=https' --tlsv1.2 -sSfL https://bencher.dev/download/install-cli.sh | sh
    - |
      MR_STATE=$(curl --header "PRIVATE-TOKEN: $CI_JOB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID" | jq -r .state)
      echo "Merge request state: $MR_STATE"
  script:
    - |
      if [ "$MR_STATE" = "closed" ]; then
        bencher archive \
        --project save-walter-white-1234abcd \
        --token "$BENCHER_API_TOKEN" \
        --branch "$CI_COMMIT_REF_NAME"
      else
        echo 'Merge request is not `closed`. Skipping archival.'
      fi
  1. Update the GitLab CI/CD file. (ex: .gitlab-ci.yml)
  2. Create a GitLab CI/CD job. (ex: archive_mr_branch)
  3. Run if the pipeline was triggered by a merge_request_event. See the GitLab CI/CD rules documentation and GitLab CI/CD predefined variables documentation for a full overview. (ex: rules: if: ...)
  4. Set the image the job will run in. See the GitLab CI/CD image documentation for a full overview. (ex: image: debian:bullseye)
  5. Install the Bencher CLI using the convenience script. (ex: before_script: curl ...)
  6. Check the MR state using the GitLab API. (ex: before_script: MR_STATE=$(...))
  7. Use the bencher archive CLI subcommand to archive the MR branch if the MR state is closed. (ex: bencher archive)
  8. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
  9. Set the --token option to the masked BENCHER_API_TOKEN environment variable. See the --token docs for more details. (ex: --token "$BENCHER_API_TOKEN")
  10. Set the --branch option to the MR branch name using a GitLab CI/CD predefined variable. (ex: --branch "$CI_COMMIT_REF_NAME")


🐰 Congrats! You have learned how to use Bencher in GitLab CI/CD! 🎉


Keep Going: Benchmarking Overview ➡


Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Sat, October 12, 2024 at 8:22:00 PM UTC