How to use Bencher in GitHub Actions

Depending on your use case, you can set up Continuous Benchmarking in GitHub Actions for your:

Make sure you have created an API token and set it as a Repository secret named BENCHER_API_TOKEN before continuing on! Navigate to Your Repo -> Settings -> Secrets and variables -> Actions -> New repository secret. Name the secret BENCHER_API_TOKEN and set the secret value to your API token.

In GitHub Actions, secrets are not passed to the runner when a workflow is triggered from a forked repository. Therefore, you will need to use a branch from the same repository when adding any of the workflows below to your repository with a pull request. If you try to add Bencher with a pull request from a fork, then the BENCHER_API_TOKEN secret will not be available. ${{ secrets.BENCHER_API_TOKEN }} will be an empty string.

Base Branch

A cornerstone of Statistical Continuous Benchmarking is having a historical baseline for your base branch. This historical baseline can then be used to detect performance regressions in Pull Requests.

on:
  push:
    branches: main


jobs:
  benchmark_base_branch:
    name: Continuous Benchmarking with Bencher
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bencherdev/bencher@main
      - name: Track base branch benchmarks with Bencher
        run: |
          bencher run \
          --project save-walter-white-1234abcd \
          --token '${{ secrets.BENCHER_API_TOKEN }}' \
          --branch main \
          --testbed ubuntu-latest \
          --adapter json \
          --err \
          bencher mock
  1. Create a GitHub Actions workflow file. (ex: .github/workflows/base_benchmarks.yml)
  2. Run on push events to the main branch. See the GitHub Actions on documentation and GitHub Actions push documentation for a full overview. (ex: on: push: branches: main)
  3. Create a GitHub Actions job. (ex: jobs: benchmark_base_branch)
  4. Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
  5. Checkout your base branch source code. (ex: uses: actions/checkout@v4)
  6. Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)
  7. Use the bencher run CLI subcommand to run your main branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
  8. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
  9. Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')
  10. Set the --branch option to the Branch name. See branch selection for a full overview. (ex: --branch main)
  11. Set the --testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)
  12. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)
  13. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)
  14. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)

Pull Requests

In order to catch performance regression in Pull Requests, you will need to run your benchmarks on PRs. If you only expect to have PRs from branches within the same repository then you can simply create another workflow to run on pull_request events from the same repository.

⚠️ This solution only works if all PRs are from the same repository! See Pull Requests from Forks below.

on:
  pull_request:
    types: [opened, reopened, edited, synchronize]


jobs:
  benchmark_pr_branch:
    name: Continuous Benchmarking PRs with Bencher
    # DO NOT REMOVE: For handling Fork PRs see Pull Requests from Forks
    if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
    permissions:
      pull-requests: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bencherdev/bencher@main
      - name: Track PR Benchmarks with Bencher
        run: |
          bencher run \
          --project save-walter-white-1234abcd \
          --token '${{ secrets.BENCHER_API_TOKEN }}' \
          --branch '${{ github.head_ref }}' \
          --branch-start-point '${{ github.base_ref }}' \
          --branch-start-point-hash '${{ github.event.pull_request.base.sha }}' \
          --branch-reset \
          --testbed ubuntu-latest \
          --adapter json \
          --err \
          --github-actions '${{ secrets.GITHUB_TOKEN }}' \
          bencher mock

  1. Create a GitHub Actions workflow file. (ex: .github/workflows/pr_benchmarks.yml)

  2. Run on pull_request events:

    • opened - A pull request was created.
    • reopened - A previously closed pull request was reopened.
    • edited - The title or body of a pull request was edited, or the base branch of a pull request was changed.
    • synchronize - A pull request’s head branch was updated. For example, the head branch was updated from the base branch or new commits were pushed to the head branch.

    See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [opened, reopened, edited, synchronize])

  3. Create a GitHub Actions job. (ex: jobs: benchmark_pr_branch)

  4. Run on pull_request events if and only if the pull request is from the same repository. ⚠️ DO NOT REMOVE THIS LINE! For handling Fork PRs see Pull Requests from Forks below. (ex: if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository)

  5. Set the permissions for the GITHUB_TOKEN to write for pull-requests. Depending on your GitHub settings, this may not be required. But for all organizations and personal repos created after 02 Feb 2023, this is the default behavior. See the GitHub documentation for a full overview. (ex: permissions: pull-requests: write)

  6. Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)

  7. Checkout the PR branch source code. (ex: uses: actions/checkout@v4)

  8. Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)

  9. Use the bencher run CLI subcommand to run your pull request branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)

  10. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)

  11. Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')

  12. Set the --branch option to the PR branch name using the GitHub Actions github context. See branch selection for a full overview. (ex: --branch '${{ github.head_ref }}')

  13. Set the --branch-start-point option to the PR base Branch start point using the GitHub Actions github context. See branch selection for a full overview. (ex: --branch-start-point '${{ github.base_ref }}')

  14. Set the --branch-start-point-hash option to the PR base Branch start point hash using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch-start-point-hash '${{ github.event.pull_request.base.sha }}')

  15. Set the --branch-reset flag to always reset the Branch to the start point. This will prevent benchmark data drift. See branch selection for a full overview. (ex: --branch-reset)

  16. Set the --testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)

  17. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)

  18. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)

  19. Set the --github-actions option to the GitHub API authentication token to post results as a comment on the Pull Request using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')

  20. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)


Pull Requests from Forks

If you plan to accept pull requests from forks, as is often the case in public open source projects, then you will need to handle things a little differently. For security reasons, secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are not available in GitHub Actions for fork PRs. That is if an external contributor opens up a PR from a fork the above example will not work. There are two options for fork PRs:

See this GitHub Security Lab write up and this blog post on preventing pwn requests for a full overview.

Benchmark Fork PR and Upload from Default Branch

This is the safe and suggested way to add Continuous Benchmarking to fork pull requests. It requires two separate workflows. The first workflow runs and caches the benchmark results in the pull_request context. No secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are available there. Then a second workflow downloads the cached benchmark results in the workflow_run context and uploads them to Bencher. This works because workflow_run runs in the context of the repository’s default branch, where secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are available. The pull request number, head branch, and base branch used in the initial pull_request workflow must also be explicitly passed into the workflow_run workflow since they are not available there. These workflows will only run if they exist on the default branch. See using data from the triggering workflow for a full overview.

name: Run and Cache Benchmarks


on:
  pull_request:
    types: [opened, reopened, edited, synchronize]


jobs:
  benchmark_fork_pr_branch:
    name: Run Fork PR Benchmarks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Mock Benchmarking
        run: |
          /bin/echo '{ "bencher::mock_0": { "latency": { "value": 1.0 } } }' > benchmark_results.json
      - name: Upload Benchmark Results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark_results.json
          path: ./benchmark_results.json
      - name: Upload GitHub Pull Request Event
        uses: actions/upload-artifact@v4
        with:
          name: event.json
          path: ${{ github.event_path }}

  1. Create a first GitHub Actions workflow file. (ex: .github/workflows/run_fork_pr_benchmarks.yml)

  2. Name this workflow so it can be referenced by the second workflow. (ex: name: Run and Cache Benchmarks)

  3. Run on pull_request events:

    • opened - A pull request was created.
    • reopened - A previously closed pull request was reopened.
    • edited - The title or body of a pull request was edited, or the base branch of a pull request was changed.
    • synchronize - A pull request’s head branch was updated. For example, the head branch was updated from the base branch or new commits were pushed to the head branch.

    See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [opened, reopened, edited, synchronize])

  4. Create a GitHub Actions job. (ex: jobs: benchmark_fork_pr_branch)

  5. Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)

  6. Checkout the fork PR branch source code. (ex: uses: actions/checkout@v4)

  7. Run your benchmarks and save the results to a file. (ex: /bin/echo '{ ... }' > benchmark_results.json)

  8. Upload the benchmark results file as an artifact. (ex: uses: actions/upload-artifact@v4)

  9. Upload the pull_request event object as an artifact. (ex: uses: actions/upload-artifact@v4)

name: Track Benchmarks with Bencher


on:
  workflow_run:
    workflows: [Run and Cache Benchmarks]
    types: [completed]


jobs:
  track_fork_pr_branch:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: ubuntu-latest
    env:
      BENCHMARK_RESULTS: benchmark_results.json
      PR_EVENT: event.json
    steps:
      - name: Download Benchmark Results
        uses: actions/github-script@v6
        with:
          script: |
            async function downloadArtifact(artifactName) {
              let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
                owner: context.repo.owner,
                repo: context.repo.repo,
                run_id: context.payload.workflow_run.id,
              });
              let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
                return artifact.name == artifactName
              })[0];
              if (!matchArtifact) {
                core.setFailed(`Failed to find artifact: ${artifactName}`);
              }
              let download = await github.rest.actions.downloadArtifact({
                owner: context.repo.owner,
                repo: context.repo.repo,
                artifact_id: matchArtifact.id,
                archive_format: 'zip',
              });
              let fs = require('fs');
              fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/${artifactName}.zip`, Buffer.from(download.data));
            }
            await downloadArtifact(process.env.BENCHMARK_RESULTS);
            await downloadArtifact(process.env.PR_EVENT);
      - name: Unzip Benchmark Results
        run: |
          unzip $BENCHMARK_RESULTS.zip
          unzip $PR_EVENT.zip
      - name: Export PR Event Data
        uses: actions/github-script@v6
        with:
          script: |
            let fs = require('fs');
            let prEvent = JSON.parse(fs.readFileSync(process.env.PR_EVENT, {encoding: 'utf8'}));
            core.exportVariable("PR_HEAD", `${prEvent.number}/merge`);
            core.exportVariable("PR_BASE", prEvent.pull_request.base.ref);
            core.exportVariable("PR_BASE_SHA", prEvent.pull_request.base.sha);
            core.exportVariable("PR_NUMBER", prEvent.number);
      - uses: bencherdev/bencher@main
      - name: Track Benchmarks with Bencher
        run: |
          bencher run \
          --project save-walter-white-1234abcd \
          --token '${{ secrets.BENCHER_API_TOKEN }}' \
          --branch '${{ env.PR_HEAD }}' \
          --branch-start-point '${{ env.PR_BASE }}' \
          --branch-start-point-hash '${{ env.PR_BASE_SHA }}' \
          --branch-reset \
          --testbed ubuntu-latest \
          --adapter json \
          --err \
          --github-actions '${{ secrets.GITHUB_TOKEN }}' \
          --ci-number '${{ env.PR_NUMBER }}' \
          --file "$BENCHMARK_RESULTS"
  1. Create a first GitHub Actions workflow file. (ex: .github/workflows/track_fork_pr_benchmarks.yml)
  2. Name this workflow second workflow. (ex: name: Track Benchmarks with Bencher)
  3. Chain the two workflows with the workflow_run event. (ex: on: workflow_run: ...)
  4. Create a GitHub Actions job. (ex: jobs: track_fork_pr_branch)
  5. Only run this job if the previous workflow’s conclusion was a success using the GitHub Actions workflow_run event. (ex: if: github.event.workflow_run.conclusion == 'success')
  6. Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)
  7. Set the benchmark results and pull_request event object file names as environment variables. (ex: env: ...)
  8. Download the cached benchmark results and pull_request event. (ex: uses: actions/github-script@v6)
  9. Extract the cached benchmark results and pull_request event. (ex: unzip ...)
  10. Export the necessary data from the pull_request event as environment variables. (ex: core.exportVariable(...))
  11. Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)
  12. Use the bencher run CLI subcommand to track your fork pull branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)
  13. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)
  14. Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')
  15. Set the --branch option to the formatted fork PR number using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch '${{ env.PR_HEAD }}')
  16. Set the --branch-start-point option to the fork PR base Branch start point using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch-start-point '${{ env.PR_BASE }}')
  17. Set the --branch-start-point-hash option to the fork PR base Branch start point hash using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch-start-point-hash '${{ env.PR_BASE_SHA }}')
  18. Set the --branch-reset flag to always reset the Branch to the start point. This will prevent benchmark data drift. See branch selection for a full overview. (ex: --branch-reset)
  19. Set the --testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)
  20. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)
  21. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)
  22. Set the --github-actions option to the GitHub API authentication token to post results as a comment on the Pull Request using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')
  23. Set the --ci-number option to the pull request number. See the --ci-number docs for more details. (ex: --ci-number '${{ env.PR_NUMBER }}')
  24. Set the --file option to the benchmark results file path. See benchmark command for a full overview. (ex: --file "$BENCHMARK_RESULTS")

Benchmark Fork PR from Target Branch with Required Reviewers

In order to guarantee that the code from a fork pull request is safe, this GitHub Action checks to see if the fork is from another repository. If the fork is from another repository, then it will need to be reviewed.

⚠️ It is very, very important to thoroughly review every fork PR before approving! Not doing so could result in a pwn request!

If you would prefer to not have that hanging over your head, see Benchmark Fork PR and Upload from Default Branch above.

In order to get this workflow configured, you need to create two GitHub Actions Environments. Navigate to Your Repo -> Settings -> Environments -> New environment. Create two new environments, internal and external. The internal environment should have no Deployment protection rules. However, the external environment should have Required reviewers set to those trusted to review fork PRs before benchmarking. See this blog post for a full overview.

This setup works because pull_request_target runs in the context of the pull request’s target branch, where secrets such as your BENCHER_API_TOKEN and the GITHUB_TOKEN are available. Therefore, this workflow will only run if it exists on the target branch. Avoid setting any secrets as environment variables, such as GITHUB_TOKEN and BENCHER_API_TOKEN. Instead explicitly pass in your secrets to bencher run.

on:
  pull_request_target:
    types: [opened, reopened, edited, synchronize]


jobs:
  fork_pr_requires_review:
    environment: ${{ (github.event.pull_request.head.repo.full_name == github.repository && 'internal') || 'external' }}
    runs-on: ubuntu-latest
    steps:
      - run: true


  benchmark_fork_pr_branch:
    needs: fork_pr_requires_review
    name: Continuous Benchmarking Fork PRs with Bencher
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          repository: ${{ github.event.pull_request.head.repo.full_name }}
          ref: ${{ github.event.pull_request.head.sha }}
          persist-credentials: false
      - uses: bencherdev/bencher@main
      - name: Track Fork PR Benchmarks with Bencher
        run: |
          bencher run \
          --project save-walter-white-1234abcd \
          --token '${{ secrets.BENCHER_API_TOKEN }}' \
          --branch '${{ github.event.number }}/merge' \
          --branch-start-point '${{ github.base_ref }}' \
          --branch-start-point-hash '${{ github.event.pull_request.base.sha }}' \
          --branch-reset \
          --testbed ubuntu-latest \
          --adapter json \
          --err \
          --github-actions '${{ secrets.GITHUB_TOKEN }}' \
          bencher mock

  1. Create a GitHub Actions workflow file. (ex: .github/workflows/pr_target_benchmarks.yml)

  2. Run on pull_request events:

    • opened - A pull request was created.
    • reopened - A previously closed pull request was reopened.
    • edited - The title or body of a pull request was edited, or the base branch of a pull request was changed.
    • synchronize - A pull request’s head branch was updated. For example, the head branch was updated from the base branch or new commits were pushed to the head branch.

    See the GitHub Actions on documentation and GitHub Actions pull_request documentation for a full overview. (ex: on: pull_request: types: [opened, reopened, edited, synchronize])

  3. Create a first GitHub Actions job to check if the workflow requires review. (ex: jobs: fork_pr_requires_review)

  4. Set the environment to internal if and only if the pull request is from the same repository. Otherwise, set the environment to external, which will require an approval from a reviewer to continue on. ⚠️ DO NOT REMOVE THIS LINE! (ex: environment: ${{ (github.event.pull_request.head.repo.full_name == github.repository && 'internal') || 'external' }})

  5. Create a second GitHub Actions job to run your benchmarks. (ex: benchmark_fork_pr_branch)

  6. Have the benchmark_fork_pr_branch job need the fork_pr_requires_review job in order to run. ⚠️ DO NOT REMOVE THIS LINE! See the GitHub Actions needs documentation for a full overview. (ex: needs: fork_pr_requires_review)

  7. Set the type of machine the job will run on. See the GitHub Actions runs-on documentation for a full overview. (ex: runs-on: ubuntu-latest)

  8. Checkout the fork PR source code. Since pull_request_target runs in the context of the pull request’s target branch, you still need to checkout the pull request branch. (ex: uses: actions/checkout@v4)

    • Specify the fork PR repository (ex: repository: ${{ github.event.pull_request.head.repo.full_name }})
    • Specify the fork PR hash (ex: ref: ${{ github.event.pull_request.head.sha }})
    • Do not persist your git credential (ex: persist-credentials: false)

  9. Install the Bencher CLI using the GitHub Action. (ex: uses: bencherdev/bencher@main)

  10. Use the bencher run CLI subcommand to run your fork pull branch benchmarks. See the bencher run CLI subcommand for a full overview. (ex: bencher run)

  11. Set the --project option to the Project slug. See the --project docs for more details. (ex: --project save-walter-white-1234abcd)

  12. Set the --token option to the BENCHER_API_TOKEN Repository secret. See the --token docs for more details. (ex: --token '${{ secrets.BENCHER_API_TOKEN }}')

  13. Set the --branch option to the formatted fork PR number using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch '${{ github.event.number }}/merge')

  14. Set the --branch-start-point option to the fork PR base Branch start point using the GitHub Actions github context. See branch selection for a full overview. (ex: --branch-start-point '${{ github.base_ref }}')

  15. Set the --branch-start-point-hash option to the fork PR base Branch start point hash using the GitHub Actions pull_request event. See branch selection for a full overview. (ex: --branch-start-point-hash '${{ github.event.pull_request.base.sha }}')

  16. Set the --branch-reset flag to always reset the Branch to the start point. This will prevent benchmark data drift. See branch selection for a full overview. (ex: --branch-reset)

  17. Set the --testbed option to the Testbed name. This should likely match the machine selected in runs-on. See the --tested docs for more details. (ex: --testbed ubuntu-latest)

  18. Set the --adapter option to the desired benchmark harness adapter. See benchmark harness adapters for a full overview. (ex: --adapter json)

  19. Set the --err flag to fail the command if an Alert is generated. See Threshold & Alerts for a full overview. (ex: --err)

  20. Set the --github-actions option to the GitHub API authentication token to post results as a comment on the Pull Request using the GitHub Actions GITHUB_TOKEN environment variable. See the --github-actions docs for more details. (ex: --github-actions '${{ secrets.GITHUB_TOKEN }}')

  21. Specify the benchmark command arguments. See benchmark command for a full overview. (ex: bencher mock)



🐰 Congrats! You have learned how to use Bencher in GitHub Actions! 🎉


Keep Going: Benchmarking Overview ➡


Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Thu, August 22, 2024 at 12:40:00 PM UTC