How to use Bencher in GitHub Actions
on:
push:
branches: main
jobs:
benchmark_with_bencher:
name: Continuous Benchmarking with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
BENCHER_TESTBED: ubuntu-latest
BENCHER_ADAPTER: json
steps:
- uses: actions/checkout@v3
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--branch main \
--err \
"bencher mock"
- Create a GitHub Actions
workflow
file. (ex:.github/workflows/benchmark.yml
) - Run on
push
events to themain
branch. See the GitHub Actionson
documentation for a full overview. Also see Pull Requests below. - Create a GitHub Actions
job
. (ex:benchmark_with_bencher
) - The Project must already exist. Set the
--project
flag or theBENCHER_PROJECT
environment variable to the Project slug or UUID (ex:BENCHER_PROJECT: save-walter-white
). - The API token must already exist. Add
BENCHER_API_TOKEN
as a Repository secret. (ex:Repo -> Settings -> Secrets and variables -> Actions -> New repository secret
) Set the--token
flag or theBENCHER_API_TOKEN
environment variable to the API token. (ex:BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
) - Optional: Set the
--testbed
flag or theBENCHER_TESTBED
environment variable to the Testbed slug or UUID. (ex:BENCHER_TESTBED: ubuntu-latest
) The Testbed must already exist. If this is not set, then thelocalhost
Testbed will be used. - Optional: Set the
--adapter
flag or theBENCHER_ADAPTER
environment variable to the desired adapter name. (ex:BENCHER_ADAPTER: json
) If this is not set, then themagic
Adapter will be used. See benchmark harness adapters for a full overview. - Checkout your source code. (ex:
uses: actions/checkout@v3
) - Install the Bencher CLI using the GitHub Action. (ex:
uses: bencherdev/bencher@main
) - Track your benchmarks with the
bencher run
CLI subcommand:- Optional: Set the
--branch
flag or theBENCHER_BRANCH
environment variable to the Branch slug or UUID. (ex:--branch main
) The Branch must already exist. If this is not set, then themain
Branch will be used. - Set the command to fail if an Alert is generated. (ex:
--err
) In order for an Alert to be generated, a Threshold must already exist. - Run your benchmarks and generate a Report from the results. (ex:
"bencher mock"
)
- Optional: Set the
Pull Requests
In order to catch performance regression in Pull Requests, you will need to run your benchmarks on PRs.
If you only expect to have PRs from branches within the same repository then you can simply modify the example above to also run on
pull_request
events.
⚠️ This solution only works if all PRs are from the same repository! See Pull Requests from Forks below.
on:
push:
branches: main
pull_request:
jobs:
benchmark_with_bencher:
name: Continuous Benchmarking with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
BENCHER_TESTBED: ubuntu-latest
BENCHER_ADAPTER: json
steps:
- uses: actions/checkout@v3
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--if-branch "$GITHUB_REF_NAME" \
--else-if-branch "$GITHUB_BASE_REF" \
--else-if-branch main \
--github-actions ${{ secrets.GITHUB_TOKEN }} \
--err \
"bencher mock"
- Run on
push
events to themain
branch and onpull_request
events. It is important to limit the run onpush
only to the select branches (ex:main
) to prevents pushes to PR branches from running twice! - Instead of always using the
main
branch, use the GitHub Action default environment variables to:- Use the current branch data if it already exists. (ex:
--if-branch "$GITHUB_REF_NAME"
) - Create a clone of the PR target branch data and thresholds if it already exists. (ex:
--else-if-branch "$GITHUB_BASE_REF"
) - Otherwise, create a clone of the
main
branch data and thresholds. (ex:--else-if-branch main
) - There are several options for setting the project branch. See branch selection for a full overview.
- Use the current branch data if it already exists. (ex:
- Set the GitHub API authentication token. (ex:
--github-actions ${{ secrets.GITHUB_TOKEN }}
) When this option is set as a part of a pull request, then the results will be added to the pull request as a comment. This uses the GitHub ActionsGITHUB_TOKEN
environment variable. - See the bencher run documentation for a full overview of all the ways to configure the pull request comment with the
--ci-*
flags.
Pull Requests from Forks
If you plan to accept pull requests from forks, as is often the case in public open source projects,
then you will need to handle things a little differently.
For security reasons, secrets such as your BENCHER_API_TOKEN
and the GITHUB_TOKEN
are not available in GitHub Actions for fork PRs.
That is if an external contributor opens up a PR from a fork the above example will not work.
There are three options for fork PRs:
- Benchmark Fork PR from Target Branch
- Benchmark Fork PR from Target Branch with Required Reviewers
- Benchmark Fork PR and Upload from Default Branch
Benchmark Fork PR from Target Branch
on:
push:
branches: main
pull_request_target:
jobs:
benchmark_main_with_bencher:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
name: Benchmark main with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
BENCHER_TESTBED: ubuntu-latest
BENCHER_ADAPTER: json
steps:
- uses: actions/checkout@v3
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--branch main \
--err \
"bencher mock"
benchmark_pr_with_bencher:
if: github.event_name == 'pull_request_target'
name: Benchmark PR with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_ADAPTER: json
BENCHER_TESTBED: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
persist-credentials: false
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--if-branch "${{ github.event.pull_request.head.ref }}" \
--else-if-branch "${{ github.event.pull_request.base.ref }}" \
--else-if-branch main \
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
--token "${{ secrets.BENCHER_API_TOKEN }}" \
--err \
"bencher mock"
- Run on
push
events to themain
branch and onpull_request_target
events. - Create a
job
that only runs forpush
events to themain
branch. Other than theif
condition, this job is nearly identical to the original example above. - Create a
job
that only runs forpull_request_target
events.- Checkout the pull request branch.
- Pass in all secrets directly. Use
--token ${{ secrets.BENCHER_API_TOKEN }}
instead of theBENCHER_API_TOKEN
environment variable. - Run and track your pull request benchmarks with
bencher run
.
This setup works because pull_request_target
runs in the context of the pull request’s target branch,
where secrets such as your BENCHER_API_TOKEN
and the GITHUB_TOKEN
are available.
Therefore, this workflow will only run if it exists on the target branch.
Avoid setting any secrets as environment variables, such as BENCHER_API_TOKEN
.
Instead explicitly pass in the API token to bencher run
. (ex: --token ${{ secrets.BENCHER_API_TOKEN }}
)
See this GitHub Security Lab write up
and this blog post
on preventing pwn requests for a full overview.
Benchmark Fork PR from Target Branch with Required Reviewers
on:
push:
branches: main
pull_request_target:
jobs:
benchmark_main_with_bencher:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
name: Benchmark main with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
BENCHER_TESTBED: ubuntu-latest
BENCHER_ADAPTER: json
steps:
- uses: actions/checkout@v3
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--branch main \
--err \
"bencher mock"
benchmark_pr_requires_review:
if: github.event_name == 'pull_request_target'
environment:
${{ (github.event.pull_request.head.repo.full_name == github.repository && 'internal') || 'external' }}
runs-on: ubuntu-latest
steps:
- run: true
benchmark_pr_with_bencher:
needs: benchmark_pr_requires_review
name: Benchmark PR with Bencher
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_ADAPTER: json
BENCHER_TESTBED: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
persist-credentials: false
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--if-branch "${{ github.event.pull_request.head.ref }}" \
--else-if-branch "${{ github.event.pull_request.base.ref }}" \
--else-if-branch main \
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
--token "${{ secrets.BENCHER_API_TOKEN }}" \
--err \
"bencher mock"
This setup is exactly the same as Benchmark Fork PR from Target Branch
with the additional requirement of approval from a Required Reviewer before each fork pull request run.
Pull requests from the same repository do not require approval.
In order to set this up, you need to create two GitHub Action Environments
(ex: Repo -> Settings -> Environments -> New environment
).
The internal
environment should have no Deployment protection rules
.
However, the external
environment should have Required reviewers
set to those trusted to review fork PRs before benchmarking.
Benchmark Fork PR and Upload from Default Branch
name: Run and Cache Benchmarks
on: pull_request
jobs:
benchmark:
name: Run Benchmarks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Mock Benchmark
run: echo '{"bencher::mock_0": { "latency": { "value": 1.0 }}}' &> benchmark_results.txt
- uses: actions/upload-artifact@v3
with:
name: benchmark_results.txt
path: ./benchmark_results.txt
- uses: actions/upload-artifact@v3
with:
name: pr_event.json
path: ${{ env.GITHUB_EVENT_PATH }}
name: Track Benchmarks
on:
workflow_run:
workflows: [Run and Cache Benchmarks]
types:
- completed
jobs:
track_with_bencher:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
env:
BENCHER_PROJECT: save-walter-white
BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
BENCHER_ADAPTER: json
BENCHER_TESTBED: ubuntu-latest
BENCHMARK_RESULTS: benchmark_results.txt
PR_EVENT: pr_event.json
steps:
- name: Download Benchmark Results
uses: actions/github-script@v6
with:
script: |
function downloadArtifact(artifactName) {
let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: context.payload.workflow_run.id,
});
let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => {
return artifact.name == artifactName
})[0];
if (!matchArtifact) {
core.setFailed(`Failed to find artifact: ${artifactName}`);
}
let download = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matchArtifact.id,
archive_format: 'zip',
});
let fs = require('fs');
fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/${artifactName}.zip`, Buffer.from(download.data));
}
downloadArtifact(process.env.BENCHMARK_RESULTS);
downloadArtifact(process.env.PR_EVENT);
- name: Unzip Benchmark Results
run: |
unzip $BENCHMARK_RESULTS.zip
unzip $PR_EVENT.zip
- name: Export PR Context
uses: actions/github-script@v6
with:
script: |
let fs = require('fs');
let prEvent = JSON.parse(fs.readFileSync(process.env.PR_EVENT, {encoding: 'utf8'}));
fs.appendFileSync(process.env.GITHUB_ENV, `PR_NUMBER=${prEvent.number}`);
fs.appendFileSync(process.env.GITHUB_ENV, `PR_HEAD=${prEvent.pull_request.head.ref}`);
fs.appendFileSync(process.env.GITHUB_ENV, `PR_BASE=${prEvent.pull_request.base.ref}`);
- uses: bencherdev/bencher@main
- name: Track Benchmarks with Bencher
run: |
bencher run \
--if-branch "${{ env.PR_HEAD }}" \
--else-if-branch "${{ env.PR_BASE }}" \
--else-if-branch main \
--github-actions "${{ secrets.GITHUB_TOKEN }}" \
--ci-number "${{ env.PR_NUMBER }}" \
--err \
--file $BENCHMARK_RESULTS
- Create a
Run and Cache Benchmarks
workflow file. - Run your benchmarks on
pull_request
events. - Save the benchmarks results to a file and upload them as an artifact.
- Upload the
pull_request
event as an artifact. - Create a second workflow file,
Track Benchmarks
. - Chain
Track Benchmarks
toRun and Cache Benchmarks
with theworkflow_run
event. - Extract necessary data from the cached
pull_request
event. - Track the cached benchmark results with
bencher run
. - Create a third workflow file and use the initial example above to run on
push
events to themain
branch.
This setup works because workflow_run
runs in the context of the repository’s default branch,
where secrets such as your BENCHER_API_TOKEN
and the GITHUB_TOKEN
are available.
Therefore, these workflows will only run if they exist on the default branch.
See using data from the triggering workflow for a full overview.
The pull request number, head branch, and base branch used in the initial workflow must be explicitly passed in since they are not available within workflow_run
.
🐰 Congrats! You have learned how to use Bencher in GitHub Actions! 🎉