Bencher Changelog

v0.6.10

  • Add a Y-Axis scale toggle to perf plots with three modes (Auto, Linear, and Log); the Auto default preserves the existing adaptive scaling, Log falls back to Auto when the data includes zero or negative values, and the selection is saved in the plot URL, restored for pinned plots, and can be set with bencher plot create --y-axis or changed with bencher plot update --y-axis
  • Selecting a pinned plot from the Plots tab in the Perf explorer now applies the plot’s saved X-Axis and Y-Axis instead of the defaults
  • Thin the Branch Version Number x-axis tick labels to fit the plot width; previously every version rendered a label, which overlapped into an unreadable smear on plots with many versions
  • Add Bencher MCP, a hosted Model Context Protocol server at https://mcp.bencher.dev/mcp (/mcp on Self-Hosted); AI agents and assistants can query benchmark data and submit runs with the same permissions as a project API key (bencher_run_*), without needing the bencher CLI installed

v0.6.9

  • Add Vitest adapter (js_vitest)
  • Add a database.cache_size server config option for the writer connection page cache in KiB (default 64 MiB)
  • Update a pinned plot in place with PATCH /v0/projects/{project}/plots/{plot}, the Perf explorer “Update Pin” button, or bencher plot update
  • Reject plot dimension UUIDs from other projects with a 404 Not Found and empty dimension lists with a 400 Bad Request
  • Add the project name and link as the first row of the PR comment report table (Thank you @OmarTawfik)
  • Create the GitHub Check with an in_progress status before benchmarks run and complete it with the results once they finish, so reruns no longer show a stale conclusion; if bencher run exits with an error before the results are posted, the check is marked as failed (Thank you @OmarTawfik)
  • Add runner update channels via runner up --update-channel <stable|canary> (default: stable); the canary channel tracks a rolling prerelease that is rebuilt whenever the runner changes on a Bencher Cloud deploy, so runner changes are field-tested on Bencher Cloud before a versioned release; canary runners report their binary checksum and converge on the published build by content rather than by version
  • Runner self-update checksum fetch failures no longer drop the WebSocket channel into a reconnect loop; the server now logs the failure, skips the update offer, and retries at the runner’s next ready poll
  • BREAKING CHANGE The reports list endpoint (GET /v0/projects/{project}/reports) now omits the results and alerts for each report by default, drastically reducing response sizes for projects with large reports; set the new expand query param to true (bencher report list --expand) to restore the previous behavior. A new counts field summarizes each report with the number of benchmarks and distinct measures per iteration and the total and active alerts. Note that older CLI versions cannot parse the new default list response, so update the CLI before using bencher report list; bencher run is unaffected

v0.6.8

  • BREAKING CHANGE Change the default self-hosted API server port from 61016 to the newly IANA-registered 6610. Deployments relying on the default now serve the API on 6610. Update clients, reverse proxies, and --host/BENCHER_HOST references (Docker Compose, devcontainer, and docs are updated to match). To stay on 61016, set server.bind_address to 0.0.0.0:61016 (or run bencher up --api-port 61016).
  • Mark the remaining user API token REST endpoints (list, view, update, and revoke) as deprecated in the OpenAPI spec and API docs; they continue to work for existing tokens

v0.6.7

  • Fix the OCI registry token endpoint for the Docker 29+ containerd image store, which sends multiple scope query parameters and an OAuth2 POST token exchange (Thank you @travishathaway)
  • Add user-scoped API keys (bencher_user_...) that authenticate as the owning user across every endpoint a JWT can reach, including docker login; the one exception is key management: a user API key cannot create more keys and can only see, update, or revoke itself, so a leaked key cannot mint credentials that outlive its own revocation, tamper with the user’s other keys, or enumerate them
  • Add bencher user key {create,list,view,update,revoke} CLI subcommands and an “API Keys” page in the Console for managing user API keys
  • --key/BENCHER_API_KEY now accepts either a project-scoped key (bencher_run_*) or a user-scoped key (bencher_user_*); the --project requirement applies only to project-scoped keys
  • DEPRECATED User API tokens are deprecated in favor of user API keys: POST /v0/users/{user}/tokens (bencher token create) now always fails with a 403 Forbidden error; existing API tokens continue to work and can still be listed, viewed, renamed, and revoked
  • Replace the Tokens entry in the Console account dropdown with Keys, and hide the “API Tokens” page entirely for accounts created after 15 June 2026
  • bencher run --github-actions now always creates a Bencher Report GitHub Check (failing on an active Alert), so it can be used as a required status check in branch protection; the check is created on a best effort basis, so check creation failures are now warnings instead of errors (Thank you @OmarTawfik)
  • Create the GitHub Check on the pull request head commit (pull_request.head.sha or workflow_run.head_sha) instead of GITHUB_SHA, so the check appears on the pull request
  • Append the --ci-id ID to the GitHub Check name (ie Bencher Report (<ID>)) so multiple bencher run invocations on the same commit get distinct, stable check names

v0.6.6

  • Show context-aware auth hints in API 404 errors: routes that accept project API keys now mention both API tokens and project keys, while token-only routes mention only API tokens
  • BREAKING CHANGE Rename usage to metrics in the organization usage API response and add runner_minutes field for bare metal runner minutes usage
  • Add --max-retry-after CLI option to cap exponential backoff between retries (default 30s) and increase default --attempts from 10 to 35 for ~15 minutes of retry coverage

v0.6.5

  • Add project scoped API keys
  • Add build time and file size tracking support for bare metal runs
  • Fix bencher run hanging on stdin when no command is provided, including for --image remote bare metal runs (Thank you @sophie-h)

v0.6.4

  • Allow a minimum sample size of 1 for the Percentage (percentage) Threshold Test (Thank you @teofr)
  • BREAKING CHANGE Switch runner key format from hex to alphanumeric encoding, reducing key length from 79 to 45 characters; existing runner keys must be rotated

v0.6.3

  • Fix sample standard deviation for Z-score, t-test, and Log Normal Thresholds to use Bessel’s correction (divide by n − 1) (Thank you @ggirol-rc)
  • Fix t-test Threshold to use the prediction interval (scale the standard deviation by √(1 + 1/n)) so the boundary tests whether a new Metric is consistent with the historical distribution rather than whether it contains the historical mean (Thank you @ggirol-rc)
  • Existing Thresholds will produce slightly wider (statistically correct) boundaries; no configuration changes required
  • Fix Alert JSON to return the Spec for the Report in which the Alert was generated instead of the Testbed’s current Spec
  • BREAKING CHANGE Rename runner token to runner key across the API, CLI, and environment variables
  • Fix race condition where concurrent branch creation could cause a “branch head is null” 500 error (Thank you @vmelamed)
  • Self-heal legacy Branches whose head_id is still NULL from the race condition above so subsequent bencher run submissions succeed instead of 500ing (Thank you @vmelamed)
  • Add API token revocation with bencher token revoke and Revoke button in the Console

v0.6.2

  • Allow unauthenticated docker push to unclaimed projects
  • Fix URL fragment placed before query string per RFC 3986 (Thank you @ricardoboss)
  • Fix --start-point-hash failure on non-existent hash by falling back to the start point branch’s latest version (Thank you @teofr)

v0.6.1

  • Buffer OCI upload network frames into configurable S3 chunks (registry.data_store.chunk_size, default 5 MB) to reduce S3 operations during Docker push
  • Add idempotency_key to bencher run to deduplicate retried submissions
  • Rename --poll-interval to --job-poll-interval for bencher run (--poll-interval is still supported as a deprecated alias)
  • Add --detach flag to bencher run to skip waiting for remote job completion

v0.6.0

  • BREAKING CHANGE Update Bencher Metric Format (BMF) to accept UUID, slug, or name for the Benchmark identifier
  • Move to soft deletion for Organizations and Projects with admin-only hard deletion
  • Add an OCI compliant image registry API
  • Add a bare metal runner
  • BREAKING CHANGE Remove the POST /v0/server/restart endpoint and CLI bencher server restart command
  • BREAKING CHANGE Remove the PUT /v0/server/config endpoint and CLI bencher server config update command
  • BREAKING CHANGE Remove the deprecated --project flag and BENCHER_PROJECT environment variable from CLI bencher threshold create command (use the positional project argument instead)
  • BREAKING CHANGE Remove deprecated support for specifying multiple --start-point values for CLI bencher run (only a single start point is now accepted)
  • BREAKING CHANGE Remove language-level adapter options (rust, cpp, go, java, c_sharp, js, python, ruby, shell, dart) from the CLI and API; use the specific tool adapter (e.g. rust_bench, cpp_google) or magic instead
  • Improve server concurrency with database connection pooling
  • Add [MemoryDiagnoser] support to the C# DotNet Adapter (Thank you @ricardoboss)
  • Add Dart benchmark_harness adapter (dart_benchmark_harness) (Thank you @Louisvranderick)

v0.5.10

  • Fix GitHub PR comment exceeding maximum length (Thank you @alexeyr-ci2)
  • Improve GitHub API error messages
  • Improve developer onboarding docs (Thank you @Xenira)
  • Update GitHub Actions docs (Thank you @pgoslatara)

v0.5.9

  • Add Single Sign-On (SSO) API (Bencher Plus Enterprise feature)

v0.5.8

  • Allow the BenchmarkDotNet namespace to be optional (Thank you @ntransw)
  • Fix Bencher Console for non-Bencher Plus builds (Thank you @flokli)

v0.5.7

  • Fix workspace level feature unification for non-Bencher Plus builds (Thank you @flokli)

v0.5.6

  • Fix Iai-Callgrind adapter parsing bug (Thank you @gamma0987)
  • Add alias for Iai-Callgrind to Gungraun rename, (rust_iai_callgrind -> rust_gungraun)

v0.5.5

  • Add Login with Google as an authentication option
  • Allow the --file option to be specified multiple times to track multiple result files

v0.5.4

  • Fix fork pull request example (Thank you @ruyadorno)
  • Improve database backup to run in the background
  • Improve server stats to run in the background
  • Add branch head version to the Console UI
  • BREAKING CHANGE Creating an on-the-fly project when the CI environment variable is set to true will now require the --ci-on-the-fly flag (Thank you @jaqx0r)

v0.5.3

  • Improve iai-callgrind adapter (Thank you gamma0987)

v0.5.2

  • Fix empty slug validation bug

v0.5.1

  • Fix horizontal wiggle in mobile view of the Console UI
  • Add daily rate limiting to all resource creation
  • Add plus.rate_limiting section to the API server configuration

v0.5.0

  • Allow bencher run to create a project if it does not exist for both anonymous and authenticated users
  • Deprecate the --no-hash flag for bencher run
  • Move to Linux, macOS, and Windows as the default Testbeds for bencher run
  • Add log scaling to Perf Images (Thank you @brianheineman)
  • Add daily rate limiting for Metrics for Organizations without a Bencher Plus plan

v0.4.37

  • Add dimension search in Threshold creation UI
  • Add Threshold list to Branch, Testbed, and Measure pages in Console UI
  • Add unit abbreviation to Perf Report tables in Console UI and PR comments (Thank you @BD103)
  • Add support for GitHub Enterprise Server to bencher CLI (Thank you @btoic)

v0.4.36

  • Fix CORS failures when API server returns an error
  • Catch errors when Console WASM fails to load
  • Add --insecure-host flag to bencher CLI to allow insecure connections to Bencher Self-Hosted API servers
  • Add --native-tls flag to bencher CLI to load TLS certificates from the platform’s native certificate store
  • Add --timeout option to bencher CLI to set the timeout for all API requests
  • Add smtp.insecure_host option to Bencher Self-Hosted API server configuration to allow insecure connections to SMTP servers (Thank you @btoic)

v0.4.35

  • Add “Dismiss All” Alerts button to Console UI
  • Fix infinite progress bar when Perf Plot is not initialized
  • Show collapsed key for Pinned Plots
  • Show selected Branches, Testbeds, and Benchmarks at the top of their Perf Plot tabs
  • Improve Reports table view
  • Show “Recent Reports” table for Branches and Testbeds
  • Use exponential scaling for Perf Plot y-axis when there is a > 10x relative difference (Thank you @brianheineman)
  • Improve new contributor onboarding experience (Thank you @brianheineman)

v0.4.34

  • Add --head option to bencher branch view CLI subcommand
  • Allow selection of up to two Measures in Perf Images
  • Fix the Perf Plot Pinned tab
  • Add --build-time and --file-size to PR comment ID
  • Omit Threshold warning from PR comment when --ci-only-thresholds is used
  • Fix flashing of “No data found” in Perf Plot

v0.4.33

  • Fix plot image y-axis labels
  • Show correctly scaled units in Perf Plot hover box
  • Factor out common dimensions in Perf Plot key
  • Allow selection of up to two Measures in Perf Plots

v0.4.32

  • Fix benchmark result units in PR comments (Thank you @dklassic)

v0.4.31

  • Fix Report measure count bug
  • Auto-scale units in Perf Plots, Perf Images, Reports, and PR comments

v0.4.30

  • Fix Bencher Self-Hosted Console build bug since v0.4.24

v0.4.29

  • Improve Pinned Plots loading fallback
  • Return 404 Not Found for all resource endpoints when checking for both existence and permissions
  • Update CLI install scripts to be able to set the BENCHER_VERSION

v0.4.28

  • Remove lazy loading of embedded perf plots due to Safari bug
  • Move Pinned Plots out of iframe tags
  • Fix custom pinned plot window size bug

v0.4.27

  • Limit GitHub Action Check to 65,535 characters
  • Fix CLI HTML table output bug for Benchmarks with different Measures

v0.4.26

  • Fix GitHub Action summary bug (Thank you @dklassic)

v0.4.25

  • Add GitHub Action summary when using --github-actions option
  • Add ability to track build times with the new --build-time flag

v0.4.24

v0.4.23

  • Fix Console Alert Perf Plot button bug
  • Fix benchmark.js adapter (js_benchmark) float parsing bug (Thank you @arv)
  • Fix Catch2 adapter (cpp_catch2) long name parse bug
  • BREAKING CHANGE The Catch2 adapter (cpp_catch2) now includes the test case name in the benchmark name (ex: Test Case: Benchmark Name)
  • Fix Benchmark results table x-overflow bug when there were several Measures

v0.4.22

  • Remove bloom filter in Perf Query when there is no branch head specified

v0.4.21

  • BREAKING CHANGE Update the Branch JSON to include the concept of a Head which now contains the Start Point
  • Changing a branch’s start point will now update the branch head instead of creating a new branch and archiving the old one
  • Deprecate --branch-start-point, --branch-start-point-hash, and --branch-reset in favor of --start-point, --start-point-hash, and --start-point-reset for bencher run
  • Introduce a max_versions for Branch Start Points, default to 255
  • Add the --start-point-max-versions option to bencher run
  • Thresholds are no longer cloned based off of a branch start point
  • Add the --start-point-clone-thresholds flag to bencher run
  • Add Thresholds to Report JSON
  • Add --threshold-measure, --threshold-test, --threshold-min-sample-size, --threshold-max-sample-size, --threshold-window, --threshold-lower-boundary, --threshold-upper-boundary, and --thresholds-reset to bencher run
  • Fix public plots pages to not require authentication
  • Add --measure option to bencher mock to mock multiple measures
  • Add a bencher archive and bencher unarchive subcommand to archive and unarchive dimensions
  • Fix broken default version tag for bencher up (Thank you @mwasplund)
  • Add --api-port and --console-port options to bencher up to set the API and Console UI ports

v0.4.20

  • Add public pages for Reports, Branches, Testbeds, Benchmarks, Measures, and Plots
  • Add links between all dimension pages
  • BREAKING CHANGE Change the Report results format to mirror Bencher Metric Format, from measures owning benchmarks to benchmarks owning measures
  • Remove user email from Report results
  • Add a table view for Reports in the Console UI
  • Update PR comments to be more concise
  • Remove deprecated statistic field from Threshold in favor of model
  • Remove deprecated server endpoint /v0/server/endpoint in favor of /v0/server/config/console
  • Track global bus events for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.19

  • Add the --format option to bencher run
  • BREAKING CHANGE Remove the Report JSON from the default bencher run output (now use --format json)
  • BREAKING CHANGE Remove the --html flag from bencher run (now use --format html)
  • Standardize all bencher CLI default values
  • Allow --branch-reset to be used with --branch-start-point
  • Update archived branches renaming scheme

v0.4.18

  • Add the concept of archived to Branches, Testbeds, Benchmarks, and Measures
  • Remove Alert stats endpoint
  • Fix + Add API token UI permissions bug
  • Filter for archived resources in the Console UI
  • Archive and unarchive resources in the Console UI
  • Make Start Points for Branches fallible
  • Parse results for multiple tools for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.17

  • Revamp UI for looking at resource details
  • Remove cached Organization when logging out

v0.4.16

  • Track Branches for pinned Plots across changes in their Start Point
  • Update pinned Plots when theme changes
  • Remove 15 second timeout for all CLI commands
  • Show total number of pages for paginated lists in Console UI
  • Add the ability to create and delete Organizations

v0.4.15

  • Fix 404 error for dedicated Metric pages in Console
  • Create Branch and Testbed, if they do not exist for a Report on the API server
  • Remove Branch and Testbed soft creation
  • Add Docker daemon check to CLI commands

v0.4.14

  • Fix iai-callgrind adapter bug
  • Obfuscate user emails in API error messages
  • Include Metric UUID in Report and Perf results
  • Add a GET Metric API endpoint and CLI subcommand
  • Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
  • Enable busy_timeout for SQLite connections to prevent database is locked errors
  • Add replica settings for disaster recovery

v0.4.13

  • Add Required to required query params in docs (Thank you eddyashton)
  • Auto-log out in Console UI for expired sessions
  • Fix wordmark color for default theme initialization
  • Add bencher plot subcommand to manage pinned plots to CLI
  • Add --api-env/--console-env and --api-volume/--console-volume to bencher up CLI command (Thank you ricardaxel)
  • Add service argument to bencher up and bencher down to only control a single container

v0.4.12

  • Fix cross-project query bug

v0.4.11

  • Reduce Console navbar, menu, and panel flickering on page load
  • Fix Add, Edit, Delete button permissions bug (Thank you eddyashton for reporting this bug)
  • Add the ability to “pin” Perf Plots to create a dashboard of pinned plots
  • Add a button to clear each dimension separately in the Perf Plot

v0.4.10

  • Expand --file-size option for bencher run to be used multiple times to track multiple file sizes
  • Add dark mode to Bencher Console UI

v0.4.9

  • Automatically find the git branch name for bencher run if one is not specified
  • Add the --file-size option to bencher run to track file size (ie binary size)
  • Allow custom Measures in Report results to be created on the fly
  • Fix parsing new libtest bench Metrics as floats instead of integers (#390)

v0.4.8

  • Add new user onboarding flow
  • Fix server stats endpoint bug
  • A very special thank you to hamirmahal for their first contribution!

v0.4.7

  • Redirect to public Perf Page for Bencher Console links if user is not logged in
  • Fix jittering in Perf Plot tabs when loading and changing pages
  • Fix PR comment tags when changing branch start points
  • Move benchmark results under a details tag for HTML comment output
  • Add an Alert summary table for HTML comment output
  • Add a warning if no Thresholds are found for HTML comment output
  • Deprecated: --ci-no-metrics flag no longer effects HTML comment output

v0.4.6

  • Add perf plot images as OpenGraph images
  • Add GET /v0/users endpoint for server admins
  • Improve perf plot query performance by 120x or more by adding indexes and a materialized view

v0.4.5

  • Move Bencher Cloud out of BETA
  • Add tab loading progress bar to perf pages
  • Add --branch-start-point and --branch-start-point-hash options to bencher run to track rebasing and other branch changes
  • Deprecated: The bencher run options and flags --if-branch, --else-if-branch, --else-branch, and --endif-branch have been deprecated in favor of using --branch and --branch-start-point
  • BREAKING CHANGE --if-branch will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without --else-branch set
  • BREAKING CHANGE --else-if-branch will only use the first branch specified as the start point and it must exist, if it is a non-empty string
  • Update branch date time plot to use local time and version plot to be discrete along the x-axis

v0.4.4

  • Fix resource contention during large Perf queries
  • Remove public query param from /v0/projects endpoint
  • BREAKING CHANGE Move all CLI commands using --org, --project, and --user over to positional arguments, except bencher run and bencher threshold create
  • Add a bencher report create CLI subcommand
  • bencher run now tries to auto-fill the git commit hash for the report
  • BREAKING CHANGE Rename statistic to model and remove API endpoint
  • Add “Click to view Threshold” for all boundary limits and add a public view Threshold page
  • Add PATCH endpoint for user account settings
  • BREAKING CHANGE Remove /v0/server/ping endpoint
  • Deprecate /v0/server/endpoint in favor of /v0/server/config/console and BREAKING CHANGE remove bencher server endpoint CLI command

v0.4.3

  • BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
  • Add shortcut URLs for user pages:
    • /console/users/tokens -> /console/users/USER-SLUG/tokens
    • /console/users/tokens/add -> /console/users/USER-SLUG/tokens/add
    • /console/users/settings -> /console/users/USER-SLUG/settings
  • Improve bencher run error messages to include the command being run (idea credit to dholroyd: #335)
  • Prevent pwn requests when using bencher run
  • Add an API version check to all Bencher CLI commands that use the backend

v0.4.2

  • Fix Perf Plot view reduce bug
  • Add a View button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
  • Add search query param to GET list endpoints for all named resources
  • Add search for a named resources in Console UI and Perf Plot dimensions
  • Add date based search for Reports in Console UI and Perf Plot
  • Add search for public projects
  • Move Bencher API server to “distroless” Docker image, bencher-api that is 10x smaller
  • Move Bencher Console UI to “distroless” Docker image, bencher-console that is 1/3 smaller
  • Fix Console server side rendering bug when using local URLs inside of Docker
  • First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter

v0.4.1

  • Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
  • Add an --exec flag to bencher run to run commands as executables (no shell) and by default when there is more than one argument
  • Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
  • Include Iteration number for Report results
  • Redirect to newly created resources in Console UI

v0.4.0

  • BREAKING CHANGES Remove backwards compatibility for renames:
    • Metric (lower_bound => lower_value and upper_bound => upper_value)
    • Measure (metric_kind => measure in Report and Threshold; metric_kinds => measures in Perf)
    • Boundary (average => baseline)
  • Add bencher up, bencher down, and bencher logs commands to CLI to easily run Bencher Self-Hosted
  • Branches and Testbeds are now created if they do not exit for bencher run if the value passed is a slug or name (UUIDs are still required to already exist)

v0.3.26

  • Add OAuth2 based login for GitHub
  • Allow users to accept invitations while logged in
  • Run smoke tests on Linux, MacOS, and Windows
  • Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
  • Add convenience script to install bencher CLI on Linux, MacOS, and Windows

v0.3.25

  • Allow custom CI ID to be an unbounded length

v0.3.24

  • Add URL fragment links to all headings 🔗
  • Fix browser navigation to replace history for automatic query args, like pagination
  • Scroll to top of page when changing paginated pages
  • Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (GET list & POST), Threshold (GET list & POST), and Branch (POST start point) endpoints
  • Validate Branch and Testbed exist before running benchmarks with bencher run

v0.3.23

  • Move CLI over to relaxed response parsing and add --strict flag to enable strict response parsing
  • Add --quiet flag to bencher run to enable quite mode and only output the final Report JSON

v0.3.22

  • The Boundary Limits object is now optional for Perf query results and Reports
  • Update all POST API endpoints with creation semantics to return a 201 status code
  • Update all PUT and PATCH API endpoints with immediate update semantics to return a 200 status code
  • Update all DELETE API endpoints to return a 204 status code and an empty body

v0.3.21

  • Add Hyperfine adapter
  • Rename Metric Kind to Measure

v0.3.20

  • Rename average to baseline for the Boundary Limits

v0.3.19

  • Include the average used for the Boundary Limits
  • Remove --ci-public-links for bencher run and simply use the project visibility to determine PR links
  • Replace --ci-with-metrics with --ci-no-metrics for bencher run and make showing benchmark metrics and boundary limits on PRs the default

v0.3.18

  • Add public Alert pages
  • Add a --ci-public-links option to bencher run to only post public links to PRs
  • Add a --ci-with-metrics option to bencher run to display benchmark metrics and boundary limits on PRs
  • Update GitHub Actions PR documentation

v0.3.17

  • Fix word break CSS in console UI
  • Add status bar for perf plot loading
  • Fix perf plot x-axis label
  • Improve embedded perf plot layout

v0.3.16

  • Improve bencher_client ergonomics for use in rustls
  • Fix navbar bug on public pages while logged in
  • Add Bencher Learn section to the site
  • Set max perf query permutations to 255
  • BREAKING CHANGE: Move from single metric_kind to array of metric_kinds for perf plot query

v0.3.15

v0.3.14

  • Move default Threshold Boundaries from 0.9 to 0.95
  • Redirect to login for an authorized users in Console UI
  • Improve API server error messages
  • Fixed the UI y-axis scale issue… again
  • Add Bencher Plus licenses

v0.3.13

  • Add ability for bencher run to read output from a file without running a command argument
  • Improve Report creation error messages
  • Add filtering Reports by Branch, Testbed, and start/end time
  • Add one month historical default when viewing Reports in the Perf Plot
  • Add endpoint /v0/server/spec to view OpenAPI spec JSON
  • Remove old UI source and scripts

v0.3.12

  • Change Metric lower_bound and upper_bound to lower_value and upper_value respectively
  • Add ability to visualize lower_value and upper_value in Perf Plot

v0.3.11

  • Add strongly typed IDs for database entities
  • Remove deprecated configuration keys (endpoint => console.url and secret_key => security.secret_key)

v0.3.10

  • Add optional error monitoring with Sentry
  • Start API error message improvement
  • Fix Threshold dimension selection bug in UI
  • Display benchmark harness results in realtime via bencher run
  • A special thank you to @robertorosmaninho for bug hunting 👾

v0.3.9

  • Move UI over to Astro with stronger TypeScript
  • Add check for failing Netlify deployments
  • Move default Threshold Boundaries from 0.95 to 0.9

v0.3.8

  • Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
  • Add a custom PR comment ID argument --ci-id to bencher run for complex scenarios
  • Another special thank you to @GitGab19 for all the feedback

v0.3.7

  • Fix slug length validation bug
  • A special thank you to @GitGab19 for all the feedback

v0.3.6

  • Fix multi-Metric Kind bug for Perf Plot Report selection
  • Require minimum and maximum sample sizes to be greater than or equal to two
  • Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
  • Comment on GitHub pull requests using --github-actions argument for bencher run
  • Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with ci-only-thresholds
  • Only start posting CI integrations when an Alert is generated with --ci-only-on-alert
  • Fix Threshold boundary calculation bug when standard deviation is zero
  • Add a warning ⚠️ in the Perf Plot UI when a Boundary Limit is not calculated
  • A special thank you to @untitaker for all the feedback

v0.3.5

  • Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
  • Add ability to dismiss and reactivate Alerts
  • Add ability to update Thresholds in the UI
  • Fix Threshold PUT bug
  • Add deletion capabilities to the UI
  • Add help page to public UI

v0.3.4

  • Revamp Quick Start tutorial
  • Add Docker Self-Hosted tutorial
  • Fix public perf pages bug
  • Add Reports tab to perf pages
  • Add ability to clear the perf plot
  • Add ability to set Project URL to null
  • Add Project permissions API endpoint
  • Remove redundant Project API endpoints
  • Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI

v0.3.3

  • Move API server config key endpoint to console.url
  • Change API from /v0/server/config/endpoint to /v0/server/endpoint to get console.url
  • Add dedicated POST API endpoint for Benchmarks
  • Add PATCH API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks
  • Add PUT API endpoint for Thresholds
  • Add DELETE API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds
  • Create two default Thresholds for all new Projects
  • Add a Go benchmark example
  • Fix Go auxillary data parse error

v0.3.2

  • Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
  • Add pagination to all GET list API endpoints, bencher CLI, and console UI
  • Move all bencher CLI API calls to using client auto-generated from OpenAPI spec
  • Improve bencher CLI error messages

v0.3.1

  • Add --backdate argument to bencher run for backdating Reports
  • Add created and modified timestamps as applicable to all entities and their JSON
  • Add ability to delete a Report
  • Add ability to delete a Project
  • A special thank you to @wuputah and the Hydra team for all the feedback

v0.3.0

  • Add demo video to landing page
  • Add bencher run documentation
  • Refactor Report, Alert, and Perf JSON to be more powerful
  • All Report and Perf metrics now show the boundary limits used if a Threshold is set
  • Display URL to visualize report in UI after bencher run
  • Change magnitude to pow for bencher mock
  • A very special thank you to @MichaelScofield for being Bencher’s second community contributor!

v0.2.46

  • Update roadmap
  • Add Iai benchmark harness adapter
  • Move from GitPod over to Dev Containers/GitHub Codespaces
  • Fix thresholds with no historical metrics bug
  • A very special thank you to @osiewicz for being Bencher’s first community contributor!

v0.2.45

  • Add servers to Swagger API docs
  • Fix perf view refresh bug
  • Add an order of --magnitude argument to bencher mock command
  • Add an API endpoint for threshold statistics and a CLI statistic subcommand
  • Upgrade the bencher perf CLI flag --table to accept an optional style parameter
  • Fix long text wrapping in UI deck view, both header and card contents
  • Autogenerate TypeScript types from Rust type definitions
  • Massive thank you to @vient for all the feedback

v0.2.44

  • Fix UI expired token error
  • Move the CLI linker over to use Zig in order to pin a specific glibc version
  • Move linker in CI over to mold for increased parallelism
  • Fix cargo workspace default-features = false warnings
  • Add project stewardship commitments
  • Massive thank you to @brainstorm for all the feedback

v0.2.43

  • Add a --table option to the bencher perf command to display query results as a table
  • Add the ability to toggle the perf plot x-axis from report date time to branch version number

v0.2.42

  • Create and publish Bencher CLI GitHub Action
  • Fix both UI and perf image y-axis sizing once and for all… hopefully

v0.2.41

  • Move perf plot images from a screenshot to server side rendering
  • Add a share mechanism to perf pages with a perf image preview

v0.2.40

  • Add an optional --file argument to bencher run to read file output
  • Move perf query API endpoint from a POST to GET
  • Add an endpoint to GET a screenshot of a public perf plot

v0.2.39

  • Update readme to add Discord invite
  • Allow certain benchmarks to be ignored by Threshold checks
  • BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
  • Add paid plans and metered billing for Bencher Cloud
  • Change perf pages to use milliseconds as opposed to date/time stamp

v0.2.38

  • Fix CLI runtime dependency error over the past five releases
  • BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
  • Add a soft branch creation option to eliminate race conditions in CI
  • Add supported languages to landing page
  • Lots of new docs!

v0.2.33

  • Add new average setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
  • Increase CLI default retry to ten attempts three seconds apart (30 second window)
  • Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for --else-if-branch in bencher run CLI
  • Moved the plot key to below the plot all the time in preparation for screenshots/embedding

v0.2.32

  • Add STARTTLS option for SMTP connections
  • Add throughput as a new default metric kind with operations / second (ops/s) as units
  • Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
  • Add resend email button to UI auth confirm page
  • Change C++ Catch2 adapter to use standard deviation for lower_bound and upper_bound
  • Fix table and perf panel formatting for long, single word values

v0.2.31

  • Prevent leaking secrets by obfuscating them in the server logs (ie ************)
  • Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
  • New harness specific adapter targets (ie rust_bench and rust_criterion)
  • Add additional options to bencher mock, --fail and --flaky to easily test parsing failing and flaky benchmarks respectively
  • Wrote continuous benchmarking and how to install the CLI pages
  • Update server framework, dropshot, and Rust to 1.67.0
  • Two new C++ adapters for Google Benchmark and Catch2
  • Add an adapter for go test -bench

v0.2.30

  • Added --if-else-branch, --else-branch, and --endif-branch flags to to bencher run CLI
  • Wrote a branch management explanation page for using bencher run
  • Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
  • Made the Diataxis convention explicit in the docs
  • Fixed bounds bug in bencher mock
  • Added a /v0/server/backup route and bencher server backup CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy

v0.2.29

  • Fixed and refixed Criterion empty benchmark name bug
  • Type level checking for benchmark names and metric kind resource IDs
  • Added documentation: Quick Start, Prior Art, and GitHub Actions how to
  • Prettier and eslint added to CI/CD
  • Stubbed out UI tests with vitest in CI/CD

v0.2.28

  • Centralized all validation logic between frontend and backend via Rust + WASM modules
  • All pages in the UI now have basic functionality
  • Added public project perf pages

v0.2.27

  • Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
  • Fixed the UI documentation + routing bug! It turns out SwaggerUIBundle was the solution.
  • Added Metric Kind list, add, and view to UI
  • In perf queries allowed for both UUIDs and slugs for branches and testbeds
  • Added bencher.dev to Google Search Console and Analytics
  • Changed API server default URL to always be localhost:3000
  • Moved Docker UI build stage into its own builder.Dockerfile, mirroring the API setup
  • Created a better default for the UI’s backend API URL, basically this-host:61016
  • Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
  • Fixed UI logout + routing “too many redirects” bug
  • Changed notifications from state based to URL query param based in the UI
  • Refactored how pathname and redirects are handled in the UI to increase encapsulation
  • Added the current version to the UI BETA tag
  • Created a repo shortcut at bencher.dev/repo

Published: Fri, October 27, 2023 at 8:40:00 AM UTC | Last Updated: Sat, July 18, 2026 at 7:00:00 AM UTC