Bencher Changelog

---

Pending v0.4.33

• Fix plot image y-axis labels

v0.4.32

• Fix benchmark result units in PR comments (Thank you @dklassic)

v0.4.31

• Fix Report measure count bug
• Auto-scale units in Perf Plots, Perf Images, Reports, and PR comments

v0.4.30

• Fix Bencher Self-Hosted Console build bug since v0.4.24

v0.4.29

• Improve Pinned Plots loading fallback
• Return 404 Not Found for all resource endpoints when checking for both existence and permissions
• Update CLI install scripts to be able to set the BENCHER_VERSION

v0.4.28

• Remove lazy loading of embedded perf plots due to Safari bug
• Move Pinned Plots out of iframe tags
• Fix custom pinned plot window size bug

v0.4.27

• Limit GitHub Action Check to 65,535 characters
• Fix CLI HTML table output bug for Benchmarks with different Measures

v0.4.26

• Fix GitHub Action summary bug (Thank you @dklassic)

v0.4.25

• Add GitHub Action summary when using --github-actions option
• Add ability to track build times with the new --build-time flag

v0.4.24

• Add GitHub Action Check when not running as a pull request (Thank you @sagudev, @dklassic, and @wusyong)

v0.4.23

• Fix Console Alert Perf Plot button bug
• Fix benchmark.js adapter (js_benchmark) float parsing bug (Thank you @arv)
• Fix Catch2 adapter (cpp_catch2) long name parse bug
• BREAKING CHANGE The Catch2 adapter (cpp_catch2) now includes the test case name in the benchmark name (ex: Test Case: Benchmark Name)
• Fix Benchmark results table x-overflow bug when there were several Measures

v0.4.22

• Remove bloom filter in Perf Query when there is no branch head specified

v0.4.21

• BREAKING CHANGE Update the Branch JSON to include the concept of a Head which now contains the Start Point
• Changing a branch’s start point will now update the branch head instead of creating a new branch and archiving the old one
• Deprecate --branch-start-point, --branch-start-point-hash, and --branch-reset in favor of --start-point, --start-point-hash, and --start-point-reset for bencher run
• Introduce a max_versions for Branch Start Points, default to 255
• Add the --start-point-max-versions option to bencher run
• Thresholds are no longer cloned based off of a branch start point
• Add the --start-point-clone-thresholds flag to bencher run
• Add Thresholds to Report JSON
• Add --threshold-measure, --threshold-test, --threshold-min-sample-size, --threshold-max-sample-size, --threshold-window, --threshold-lower-boundary, --threshold-upper-boundary, and --thresholds-reset to bencher run
• Fix public plots pages to not require authentication
• Add --measure option to bencher mock to mock multiple measures
• Add a bencher archive and bencher unarchive subcommand to archive and unarchive dimensions
• Fix broken default version tag for bencher up (Thank you @mwasplund)
• Add --api-port and --console-port options to bencher up to set the API and Console UI ports

v0.4.20

• Add public pages for Reports, Branches, Testbeds, Benchmarks, Measures, and Plots
• Add links between all dimension pages
• BREAKING CHANGE Change the Report results format to mirror Bencher Metric Format, from measures owning benchmarks to benchmarks owning measures
• Remove user email from Report results
• Add a table view for Reports in the Console UI
• Update PR comments to be more concise
• Remove deprecated statistic field from Threshold in favor of model
• Remove deprecated server endpoint /v0/server/endpoint in favor of /v0/server/config/console
• Track global bus events for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.19

• Add the --format option to bencher run
• BREAKING CHANGE Remove the Report JSON from the default bencher run output (now use --format json)
• BREAKING CHANGE Remove the --html flag from bencher run (now use --format html)
• Standardize all bencher CLI default values
• Allow --branch-reset to be used with --branch-start-point
• Update archived branches renaming scheme

v0.4.18

• Add the concept of archived to Branches, Testbeds, Benchmarks, and Measures
• Remove Alert stats endpoint
• Fix + Add API token UI permissions bug
• Filter for archived resources in the Console UI
• Archive and unarchive resources in the Console UI
• Make Start Points for Branches fallible
• Parse results for multiple tools for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.17

• Revamp UI for looking at resource details
• Remove cached Organization when logging out

v0.4.16

• Track Branches for pinned Plots across changes in their Start Point
• Update pinned Plots when theme changes
• Remove 15 second timeout for all CLI commands
• Show total number of pages for paginated lists in Console UI
• Add the ability to create and delete Organizations

v0.4.15

• Fix 404 error for dedicated Metric pages in Console
• Create Branch and Testbed, if they do not exist for a Report on the API server
• Remove Branch and Testbed soft creation
• Add Docker daemon check to CLI commands

v0.4.14

• Fix iai-callgrind adapter bug
• Obfuscate user emails in API error messages
• Include Metric UUID in Report and Perf results
• Add a GET Metric API endpoint and CLI subcommand
• Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
• Enable busy_timeout for SQLite connections to prevent database is locked errors
• Add replica settings for disaster recovery

v0.4.13

• Add Required to required query params in docs (Thank you eddyashton)
• Auto-log out in Console UI for expired sessions
• Fix wordmark color for default theme initialization
• Add bencher plot subcommand to manage pinned plots to CLI
• Add --api-env/--console-env and --api-volume/--console-volume to bencher up CLI command (Thank you ricardaxel)
• Add service argument to bencher up and bencher down to only control a single container

v0.4.12

• Fix cross-project query bug

v0.4.11

• Reduce Console navbar, menu, and panel flickering on page load
• Fix Add, Edit, Delete button permissions bug (Thank you eddyashton for reporting this bug)
• Add the ability to “pin” Perf Plots to create a dashboard of pinned plots
• Add a button to clear each dimension separately in the Perf Plot

v0.4.10

• Expand --file-size option for bencher run to be used multiple times to track multiple file sizes
• Add dark mode to Bencher Console UI

v0.4.9

• Automatically find the git branch name for bencher run if one is not specified
• Add the --file-size option to bencher run to track file size (ie binary size)
• Allow custom Measures in Report results to be created on the fly
• Fix parsing new libtest bench Metrics as floats instead of integers (#390)

v0.4.8

• Add new user onboarding flow
• Fix server stats endpoint bug
• A very special thank you to hamirmahal for their first contribution!

v0.4.7

• Redirect to public Perf Page for Bencher Console links if user is not logged in
• Fix jittering in Perf Plot tabs when loading and changing pages
• Fix PR comment tags when changing branch start points
• Move benchmark results under a details tag for HTML comment output
• Add an Alert summary table for HTML comment output
• Add a warning if no Thresholds are found for HTML comment output
• Deprecated: --ci-no-metrics flag no longer effects HTML comment output

v0.4.6

• Add perf plot images as OpenGraph images
• Add GET /v0/users endpoint for server admins
• Improve perf plot query performance by 120x or more by adding indexes and a materialized view

v0.4.5

• Move Bencher Cloud out of BETA
• Add tab loading progress bar to perf pages
• Add --branch-start-point and --branch-start-point-hash options to bencher run to track rebasing and other branch changes
• Deprecated: The bencher run options and flags --if-branch, --else-if-branch, --else-branch, and --endif-branch have been deprecated in favor of using --branch and --branch-start-point
• BREAKING CHANGE --if-branch will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without --else-branch set
• BREAKING CHANGE --else-if-branch will only use the first branch specified as the start point and it must exist, if it is a non-empty string
• Update branch date time plot to use local time and version plot to be discrete along the x-axis

v0.4.4

• Fix resource contention during large Perf queries
• Remove public query param from /v0/projects endpoint
• BREAKING CHANGE Move all CLI commands using --org, --project, and --user over to positional arguments, except bencher run and bencher threshold create
• Add a bencher report create CLI subcommand
• bencher run now tries to auto-fill the git commit hash for the report
• BREAKING CHANGE Rename statistic to model and remove API endpoint
• Add “Click to view Threshold” for all boundary limits and add a public view Threshold page
• Add PATCH endpoint for user account settings
• BREAKING CHANGE Remove /v0/server/ping endpoint
• Deprecate /v0/server/endpoint in favor of /v0/server/config/console and BREAKING CHANGE remove bencher server endpoint CLI command

v0.4.3

• BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
• Add shortcut URLs for user pages:
  • /console/users/tokens -> /console/users/USER-SLUG/tokens
  • /console/users/tokens/add -> /console/users/USER-SLUG/tokens/add
  • /console/users/settings -> /console/users/USER-SLUG/settings
• Improve bencher run error messages to include the command being run (idea credit to dholroyd: #335)
• Prevent pwn requests when using bencher run
• Add an API version check to all Bencher CLI commands that use the backend

v0.4.2

• Fix Perf Plot view reduce bug
• Add a View button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
• Add search query param to GET list endpoints for all named resources
• Add search for a named resources in Console UI and Perf Plot dimensions
• Add date based search for Reports in Console UI and Perf Plot
• Add search for public projects
• Move Bencher API server to “distroless” Docker image, bencher-api that is 10x smaller
• Move Bencher Console UI to “distroless” Docker image, bencher-console that is 1/3 smaller
• Fix Console server side rendering bug when using local URLs inside of Docker
• First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter

v0.4.1

• Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
• Add an --exec flag to bencher run to run commands as executables (no shell) and by default when there is more than one argument
• Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
• Include Iteration number for Report results
• Redirect to newly created resources in Console UI

v0.4.0

• BREAKING CHANGES Remove backwards compatibility for renames:
  • Metric (lower_bound => lower_value and upper_bound => upper_value)
  • Measure (metric_kind => measure in Report and Threshold; metric_kinds => measures in Perf)
  • Boundary (average => baseline)
• Add bencher up, bencher down, and bencher logs commands to CLI to easily run Bencher Self-Hosted
• Branches and Testbeds are now created if they do not exit for bencher run if the value passed is a slug or name (UUIDs are still required to already exist)

v0.3.26

• Add OAuth2 based login for GitHub
• Allow users to accept invitations while logged in
• Run smoke tests on Linux, MacOS, and Windows
• Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
• Add convenience script to install bencher CLI on Linux, MacOS, and Windows

v0.3.25

• Allow custom CI ID to be an unbounded length

v0.3.24

• Add URL fragment links to all headings 🔗
• Fix browser navigation to replace history for automatic query args, like pagination
• Scroll to top of page when changing paginated pages
• Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (GET list & POST), Threshold (GET list & POST), and Branch (POST start point) endpoints
• Validate Branch and Testbed exist before running benchmarks with bencher run

v0.3.23

• Move CLI over to relaxed response parsing and add --strict flag to enable strict response parsing
• Add --quiet flag to bencher run to enable quite mode and only output the final Report JSON

v0.3.22

• The Boundary Limits object is now optional for Perf query results and Reports
• Update all POST API endpoints with creation semantics to return a 201 status code
• Update all PUT and PATCH API endpoints with immediate update semantics to return a 200 status code
• Update all DELETE API endpoints to return a 204 status code and an empty body

v0.3.21

• Add Hyperfine adapter
• Rename Metric Kind to Measure

v0.3.20

• Rename average to baseline for the Boundary Limits

v0.3.19

• Include the average used for the Boundary Limits
• Remove --ci-public-links for bencher run and simply use the project visibility to determine PR links
• Replace --ci-with-metrics with --ci-no-metrics for bencher run and make showing benchmark metrics and boundary limits on PRs the default

v0.3.18

• Add public Alert pages
• Add a --ci-public-links option to bencher run to only post public links to PRs
• Add a --ci-with-metrics option to bencher run to display benchmark metrics and boundary limits on PRs
• Update GitHub Actions PR documentation

v0.3.17

• Fix word break CSS in console UI
• Add status bar for perf plot loading
• Fix perf plot x-axis label
• Improve embedded perf plot layout

v0.3.16

• Improve bencher_client ergonomics for use in rustls
• Fix navbar bug on public pages while logged in
• Add Bencher Learn section to the site
• Set max perf query permutations to 255
• BREAKING CHANGE: Move from single metric_kind to array of metric_kinds for perf plot query

v0.3.15

• Embedded Perf Plots
• Move default Threshold Boundaries from 0.95 to 0.98
• Add server statistics collection to Bencher Self-Hosted instances

v0.3.14

• Move default Threshold Boundaries from 0.9 to 0.95
• Redirect to login for an authorized users in Console UI
• Improve API server error messages
• Fixed the UI y-axis scale issue… again
• Add Bencher Plus licenses

v0.3.13

• Add ability for bencher run to read output from a file without running a command argument
• Improve Report creation error messages
• Add filtering Reports by Branch, Testbed, and start/end time
• Add one month historical default when viewing Reports in the Perf Plot
• Add endpoint /v0/server/spec to view OpenAPI spec JSON
• Remove old UI source and scripts

v0.3.12

• Change Metric lower_bound and upper_bound to lower_value and upper_value respectively
• Add ability to visualize lower_value and upper_value in Perf Plot

v0.3.11

• Add strongly typed IDs for database entities
• Remove deprecated configuration keys (endpoint => console.url and secret_key => security.secret_key)

v0.3.10

• Add optional error monitoring with Sentry
• Start API error message improvement
• Fix Threshold dimension selection bug in UI
• Display benchmark harness results in realtime via bencher run
• A special thank you to @robertorosmaninho for bug hunting 👾

v0.3.9

• Move UI over to Astro with stronger TypeScript
• Add check for failing Netlify deployments
• Move default Threshold Boundaries from 0.95 to 0.9

v0.3.8

• Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
• Add a custom PR comment ID argument --ci-id to bencher run for complex scenarios
• Another special thank you to @GitGab19 for all the feedback

v0.3.7

• Fix slug length validation bug
• A special thank you to @GitGab19 for all the feedback

v0.3.6

• Fix multi-Metric Kind bug for Perf Plot Report selection
• Require minimum and maximum sample sizes to be greater than or equal to two
• Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
• Comment on GitHub pull requests using --github-actions argument for bencher run
• Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with ci-only-thresholds
• Only start posting CI integrations when an Alert is generated with --ci-only-on-alert
• Fix Threshold boundary calculation bug when standard deviation is zero
• Add a warning ⚠️ in the Perf Plot UI when a Boundary Limit is not calculated
• A special thank you to @untitaker for all the feedback

v0.3.5

• Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
• Add ability to dismiss and reactivate Alerts
• Add ability to update Thresholds in the UI
• Fix Threshold PUT bug
• Add deletion capabilities to the UI
• Add help page to public UI

v0.3.4

• Revamp Quick Start tutorial
• Add Docker Self-Hosted tutorial
• Fix public perf pages bug
• Add Reports tab to perf pages
• Add ability to clear the perf plot
• Add ability to set Project URL to null
• Add Project permissions API endpoint
• Remove redundant Project API endpoints
• Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI

v0.3.3

• Move API server config key endpoint to console.url
• Change API from /v0/server/config/endpoint to /v0/server/endpoint to get console.url
• Add dedicated POST API endpoint for Benchmarks
• Add PATCH API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks
• Add PUT API endpoint for Thresholds
• Add DELETE API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds
• Create two default Thresholds for all new Projects
• Add a Go benchmark example
• Fix Go auxillary data parse error

v0.3.2

• Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
• Add pagination to all GET list API endpoints, bencher CLI, and console UI
• Move all bencher CLI API calls to using client auto-generated from OpenAPI spec
• Improve bencher CLI error messages

v0.3.1

• Add --backdate argument to bencher run for backdating Reports
• Add created and modified timestamps as applicable to all entities and their JSON
• Add ability to delete a Report
• Add ability to delete a Project
• A special thank you to @wuputah and the Hydra team for all the feedback

v0.3.0

• Add demo video to landing page
• Add bencher run documentation
• Refactor Report, Alert, and Perf JSON to be more powerful
• All Report and Perf metrics now show the boundary limits used if a Threshold is set
• Display URL to visualize report in UI after bencher run
• Change magnitude to pow for bencher mock
• A very special thank you to @MichaelScofield for being Bencher’s second community contributor!

v0.2.46

• Update roadmap
• Add Iai benchmark harness adapter
• Move from GitPod over to Dev Containers/GitHub Codespaces
• Fix thresholds with no historical metrics bug
• A very special thank you to @osiewicz for being Bencher’s first community contributor!

v0.2.45

• Add servers to Swagger API docs
• Fix perf view refresh bug
• Add an order of --magnitude argument to bencher mock command
• Add an API endpoint for threshold statistics and a CLI statistic subcommand
• Upgrade the bencher perf CLI flag --table to accept an optional style parameter
• Fix long text wrapping in UI deck view, both header and card contents
• Autogenerate TypeScript types from Rust type definitions
• Massive thank you to @vient for all the feedback

v0.2.44

• Fix UI expired token error
• Move the CLI linker over to use Zig in order to pin a specific glibc version
• Move linker in CI over to mold for increased parallelism
• Fix cargo workspace default-features = false warnings
• Add project stewardship commitments
• Massive thank you to @brainstorm for all the feedback

v0.2.43

• Add a --table option to the bencher perf command to display query results as a table
• Add the ability to toggle the perf plot x-axis from report date time to branch version number

v0.2.42

• Create and publish Bencher CLI GitHub Action
• Fix both UI and perf image y-axis sizing once and for all… hopefully

v0.2.41

• Move perf plot images from a screenshot to server side rendering
• Add a share mechanism to perf pages with a perf image preview

v0.2.40

• Add an optional --file argument to bencher run to read file output
• Move perf query API endpoint from a POST to GET
• Add an endpoint to GET a screenshot of a public perf plot

v0.2.39

• Update readme to add Discord invite
• Allow certain benchmarks to be ignored by Threshold checks
• BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
• Add paid plans and metered billing for Bencher Cloud
• Change perf pages to use milliseconds as opposed to date/time stamp

v0.2.38

• Fix CLI runtime dependency error over the past five releases
• BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
• Add a soft branch creation option to eliminate race conditions in CI
• Add supported languages to landing page
• Lots of new docs!

v0.2.33

• Add new average setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
• Increase CLI default retry to ten attempts three seconds apart (30 second window)
• Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for --else-if-branch in bencher run CLI
• Moved the plot key to below the plot all the time in preparation for screenshots/embedding

v0.2.32

• Add STARTTLS option for SMTP connections
• Add throughput as a new default metric kind with operations / second (ops/s) as units
• Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
• Add resend email button to UI auth confirm page
• Change C++ Catch2 adapter to use standard deviation for lower_bound and upper_bound
• Fix table and perf panel formatting for long, single word values

v0.2.31

• Prevent leaking secrets by obfuscating them in the server logs (ie ************)
• Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
• New harness specific adapter targets (ie rust_bench and rust_criterion)
• Add additional options to bencher mock, --fail and --flaky to easily test parsing failing and flaky benchmarks respectively
• Wrote continuous benchmarking and how to install the CLI pages
• Update server framework, dropshot, and Rust to 1.67.0
• Two new C++ adapters for Google Benchmark and Catch2
• Add an adapter for go test -bench

v0.2.30

• Added --if-else-branch, --else-branch, and --endif-branch flags to to bencher run CLI
• Wrote a branch management explanation page for using bencher run
• Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
• Made the Diataxis convention explicit in the docs
• Fixed bounds bug in bencher mock
• Added a /v0/server/backup route and bencher server backup CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy

v0.2.29

• Fixed and refixed Criterion empty benchmark name bug
• Type level checking for benchmark names and metric kind resource IDs
• Added documentation: Quick Start, Prior Art, and GitHub Actions how to
• Prettier and eslint added to CI/CD
• Stubbed out UI tests with vitest in CI/CD

v0.2.28

• Centralized all validation logic between frontend and backend via Rust + WASM modules
• All pages in the UI now have basic functionality
• Added public project perf pages

v0.2.27

• Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
• Fixed the UI documentation + routing bug! It turns out SwaggerUIBundle was the solution.
• Added Metric Kind list, add, and view to UI
• In perf queries allowed for both UUIDs and slugs for branches and testbeds
• Added bencher.dev to Google Search Console and Analytics
• Changed API server default URL to always be localhost:3000
• Moved Docker UI build stage into its own builder.Dockerfile, mirroring the API setup
• Created a better default for the UI’s backend API URL, basically this-host:61016
• Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
• Fixed UI logout + routing “too many redirects” bug
• Changed notifications from state based to URL query param based in the UI
• Refactored how pathname and redirects are handled in the UI to increase encapsulation
• Added the current version to the UI BETA tag
• Created a repo shortcut at bencher.dev/repo