Allow bencher run to create a project if it does not exist for both anonymous and authenticated users
Deprecate the --no-hash flag for bencher run
Move to Linux, macOS, and Windows as the default Testbeds for bencher run
Add log scaling to Perf Images (Thank you @brianheineman)
Add daily rate limiting for Metrics for Organizations without a Bencher Plus plan
v0.4.37
Add dimension search in Threshold creation UI
Add Threshold list to Branch, Testbed, and Measure pages in Console UI
Add unit abbreviation to Perf Report tables in Console UI and PR comments (Thank you @BD103)
Add support for GitHub Enterprise Server to bencher CLI (Thank you @btoic)
v0.4.36
Fix CORS failures when API server returns an error
Catch errors when Console WASM fails to load
Add --insecure-host flag to bencher CLI to allow insecure connections to Bencher Self-Hosted API servers
Add --native-tls flag to bencher CLI to load TLS certificates from the platform’s native certificate store
Add --timeout option to bencher CLI to set the timeout for all API requests
Add smtp.insecure_host option to Bencher Self-Hosted API server configuration to allow insecure connections to SMTP servers (Thank you @btoic)
v0.4.35
Add “Dismiss All” Alerts button to Console UI
Fix infinite progress bar when Perf Plot is not initialized
Show collapsed key for Pinned Plots
Show selected Branches, Testbeds, and Benchmarks at the top of their Perf Plot tabs
Improve Reports table view
Show “Recent Reports” table for Branches and Testbeds
Use exponential scaling for Perf Plot y-axis when there is a > 10x relative difference (Thank you @brianheineman)
Improve new contributor onboarding experience (Thank you @brianheineman)
v0.4.34
Add --head option to bencher branch view CLI subcommand
Allow selection of up to two Measures in Perf Images
Fix the Perf Plot Pinned tab
Add --build-time and --file-size to PR comment ID
Omit Threshold warning from PR comment when --ci-only-thresholds is used
Fix flashing of “No data found” in Perf Plot
v0.4.33
Fix plot image y-axis labels
Show correctly scaled units in Perf Plot hover box
Factor out common dimensions in Perf Plot key
Allow selection of up to two Measures in Perf Plots
v0.4.32
Fix benchmark result units in PR comments (Thank you @dklassic)
v0.4.31
Fix Report measure count bug
Auto-scale units in Perf Plots, Perf Images, Reports, and PR comments
v0.4.30
Fix Bencher Self-Hosted Console build bug since v0.4.24
v0.4.29
Improve Pinned Plots loading fallback
Return 404 Not Found for all resource endpoints when checking for both existence and permissions
Update CLI install scripts to be able to set the BENCHER_VERSION
v0.4.28
Remove lazy loading of embedded perf plots due to Safari bug
Move Pinned Plots out of iframe tags
Fix custom pinned plot window size bug
v0.4.27
Limit GitHub Action Check to 65,535 characters
Fix CLI HTML table output bug for Benchmarks with different Measures
v0.4.26
Fix GitHub Action summary bug (Thank you @dklassic)
v0.4.25
Add GitHub Action summary when using --github-actions option
Add ability to track build times with the new --build-time flag
v0.4.24
Add GitHub Action Check when not running as a pull request (Thank you @sagudev, @dklassic, and @wusyong)
v0.4.23
Fix Console Alert Perf Plot button bug
Fix benchmark.js adapter (js_benchmark) float parsing bug (Thank you @arv)
Fix Catch2 adapter (cpp_catch2) long name parse bug
BREAKING CHANGE The Catch2 adapter (cpp_catch2) now includes the test case name in the benchmark name (ex: Test Case: Benchmark Name)
Fix Benchmark results table x-overflow bug when there were several Measures
v0.4.22
Remove bloom filter in Perf Query when there is no branch head specified
v0.4.21
BREAKING CHANGE Update the Branch JSON to include the concept of a Head which now contains the Start Point
Changing a branch’s start point will now update the branch head instead of creating a new branch and archiving the old one
Deprecate --branch-start-point, --branch-start-point-hash, and --branch-reset in favor of --start-point, --start-point-hash, and --start-point-reset for bencher run
Introduce a max_versions for Branch Start Points, default to 255
Add the --start-point-max-versions option to bencher run
Thresholds are no longer cloned based off of a branch start point
Add the --start-point-clone-thresholds flag to bencher run
Add Thresholds to Report JSON
Add --threshold-measure, --threshold-test, --threshold-min-sample-size, --threshold-max-sample-size, --threshold-window, --threshold-lower-boundary, --threshold-upper-boundary, and --thresholds-reset to bencher run
Fix public plots pages to not require authentication
Add --measure option to bencher mock to mock multiple measures
Add a bencher archive and bencher unarchive subcommand to archive and unarchive dimensions
Fix broken default version tag for bencher up (Thank you @mwasplund)
Add --api-port and --console-port options to bencher up to set the API and Console UI ports
v0.4.20
Add public pages for Reports, Branches, Testbeds, Benchmarks, Measures, and Plots
Add links between all dimension pages
BREAKING CHANGE Change the Report results format to mirror Bencher Metric Format, from measures owning benchmarks to benchmarks owning measures
Remove user email from Report results
Add a table view for Reports in the Console UI
Update PR comments to be more concise
Remove deprecated statistic field from Threshold in favor of model
Remove deprecated server endpoint /v0/server/endpoint in favor of /v0/server/config/console
Track global bus events for iai-callgrind adapter (Thank you @OmarTawfik)
v0.4.19
Add the --format option to bencher run
BREAKING CHANGE Remove the Report JSON from the default bencher run output (now use --format json)
BREAKING CHANGE Remove the --html flag from bencher run (now use --format html)
Standardize all bencher CLI default values
Allow --branch-reset to be used with --branch-start-point
Update archived branches renaming scheme
v0.4.18
Add the concept of archived to Branches, Testbeds, Benchmarks, and Measures
Remove Alert stats endpoint
Fix + Add API token UI permissions bug
Filter for archived resources in the Console UI
Archive and unarchive resources in the Console UI
Make Start Points for Branches fallible
Parse results for multiple tools for iai-callgrind adapter (Thank you @OmarTawfik)
v0.4.17
Revamp UI for looking at resource details
Remove cached Organization when logging out
v0.4.16
Track Branches for pinned Plots across changes in their Start Point
Update pinned Plots when theme changes
Remove 15 second timeout for all CLI commands
Show total number of pages for paginated lists in Console UI
Add the ability to create and delete Organizations
v0.4.15
Fix 404 error for dedicated Metric pages in Console
Create Branch and Testbed, if they do not exist for a Report on the API server
Remove Branch and Testbed soft creation
Add Docker daemon check to CLI commands
v0.4.14
Fix iai-callgrind adapter bug
Obfuscate user emails in API error messages
Include Metric UUID in Report and Perf results
Add a GET Metric API endpoint and CLI subcommand
Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
Enable busy_timeout for SQLite connections to prevent database is locked errors
Add replica settings for disaster recovery
v0.4.13
Add Required to required query params in docs (Thank you eddyashton)
Auto-log out in Console UI for expired sessions
Fix wordmark color for default theme initialization
Add bencher plot subcommand to manage pinned plots to CLI
Add --api-env/--console-env and --api-volume/--console-volume to bencher up CLI command (Thank you ricardaxel)
Add service argument to bencher up and bencher down to only control a single container
v0.4.12
Fix cross-project query bug
v0.4.11
Reduce Console navbar, menu, and panel flickering on page load
Fix Add, Edit, Delete button permissions bug (Thank you eddyashton for reporting this bug)
Add the ability to “pin” Perf Plots to create a dashboard of pinned plots
Add a button to clear each dimension separately in the Perf Plot
v0.4.10
Expand --file-size option for bencher run to be used multiple times to track multiple file sizes
Add dark mode to Bencher Console UI
v0.4.9
Automatically find the git branch name for bencher run if one is not specified
Add the --file-size option to bencher run to track file size (ie binary size)
Allow custom Measures in Report results to be created on the fly
Fix parsing new libtest bench Metrics as floats instead of integers (#390)
v0.4.8
Add new user onboarding flow
Fix server stats endpoint bug
A very special thank you to hamirmahal for their first contribution!
v0.4.7
Redirect to public Perf Page for Bencher Console links if user is not logged in
Fix jittering in Perf Plot tabs when loading and changing pages
Fix PR comment tags when changing branch start points
Move benchmark results under a details tag for HTML comment output
Add an Alert summary table for HTML comment output
Add a warning if no Thresholds are found for HTML comment output
Deprecated: --ci-no-metrics flag no longer effects HTML comment output
v0.4.6
Add perf plot images as OpenGraph images
Add GET /v0/users endpoint for server admins
Improve perf plot query performance by 120x or more by adding indexes and a materialized view
v0.4.5
Move Bencher Cloud out of BETA
Add tab loading progress bar to perf pages
Add --branch-start-point and --branch-start-point-hash options to bencher run to track rebasing and other branch changes
Deprecated: The bencher run options and flags --if-branch, --else-if-branch, --else-branch, and --endif-branch
have been deprecated in favor of using --branch and --branch-start-point
BREAKING CHANGE--if-branch will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without --else-branch set
BREAKING CHANGE--else-if-branch will only use the first branch specified as the start point and it must exist, if it is a non-empty string
Update branch date time plot to use local time and version plot to be discrete along the x-axis
v0.4.4
Fix resource contention during large Perf queries
Remove public query param from /v0/projects endpoint
BREAKING CHANGE Move all CLI commands using --org, --project, and --user over to positional arguments, except bencher run and bencher threshold create
Add a bencher report create CLI subcommand
bencher run now tries to auto-fill the git commit hash for the report
BREAKING CHANGE Rename statistic to model and remove API endpoint
Add “Click to view Threshold” for all boundary limits and add a public view Threshold page
Add PATCH endpoint for user account settings
BREAKING CHANGE Remove /v0/server/ping endpoint
Deprecate /v0/server/endpoint in favor of /v0/server/config/console and BREAKING CHANGE remove bencher server endpoint CLI command
v0.4.3
BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
Improve bencher run error messages to include the command being run (idea credit to dholroyd: #335)
Prevent pwn requests when using bencher run
Add an API version check to all Bencher CLI commands that use the backend
v0.4.2
Fix Perf Plot view reduce bug
Add a View button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
Add search query param to GET list endpoints for all named resources
Add search for a named resources in Console UI and Perf Plot dimensions
Add date based search for Reports in Console UI and Perf Plot
Add search for public projects
Move Bencher API server to “distroless” Docker image, bencher-api that is 10x smaller
Move Bencher Console UI to “distroless” Docker image, bencher-console that is 1/3 smaller
Fix Console server side rendering bug when using local URLs inside of Docker
First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter
v0.4.1
Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
Add an --exec flag to bencher run to run commands as executables (no shell) and by default when there is more than one argument
Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
Include Iteration number for Report results
Redirect to newly created resources in Console UI
v0.4.0
BREAKING CHANGES Remove backwards compatibility for renames:
Metric (lower_bound => lower_value and upper_bound => upper_value)
Measure (metric_kind => measure in Report and Threshold; metric_kinds => measures in Perf)
Boundary (average => baseline)
Add bencher up, bencher down, and bencher logs commands to CLI to easily run Bencher Self-Hosted
Branches and Testbeds are now created if they do not exit for bencher run if the value passed is a slug or name (UUIDs are still required to already exist)
v0.3.26
Add OAuth2 based login for GitHub
Allow users to accept invitations while logged in
Run smoke tests on Linux, MacOS, and Windows
Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
Add convenience script to install bencher CLI on Linux, MacOS, and Windows
Fix browser navigation to replace history for automatic query args, like pagination
Scroll to top of page when changing paginated pages
Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (GET list & POST), Threshold (GET list & POST), and Branch (POST start point) endpoints
Validate Branch and Testbed exist before running benchmarks with bencher run
v0.3.23
Move CLI over to relaxed response parsing and add --strict flag to enable strict response parsing
Add --quiet flag to bencher run to enable quite mode and only output the final Report JSON
v0.3.22
The Boundary Limits object is now optional for Perf query results and Reports
Update all POST API endpoints with creation semantics to return a 201 status code
Update all PUT and PATCH API endpoints with immediate update semantics to return a 200 status code
Update all DELETE API endpoints to return a 204 status code and an empty body
v0.3.21
Add Hyperfine adapter
Rename Metric Kind to Measure
v0.3.20
Rename average to baseline for the Boundary Limits
v0.3.19
Include the average used for the Boundary Limits
Remove --ci-public-links for bencher run and simply use the project visibility to determine PR links
Replace --ci-with-metrics with --ci-no-metrics for bencher run and make showing benchmark metrics and boundary limits on PRs the default
v0.3.18
Add public Alert pages
Add a --ci-public-links option to bencher run to only post public links to PRs
Add a --ci-with-metrics option to bencher run to display benchmark metrics and boundary limits on PRs
Massive thank you to @brainstorm for all the feedback
v0.2.43
Add a --table option to the bencher perf command to display query results as a table
Add the ability to toggle the perf plot x-axis from report date time to branch version number
v0.2.42
Create and publish Bencher CLI GitHub Action
Fix both UI and perf image y-axis sizing once and for all… hopefully
v0.2.41
Move perf plot images from a screenshot to server side rendering
Add a share mechanism to perf pages with a perf image preview
v0.2.40
Add an optional --file argument to bencher run to read file output
Move perf query API endpoint from a POST to GET
Add an endpoint to GET a screenshot of a public perf plot
v0.2.39
Update readme to add Discord invite
Allow certain benchmarks to be ignored by Threshold checks
BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
Add paid plans and metered billing for Bencher Cloud
Change perf pages to use milliseconds as opposed to date/time stamp
v0.2.38
Fix CLI runtime dependency error over the past five releases
BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
Add a soft branch creation option to eliminate race conditions in CI
Add supported languages to landing page
Lots of new docs!
v0.2.33
Add new average setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
Increase CLI default retry to ten attempts three seconds apart (30 second window)
Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for --else-if-branch in bencher run CLI
Moved the plot key to below the plot all the time in preparation for screenshots/embedding
v0.2.32
Add STARTTLS option for SMTP connections
Add throughput as a new default metric kind with operations / second (ops/s) as units
Added a /v0/server/backup route and bencher server backup CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy
v0.2.29
Fixed and refixed Criterion empty benchmark name bug
Type level checking for benchmark names and metric kind resource IDs
Added documentation: Quick Start, Prior Art, and GitHub Actions how to
Prettier and eslint added to CI/CD
Stubbed out UI tests with vitest in CI/CD
v0.2.28
Centralized all validation logic between frontend and backend via Rust + WASM modules
All pages in the UI now have basic functionality
Added public project perf pages
v0.2.27
Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
Fixed the UI documentation + routing bug! It turns out SwaggerUIBundle was the solution.
Added Metric Kind list, add, and view to UI
In perf queries allowed for both UUIDs and slugs for branches and testbeds
Added bencher.dev to Google Search Console and Analytics
Changed API server default URL to always be localhost:3000
Moved Docker UI build stage into its own builder.Dockerfile, mirroring the API setup
Created a better default for the UI’s backend API URL, basically this-host:61016
Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
Fixed UI logout + routing “too many redirects” bug
Changed notifications from state based to URL query param based in the UI
Refactored how pathname and redirects are handled in the UI to increase encapsulation