Bencher Changelog

v0.4.31

  • Fix Report measure count bug
  • Auto-scale units in Perf Plots, Perf Images, Reports, and PR comments

v0.4.30

  • Fix Bencher Self-Hosted Console build bug since v0.4.24

v0.4.29

  • Improve Pinned Plots loading fallback
  • Return 404 Not Found for all resource endpoints when checking for both existence and permissions
  • Update CLI install scripts to be able to set the BENCHER_VERSION

v0.4.28

  • Remove lazy loading of embedded perf plots due to Safari bug
  • Move Pinned Plots out of iframe tags
  • Fix custom pinned plot window size bug

v0.4.27

  • Limit GitHub Action Check to 65,535 characters
  • Fix CLI HTML table output bug for Benchmarks with different Measures

v0.4.26

  • Fix GitHub Action summary bug (Thank you @dklassic)

v0.4.25

  • Add GitHub Action summary when using --github-actions option
  • Add ability to track build times with the new --build-time flag

v0.4.24

v0.4.23

  • Fix Console Alert Perf Plot button bug
  • Fix benchmark.js adapter (js_benchmark) float parsing bug (Thank you @arv)
  • Fix Catch2 adapter (cpp_catch2) long name parse bug
  • BREAKING CHANGE The Catch2 adapter (cpp_catch2) now includes the test case name in the benchmark name (ex: Test Case: Benchmark Name)
  • Fix Benchmark results table x-overflow bug when there were several Measures

v0.4.22

  • Remove bloom filter in Perf Query when there is no branch head specified

v0.4.21

  • BREAKING CHANGE Update the Branch JSON to include the concept of a Head which now contains the Start Point
  • Changing a branch’s start point will now update the branch head instead of creating a new branch and archiving the old one
  • Deprecate --branch-start-point, --branch-start-point-hash, and --branch-reset in favor of --start-point, --start-point-hash, and --start-point-reset for bencher run
  • Introduce a max_versions for Branch Start Points, default to 255
  • Add the --start-point-max-versions option to bencher run
  • Thresholds are no longer cloned based off of a branch start point
  • Add the --start-point-clone-thresholds flag to bencher run
  • Add Thresholds to Report JSON
  • Add --threshold-measure, --threshold-test, --threshold-min-sample-size, --threshold-max-sample-size, --threshold-window, --threshold-lower-boundary, --threshold-upper-boundary, and --thresholds-reset to bencher run
  • Fix public plots pages to not require authentication
  • Add --measure option to bencher mock to mock multiple measures
  • Add a bencher archive and bencher unarchive subcommand to archive and unarchive dimensions
  • Fix broken default version tag for bencher up (Thank you @mwasplund)
  • Add --api-port and --console-port options to bencher up to set the API and Console UI ports

v0.4.20

  • Add public pages for Reports, Branches, Testbeds, Benchmarks, Measures, and Plots
  • Add links between all dimension pages
  • BREAKING CHANGE Change the Report results format to mirror Bencher Metric Format, from measures owning benchmarks to benchmarks owning measures
  • Remove user email from Report results
  • Add a table view for Reports in the Console UI
  • Update PR comments to be more concise
  • Remove deprecated statistic field from Threshold in favor of model
  • Remove deprecated server endpoint /v0/server/endpoint in favor of /v0/server/config/console
  • Track global bus events for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.19

  • Add the --format option to bencher run
  • BREAKING CHANGE Remove the Report JSON from the default bencher run output (now use --format json)
  • BREAKING CHANGE Remove the --html flag from bencher run (now use --format html)
  • Standardize all bencher CLI default values
  • Allow --branch-reset to be used with --branch-start-point
  • Update archived branches renaming scheme

v0.4.18

  • Add the concept of archived to Branches, Testbeds, Benchmarks, and Measures
  • Remove Alert stats endpoint
  • Fix + Add API token UI permissions bug
  • Filter for archived resources in the Console UI
  • Archive and unarchive resources in the Console UI
  • Make Start Points for Branches fallible
  • Parse results for multiple tools for iai-callgrind adapter (Thank you @OmarTawfik)

v0.4.17

  • Revamp UI for looking at resource details
  • Remove cached Organization when logging out

v0.4.16

  • Track Branches for pinned Plots across changes in their Start Point
  • Update pinned Plots when theme changes
  • Remove 15 second timeout for all CLI commands
  • Show total number of pages for paginated lists in Console UI
  • Add the ability to create and delete Organizations

v0.4.15

  • Fix 404 error for dedicated Metric pages in Console
  • Create Branch and Testbed, if they do not exist for a Report on the API server
  • Remove Branch and Testbed soft creation
  • Add Docker daemon check to CLI commands

v0.4.14

  • Fix iai-callgrind adapter bug
  • Obfuscate user emails in API error messages
  • Include Metric UUID in Report and Perf results
  • Add a GET Metric API endpoint and CLI subcommand
  • Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
  • Enable busy_timeout for SQLite connections to prevent database is locked errors
  • Add replica settings for disaster recovery

v0.4.13

  • Add Required to required query params in docs (Thank you eddyashton)
  • Auto-log out in Console UI for expired sessions
  • Fix wordmark color for default theme initialization
  • Add bencher plot subcommand to manage pinned plots to CLI
  • Add --api-env/--console-env and --api-volume/--console-volume to bencher up CLI command (Thank you ricardaxel)
  • Add service argument to bencher up and bencher down to only control a single container

v0.4.12

  • Fix cross-project query bug

v0.4.11

  • Reduce Console navbar, menu, and panel flickering on page load
  • Fix Add, Edit, Delete button permissions bug (Thank you eddyashton for reporting this bug)
  • Add the ability to “pin” Perf Plots to create a dashboard of pinned plots
  • Add a button to clear each dimension separately in the Perf Plot

v0.4.10

  • Expand --file-size option for bencher run to be used multiple times to track multiple file sizes
  • Add dark mode to Bencher Console UI

v0.4.9

  • Automatically find the git branch name for bencher run if one is not specified
  • Add the --file-size option to bencher run to track file size (ie binary size)
  • Allow custom Measures in Report results to be created on the fly
  • Fix parsing new libtest bench Metrics as floats instead of integers (#390)

v0.4.8

  • Add new user onboarding flow
  • Fix server stats endpoint bug
  • A very special thank you to hamirmahal for their first contribution!

v0.4.7

  • Redirect to public Perf Page for Bencher Console links if user is not logged in
  • Fix jittering in Perf Plot tabs when loading and changing pages
  • Fix PR comment tags when changing branch start points
  • Move benchmark results under a details tag for HTML comment output
  • Add an Alert summary table for HTML comment output
  • Add a warning if no Thresholds are found for HTML comment output
  • Deprecated: --ci-no-metrics flag no longer effects HTML comment output

v0.4.6

  • Add perf plot images as OpenGraph images
  • Add GET /v0/users endpoint for server admins
  • Improve perf plot query performance by 120x or more by adding indexes and a materialized view

v0.4.5

  • Move Bencher Cloud out of BETA
  • Add tab loading progress bar to perf pages
  • Add --branch-start-point and --branch-start-point-hash options to bencher run to track rebasing and other branch changes
  • Deprecated: The bencher run options and flags --if-branch, --else-if-branch, --else-branch, and --endif-branch have been deprecated in favor of using --branch and --branch-start-point
  • BREAKING CHANGE --if-branch will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without --else-branch set
  • BREAKING CHANGE --else-if-branch will only use the first branch specified as the start point and it must exist, if it is a non-empty string
  • Update branch date time plot to use local time and version plot to be discrete along the x-axis

v0.4.4

  • Fix resource contention during large Perf queries
  • Remove public query param from /v0/projects endpoint
  • BREAKING CHANGE Move all CLI commands using --org, --project, and --user over to positional arguments, except bencher run and bencher threshold create
  • Add a bencher report create CLI subcommand
  • bencher run now tries to auto-fill the git commit hash for the report
  • BREAKING CHANGE Rename statistic to model and remove API endpoint
  • Add “Click to view Threshold” for all boundary limits and add a public view Threshold page
  • Add PATCH endpoint for user account settings
  • BREAKING CHANGE Remove /v0/server/ping endpoint
  • Deprecate /v0/server/endpoint in favor of /v0/server/config/console and BREAKING CHANGE remove bencher server endpoint CLI command

v0.4.3

  • BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
  • Add shortcut URLs for user pages:
    • /console/users/tokens -> /console/users/USER-SLUG/tokens
    • /console/users/tokens/add -> /console/users/USER-SLUG/tokens/add
    • /console/users/settings -> /console/users/USER-SLUG/settings
  • Improve bencher run error messages to include the command being run (idea credit to dholroyd: #335)
  • Prevent pwn requests when using bencher run
  • Add an API version check to all Bencher CLI commands that use the backend

v0.4.2

  • Fix Perf Plot view reduce bug
  • Add a View button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
  • Add search query param to GET list endpoints for all named resources
  • Add search for a named resources in Console UI and Perf Plot dimensions
  • Add date based search for Reports in Console UI and Perf Plot
  • Add search for public projects
  • Move Bencher API server to “distroless” Docker image, bencher-api that is 10x smaller
  • Move Bencher Console UI to “distroless” Docker image, bencher-console that is 1/3 smaller
  • Fix Console server side rendering bug when using local URLs inside of Docker
  • First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter

v0.4.1

  • Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
  • Add an --exec flag to bencher run to run commands as executables (no shell) and by default when there is more than one argument
  • Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
  • Include Iteration number for Report results
  • Redirect to newly created resources in Console UI

v0.4.0

  • BREAKING CHANGES Remove backwards compatibility for renames:
    • Metric (lower_bound => lower_value and upper_bound => upper_value)
    • Measure (metric_kind => measure in Report and Threshold; metric_kinds => measures in Perf)
    • Boundary (average => baseline)
  • Add bencher up, bencher down, and bencher logs commands to CLI to easily run Bencher Self-Hosted
  • Branches and Testbeds are now created if they do not exit for bencher run if the value passed is a slug or name (UUIDs are still required to already exist)

v0.3.26

  • Add OAuth2 based login for GitHub
  • Allow users to accept invitations while logged in
  • Run smoke tests on Linux, MacOS, and Windows
  • Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
  • Add convenience script to install bencher CLI on Linux, MacOS, and Windows

v0.3.25

  • Allow custom CI ID to be an unbounded length

v0.3.24

  • Add URL fragment links to all headings 🔗
  • Fix browser navigation to replace history for automatic query args, like pagination
  • Scroll to top of page when changing paginated pages
  • Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (GET list & POST), Threshold (GET list & POST), and Branch (POST start point) endpoints
  • Validate Branch and Testbed exist before running benchmarks with bencher run

v0.3.23

  • Move CLI over to relaxed response parsing and add --strict flag to enable strict response parsing
  • Add --quiet flag to bencher run to enable quite mode and only output the final Report JSON

v0.3.22

  • The Boundary Limits object is now optional for Perf query results and Reports
  • Update all POST API endpoints with creation semantics to return a 201 status code
  • Update all PUT and PATCH API endpoints with immediate update semantics to return a 200 status code
  • Update all DELETE API endpoints to return a 204 status code and an empty body

v0.3.21

  • Add Hyperfine adapter
  • Rename Metric Kind to Measure

v0.3.20

  • Rename average to baseline for the Boundary Limits

v0.3.19

  • Include the average used for the Boundary Limits
  • Remove --ci-public-links for bencher run and simply use the project visibility to determine PR links
  • Replace --ci-with-metrics with --ci-no-metrics for bencher run and make showing benchmark metrics and boundary limits on PRs the default

v0.3.18

  • Add public Alert pages
  • Add a --ci-public-links option to bencher run to only post public links to PRs
  • Add a --ci-with-metrics option to bencher run to display benchmark metrics and boundary limits on PRs
  • Update GitHub Actions PR documentation

v0.3.17

  • Fix word break CSS in console UI
  • Add status bar for perf plot loading
  • Fix perf plot x-axis label
  • Improve embedded perf plot layout

v0.3.16

  • Improve bencher_client ergonomics for use in rustls
  • Fix navbar bug on public pages while logged in
  • Add Bencher Learn section to the site
  • Set max perf query permutations to 255
  • BREAKING CHANGE: Move from single metric_kind to array of metric_kinds for perf plot query

v0.3.15

v0.3.14

  • Move default Threshold Boundaries from 0.9 to 0.95
  • Redirect to login for an authorized users in Console UI
  • Improve API server error messages
  • Fixed the UI y-axis scale issue… again
  • Add Bencher Plus licenses

v0.3.13

  • Add ability for bencher run to read output from a file without running a command argument
  • Improve Report creation error messages
  • Add filtering Reports by Branch, Testbed, and start/end time
  • Add one month historical default when viewing Reports in the Perf Plot
  • Add endpoint /v0/server/spec to view OpenAPI spec JSON
  • Remove old UI source and scripts

v0.3.12

  • Change Metric lower_bound and upper_bound to lower_value and upper_value respectively
  • Add ability to visualize lower_value and upper_value in Perf Plot

v0.3.11

  • Add strongly typed IDs for database entities
  • Remove deprecated configuration keys (endpoint => console.url and secret_key => security.secret_key)

v0.3.10

  • Add optional error monitoring with Sentry
  • Start API error message improvement
  • Fix Threshold dimension selection bug in UI
  • Display benchmark harness results in realtime via bencher run
  • A special thank you to @robertorosmaninho for bug hunting 👾

v0.3.9

  • Move UI over to Astro with stronger TypeScript
  • Add check for failing Netlify deployments
  • Move default Threshold Boundaries from 0.95 to 0.9

v0.3.8

  • Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
  • Add a custom PR comment ID argument --ci-id to bencher run for complex scenarios
  • Another special thank you to @GitGab19 for all the feedback

v0.3.7

  • Fix slug length validation bug
  • A special thank you to @GitGab19 for all the feedback

v0.3.6

  • Fix multi-Metric Kind bug for Perf Plot Report selection
  • Require minimum and maximum sample sizes to be greater than or equal to two
  • Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
  • Comment on GitHub pull requests using --github-actions argument for bencher run
  • Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with ci-only-thresholds
  • Only start posting CI integrations when an Alert is generated with --ci-only-on-alert
  • Fix Threshold boundary calculation bug when standard deviation is zero
  • Add a warning ⚠️ in the Perf Plot UI when a Boundary Limit is not calculated
  • A special thank you to @untitaker for all the feedback

v0.3.5

  • Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
  • Add ability to dismiss and reactivate Alerts
  • Add ability to update Thresholds in the UI
  • Fix Threshold PUT bug
  • Add deletion capabilities to the UI
  • Add help page to public UI

v0.3.4

  • Revamp Quick Start tutorial
  • Add Docker Self-Hosted tutorial
  • Fix public perf pages bug
  • Add Reports tab to perf pages
  • Add ability to clear the perf plot
  • Add ability to set Project URL to null
  • Add Project permissions API endpoint
  • Remove redundant Project API endpoints
  • Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI

v0.3.3

  • Move API server config key endpoint to console.url
  • Change API from /v0/server/config/endpoint to /v0/server/endpoint to get console.url
  • Add dedicated POST API endpoint for Benchmarks
  • Add PATCH API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks
  • Add PUT API endpoint for Thresholds
  • Add DELETE API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds
  • Create two default Thresholds for all new Projects
  • Add a Go benchmark example
  • Fix Go auxillary data parse error

v0.3.2

  • Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
  • Add pagination to all GET list API endpoints, bencher CLI, and console UI
  • Move all bencher CLI API calls to using client auto-generated from OpenAPI spec
  • Improve bencher CLI error messages

v0.3.1

  • Add --backdate argument to bencher run for backdating Reports
  • Add created and modified timestamps as applicable to all entities and their JSON
  • Add ability to delete a Report
  • Add ability to delete a Project
  • A special thank you to @wuputah and the Hydra team for all the feedback

v0.3.0

  • Add demo video to landing page
  • Add bencher run documentation
  • Refactor Report, Alert, and Perf JSON to be more powerful
  • All Report and Perf metrics now show the boundary limits used if a Threshold is set
  • Display URL to visualize report in UI after bencher run
  • Change magnitude to pow for bencher mock
  • A very special thank you to @MichaelScofield for being Bencher’s second community contributor!

v0.2.46

  • Update roadmap
  • Add Iai benchmark harness adapter
  • Move from GitPod over to Dev Containers/GitHub Codespaces
  • Fix thresholds with no historical metrics bug
  • A very special thank you to @osiewicz for being Bencher’s first community contributor!

v0.2.45

  • Add servers to Swagger API docs
  • Fix perf view refresh bug
  • Add an order of --magnitude argument to bencher mock command
  • Add an API endpoint for threshold statistics and a CLI statistic subcommand
  • Upgrade the bencher perf CLI flag --table to accept an optional style parameter
  • Fix long text wrapping in UI deck view, both header and card contents
  • Autogenerate TypeScript types from Rust type definitions
  • Massive thank you to @vient for all the feedback

v0.2.44

  • Fix UI expired token error
  • Move the CLI linker over to use Zig in order to pin a specific glibc version
  • Move linker in CI over to mold for increased parallelism
  • Fix cargo workspace default-features = false warnings
  • Add project stewardship commitments
  • Massive thank you to @brainstorm for all the feedback

v0.2.43

  • Add a --table option to the bencher perf command to display query results as a table
  • Add the ability to toggle the perf plot x-axis from report date time to branch version number

v0.2.42

  • Create and publish Bencher CLI GitHub Action
  • Fix both UI and perf image y-axis sizing once and for all… hopefully

v0.2.41

  • Move perf plot images from a screenshot to server side rendering
  • Add a share mechanism to perf pages with a perf image preview

v0.2.40

  • Add an optional --file argument to bencher run to read file output
  • Move perf query API endpoint from a POST to GET
  • Add an endpoint to GET a screenshot of a public perf plot

v0.2.39

  • Update readme to add Discord invite
  • Allow certain benchmarks to be ignored by Threshold checks
  • BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
  • Add paid plans and metered billing for Bencher Cloud
  • Change perf pages to use milliseconds as opposed to date/time stamp

v0.2.38

  • Fix CLI runtime dependency error over the past five releases
  • BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
  • Add a soft branch creation option to eliminate race conditions in CI
  • Add supported languages to landing page
  • Lots of new docs!

v0.2.33

  • Add new average setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
  • Increase CLI default retry to ten attempts three seconds apart (30 second window)
  • Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for --else-if-branch in bencher run CLI
  • Moved the plot key to below the plot all the time in preparation for screenshots/embedding

v0.2.32

  • Add STARTTLS option for SMTP connections
  • Add throughput as a new default metric kind with operations / second (ops/s) as units
  • Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
  • Add resend email button to UI auth confirm page
  • Change C++ Catch2 adapter to use standard deviation for lower_bound and upper_bound
  • Fix table and perf panel formatting for long, single word values

v0.2.31

  • Prevent leaking secrets by obfuscating them in the server logs (ie ************)
  • Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
  • New harness specific adapter targets (ie rust_bench and rust_criterion)
  • Add additional options to bencher mock, --fail and --flaky to easily test parsing failing and flaky benchmarks respectively
  • Wrote continuous benchmarking and how to install the CLI pages
  • Update server framework, dropshot, and Rust to 1.67.0
  • Two new C++ adapters for Google Benchmark and Catch2
  • Add an adapter for go test -bench

v0.2.30

  • Added --if-else-branch, --else-branch, and --endif-branch flags to to bencher run CLI
  • Wrote a branch management explanation page for using bencher run
  • Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
  • Made the Diataxis convention explicit in the docs
  • Fixed bounds bug in bencher mock
  • Added a /v0/server/backup route and bencher server backup CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy

v0.2.29

  • Fixed and refixed Criterion empty benchmark name bug
  • Type level checking for benchmark names and metric kind resource IDs
  • Added documentation: Quick Start, Prior Art, and GitHub Actions how to
  • Prettier and eslint added to CI/CD
  • Stubbed out UI tests with vitest in CI/CD

v0.2.28

  • Centralized all validation logic between frontend and backend via Rust + WASM modules
  • All pages in the UI now have basic functionality
  • Added public project perf pages

v0.2.27

  • Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
  • Fixed the UI documentation + routing bug! It turns out SwaggerUIBundle was the solution.
  • Added Metric Kind list, add, and view to UI
  • In perf queries allowed for both UUIDs and slugs for branches and testbeds
  • Added bencher.dev to Google Search Console and Analytics
  • Changed API server default URL to always be localhost:3000
  • Moved Docker UI build stage into its own builder.Dockerfile, mirroring the API setup
  • Created a better default for the UI’s backend API URL, basically this-host:61016
  • Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
  • Fixed UI logout + routing “too many redirects” bug
  • Changed notifications from state based to URL query param based in the UI
  • Refactored how pathname and redirects are handled in the UI to increase encapsulation
  • Added the current version to the UI BETA tag
  • Created a repo shortcut at bencher.dev/repo

Published: Sat, August 12, 2023 at 4:07:00 PM UTC | Last Updated: Wed, April 3, 2024 at 11:23:00 AM UTC