Bencher Changelog


Pending v0.4.3

  • BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
  • Add shortcut URLs for user pages:
    • /console/users/tokens -> /console/users/USER-SLUG/tokens
    • /console/users/tokens/add -> /console/users/USER-SLUG/tokens/add
    • /console/users/settings -> /console/users/USER-SLUG/settings
  • Improve bencher run error messages to include the command being run (idea credit to dholroyd: #335)
  • Prevent pwn requests when using bencher run
  • Add an API version check to all Bencher CLI commands that use the backend

v0.4.2

  • Fix Perf Plot view reduce bug
  • Add a View button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
  • Add search query param to GET list endpoints for all named resources
  • Add search for a named resources in Console UI and Perf Plot dimensions
  • Add date based search for Reports in Console UI and Perf Plot
  • Add search for public projects
  • Move Bencher API server to “distroless” Docker image, bencher-api that is 10x smaller
  • Move Bencher Console UI to “distroless” Docker image, bencher-console that is 1/3 smaller
  • Fix Console server side rendering bug when using local URLs inside of Docker
  • First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter

v0.4.1

  • Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
  • Add an --exec flag to bencher run to run commands as executables (no shell) and by default when there is more than one argument
  • Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
  • Include Iteration number for Report results
  • Redirect to newly created resources in Console UI

v0.4.0

  • BREAKING CHANGES Remove backwards compatibility for renames:
    • Metric (lower_bound => lower_value and upper_bound => upper_value)
    • Measure (metric_kind => measure in Report and Threshold; metric_kinds => measures in Perf)
    • Boundary (average => baseline)
  • Add bencher up, bencher down, and bencher logs commands to CLI to easily run Bencher Self-Hosted
  • Branches and Testbeds are now created if they do not exit for bencher run if the value passed is a slug or name (UUIDs are still required to already exist)

v0.3.26

  • Add OAuth2 based login for GitHub
  • Allow users to accept invitations while logged in
  • Run smoke tests on Linux, MacOS, and Windows
  • Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
  • Add convenience script to install bencher CLI on Linux, MacOS, and Windows

v0.3.25

  • Allow custom CI ID to be an unbounded length

v0.3.24

  • Add URL fragment links to all headings 🔗
  • Fix browser navigation to replace history for automatic query args, like pagination
  • Scroll to top of page when changing paginated pages
  • Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (GET list & POST), Threshold (GET list & POST), and Branch (POST start point) endpoints
  • Validate Branch and Testbed exist before running benchmarks with bencher run

v0.3.23

  • Move CLI over to relaxed response parsing and add --strict flag to enable strict response parsing
  • Add --quiet flag to bencher run to enable quite mode and only output the final Report JSON

v0.3.22

  • The Boundary Limits object is now optional for Perf query results and Reports
  • Update all POST API endpoints with creation semantics to return a 201 status code
  • Update all PUT and PATCH API endpoints with immediate update semantics to return a 200 status code
  • Update all DELETE API endpoints to return a 204 status code and an empty body

v0.3.21

  • Add Hyperfine adapter
  • Rename Metric Kind to Measure

v0.3.20

  • Rename average to baseline for the Boundary Limits

v0.3.19

  • Include the average used for the Boundary Limits
  • Remove --ci-public-links for bencher run and simply use the project visibility to determine PR links
  • Replace --ci-with-metrics with --ci-no-metrics for bencher run and make showing benchmark metrics and boundary limits on PRs the default

v0.3.18

  • Add public Alert pages
  • Add a --ci-public-links option to bencher run to only post public links to PRs
  • Add a --ci-with-metrics option to bencher run to display benchmark metrics and boundary limits on PRs
  • Update GitHub Actions PR documentation

v0.3.17

  • Fix word break CSS in console UI
  • Add status bar for perf plot loading
  • Fix perf plot x-axis label
  • Improve embedded perf plot layout

v0.3.16

  • Improve bencher_client ergonomics for use in rustls
  • Fix navbar bug on public pages while logged in
  • Add Bencher Learn section to the site
  • Set max perf query permutations to 255
  • BREAKING CHANGE: Move from single metric_kind to array of metric_kinds for perf plot query

v0.3.15

v0.3.14

  • Move default Threshold Boundaries from 0.9 to 0.95
  • Redirect to login for an authorized users in Console UI
  • Improve API server error messages
  • Fixed the UI y-axis scale issue… again
  • Add Bencher Plus licenses

v0.3.13

  • Add ability for bencher run to read output from a file without running a command argument
  • Improve Report creation error messages
  • Add filtering Reports by Branch, Testbed, and start/end time
  • Add one month historical default when viewing Reports in the Perf Plot
  • Add endpoint /v0/server/spec to view OpenAPI spec JSON
  • Remove old UI source and scripts

v0.3.12

  • Change Metric lower_bound and upper_bound to lower_value and upper_value respectively
  • Add ability to visualize lower_value and upper_value in Perf Plot

v0.3.11

  • Add strongly typed IDs for database entities
  • Remove deprecated configuration keys (endpoint => console.url and secret_key => security.secret_key)

v0.3.10

  • Add optional error monitoring with Sentry
  • Start API error message improvement
  • Fix Threshold dimension selection bug in UI
  • Display benchmark harness results in realtime via bencher run
  • A special thank you to @robertorosmaninho for bug hunting 👾

v0.3.9

  • Move UI over to Astro with stronger TypeScript
  • Add check for failing Netlify deployments
  • Move default Threshold Boundaries from 0.95 to 0.9

v0.3.8

  • Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
  • Add a custom PR comment ID argument --ci-id to bencher run for complex scenarios
  • Another special thank you to @GitGab19 for all the feedback

v0.3.7

  • Fix slug length validation bug
  • A special thank you to @GitGab19 for all the feedback

v0.3.6

  • Fix multi-Metric Kind bug for Perf Plot Report selection
  • Require minimum and maximum sample sizes to be greater than or equal to two
  • Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
  • Comment on GitHub pull requests using --github-actions argument for bencher run
  • Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with ci-only-thresholds
  • Only start posting CI integrations when an Alert is generated with --ci-only-on-alert
  • Fix Threshold boundary calculation bug when standard deviation is zero
  • Add a warning ⚠️ in the Perf Plot UI when a Boundary Limit is not calculated
  • A special thank you to @untitaker for all the feedback

v0.3.5

  • Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
  • Add ability to dismiss and reactivate Alerts
  • Add ability to update Thresholds in the UI
  • Fix Threshold PUT bug
  • Add deletion capabilities to the UI
  • Add help page to public UI

v0.3.4

  • Revamp Quick Start tutorial
  • Add Docker Self-Hosted tutorial
  • Fix public perf pages bug
  • Add Reports tab to perf pages
  • Add ability to clear the perf plot
  • Add ability to set Project URL to null
  • Add Project permissions API endpoint
  • Remove redundant Project API endpoints
  • Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI

v0.3.3

  • Move API server config key endpoint to console.url
  • Change API from /v0/server/config/endpoint to /v0/server/endpoint to get console.url
  • Add dedicated POST API endpoint for Benchmarks
  • Add PATCH API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks
  • Add PUT API endpoint for Thresholds
  • Add DELETE API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds
  • Create two default Thresholds for all new Projects
  • Add a Go benchmark example
  • Fix Go auxillary data parse error

v0.3.2

  • Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
  • Add pagination to all GET list API endpoints, bencher CLI, and console UI
  • Move all bencher CLI API calls to using client auto-generated from OpenAPI spec
  • Improve bencher CLI error messages

v0.3.1

  • Add --backdate argument to bencher run for backdating Reports
  • Add created and modified timestamps as applicable to all entities and their JSON
  • Add ability to delete a Report
  • Add ability to delete a Project
  • A special thank you to @wuputah and the Hydra team for all the feedback

v0.3.0

  • Add demo video to landing page
  • Add bencher run documentation
  • Refactor Report, Alert, and Perf JSON to be more powerful
  • All Report and Perf metrics now show the boundary limits used if a Threshold is set
  • Display URL to visualize report in UI after bencher run
  • Change magnitude to pow for bencher mock
  • A very special thank you to @MichaelScofield for being Bencher’s second community contributor!

v0.2.46

  • Update roadmap
  • Add Iai benchmark harness adapter
  • Move from GitPod over to Dev Containers/GitHub Codespaces
  • Fix thresholds with no historical metrics bug
  • A very special thank you to @osiewicz for being Bencher’s first community contributor!

v0.2.45

  • Add servers to Swagger API docs
  • Fix perf view refresh bug
  • Add an order of --magnitude argument to bencher mock command
  • Add an API endpoint for threshold statistics and a CLI statistic sub-command
  • Upgrade the bencher perf CLI flag --table to accept an optional style parameter
  • Fix long text wrapping in UI deck view, both header and card contents
  • Autogenerate TypeScript types from Rust type definitions
  • Massive thank you to @vient for all the feedback

v0.2.44

  • Fix UI expired token error
  • Move the CLI linker over to use Zig in order to pin a specific glibc version
  • Move linker in CI over to mold for increased parallelism
  • Fix cargo workspace default-features = false warnings
  • Add project stewardship commitments
  • Massive thank you to @brainstorm for all the feedback

v0.2.43

  • Add a --table option to the bencher perf command to display query results as a table
  • Add the ability to toggle the perf plot x-axis from report date time to branch version number

v0.2.42

  • Create and publish Bencher CLI GitHub Action
  • Fix both UI and perf image y-axis sizing once and for all… hopefully

v0.2.41

  • Move perf plot images from a screenshot to server side rendering
  • Add a share mechanism to perf pages with a perf image preview

v0.2.40

  • Add an optional --file argument to bencher run to read file output
  • Move perf query API endpoint from a POST to GET
  • Add an endpoint to GET a screenshot of a public perf plot

v0.2.39

  • Update readme to add Discord invite
  • Allow certain benchmarks to be ignored by Threshold checks
  • BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
  • Add paid plans and metered billing for Bencher Cloud
  • Change perf pages to use milliseconds as opposed to date/time stamp

v0.2.38

  • Fix CLI runtime dependency error over the past five releases
  • BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
  • Add a soft branch creation option to eliminate race conditions in CI
  • Add supported languages to landing page
  • Lots of new docs!

v0.2.33

  • Add new average setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
  • Increase CLI default retry to ten attempts three seconds apart (30 second window)
  • Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for --else-if-branch in bencher run CLI
  • Moved the plot key to below the plot all the time in preparation for screenshots/embedding

v0.2.32

  • Add STARTTLS option for SMTP connections
  • Add throughput as a new default metric kind with operations / second (ops/s) as units
  • Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
  • Add resend email button to UI auth confirm page
  • Change C++ Catch2 adapter to use standard deviation for lower_bound and upper_bound
  • Fix table and perf panel formatting for long, single word values

v0.2.31

  • Prevent leaking secrets by obfuscating them in the server logs (ie ************)
  • Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
  • New harness specific adapter targets (ie rust_bench and rust_criterion)
  • Add additional options to bencher mock, --fail and --flaky to easily test parsing failing and flaky benchmarks respectively
  • Wrote continuous benchmarking and how to install the CLI pages
  • Update server framework, dropshot, and Rust to 1.67.0
  • Two new C++ adapters for Google Benchmark and Catch2
  • Add an adapter for go test -bench

v0.2.30

  • Added --if-else-branch, --else-branch, and --endif-branch flags to to bencher run CLI
  • Wrote a branch management explanation page for using bencher run
  • Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
  • Made the Diataxis convention explicit in the docs
  • Fixed bounds bug in bencher mock
  • Added a /v0/server/backup route and bencher server backup CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy

v0.2.29

  • Fixed and refixed Criterion empty benchmark name bug
  • Type level checking for benchmark names and metric kind resource IDs
  • Added documentation: Quick Start, Prior Art, and GitHub Actions how to
  • Prettier and eslint added to CI/CD
  • Stubbed out UI tests with vitest in CI/CD

v0.2.28

  • Centralized all validation logic between frontend and backend via Rust + WASM modules
  • All pages in the UI now have basic functionality
  • Added public project perf pages

v0.2.27

  • Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
  • Fixed the UI documentation + routing bug! It turns out SwaggerUIBundle was the solution.
  • Added Metric Kind list, add, and view to UI
  • In perf queries allowed for both UUIDs and slugs for branches and testbeds
  • Added bencher.dev to Google Search Console and Analytics
  • Changed API server default URL to always be localhost:3000
  • Moved Docker UI build stage into its own builder.Dockerfile, mirroring the API setup
  • Created a better default for the UI’s backend API URL, basically this-host:61016
  • Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
  • Fixed UI logout + routing “too many redirects” bug
  • Changed notifications from state based to URL query param based in the UI
  • Refactored how pathname and redirects are handled in the UI to increase encapsulation
  • Added the current version to the UI BETA tag
  • Created a repo shortcut at bencher.dev/repo