Bencher Changelog
v0.4.30
- Fix Bencher Self-Hosted Console build bug since
v0.4.24
v0.4.29
- Improve Pinned Plots loading fallback
- Return 404 Not Found for all resource endpoints when checking for both existence and permissions
- Update CLI install scripts to be able to set the
BENCHER_VERSION
v0.4.28
- Remove lazy loading of embedded perf plots due to Safari bug
- Move Pinned Plots out of
iframe
tags
- Fix custom pinned plot window size bug
v0.4.27
- Limit GitHub Action Check to 65,535 characters
- Fix CLI HTML table output bug for Benchmarks with different Measures
v0.4.26
- Fix GitHub Action summary bug (Thank you @dklassic)
v0.4.25
- Add GitHub Action summary when using
--github-actions
option
- Add ability to track build times with the new
--build-time
flag
v0.4.24
v0.4.23
- Fix Console Alert Perf Plot button bug
- Fix benchmark.js adapter (
js_benchmark
) float parsing bug (Thank you @arv)
- Fix Catch2 adapter (
cpp_catch2
) long name parse bug
- BREAKING CHANGE The Catch2 adapter (
cpp_catch2
) now includes the test case name in the benchmark name (ex: Test Case: Benchmark Name
)
- Fix Benchmark results table x-overflow bug when there were several Measures
v0.4.22
- Remove bloom filter in Perf Query when there is no branch head specified
v0.4.21
- BREAKING CHANGE Update the Branch JSON to include the concept of a Head which now contains the Start Point
- Changing a branch’s start point will now update the branch head instead of creating a new branch and archiving the old one
- Deprecate
--branch-start-point
, --branch-start-point-hash
, and --branch-reset
in favor of --start-point
, --start-point-hash
, and --start-point-reset
for bencher run
- Introduce a
max_versions
for Branch Start Points, default to 255
- Add the
--start-point-max-versions
option to bencher run
- Thresholds are no longer cloned based off of a branch start point
- Add the
--start-point-clone-thresholds
flag to bencher run
- Add Thresholds to Report JSON
- Add
--threshold-measure
, --threshold-test
, --threshold-min-sample-size
, --threshold-max-sample-size
, --threshold-window
, --threshold-lower-boundary
, --threshold-upper-boundary
, and --thresholds-reset
to bencher run
- Fix public plots pages to not require authentication
- Add
--measure
option to bencher mock
to mock multiple measures
- Add a
bencher archive
and bencher unarchive
subcommand to archive and unarchive dimensions
- Fix broken default version tag for
bencher up
(Thank you @mwasplund)
- Add
--api-port
and --console-port
options to bencher up
to set the API and Console UI ports
v0.4.20
- Add public pages for Reports, Branches, Testbeds, Benchmarks, Measures, and Plots
- Add links between all dimension pages
- BREAKING CHANGE Change the Report results format to mirror Bencher Metric Format, from measures owning benchmarks to benchmarks owning measures
- Remove user email from Report results
- Add a table view for Reports in the Console UI
- Update PR comments to be more concise
- Remove deprecated
statistic
field from Threshold in favor of model
- Remove deprecated server endpoint
/v0/server/endpoint
in favor of /v0/server/config/console
- Track global bus events for
iai-callgrind
adapter (Thank you @OmarTawfik)
v0.4.19
- Add the
--format
option to bencher run
- BREAKING CHANGE Remove the Report JSON from the default
bencher run
output (now use --format json
)
- BREAKING CHANGE Remove the
--html
flag from bencher run
(now use --format html
)
- Standardize all
bencher
CLI default values
- Allow
--branch-reset
to be used with --branch-start-point
- Update archived branches renaming scheme
v0.4.18
- Add the concept of
archived
to Branches, Testbeds, Benchmarks, and Measures
- Remove Alert stats endpoint
- Fix
+ Add
API token UI permissions bug
- Filter for
archived
resources in the Console UI
- Archive and unarchive resources in the Console UI
- Make Start Points for Branches fallible
- Parse results for multiple tools for
iai-callgrind
adapter (Thank you @OmarTawfik)
v0.4.17
- Revamp UI for looking at resource details
- Remove cached Organization when logging out
v0.4.16
- Track Branches for pinned Plots across changes in their Start Point
- Update pinned Plots when theme changes
- Remove 15 second timeout for all CLI commands
- Show total number of pages for paginated lists in Console UI
- Add the ability to create and delete Organizations
v0.4.15
- Fix 404 error for dedicated Metric pages in Console
- Create Branch and Testbed, if they do not exist for a Report on the API server
- Remove Branch and Testbed
soft
creation
- Add Docker daemon check to CLI commands
v0.4.14
- Fix iai-callgrind adapter bug
- Obfuscate user emails in API error messages
- Include Metric UUID in Report and Perf results
- Add a
GET
Metric API endpoint and CLI subcommand
- Add a dedicated Metric page when clicking on a Metric dot in a Perf Plot
- Enable
busy_timeout
for SQLite connections to prevent database is locked
errors
- Add replica settings for disaster recovery
v0.4.13
- Add
Required
to required query params in docs (Thank you eddyashton)
- Auto-log out in Console UI for expired sessions
- Fix wordmark color for default theme initialization
- Add
bencher plot
subcommand to manage pinned plots to CLI
- Add
--api-env
/--console-env
and --api-volume
/--console-volume
to bencher up
CLI command (Thank you ricardaxel)
- Add
service
argument to bencher up
and bencher down
to only control a single container
v0.4.12
- Fix cross-project query bug
v0.4.11
- Reduce Console navbar, menu, and panel flickering on page load
- Fix
Add
, Edit
, Delete
button permissions bug (Thank you eddyashton for reporting this bug)
- Add the ability to “pin” Perf Plots to create a dashboard of pinned plots
- Add a button to clear each dimension separately in the Perf Plot
v0.4.10
- Expand
--file-size
option for bencher run
to be used multiple times to track multiple file sizes
- Add dark mode to Bencher Console UI
v0.4.9
- Automatically find the
git
branch name for bencher run
if one is not specified
- Add the
--file-size
option to bencher run
to track file size (ie binary size)
- Allow custom Measures in Report results to be created on the fly
- Fix parsing new libtest bench Metrics as floats instead of integers (#390)
v0.4.8
- Add new user onboarding flow
- Fix server stats endpoint bug
- A very special thank you to hamirmahal for their first contribution!
v0.4.7
- Redirect to public Perf Page for Bencher Console links if user is not logged in
- Fix jittering in Perf Plot tabs when loading and changing pages
- Fix PR comment tags when changing branch start points
- Move benchmark results under a
details
tag for HTML comment output
- Add an Alert summary table for HTML comment output
- Add a warning if no Thresholds are found for HTML comment output
- Deprecated:
--ci-no-metrics
flag no longer effects HTML comment output
v0.4.6
- Add perf plot images as OpenGraph images
- Add GET
/v0/users
endpoint for server admins
- Improve perf plot query performance by 120x or more by adding indexes and a materialized view
v0.4.5
- Move Bencher Cloud out of BETA
- Add tab loading progress bar to perf pages
- Add
--branch-start-point
and --branch-start-point-hash
options to bencher run
to track rebasing and other branch changes
- Deprecated: The
bencher run
options and flags --if-branch
, --else-if-branch
, --else-branch
, and --endif-branch
have been deprecated in favor of using --branch
and --branch-start-point
- BREAKING CHANGE
--if-branch
will now hard fail instead of skipping the run if not specified or if an empty string is provided, and it will always be created even without --else-branch
set
- BREAKING CHANGE
--else-if-branch
will only use the first branch specified as the start point and it must exist, if it is a non-empty string
- Update branch date time plot to use local time and version plot to be discrete along the x-axis
v0.4.4
- Fix resource contention during large Perf queries
- Remove
public
query param from /v0/projects
endpoint
- BREAKING CHANGE Move all CLI commands using
--org
, --project
, and --user
over to positional arguments, except bencher run
and bencher threshold create
- Add a
bencher report create
CLI subcommand
bencher run
now tries to auto-fill the git commit hash for the report
- BREAKING CHANGE Rename
statistic
to model
and remove API endpoint
- Add “Click to view Threshold” for all boundary limits and add a public view Threshold page
- Add PATCH endpoint for user account settings
- BREAKING CHANGE Remove
/v0/server/ping
endpoint
- Deprecate
/v0/server/endpoint
in favor of /v0/server/config/console
and BREAKING CHANGE remove bencher server endpoint
CLI command
v0.4.3
- BREAKING CHANGE Limit branch names to 256 characters and all slug lengths to 64 characters. Any existing values greater than these new limits are set to their respective UUID.
- Add shortcut URLs for user pages:
/console/users/tokens
-> /console/users/USER-SLUG/tokens
/console/users/tokens/add
-> /console/users/USER-SLUG/tokens/add
/console/users/settings
-> /console/users/USER-SLUG/settings
- Improve
bencher run
error messages to include the command being run (idea credit to dholroyd: #335)
- Prevent pwn requests when using
bencher run
- Add an API version check to all Bencher CLI commands that use the backend
v0.4.2
- Fix Perf Plot view reduce bug
- Add a
View
button to Perf Plot for Reports, Branches, Testbeds, and Benchmarks
- Add
search
query param to GET
list endpoints for all named resources
- Add search for a named resources in Console UI and Perf Plot dimensions
- Add date based search for Reports in Console UI and Perf Plot
- Add search for public projects
- Move Bencher API server to “distroless” Docker image,
bencher-api
that is 10x smaller
- Move Bencher Console UI to “distroless” Docker image,
bencher-console
that is 1/3 smaller
- Fix Console server side rendering bug when using local URLs inside of Docker
- First time contributor JonasWanke added the Iai-Callgrind benchmark harness adapter
v0.4.1
- Add a “Show all”/“Hide all” toggle for perf plot results (idea credit to jerome-benoit)
- Add an
--exec
flag to bencher run
to run commands as executables (no shell) and by default when there is more than one argument
- Add new Thresholds: Static, Percentage, Log Normal, Interquartile Range, and Delta Interquartile Range
- Include Iteration number for Report results
- Redirect to newly created resources in Console UI
v0.4.0
- BREAKING CHANGES Remove backwards compatibility for renames:
Metric
(lower_bound
=> lower_value
and upper_bound
=> upper_value
)
Measure
(metric_kind
=> measure
in Report
and Threshold
; metric_kinds
=> measures
in Perf
)
Boundary
(average
=> baseline
)
- Add
bencher up
, bencher down
, and bencher logs
commands to CLI to easily run Bencher Self-Hosted
- Branches and Testbeds are now created if they do not exit for
bencher run
if the value passed is a slug or name (UUIDs are still required to already exist)
v0.3.26
- Add OAuth2 based login for GitHub
- Allow users to accept invitations while logged in
- Run smoke tests on Linux, MacOS, and Windows
- Release Linux, MacOS, and Windows CLI builds for both x86_64 and ARM64
- Add convenience script to install
bencher
CLI on Linux, MacOS, and Windows
v0.3.25
- Allow custom CI ID to be an unbounded length
v0.3.24
- Add URL fragment links to all headings 🔗
- Fix browser navigation to replace history for automatic query args, like pagination
- Scroll to top of page when changing paginated pages
- Allow Branch, Testbed, and Measure names to be used in addition to slugs and UUIDs for Report (
GET
list & POST
), Threshold (GET
list & POST
), and Branch (POST
start point) endpoints
- Validate Branch and Testbed exist before running benchmarks with
bencher run
v0.3.23
- Move CLI over to relaxed response parsing and add
--strict
flag to enable strict response parsing
- Add
--quiet
flag to bencher run
to enable quite mode and only output the final Report JSON
v0.3.22
- The Boundary Limits object is now optional for Perf query results and Reports
- Update all
POST
API endpoints with creation semantics to return a 201
status code
- Update all
PUT
and PATCH
API endpoints with immediate update semantics to return a 200
status code
- Update all
DELETE
API endpoints to return a 204
status code and an empty body
v0.3.21
- Add Hyperfine adapter
- Rename
Metric Kind
to Measure
v0.3.20
- Rename
average
to baseline
for the Boundary Limits
v0.3.19
- Include the
average
used for the Boundary Limits
- Remove
--ci-public-links
for bencher run
and simply use the project visibility to determine PR links
- Replace
--ci-with-metrics
with --ci-no-metrics
for bencher run
and make showing benchmark metrics and boundary limits on PRs the default
v0.3.18
- Add public Alert pages
- Add a
--ci-public-links
option to bencher run
to only post public links to PRs
- Add a
--ci-with-metrics
option to bencher run
to display benchmark metrics and boundary limits on PRs
- Update GitHub Actions PR documentation
v0.3.17
- Fix word break CSS in console UI
- Add status bar for perf plot loading
- Fix perf plot x-axis label
- Improve embedded perf plot layout
v0.3.16
- Improve
bencher_client
ergonomics for use in rustls
- Fix navbar bug on public pages while logged in
- Add Bencher Learn section to the site
- Set max perf query permutations to
255
- BREAKING CHANGE: Move from single
metric_kind
to array of metric_kinds
for perf plot query
v0.3.15
v0.3.14
- Move default Threshold Boundaries from
0.9
to 0.95
- Redirect to login for an authorized users in Console UI
- Improve API server error messages
- Fixed the UI y-axis scale issue… again
- Add Bencher Plus licenses
v0.3.13
- Add ability for
bencher run
to read output from a file without running a command argument
- Improve Report creation error messages
- Add filtering Reports by Branch, Testbed, and start/end time
- Add one month historical default when viewing Reports in the Perf Plot
- Add endpoint
/v0/server/spec
to view OpenAPI spec JSON
- Remove old UI source and scripts
v0.3.12
- Change Metric
lower_bound
and upper_bound
to lower_value
and upper_value
respectively
- Add ability to visualize
lower_value
and upper_value
in Perf Plot
v0.3.11
- Add strongly typed IDs for database entities
- Remove deprecated configuration keys (
endpoint
=> console.url
and secret_key
=> security.secret_key
)
v0.3.10
- Add optional error monitoring with Sentry
- Start API error message improvement
- Fix Threshold dimension selection bug in UI
- Display benchmark harness results in realtime via
bencher run
- A special thank you to @robertorosmaninho for bug hunting 👾
v0.3.9
- Move UI over to Astro with stronger TypeScript
- Add check for failing Netlify deployments
- Move default Threshold Boundaries from
0.95
to 0.9
v0.3.8
- Segment PR comments by the combination of Project, Branch, Testbed, and Adapter
- Add a custom PR comment ID argument
--ci-id
to bencher run
for complex scenarios
- Another special thank you to @GitGab19 for all the feedback
v0.3.7
- Fix slug length validation bug
- A special thank you to @GitGab19 for all the feedback
v0.3.6
- Fix multi-Metric Kind bug for Perf Plot Report selection
- Require minimum and maximum sample sizes to be greater than or equal to two
- Allow system Metric Kinds (ie Latency, Throughput, etc) to be deleted and recreated via a later Report
- Comment on GitHub pull requests using
--github-actions
argument for bencher run
- Only post CI integrations when a Threshold is set for a Metric Kind, Branch, and Testbed with
ci-only-thresholds
- Only start posting CI integrations when an Alert is generated with
--ci-only-on-alert
- Fix Threshold boundary calculation bug when standard deviation is zero
- Add a warning ⚠️ in the Perf Plot UI when a Boundary Limit is not calculated
- A special thank you to @untitaker for all the feedback
v0.3.5
- Visualize Threshold lower and upper boundaries and Alerts in Perf Plot
- Add ability to dismiss and reactivate Alerts
- Add ability to update Thresholds in the UI
- Fix Threshold
PUT
bug
- Add deletion capabilities to the UI
- Add help page to public UI
v0.3.4
- Revamp Quick Start tutorial
- Add Docker Self-Hosted tutorial
- Fix public perf pages bug
- Add Reports tab to perf pages
- Add ability to clear the perf plot
- Add ability to set Project URL to
null
- Add Project permissions API endpoint
- Remove redundant Project API endpoints
- Add ability to update Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks in UI
v0.3.3
- Move API server config key
endpoint
to console.url
- Change API from
/v0/server/config/endpoint
to /v0/server/endpoint
to get console.url
- Add dedicated
POST
API endpoint for Benchmarks
- Add
PATCH
API endpoints for Organizations, Projects, Metric Kinds, Branches, Testbeds, and Benchmarks
- Add
PUT
API endpoint for Thresholds
- Add
DELETE
API endpoint for Metric Kinds, Branches, Testbeds, Benchmarks, and Thresholds
- Create two default Thresholds for all new Projects
- Add a Go benchmark example
- Fix Go auxillary data parse error
v0.3.2
- Decrement all proceeding Branch versions after the final Report is deleted for a Branch version
- Add pagination to all GET list API endpoints,
bencher
CLI, and console UI
- Move all
bencher
CLI API calls to using client auto-generated from OpenAPI spec
- Improve
bencher
CLI error messages
v0.3.1
- Add
--backdate
argument to bencher run
for backdating Reports
- Add
created
and modified
timestamps as applicable to all entities and their JSON
- Add ability to delete a Report
- Add ability to delete a Project
- A special thank you to @wuputah and the Hydra team for all the feedback
v0.3.0
- Add demo video to landing page
- Add
bencher run
documentation
- Refactor Report, Alert, and Perf JSON to be more powerful
- All Report and Perf metrics now show the boundary limits used if a Threshold is set
- Display URL to visualize report in UI after
bencher run
- Change
magnitude
to pow
for bencher mock
- A very special thank you to @MichaelScofield for being Bencher’s second community contributor!
v0.2.46
- Update roadmap
- Add Iai benchmark harness adapter
- Move from GitPod over to Dev Containers/GitHub Codespaces
- Fix thresholds with no historical metrics bug
- A very special thank you to @osiewicz for being Bencher’s first community contributor!
v0.2.45
- Add servers to Swagger API docs
- Fix perf view refresh bug
- Add an order of
--magnitude
argument to bencher mock
command
- Add an API endpoint for threshold statistics and a CLI
statistic
subcommand
- Upgrade the
bencher perf
CLI flag --table
to accept an optional style parameter
- Fix long text wrapping in UI deck view, both header and card contents
- Autogenerate TypeScript types from Rust type definitions
- Massive thank you to @vient for all the feedback
v0.2.44
- Fix UI expired token error
- Move the CLI linker over to use Zig in order to pin a specific
glibc
version
- Move linker in CI over to
mold
for increased parallelism
- Fix cargo workspace
default-features = false
warnings
- Add project stewardship commitments
- Massive thank you to @brainstorm for all the feedback
v0.2.43
- Add a
--table
option to the bencher perf
command to display query results as a table
- Add the ability to toggle the perf plot x-axis from report date time to branch version number
v0.2.42
- Create and publish Bencher CLI GitHub Action
- Fix both UI and perf image y-axis sizing once and for all… hopefully
v0.2.41
- Move perf plot images from a screenshot to server side rendering
- Add a share mechanism to perf pages with a perf image preview
v0.2.40
- Add an optional
--file
argument to bencher run
to read file output
- Move perf query API endpoint from a POST to GET
- Add an endpoint to GET a screenshot of a public perf plot
v0.2.39
- Update readme to add Discord invite
- Allow certain benchmarks to be ignored by Threshold checks
- BREAKING CHANGE: Make Project names unique within an organization and delete existing duplicates
- Add paid plans and metered billing for Bencher Cloud
- Change perf pages to use milliseconds as opposed to date/time stamp
v0.2.38
- Fix CLI runtime dependency error over the past five releases
- BREAKING CHANGE: Make Metric Kind, Branch, and Testbed names unique and delete existing duplicates
- Add a soft branch creation option to eliminate race conditions in CI
- Add supported languages to landing page
- Lots of new docs!
v0.2.33
- Add new
average
setting for adapters to choose between mean/standard deviation and median/interquartile range when possible
- Increase CLI default retry to ten attempts three seconds apart (30 second window)
- Add ability to clone thresholds when creating a new branch from a start point and make this the default behavior for
--else-if-branch
in bencher run
CLI
- Moved the plot key to below the plot all the time in preparation for screenshots/embedding
v0.2.32
- Add STARTTLS option for SMTP connections
- Add
throughput
as a new default metric kind with operations / second (ops/s)
as units
- Add Java JMH, C# BenchmarkDotNet, JavaScript Benchmark.js, JavaScript console.time/timeEnd, Python airspeed velocity, Python pytest-benchmark, and Ruby Benchmark adapters
- Add resend email button to UI auth confirm page
- Change C++ Catch2 adapter to use standard deviation for
lower_bound
and upper_bound
- Fix table and perf panel formatting for long, single word values
v0.2.31
- Prevent leaking secrets by obfuscating them in the server logs (ie
************
)
- Move run failure detection to the CLI, adapter parsing will no longer attempt error detection
- New harness specific adapter targets (ie
rust_bench
and rust_criterion
)
- Add additional options to
bencher mock
, --fail
and --flaky
to easily test parsing failing and flaky benchmarks respectively
- Wrote continuous benchmarking and how to install the CLI pages
- Update server framework, dropshot, and Rust to
1.67.0
- Two new C++ adapters for Google Benchmark and Catch2
- Add an adapter for
go test -bench
v0.2.30
- Added
--if-else-branch
, --else-branch
, and --endif-branch
flags to to bencher run
CLI
- Wrote a branch management explanation page for using
bencher run
- Completely refactored docs UI configuration to make it easier to add new pages, one place instead of three
- Made the Diataxis convention explicit in the docs
- Fixed bounds bug in
bencher mock
- Added a
/v0/server/backup
route and bencher server backup
CLI command to perform database backups with options to compress, push to a remote data store (currently only AWS S3), and remove the local copy
v0.2.29
- Fixed and refixed Criterion empty benchmark name bug
- Type level checking for benchmark names and metric kind resource IDs
- Added documentation: Quick Start, Prior Art, and GitHub Actions how to
- Prettier and eslint added to CI/CD
- Stubbed out UI tests with
vitest
in CI/CD
v0.2.28
- Centralized all validation logic between frontend and backend via Rust + WASM modules
- All pages in the UI now have basic functionality
- Added public project perf pages
v0.2.27
- Fixed the UI y-axis scale cutoff issue! Note to self: RTFD
- Fixed the UI documentation + routing bug! It turns out
SwaggerUIBundle
was the solution.
- Added Metric Kind list, add, and view to UI
- In perf queries allowed for both UUIDs and slugs for branches and testbeds
- Added bencher.dev to Google Search Console and Analytics
- Changed API server default URL to always be localhost:3000
- Moved Docker UI build stage into its own
builder.Dockerfile
, mirroring the API setup
- Created a better default for the UI’s backend API URL, basically
this-host:61016
- Created a walking skeleton of adding Rust + WASM modules to the UI in both dev and CI/CD
- Fixed UI logout + routing “too many redirects” bug
- Changed notifications from state based to URL query param based in the UI
- Refactored how pathname and redirects are handled in the UI to increase encapsulation
- Added the current version to the UI
BETA
tag
- Created a repo shortcut at bencher.dev/repo
Published: Fri, October 27, 2023 at 8:40:00 AM UTC | Last Updated: Wed, April 3, 2024 at 11:23:00 AM UTC