Arte Anterior

Benchmark Tracking Tools

General
- Conbench - Language-independent Continuous Benchmarking (CB) Framework
- Continuous Benchmark - GitHub Action for continuous benchmarking to keep performance
- Horreum - Benchmark results repository service
- ReBench - Execute and document benchmarks reproducibly
Benchmark Harness Specific
- airspeed-velocity/asv - Airspeed Velocity: A simple Python benchmarking tool with web-based reporting
- bobheadxi/gobenchdata - Run Go benchmarks, publish results to an interactive web app, and check for performance regressions in your pull requests
- codingberg/benchgraph - Visualization of Golang benchmark output using Google charts
- cob - Continuous Benchmark for Go Project
- Criterion.rs - Statistics-driven benchmarking library for Rust
- Cybench - Continuous performance regression testing for Java CI/CD pipelines
- dandavison/chronologer - Visualize performance benchmarks over git commit history
- distributed-system-analysis/pbench - A benchmarking and performance analysis framework
- JfrUnit - A JUnit extension for asserting JDK Flight Recorder events
- jsperf/jsperf.com - jsPerf aims to provide an easy way to create and share test cases, comparing the performance of different JavaScript snippets by running benchmarks.
- NimbleDroid - Functional Performance Testing for Android & iOS
- novadiscovery/benchgraph - A lightewight tool for visualizing your benchmarks history
- OctoPerf - Simplify your load testing experience
- ocurrent/current-bench - Experimental continuous benchmarking infrastructure using OCurrent pipelines
- OpenBenchmarking.org - Storage of Phoronix Test Suite benchmark result data (including optional system logs, etc)
- Orijtech bencher - Continuous benchmarking for the Go programming language
- tobami/codespeed - A web application to monitor and analyze the performance of your code
- python/codespeed - A fork of Codespeed that includes the instances run at [[https://speed.python.org/]] and [[https://speed.pypy.org]]
- Unity-Technologies/PerformanceBenchmarkReporter - Establish benchmark samples and measurements using the Performance Testing package, then use these benchmark values to compare subsequent performance test results in an html output utilizing graphical visualizations
- BenchHub - A service for running database benchmarks and saving the result
- bench-bot - Benchmark runner robot. Continuous benchmarking for benchmarkify the benchmark framework for NodeJS
- CodSpeed - CodSpeed provides integrated CI tools for software engineering teams to anticipate the impacts of the next delivery on system performance.
- OpenEBench - OpenEBench is the ELIXIR gateway to benchmarking communities, software monitoring, and quality metrics for life sciences tools and workflows.
- callstack/reassure - Performance testing companion for React and React Native
- bamlab/flashlight - Flashlight is a Lighthouse-like tool for mobile apps. No installation required.
- Perfbench - Perfbench is an interactive online C++ code profiling tool.
Web Specific
- Contentsquare - Speed Analysis is not only about Synthetic Monitoring but also offers powerful Real User Monitoring capabilities, a great fit for brands leading the way on customer experience across the world.
- Iron/Out - Faster websites equals better business results: improve user experience, increase conversion rate and page experience ranking, lower the bounce rate
- MeasureWorks - Track user behavior with End-2-End observability to continuously optimize your online performance
- SpeedCurve - See how people experience the speed of your website, then identify and fix performance issues
- UBenchan - Beautiful browser benchmarks
- WebPageTest - Run a free website speed test from around the globe using real browsers at consumer connection speeds with detailed optimization recommendations
Project Specific
- BrowserBench - Speedometer is a browser benchmark that measures the responsiveness of web applications
- Elasticsearch Benchmarks - The results of the Elasticsearch nightly benchmarks based on the main branch as of that point in time
- deno.land benchmarks - As part of Deno’s continuous integration and testing pipeline we measure the performance of certain key metrics of the runtime. You can view these benchmarks here.
- golang/benchmarks - Benchmarks for the Go perf dashboard
- Lucene Nightly Benchmarks - Each night, an automated Python tool checks out the Lucene/Solr trunk source code and runs multiple benchmarks
- python/pyperformance - Python Performance Benchmark Suite
- pytorch/benchmark - TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.
- PyPy Speed Center - A fork of Codespeed of PyPy
- Python Speed Center - A fork of Codespeed for Python
- rust-lang/rustc-perf - Website for graphing performance of rustc
- Shopify/yjit-bench - Set of benchmarks for the YJIT CRuby JIT compiler and other Ruby implementations seen at speed.yjit.org
- Skia Perf - Skia Perf is a web application for analyzing and viewing performance metrics produced by Skia’s testing infrastructure
- Supabench - App by Supabase team to run benchmarks automatically in CI
- Vitess Are We Fast Yet - Automated benchmarking system that tests Vitess’s performance on a nightly basis
- Benchmark for Parse Server - Parse Server Continuous Benchmark
- Google Chrome V8 - CSuite: Local benchmarking for V8 performance analysis
- diesel-rs/metrics - The numbers collected by diesel continuous scheduled benchmark actions to track changes over time

🐰 A special thank you to Continuous Benchmark! Some of their test files are used for our Benchmark Harness Adapters.

Benchmark Tracking Posts

Benchmark Comparisons

CH-benCHmark - Operational and real-time Business Intelligence (BI) mixed workload SQL benchmarks
denosaurs/bench - Comparing deno & node HTTP frameworks
HewlettPackard/netperf - Netperf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput, and end-to-end latency
krausest/js-framework-benchmark - A comparison of the performance of a few popular javascript frameworks
TechEmpower/FrameworkBenchmarks - In the following tests, we have measured the performance of several web application platforms, full-stack frameworks, and micro-frameworks (collectively, “frameworks”).
The Computer Language Benchmarks Game - Which programming language is fastest?
Programming Language Benchmarks - Yet another implementation of computer language benchmarks game
Top500
PerfKit Benchmarker (PKB) - A set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see.

Benchmark Harnesses

General
- ApacheBench - A tool for benchmarking your Hypertext Transfer Protocol (HTTP) server
- bench
- hyperfine - A command-line benchmarking tool
- Phoronix Test Suite - Open-Source, Automated Benchmarking
Android
- Macrobenchmark - Writing a Macrobenchmark
- Microbenchmark - Microbenchmark Overview
C
- Linux kernel perf bench - This command includes a number of multi-threaded microbenchmarks to exercise different subsystems in the Linux kernel and system calls.
- MMTests - Benchmarking framework primarily aimed at Linux kernel testing
- Sysbench - Scriptable database and system performance benchmark
- LEBench - An analysis of performance evolution of Linux’s core operations
C++
- catchorg/Catch2 - A modern, C++-native, test framework for unit-tests, TDD, and BDD - using C++14, C++17, and later
- google/benchmark - A microbenchmark support library for C++
C#
- donet/BenchmarkDotNet - Powerful .NET library for benchmarking
- Unity-Technologies/PerformanceBenchmarkReporter - Establish benchmark samples and measurements using the Performance Testing package, then use these benchmark values to compare subsequent performance test results in an html output utilizing graphical visualizations
- xunit/xunit - xUnit.net is a free, open source, community-focused unit testing tool for .NET
Elixir
- alco/benchfella - Microbenchmarking tool for Elixir
- bencheeorg/benchee - Easy and extensible benchmarking in Elixir providing you with lots of statistics!
Go
- Benchmarks - Go test benchmarks
Haskell
- haskell/criterion - A powerful but simple library for measuring the performance of Haskell code
Java
- openjdk/jmh - JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM
- JfrUnit - A JUnit extension for asserting JDK Flight Recorder events
JavaScript
- bestiejs/benchmark.js - A benchmarking library. As used on jsPerf.com
- console.time/console.timeEnd - A method to start/stop a timer you can use to track how long an operation takes
- Node.js Performance Measurement API - This module provides an implementation of a subset of the W3C Web Performance APIs as well as additional APIs for Node.js-specific performance measurements.
- tinylibs/tinybench - A simple, tiny and lightweight benchmarking library!
- vitest bench - vitest bench uses tinybench library under the hood
- deno bench - Deno has a built-in benchmark runner that you can use for checking performance of JavaScript or TypeScript code.
- callstack/reassure - Performance testing companion for React and React Native
Julia
- BenchmarkTools.jl - A benchmarking framework for the Julia language
Python
- airspeed-velocity/asv - Airspeed Velocity: A simple Python benchmarking tool with web-based reporting
- pytest-benchmark - py.test fixture for benchmarking code
- timeit - This module provides a simple way to time small bits of Python code
Ruby
- ruby/benchmark - Methods for benchmarking Ruby code, giving detailed reports on the time taken for each task
Rust
- bheisler/criterion.rs - Statistics-driven benchmarking library for Rust
- bheisler/iai - Experimental one-shot benchmarking/profiling harness for Rust
- bluss/bencher - bencher is just a port of the libtest (unstable) benchmark runner to Rust stable releases. cargo bench on stable. “Not a better bencher!” = No feature development. Go build a better stable benchmarking library.
- BurntSushi/cargo-benchcmp - A small utility to compare Rust micro-benchmarks
- jbreitbart/criterion-perf-events - A plugin for Criterion.rs to measure Linux perf events
- libtest bench - The libtest harness supports running benchmarks for functions annotated with the #[bench] attribute. Benchmarks are currently unstable, and only available on the nightly channel.
- bazhenov/tango - Rust pairwise microbenchmarking harness
Swift
- apple/swift - The Swift Benchmark Suite
- google/swift-benchmark - A swift library to benchmark code snippets

Test Reliability

Tools
- Buildkite Test Analytics - Identify wasted compute, improve reliability, and deploy with confidence using Buildkite Test Analytics, an observability platform for your test suites.
- trytouca/trytouca - Open Source Continuous Regression Testing for Engineering Teams
- Unflakable - Flaky Tests Are Inevitable. Keep Builds Green While You Fix Them.
Posts