Building Bencher in 2026: Open Source, Bootstrapped, and Agent Pilled
Everett Pompeii
This essay was originally submitted to The Pragmatic Engineerās How Software Engineering is Changing Essay Challenge on 04 October 2026.
This year has marked a radical shift in the way that I build software. Iām a staff software engineer. In the past six years, Iāve worked professionally in Rust for both a YC and a16z backed unicorn and a Fortune 500 company. As a recovering C++ developer, I found myself drawn to the depth and skill Rust required. I had become an expert craftsman at hand chiseling real-time systems software. In the past ten months, Iāve had to bury that identity. While my job title still says āsoftware engineerā, my job is now that of a ācontext engineerā.
Over the past four years, Iāve spent my nights and weekends building an open source, bootstrapped business, Bencher. Bencher is a continuous benchmarking platform that helps systems software engineers catch performance regressions before they merge. It takes a suite of tools to accomplish this: a CLI, an API server, and a frontend web console UI. Each of these components was written, within a rounding error, by me. Iāve had a few notable open source contributors over the years, but the vast majority of Bencher the open source project and 100% of Bencher the business was designed and implemented by myself.
In these four years, I didnāt just live under a rock. I got engaged, married, and had my first child. It would take weeks and sometimes months for me to work through a large feature implementation and roll it out. I had to juggle both development and support. Bencher already had many users, so I would spend quite a bit of time answering questions and fixing the occasional bug.
Back in 2022 my use of LLMs consisted of the Rust freeware version of TabNine, a tab to autocomplete developer tool built by a student and early intern at OpenAI working on this funny little project named Codex. Once GitHub Copilot launched and unofficially added Rust support, I ponied up the $10/month for a personal license.
It was rough. But it still felt a bit like magic. I remember being wowed shortly after chat was added to the GitHub Copilot VSCode extension by how good it was at creating Rust procedural macros. Usually I had to tweak the output some, but a few times, it managed to oneshot it. Bananas! This was cool and probably helped save some time, but I still predominantly worked in the same way as before.
From that point onward, I had this mindset that Microsoft was in the vanguard with GitHub Copilot. Even if these newfangled labs were throwing āagentsā out there, I could trust that GitHub Copilot would keep up if there was anything worthwhile. I would hear folks like Dan Shipper on Lennyās Podcast, and their telling of the modelās capabilities just did not square with my experience nor quality bar.
Early last year, I wrote a post reflecting on three years of Bencher development, Engineering Review: 2025 Edition. In that post I covered all of my major engineering decisions to build Bencher spanning frontend, backend, developer environment, CI, you name it. I spent an entire paragraph venting about how hard of a time I had trying to wrap my head around CUE lang two years prior. Yet, thereās not a single mention of AI, LLMs, or even Copilot anywhere in that 3,500 word write up.
Fast forward to less than a year later, and Iām commiserating with We mourn our craft.
Claude Helps Save Christmas
Like many others, over the 2025 winter holiday I had a chance to really use Claude Opus 4.5. I was in London visiting my sisterās family with my wife and child. My sister had recently given birth to twins, so I was supposed to be there to help out. Instead, I was hunkered down upstairs trying to fight off email spam and DDoS attacks against Bencher. Claude was super helpful in diagnosing what was happening and ideating possible solutions.
I ended up migrating Bencher behind Cloudflare and setting up reCAPTCHA. It was still me doing most of the work, but Opus 4.5 was a noticeable step up in its problem solving capabilities across many competing design constraints. Though, it did suggest that I make a change that bricked the convenience script for installing the Bencher CLI for half a day. Oops.
New Year, New Harness
By January of this year, Iād been hearing about how impressive Claude Code was for months. Iād held off trying it previously because Opus 4 just didnāt meet my quality bar to trust its work. Now that Opus 4.5 had met that bar, I wanted to give Claude Code a try as well. The level of improvement I saw going from using Copilot as a harness to Claude Code was just as large of a leap as going from using Opus 4/GPT-5.2/Gemini 2.5 Pro as a model to Opus 4.5.
My very first weekend with Claude Code I did a not so controlled test against Copilot. I wanted Copilot to turn a PDF into an audiobook. Even in āAgent Modeā Copilot would get five or six steps in and then just stop until I poked it again. At the same time, Claude Code was cranking away trying to build a custom hypervisor, after going through a thorough Plan mode. I became absolutely convinced that the level of abstraction had moved up from the model to the harness and that Copilot was never going to catch up again.
Dreaming of Bencher Bare Metal
Since the very early days of Bencher, I had been dreaming of building a bare metal benchmarking fleet. Traditional CI runners, by virtue of being virtual machines, are too noisy to give you consistent benchmark results. You could detect say a 200% increase in latency, but you would never be able to detect a 20% increase reliably. Instruction counts are one possibility for benchmarking in CI. They are mostly deterministic. However, instruction counts are a proxy for the thing you actually care about, wall-clock time. Anything that runs in a āCPU simulationā and splits out instruction counts can only correlate with speed. The problem is you never know by how much or how little it is off by unless you actually measure on bare metal.
One of the very first users of Bencher was the Rustls project. I helped contribute to their Rustls bench runner. However, this was not a good foundation to build a bare metal benchmarking fleet on top of. It sat on a single host and treated all benchmark workloads as trusted. That means no PRs from forks and no multitenancy. Plus, most teams donāt want to pay $100+ per month per bare metal benchmarking server.
I had considered using GitHub Actions for the orchestration layer, but there were a few fundamental shortcomings of this approach. To start, the GitHub Actions Runner is written in C#. This means its garbage collector could kick on in the middle of your run, skewing your benchmark results. Oh and make sure to turn the auto-updater off so that it doesnāt kick on mid-run, but donāt forget about it for too long otherwise your jobs will stop running!
Even if the GitHub Actions Runner was a nice, quiet Rust binary the more fundamental issue was that it was tied to GitHub Actions. Bencher since the beginning has been CLI first. For better or worse I have not wanted to tie Bencher too deeply to any one CI provider. Thereās a Bencher CLI GitHub Action, but all that does is install the CLI for you. I wanted Bencher Bare Metal to work from anywhere the CLI does, not just GitHub Actions.
The idea I came up with was using OCI images to bundle up the benchmarking workload and an optional, very small sandbox to only ever run one job at a time on a particular bare metal host. The prospect of building this out on my own over nights and weekends was daunting. With a day job and a family, it would have taken at least a year chipping away at it by myself.
Building Bencher Bare Metal
With Opus 4.5 and Claude Code in-hand in January of this year, I felt like I could finally see the practical path forward for building out Bencher Bare Metal. I once again ponied up, this time to the tune of $200/month for the Claude Max 20x plan. Probably the best business expense ever.
I started to absolutely crank and tried my best to max out my Claude plan usage each week. I stopped handwriting code, both on Bencher and at my day job. At this point I was still iterating in pretty tight loops, using Plan mode, reading all of the output, and running in Accept edits mode. I didnāt trust the agent to touch git/jj at this point, so I would handle all version control myself. I thought of myself as a human-in-the-loop Ralph Loop. At about five agents running in parallel, my brain would start to melt.
One suggestion that I had heard was to have separate agents do the implementation and the review, so I started to do separate adversarial reviews. These adversarial reviews really seemed to help improve the quality of the output. I created my own GitHub Actions review bot that used my Claude Code subscription API key. I would then have my local agent iterate with this CI reviewer in order to save cycles on my local machine.
At one point in there, I thought I was going to have Claude build its own hypervisor for the benchmarking sandbox. I then asked Claude to do an adversarial review of the plan as Linus Torvalds, and it told me I was stupid and this was a very, very, very dumb idea. No sycophancy when Linus is in your context! So I scrapped that plan and went with Firecracker.
The enemy of good benchmarking is variance, not speed per se. So even though Firecracker adds overhead as a hypervisor it is very consistent overhead, especially with only one microVM ever running at a time. This allows shared Bencher Bare Metal runners to have both multitenancy and very consistent benchmarking results. I also worked with Claude using Plan mode to design things so that the sandbox was optional. If a user wants to run on their own dedicated server where they do trust every workload, then they can run without Firecracker with near zero overhead.
Developing the Bare Metal runner was a big learning lesson for me about how important it is for agents to have a tight feedback loop. Even after Opus 4.6 came out, Claude seemed to be struggling to get everything working consistently. Once I gave it access to a development environment with KVM enabled (which is required to run Firecracker), things really started to click. These agents arenāt omniscient, theyāre just really persistent.
The level of scrutiny that I put Claudeās work under depended on the part of the code base it was working in. The code that was purely for Bare Metal such as the OCI registry and the runner binary I only lightly reviewed. As long as it passed adversarial review, the end-to-end tests passed, and things didnāt look too terrible, I merged it. For the code that touched the existing core, job queuing and report processing, I was as nitpicky as ever.
At the end of March this year, I released Bencher Bare Metal as a part of v0.6.0. For the public bare metal fleet, I was considering using AWS EC2 Bare Metal. However, these machines are both massive and ARM only. To match most real world workloads, I wanted working class Intel x86 machines. I researched with Claude to find the best home for the fleet that would meet the design constraints. We ended up being able to offer the intel-v1 Spec on Bencher Cloud that gives users 4 vCPUs, 48 GB memory, and 128 GB disk billed to the minute. On well structured benchmarks, I was consistently getting <2% variance between runs across the fleet.
Agent Authentication
Up until Bencher Bare Metal, all authentication had been JWTs. Big, beautiful, incomprehensible JWTs. The authentication for Bare Metal runners was a name prefixed API key, ex bencher_runner_abc123....
On the client side, I wanted to give agents something to work with other than a userās own API token JWT. The middle ground that I struck was finally implementing a long requested feature, project-scoped authentication. Claude helped me create secure project-scoped, prefixed API keys. These API keys are limited to the permissions equivalent to what the bencher run CLI command performs, project-scoped reads and non-destructive writes. This also makes these API keys a natural fit for use in continuous benchmarking CI workflows.
Later when I added the Bencher MCP, these project-scoped API keys were also used for the authentication there. The MCP standard names OAuth 2.1, however it is still in draft, so I didnāt want to implement it yet. Iām honestly a bit of an MCP skeptic. I still think a good CLI + skills is a more effective and portable way to expose a service to an agent. The biggest issue with CLIs though is their lack of a standard authentication mechanism. So even though OAuth 2.1 is still in draft, at least MCP has that covered. And a good authn story really matters to enterprise.
Claude the Polyglot
Back in 2023, I decided that I wanted the Bencher docs to be available in 9 languages: English, Chinese, Spanish, Brazilian Portuguese, Russian, Japanese, French, German, and Korean. How did I pick those nine languages? Theyāre the languages the GitHub docs were in, so I figured that was a good enough approximation. The only problem is Iām only fluent in English. So I had GPT-4 translate the docs for me. At the time GPT-4 only had a context window of 8k tokens. This meant that I had to break down all of my doc pages into āchunksā under 8k tokens in size. Then I had to create a CLI tool to handle translating and very often retranslating chunks after they had been updated.
This was a very tedious and error prone process. I would have to hand edit the translated files because even with some very explicit system prompt instructions the model would still botch import paths or stick everything inside of another layer of a Markdown block, especially in Japanese for some reason.
Once I started to use Claude Code with Opus 4.5, this all went away. Now the agent just does it. The convention for what constitutes a chunk is there in the codebase, and Claude just follows it. When I was writing the technical documentation for Bencher Bare Metal, Claude just handled the chunking and orchestrating the translations. Even though the context size is now more than large enough to handle an entire page, I like having the chunks. It helps keep changes isolated, so only the relevant sections of a page get retranslated during a revision.
Who Didnāt Build a Harness in 2026?
While building Bencher Bare Metal, I also had Claude create tooling for itself to manage the fleet. This consisted of two parts, a CLI in the bencher repo for routine tasks and a new meta-repository (meta-repo). This meta-repo holds all of the docs and context for Bencher as a product. Every other repo in the bencherdev GitHub organization is a submodule in this meta-repo. The meta-repo allows the agent to easily work on a task that may span multiple repositories and write docs that I would prefer not to be public (product strategy, incident reports, etc).
I wanted to build my own agent on top of the Claude Code CLI that would take this meta-repo from an artifact to a proactive part of the development cycle. Once Fable 5 became available, I put a lot of my focus into building this agent. I wanted to be able to trust this agent to run autonomously. A big part of this was security (ie mitigating prompt injections). I designed it so that worker agents never hold any secrets. All secrets are injected by a network proxy and restricted to allow-listed endpoints. The agents themselves ran in yet another Firecracker microVM, this time with parallelism. The agent control box and the network proxy + sandboxes ran on completely different hosts.
My grand vision was for the interface to be a queue ranked by task priority. That is, the goal of this agent would be to show me the most important task in that given moment. Where the Facebook/Instagram/Twitter/X/TikTok feeds are optimized for engagement, this feed would be optimized for importance to me.
I wanted to create the tool that would put myself below the API. I had found that I spent most of my time context switching and hopping between threads. My hope was that the agent could help alleviate this juggling by rank ordering things for me. With a Reddit-like up vote/down vote system, I could train the agent to know whether that new Slack message was really the most important thing for me to pay attention to.
Iāve also found that a lot of my job now is being my agentās Rubeus Hagrid, its ākeeper of the keysā. Injection at the network proxy level with a restricted endpoint allow-list was the best way that Claude and I could come up with to both keep the agent from ever having access to a secret and still being able to use tools like the gh CLI out of the box.
The other headline goal was to never leave the agent interface. Navigating away from the agent for any reason is seen as a failure of the harness. I created a procedural UI component system so that the agent working on a task could build a novel interface on the fly or choose one of the existing components in the library based on the task at hand. Code, email, Slack, calendar, production telemetry data, error monitoring, everything all in one interface, all in one prioritized queue.
I got a minimal version of all of this up and running. However, thereās still a lot of rough edges. Iām currently much more productive if I just use the Claude desktop app. And at the present moment, I need as much immediate productivity as I can get to keep up with demand.
What happens when the number of developers 100x?
Back in 2022, I was looking for a new side project. I had shutdown my previous wantrepreneurial endeavor back in 2020. LPT: Listen to Jason Cohen, donāt try to bootstrap a marketplace.
I was building a static bash parser and analyzer in Rust at my day job, and I found I was having to use a spreadsheet to track my benchmark results. The Rust project has rustc-perf to track the performance of the compiler itself. Yet, there wasnāt a good tool to do this on my own code.
I did a lot of research into prior art within the field. Over the course of a few months, I had calls with several developers who had either built or advocated building continuous benchmarking tools. Everyone was reinventing the wheel. I thought I had found a nice quiet node in the water of the marketplace. There was an obvious, ongoing need. Many teams were already paying more than $100/month for their own bare metal runners. So there was willingness to pay. Yet, the use case was niche. Most modern software is deployed as a service, so shifting performance testing left wasnāt as much of a priority for most teams. Theyād just set up alerts in Grafana post deploy and post hoc catch anything that slipped through.
With highly capable agents coming online though, two factors have greatly increased the usage of Bencher:
- The need to adopt software engineering best practices in order to deterministically test the output of agents
- The ease of adopting software engineering best practices with the help of agents
LLMs and by extension agents are nondeterministic, stochastic systems. Like human developers, they need guardrails in order to make sure they maintain code quality. The ways to do this are not new: a robust type system, a standard formatter, a linter turned up to 11, tests, tests, and more tests. All run in CI. For performance sensitive workloads, these tests include benchmarks that are tracked. Tracking software performance over time allows you to catch performance regressions in CI. This is called continuous benchmarking. Implementing continuous benchmarking was always a bit of a chore. Only teams that really, really cared about performance would do it, and they made great Bencher users!
With the advent of agents though, the overhead of adding continuous benchmarking to your workflow using Bencher became just a short prompt away. Itās hard to track completely, but I would wager the vast majority of Bencherās inbound signups now come from the recommendation of an agent or an agent doing it themselves. This will only become more true over time, as more developers adopt agents and empower them with agentic commerce. It may or may not be the x402 protocol, but I think a lot of devtool spend in the future will be mediated by agents. I recommend that you reserve your Cloudflare Wallet and Stripe Wallet, if you havenāt already.
Nearly all devtools will need to move to usage-based pricing in the agentic era. Seat-based pricing does not make sense when agents are the ones using your service. Bencher has always had usage-based pricing. However, with the help of Claude I recently revamped the pricing model to be even more agent friendly. Previously, Bencher would charge per Metric. A Metric is a single point-in-time benchmark result. For example, if you have 1 benchmark and you run it 5 times, that creates 5 Metrics. Claude helped me analyze user feedback and usage patterns. I came to the conclusion that the unit we were using for our usage-based pricing model needed to change. Now Bencher charges per Benchmark Series. A Benchmark Series is a benchmark tracked over time. Going back to our example, if you have 1 benchmark and you run it 5 times, that is 1 Benchmark Series. This new pricing model gives users a lot more predictability. You can hand an agent an optimization problem, and it can hill climb on your historical benchmarks without incurring any additional cost.
One of the things that I have not seen change this past year are open source contributions. Bencher has received a couple of AI slop PRs. The main issue with these PRs was that they fixed random, unrelated bugs they happened to find along the way. They were still genuine bugs though! So I just sort of shrugged and merged the PRs after having my own agent review them of course. It is a bit sad, but I no longer see much benefit to courting open source contributors. Whether itās their tokens or mine, I still need to put the same amount of effort into reviewing the code. Itās now much more energy to review a contributorās PR than to iterate with Claude myself.
What is the future of Continuous Benchmarking?
When trying to divine what the future holds, the pattern seems to be that we should look at what the frontier labs themselves are doing. Weāll all be there in six to twelve months it seems. The Pragmatic Engineerās Inside OpenAIās agentic software factory shines an interesting light on this from a continuous benchmarking perspective:
New: a āperf harnessā: the agent also uses a perf harness to send problematic PRs to the Synthetics A/B framework for evaluating performance implications. As mentioned above, the load upon CI systems has increased greatly in the past six months.
OpenAIās āperf harnessā seems very similar to the established practice of relative continuous benchmarking. The novel AI piece is the synthetic part of the A/B framework. Synthetic test data is not new. But being able to autonomously shape those synthetic data to evaluate the performance of the changeset at hand certainly is new! Getting a developer to actually write the benchmarks has always been the struggle. Now itās automated. In the same way that agents have drastically increased feature test coverage, theyāre also enabling us to drastically increase performance test coverage.
OpenAI is not alone in this pursuit. Anthropic soon thereafter published How we made claude.ai 3x faster in two weeks. In it they also describe building benchmarks on the fly. One key difference was that Anthropic found deterministic counts that correlate with performance: instructions, commits per interaction, function calls, style recalculations, and DOM mutations. Claude then hill climbed these counts to optimize the performance of their app. At the end of the optimization pass, they checked that their count based optimizations actually resulted in real-world wall-clock time improvements.
These methodologies line up with what Iāve personally been experiencing working on Bencher. Though, my A/B performance tests have all been run locally or on an agentās work machine, not in a CI loop. Whenever Iām about to make a large change to a query or a meaningful schema migration, I have Claude do an exhaustive performance evaluation of the change. This always includes wall-clock time and occasionally some corollary deterministic counts (ex SQLite virtual machine instructions).
Due to noisy neighbor issues, these agent led performance optimization passes have had pretty wide error bars on them. Iām looking to add fixtures (additional benchmark inputs) and artifacts (additional benchmark outputs) to Bencher Bare Metal in order to better enable my agent to run these performance optimization tasks with a higher confidence. The agent will still do most of its optimization work locally, hill climbing the best it can on wall-clock time or deterministic counts. Then itāll use Bencher Bare Metal to check its work, as a sort of wall-clock caliper.
Even though I think most āusersā of Bencher will be agents within a year or so, I still think most humans will want a visual interface for when they check in. Iām currently neck deep in a total redesign of the Bencher UI with the help of Claude. As a Rust engineer writing Anyscript during the days of GitHub Copilot tab autocomplete, the current UI leaves much to be desired. The UI rewrite is blocked on another even more ambitious overhaul though, a new version of Bencher Metric Format (BMF).
BMF is the core format that benchmark results are normalized to within Bencher. It runs through everything. This new version of BMF will allow for benchmarks to be parameterized. Youāll be able to say my_benchmark with the parameters size: 16 MB and size: 32 MB, instead of having two separate benchmarks, my_benchmark_16_mb and my_benchmark_32_mb. The new version of BMF will also allow for arbitrary metric names. Youāll be able to provide mean, p95, and p99 metrics for your Latency measurements, instead of just having mean or needing to make Latency (mean), Latency (p95), and Latency (p99) as separate measurements.
Supporting this new BMF version is a massive overhaul to the core of Bencher. I would not have felt comfortable doing this work with anything prior to Fable 5. As discussed above, most of the Bare Metal work sat orthogonal to the core of Bencher: tracking performance over time and alerting on regressions. BMF is central to that core value proposition. My trust in Fable 5 and even more so Fable 5.1 and Opus 5.5 has again allowed me to start on a project that would have taken years of nights and weekends to complete. I previously had to spend quite a bit of time optimizing SQLite for Bencher. Now every time I evolve the model, I just ask Claude to spin up an agent to do a thorough performance investigation to make sure everything is still performant. In the future, I donāt even think Iāll have to ask.
How I Code Today
Back when I was using GitHub Copilot, it was an extension in VSCode. When I first switched to Claude Code, I was using the CLI. I would have three terminal windows each in their own macOS Space. Each terminal window would have two tabs. One tab was for Claude the other was for Jujutsu (jj). I was still managing the version control and jj workspaces myself.
Nowadays, I just use the Claude Code desktop app. Every new session that I start is from my meta-repo with the worktree checkbox enabled. I stopped using Plan mode a few months back. I never installed Matt Pocockās /grill-me skill. I just ask Claude to āgrill me on all design decisionsā in the prompt, and at this point the model knows what to do. My CLAUDE.mdAGENTS.md is pretty well tuned to how I want the model to behave.
Iāve all but decommissioned my Claude Review CI workflow. The agent now handles spinning up an adversarial reviewer locally without me having to even ask. Then they iterate until they converge, sometimes up to a half dozen rounds. Iāve started to experiment with offloading development work to external work boxes and just leaving my local machine as the orchestrator.
The only time I ever touch version control myself is when Iām trying to check out a build to test things myself or when Iām creating a tagged, versioned release of Bencher. These happen about once per week and once per month respectively. Other than that, I just let Claude handle it now. I make ample use of draft PRs and now that theyāre available on GitHub, stacked PRs. I donāt look at the code until the PR is open. I will still scroll through the code for Bencher, but itās more of a sanity check than a real code review at this point.
When I look back in a year and reread what Iāve written here, will I have another āYou didnāt even mention AIā½ā moment? Half of me thinks how awesome that would be. The other half is absolutely terrified.
Bencher: Continuous Benchmarking
Bencher is a suite of continuous benchmarking tools. Have you ever had a performance regression impact your users? Bencher could have prevented that from happening. Bencher allows you to detect and prevent performance regressions before they merge.
- Run: Run your benchmarks locally or in CI using the exact same bare metal runners and your favorite benchmarking tools. The
bencherCLI orchestrates running your benchmarks on bare metal and stores the results. - Track: Track the results of your benchmarks over time. Monitor, query, and graph the results using the Bencher web console based on the source branch, testbed, benchmark, and measure.
- Catch: Catch performance regressions locally or in CI using the exact same bare metal hardware. Bencher uses state of the art, customizable analytics to detect performance regressions before they merge.
For the same reasons that unit tests are run to prevent feature regressions, benchmarks should be run with Bencher to prevent performance regressions. Performance bugs are bugs!
Start catching performance regressions before they merge ā try Bencher Cloud for free.