Bencher Privacy Policy

This policy explains what personal data Bencher collects, why Bencher uses it, who receives it, how long Bencher keeps it, and the controls you have. If you have a question, email legal@bencher.dev.

Scope

This policy covers Bencher’s services (the “Services”):

  • the bencher.dev website and the Bencher Console
  • the Bencher API
  • the bencher CLI
  • Bencher Cloud bare metal runners
  • the Bencher MCP server and its OAuth sign-in
  • the usage statistics that self-hosted Bencher servers send to Bencher Cloud

Data Bencher Collects

Account and Sign-In

  • Your email address, name, and slug (the short name in your profile URL).
  • You sign in with a link that Bencher emails to you, or with GitHub or Google. If you sign in with GitHub, Bencher receives your GitHub login and your primary verified email address. If you sign in with Google, Bencher receives your name and your verified email address.
  • The organizations you belong to and your role in each.
  • When you invite someone to an organization, the email address you invite.

Bencher uses this data to identify you, sign you in, send you sign-in links, confirmations, and invitations, and let you work with the other members of your organizations. When someone signs up, Bencher’s administrators receive an email with the new user’s name, email address, and sign-up method. Bencher sends no marketing email.

Bot Protection

The sign-up, sign-in, and confirmation pages run a bot protection check. Bencher’s server sends the token from that check and your IP address to the bot protection provider to verify that you are not a bot.

Billing

  • To manage a paid plan, Bencher sends your name, email address, and account identifiers to a payment processor, along with usage quantities for metered plans.
  • You enter card details only on the payment processor’s hosted pages. Bencher does not store card numbers. The Console shows your card’s brand, last four digits, and expiration date.

API Keys

Bencher stores user and project API keys as hashes.

Project Data

  • Projects and their reports, benchmark results, branches, git commit hashes, testbeds, benchmarks, measures, thresholds, alerts, and plots.
  • The benchmark output you submit. It is whatever your benchmark harness prints, so it holds anything your benchmarks write.
  • Who submitted each report: your user account or a project API key.

Bencher uses project data to provide the Services: to track your benchmark results, detect performance regressions, and show your results.

CLI Run Context

Every bencher run sends a run context with the following, where available:

  • the repository name
  • the hash of the repository’s root commit
  • the branch name
  • the HEAD commit hash
  • the operating system
  • a machine fingerprint derived from the machine’s ID

Bencher uses the run context to choose the default branch, commit hash, and testbed for the run. For a run that names no project, Bencher creates a project named after the repository, with a slug built from the repository name, the root commit hash, and the machine fingerprint.

Anonymous Runs

Anyone can submit runs to an unclaimed project without an account. Bencher records the IP address of these requests and uses it to rate limit them.

Bare Metal Jobs

When you run benchmarks on Bencher Cloud bare metal runners, Bencher stores:

  • the job configuration: container image, entrypoint and command, the environment variables you supply, file paths, and timeout
  • the IP address the job was submitted from
  • the job output: exit code, standard output, standard error, the files the job collects, and any error
  • the callback URL and its credentials, if you configure a callback. Bencher stores the callback encrypted and clears it once the callback is delivered or fails for good.

Bencher uses this data to run the job, return its results to you, and notify your CI when the job finishes. A runner pulls the container image you name from its container image registry.

Self-Hosted Usage Statistics

A self-hosted Bencher server notifies Bencher Cloud when it starts. By default, it also sends usage statistics to Bencher Cloud every day. The statistics include the names and email addresses of the server’s administrators, the server’s organizations, the names of its most active projects, and counts of its users, projects, and other usage. A server with a valid Bencher Plus license can turn the daily statistics off. Bencher uses these statistics to enforce the Bencher License and to understand how self-hosted servers are used, and emails them to Bencher Cloud’s administrators.

Usage and Log Data

Bencher’s servers log requests. The logs hold IP addresses, the requested URLs, the user identifier of signed-in requests, the email addresses Bencher sends email to, the run requests submitted, including their benchmark results and run context, and the self-hosted usage statistics Bencher receives. Bencher uses IP addresses to rate limit requests and stop abuse, and uses the logs to diagnose problems. When your browser loads Bencher’s pages, the hosting, content delivery, analytics, error monitoring, bot protection, and embedded video and slide providers also receive your IP address and your browser’s user agent.

Website Analytics

Most bencher.dev pages, including the documentation and parts of the Console, load a website analytics service. It records the pages you view and how you use them, along with your browser details and IP address. It does not run on the page where you approve an AI client’s access. Bencher uses analytics to understand how the website is used. Bencher shows no ads and does not use analytics data for advertising.

Error Monitoring and Session Replay

The Bencher API and website send error reports to an error monitoring provider, and the website also sends performance traces. An error report describes the failure and can include the request or page that failed and data from it. The website also records session replays, recordings of the page and your interactions with it, for a sample of sessions and for every session that hits an error. Bencher uses this data to find and fix problems.

In-App Feedback

The Console’s feedback dialog sends your message to the error monitoring provider. By default, it also sends the page URL and title, the project, your browser, the Console build, a link to your session replay, and your email address, along with any image you attach. You can untick every item except the message before you send it.

Browser Storage

Bencher stores data in your browser:

  • Local storage holds your signed-in session, with your name, slug, and email address, and your preferences, such as theme.
  • Session storage holds short-lived sign-in and onboarding data, such as the email address you are signing in with.
  • IndexedDB caches API responses for up to 7 days.

MCP Server and AI Clients

The Bencher MCP server at https://mcp.bencher.dev/mcp is a remote MCP server. It lets an AI client that you connect, such as ChatGPT or Claude, read and act on your Bencher projects through tools.

Tool Inputs

The arguments an AI client sends to a tool are the same as Bencher API requests: project, branch, testbed, benchmark, and measure names; query filters; benchmark results; run context; and bare metal job settings, such as the container image, command, environment variables, and callback. Free text that you or the AI client puts into these arguments can contain personal data. Bencher stores what a tool submits exactly as it stores the same Bencher API request.

Tool Outputs

A tool returns Bencher API data to the AI client: projects, reports, branches, testbeds, benchmarks, measures, thresholds, alerts, metrics, performance query results and chart images, plots, and bare metal jobs. Reports include the name and slug of the user who submitted them. For a signed-in AI client, viewing a finished bare metal job returns its output: exit code, standard output, standard error, and collected files. Tool outputs do not include a job’s environment variables or the IP address it was submitted from. Once a tool returns data to the AI client, the AI client’s provider handles that data under its own privacy policy.

What the MCP Server Does Not Do

  • It does not request or receive your conversation history.
  • It does not request your location.
  • It keeps no MCP session.
  • It does not store the user or conversation identifiers an AI client may attach to a request.
  • It does not log tool results. Bencher logs MCP requests the same way as Bencher API requests: the IP address, the user identifier, and the project, and for a submitted run, the run request, including its benchmark results and run context.

OAuth Sign-In

When a tool needs you to be signed in, the AI client asks you to sign in to Bencher and approve its access. You can deny the request. Bencher fetches the AI client’s published client metadata to show you which app is asking. The AI client can request these scopes:

Scope What it allows
projects:read Read the projects you can read
projects:write Create and update data in the projects you can write
openid Your Bencher user identifier
email Your email address
profile Your name and slug

A connected app always receives your Bencher user identifier. It receives your email address only with the email scope, and your name and slug only with the profile scope. For each connected app, Bencher keeps the app’s name and client ID, the scopes you granted, and when the grant expires. You can see and revoke your connected apps in Connected Apps.

Anonymous Use

Without signing in, an AI client can use read tools on public projects and submit runs, including bare metal jobs, to unclaimed projects. Bencher rate limits these requests by IP address.

Public Information

Projects are public by default. Private projects require Bencher Plus. Anyone, including AI clients, can see a public project’s data, including the name and slug of the user who submitted each report. Any signed-in Bencher user can also see the output of a public project’s bare metal jobs. Bencher submits public project URLs to search engines for indexing. The members of an organization see each other’s name, slug, email address, and role.

How Bencher Uses Data

  • Account and sign-in data: to identify you, sign you in, send you sign-in links, confirmations, and invitations, and notify Bencher’s administrators of new users.
  • Bot protection data: to stop automated sign-ups and sign-ins.
  • Billing data: to run subscriptions and bill metered usage.
  • Project data, run context, and git hashes: to provide the Services and to name new projects.
  • Bare metal job data: to run jobs, return their results, and notify your CI.
  • Self-hosted usage statistics: to enforce the Bencher License and understand how self-hosted servers are used.
  • IP addresses and logs: to rate limit requests, stop abuse, and diagnose problems.
  • Analytics, error monitoring, session replay, and feedback: to understand how the Services are used, find and fix problems, and act on your feedback.
  • OAuth grants: to let the AI clients you connect act within the scopes you granted.
  • Public project URLs: to have search engines index public projects.

Bencher does not sell personal data and does not use it for advertising.

Who Receives Data

Service Providers

Bencher shares data with service providers that help operate the Services, only as needed for their work:

  • hosting and infrastructure
  • content delivery
  • payment processing
  • error monitoring and session diagnostics
  • website analytics
  • bot protection
  • email delivery
  • search engines, which receive public project URLs for indexing
  • bare metal runner hosting
  • container image registries
  • AI service providers, which process data only to help Bencher operate and support the Services, for example to debug problems, under terms that bar them from training models on that data

Others

  • Other users and the public: anyone can see public projects, and the members of an organization see each other.
  • Sign-in providers: if you sign in with GitHub or Google, that provider takes part in the sign-in.
  • AI clients you connect: they receive the tool outputs described above.
  • Your CI provider: when you configure a bare metal job callback, such as one to GitHub, Bencher notifies its URL when the job finishes.
  • Legal requests: Bencher may disclose personal data when required by law, such as a valid subpoena, warrant, or court order, or when needed to enforce its agreements, collect amounts owed, protect the rights, property, or safety of Bencher or its users, or protect users from fraudulent, abusive, or unlawful use of the Services.
  • Business transfers: personal data may be transferred as part of a merger, acquisition, financing, sale of assets, or insolvency. Bencher will try to notify you before your personal data is transferred.

Retention

Data Kept
Account data While your account exists, until you ask Bencher to delete it
Project data (reports, results, benchmarks, and bare metal job configuration, outputs, and the IP address a job was submitted from) Until you ask Bencher to delete it
Billing records As long as tax and accounting law requires, then deleted
Self-hosted usage statistics While the server’s license or use continues
Backups Up to 12 months
Server logs, which hold IP addresses, user identifiers, and the other log data above Up to 30 days
Sign-in links 30 minutes
Console sessions 30 days
MCP OAuth authorization codes 5 minutes
MCP OAuth access tokens 1 hour
MCP OAuth refresh tokens 30 days
A connected app’s grant Up to 1 year, or until you revoke it
Bare metal job callback URL and credentials Until the callback is delivered or fails for good
Browser API cache Up to 7 days, or until you sign out
Analytics, error monitoring, and bot protection data Kept by those providers under their retention periods

Your Controls

  • Change your name, slug, and email address in the Console. A new email address takes effect once you confirm it.
  • Create and revoke user and project API keys.
  • See and revoke the apps you have connected in Connected Apps, in the Console’s account menu. Revoking an app stops its access at once.
  • Deny an AI client’s request when it asks for access.
  • Delete reports.
  • Remove members from your organizations.
  • Make a project private with Bencher Plus.
  • Cancel a paid plan, and manage your payment details through the payment processor.
  • Untick any item except the message in the in-app feedback dialog.
  • Block cookies and browser storage in your browser’s settings.
  • Turn off a self-hosted server’s daily usage statistics with a valid Bencher Plus license.
  • Delete your account: email legal@bencher.dev. Bencher deletes your account and its personal data within 30 days, except records that law or billing requires Bencher to keep. Backups that hold your data age out within 12 months.

Cookies and Browser Storage

Bencher sets no first-party cookies. Your signed-in session and your preferences live in your browser’s local storage. Signing out clears your session and the cached API data. The third-party analytics, bot protection, error monitoring, and embedded video and slide services on Bencher’s pages may set their own cookies. You can block or clear cookies and browser storage in your browser’s settings.

Security

Bencher serves the Services over HTTPS. It stores user and project API keys as hashes and stores bare metal job callbacks encrypted. No security measure is perfect. If you think an API key has leaked, revoke it and email legal@bencher.dev.

Children

The Services are not intended for children under the age of 13, and the Terms of Use prohibit such use. Bencher does not knowingly collect personal data from children under 13. If you become aware that a child has provided Bencher with personal data, please email legal@bencher.dev.

Changes to This Policy

Bencher may change this policy at any time and always posts the current version on its website. If a change materially affects your rights, Bencher will notify you, for example by email to your account’s email address.

Contact

For questions about this policy or Bencher’s privacy practices, or to correct or delete your personal data, email legal@bencher.dev.