How we compare CI tools

The seven criteria

Criterion	What it means	What to look for
Push-to-verdict time	Minutes from `git push` to the pass or fail you act on: queue, runner start, setup, cache restore, build, tests, report.	What the tool changes in each of those parts, not a headline multiple.
Caching	How dependencies and build output survive between jobs, and how long restoring them takes.	Whether caches are already on the machine when a job starts or are downloaded each time; storage limits; scoping by branch and repository.
Parallelism	How work is spread across machines, and how many tests each change runs.	Built-in test splitting, concurrency limits, and whether the tool can run only the tests a change reaches without ever hiding a failure.
Pricing model	What you are billed for: minutes, credits, vCPU-minutes, seats or your own servers.	The unit, the minimum, what is included, and how the bill grows with change volume. We link to pricing pages instead of copying prices that change.
Migration effort	What changes in your repositories to move in, and to move back.	A one-line `runs-on` change, a converter, or a full rewrite; secrets, caches and runners to move.
Security	Where jobs run, how they are kept apart, and how secrets reach them.	Fresh machine per job, isolation between tenants, handling of fork pull requests, OIDC instead of long-lived keys, what the vendor keeps.
Agent-heavy workloads	How the tool holds up when coding agents push many more changes a day.	Queue time under bursts, test cost per change, and whether the verdict stays as trustworthy as volume rises.

Where the facts come from

Each row in the comparison names its sources: the vendor's documentation and pricing pages. Speed claims a vendor makes are reported as theirs, never as ours.
Prices, included minutes and limits change often. Where we name one, it is what the vendor's page said when we checked; where we do not, we say where to check it.
We publish no ratings, scores or reviews. The criteria are the comparison.

Benchmark against your own baseline

A vendor's benchmark is someone else's workload. The only number that decides is your own push-to-verdict time before and after. Run this on one repository that represents your slow path.

Pick the jobs. Choose the workflow your team waits on most, usually build and test on pull requests. Write down its jobs and their runner labels.
Record a baseline. Take at least two weeks of runs (or 50 or more) from your current CI: queue time, total duration and conclusion per run. On GitHub Actions, the REST API's workflow runs and jobs endpoints return start and end times; gh run list --json is a quick way in.
Fix the inputs. Compare the same commits, or the same kinds of change (a one-file fix, a dependency bump, a wide refactor). Note cold-cache and warm-cache runs separately.
Run the candidate side by side. Point a copy of the workflow, or a share of branches, at the new tool for the same window. Keep the old one running so both see the same changes.
Compare verdicts first. For every commit both ran, the pass or fail must match. A faster tool that turned a failure into a pass is not faster; it is wrong.
Then compare time and cost. Report the median and the 90th percentile of push-to-verdict time, and the bill for the window in the vendor's own unit.
Write down the scope. Repository, jobs, window, number of runs and cache state. A number without its scope cannot be checked.

Testing a runner layer for GitHub Actions is the cheapest benchmark to run: change runs-on on one branch and compare against the same workflow on your current runners. Graphide CI's docs show how a run reads.

How we compare CI tools

The seven criteria

Where the facts come from

Benchmark against your own baseline

Read next

Get an email when these comparisons change