Run the replay in CI
In CI, the job installs Keploy and replays your tests with no database or
service running. Every call is answered from the recordings that the commit's
keploy.lock pins. On a pull request, the job also comments which tests the
pull request re-records.
Every output here is from Keploy 3.9.4 on Linux. The jobs ran the workflow on
this page step by step, as a GitHub-hosted runner runs it: a fresh clone, a user
with passwordless sudo, no Keploy installed and every dependency switched
off. The output is trimmed: timestamps, run folders and lines that don't matter
are cut.
What you need
keploy.ymlandkeploy.lockcommitted, from a record withKEPLOY_PROVEN=1. The Go quickstart and keploy.yml and keploy.lock show how.- The app's recording pin on, so every replay reads
keploy.lock: Turn on the app's recording pin. - A Linux runner where the job can use
sudowith no password, or runs as root. Keploy starts its agent withsudo. GitHub-hosted runners allow it. - A CI key, which an admin of your Keploy organization makes (step 1).
Step 1: Make a CI key
The replay signs in with a read-only key of a service account. An admin of your Keploy organization makes it:
- In app.keploy.io, open Account → Service Accounts.
- Create a service account with New service account.
- Under Access → Grant access, give it the MOCK_READER role on the app.
- Create a key under Keys, and copy it.
A CI key can replay, but it can't record: see Re-record in CI.
Step 2: Save the key as a repository secret
Save the key as the repository secret KEPLOY_API_KEY, under the repository's
Settings → Secrets and variables → Actions. Or use the GitHub CLI, which
asks for the value, so it stays out of your shell history:
gh secret set KEPLOY_API_KEY
GitHub gives no repository secrets to a pull request from a fork, so its replay can't sign in.
Step 3: Add the workflow
With the VS Code extension,
run Keploy: Wire CI (GitHub Actions). It writes
.github/workflows/keploy.yml for a Go or Node project that has a
keploy.lock, and never over a file that is already there. For a project in a
folder below the repository's root, the workflow's steps run in that folder.
Without the extension, add the file yourself. For a Go module at the repository's root, it is:
# Written by the Keploy VS Code extension. On every pull request and every
# push to main, it replays the tests on the recordings keploy.lock pins, with
# every dependency switched off (keploy.yml's mock.command runs them), and on
# a pull request it comments what the pull request's keploy.lock changes.
# It needs one repository secret, KEPLOY_API_KEY: a read-only CI key from
# https://app.keploy.io/settings/service-accounts
name: keploy
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
pull-requests: write
jobs:
replay:
runs-on: ubuntu-latest
env:
KEPLOY_API_KEY: ${{ secrets.KEPLOY_API_KEY }}
KEPLOY_INVOKER: ci
KEPLOY_PROVEN: "1"
steps:
- uses: actions/checkout@v4
with:
# keploy mock diff compares keploy.lock with its merge base.
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Install the project's dependencies (the replay downloads nothing)
run: |
go mod download
- name: Install Keploy
run: |
keploy_rc=0; keploy_sh="$(mktemp)" && curl -fsSL https://keploy.io/ent/install.sh -o "$keploy_sh" && bash "$keploy_sh" || keploy_rc=$?; rm -f "$keploy_sh"; [ "$keploy_rc" -eq 0 ]
- name: Check the Keploy key
run: |
if [ -z "$KEPLOY_API_KEY" ]; then
echo "::error::Add a repository secret named KEPLOY_API_KEY, a read-only CI key from https://app.keploy.io/settings/service-accounts. A pull request from a fork gets no secrets."
exit 1
fi
- name: Replay on the recordings keploy.lock pins
# --lock fails the job when keploy.lock is missing, and --on-miss fail
# fails a call the recordings do not hold instead of letting it out.
run: |
keploy mock replay --lock keploy.lock --on-miss fail
- name: Comment what keploy.lock changes
if: ${{ always() && github.event_name == 'pull_request' }}
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
run: |
body="$RUNNER_TEMP/keploy-mock-diff.md"
{ echo '<!-- keploy mock diff -->'; keploy mock diff --markdown --base "origin/$GITHUB_BASE_REF"; } > "$body"
id=$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues/$PR/comments" --jq '.[] | select(.body | startswith("<!-- keploy mock diff -->")) | .id' | head -n 1)
if [ -n "$id" ]; then
gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$id" -F body=@"$body" > /dev/null
else
gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body-file "$body"
fi
The replay lets no download out, so the toolchain and the project's
dependencies are set up before it, in the folder the test command runs in (the
repository's root, or the folder its first cd goes to). The extension reads
the toolchain from the program the test command runs, or, for a command such as
make test, from that folder's files:
- Go (
go, or ago.mod):actions/setup-go@v5with thego.mod's version, thengo mod download, as above. - Node (
npm,npx,pnpm,yarn,node,jestorplaywright, or apackage.jsonwith its lockfile):actions/setup-node@v4withnode-version: lts/*, then an install from the lockfile:npm ci, or pnpm's or Yarn's frozen install through corepack (npm installwith no lockfile).
For any other project, such as a Python or Java one, a test command that runs several toolchains, or one whose toolchain it can't tell, the extension writes no workflow and says that CI setup is available on request: contact the Keploy team at support@keploy.io and we'll set it up with you. Other CI systems lists what any job needs.
What each part does:
KEPLOY_INVOKER: cimakes a missing key fail the job at once instead of waiting for a sign-in, and the key check before the replay says what to add.--lockfails the job whenkeploy.lockis missing, and--on-miss failfails a call the recordings don't hold instead of letting it reach the network.- On a pull request, the last step comments the table
keploy mock diff --markdownprints: the tests the pull request re-records, read from itskeploy.lockagainst the base branch's. A later push updates that comment. That step needs the full history (fetch-depth: 0) andpull-requests: write, and a failure in it never fails the job.
Commit .github/workflows/keploy.yml with keploy.yml and keploy.lock, and
push.
Step 4: Read the job
Here is a pull request that adds a test, TestTotalInGBP, and records it. Its
job, trimmed:
Install Keploy prints the version it installed (Keploy: 3.9.4),
Keploy's help and its next steps.
Replay on the recordings keploy.lock pins downloads the recordings and runs
go test against them:
API key verified successfully!
INFO Using branch for test suites {"branch": "add-gbp", "appId": "docs-ci-orders"}
INFO pulled set "orders" pinned by …/orders/keploy.lock: 3 tests, 0 cases; recordings 4 fetched, 0 cached, 0 skipped by keploy.lock and not fetched; packs 4 fetched, 0 cached {"mock-set": "orders"}
INFO Replaying mocks for your test command {"mock-set": "orders", "program": "go", "args": 6, "on-miss": "fail"}
INFO mock replay summary {"loaded": 13, "consumed": 11, "missed": 0}
INFO this replay ran with every dependency answered from the recording {"mock-set": "orders", "served": 11}
keploy: orders: 3 test(s) ran: 3 passed, 0 failed (0 held by keploy), 0 skipped by the test; 0 quarantined (skipped by keploy)
MOCKED .::TestTotalInGBP: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInGBP: [::1]:8090 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInUSD: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInEUR: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
VERDICT passed
View test report at: https://app.keploy.io/tr/<report id>?appId=<app id>
- Each
MOCKEDline is a destination that a test called and the recordings answered: here PostgreSQL on port 5432 and the rates API on port 8090. If one of them is your own app, start it in the job, before the test command, and record again. VERDICT passedmeans every test that ran passed with no real dependency behind it. The step exits with the test command's exit code.
Comment what keploy.lock changes posts this comment on the pull request:
<!-- keploy mock diff -->
**e2e tests** keploy.lock vs origin/main (44e7a8263360) — 0 changed, 1 added, 0 removed
| set | test | status | cases | mocks |
|---|---|---|---|---|
| . | .::TestTotalInGBP | added | +0 -0 ~0 | +3 -0 ~0 |
Branches and pull requests explains the table, and what happens when the pull request merges.
Pushes to main
When the pull request merges, the push to main runs the same job. After a green
replay, it lists main's keploy.lock as Keploy main in Keploy Cloud, and names
the branches it brought into main:
VERDICT passed
keploy: listed keploy.lock of commit 50494ee09825 as Keploy main in Keploy Cloud (4 recorded, 0 removed, 0 unchanged); now in main, as main's keploy.lock holds or held each one's own recording of every test it changed: add-gbp: https://app.keploy.io/recordings?appId=<app id>&view=e2e
The CI key's read access is enough for this. Branches and pull requests says when main isn't listed.
Cache the recordings
The replay downloads only the recordings of the tests it runs. Each one is named
by its hash and kept in ~/.cache/keploy/mocks. To save the downloads on the
next run, cache that folder, keyed on keploy.lock, with a step before the
replay:
- uses: actions/cache@v4
with:
path: ~/.cache/keploy/mocks
key: keploy-mocks-${{ hashFiles('keploy.lock') }}
restore-keys: keploy-mocks-
Cache that folder itself, never a parent such as ~/.cache. With the cache
restored, the same replay downloads nothing:
INFO pulled set "orders" pinned by …/orders/keploy.lock: 3 tests, 0 cases; recordings 0 fetched, 4 cached, 0 skipped by keploy.lock and not fetched; packs 0 fetched, 4 cached {"mock-set": "orders"}
Set KEPLOY_MOCK_CACHE_DIR to keep the cache in another folder, and cache that
folder instead.
What fails the job
The replay step exits with your test command's exit code, so a failing test fails the job. These fail it too.
No CI key
When the KEPLOY_API_KEY secret is missing or empty, as on a pull request from
a fork, the key check stops the job before the replay, and GitHub shows this
error on the run:
Add a repository secret named KEPLOY_API_KEY, a read-only CI key from https://app.keploy.io/settings/service-accounts. A pull request from a fork gets no secrets.
Without that step, the replay itself stops with exit code 5:
Error: not signed in and running non-interactively: run `keploy login` first
Add the secret (step 2).
No keploy.lock
A job that runs before keploy.lock is committed, or in a folder without one,
replays nothing and exits 1:
NOT_JUDGED keploy did not complete the run: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed; the tests are not to blame
VERDICT not judged: keploy did not complete the run: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed; the tests are not to blame
Error: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed
Record with KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock on your
machine, and commit keploy.lock. Keploy 3.9.4 also prints a NEXT debug:
line here; the fix is the commit, not the debug run.
A call the recordings don't hold
Here the pull request changes TestTotalInEUR to insert and price a new order,
and doesn't record it again. The new INSERT has no recording, so
--on-miss fail fails the test, and the job exits 1:
WARN no recorded mock matched an outgoing call {"protocol": "PostgreSQL", "call": "INSERT INTO orders VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING", "destination": "[::1]:5432", …}
keploy: orders: 3 test(s) ran: 2 passed, 1 failed (1 held by keploy), 0 skipped by the test; 0 quarantined (skipped by keploy)
FAILED_BY_KEPLOY .::TestTotalInEUR (failed): keploy/unmocked_call
MISS PostgreSQL INSERT INTO orders VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING to [::1]:5432 x1 match stopped at no_schema_candidates
next: None of the mocks this query was matched against at this point of the run holds a recorded invocation of this SQL. Either the application issued a statement it did not issue while recording, or the recording holds it at a different point of the run (in another test, or before or between tests). If the call is intended here, re-record the mock set with 'KEPLOY_PROVEN=1 keploy mock record'.
UNCONSUMED mock-5: recorded for this test, never called
UNCONSUMED mock-6: recorded for this test, never called
VERDICT failed: keploy held evidence against 1 test that ran: a call it could not answer or capture, or a recorded call left unmade
NEXT re-record: KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock
NEXT quarantine: KEPLOY_PROVEN=1 keploy mock quarantine TestTotalInEUR --reason 'keploy held evidence against it in CI' --name orders
When the change is intended, record the test again on your machine, with the
dependencies up, and push the new keploy.lock:
Partial re-record records just that test.
Troubleshooting covers the other ways a replay that passes on your machine fails in CI, such as a test command that downloads packages.
Re-record in CI
Keep CI for replays, and record on your machine. A CI key can't record: Keploy Cloud refuses the recordings that a key without write access uploads.
With KEPLOY_PROVEN=1, as this guide sets it, Keploy 3.9.4 stops a record
before it starts when the key's role is CI. It doesn't stop a record signed
in with a service-account key, like the one step 1
makes, or a record without KEPLOY_PROVEN=1. Such a record runs your tests
against the real dependencies, and then Keploy Cloud refuses its recordings
at the upload: see
A CI key can't record.
A job that must re-record, such as a scheduled refresh, needs the dependencies up and an API key that a person made. Run it on a branch, not on main, and open a pull request from it, so the new recordings are reviewed like any other change:
git switch -c refresh-recordings
KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock
git commit -am "Refresh the recordings"
git push origin refresh-recordings
The record ends with what it re-recorded, and lists the new keploy.lock under
the branch:
keploy: keploy.lock: re-recorded .::.::TestTotalInEUR, .::.::TestTotalInGBP, .::.::TestTotalInUSD (3), kept 0, start-up re-recorded; recordings 4 uploaded, 0 already on the server; packs 4 uploaded, 0 already on the server
keploy: commit keploy.lock with your change
keploy: listed keploy.lock in Keploy Cloud under branch refresh-recordings (4 recorded, 0 removed, 0 unchanged): https://app.keploy.io/recordings?appId=<app id>&view=e2e&branch=refresh-recordings
A record on main lists nothing: Keploy lists the lock of a branch that merges into main, never main's own.
Recordings in the repository
With mock.local: true, the recordings come with the checkout and there is no
keploy.lock. Drop KEPLOY_API_KEY, the key check and the comment step, and
run keploy mock replay --local instead.
Other CI systems
GitLab CI and other CI systems are available on request: contact the Keploy team at support@keploy.io and we'll set it up with you. Any job needs the same steps as the workflow above:
-
A Linux runner where the job runs as root, or as a user with passwordless
sudo. -
The project's toolchain, and its dependencies downloaded, such as
go mod downloadornpm ci. The replay lets no download out. -
Keploy, from
https://keploy.io/ent/install.sh. In a pipeline, it installs in/usr/local/bin, which takes root or passwordlesssudo. Without them, the install fails:sudo: a password is required
Error: could not install the keploy binary -
The CI key in
KEPLOY_API_KEY, from the CI system's secrets. -
The replay, in the project's folder:
KEPLOY_INVOKER=ci KEPLOY_PROVEN=1 keploy mock replay --lock keploy.lock --on-miss fail
On GitLab CI and Woodpecker, a replay of a push to the default branch lists
main's keploy.lock as Keploy main, as on GitHub Actions. On any other CI
system, pass --list-main on:
List main in Keploy Cloud.
Related
- keploy.yml and keploy.lock: what to commit, and the recording pin.
- Branches and pull requests: the pull request's table, merging, and Keploy main.
- Troubleshooting: a replay that passes on your machine but fails in CI.
- Reference: the replay's exit codes.