Skip to main content
View as Markdown

Run the replay in CI​

In CI, the job installs Keploy and replays your tests with no database or service running. Every call is answered from the recordings that the commit's keploy.lock pins. On a pull request, the job also comments which tests the pull request re-records.

Every output here is from Keploy 3.9.4 on Linux. The jobs ran the workflow on this page step by step, as a GitHub-hosted runner runs it: a fresh clone, a user with passwordless sudo, no Keploy installed and every dependency switched off. The output is trimmed: timestamps, run folders and lines that don't matter are cut.

What you need​

  • keploy.yml and keploy.lock committed, from a record with KEPLOY_PROVEN=1. The Go quickstart and keploy.yml and keploy.lock show how.
  • The app's recording pin on, so every replay reads keploy.lock: Turn on the app's recording pin.
  • A Linux runner where the job can use sudo with no password, or runs as root. Keploy starts its agent with sudo. GitHub-hosted runners allow it.
  • A CI key, which an admin of your Keploy organization makes (step 1).

Step 1: Make a CI key​

The replay signs in with a read-only key of a service account. An admin of your Keploy organization makes it:

  1. In app.keploy.io, open Account → Service Accounts.
  2. Create a service account with New service account.
  3. Under Access → Grant access, give it the MOCK_READER role on the app.
  4. Create a key under Keys, and copy it.

A CI key can replay, but it can't record: see Re-record in CI.

Step 2: Save the key as a repository secret​

Save the key as the repository secret KEPLOY_API_KEY, under the repository's Settings → Secrets and variables → Actions. Or use the GitHub CLI, which asks for the value, so it stays out of your shell history:

gh secret set KEPLOY_API_KEY

GitHub gives no repository secrets to a pull request from a fork, so its replay can't sign in.

Step 3: Add the workflow​

With the VS Code extension, run Keploy: Wire CI (GitHub Actions). It writes .github/workflows/keploy.yml for a Go or Node project that has a keploy.lock, and never over a file that is already there. For a project in a folder below the repository's root, the workflow's steps run in that folder.

Without the extension, add the file yourself. For a Go module at the repository's root, it is:

# Written by the Keploy VS Code extension. On every pull request and every
# push to main, it replays the tests on the recordings keploy.lock pins, with
# every dependency switched off (keploy.yml's mock.command runs them), and on
# a pull request it comments what the pull request's keploy.lock changes.
# It needs one repository secret, KEPLOY_API_KEY: a read-only CI key from
# https://app.keploy.io/settings/service-accounts
name: keploy
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
pull-requests: write
jobs:
replay:
runs-on: ubuntu-latest
env:
KEPLOY_API_KEY: ${{ secrets.KEPLOY_API_KEY }}
KEPLOY_INVOKER: ci
KEPLOY_PROVEN: "1"
steps:
- uses: actions/checkout@v4
with:
# keploy mock diff compares keploy.lock with its merge base.
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Install the project's dependencies (the replay downloads nothing)
run: |
go mod download
- name: Install Keploy
run: |
keploy_rc=0; keploy_sh="$(mktemp)" && curl -fsSL https://keploy.io/ent/install.sh -o "$keploy_sh" && bash "$keploy_sh" || keploy_rc=$?; rm -f "$keploy_sh"; [ "$keploy_rc" -eq 0 ]
- name: Check the Keploy key
run: |
if [ -z "$KEPLOY_API_KEY" ]; then
echo "::error::Add a repository secret named KEPLOY_API_KEY, a read-only CI key from https://app.keploy.io/settings/service-accounts. A pull request from a fork gets no secrets."
exit 1
fi
- name: Replay on the recordings keploy.lock pins
# --lock fails the job when keploy.lock is missing, and --on-miss fail
# fails a call the recordings do not hold instead of letting it out.
run: |
keploy mock replay --lock keploy.lock --on-miss fail
- name: Comment what keploy.lock changes
if: ${{ always() && github.event_name == 'pull_request' }}
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
run: |
body="$RUNNER_TEMP/keploy-mock-diff.md"
{ echo '<!-- keploy mock diff -->'; keploy mock diff --markdown --base "origin/$GITHUB_BASE_REF"; } > "$body"
id=$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues/$PR/comments" --jq '.[] | select(.body | startswith("<!-- keploy mock diff -->")) | .id' | head -n 1)
if [ -n "$id" ]; then
gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$id" -F body=@"$body" > /dev/null
else
gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body-file "$body"
fi

The replay lets no download out, so the toolchain and the project's dependencies are set up before it, in the folder the test command runs in (the repository's root, or the folder its first cd goes to). The extension reads the toolchain from the program the test command runs, or, for a command such as make test, from that folder's files:

  • Go (go, or a go.mod): actions/setup-go@v5 with the go.mod's version, then go mod download, as above.
  • Node (npm, npx, pnpm, yarn, node, jest or playwright, or a package.json with its lockfile): actions/setup-node@v4 with node-version: lts/*, then an install from the lockfile: npm ci, or pnpm's or Yarn's frozen install through corepack (npm install with no lockfile).

For any other project, such as a Python or Java one, a test command that runs several toolchains, or one whose toolchain it can't tell, the extension writes no workflow and says that CI setup is available on request: contact the Keploy team at support@keploy.io and we'll set it up with you. Other CI systems lists what any job needs.

What each part does:

  • KEPLOY_INVOKER: ci makes a missing key fail the job at once instead of waiting for a sign-in, and the key check before the replay says what to add.
  • --lock fails the job when keploy.lock is missing, and --on-miss fail fails a call the recordings don't hold instead of letting it reach the network.
  • On a pull request, the last step comments the table keploy mock diff --markdown prints: the tests the pull request re-records, read from its keploy.lock against the base branch's. A later push updates that comment. That step needs the full history (fetch-depth: 0) and pull-requests: write, and a failure in it never fails the job.

Commit .github/workflows/keploy.yml with keploy.yml and keploy.lock, and push.

Step 4: Read the job​

Here is a pull request that adds a test, TestTotalInGBP, and records it. Its job, trimmed:

Install Keploy prints the version it installed (Keploy: 3.9.4), Keploy's help and its next steps.

Replay on the recordings keploy.lock pins downloads the recordings and runs go test against them:

API key verified successfully!
INFO Using branch for test suites {"branch": "add-gbp", "appId": "docs-ci-orders"}
INFO pulled set "orders" pinned by …/orders/keploy.lock: 3 tests, 0 cases; recordings 4 fetched, 0 cached, 0 skipped by keploy.lock and not fetched; packs 4 fetched, 0 cached {"mock-set": "orders"}
INFO Replaying mocks for your test command {"mock-set": "orders", "program": "go", "args": 6, "on-miss": "fail"}
INFO mock replay summary {"loaded": 13, "consumed": 11, "missed": 0}
INFO this replay ran with every dependency answered from the recording {"mock-set": "orders", "served": 11}
keploy: orders: 3 test(s) ran: 3 passed, 0 failed (0 held by keploy), 0 skipped by the test; 0 quarantined (skipped by keploy)
MOCKED .::TestTotalInGBP: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInGBP: [::1]:8090 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInUSD: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
MOCKED .::TestTotalInEUR: [::1]:5432 (mocked: if this is your app, start it from the runner, then re-record)
VERDICT passed
View test report at: https://app.keploy.io/tr/<report id>?appId=<app id>
  • Each MOCKED line is a destination that a test called and the recordings answered: here PostgreSQL on port 5432 and the rates API on port 8090. If one of them is your own app, start it in the job, before the test command, and record again.
  • VERDICT passed means every test that ran passed with no real dependency behind it. The step exits with the test command's exit code.

Comment what keploy.lock changes posts this comment on the pull request:

<!-- keploy mock diff -->
**e2e tests** keploy.lock vs origin/main (44e7a8263360) — 0 changed, 1 added, 0 removed
| set | test | status | cases | mocks |
|---|---|---|---|---|
| . | .::TestTotalInGBP | added | +0 -0 ~0 | +3 -0 ~0 |

Branches and pull requests explains the table, and what happens when the pull request merges.

Pushes to main​

When the pull request merges, the push to main runs the same job. After a green replay, it lists main's keploy.lock as Keploy main in Keploy Cloud, and names the branches it brought into main:

VERDICT  passed
keploy: listed keploy.lock of commit 50494ee09825 as Keploy main in Keploy Cloud (4 recorded, 0 removed, 0 unchanged); now in main, as main's keploy.lock holds or held each one's own recording of every test it changed: add-gbp: https://app.keploy.io/recordings?appId=<app id>&view=e2e

The CI key's read access is enough for this. Branches and pull requests says when main isn't listed.

Cache the recordings​

The replay downloads only the recordings of the tests it runs. Each one is named by its hash and kept in ~/.cache/keploy/mocks. To save the downloads on the next run, cache that folder, keyed on keploy.lock, with a step before the replay:

- uses: actions/cache@v4
with:
path: ~/.cache/keploy/mocks
key: keploy-mocks-${{ hashFiles('keploy.lock') }}
restore-keys: keploy-mocks-

Cache that folder itself, never a parent such as ~/.cache. With the cache restored, the same replay downloads nothing:

INFO  pulled set "orders" pinned by …/orders/keploy.lock: 3 tests, 0 cases; recordings 0 fetched, 4 cached, 0 skipped by keploy.lock and not fetched; packs 0 fetched, 4 cached  {"mock-set": "orders"}

Set KEPLOY_MOCK_CACHE_DIR to keep the cache in another folder, and cache that folder instead.

What fails the job​

The replay step exits with your test command's exit code, so a failing test fails the job. These fail it too.

No CI key​

When the KEPLOY_API_KEY secret is missing or empty, as on a pull request from a fork, the key check stops the job before the replay, and GitHub shows this error on the run:

Add a repository secret named KEPLOY_API_KEY, a read-only CI key from https://app.keploy.io/settings/service-accounts. A pull request from a fork gets no secrets.

Without that step, the replay itself stops with exit code 5:

Error: not signed in and running non-interactively: run `keploy login` first

Add the secret (step 2).

No keploy.lock​

A job that runs before keploy.lock is committed, or in a folder without one, replays nothing and exits 1:

NOT_JUDGED  keploy did not complete the run: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed; the tests are not to blame
VERDICT not judged: keploy did not complete the run: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed; the tests are not to blame
Error: could not pull the recordings of orders: there is no keploy.lock at …/orders/keploy.lock, which this run was told pins its recordings: nothing was replayed

Record with KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock on your machine, and commit keploy.lock. Keploy 3.9.4 also prints a NEXT debug: line here; the fix is the commit, not the debug run.

A call the recordings don't hold​

Here the pull request changes TestTotalInEUR to insert and price a new order, and doesn't record it again. The new INSERT has no recording, so --on-miss fail fails the test, and the job exits 1:

WARN  no recorded mock matched an outgoing call  {"protocol": "PostgreSQL", "call": "INSERT INTO orders VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING", "destination": "[::1]:5432", …}
keploy: orders: 3 test(s) ran: 2 passed, 1 failed (1 held by keploy), 0 skipped by the test; 0 quarantined (skipped by keploy)
FAILED_BY_KEPLOY .::TestTotalInEUR (failed): keploy/unmocked_call
MISS PostgreSQL INSERT INTO orders VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING to [::1]:5432 x1 match stopped at no_schema_candidates
next: None of the mocks this query was matched against at this point of the run holds a recorded invocation of this SQL. Either the application issued a statement it did not issue while recording, or the recording holds it at a different point of the run (in another test, or before or between tests). If the call is intended here, re-record the mock set with 'KEPLOY_PROVEN=1 keploy mock record'.
UNCONSUMED mock-5: recorded for this test, never called
UNCONSUMED mock-6: recorded for this test, never called
VERDICT failed: keploy held evidence against 1 test that ran: a call it could not answer or capture, or a recorded call left unmade
NEXT re-record: KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock
NEXT quarantine: KEPLOY_PROVEN=1 keploy mock quarantine TestTotalInEUR --reason 'keploy held evidence against it in CI' --name orders

When the change is intended, record the test again on your machine, with the dependencies up, and push the new keploy.lock: Partial re-record records just that test.

Troubleshooting covers the other ways a replay that passes on your machine fails in CI, such as a test command that downloads packages.

Re-record in CI​

Keep CI for replays, and record on your machine. A CI key can't record: Keploy Cloud refuses the recordings that a key without write access uploads.

With KEPLOY_PROVEN=1, as this guide sets it, Keploy 3.9.4 stops a record before it starts when the key's role is CI. It doesn't stop a record signed in with a service-account key, like the one step 1 makes, or a record without KEPLOY_PROVEN=1. Such a record runs your tests against the real dependencies, and then Keploy Cloud refuses its recordings at the upload: see A CI key can't record.

A job that must re-record, such as a scheduled refresh, needs the dependencies up and an API key that a person made. Run it on a branch, not on main, and open a pull request from it, so the new recordings are reviewed like any other change:

git switch -c refresh-recordings
KEPLOY_PROVEN=1 keploy mock record --lock keploy.lock
git commit -am "Refresh the recordings"
git push origin refresh-recordings

The record ends with what it re-recorded, and lists the new keploy.lock under the branch:

keploy: keploy.lock: re-recorded .::.::TestTotalInEUR, .::.::TestTotalInGBP, .::.::TestTotalInUSD (3), kept 0, start-up re-recorded; recordings 4 uploaded, 0 already on the server; packs 4 uploaded, 0 already on the server
keploy: commit keploy.lock with your change
keploy: listed keploy.lock in Keploy Cloud under branch refresh-recordings (4 recorded, 0 removed, 0 unchanged): https://app.keploy.io/recordings?appId=<app id>&view=e2e&branch=refresh-recordings

A record on main lists nothing: Keploy lists the lock of a branch that merges into main, never main's own.

Recordings in the repository​

With mock.local: true, the recordings come with the checkout and there is no keploy.lock. Drop KEPLOY_API_KEY, the key check and the comment step, and run keploy mock replay --local instead.

Other CI systems​

GitLab CI and other CI systems are available on request: contact the Keploy team at support@keploy.io and we'll set it up with you. Any job needs the same steps as the workflow above:

  1. A Linux runner where the job runs as root, or as a user with passwordless sudo.

  2. The project's toolchain, and its dependencies downloaded, such as go mod download or npm ci. The replay lets no download out.

  3. Keploy, from https://keploy.io/ent/install.sh. In a pipeline, it installs in /usr/local/bin, which takes root or passwordless sudo. Without them, the install fails:

    sudo: a password is required
    Error: could not install the keploy binary
  4. The CI key in KEPLOY_API_KEY, from the CI system's secrets.

  5. The replay, in the project's folder:

    KEPLOY_INVOKER=ci KEPLOY_PROVEN=1 keploy mock replay --lock keploy.lock --on-miss fail

On GitLab CI and Woodpecker, a replay of a push to the default branch lists main's keploy.lock as Keploy main, as on GitHub Actions. On any other CI system, pass --list-main on: List main in Keploy Cloud.