Resilience testing checks whether software keeps serving users when a dependency breaks. Teams inject timeouts, error responses, and slow databases, then measure the impact and the time to recover. Keploy brings this into CI: its recorded mocks can be edited into failure responses and replayed on every pull request.

Resilience testing gets skipped for the same reason outages run long: most systems are built for a dependency that goes down, not one that gets slow. A database that answers slowly still passes its health check. Meanwhile, every request handler waits on it, queues fill up, and a service that looks healthy on the dashboard starts turning users away.

Failover plans share that blind spot. They’re usually written for clean outages, so they wait for a dependency to disappear while users are already stuck behind one that’s only degraded. The distance between the failure a team planned for and the one that actually arrives is where an incident’s first hour usually goes.

What Is Resilience Testing?

Resilience testing, also called resiliency testing, is a type of non-functional software testing that deliberately introduces failures into a system, such as dependency outages, latency, error responses, or resource loss, to verify that it degrades gracefully and recovers within agreed limits.

Software resiliency is the property being measured, and resilience testing is how you prove it. You’ll also see the practice called fault tolerance testing or software resilience testing, and it sits under the broader discipline of resilience engineering.

The pass criteria aren’t "nothing broke." Instead, a resilience test asks whether the system stayed within its service level objectives (SLOs) while something was broken, and whether it came back without manual help.

Why Does Resilience Testing Matter?

Resilience testing matters because real dependencies rarely fail cleanly, and the recovery code written for those moments almost never runs until an incident forces it to.

  • Partial failures are the dangerous ones. A dependency that’s slow or erroring intermittently is harder to detect than one that’s fully down, so failover logic often misses it.
  • Retries can turn a blip into an outage. Clients that retry aggressively without backoff multiply the load on a service that’s already struggling.
  • Failures travel in microservices. One slow service holds connections open in every caller, so the degradation moves upstream, one hop at a time.
  • Recovery paths go untested. Fallbacks, circuit breakers, and failover scripts only run during incidents, which is the worst possible time to learn they don’t work.

That’s why a good resilience suite tests the recovery path as carefully as the failure itself.

Types of Resilience Testing

Types of Resilience Testing

Each type targets a different way a system can fail. Most teams start with dependency failure and latency tests, since those cover the incidents that happen most often in API services.

Dependency Failure Testing

A downstream API, database, or queue returns errors or stops responding entirely.

  • What you check: fallback responses, clear error messages for users, and circuit breakers that open instead of hammering the failed dependency.

Latency and Timeout Testing

A dependency still answers, just slowly. In practice, this is where to start, because default timeouts are often far longer than any user will wait.

  • What you check: timeouts are set and enforced, connection pools don’t run dry, and users get a fast failure instead of a hanging request.

Failover Testing

The primary instance of a component is removed or degraded, and traffic should shift to a backup.

  • What you check: failover triggers on degraded performance, not only on a total outage, and the switch finishes within your recovery target.

Recovery Testing

The failure is removed, and the system should return to normal without anyone stepping in.

  • What you check: recovery time, data consistency after the event, and whether queued work drains without overloading the dependency again.

Load-Induced Failure Testing

Failures happen while the system is under realistic traffic, which is when retries and restarts do the most damage.

  • What you check: retries use exponential backoff with jitter, load shedding protects core paths, and rate limits hold.

Resilience Testing Techniques

Fault Injection

Fault injection introduces a specific, controlled fault at a defined point: an error response, added latency, a dropped connection, or a killed process. Almost every resilience test is built on it, so it’s the technique to learn first.

Chaos Experiments

Chaos experiments run fault injection as a structured experiment. You define a steady state, form a hypothesis, break something, and check whether the steady state held. This loop is the core of chaos testing, and it often runs in production-like environments or in production itself.

Service Virtualization for Failure Simulation

Service virtualization replaces a real dependency with a virtual one that returns failures on demand. As a result, you can test how your code handles a partner API outage without touching the partner, and the same test runs safely on every build. It also fits naturally with dependency mocking when a team already mocks external calls in CI.

How to Do Resilience Testing

How to Do Resilience Testing

  1. Define steady state. Pick the metrics that show the system is healthy, such as error rate, p95 latency, and throughput, and tie them to your SLOs.
  2. List the failures that matter. Work from your dependency map and past incidents. Every external API, database, cache, and queue is a candidate.
  3. Decide the expected behavior first. For each failure, write down what should happen before you run anything: a fallback, a cached response, a graceful error, or a failover.
  4. Simulate the failure in a safe environment. Start with mocks or virtual services in CI, then move to a staging test environment. Production experiments come last, with a limited blast radius and a clear stop condition.
  5. Measure degradation and recovery. Compare results against steady state, and record how long the system took to recover.
  6. Automate the repeatable tests. Dependency failure and timeout tests belong in CI on every pull request, while larger chaos experiments can run on a schedule.

Resilience Testing Examples

A useful resilience test names three things: the failure you inject, the behavior you expect, and the number that tells you whether it held. These examples cover the failures API teams run into most.

Failure injected

Expected behavior

What to measure

Payment API returns 503

Checkout shows a clear error, retries with backoff, and never creates a duplicate order

Error rate, retry count, duplicate orders

Database responds slowly

Requests time out fast instead of queuing, and failover triggers on latency

p95 latency, connection pool usage, time to failover

Auth token service is down

Existing sessions keep working from cached tokens

Share of requests still served

Cache cluster is lost

Requests fall back to the database without overloading it

Database load, response time

Many instances restart at once

Jittered backoff keeps startup traffic to shared dependencies within limits

Dependency load, total recovery time

Message queue rejects writes

Events are buffered or retried later, and none are lost

Lost messages, backlog drain time

Downstream API returns malformed JSON

The parser rejects it safely and the service falls back

5xx rate, fallback rate

Resilience Testing Tools

Most teams pair a tool that simulates dependency failures close to the code with one that breaks infrastructure on purpose.

Tool

Layer

What it simulates

License

Keploy

API and dependencies

Error responses and unexpected payloads from recorded mocks

Open-source core

Gremlin

Infrastructure

Resource, network, and host failures

Commercial

LitmusChaos

Kubernetes

Pod, node, and network faults

Open source

Chaos Mesh

Kubernetes

Pod, network, I/O, and time faults

Open source

Toxiproxy

Network (TCP)

Latency, timeouts, and dropped connections

Open source

AWS Fault Injection Service

AWS resources

Faults in EC2, ECS, EKS, and RDS

Pay as you go

k6

Load

Realistic traffic while faults run

Open source, paid cloud

Keploy

Keploy records API calls along with the database queries and downstream calls behind them, then turns them into tests and mocks. Because the mocks are plain files, a recorded success can be edited into a failure, such as a 503 or a malformed payload, and replayed in CI.

  • Best for: testing how an API handles failing dependencies on every pull request, without a live environment
  • Simulates: error responses and unexpected payloads from databases and downstream APIs
  • Limitation: works at the application dependency level, so it doesn’t kill nodes or degrade networks
  • License: open-source core under Apache 2.0, with paid plans

Gremlin

Gremlin is a commercial chaos engineering platform for running controlled failure experiments across hosts, containers, and Kubernetes.

  • Best for: teams that want managed chaos experiments with built-in safety controls
  • Simulates: CPU, memory, and disk exhaustion, network latency and packet loss, and process or host shutdown
  • Limitation: infrastructure-level faults need careful blast radius planning, especially in production
  • License: commercial

LitmusChaos

LitmusChaos is an open-source, CNCF chaos engineering project for Kubernetes that defines experiments as Kubernetes resources.

  • Best for: Kubernetes teams that want chaos experiments managed like other cluster resources
  • Simulates: pod deletion, node failures, network latency, and resource stress
  • Limitation: focused on Kubernetes, so it’s less useful for services running elsewhere
  • License: open source

Chaos Mesh

Chaos Mesh is another open-source CNCF project that injects faults into Kubernetes workloads.

  • Best for: fine-grained fault types inside Kubernetes clusters
  • Simulates: pod failures, network partitions and delays, I/O faults, and clock skew
  • Limitation: Kubernetes only, and cluster-wide permissions need careful control
  • License: open source

Toxiproxy

Toxiproxy, from Shopify, is a TCP proxy that sits between an application and its dependencies to simulate network conditions in tests.

  • Best for: lightweight latency and timeout tests in local development and CI
  • Simulates: added latency, bandwidth limits, timeouts, and dropped connections
  • Limitation: works at the TCP level, so it can’t return HTTP-specific failures like a 503
  • License: open source

AWS Fault Injection Service

AWS Fault Injection Service is a managed service for running fault injection experiments against AWS resources, with stop conditions tied to CloudWatch alarms.

  • Best for: teams running on AWS that want managed experiments with automatic stop conditions
  • Simulates: faults in EC2, ECS, EKS, and RDS, including instance stops and network disruption
  • Limitation: only covers AWS resources
  • License: pay as you go, based on usage

k6

k6 is an open-source load testing tool from Grafana Labs. It doesn’t inject faults by itself, but it generates the realistic traffic that load-induced failure tests need, and its thresholds can fail a CI job.

  • Best for: testing retries, backoff, and load shedding under realistic traffic
  • Simulates: user load, which pairs with a fault injection tool running at the same time
  • Limitation: needs another tool to create the failure
  • License: open source, with a paid cloud service

Resilience Testing vs Performance Testing vs Chaos Engineering

Resilience Testing vs Performance Testing vs Chaos Engineering

Aspect

Resilience testing

Performance testing

Chaos engineering

Main question

Does it degrade gracefully and recover?

Is it fast enough under load?

What unknown weakness appears when something breaks?

Input

Planned, specific failures

Load profiles

Experiments with a hypothesis

Where it runs

CI and staging

Staging or a performance environment

Staging and production

Pass criteria

SLOs held and recovery time met

Latency and throughput targets met

Steady state held

Performance testing asks how the system behaves when everything works but demand is high. Resilience testing, however, asks how it behaves when something doesn’t work at all, so the two often run together in load-induced failure tests.

Conclusion

Dependency graphs keep getting wider. AI features add model APIs to request paths, and AI agents call other services with their own retry logic, so every new integration is another partial failure waiting to happen.

Resilience tests that only run during an annual game day can’t keep up with that. The teams that stay ahead will run the cheap tests (timeouts, error responses, backoff) on every pull request and save the big experiments for failures that truly need real infrastructure.

FAQs

Is resilience testing functional or non-functional testing?

Resilience testing is non-functional testing. It doesn’t check whether a feature returns the right result. Instead, it checks how the system behaves, and how quickly it recovers, when something it depends on fails.

What is the difference between resilience testing and recovery testing?

Recovery testing is one type of resilience testing. It focuses only on what happens after a failure ends, such as recovery time and data consistency. Resilience testing also covers how the system behaves while the failure is still happening.

What is the difference between resilience testing and reliability testing?

Reliability testing measures how consistently a system works over time under normal conditions. Resilience testing measures how it behaves when conditions are not normal, such as during a dependency outage or a sudden slowdown.

Can resilience testing run in CI/CD?

Yes. Dependency failure and timeout tests run well in CI when dependencies are mocked or virtualized, because they don’t need real infrastructure to break. Larger chaos experiments usually run on a schedule in staging or production.



More Stories

No posts found matching ""