Skip to main content

Buggy Test Suites​

FreeTeamsScaleEnterpriseSelf-HostedDedicated

When Keploy generates tests, some test cases might fail due to various reasons such as endpoint issues, data mismatches, or API changes. The buggy test suites page helps you identify, understand, and fix these failing tests.

Viewing Buggy Test Suites​

Navigate to the test suites generated with the red exclamation icon to view all test suites that contain failing test cases. Each buggy suite displays:

  • Suite Name: The name of the test suite containing failed tests
  • Test Steps: Steps in the suite
  • Failure Reason: Reason why the test suite is buggy

Understanding Failure Reasons​

For each buggy test suite, you can find the detailed explanations of why tests are marked as buggy. Common failure reasons include:

1. Endpoint Not Found (404 Errors)​

Example Failure Reason:

The response returned a 404 status code for the 'Create Owner' step, indicating the endpoint '/owners' was not found. This contradicts the documented cURL examples and schema, which show that this endpoint should exist and return a 201 status code upon successful creation.

What this means:

  • The API endpoint that was working during recording is no longer available
  • The endpoint URL might have changed
  • The API server might be down or misconfigured

How to fix:

  1. Verify the endpoint URL is correct
  2. Check if the API server is running
  3. Review API documentation for any endpoint changes
  4. Update the test suite if the endpoint has moved

2. Schema Validation Failures​

Example Failure Reason:

Response schema validation failed. Expected property 'id' of type 'number' but received 'string'. The API response structure has changed from the recorded version.

What this means:

  • The API response format has changed since recording
  • Data types don't match the expected schema
  • New required fields might have been added

3. Authentication Issues​

Example Failure Reason:

Authentication failed with 401 Unauthorized. The API key or token used during recording may have expired or been revoked.

What this means:

  • API credentials have expired or changed
  • Authentication method has been updated
  • Permission levels may have changed

Assertion Failures​

The buggy suites page provides detailed assertion failure information to help you understand exactly what went wrong:

Response Status Assertions​

Expected: 201 Created
Actual: 404 Not Found
Assertion: status_code_equals
Message: The endpoint returned an unexpected status code

Response Body Assertions​

Expected: {"id": 123, "name": "John Doe", "email": "john@example.com"}
Actual: {"error": "User not found", "code": 404}
Assertion: json_body_equals
Message: Response body structure completely different from expected

Debugging Actions​

For each buggy test case, you can take several debugging actions:

1. View Full Test Details​

Click on any failed test to see:

  • Complete request details (URL, headers, body)
  • Full response details (status, headers, body)
  • All assertion results with expected vs actual values

2. Compare with Recorded Version​

View the side-by-side comparison between:

  • Original Recording: The request/response captured during recording
  • Current Execution: The actual request/response during test execution
  • Differences Highlighted: Visual indicators showing what changed

3. Manual Test Execution​

Test the endpoint manually to verify:

# Example manual cURL test
curl -X POST \
  'https://api.example.com/owners' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer your-token' \
  -d '{
    "name": "John Doe",
    "email": "john@example.com"
  }'

4. Update Test Expectations​

If the API behavior has legitimately changed:

  1. Re-record the test: Capture new expected behavior
  2. Update assertions: Modify expected values to match new API
  3. Add new test cases: Cover additional scenarios if needed

Getting Help​

If you're unable to resolve buggy test suites:

  1. Check Documentation: Review API documentation for recent changes
  2. Contact Support: Reach out to the development team for API-related issues
  3. Community Forums: Ask questions in Keploy community channels
  4. Share Test Details: Provide complete test execution logs when seeking help

Remember, buggy test suites often indicate real issues with your API or environment. Use them as an early warning system to maintain API quality and reliability.