Canary Testing: A Comprehensive Guide For Developers

Cover Image for Canary Testing: A Comprehensive Guide for Developers
Background

Animesh Pathak

Table of Contents
Table of Contents

What’s Canary Testing, Anyway?

Imagine you’re a miner with a canary in a cage. If the air is toxic, the canary reacts first, giving you a heads-up. Canary testing works similarly for your software. Instead of releasing the whole flock (users) into a potentially toxic or buggy environment, you release just one canary (a small subset of users) to test the waters.

Once you have a new release ready, you can deploy it to one of the environments. Then, you can direct a small portion of your users (around 5% is recommended) to this canary release. These users will experience the new features, while the other group will not encounter any changes.

Why Canary?

In the coal mining days, miners used canaries to detect dangerous gases. If the canary stopped singing, it was a sign that something was wrong. In our tech world, our "canaries" are the early adopters who help us sniff out any issues before a full-scale release.

Developers create automated tests for their software’s new features and modifications. The changes are deployed to a testing environment where others can explore and interact with the new features. If everything goes smoothly, the new software update is rolled out to the production environment, allowing end users to benefit from the newly added feature.

However, given the nature of software, bugs tend to move into production. As humans, it is impossible to anticipate every potential edge case. Moreover, deadlines and budget constraints add to the pressure.

Setting Up Your Canary Test

To execute the canary test, two approaches are mainly implemented to achieve reliable outcomes. We’ll be using GoLang for our examples because, well, Go is awesome. Here are those two approaches:

Feature Flags

Start by incorporating feature flags into your code. These nifty toggles allow you to enable or disable certain features at runtime. Perfect for controlling who gets to see your new changes.

With the feature flag, you can limit the release to 5% of the users and monitor the key metrics. This approach is handy for business stakeholders who need to test new features before implementing them for everyone. However, while performing a canary test, if any issue is detected during the deployment method, you can easily disable the new features by turning the feature off.

package main

import (
    "fmt"
    "feature" // Assuming a feature package for checking feature status
)

func main() {
    if feature.IsFeatureEnabled("new-feature") {
        // User Input and Processing
        var userInput string
        fmt.Println("Enter your name:")
        _, err := fmt.Scanln(&userInput) // Read user input
        if err != nil {
            fmt.Println("Error reading input:", err)
        } else {
            fmt.Println("Hello,", userInput + "!") // Process input
        }
    } else {
        // Code for the old behaviour
        fmt.Println("Hello, User!")
    }
}

In this code, the IsFeatureEnabled function is expected to perform the actual check for the feature’s status. Depending on the result, the program executes either the code intended for the new feature or the code representing the old behaviour.

Time to deploy! Let’s push our changes but only to a small fraction of our users. In Go, we can use something like this:

// deployment.go
package main

import "github.com/my/deployment/package"

func main() {
    // Deploy to 5% of users
    deployment.Rollout("new-feature", 5)
}

Blue-Green Deployment

Blue-green deployment is a software release strategy that minimizes downtime and risk by running two identical production environments, often referred to as "blue" and "green."

Once the new version is deployed to the green environment, a series of comprehensive tests are conducted to ensure its functionality, performance, and integration meet expectations. This meticulous testing phase helps identify potential issues before exposing the new version to users. If the green environment proves stable and successful in the tests, traffic is seamlessly switched from the blue to the green environment, making the latter the new production environment.

To implement blue-green deployment successfully, organisations often leverage automated deployment tools and adhere to continuous integration practices to streamline the process and ensure a smooth transition between environments.

package main

import (
    "fmt"
    "net/http"
    "os"
)

var isBlue bool

func main() {
    isBlue = true
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

func handler(w http.ResponseWriter, r *http.Request) {
    if isBlue {
        fmt.Fprint(w, "Blue Environment - Old Behaviour\n")
    } else {
        fmt.Fprint(w, "Green Environment - New Feature\n")
    }

    isBlue = !isBlue
}
func switchEnvironment() {
    isBlue = !isBlue
    fmt.Printf("Switched to %s environment\n", map[bool]string{true: "blue", false: "green"}[isBlue])
}

func init() {
    switchEnvironment()
}

The handler function serves different responses based on whether the environment is in the "blue" or "green" state. The switchEnvironment function allows you to switch between blue and green environments manually.

Monitoring Like a Hawk

Once your canaries are out there singing, you need to listen carefully. Monitoring is key. Keep a close eye on your system’s performance, error rates, and user feedback.

Logging and Metrics

Enhance your logging game. Log relevant information about the new feature and monitor metrics that matter. Go’s standard library makes this a breeze.

// logger.go
package main

import "log"

func LogFeatureUsage(userID string) {
    log.Printf("User %s is using the new feature", userID)
}

Error Tracking

Integrate error-tracking tools. Identify and squash those bugs before they spread.

Gradual Rollouts

Once the deployment of the new feature has been successful, we can enable the feature flag and all users can enjoy the new feature.

// deployment.go
package main

import "github.com/your/deployment/package"

func main() {
    // Increase to 20% of users
    deployment.Rollout("new-feature", 20)
}

We can then monitor the usage of the new feature with our logs or analytics dashboards to see if users are adopting the feature and how they’re using it. The release of the feature has been completely independent of the app’s deployment!

Rollbacks

Oops, something went wrong. No worries! Quickly roll back to the previous version using your deployment tool.

// deployment.go
package main

import "github.com/your/deployment/package"

func main() {
    // Rollback to the previous version
    deployment.Rollback("new-feature")
}

Monitoring and Observability Tools

Monitoring and observability are crucial components in overseeing canary releases and ensuring effective testing. These tools help teams gain insights into the performance, health, and behaviour of applications during canary deployments. Here are some notable monitoring and observability tools that can aid in this process:

  1. Prometheus:Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It excels in collecting metrics, making it suitable for tracking the performance of canary releases. Prometheus is particularly valuable for its support of multi-dimensional data collection and querying.
  2. Grafana:Grafana is a popular open-source platform for monitoring and observability that works seamlessly with data sources like Prometheus. It provides a customizable and interactive dashboard, allowing teams to visualise key metrics, logs, and traces. Grafana is valuable for gaining insights into application behaviour during canary releases.
  3. Datadog:Datadog is a cloud-based monitoring and analytics platform that offers comprehensive observability solutions. It provides real-time monitoring, logs, and traces, enabling teams to correlate data and troubleshoot issues effectively. Datadog supports canary releases by offering end-to-end visibility into application performance.
  4. New Relic:New Relic is a cloud-based observability platform that provides insights into application performance, errors, and infrastructure. It offers APM (Application Performance Monitoring) capabilities and supports distributed tracing, making it valuable for monitoring canary releases and identifying any performance regressions.
  5. Istio:Istio is an open-source service mesh that enhances visibility and control over microservices-based applications. It includes features like traffic management, security, and observability. Istio’s observability tools, such as Prometheus integration and distributed tracing, are beneficial for monitoring canary releases in complex, microservices architectures.

When implementing canary releases, a combination of monitoring and observability tools ensures that teams have comprehensive insights into the impact of changes on application performance, allowing for effective testing and rapid identification of any issues that may arise during the release process.

Challenges and Solutions of Canary testing

Cons of Canary Testing

Canary testing presents challenges such as the risk of false positives or negatives, where issues in the canary group may not accurately mirror the broader user base, leading to an incomplete understanding of the impact of changes. Managing multiple environments demands meticulous coordination, adding operational overhead. Striking a balance between the canary group’s size and its representativeness is crucial, and requires constant attention. Improper execution can yield skewed results, hindering accurate assessments of changes in the overall system.

Pros of Canary Testing

Canary testing minimizes software deployment risks by releasing changes to a limited user subset before wider adoption. This controlled exposure enables early identification and resolution of potential issues, reducing the impact on a larger audience and enhancing overall system stability. Valuable insights into real-world performance are gained, informing informed decisions based on actual user interactions. Aligned with continuous delivery practices, canary testing cultivates an iterative and cautious approach, fostering a culture of continuous improvement and innovation within development teams.

Conclusion

The concept of canary testing is a strategic approach to introducing changes in a controlled manner by initially deploying them to a limited subset of users or systems.

The successful implementation of canary testing hinges on careful planning. A well-thought-out strategy is crucial to ensure a smooth rollout process, encompassing considerations for the deployment mechanism, monitoring protocols, and contingency plans in case issues arise during the limited release.

Additionally, effective communication is paramount. Transparent and clear communication among development, operations, and other involved teams establishes a collaborative environment where stakeholders are informed about the canary testing strategy, progress, and any necessary adjustments.

FAQ’s

How is canary testing different from A/B testing?

Canary testing involves deploying changes incrementally to a small subset of users or systems before rolling them out to the entire user base. It focuses on detecting potential issues early by monitoring metrics and user feedback.

In contrast, A/B testing compares two or more versions of a feature or webpage to determine which one performs better based on predefined metrics, often involving larger user segments in a randomized manner to assess differences in user behaviour or preferences.

What are some successful examples of canary testing in real-world applications?

Google employs a sophisticated canary analysis system to gradually roll out changes across its vast infrastructure, allowing for early detection of issues before they affect a significant portion of users. By carefully monitoring key metrics such as latency, error rates, and user engagement, Google can automatically scale back or halt deployments if abnormalities are detected during the canary phase.

This approach enables Google to maintain high availability and reliability while continuously delivering updates and improvements to its services.

How do you set up a canary deployment pipeline?

To set up a canary deployment pipeline, first, define criteria for the canary release, such as metrics for performance and stability. Next, configure deployment tools like Spinnaker or Kubernetes to automate the deployment process. Integrate monitoring tools like Prometheus and Grafana to collect and analyze metrics during the canary release, enabling informed decisions about proceeding or rolling back changes.

Author

Author Avatar
Author Details
Author Name: Animesh pathak
Author Description:

  • More Stories

    Cover Image for Canary Testing: A Comprehensive Guide for Developers

    Canary Testing: A Comprehensive Guide for Developers

    Animesh Pathak

    Table of Contents What’s Canary Testing, Anyway? Imagine you’re a miner with a canary in a cage. If the air...

    Cover Image for Mock vs Stub vs Fake: Understand the difference

    Mock vs Stub vs Fake: Understand the difference

    Arindam

    Table of Contents Introduction Testing software is like putting it through a series of challenges to make sure it’s tough...

    Cover Image for Writing test cases for Cron Job Testing

    Writing test cases for Cron Job Testing

    Animesh Pathak

    Table of Contents Understanding Cron Jobs: A Quick Recap Cron is a time-based job scheduler in Unix-like operating systems. It...

    Cover Image for Improving Code Quality and Accelerating Development: The Continuous Testing Way

    Improving Code Quality and Accelerating Development: The Continuous Testing Way

    Prajwal

    Table of Contents Introduction In the fast-changing world of software development, teams struggle to maintain good code quality while shortening...

    Cover Image for Understanding Testing in production

    Understanding Testing in production

    Arindam

    Table of Contents Introduction Testing in production was previously ignored by Product Developers, But recently it gaining Popularity Again! Even,...

    Cover Image for 5 Unit Testing Tools You Must Know in 2024

    5 Unit Testing Tools You Must Know in 2024

    Arindam

    Table of Contents Introduction: Unit testing is one of the most important areas to ensure code coverage and basic testing...

    Cover Image for Exploring Various Protocols : HTTP to Databases

    Exploring Various Protocols : HTTP to Databases

    Shivam

    Table of Contents Table of Contents Unraveling the Mystery of various Protocols The Role of Protocols in Network Communication Examples:...

    Cover Image for Demystifying Cron Job Testing

    Demystifying Cron Job Testing

    Animesh Pathak

    Table of Contents What is Cron Job? Before we dive into the world of testing Cron jobs, let’s first know...

    Cover Image for Building Custom YAML-DSL in Python

    Building Custom YAML-DSL in Python

    Animesh Pathak

    Table of Contents In this blog post, I will guide you through the process of building a custom DSL in...

    Cover Image for eBPF, Service Mesh and Sidecar

    eBPF, Service Mesh and Sidecar

    Animesh Pathak

    Table of Contents The operating system is like the boss of your computer, handling security, networking, and keeping an eye...

    Cover Image for MongoDB in Mock Mode: Acting the Server Part

    MongoDB in Mock Mode: Acting the Server Part

    Ritik

    Table of Contents In the contemporary software development landscape, unit tests have become paramount for ensuring software quality. A prevalent...

    Cover Image for Capture gRPC Traffic going out from a Server

    Capture gRPC Traffic going out from a Server

    Mehfooz

    Table of Contents How does gRPC work? A quick Google search would tell you that it uses HTTP/2.0 under the...

    Cover Image for Integration vs E2E Testing: What worked for me as a charm

    Integration vs E2E Testing: What worked for me as a charm

    Sarthak Shyngle

    Table of Contents When it comes to testing software applications, various testing techniques can be employed. Three common testing methods...

    Cover Image for Automated E2E tests using Property Based Testing  | Part II

    Automated E2E tests using Property Based Testing | Part II

    charan

    Table of Contents If you haven’t visited Part I, I highly recommend you go through it for a better understanding...

    Cover Image for Automated End to End tests using Property Based Testing  | Part I

    Automated End to End tests using Property Based Testing | Part I

    charan

    Table of Contents " Engineers call them edge cases. I call them: what our users do " – Noah Sussman...

    Cover Image for Go Mocks and Stubs generator Made Easy

    Go Mocks and Stubs generator Made Easy

    Jain

    Table of Contents Testing network stuff like APIs and database calls can be a real pain: I find myself burning...