deployment doesn’t equal release
As part of continuous deployment, we have to build an automated deployment pipeline that allows us to frequently deploy and test new functionality and bug fixes. We will first test the deployment on a staging environment. Only if a deployment passed all tests, we will release it to the production environment:
- running unit tests (triggered by check-in)
- running integration builds
- running integration tests
- deploy the artifacts to the staging environment
- perform more tests (e.g. smoke and acceptance tests)
- If all tests are passed, the new release can be rolled out across the production environment.
Having such a deployment strategy in place becomes very handy when instant releases (e.g. bug fixes) are required. The goal is to fully automate this deployment pipeline to shorten the time (and pain) from check-in to release. While doing so, the solution needs to be able to respond to requests at all times, even when in the process of deploying or testing a new release. To achieve zero downtime, we commonly take advantage of the two deployment strategies “blue-green deployment” and “canary releasing”.
It is important to understand that the risk is exponentially greater the more check-ins that occur between releases. So actually, launching with new releases more frequently is less risky because the scope of the changes are better understood. This is counterintuitive to many people.
Blue-green deployments are based on two identical deployment environments – one for production and one for staging. The key is to ensure that the two environments are truly identical, including the data it manages. Zero downtime releases are achieved by deploying to the staging environment. After smoke testing the new deployment, traffic will be routed to the staging environment which now becomes the production environment. While blue-green deployments provide a simple and powerful way to test a deployment before going into production, it might require staging environments of similar size and capacity to perform capacity tests, which might not be an economically feasible option for large scale services. Microsoft Azure Web Sites and its staging slots provide an out-of-the-box experience for “blue-green deployments”. It basically provides two deployment slots which can be swapped. In most scenarios, this will be the default deployment strategy.
Canary releasing addresses the challenge of testing a new release with only a subset of the servers. This approach can also be used for A/B testing: small percentage of the users will be routed to the new service while the majority still works against the “old” version. This allows the team to get direct feedback without being at risk of impacting the majority of users. It is actually possible to have multiple versions running in parallel. The same approach can be used to perform capacity tests without routing actual users to the release which is under test (basically test the new version in production without routing actual users to it). While “blue-green deployments” are simple, doing “canary releases” is more complicated, because all instances within a Web Site are based on the same deployment. As part of this series, I will discuss the use of a custom router which acts as a reverse proxy. Using this approach allows to route certain users to the “canary deployment” while the majority of users work against older releases.