Deployment and release strategies for Microsoft Azure Web Sites

One of the easiest ways to implement continuous deployment with web sites is to use Git. Developers can write Git hooks that push the deployable repository to the web site repository. When we take this approach, it is important to fully script the creation and configuration of the web site. It is not a good practice to “manually” create and configure it. This might not be apparent, but it is crucial for supporting disaster recovery, creating parallel versions of different releases, or deploying releases to additional data centers. Further, the separation of configuration and settings from the deployable artifacts makes it easy to guard certificates and other secrets, such as connection strings.
The proposed approach is to create a web site (including staging slot) for each releasable branch. This allows deployment of new release candidates by simply pushing the Git repository to the staging web site. After testing, this can be switched to the production environment.
As described above, it is recommended that we create two repositories, one for the creation and configuration of the web site and one for the deployable artifacts. This allows us to restrict access to sensitive data stored in the configuration repository. The configuration script must be idempotent, so it produces the same outcome regardless of if it runs the first or the hundredth time. Once the web site has been created and configured, the deployable artifacts can be deployed using Git push to the staging web site’s Git repository. This push should take place with every commit to the release repository.
It is important that all web site dependencies, such as connection strings and URLs, are sourced from the web site’s application and connection string settings. (Do not make them part of the deployable artifacts!) This allows us to deploy the same artifacts across different web sites without interference. For this example, assume we have an application that consists of two sites, one serving as the frontend and the other as the backend. The backend site also uses storage services (Figure 1).

Application consisting of two sites

Figure 1: Application consisting of two sites

The first step is to split the application into independent deployable components. Each component has its own source repository. Because the backend is the only component that accesses the storage service, we can group them together. The configuration script creates the web site for each component as well as the containing resources, such as storage accounts or databases. Further, it configures all dependencies. In the example below, the script for site 1 will configure the site 2 URL as an application setting. Splitting an application into independent deployable components (Figure 2).

Splitting an application into independent deployable components

Figure 2: Splitting an application into independent deployable components

There are different strategies to handle code branches when releasing new functionality. The following two are commonly used:

  • Keep the master always deployable and use short-lived branches for feature work.
  • Create long-lived branches for releases and integrate feature work directly into the master.

In this series of posts I will focus on the second approach—creating long-lived branches for every new release. The benefit of this approach resides in the fact that there is a 1:1 relationship between a specific release and its corresponding web site creation and configuration script. This makes deploying previous versions extremely simple because we just run the respective script and then deploy the component. It also allows us to easily run multiple releases of the same component in parallel, which is great for A/B testing.

The next posts will cover how to manage long-lived branches for releases while working on features on master. So stay tuned…

Deployment and release strategies

deployment doesn’t equal release

As part of continuous deployment, we have to build an automated deployment pipeline that allows us to frequently deploy and test new functionality and bug fixes. We will first test the deployment on a staging environment. Only if a deployment passed all tests, we will release it to the production environment:

  1. running unit tests (triggered by check-in)
  2. running integration builds
  3. running integration tests
  4. deploy the artifacts to the staging environment
  5. perform more tests (e.g. smoke and acceptance tests)
  6. If all tests are passed, the new release can be rolled out across the production environment.

Having such a deployment strategy in place becomes very handy when instant releases (e.g. bug fixes) are required. The goal is to fully automate this deployment pipeline to shorten the time (and pain) from check-in to release. While doing so, the solution needs to be able to respond to requests at all times, even when in the process of deploying or testing a new release. To achieve zero downtime, we commonly take advantage of the two deployment strategies “blue-green deployment” and “canary releasing”.

It is important to understand that the risk is exponentially greater the more check-ins that occur between releases. So actually, launching with new releases more frequently is less risky because the scope of the changes are better understood. This is counterintuitive to many people.

Blue-green deployment
Blue-green deployments are based on two identical deployment environments – one for production and one for staging. The key is to ensure that the two environments are truly identical, including the data it manages. Zero downtime releases are achieved by deploying to the staging environment. After smoke testing the new deployment, traffic will be routed to the staging environment which now becomes the production environment. While blue-green deployments provide a simple and powerful way to test a deployment before going into production, it might require staging environments of similar size and capacity to perform capacity tests, which might not be an economically feasible option for large scale services. Microsoft Azure Web Sites and its staging slots provide an out-of-the-box experience for “blue-green deployments”. It basically provides two deployment slots which can be swapped. In most scenarios, this will be the default deployment strategy.

Canary releasing
Canary releasing addresses the challenge of testing a new release with only a subset of the servers. This approach can also be used for A/B testing: small percentage of the users will be routed to the new service while the majority still works against the “old” version. This allows the team to get direct feedback without being at risk of impacting the majority of users. It is actually possible to have multiple versions running in parallel. The same approach can be used to perform capacity tests without routing actual users to the release which is under test (basically test the new version in production without routing actual users to it). While “blue-green deployments” are simple, doing “canary releases” is more complicated, because all instances within a Web Site are based on the same deployment. As part of this series, I will discuss the use of a custom router which acts as a reverse proxy. Using this approach allows to route certain users to the “canary deployment” while the majority of users work against older releases.

Continuous deployment – automation is key

During the last couple of months I had many discussions on DevOps and especially continuous deployment. These discussions were not only about technology – they were also about a cultural change:

Delivering a service instead of shipping software requires a cultural change in how development teams and operation teams interact – there has to be a joint accountability to deliver the SLA as one team. A central part of continuous delivery is automation, from the check-in of new code to build, test and deploy – automation is the key to continuously deliver a service. Gone are the days of the waterfall process where the developer hands the code to the test department to hand it to the ops guys. It is one team that is responsible to develop, test and operate the service.

Over the next couple of weeks I plan to blog concrete guidance on how to build an automated deployment pipeline using Azure Web Sites. Here’s a rough outline of the topics to come:

Stay tuned …