Continuous integration and web deployment workflow

Question

We are seeking to improve several aspects of our SDLC. What we have right now is the following, and these are fairly new-ish and light years beyond what we had a couple years ago:

• Code: we are predominantly a .Net shop. We develop mostly web apps (MVC/SQL Server). But we also have some batch console apps.
• SCM: Git and Enterprise GitHub. Most work is done directly in the master branch (queue boo-hiss) but we recognize the need to move to a branching model. We are going to implement some flavor of Vincent Driessen's model .
• CI server: Team City. Our immediate need a year or two ago was to get deployment automation set up to avoid hazard-laden manual scripts. We now use TeamCity to kick off deployment scripts via PowerShell. (It uses msbuild to simply build targeted configs -- standard is Debug and Release; we've set up Dev, QA, and Prod configs in each solution -- defined in the Visual Studio solutions.) Sadly, we have yet to actually use TeamCity for it's intended purpose, which is continuous integration. For that, we need to implement TDD. The deployment automation pulls from the master branch for each repository.
• Environments: we have 3 environments, each with dedicated web, batch, and database servers. We call them Dev, QA, and Prod. Dev is our testing environment. QA is for end users to test. Prod is prod. Our deployment automation, in general, pushes each app to each environment one time. It cannot currently handle pushing the same app to multiple places in a single environment.

Note that we are starting to have situations where we need to do parallel development: one feature is being worked on in one branch; another in another branch; and a hotfix in another. Obviously doing everything in master is unacceptable.

Here is where I'd like to go:

• Begin a branching model in Git.
• Move the deployment automation out of Team City and into a custom, robust deployment tool.
• Implement TDD.
• Use TeamCity for its intended purpose, continuous integration.

Those are the high level goals, and I have rough ideas on how to start doing this (especially with the deployment automation). However, I have some rather large outstanding issues that are frustrating my (woo-ha-ha) master plan. So here are the items I'd love input from SO community on:

1. When you do CI and kick off automated tests, what branch(es) do you do this against? I'm thinking that, in theory, you don't want "broken" code in your "develop" and (obviously) "master" branches. Do you kick of CI against "develop" or all branches?
2. Do you merge into master before or after you move your code to prod? Put another way, do you push to prod from develop or master?
3. Sometimes we will need to test (at least in QA, maybe in Dev) parallel changes, like a bugfix and a feature. Do you literally set up multiple test websites (eg myappqa1.blah.com and myappqa2.blah.com, etc.)? How do these multiple environments play into your deployment automation?
4. Just of out of curiosity, do you do nightly builds -- if so, against what branch? Do you deploy this nightly build anywhere?
5. How do you package your apps (ie store releases)? Right now, we use TeamCity and PowerShell to run msbuild against config's defined in the VS. It will put all environment builds into a zip file. This is stored as an artifact of the TeamCity build. Here is a visual of what I'm talking about: Then another script will pick the right environments out of the zip file and deploy. Should we store the release package in Team City or Git (using the "release" feature) somehow?
6. Related, with your deployment automation: do you deploy a previously packaged/built app, or do you build and deploy all in one fell swoop?

Your feedback much appreciated!

Thanks Tom

Answer 1

As a developer for BuildMaster (a tool that fits in just after TeamCity ends and may be what you're looking for), I can try to answer most of these.

When you do CI and kick off automated tests, what branch(es) do you do this against? I'm thinking that, in theory, you don't want "broken" code in your "develop" and (obviously) "master" branches. Do you kick of CI against "develop" or all branches?

We follow the "branch-by-rule" and "branch-by-exception" models, where development is done either against a "release" branch, or it is done against trunk/master. Feature branches can exist as well for local development, but are not by themselves built and released into integration and beyond (ie they are merged in with the "release" branch, whether it's on trunk/master or separate).

You can read more about this process in Source Control Done Right .

Do you merge into master before or after you move your code to prod? Put another way, do you push to prod from develop or master?

It is strongly recommended that you do not keep separate branches for individual environments. I know this is becoming a trend lately with all the distributed source control systems but from our experience, it leads to many problems if not done with exceptional discipline. The main problems are: missing/forgotten merges, incompatible merges from one environment to the next, untested code merged into a new environment accidentally, and so on.

So to answer the question, either the branch code (for "branch-by-rule") or trunk/master code (for "branch-by-exception") is what is pushed into production after it has been released to the previous testing environments.

Sometimes we will need to test (at least in QA, maybe in Dev) parallel changes, like a bugfix and a feature. Do you literally set up multiple test websites (eg myappqa1.blah.com and myappqa2.blah.com, etc.)? How do these multiple environments play into your deployment automation?

This type of test is essentially a pre-integration test, as testers are testing features without them being integrated with other development. The fully integrated code is what needs to be tested. You can have any number of pre-integration environments just like you can have any number of local development environments.

Just of out of curiosity, do you do nightly builds -- if so, against what branch? Do you deploy this nightly build anywhere?

Yes, but they are essentially useless for our team because once you've automated the build & release process, it's trivial to create new builds as they are necessary. For larger teams or public-facing downloads it can of course be helpful. The build is aptly deployed to the "integration" environment. The branch that is used is determined by the release that the build is created for.

How do you package your apps (ie store releases)? ... Should we store the release package in Team City or Git (using the "release" feature) somehow?

Separate builds per environment is an anti-pattern. The unit of deployment should be the same for every environment you deploy to, with the exceptions of course being:

• Application Configuration - such as for .NET, web.config / App.config files
• Infrastructure Deployment - such as setting up IIS/website (this is orthogonal to application deployment and is typically handled by an Ops team thus the need for DevOps, but that's an entirely different subject)
• Database Changes - these are special in their own right because once you DROP a column, it can't be dropped again. You can read more about these in: Database Changes Done Right

Related, with your deployment automation: do you deploy a previously packaged/built app, or do you build and deploy all in one fell swoop?

The common approach is to separate build and deployment. For simple websites (ie brochure sites) there's no need to make this distinction and "continuous production" is good enough.

This can of course get murky when other pieces and parts need to come together. As an example, releasing our BuildMaster software requires an installer for the public to use. The installer is not used at all until the later environments, the deployment to the earlier environments is done via the pre-packaged build artifacts (which the installer uses as well). This is where keeping a single deployment unit (ie the set of artifacts) from the beginning makes it easy to deploy to any environment with or without an installer.

TL,DR - It's really hard to summarize such a broad topic... but I recommend checking out this new Incremental Continuous Delivery paper authored by a colleague if you're looking to get started without investing heavily up-front.