optimize compiletime in Continuous Integration

Question

Maybe that's just a mad man's dream, but..

In my company we have a big C# .NET project, with ~25 solutions (very old) and ~3.5 mio. loc. The problems I'm facing is: too slow build times, right now it takes 7 minutes with SSD (dev machines), 15 minutes+ in VM with normal harddrives (would be the TeamCity build system I'd like to get deployed). I know, build system should be fastest, but that's nothing I can change in the short term.

I want to shorten the commit-build-unittest feedback-loop for the devs (preferably on Teamcity machine right now) by just compiling the project(s) which were touched by the last commit, taking all other assemblies from eg a local nuget server (teamcity server itself with version 7.0).

Now that would immensely cut down the feedback loop (15 minutes to less then a minute, given real unit tests) for small commits.

I know the problem of such a partial compile is the possibility of skipping compile errors (mismatching interfaces could go unnoticed), but that would be mitigated by running a second (Teamcity?) build server instance which runs the whole enchilada, in parallel. But getting first feedback immediately is very important to me.

Now my question: is there any build system/continuous integration system which can handle this task? Or would I have to write my own commit-aware background service? Which would be a bit nasty, as we use FinalBuilder Scripts, and that Format doesn't seem to be readable by any API (but didn't dig deep enough into that).

PS: Also, I would like to only run the unit tests for the projects which were changed by the last commit, or at least prioritize them. But that's an afterthought.

Answer 1

Most of the available CI engines adopts a deployment pipeline process, which is specifically designed to reduce the feedback time in the development loop. It works like this, with a FAIL status immediately if any of the steps went wrong.

It is suggested ( by this book ) to have the first 4 steps to be less then 2-5 mins even for the most complicated projects, if it goes above that then there is a problem with your configuration and the way you use CI process.

Code commit
 triggers --->

 Step 1. Automatic checkout on CI side
 Step 2. Compile code, ideally 1-2 mins  
 Step 3. Save binaries to the artifact repository
 Step 4. Unit test, ideally 1-2 mins
 Step 5. Deploy to staging
 Step 6. Automated integration testing
 Step 7. Automated acceptance testing
 ------------------------------------
 Manual testing
 Manual deploy to production

Specifically to Step 2 you can:

a. Split large solution into separate tiers. Each tier would have it's own Visual Studio solution and only projects relevant to that tier, in a sense you perform decentralisation of the initial bulky solution. On step 5 you would know how to assemble tiers into usable application.

b. In TeamCity configuration you can specify whether to perform a clean checkout or use the source already available (can save time on step 1). Check that the target to MSBuild is set to Build which will pick up only the source files that have changed since the last build (save time on step 2).

From personal experience option a) is the best one, but if needed you can also use both a) and b), which however may bring some latent bugs with old files being kept for longer than needed. Teamcity also supports several agents (up to 3 in free edition) which will allow you to run tasks in parallel.

Answer 2

Use a system of dependent builds. If you are using SVN with each project in its own folder you can set up a CI project for each of your c# projects that builds just that project and notifies you very quickly whether it succeeds or fails.

Set up a second CI project that has a Build Trigger so that they second CI project builds if your first one succeeds and have the second project run your test cases. I have done this using Jenkins with good success.

For a complete build you could have another CI project that watches the root folder for any changes and kicks off a build of the entire solution.

Setting it up like this you can have each project build and return results quickly when code is checked in and a slower return on tests for the project

Answer 3

MSBuild does differentiate between "build" and "rebuild" for the build script targets - the normal build only builds projects it sees as being changed since the previous builds.

That said, it should suffice not to clean out the build directory of the TeamCity agent when getting the source from source control (the abiity to do so may depend on the SCM - it seems to work fine with Mercurial) and just trigger a build, not a rebuild.

Regarding unit tests, TeamCIty can execute the failed ones first, but figuring out which tests address which source code parts seems like a difficult task to me, and not one supported by TeamCity AFAIK.