Fabric: How can I unit test my fabfile?

Question

In the previous project I was working on, our fabfile got out of control. While the rest of our project was well-tested, we didn't write a single test for our fabfile. Refactoring was scary, and we weren't confident a fabric command would work how we expected until we ran the command.

I'm starting a new project, and I'd like to make sure our fabfile is well-tested from the beginning. Obey the Testing Goat has a great article discussing some possible strategies, yet it has more questions than answers. Using fabtest is a possibility, although it seems to be dead.

Has anyone successfully unit tested their fabfile? If so, how?

Answer 1

1. run your Fabfile task in a Docker instance

2. use docker diff to verify that the right files were changed by the Fabfile.

This is still quite a bit of work, but it allows testing without excessive Fabfile modifications.

Answer 2

Have you tried python-vagrant ? It seems to do the same thing that fabtest does, but it includes some Fabric demos and is still used and maintained.

Answer 3

The slides - mentioned by Henrik Andersson - from back then are available here

Robin Kåveland Hansen replied to me:

There are some examples of the types of refactoring that we did in order to keep our fabric code well-tested there.

In general I would say the best advice is to try avoiding low-level code such as shell commands in higher level code that makes decisions about what code to run, eg. isolate effect-full code from code that makes decisions.

Branching increases the amount of test-cases that you need and it's a lot more effort to write good test-cases for code that changes state on some server.

At the time, we used mock to mock out fabric to write test-cases for branch-less code that has side-effects on the server, so the code + tests would look a lot like this
Obviously this has the weakness that it won't pick up bugs in the shell commands themselves. My experience is that this is rarely the cause of serious problems, though.

Other options using mock would be to use the following idea to run the tests locally on your machine instead of remotely

Maybe the most robust approach is to run the tests in vagrant, but that has the disadvantage of requiring lots of setup and has a tendency to make the tests slower.
I think it's important to have fast tests, because then you can run them all the time and they give you a really nice feedback-loop.

The deploy-script I've written for my current employer has ~150 test cases and runs in less than 0.5 seconds, so the deploy-script will actually do a self-test before deploying.
This ensures that it is tested on each developer machine all the time, which has picked up a good few bugs for example for cases where linux and mac osx behave differently.