Unit Testing: Correctly wrapping library calls

Question

I've been reading a lot about unit testing recently. I'm currently reading The Art of Unit Testing by Roy Osherove . But one problem isn't properly addressed in the book: How do you ensure that your stubs behave exactly like the "real thing" does?

For example, I'm trying to create an ImageTagger application.There, I have a class ImageScanner which job it is to find all images inside a folder. The method I want to test has the following signature: IEnumerable<Image> FindAllImages(string folder) . If there aren't any Images inside that folder, the method is supposed to return null. The method itself issues a call to System.IO.Directory.GetFiles(..) in order to find all the images inside that folder.

Now, I want to write a test which ensures that FindAllImages(..) returns null if the folder is empty. As Osherove writes in his book, I should extract an Interface IDirectory which has a single method GetFiles(..) . This interface is injected into my ImageScanner class. The actual implementation just calls System.IO.Directory.GetFiles(..) The Interface however allows me to create stubs which can simulate Directory.GetFiles() behavior.

Directory.GetFiles returns an empty array if there aren't any files present. So my stub would look like this:

   class EmptyFolderStub : IDirectory
   {
     string[] GetFiles(string path)
     {
       return new string[]{};
     }
   }

I can just inject EmptyFolderStub into my ImageScanner class and test whether it returns null. Now, what if I decide that there's a better library for searching for files? But its GetFiles(..) method throws an exception or returns null if there are no files to be found. My test still passes since the stub simulates the old GetFiles(..) behavior. However, the production code will fail since it isn't prepared to handle a null or an exception from the new library.

Of course you could say that by extracting an Interface IDirectory there also exists a contract which guarantees that the IDirectory.GetFiles(..) method is supposed to return an empty array. So technically I have to test whether the actual implementation satisfies that contract. But apparently, aside from Integration Testing, there's no way to tell if the method actually behaves this way. I could, of course, read the API specification and make sure that it returns an empty array, but don't think this is the point of unit testing.

How can I overcome this problem? Is this even possible with unit testing or do I have to rely on integration tests to capture edge cases like this? I feel like I'm testing something which I don't have any control over. But I would at least expect that there is a test which breaks when I introduce a new incompatible library.

Answer 1

Of course you could say that by extracting an Interface IDirectory there also exists a contract which guarantees that the IDirectory.GetFiles(..) method is supposed to return an empty array. ... I could, of course, read the API specification and make sure that it returns an empty array, but don't think this is the point of unit testing.

Yes, I would say that. You didn't change the signature, but you're changing the contract. An interface is a contract, not just a signature. And that's why that is the point of unit testing - if the contract does its part, this unit does its part.

If you wanted extra peace of mind, you could write unit tests against IDirectory, and in your example, those would break (expecting empty string array). This would alert you to contract changes if you change implementations (or a new version of current implementation breaks it).

Answer 2

In unit testing, identifying SUT (system under test) is very important. As soon as it is identified, its dependencies should be replaced them with stubs. Why? Because we want to pretend that we are living in a perfect world of bug-free collaborators, and under this condition we want to check how SUT only behaves.

Your SUT is surely FindAllImages . Stick to this to avoid being lost. All stubs are actually replacements of dependencies (collaborators) that should work perfectly without any bit of failure (this is the reason of their existence). Stubs cannot fail a test. Stubs are imaginary perfect objects. Pay attention: this configurations has an important meaning. It has a philosophy behind:

*If a test passes, given that all of its dependencies work fine (either by using stubs or actual objects), the SUT is guaranteed to behave as expected. In other words, if dependencies work, then SUT works hence green test retain its meaning in any environment : If A --> Then B

But it doesn't say if dependencies fail then SUT test should pass or not or something. IMHO, any further logical interpretation is misleading. If Not A --> Then ??? (we can't say anything)

In summary: Failing actual dependency and how SUT should react to is another test scenario. You may design a stub which throws exception and check for SUT's expected behavior.