The Art of Unit Testing: with examples in C#
All
Stack Overflow 10
This Year
Stack Overflow 5
This Month
Stack Overflow 1
Unit testing is easy for beginners. Unit testing done well is much less so. This results in many inexperienced people writing poor unit tests.
In my last job, I worked on a legacy C++ code base. 500K lines of thorough testing, but none was a unit test. It took 10 hours to run the full test suite. I set about the task of converting portions for unit testability.
I was surprised how much of a challenge it was (and I learned a lot). There were few resources on proper unit testing in C++. Mostly learned from Stack Overflow. Lessons learned:
1. If your code did not have unit tests, then likely your code will need a lot of rearchitecting just to enable a single unit test. The good news is that the rearchitecting made the code better.
As a corollary: You'll never know how much coupling there is in your code until you write unit tests for it. Our code looked fairly good, but when I wanted to test one part, I found too many dependencies I needed to bring in just to test it. In the end, to test one part, I was involving code from all over the code base. Not good. Most of my effort was writing interface classes to separate the parts so that I could unit test them.
2. For C++, this means your code will look very enterprisey. For once, this was a good thing.
3. Mocking is an art. There's no ideal rule/guideline for it. Overmock and you are just encoding bugs into your tests. Undermock and you are testing too many things at once.
4. For the love of God, don't do a 1:1 mapping between functions/methods and tests. It's OK if your test involves more than one method. Even Robert Martin (Uncle Bob) says so. I know I go against much dogma, but make your unit tests test features, not functions.
5. If your unit tests keep breaking due to trivial refactors, then architect your unit tests to be less sensitive to refactors.
6. For classes, don't test private methods directly. Test them through your public interface. If you cannot reach some code in a private method via the public interface, throw the code away!
7. Perhaps the most important: Assume a hostile management (which is what I had). For every code change you make so that you can write a unit test, can you justify that code change assuming your project never will have unit tests? There are multiple ways to write unit tests - many of them are bad. This guideline will keep you from taking convenient shortcuts.
This advice is all about unit tests, and not TDD. With TDD, it is not hard to test yourself into a corner where you then throw everything away and restart. If you insist on TDD, then at least follow Uncle Bob's heuristic. For your function/feature, think of the most complicated result/boundary input, and make that your first unit test. This way you're less likely to develop the function into the wrong corner.
When I completed my proof of work, the team rejected it. The feedback was it required too much skill for some of the people in the team (40+ developers across 4 sites), and the likelihood that all of them will get it was miniscule. And too much of the code would need to change to add unit tests.
I later read this book:
https://www.amazon.com/Art-Unit-Testing-examples/dp/16172908...
And it contains quite a bit of what I learned from unit testing.
Let's see at the following statement in
Module.ApplicationEndRequest()
method:When this code is executed from Unit Test,
context.Response
is a mock that you set up in MoqHttpContext.CreateBaseMocks():You can't expect that you call a
Write()
method on a mock and then can read the same data back. Mock is a fake object. Its default implementation ofWrite()
method does nothing, and passed data is just lost.To fix the problem, you could setup a callback on
Response
mock that will write passed data to a stream and then return it back on read. You are actually very close to it.In
MoqHttpContext
class declare a stream where you will save the data:Then in
CreateBaseMocks()
method setup a callback:You also should remove a line that sets
inputStream
position to0
inMoqHttpContextExtensions.StreamWrite()
, so that html data that you write inUnitTest1.TestMethod1()
is appended, not overwritten:That's it. Now if you check value of
responseRead
in the test, you will see that data appended by Http module is there.UPDATE (Fixing problem with a filter)
There are 3 different issues with current code that prevent correct work of a filter from UT.
You tried handful options for mocking
Filter
property, however none of them seems correct. The correct way to mock property getter with Moq is:Remove all other statements for mocking
response.Filter
, but don't add above statement yet, it's not a final version.You have following check in
Module.ApplicationEndRequest()
method:When UT is executed,
context.Response.Filter
is aMemoryStream
not aResponseSniffer
. Setter that is called inModule
constructor:will not actually affect value returned by
Filter
getter because it's a mock that currently always return instance ofMemoryStream
that you setup withSetupGet
. To fix this problem you should actually emulate property behavior: save the value passed to setter and return it in the getter. Here is a final setup ofresponse.Filter
property:Make sure you have deleted all other mocks of
response.Filter
property.The final problem that you should fix - is the order of
Module
invocations from UT. Currently the order is the following:But
PreRequestHandlerExecute
setsResponse.Filter
with an instance ofResponseSniffer
. So whenhttpContext.StreamWrite
above it is called,httpContext.Response.Filter
holds actually instance ofMemoryStream
, notResponseSniffer
. So the last fix you should make is to change the order of statements in UT body:UPDATE (UT Redesign)
At this point your UT should work. However current test is very cumbersome. The fact that it takes so much time to understand why it does not work proves it. Such tests are very hard to maintain and fix, they become a real pain over time.
Moreover it's rather Integration test than Unit test, because it invokes several of classes with different functionality -
ResponseSniffer
andModule
.You should strongly consider redesign of current test. And the good start is to make separate tests for
ResponseSniffer
andModule
classes.Most valuable test for
ResponseSniffer
is the one that verifies that written data is registered inRecordStream
:As regards
Module
class, there are several checks that should be made:PreRequestHandlerExecute()
setsResponse.Filter
with instance ofResponseSniffer
.ApplicationBeginRequest()
addsStopwatch
tocontext.Items
dictionary.ApplicationEndRequest()
writes request info to the response.UT approach implies checking of these facts in separate tests. Here are samples of such 3 tests:
As you see, the tests are pretty simple and straightforward. You don't need those
MoqHttpContext
andMoqHttpContextExtensions
with a lot of mocks and helpers anymore. Another benefit - if some of the tests get broken, it's much easier to identify the root cause and fix it.If you are new to Unit Testing and are looking for good source of info on it, I strongly suggest book The Art of Unit Testing by Roy Osherove.
To illustrate the usage of stubs and mocks, I would like to also include an example based on Roy Osherove's "The Art of Unit Testing".
Imagine, we have a LogAnalyzer application which has the sole functionality of printing logs. It not only needs to talk to a web service, but if the web service throws an error, LogAnalyzer has to log the error to a different external dependency, sending it by email to the web service administrator.
Here’s the logic we’d like to test inside LogAnalyzer:
How do you test that LogAnalyzer calls the email service correctly when the web service throws an exception? Here are the questions we’re faced with:
How can we replace the web service?
How can we simulate an exception from the web service so that we can test the call to the email service?
How will we know that the email service was called correctly or at all?
We can deal with the first two questions by using a stub for the web service. To solve the third problem, we can use a mock object for the email service.
A fake is a generic term that can be used to describe either a stub or a mock.In our test, we’ll have two fakes. One will be the email service mock, which we’ll use to verify that the correct parameters were sent to the email service. The other will be a stub that we’ll use to simulate an exception thrown from the web service. It’s a stub because we won’t be using the web service fake to verify the test result, only to make sure the test runs correctly. The email service is a mock because we’ll assert against it that it was called correctly.
The standard approach to testing code that runs SQL queries is to unit-test it. (There are higher-level kinds of testing than unit testing, but it sounds like your problem is with a small, specific part of your application so don't worry about higher-level testing yet.) Don't try to test the queries directly, but test the result of the queries. That is, write unit tests for each of the C# methods that runs a query. Each unit test should insert known data into the database, call the method, and assert that it returns the expected result.
The two most common approaches to unit testing in C# are to use the Visual Studio unit test tools or NUnit. How to write unit tests is a big topic. Roy Osherove's "Art of Unit Testing" should be a good place to get started.
About testability
Due to the use of singletons and static classes MyViewModel isn't testable. Unit testing is about isolation. If you want to unit test some class (for example, MyViewModel) you need to be able to substitute its dependencies by test double (usually stub or mock). This ability comes only when you provide seams in your code. One of the best techniques used to provide seams is Dependency Injection. The best resource for learning DI is this book from Mark Seemann (Dependency Injection in .NET).
You can't easily substitute calls of static members. So if you use many static members then your design isn't perfect.
Of course, you can use unconstrained isolation framework such as Typemock Isolator, JustMock or Microsoft Fakes to fake static method calls but it costs money and it doesn't push you to better design. These frameworks are great for creating test harness for legacy code.
About design
About testing framework
You can use any unit testing framework you like. Even MSTest, but personally I don't recommend it. NUnit and xUnit.net are MUCH better.
Further reading
Sample (using MvvmLight, NUnit and NSubstitute)
What you gained here is that when the dependency
Foo
itself gets any dependencies of its own, or requires a different lifestyle, you can make this change without having to do sweeping changes throughout all consumers ofFoo
.You can't just ignore unit testing in this. As Roy Osherove explained a long time ago, your test suite is another (equally important) consumer of your application with its own requirements. If adding the abstraction simplifies testing, you shouldn't need another reason for creating it.
You won't violate YAGNI if you create this abstraction for testing. In that case YNI (You need it). By not creating the abstraction you you are optimizing locally within your production code. This is a local optimum instead of a global optimum, since this optimization doesn't take all the other (equally important) code that needs to be maintained (i.e. your test code) into consideration.
There isn't anything wrong per see to inject a concrete instance, although -as said- creating an abstraction could simplify testing. If it doesn't simplify testing and letting the consumer take a hard dependency on the implementation could be fine. But do note that depending on a concrete type can have its downsides. For instance, it becomes harder to replace it with a different instance (such as an interceptor or decorator) without having to make changes to the consumer(s). If this is not a problem, you might as well use the concrete type.
First of all, it is important to understand that
Verify
-family methods are there for a reason -- they allow you to test unobservable1 behavior of your system. What do I mean by that? Consider simple example of application generating and sending reports. Your final component will most likely look like this:How do you test this method? It doesn't return anything, thus you cannot check values. It doesn't change state of the system (like, flipping some flag), thus you cannot check that either. The only visible result of
SendReport
being called is the fact that report was sent viaSendEmailToSubscribers
invocation. This is the main responsibility ofSendReport
method -- and this is what unit tests should verify.Of course, your unit tests should not and will not check whether some email was sent or delivered. You will verify mock of
reportSender
. And this is where you useVerify
methods. To check that some call to some mock actually took place.As a final note, Roy Osherove in his book Art Of Unit Testing (2nd edition) separates unit tests into three categories, depending on what can be checked:
Last category is where you use mocks and
Verify
methods on them. For other two, stubs are enough (Setup
methods).When your code is designed correctly, such test will (last category) be in minority in your code base, somewhere in 5% - 10% range (number taken from Roy's book, in line with my observations).
1: Unobservable as in that caller cannot easily verify what exactly happend after the call.
You're approaching this the wrong way. Just test your functionality: if an exception is thrown the test will automatically fail. If no exception is thrown, your tests will all turn up green.
I have noticed this question garners interest from time to time so I'll expand a little.
Background to unit testing
When you're unit testing it's important to define to yourself what you consider a unit of work. Basically: an extraction of your codebase that may or may not include multiple methods or classes that represents a single piece of functionality.
Or, as defined in The art of Unit Testing, 2nd Edition by Roy Osherove, page 11:
What is important to realize is that one unit of work usually isn't just one method but at the very basic level it is one method and after that it is encapsulated by other unit of works.
Ideally you should have a test method for each separate unit of work so you can always immediately view where things are going wrong. In this example there is a basic method called
getUserById()
which will return a user and there is a total of 3 unit of works.The first unit of work should test whether or not a valid user is being returned in the case of valid and invalid input.
Any exceptions that are being thrown by the datasource have to be handled here: if no user is present there should be a test that demonstrates that an exception is thrown when the user can't be found. A sample of this could be the
IllegalArgumentException
which is caught with the@Test(expected = IllegalArgumentException.class)
annotation.Once you have handled all your usecases for this basic unit of work, you move up a level. Here you do exactly the same, but you only handle the exceptions that come from the level right below the current one. This keeps your testing code well structured and allows you to quickly run through the architecture to find where things go wrong, instead of having to hop all over the place.
Handling a tests' valid and faulty input
At this point it should be clear how we're going to handle these exceptions. There are 2 types of input: valid input and faulty input (the input is valid in the strict sense, but it's not correct).
When you work with valid input you're setting the implicit expectancy that whatever test you write, will work.
Such a method call can look like this:
existingUserById_ShouldReturn_UserObject
. If this method fails (e.g.: an exception is thrown) then you know something went wrong and you can start digging.By adding another test (
nonExistingUserById_ShouldThrow_IllegalArgumentException
) that uses the faulty input and expects an exception you can see whether your method does what it is supposed to do with wrong input.TL;DR
You were trying to do two things in your test: check for valid and faulty input. By splitting this into two method that each do one thing, you will have much clearer tests and a much better overview of where things go wrong.
By keeping the layered unit of works in mind you can also reduce the amount of tests you need for a layer that is higher in the hierarchy because you don't have to account for every thing that might have gone wrong in the lower layers: the layers below the current one are a virtual guarantee that your dependencies work and if something goes wrong, it's in your current layer (assuming the lower layers don't throw any errors themselves).