Here is the updated presentation I did a few years ago. I updated the content with a few more relevant items and suggested material. Here are the PDF and PPTX format publications. Please present and distribute as needed.

A Practical Guide to Unit Testing12.pdf (2.86 MB)
A Practical Guide to Unit Testing.pptx (440.26 KB)

Do we really trust our tests? If we have a test suite, and most run fine but sometimes, some tests fail for predictable or unpredictable reasons... should we still keep them in the suite? Here is a scenario. We have a test suite of almost 5000 tests, some of which are integration tests, some functional tests, and a lot of unit tests. They are organized into a set of lets say 25 assemblies total. Many teams (lets say 10) are working on the code base. Sometimes some of the tests that were written depending on data in the database fail, because the database is shared between all the 10 teams AND the CI build server. Sometimes a test or a few will fail, breaking the build. Most of the time, just re-initiating the CI build process will result in success the second time. So do we trust these tests? Do we believe in their results?

When using TDD, we first test the test itself, making sure it fails when we expect it to, and passes when we expect it to. Tests must set up and tear down anything and everything (including data in a database for integration tests) in order to be reliable. If we have legacy tests that weren't written this way, are they valid? They certainly don't meet the standard of TDD in my view.

Lets look at the reason we write tests at all...

• To mitigate risk

that's it... in a nutshell.

If a test runs successfully sometimes but not others, does it really mitigate risk? Does it mitigate anything at all if we don't trust its result? I think NO. Are we better off leaving these tests as part of the suite? or are we better off just deleting them from the codebase - or... fixing them so that they are reliable. If they mitigate risk when they work, and they test a portion of the code that is valuable to ensure is working correctly, then they have to be fixed. Otherwise, they should just be removed from the test suite.

We depend on our test suites to tell us the state of the state for our working software. If they can't do this, then I would argue they are of no value to us in the overall big picture. Ensure that all tests are valuable, valid, and trustworthy. Make sure that things like time of day, data in a database (or lack thereof), or other transient factors aren't able to influence the outcome of the test. Only when we can rely on our test coverage to tell us the Real state of the code, can we deliver reliable, working software.

I have long felt that one good reason for unit tests should be that of "Executable Documentation." Unit tests should illustrate the intent of the developers, as they wrote the code, so that we can make sure the code does what was intended. In contrast to other types of documentation, these tests are actually compilable and runable. There is very little chance that they can be out of sync with the actual product code, as long as there is a Continuous Integration [CI] system in place to execute the tests. This is in contract to documents, and other written forms. The only document that is kept in sync with source is one that is automatically extracted from the code.

All project documentation has a cost associated with it - both for creating it, as well as maintaining it. It's these ongoing costs that we on Agile project would like to keep to a minimum. We need only as much documentation as required by stakeholders, and no more. Unit code "documentation" is usually required by the "development management" stakeholder, usually implicitly. Anything else should probably be specified directly by the product owner, and should be a specific task on the sprint backlog to produce.

Documentation that is in sync with the product code, such as unit tests and automatically extracted documentation, or even release notes, has the lowest total cost of ownership [TCO] for the project long-term. See what you can do on your current or next project to lower your TCO for documentation with automation and Executable Documentation.

These are some qualities of a good unit test: fast, small, focused (atomic), and isolated (order-independent).

A good unit test will execute in MILLISECONDS. NOT seconds. If a test takes more than a few hundred milliseconds to execute, it is probably too slow. This is not just an arbitrary performance metric...

Let me explain the philosophy behind this tenet.

A unit test suite should contain dozens if not hundreds of tests, organized into classes that have specific testing focuses. If a test runs in 1 SECOND (1000 ms) and there are 100 of these, that is almost a 2-minute wait for a developer to wait for the results of the suite. And we do this over and over and over... If we have to wait more than a few seconds (7 seconds is about the maximum time to keep the attention of the average developer) then the mind wanders, we get distracted, and it interrupts our workflow.

Not to mention the lost time... unit tests as a suite are (or SHOULD be) run frequently. Just imagine multiplying 2 minutes x 15 times per day (conservatively) x number of developers x developer rate per hour... This turns out to be quite an expense. Not to mention that the expense grows over time as new tests are added.

Tips:
- any unit tests that need to hit a database are not unit tests...
- any tests that need a network connection are not unit tests...
- some even say any tests that need a disk FILE are not unit tests...

Any of these issues can be resolved by the use of dependency injection and substituting mock objects in place of the database, service, or network resource. Using these objects will decouple the code from the service, and allow it to execute in isolation without requiring the actual resource. And, it will be FAST.

It's really worth doing, when we look at all the benefits. Take your unit tests to the next level and remove the integration with external resources. Make the tests fast, and everyone benefits.

Test-driven Development/Design when practiced properly, yields organic design growth as tests cause feature code to be written in response to failing tests. In *proper* practice, the entire code base is refactored as needed to ensure that all code and design smells are eliminated (while keeping tests passing). This essentially causes the design at any given time to integrate all the features written up to that point. Once this process is completed, the feature code should be relatively complete, if the tests were written well to dictate the behavior required by the customer. If Acceptance-Test Driven Development is used, the Acceptance Tests for the feature passing will indicate that it’s ready to ship. At any given point, the code base that is checked in should withstand the scrutiny of an outside code reviewer, and should embody all the best design principles. If it doesn’t, the “refactor” stage has not yet been completed.

Refactoring to Remove Code and Design Smells

I find many teams that either poorly implement or flat-out skip the step of refactoring the code base and overall code design as part of this cycle. This kind of organic development - sticking a new feature on a code base without integrating it into the design and making changes as required is like sticking a wad of gum on a bowling ball. It tends to be the entire code base lumpy, obtuse, and generally not pretty to look at. Teams can get away with this kind of development for a while more or less successfully (unfortunately). However this behavior causes LARGE problems for the team or those inheriting maintenance and new feature work on this code base. Often the team will have to stop to rewrite the whole code base at some point, because it’s too problematic to maintain the bowling ball and gum code. In practical business situations, it’s far far easier to maintain the overall design and integrate features into the design, course-correcting the design as needed to absorb all of the features into a coherent architecture.

This concept of going through the code base and refactoring all of it to embody all the features in it, and removing design and code smells does imply that if the design changes, so must the tests that correspond to the design. If the tests are tightly coupled to the code base, some design changes can be painful in terms of test maintenance overhead. There are methodologies that assist in decoupling the test from the code, and have the test just confirm the behavior without much knowledge of how the implementation is done. The use of polymorphism, mock objects to decouple behaviors from implementation, and the principle of dependency injection in general can assist with decoupling, and make test maintenance less painful (although perhaps somewhat tougher to write at the outset). There is overhead still however with design changes, and that should be expected, embraced, and *estimated* in planning…

Plan for Change

In iteration planning, changes to the design and implementation, along with test maintenance should be factored in to the estimates for work being done. When breaking out tasks for a story, I like to at least mention if not record a specific task for the design refactoring, and another for the test maintenance for design changes. I personally think these should be more or less estimable by most teams with at least rudimentary skills. If the team is new, add a task with a ballpark guess for the time, but be sure to add the tasks so they have visibility… Scoping the effort for the design changes and test maintenance will be easier and easier going forward as the team gains understanding of what impact new features have. And it is likely that for a team with a good solid design, these changes will be minimal if they are done incrementally.

Some teams prefer to finish the tasks for the features and their tests, and then have a single story for the design and test maintenance per iteration, and keep copying it from one iteration to the next. This works too, however I have seen issues where the iteration tasks don’t get completed, and the maintenance story gets bumped off or not completed in the sprint. This engineering debt is carried forward, and the code that is checked in for “completed” features for the sprint now carries debt with it as well.

In my experience, I have seen more success with making the elimination of design and code smells part of the “Done” criteria for each story, and in fact pre-check-in criteria. This way we can’t call a story “Done” [meaning completed, and shippable] until it *really* is. Design and test maintenance changes on a per-story level should be small, and usually bite-size enough for a team to be able to handle operating this way even in a short iteration period.

Don’t omit the design and test maintenance, plan for it specifically each iteration. This practice will help keep engineering debt from accruing from iteration to iteration. It is a key practice that will help keep a team, a project, and a code base (and a business) all on a successful course for the short, and the longer term.

When we are unit testing a piece of code, sometimes there are modules or even systems that we need to rely on outside the scope of the test. It is usually very difficult or expensive to spin up these large pieces just to test one small piece. Even attempting to do this can really slow down the execution of the tests, and that becomes a problem of its own. When possible, we'd like to separate the implementation from the caller's execution so that the control is really in the called module doing the work. This is the "Inversion of Control" or IoC principle, as it's often referred.

Dependency Injection is one mechanism where we can design our code in this way. Dependency injection [DI] is like "injecting" a small bit of test-controlled code into the main-line execution space while the test is running. The test can control the specific execution of the "dependency" code, and cause it to behave in a specific way so as to elicit a specific behavior from the mainline code under test.

My favorite mechanism to accomplish DI is to refactor code under test, extracting the logic of the dependency into a class that implements an interface. Then, we can create a test mechanism that implements the interface and behaves in the way that the test can control. We then replace the mainline "real" code with the test version at test time, bypassing the dependency and allowing us to remain focused on the code under test.
We can implement this with simple classes, or with Mock objects. The mock object frameworks out there today are quite robust and relatively easy to use, given an interface to implement for the dependency class. Rhino Mocks and Moq are some examples of frameworks that can be used for this type of test. Here is a simple example of some code under test:

using System;

namespace DIexample
{
    public class ClassUnderTest
    {
        internal IDependOnMe limit = new Limiter();

        public int HasDependencyOnExternalClass(int value)
        {
            if (value < 0)
            {
                return limit.LogicFunction(value);
            }
            else
            {
                return value;
            }
        }
    }

    interface IDependOnMe
    {
        int LogicFunction(int value);
    }

    public class Limiter : IDependOnMe
    {
        public int LogicFunction(int value)
        {
            if (value < -10)
            {
                return value;
            }
            else
            {
                return -10;
            }
        }
    }
}

At test time, we can replace the internal variable "limit" with a test-created instance of an object that implements the "DependOnMe" interface, and behaves in a specific way (like just returning the value perhaps). The variable "limit" is declared as "internal" scope, because we want the test code to be able to access it and replace its "real" object instance with a test-specific version. I keep the test assembly separate from the mainline code. However, I don't like the idea of making these mechanisms public unless they need to be, so instead in the mainline code, I use the internals visible to assembly attribute like this:

using System.Runtime.CompilerServices;

[assembly: InternalsVisibleTo("TestAssembly")]

This allows all "internal" scoped methods to be accessed by the test assembly, and keeps all the test code out of the mainline code, and maintains at least a level of security that being public doesn't offer.

Mock frameworks can be a big help in this type of DI testing. Rhino Mocks is here: http://ayende.com/projects/rhino-mocks.aspx it's a good free framework and fully featured. I've not used it but heard good things about MOQ: http://code.google.com/p/moq/

Remember to keep the execution isolated from the implementation whenever possible, and it will make unit testing much easier.

OK people. STOP putting cowboy code on my desk. Seriously. TEST your software. REALLY. Please STOP writing BRAND NEW LEGACY code.
< /Rant >

OK I feel better. But not really. Just because a system is not considered to be a "production release" (meaning that people outside the particular organization or business won't have access to it) it DOES NOT mean that we can cowboy it up and plop a steaming pile of untested code out there for our colleagues or coworkers to consume.

ALL CODE NEEDS TESTS. Period.

Untested code is NOT finished. Please don't just assume that if the code "works" that it's ready to inflict on other people (or even yourself). Self-inflicted legacy code casualties are the second highest cause of career paralysis, according to 4 out of 5 dentists. By definition, code is "legacy" if it doesn't have tests.

Here's a plan to get your tests caught up.

Start Small
At least we need a set of "sanity" checks... if nothing else. Create a "test suite." Use xUnit or the like, keep it simple. Start with the checks for the obvious output... if that doesn't show up, then something is clearly broken.

Add Some More
One at a time if you have to, add another test for something. Any kind of test will do... unit, functional, acceptance... I recommend starting with acceptance and working backward. We already blew the op to be able to do TDD and get unit testing done, so now let's just make sure the surface behavior is adequate. Definitely do go in deeper if time permits.

New Feature
Each time a new feature is added, update the test suite to make sure everything is happy with changes that ripple out from the effect of the feature. Again - at least make sure that the obvious functionality is tested on the new code. Don't slack on it, keep at least the new code from being legacy.

Lather, Rinse, Repeat
Pick a time each week (say 4:30PM on Wednesdays) to put in a half-hour on writing one new test. Schedule it. Block the time out on your calendar so someone can't schedule a meeting... And hide if you have to - to avoid distractions.

What to Write
Focus on happy-path code tests only if there are none for that feature. Otherwise, do alternate and failure path tests, so that the behavior of the system is understood in circumstances other than the best-case. 90% of bugs occur in the not-happy path.

Using this strategy will get testing started, and contribute to a more robust and reliable product, no matter what it is. Remember that most "side projects" or "utilities" that are developed in-house will be used by your own people. We want them to have the best.

"But it's just a spreadsheet." OK, yes it is. I say we need to make sure to test even the lowly spreadsheet. Because, next week someone in Finance is going to discover it and begin to use it for their payroll planning... if there is a critical flaw in it, it might be YOUR paycheck that gets eliminated!

We should realize that even internal tools and web sites are usually used to make business decisions of some kind. We want to make sure that they are made using reliable tools, to the extent possible. Business relies on its data, now more than ever. Competition is fierce in a down economy - even small mistakes can be a big problem.

"Bread's not done until it's baked" - and neither is code! SO PLEASE: Test it. Bake it. Slice it... and don't get burned!

Here is a presentation I wrote on what Unit Testing is all about, and how TDD fits into the ATDD cycle.

There are specific things here on testing the UI code with Selenium and JSUnit, and recommendations on how to do unit testing on your database code.

This presentation is in PDF format, but I can post the PPTX format also if needed.

A Practical Guide to Unit Testing1.pdf (503.29 KB)

Do you ever wonder why there is an extra "False" printed at the end of each run for unit tests?

C:\Code\sample>ipy UnitTests.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.395s

OK
False

It looks like there is an exit statement buried inside the unit test framework. We can eliminate the problem pretty easily with the addition of a try/except around the unit test execution. Here is the code to put at the end of your unit test script:

if __name__ == '__main__':
    try:
        unittest.main()
    except SystemExit:
        pass

This catches and eats the SystemExit exception, and runs without printing the "False" at the end of the execution:

C:\Code\sample>ipy UnitTests.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.387s

OK

cheers!

  information_schema.columns cols

  INNER JOIN sys.columns sc ON

    cols.column_name = sc.name

    AND OBJECT_NAME(sc.object_id) = @table_name