As Esther Derby's article discusses, it's the area of integration testing or cross-context testing where we most often discover the discrepancies in interfacing systems to each other. Automated acceptance testing, or automated integration tests are a key point for winning the bug battle in larger systems. If the teams and/or organizations negotiate a contract on test automation across these integration points, they can collaborate to create test frameworks and test automation that can be used to ensure proper functionality. These acceptance tests (or end-to-end tests as some call them) can and should be demonstrated to key stakeholders to make sure that the business understands what development is delivering, and that it meets the intended need.

As I often say, if we have automated acceptance tests, we can do ATDD: First, we automate the acceptance test, which fails because there is no code yet to test. Then, we write unit tests and code, more unit tests, and more code - striving ONLY to make the acceptance test pass. We should focus on just the one failing acceptance test, and not stray off into other functionality. Once we do, we can refactor to make sure we have the best solution, then we start all over again with writing more automated acceptance tests. That's ATDD!

Test automation is such a great thing to have on a project. It makes me smile each time an automated test catches a bug that would have otherwise gone unnoticed and perhaps even shipped to production.

This will be my last article for 2011, hope you had a great year! Best wishes for a fantastic 2012!

Lately I have been doing work on WinForms applications and have been looking at ways to test them. We use unit tests to test at the method level, and [ideally] our code is written to be able to separate the logic code from the UI code. However, not *all* applications are written this way (unfortunately). Today I am in a situation where I must be able to test an application at the UI level. I need to bring up forms, type in text boxes, poke buttons, and inspect results in labels, etc.

The answer is NUnitForms! This fabulous test framework is an open source project that extends the functionality of NUnit, and allows us to test applications at the UI level. We don't consider this to be a unit test or even integration test mechanism, but rather a functional test mechanism. Rather than having testers manually go through screen after screen in the application just to see if it all works, we'll write some NUnit code that will automate it all for us.

So How do I do it?

SourceForge bits: http://sourceforge.net/projects/nunitforms/files/nunitforms/

First, download the NUnitForms source code and add it to your solution. Then, write an NUnit test class, add a reference to the NUnitForms project and make your test class inherit from the NUnitFormsTest class. Add a reference to your application classes, and create a test. In the test, open your form with Form.Show() (so that it is visible). Then use a ButtonTester, a LabelTester, TextBoxTester, etc. to find each of the controls by name, and do the appropriate action to them. Then, after all of your actions are completed, call Application.DoEvents() to make sure to let everything have time to execute. You can then grab those text boxes, labels, etc. and Assert that the values are correct. That's it! Build it up step by step, and you have a fully automated end-to-end test!

If your forms are simple enough, you might even be able to use the Recorder application that comes with the codebase to record your keystroke and mouse actions as a C# code test. If your app has some startup code to accomplish, refactor it so that it's in simple methods, and include a hidden form in the app that your Recorder can open to accomplish the startup before opening other forms.

I have written and included sample code in a Visual Studio 2010 project in the zip file attached. This is a really great way to automate acceptance, functional, and end-to-end tests for a WinForms application project.

Sample code Here (5.5MB)

Here is the updated presentation I did a few years ago. I updated the content with a few more relevant items and suggested material. Here are the PDF and PPTX format publications. Please present and distribute as needed.

A Practical Guide to Unit Testing12.pdf (2.86 MB)
A Practical Guide to Unit Testing.pptx (440.26 KB)

I have long felt that one good reason for unit tests should be that of "Executable Documentation." Unit tests should illustrate the intent of the developers, as they wrote the code, so that we can make sure the code does what was intended. In contrast to other types of documentation, these tests are actually compilable and runable. There is very little chance that they can be out of sync with the actual product code, as long as there is a Continuous Integration [CI] system in place to execute the tests. This is in contract to documents, and other written forms. The only document that is kept in sync with source is one that is automatically extracted from the code.

All project documentation has a cost associated with it - both for creating it, as well as maintaining it. It's these ongoing costs that we on Agile project would like to keep to a minimum. We need only as much documentation as required by stakeholders, and no more. Unit code "documentation" is usually required by the "development management" stakeholder, usually implicitly. Anything else should probably be specified directly by the product owner, and should be a specific task on the sprint backlog to produce.

Documentation that is in sync with the product code, such as unit tests and automatically extracted documentation, or even release notes, has the lowest total cost of ownership [TCO] for the project long-term. See what you can do on your current or next project to lower your TCO for documentation with automation and Executable Documentation.

Value Stream Mapping [VSM] is a lean concept that helps us improve our processes by focusing on the whole process and making stepwise refinements on smaller parts of the process, to gain improvement for the whole. It is in a sense, "Refactoring" a process.

In an ideal case, a process can be thought of as a workflow, and perhaps we might be able to represent it using the WF Windows Workflow Foundation code delivered in .NET 3.5. It would be nice to show a code example here of how we can represent a process with workflow steps, but I didn't have time to write up an example as of yet.

With VSM, we are trying to improve a process by mapping it into parts, each of which we can study. We can gather metrics on each of the process steps, and metrics on how long things wait between steps. The waiting between steps is usually far more significant time-wise than any of the steps themselves. WF can be instrumental in providing an automated framework for collecting and reporting these metrics.

Indeed there are also other workflow alternatives of course. BizTalk Server 2006 R2 is a fine candidate for workflow representation as well, when adapters are available for systems that interoperate. I have a specific Microsoft focus in my career, but I have used CapeClear as a Java-based workflow system, and there are others in the ESB space which have features that would allow us to collect metrics on an automated representation of our processes.

Automation is key to refactoring, as in any development mechanism. We can't fix what we don't know is broken. With VSM, we focus on the entire process as a whole, so we need to collect measurements on where it is today, and then we can see total improvement on the whole as we make small changes (refactoring the process).

Many enterprises today use Service-Oriented Architecture [SOA] as a design pattern for the construction of enterprise systems. This architecture lends itself well to the use of WF or BizTalk (substiitute your favorite flavor here) workflow components. SOA helps us to map the entire process to blocks of functionality that we can measure and monitor easily, contributing to our VSM overall picture.

My theory is that like with software development, process improvement can also have a test-driven, scientific approach. VSM is a tool that helps us change the right things to improve our overall process, but it typically has measurements in large units - days, even weeks. With the right automation framework contributing to very rapid feedback with the metrics collected on processes, we can write tests that indicate Red or Green for not only each component being analyzed, but for the entire process as well.

Tests don't have to be in the form of an NUnit DLL or test fixture. We can use simple reports actually to give us the feedback we need to make decisions on how we refactor. Think of it as moving test-driven concepts a couple elevations higher in the plan view. From the 500 foot view to the 5000 foot view if you will. It's the exact same concept, just applied at a somewhat higher level of abstraction.

For each of our services we monitor the time taken in the process and the time between processes, and compare them to our initial baseline. We might even be able to get real-time data on a visible dashboard in the best of cirumstances. When we are better than our initial baseline, we are Green, if not, then we're Red. There is no yellow - same result means we haven't improved so it's still red. This concept is brutally simple, but it can be very effective, especially if the Red/Green report is made visible on a web-based report.

Mapping the workflow using WF, BizTalk, or other tools gives us a way to take automated measurements and compare them to our baseline to render a decision on whether our process has improved or not, and whether we are reaching our overall goal of total improvement. Individual portions of the whole can still be red as long as we achieve green for the overall process.

A business process really is a workflow, we just need to map it as such using software and automation. In this way we can apply the fine concepts of test-driven and refactoring to process improvement, once again using automation and measurement as our top key tools for leveraging our own technologies to help us improve them.

Code examples here would be good of course... Stay tuned for future posts with WF code examples of a process flow that has metrics and baseline comparisons for Red/Green.

I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|


I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

I have been asked recently about tools I prefer to use in my every-day development. Here is a list of tools, and where to get them.

Visual Studio 2008      Development IDE                   Microsoft
TestDriven.net           Test Runner                         TestDriven.net
WinMerge                  Diff / Merge tool                   WinMerge.org integrates with VS and Tortoise too!
nUnit                        Unit Testing Framework          nUnit.org     Also see mbUnit and Gallio
Selenium                   UI test framework                 OpenQA       Also see WatiN
ReSharper                  Integrated toolkit for VS        JetBrains     (OK I don't actually use it but it's good.)
Tortoise SVN              Shell Integration with SVN      Tigris
Ankh SVN                  Visual Studio Plugin for SVN     CollabNet
Subversion (Server)     Version Control System          VisualSVN
Cruise Control.net       CI system                            ThoughtWorks
RhinoMocks                Mock Object System              Ayende
nAnt                         .NET Build Tool                     nAnt
Fitnesse                    Acceptance Test Tool            Fitnesse
STIQ                         Story Test Tool                    Solutions IQ
GIMP                         GNU Image Manipulation Prog. SourceForge
Notepad++                 Smart Text Editor                 SourceForge UK when Visual Studio just won't do...

Not development tools exactly, but extremely handy:
Process Explorer          Smarter Task Manager           SysInternals
FileZilla                      Upload/FTP client                  FileZilla
DivX                          Decoder                              DivX         Because sometimes we need to watch movies...

That's all I can think of at the moment, but am probably missing some things. I'm sure you'll all (please) chime in with what I forgot...

We the practitioners of Agile and users of Beer, hereby proclaim Agile Beer Night.

The Agile Beer Users Group (Seattle Chapter) will be meeting on Thursday Aug 21, for information sharing, networking, agile discussions, a brief topic presentation on Automated Acceptance Testing, oh - and BEER. Come meet us for a discussion and brief presentation on Agile practices each month.

There is good food, good beer, and a good chance to meet some of the folks who do agile development in the Pacific Northwest, get to know their tricks, and share in their success. The Owl has a fine selection of food, drinks, and spirits for those who choose them.

ABN next meeting will be held in Seattle at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle from 5PM to 8PM on Thursday August 21 2008.

The September ABN meeting will be on the East side in Redmond or Bellevue, but plans are not firm yet - stay tuned...


Please link this post, track it back, and let everyone know!

Next ABN on Thursday August 21 2008 5PM to 8PM at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle

I learned a valuable lesson today using Selenium. It seems that the DragAndDropToObject() function doesn't scale well when the HTML is being updated in a loop by the mainline code... This method uses two locator strings, one to find the object to drag, and the other to find the object to drop.

It apparently has to examine the entire XML DOM for each locator, each time through. So, when I created a loop that did a drag and drop 250 times on a page, the first few were not too bad only a couple hundred milliseconds. However, as more and more DHTML was added to the page, it took longer and longer to drop the next objects. I measured this time, and when I charted the data, it was a moderately increasing exponential slope. Definitely not even linear...

It seems that the more HTML was added to the DOM, the longer it took both of the locators to locate their objects, even though those objects were static on the page. It walked the DOM each time, for each locator. Thus the increasing non-linear time slope, because the DOM grows with every loop iteration.

By the time it got to the 50th iteration through the loop, the time taken was almost 2 orders of magnitude higher than the initial one. It made testing this particular feature not doable through this framework. And in measuring how long it took the actual javascript to render the new DHTML, it was just around 100ms, when the Selenium locates for the command took upwards of 7000ms for the 50th loop, but was only 200ms at the initial loop.

The Solution
I didn't find a complete solution to the excruciating amount of time the locator code needs to walk the DOM, but I did end up switching from the DragAndDropToObject() method to the DragAndDrop() method. The latter uses one locator string and a relative X,Y coordinate string (NOT a locator). It's just about zero time for it to process the relative coordinate string rather than the locator. So, in my specific case with this test, I was able to cut the time down to 2.5 seconds for the 250th drag and drop. Still a very LONG running test, but it worked (oh yeah and it found a nice juicy bug too...).

OpenQA people - it might behoove you to take a look at optimizing how you interact with a DOM that is growing or being updated by the code under test... A bit of performance gain in this area would really come in handy.

Both Selenium and WatiN have been around for a while, and both are fairly good tools when it comes to UI automated testing. Both tools now drive both IE7 and FF2, and are free to download. I have been using Selenium RC for a while now, and it seems to be fairly robust, although sometimes rather frustrating and somewhat slow. Both have a recorder mechanism, and both emit C# code for me. However the WatiN recorder did blow sky high with an unhandled exception when a pop-up window came up, but the recorder code was written by someone else so I won't hold it against WatiN.

I tried WatiN for the first time today, and decided to go with the V.2 CTP instead of the .NET 1.1 release. It seems that some of the API's have been refactored into more sensible structures (factory patterns, etc.), but somehow the things that would have made the most sense to me in the API were structured much differently that I would have expected. There were some problems with my IE (it kept crashing unless I ran the debug version of W2). However, I blame this on IE, not on W2. FF has a plug-in that needs to be manually installed. It comes with the CTP, but doesn't get installed. Hopefully this will be integrated by release.

Most of the basic features are similar in function but not implementation. I can see writing an interface class and a couple of wrapper classes that implement the interface, one each for WatiN and Selenium, so we can have the best of both worlds.

WatiN Pro's

• No proxy server required. It just runs. Runs both FF and IE, switchable with the parameter to the factory method that creates the instance.
  

WatiN Con's

• Documentation is less than sparse.
• I could not find a way to do a "drag and drop" which is easily accomplished with Selenium.
• Does not handle pop-up windows. This was the show-stopper for me.

Selenium Pro's

• polished code
• well-documented functionality
• lots of examples
• handles lots of weird types of conditions and situations (most of which we will all need at some point.)
  

Selenium Con's

• have to run a java proxy in a separate process and set it up and tear it down
  
• slow in places, due to proxy and not-so-well-written sections of code
• locators aren't always easy to figure out. Tools like Firebug and XPather help identify locator strings.
  

Since it does not handle pop-up windows (or I just couldn't figure it out in about 4h of research), this was the end of the line for it. Selenium does handle them, and has been able to accomplish 100% of the weird scenario tests I have needed so far. Loads of tweaking and frustrating trial and error notwithstanding.

So, until we have a bit closer to release for WatiN, Selenium still wins (for now). However, I definitely plan to write the interface and shim both out so I can use either or both later on. Stay tuned for that posting, that will be the money shot...

In closing, let me just reiterate that every developer should be thinking about UI automated testing for web code. Each user-interactive element on the page NEEDS AN ID. That's every link, every button, every text box, checkbox, radio button, and other user input elements. The ID's need to be simple, and either fixed, or predictable. With these things in place in the production code, it will be testable and easier to validate. Better quality code will make it out to the hands of the users, and all will be right with the world. Amen.

<steps down from soapbox./>

"Is it VSTS or Selenium that is going to be our standard tool for testing UI elements and automated acceptance/functional/scenario testing?"

That's like saying "do we want steak or lobster?" The answer is YES. There is no reason we can't do both. Both test tools have their merits, and their drawbacks. In my tool belt, I have both a Phillips head and a flat blade screwdriver. Like my grandfather taught me long ago, use the right tool for the right job.

Selenium actually operates a browser, so it executes JavaScript and gives the test the same experience as the user would get. We can measure response times to render a page, inspect UI elements for formatting, values, and location, and generally things take longer to execute because a browser has to render the page.

VSTS tests use only HTTP level inputs and outputs, so they couldn't care less about JavaScript or having the browser render something. They are typically much faster, because they only test the server and its communication with the client. Conversely, they are much more difficult to hand-code. If we can record a session and alter the code to provide data-driven inputs to the tests, this is an effective way to test that the server is doing the right thing for each request.

Both kinds of tests can be run under both MSTest and NUnit unit test frameworks. There is no reason we have to choose one over the other. When we are talking about UI testing, we are not talking about unit tests anymore, even though we can leverage the same mechanism for executing the tests. Most of the unit test rules do not apply, like testing times, and focused testing. We tend to have scenarios and sometimes fairly elaborate ones that run in these types of test suites.

Let me suggest a course of action for those wishing to have their cake and eat it too. In a single test solution, we can have three separate projects, the Selenium test project, the VSTS test project, and a simple DLL libraray where we can place common helper functions and test setup functions for both types of tests.

If we are careful about how we construct our scenario tests, we can build a complete beginning to end login, accomplish a set of tasks, and log out as a complete user scenario or use case. Then we can refactor the specific things out for re-use, such as log in and log out, and probably each of the specific tasks. This will help us to be able to construct scenarios much easier and quicker, using these building blocks. Then, any "common" non-UI related functionality can be refactored out in to the "helper" classes in the DLL project, making them available for all test types.

Using these techniques, I was able to construct a complete end to end scenario to accomplish a task (and verify the existence (or not) of a specific bug in about 4 hours using Selenium and its IDE. [Note: it shouldn't have taken this long - there were some mainline code change I had to make to be able to get Selenium to be able to locate the controls I needed to operate for the tests.] I was able to then refactor the code in about another hour into these basic blocks. The following scenario and bug confirmation tests took only a few minutes a piece to write for complete scenarios, each of which created their own per-test data using the building blocks and common code.

Granted, some of the per-test data should have been refactored into common test functions that entered data directly into the database rather than going through the UI to create the data, but that sometimes has its problems, and since I assert that everything is going smoothly at each step along the way of the UI process, any BVT-breaking bugs will show up there as those tasks will fail for the setup for those tests as well.

Just a side note, if you plan to use Selenium, the most complicated part will be how to fire up and tear down the Java proxy server for Selenium RC to talk to when it is testing. My suggestion is that in the assembly that houses the Selenium tests, we place Java startup code in a method with a TestFixtureSetup or ClassInitialize attribute [MSTest also has the AssemblyInitialize attribute which can be handy. Here is a reference to a comparison between MSTest and NUnit attributes for those of us who do both.] The Java.exe file will need to be located at run time by searching the file system, then its path can be used to launch it with the jar files that make the proxy work. Make sure to check the running tasks before launching it, just in case someone started it manually. At tear down time, we simply look for the Java.exe task using the process information API, and kill it if it's running (and only if the test system started it).

Using these concepts, we should be able to generate a completely automated suite that we can use for build verification, functional testing, scenario tests, and acceptance tests. We can use both Selenium and MSTest, each where appropriate. All of them can be wrapped up using the unit test framework we choose, and can be run automatically by our build system once the new build has been propagated to the test server. Some folks do this only daily, but in my experience, every couple of hours is a bit better time mark to shoot for.

Comments, suggestions, or other experiences? Let's all please collaborate and enter suggestions below - we'll all benefit!