Shawn Neal has a great post (ok it might be a rant) on organizing test code. I agree with his article, on the separation of different kinds of tests into separate assemblies that perhaps can be run at different times (not always with the build). Only the true unit tests should probably be running with each build, and for each developer. If there are other tests that run very fast (by my definition, a fast test either passes or fails in under 1/10 second [100ms]) they too could be included in per-build and per-developer runs, provided that they don't bog down the total test time too much, and provided that they are in separate assemblies.

Someday, when technology allows for it, we should even be able to run our tests in parallel threads or parallel processes (see the 2.5 target features for NUnit). Speed is truly important. If your development team has 10 developers, every minute they wait for a build to complete or a test run to complete is 10 minutes a day of development time wasted. If they build and test 6 times a day each (that's not very much BTW) that is one team development hour per day. Realistically it's likely to be more like 5 or 6 development hours a day.

So if someone is paying attention to how time is spent, we could say perhaps that spending a couple of development hours on the tests to make them run faster, and save one minute per execution would more than pay for itself in a single day.

The moral of this story is keep the tests fast, and when they aren't, fix them or move them out of the main test path.

Acceptance criteria are the real key to being able to develop software in the agile model. Acceptance criteria is not really adequately represented by a use case, even with several alternate path scenarios... I have seen this and it just doesn't do the trick. A scenario is good, but it's really not acceptance criteria. We can however, analyze these scenarios and alternate paths, and usually come up with a pretty good set of criteria.

Beware of the incomplete criteria. It's darn hard to get it all. We know this. But, make an attempt anyway. Use the thinking that if it isn't in the criteria it won't be delivered, and that the developers can implement it any way they want... This thought should be scary enough for a customer that they will be forced to think hard about what they really want the software to do.

When we think we know what we are doing, we can write some tests, write some code. We think we have the idea, only to find out later that we had misunderstood or didn't have all the requirements. Acceptance criteria are absolutely the must-have thing for agile processes to succeed. We need the Customer or product owner [PO] to provide clear, concise, and specific criteria for our software.

I'm not saying that generating/gathering acceptance criteria is easy. It takes practice. It takes cooperation. It takes understanding on both the development team and the customer/PO that "if it isn't requested or specified in the acceptance criteria, it isn't going to be delivered." It's really kind of harsh actually to practice this, so I wouldn't advocate it. However, it might be a good idea to talk about it in those kind of terms while in the sprint planning or story negotiation phase.

Without the ability to know that we are doing the right thing, surely we will fail to deliver it. The customer knows exactly what they want. OK, most customers really don't know what they want. It is up to us as a development team to be able to know this, and help them out by asking lots of questions, and examining exactly what we propose to deliver. After all, that is why they hired us right?

Train your team. Let them know that the ancient role of "Business Analyst" is no place to be found in an agile team, and the whole team is really responsible for making sure to ask the right questions. I do like the whole user story concept, but it is far more practical in sprint planning to capture some of the actual acceptance criteria and store it attached to the story somehow, so that it is accessible by the team and can be implemented as automated tests.

Test automation is critical, mandatory, required, and not optional. Did I mention that test automation is required. Perhaps I should mention it again, in case my previous sentences weren't clear. Automate, Automate, Automate. Double-click run test suite, should be the demo in the sprint review and customer demonstration at the end of the sprint. This is really and truly the only way that agile teams can operate successfully.

If you have people doing any kind of manual testing (other than exploratory), this is a great opportunity to get the team to be far more productive by assisting the manual testers to get these tasks automated. Sometimes it is not practical to automate some manual tests. But in my experience - NOT OFTEN. If it looks hard to automate, someone should be asking why the code is so "untestable"... In most cases, the three or four dev or test hours it takes to automate the most difficult manual test case scenario is more than paid back in the number of times the automation can be run from then on. The number of bugs and breaks that it will prevent is a definite value add to the organization, and in my opinion, money in the bank.

The next Agile Beer Night is coming up on Tuesday September 30, at the Celtic Bayou pub in Redmond from 5PM to 7PM. Be there or be square...

I posted a simple sample of accepance test code in Selenium and WatiN along with a sample web site to test. You can download the zip file here.

I also have posted a Fitnesse fixture in that zip file that illustrates how we can create a simple test fixture for Fitnesse acceptance testing. The Fitnesse tests aren't in the set, but here is the page wiki code that makes use of the fixture:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}
!path dotnet\*.dll

Here is an acceptance test using our BusinessObjectTestFixture test class:

!|FitnesseFixture.BusinessObjectTestFixture|
|UserId|Password|Authenticate()|
|administrator|secret0|ADMIN|
|admin|secret0|NONE|
|administrator|secret|NONE|
|user11|secret11|USER|
|user11|secret0|NONE|
|user|secret|NONE|


I also created some STIQ tests, here is the code for the tests and components. Extract this zip file under repository\ProjectRoot folder and it should be able to test the sample site also.

ATDDSTIQ.zip (3.57 KB)

I have been asked recently about tools I prefer to use in my every-day development. Here is a list of tools, and where to get them.

Visual Studio 2008      Development IDE                   Microsoft
TestDriven.net           Test Runner                         TestDriven.net
WinMerge                  Diff / Merge tool                   WinMerge.org integrates with VS and Tortoise too!
nUnit                        Unit Testing Framework          nUnit.org     Also see mbUnit and Gallio
Selenium                   UI test framework                 OpenQA       Also see WatiN
ReSharper                  Integrated toolkit for VS        JetBrains     (OK I don't actually use it but it's good.)
Tortoise SVN              Shell Integration with SVN      Tigris
Ankh SVN                  Visual Studio Plugin for SVN     CollabNet
Subversion (Server)     Version Control System          VisualSVN
Cruise Control.net       CI system                            ThoughtWorks
RhinoMocks                Mock Object System              Ayende
nAnt                         .NET Build Tool                     nAnt
Fitnesse                    Acceptance Test Tool            Fitnesse
STIQ                         Story Test Tool                    Solutions IQ
GIMP                         GNU Image Manipulation Prog. SourceForge
Notepad++                 Smart Text Editor                 SourceForge UK when Visual Studio just won't do...

Not development tools exactly, but extremely handy:
Process Explorer          Smarter Task Manager           SysInternals
FileZilla                      Upload/FTP client                  FileZilla
DivX                          Decoder                              DivX         Because sometimes we need to watch movies...

That's all I can think of at the moment, but am probably missing some things. I'm sure you'll all (please) chime in with what I forgot...

500 lines of code in one method? uh... U R DOING IT WRONG!

Long methods are a code smell. And a loud one at that. Methods should be focused on doing ONE thing. That thing can be simple or complex, but inner complexity should be refactored out into other (small) methods, keeping them each small and focused on a single task. A method with 4 nested levels of for/foreach/while statements is just so wrong in so many ways... Keep it simple. Keep it straightforward. Did I mention keep it simple?

Jeremy Miller posted a nice article a while back about short methods, and Ward Cunningham has another on C2 that Jeremy references.
We definitely need a Visual Studio plug-in to turn the background yellow when a method exceeds 20 lines, and turn red when it goes over 40 lines...

Here is a simple example. Code by intent means writing code by declaring what it does by well named (and usually long-named) methods. This method is only three lines long, but it conveys that there are two steps to be done, the second with the result of the first. This should be clear to anyone reading it. Further, the XML comment summary (missing from this example) should provide the reader with any information needed not conveyed with the title.

public void DoOneThing()
{
   int n = 0;
   n = DoStep1ReturnNumber(n);
   DoStep2WithResultOfStep1(n);
}

private void DoStep1ReturnNumber(int number)
{
   // simple method
}

private void DoStep2WithResultOfStep1(int number)
{
   // more simple stuff
}

Seriously, if there is something out there already that does this, please let me know! If not, I am going to have to write one...

We the practitioners of Agile and users of Beer, hereby proclaim Agile Beer Night.

The Agile Beer Users Group (Seattle Chapter) will be meeting on Thursday Aug 21, for information sharing, networking, agile discussions, a brief topic presentation on Automated Acceptance Testing, oh - and BEER. Come meet us for a discussion and brief presentation on Agile practices each month.

There is good food, good beer, and a good chance to meet some of the folks who do agile development in the Pacific Northwest, get to know their tricks, and share in their success. The Owl has a fine selection of food, drinks, and spirits for those who choose them.

ABN next meeting will be held in Seattle at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle from 5PM to 8PM on Thursday August 21 2008.

The September ABN meeting will be on the East side in Redmond or Bellevue, but plans are not firm yet - stay tuned...


Please link this post, track it back, and let everyone know!

Next ABN on Thursday August 21 2008 5PM to 8PM at the Owl and Thistle Irish Pub and Restaurant at 808 Post Avenue in downtown Seattle

I learned a valuable lesson today using Selenium. It seems that the DragAndDropToObject() function doesn't scale well when the HTML is being updated in a loop by the mainline code... This method uses two locator strings, one to find the object to drag, and the other to find the object to drop.

It apparently has to examine the entire XML DOM for each locator, each time through. So, when I created a loop that did a drag and drop 250 times on a page, the first few were not too bad only a couple hundred milliseconds. However, as more and more DHTML was added to the page, it took longer and longer to drop the next objects. I measured this time, and when I charted the data, it was a moderately increasing exponential slope. Definitely not even linear...

It seems that the more HTML was added to the DOM, the longer it took both of the locators to locate their objects, even though those objects were static on the page. It walked the DOM each time, for each locator. Thus the increasing non-linear time slope, because the DOM grows with every loop iteration.

By the time it got to the 50th iteration through the loop, the time taken was almost 2 orders of magnitude higher than the initial one. It made testing this particular feature not doable through this framework. And in measuring how long it took the actual javascript to render the new DHTML, it was just around 100ms, when the Selenium locates for the command took upwards of 7000ms for the 50th loop, but was only 200ms at the initial loop.

The Solution
I didn't find a complete solution to the excruciating amount of time the locator code needs to walk the DOM, but I did end up switching from the DragAndDropToObject() method to the DragAndDrop() method. The latter uses one locator string and a relative X,Y coordinate string (NOT a locator). It's just about zero time for it to process the relative coordinate string rather than the locator. So, in my specific case with this test, I was able to cut the time down to 2.5 seconds for the 250th drag and drop. Still a very LONG running test, but it worked (oh yeah and it found a nice juicy bug too...).

OpenQA people - it might behoove you to take a look at optimizing how you interact with a DOM that is growing or being updated by the code under test... A bit of performance gain in this area would really come in handy.

Both Selenium and WatiN have been around for a while, and both are fairly good tools when it comes to UI automated testing. Both tools now drive both IE7 and FF2, and are free to download. I have been using Selenium RC for a while now, and it seems to be fairly robust, although sometimes rather frustrating and somewhat slow. Both have a recorder mechanism, and both emit C# code for me. However the WatiN recorder did blow sky high with an unhandled exception when a pop-up window came up, but the recorder code was written by someone else so I won't hold it against WatiN.

I tried WatiN for the first time today, and decided to go with the V.2 CTP instead of the .NET 1.1 release. It seems that some of the API's have been refactored into more sensible structures (factory patterns, etc.), but somehow the things that would have made the most sense to me in the API were structured much differently that I would have expected. There were some problems with my IE (it kept crashing unless I ran the debug version of W2). However, I blame this on IE, not on W2. FF has a plug-in that needs to be manually installed. It comes with the CTP, but doesn't get installed. Hopefully this will be integrated by release.

Most of the basic features are similar in function but not implementation. I can see writing an interface class and a couple of wrapper classes that implement the interface, one each for WatiN and Selenium, so we can have the best of both worlds.

WatiN Pro's

• No proxy server required. It just runs. Runs both FF and IE, switchable with the parameter to the factory method that creates the instance.
  

WatiN Con's

• Documentation is less than sparse.
• I could not find a way to do a "drag and drop" which is easily accomplished with Selenium.
• Does not handle pop-up windows. This was the show-stopper for me.

Selenium Pro's

• polished code
• well-documented functionality
• lots of examples
• handles lots of weird types of conditions and situations (most of which we will all need at some point.)
  

Selenium Con's

• have to run a java proxy in a separate process and set it up and tear it down
  
• slow in places, due to proxy and not-so-well-written sections of code
• locators aren't always easy to figure out. Tools like Firebug and XPather help identify locator strings.
  

Since it does not handle pop-up windows (or I just couldn't figure it out in about 4h of research), this was the end of the line for it. Selenium does handle them, and has been able to accomplish 100% of the weird scenario tests I have needed so far. Loads of tweaking and frustrating trial and error notwithstanding.

So, until we have a bit closer to release for WatiN, Selenium still wins (for now). However, I definitely plan to write the interface and shim both out so I can use either or both later on. Stay tuned for that posting, that will be the money shot...

In closing, let me just reiterate that every developer should be thinking about UI automated testing for web code. Each user-interactive element on the page NEEDS AN ID. That's every link, every button, every text box, checkbox, radio button, and other user input elements. The ID's need to be simple, and either fixed, or predictable. With these things in place in the production code, it will be testable and easier to validate. Better quality code will make it out to the hands of the users, and all will be right with the world. Amen.

<steps down from soapbox./>