Choosing Twist - Saying goodbye to Fitnesse
I wont go into the whys and how’s in this post, but my team and i found ourselves grappling with FitNesse. It was a huge undertaking for us. Something of a torrid affair, all love hate and very emotive. But now we have embarked on a new journey of discovery and left FitNesse behind moving to twist. Here in this post, I will hopefully capture why and how of the transition.
Before I go any further I should perhaps lay out my stall. We are a large website operating in the UK. When I say large, I’m talking in terms of the footfall on the site and the revenue generated. Around 500,000,000 page views a month genberated by 10% of the UK population. Traditionally all of our testing had been manual and we were a Prince II-ish development house; aka the waterfall approach.
All of our testers are career testers and should not be confused with TDD developers, or BA’s or UAT-ers. Our testers are plain old fashioned, passionate (ISEB accredited) functional and non functional software testers. To us, testing is our career not a stepping stone to project management.
Our scope was varied. We would test from just above Unit testing all the way up to and including UAT (that is most of the right hand side of the V model). Yes we are testers, but we are technical testers; in that we understand HTTP, and Ajax and CSS and load balancers, edge suites, 3DNS, PL-SQL, Jboss, Apache, J2EE etc etc. Experience has taught us that on the whole we understand a lot more than your average "web tester" . However we don’t do code, ever, never ever.
In the past we have used a number of what could be described as "de-facto" test tools, most notably the Microsoft Office suite (Excel spreadsheets, Word templates and so on). Of course there has been much talk and pitches from prospective Testing Tool vendors over the years, and when my team was first learning how to get Fit with FitNesse another department was seriously evaluating QTP and Axe, more on that later.
We were originally introduced to FitNesse through a project that we had decided we would run using Agile principles (note the big A, we went all out). We had some guidance from Agile partners ThoughtWorks who were helping us find our way in the Agile world, and they said we would need to seriously consider test automation, and suggested a few tools, one of which was FitNesse.
I have made a previous comment that the first time we used FitNesse, we felt like the Apes in the beginning of A Space Odyssey 2001. There we were staring up at this great monolith, with no idea what to do with it, but we knew it was important.
Through some heavy googling we settled on the Webtest fixture by Gojko Adzic. We soon became comfortable with FitNesse and Gojkos fixture. We rapidly begun to out grow the webtest fixture. Before we knew it, the thing we called the webtest fixture contained very little of Gojkos code and had code specific to some of the idiosyncrasies of our website; it had become our DSL. Yes, we were writing code. This was quite scary for us, because who tests the testers tests? However no sooner had we begun to feel very comfortable with writing the fixtures we needed to drive selenium or execute a stored procedure than we started to feel the pain of FitNesse. Here is a short (by no means exhaustive) in no particular order list of our pain points:
Version control
Version control in FitNesse or with FitNesse, is one thing that still sends shudders down my spine. I know it can be done (we achieved it to some extent) but we had little faith in it, and consequently the team would keep local revisions on their machines.
Duplicate code
The code was continually being replicated/duplicated and different testers would write different methods to do the same thing, e.g
- The user navigates to <url>
- The user Opens <url>
- The URL <url> is opened
- Open the Bowser and Types <url>
Despite our best efforts to keep an internal dictionary (and an ant task to generate html JavaDocs) we would still find duplicate (or very similar) methods.
Wiki Markup
The wiki mark-up language. I know its fairly straight forward, but when you have to escape words that you need to use in your test so that FitNesse doesn’t try and parse it, it gets overly complicated quickly. I want to write a test quickly, not have to remember the syntax for bold italics.
Metrics
Lack of metrics, seriously is this a test tool? Not having metrics available after the test had run was and still is the Achilles heel of FitNesse.
Setup and Teardown
Setup and Teardown – this seemed great at first, until we discovered that when we wanted to pass parameters around the tests based on an initial value that was discovered in a setup routine we had to jump through hoops. Things are further compounded when you have a large suite that needs multiple setups and teardowns to allow the tests to be independent.
Lack of Documentation
While there is some documentation out there a lot of it confusing. Even the FitNesse website is confusing running inside FitNesse it has pages that lead nowhere.
Tables
We found that because of the nature of the beast we needed to express all of our tests as tables. We learnt how to use Fitlibrary's DoFixture to help us get around that limitation. however, once we had gained a better understanding of the what we wanted to test we would end up back in a table. The downside of this is that we may have wanted a test to read:
"a registered user returns to the site and successfully logs in."
The tabular nature of the test would coerce us into using each row in the table as a step to achieve the goal.
The whole table thing was such a PITA that i looked at using David Peterson's Concordion, it seems like a good alternative because it doesn't have the same dependence on tables, but i didn't want to disappear Alice like down a the rabbit hole.
Throughout this painful period we just kept thinking it’s our fault because we don’t know how to use Fit and FitNesse is just exasperating that. However there became point in time when we realised that we did posses more FitNesse knowledge than we gave ourselves credit for and that realisation galvanised the thought that FitNesse was unsuitable for what we wanted. Ergo, there must be a better way.
In short, we had a real love hat relationship with FitNesse. While FitNesse seemed to present us with more challenges than benefits, the benefits outweighed the effort to make it all work, and to that end, I became very proactive, and begun selling FitNesse to the business at large.
Remember that evaluation of QTP I was talking about? This was partly driven by a new development manager who was eager to influence the department with his experiences of continuous integration; not having any existing automated tests was a non starter for him.
We spent a significant amount of money on some POC work to see if QTP and Odin’s AXE would work for us. It took 5 consultants 1 week to complete that proof if concept and demo their findings, however that work was quickly put to shame by a tester who had never used FitNesse or selenium, but had seen its potential on our project. In one evening at home he was able to create an end to end test for his product, whereas the consultancy team who conducted the POC work (hint: this was their product) were unable to automate more than one fifth of the application in that week, and even then they could only drive Microsoft Internet Explorer. In one fell swoop FitNesse and Selenium saved the department £40,000 and another team adopted FitNesse.
Meanwhile our life with FitNesse was coming to an end, or so we thought. Our toe dipping exercise into the world of Agile was hailed by all as a success, and so it was decreed that from now on it was the Agile way or no way.
End of Part One
