Friday 27 February 2009

Choosing Twist - Part One


Choosing Twist - Saying goodbye to Fitnesse

I wont go into the whys and how’s in this post, but my team and i found ourselves grappling with FitNesse. It was a huge undertaking for us. Something of a torrid affair, all love hate and very emotive. But now we have embarked on a new journey of discovery and left FitNesse behind moving to twist. Here in this post, I will hopefully capture why and how of the transition.

Before I go any further I should perhaps lay out my stall. We are a large website operating in the UK. When I say large, I’m talking in terms of the footfall on the site and the revenue generated. Around 500,000,000 page views a month genberated by 10% of the UK population. Traditionally all of our testing had been manual and we were a Prince II-ish development house; aka the waterfall approach.

All of our testers are career testers and should not be confused with TDD developers, or BA’s or UAT-ers. Our testers are plain old fashioned, passionate (ISEB accredited) functional and non functional software testers. To us, testing is our career not a stepping stone to project management.

Our scope was varied. We would test from just above Unit testing all the way up to and including UAT (that is most of the right hand side of the V model). Yes we are testers, but we are technical testers; in that we understand HTTP, and Ajax and CSS and load balancers, edge suites, 3DNS, PL-SQL, Jboss, Apache, J2EE etc etc. Experience has taught us that on the whole we understand a lot more than your average "web tester" . However we don’t do code, ever, never ever.

In the past we have used a number of what could be described as "de-facto" test tools, most notably the Microsoft Office suite (Excel spreadsheets, Word templates and so on). Of course there has been much talk and pitches from prospective Testing Tool vendors over the years, and when my team was first learning how to get Fit with FitNesse another department was seriously evaluating QTP and Axe, more on that later.

We were originally introduced to FitNesse through a project that we had decided we would run using Agile principles (note the big A, we went all out). We had some guidance from Agile partners ThoughtWorks who were helping us find our way in the Agile world, and they said we would need to seriously consider test automation, and suggested a few tools, one of which was FitNesse.

I have made a previous comment that the first time we used FitNesse, we felt like the Apes in the beginning of A Space Odyssey 2001. There we were staring up at this great monolith, with no idea what to do with it, but we knew it was important.

Through some heavy googling we settled on the Webtest fixture by Gojko Adzic. We soon became comfortable with FitNesse and Gojkos fixture. We rapidly begun to out grow the webtest fixture. Before we knew it, the thing we called the webtest fixture contained very little of Gojkos code and had code specific to some of the idiosyncrasies of our website; it had become our DSL. Yes, we were writing code. This was quite scary for us, because who tests the testers tests? However no sooner had we begun to feel very comfortable with writing the fixtures we needed to drive selenium or execute a stored procedure than we started to feel the pain of FitNesse. Here is a short (by no means exhaustive) in no particular order list of our pain points:

Version control
Version control in FitNesse or with FitNesse, is one thing that still sends shudders down my spine. I know it can be done (we achieved it to some extent) but we had little faith in it, and consequently the team would keep local revisions on their machines.

Duplicate code
The code was continually being replicated/duplicated and different testers would write different methods to do the same thing, e.g

  • The user navigates to <url>

  • The user Opens <url>

  • The URL <url> is opened

  • Open the Bowser and Types <url>


Despite our best efforts to keep an internal dictionary (and an ant task to generate html JavaDocs) we would still find duplicate (or very similar) methods.

Wiki Markup
The wiki mark-up language. I know its fairly straight forward, but when you have to escape words that you need to use in your test so that FitNesse doesn’t try and parse it, it gets overly complicated quickly. I want to write a test quickly, not have to remember the syntax for bold italics.

Metrics
Lack of metrics, seriously is this a test tool? Not having metrics available after the test had run was and still is the Achilles heel of FitNesse.

Setup and Teardown
Setup and Teardown – this seemed great at first, until we discovered that when we wanted to pass parameters around the tests based on an initial value that was discovered in a setup routine we had to jump through hoops. Things are further compounded when you have a large suite that needs multiple setups and teardowns to allow the tests to be independent.

Lack of Documentation

While there is some documentation out there a lot of it confusing. Even the FitNesse website is confusing running inside FitNesse it has pages that lead nowhere.

Tables
We found that because of the nature of the beast we needed to express all of our tests as tables. We learnt how to use Fitlibrary's DoFixture to help us get around that limitation. however, once we had gained a better understanding of the what we wanted to test we would end up back in a table. The downside of this is that we may have wanted a test to read:

"a registered user returns to the site and successfully logs in."

The tabular nature of the test would coerce us into using each row in the table as a step to achieve the goal.

The whole table thing was such a PITA that i looked at using David Peterson's Concordion, it seems like a good alternative because it doesn't have the same dependence on tables, but i didn't want to disappear Alice like down a the rabbit hole.

Throughout this painful period we just kept thinking it’s our fault because we don’t know how to use Fit and FitNesse is just exasperating that. However there became point in time when we realised that we did posses more FitNesse knowledge than we gave ourselves credit for and that realisation galvanised the thought that FitNesse was unsuitable for what we wanted. Ergo, there must be a better way.
In short, we had a real love hat relationship with FitNesse. While FitNesse seemed to present us with more challenges than benefits, the benefits outweighed the effort to make it all work, and to that end, I became very proactive, and begun selling FitNesse to the business at large.

Remember that evaluation of QTP I was talking about? This was partly driven by a new development manager who was eager to influence the department with his experiences of continuous integration; not having any existing automated tests was a non starter for him.
We spent a significant amount of money on some POC work to see if QTP and Odin’s AXE would work for us. It took 5 consultants 1 week to complete that proof if concept and demo their findings, however that work was quickly put to shame by a tester who had never used FitNesse or selenium, but had seen its potential on our project. In one evening at home he was able to create an end to end test for his product, whereas the consultancy team who conducted the POC work (hint: this was their product) were unable to automate more than one fifth of the application in that week, and even then they could only drive Microsoft Internet Explorer. In one fell swoop FitNesse and Selenium saved the department £40,000 and another team adopted FitNesse.

Meanwhile our life with FitNesse was coming to an end, or so we thought. Our toe dipping exercise into the world of Agile was hailed by all as a success, and so it was decreed that from now on it was the Agile way or no way.

End of Part One

Wednesday 11 February 2009

Debian - Backup and restore instaled software

- This is just a quick post so i can remember how i did something -

So I have taken an old Ferrograph SDX display and put new firmware on it ready to integrate it into an XFD (more on this in a later post).

I have decided to drive the display using Perl as there is a requirement to get RSS and xml and parse it appropriately build the packets for the sign and send them to the serial port of the host PC. I chose Perl because it has all of that stuff ready to go and i can develop on my local pc (windows) and port it to Linux fairly easily.

Anyhoo, i had already built an IRC server using an old junked desktop machine for the teams to use as a communication tool, and i built that on top of Debian etch. I couldn't use the same machine for the XFD because it doesn't have a serial port. I got hold of another junked desktop machine with a serial port and installed ebian etch on there too, and soon realised i couldn't remember what software i had installed on the IRC box. I wanted to keep them in sync.

Backup list of installed software

to find out what software i had installed i used the dpkg command to list installed software:


$ dpkg --get-selections


I then redirected the output into a file to store the list of installed software.

$ dpkg --get-selections > ~/installed-software.log

on the other box i used sftp to fetch the file, and used that to select the software i wanted to install

All i had to do is type following two commands:

# dpkg --set-selections < installed-software.log

Now my list is imported i need to use dselect to install the package.

# dselect

Select 'i' at the menu for install the software, answering Y to the question about additional space.

Job done!

Wednesday 4 February 2009

going round the Twist

I haven't posted for a while, and that's because we have been up to our ears in Twist.

Twist is a tool made by ThoughtWorks that take a lot of the good ideas from FitNesse and bolts them on top of eclipse.

I cant be too harsh on Twist because it is still in beta, however we have had a lot of pain to deal with.

For one thing, ThoughtWorks appear to have thrown the baby out with the bath water as the supplied selenium appears to be altogether different.

So this is where we are.

We have Twist installed on all the testers machines, and we have installed subclipse into Twist. This takes care of the version control issues we had with fitnesse.

One issue we did run into was getting the Twist project to play nice within the main project Trunk (more on this later).

The guys are writing the prose like business speak heavy tests quickly (the Twist Scenarios) however the instrumentation of those tests has become a little bottle neck, and some dev guys have come on board to help us with that.

What was interesting was seeing the dev guys pulling their hair out over Twist too.

I will post a fuller write up soon that explains how we run the twist tests (they are running as part of the CI build) and how we get reports from those tests.

Stuart.