Working Thought - A testers Blog

Testers - we don't need that any more.

2010-07-17T11:52:00.000+01:00

In the past Uncle Bob and Martin Folwer have both pissed me off with their "we don't need testers anymore" messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn't met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed.

So O.K, let me stand back for a moment and reflect on what XP teaches us in that a highly disciplined team should be able to achieve zero defects, so its easy to say, zero defects means we don't need testers.
Moreover, at the SIGIST conference in June, Julian Harty (ex Google senior test engineer) delivered a presentation that posed the question, do we need testers. What if we stop testing? After all these days people seem to want speed first and fully working functionality secondary. The future of testing?
So yes the playing field has changed, its true. But that doesn't mean i have to agree with it.

If the software is crummy, your users will only tolerate it to a point. A great example of this is reading the feedback comments on iPhone App store. Even when a piece of software is offered for free, people leave bad feedback complaining about the uselessness or poor UI etc.

The crux of the matter for me is this. Even if we did find the magic incantation that could give us defect free code, it still wouldn't mean the business would get quality software.

I'm signed up to Bachs ideal, but i'm embedded in an XP/Lean team. We have 90% test covereage across 903 classes and 201,000 lines of code (and counting). Those thousands of tests run in seconds, before they make their way up the CI pipeline to run UI and performace tests. No nightly build, we test every checkin.

CI had my support 100%, because it should mitigate lots of manual testing. In fact it should mitigate lots of testing. As the product we work on has matured, the CI has formed a regression pack. Its sound great right? So why do I still have a team of 8 testers and a 15% defect rate?? Because developers are humans not robots. Because developers focus on what "Done" looks like, and not the bigger picture. Because they dont understand the context within which the story was written and thus make an assumption. Because they are testing in the small. Because they didn't have the right data in the development environment. Because there exists two different mindsets between good developers and good testers. But non of those reasons are new, or particular to Agile. Its all old news.

But then i read this by Chief Frank C. Montagna

"Firefighters, as all humans, make mistakes. When firefighters make a mistake on the job, however, it can be life-threatening to themselves, to their coworkers, and to the public they serve. Nonetheless, firefighters will continue to make mistakes and on occasion will repeat a mistake.

Our goal should be to learn from each mistake and to try not to repeat it. We should also teach others not to make the same mistake we made. To do this, we have to admit our mistake publicly by telling others about it. This is not always easy to do. "

This Agile vs Tester friction just needs us, the testers to be pragmatic and professional; we must continue to share best practice, collaborate and talk to the developers earlier in story life-cycle. We should shout STOP! sooner rather than later, jump in feet first and embrace what may feel like an awkward discussion.

We testers are not here to simply find defects, appoint blame, or hold a "who sucks the most" post mortem. We are here to add value, and if we don't, then why do we need testers?

A defect report is pretty meaningless. You cant sell it or market it to your customers. Worse, it has overheads. So why have one? Why not have a discussion with a developer and a product owner instead?

We (Developers and QA) both work for the same company, so why aren't we working together, and demonstrating to Martin and Uncle Bob, that we are valuable and that testing and testers should be taken seriously.

The Tester's Headache: Tester Certification - my take...

2010-05-19T08:14:00.001+01:00

Reading Simons Blog post The Tester's Headache: Tester Certification - my take... got me simmering again.

This something I have been bending my peers ears with and blogged in passing for over a year now.

For me the value in the foundation cert. Was two fold (btw I'm old school ISEB), it gave testers a commonon taxonomy and the course provided valuable networking.

Today too many of the candidates I interview simply see the certificateas a corgi gas fiiters badge. I have the cert. Therefore I'm a tester.

Worse still many candidates have self studied, and there the only aim is to complete practice exam papers or question banks.

I was chilled to the core when speaking with Dave Evans from SQS as I thought the Agile Tester cert. I had heard about was a joke, or tester lore. It seems istqb are serious.

All that has happened is that the cert. Has been de valued, and so I don't make it a "must have" in the job spec anymore.

Has the calibre of applicant decreased? Nope, not one bit, it's just increased the volume.

I still have the blisters from the ISEB practitioner exam, but I can't think of anything I use from the sylabus now we are all out Agile.

Stuart

Making the Ferrograph SDX 63 Aurora XML aware

2010-05-18T15:46:00.000+01:00

I mentioned in a previous post how we have used an old Ferrograph SDX 63 LED moving message display / wallboard and Robert Cowards excellent ADF firmware to give us a sign that is to be used for our Extreme Feedback device.

Once i had got my head round the Alpha Protocol, i started to knit a huge perl module together to control the sign. Essentially all it did was create an alpha protocol formated string and send it to the sign. At first this seed to be the logical way to approach the problem. I had already written an IRC bot that announces the build status into the channel and it too was written in about 10 lines of perl, so giving the bot access to the the sign module would allow the bot to relay messages to the sign.

However, once you start send messages to the sign you realise that what you actually want is an SGML way of writing message to the sign, that is to say, create your message in your mark up then send it to the sign. The sign itself only has a Z80 processor and its already overburdened with the task of displaying data on the LEDs, so making the sign understand SGML was out of the question. So how about using XML and transforming the XML through an XSLT that would result in a stream of data the sign could understand? At last i found something i actually made real use of XML transformation.

XSLT is commonly used to turn XML into html, all i had to do was create an XSL with the appropriate taxonomy for the ADF firmware and then build up my message using my own SGML.

I've never found a use for XML before, I've always found it to be a bit pointless, but then that s probably to do with my age. Once upon a time we only had 8-bit processors and 1k of RAM, and those sort of constraints make you think about how you are going to deal with your data very carefully. The last thing i would do is transmit 80 bytes of data in a 4k document, just because it can specify its contents.

SO no sooner had I finished my work when as usual i found i wasn't the first, and software already exists that can speak the Alpha protocol and uses XML and XSLT its called bbxml (BB from the betabrite signs). Bugger!

Twist behind a proxy

2010-05-18T15:35:00.000+01:00

So, we had this issue when we used Selenium 1.0 so it was bound to crop up with Twist (after all it just runs selenium, right?).

We need to make selenium proxy through our corporate proxy.

http://studio.thoughtworks.com/discussion/forums/21/topics/970

We have statistics and charts to prove it so f**k off!

2010-05-18T15:28:00.000+01:00

Its been a while since i've posted, and that has been due to partly due to the pressures of work, but but also because i didnt think i had anything significant to share. That is until this week when i had a moment of WTF!!.

I want to share with you something that is seemingly innocuous but actually very toxic; namley using velocity as a performance metric!

I have never been a big fan of metrics for metrics sake, because a lot of them a pretty meaningless. We have a focus, and that is the delivery of software. All the business wants to know is that what they have asked for is to be delievered, on time and hopefully to budget. Why bother producing Burn down or burn up charts if we dont factor in defects or Tech tasks and besides what is the point of any of it if one story can block 20 others?

The old software delivery problem still exists right?

Project Manager: How much have you got left to test?

Tester: Just one story out of the 40 we had.

Project Manager: Cool, the testing guys are nearly done.

- One week later....

Project Manager: How much have you got left to test?

Tester: Just that one story, but boy-oh-boy its a doozy.

For me metrics fall into two piles:

1. Diagnostics Metrics: These are informative indicators chosen by the team that are used to evaluate and improve the teams process. We collect diagnostic metrics to give us some insight into where we can improve, and to track whether the improvements have any effect. Simply informative, never to acertain how much value is being delieverd and of a fairly short life; if the process improves, we move on and identify other sub-optimal areas. The measurmnent is dropped a that point, its served its purpose.

2. Performance Metrics: This the measurement that details how much value is being delievered. These are used to track our performance, and are chosen carefully. There should be fewer rather than more of these metrics. Wherever possible these metrics should be outside of the teams control, that way the numbers can not be perversed. E.g User or customer feedback. "how satisfied are you with the service/product" or "how likely are you to recommend the service offered by the team to another?".

The problem i have at the moment is that our velocity is being used as a performance metric rather than a diagnostic aid. Incredible i know, but true. Why? Because velocity cant be used as a performance measure, because it is unique to the team being measured at that point in time and only within that context. It changes, like the ebb and flow of a tide.

Not only is someone trying to use velocity as an indication of the teams performance, i have also heard the following remarks made.

Why is Team A slower than Team B? Wow, where do i start. Because they estimate in different scales, or maybe their iteration length is different, possibly the team composition is different? There are so many factors that can influence velocity that it’s only useful to compare it within the same team, and even then just to identify trends. The absolute value doesn’t mean anything.
We only did 28 last iteration, can we push for 38, and then 58? The beginning of a project can have a naturaly low velocity, and this should be expected, however it should begin to increase after a number of iterations. However we have been asked to push and push to improve velocity when we reach a natural plateau, and we have pushed, after all, we want to go faster, because we rock, right? Because velocity indicates the capability of our teams to deliver it will settle down itself (because our process is stable, and we haven't employed any creative accounting). One my heroes is Deming who noted that in a stable process, the way to improve is to change the process.

It’s important that as journeymen (and women) of the Agile craft we remember that velocity is just a by-product of our current reality (our teams, our processes and our tools). We can only improve our process once it’s stable and we know the current capacity.
Velocity is little more than a health-check that help inform us about our team’s capability. It can not tell us about how much value is being delivered or for that matter how fast its being delivered. We can deliver a lot of story points but only through making a trade-offs with our quality which, however we measure it, will impact our ability to deliver more. As uncle Bob Martin says:

“The way to go fast, is to go well"

Never use velocity to measure performance. After all the thing being tracked is a gumi bear, or pink elephant. Simply look at velocity as a diagnostic metric to help us improve our software delivery process.

Life after ThoughtWorks Part Deux

2010-04-22T18:18:00.002+01:00

Last year i wrote a blog post Life After ThoughtWorks and this year i was reminded of it by an (now ex) ThoughtWorker Chris Read.

I re read the old post and marvelled at how far i had come in twelve months, and also how my attitude towards TW has changed too.

Now, twelve months on from that last post and I'm older for sure, more knowledgeable too, but i'm still passionate about Agile software delivery. So passionate in fact it often leaks out into my daily life. I have jokingly called my self evangelical about it. I'm often challenged by people who have had a less than optimal experience with Agile as a software delivery methodology, and sometimes i can feel their hate and anger for this stupid thing called Agile that everyone is talking about.

So, first up an apology, to a good friend of mine Keith Henry. Keith was talking to me about this weird and wonderful stuff called Agile at least 6 years ago, and i dismissed him as a nutter, while he just nodded and grinned; he knew I would succumb eventually, then he could (rightfully) say i told you so. Keith, i'm sorry it was I that was the nutter.

Next up a thank you. To several ThoughtWorkers who have changed my perceptions, shared their wealth of knowledge with me, and listened to my many, many criticisms, scepticism and pessimisms. They taught me many things (often through observation and not directly), and so, in no particular order:

Luke Barrett - for many things, the consummate consultant and voice of reason.
Graham "Wookie" Brooks - for teaching me that you can teach an old dog new tricks.
Chris "Dude" Read - for the filth and the energy.
Tom "Boobies" Scott - for teaching me to get my game face on, and never high 5 in public.
Richard "ffs" Fillippi - for the "Victor Kiam" moment, classic.
Sam Newman - don't over promise and under deliver.
Jo Cranford - for making nice story cards and making coke come out of Chris Reads nose.
Pat Kua - Agile and performance testing can work.

I worked with many, many more ThoughtWorkers, but those above are the few who have shaped who I am twelve months on.

The big Question - Are we coping?

So TW roll off, hows it all hanging, are we still Agile? I think we mostly are, there have been a few times were the edges became a little frayed, but I think that in the past we may have stood by and watched as the fabric fell apart. This time we were able to tell that we needed to act, and were able to put in place a fix that was secure enough to have the required longevity and light enough as so not to deplete us of time and resource, and of course, we left the repair until the last responsible moment.

Our big challenge is not staying Agile, but keeping one team, that is distributed across two locations in sync. More than that, keep the synergy. That is not a software delivery problem, that's a people problem.

Painting over the rust - Preparation, preparation, preparation.

2010-04-22T18:17:00.001+01:00

I thought I would follow on from my post painting over the rust.
It would be too easy for me just to say, hey our business people don’t get Agile, boo hoo. So I thought I would try and address the issue by thinking about how I felt when I was first introduced to Agile. Thinking about it this way allowed me to take a step back and understand why they don’t get it.

1. All Requirements are not needed up front.
This ones a biggie. I wrestled with it for ages, but then I would, I’m a tester and I need those requirements to do my job; after all the V-Model is nothing without upfront requirements, right?
How did I overcome this? Easy (for me at least), in that in our old way of working, even when the requirements were available early, they often didn’t reflect what was actually delivered. This is software delivery after all; we expect the customer to change its mind during the life cycle. So if I’m used to finding that the requirements documentation don’t match what’s delivered into test, why am I so worked up now? Oh wait, I’m not after all. It just felt uncomfortable because its new to me.

2. Story Points are a guess but the deadline is still firm (if not impossible).
This one is a bit of a paradox. Both are true. During estimation we size the stories with t-shirt sizes or points. The point or size at that stage is no more than an informed/educated guess. Again I had a hard time with this, until I realised that it is just a concept that allows us humans to gauge the relative size of the stories. Wrap your head around the simplicity of that device, and your home free. Sure it’s a guesstimate, but we can still meet the deadline, knowing how much work is in there. Once you get the fact that its just about relative sizing and not a unit of currency for determining performance, it becomes second nature. Its all about making sure the team gets the same amount of work passed down the conveyor belt, and that they don’t get under or over utilised. Once you know they can knock out two 4 pointers and a 6 per iteration, you know your capacity (well mostly).

3. Tasks do not get assigned at Iteration Kick off.
I didn’t struggle with this too much, in that previously a task could sit with someone that didn’t have the bandwidth to complete the task for a very long time. One important protocol is leaving things until “the last responsible moment”. By not assigning tasks we can leverage the flexibility that the last responsible moment affords. Also, during the iteration planning, the team can agree what tasks make up a story and perhaps identify the required SMEs. The whole team takes Responsibility for promises the team made. We will deliver value.

4. An increase in speed comes from maximising Quality.
It took me a while to fully understand the power of this. If the team gets it right first time, then it spend less time fixing broken code, or miss understood/poorly captured requirements. I’m sure some Kanbaners out there will be nodding sagely. But this is one of the core principals that Toyota follow, they claim to produce one brand new, fully tested, inspected and passed vehicle in the same time other manufacturers spend snag fixing at the end of their lines.
Traditionally in software delivery, code fixes are not time boxed, and a defect can loop through QA and DEV several times before its fixed. One pair can sink a load of time into fixing the defect while QA source the effort around regression. It’s a massive waste. Enforce a Zero Defect policy; get the business to sign up to it, get buy in from the operational functions (Helpdesk, System Administrators etc) and say goodbye to the defect backlog. Get it right first time and watch throughput soar. Write your tests, then your code, then apply your design.

In Summary

It was difficulty for me to adjust to being a tester in an Agile world, and it took me time. I learnt through an apprenticeship with ThoughtWorks. They may have their own flavour of Agile, regardless its not something you can learn in a book or on a course. Much of what i learnt about Agile is more about your mindset, and self discipline.

Several times along the way, I thought, this is rubbish, this is causing pain and this doesn’t work. But hey that’s Agile right? If it doesn’t work for you, you can stop, change, and monitor for improvement. Once you appreciate that, you become empowered. You can leverage the power of one.

But, we had left our business behind. We (the software delivery team) decided we had to be Agile to deliver a project, without considering the impact to the business. Worse than that, we undertook an apprenticeship with ThoughtWorks and expected the business to understand story points, velocity, waste, lean, eXtreme Programming... the list goes on.

If you haven’t taken the time to bring your business along for the ride you cant expect them just to get it.
If you use enough filler, you can paint over the rust.

Painting Over the Rust

2010-03-18T13:33:00.001Z

Having Story Cards and Stand-Ups doesn't make you Agile.

It is with a heavy heart that i watch the team descend into the latest form of chaos. Looking down on it now, i can see how we arrived here. One of the developers used the phrase "painting over the rust" to describe his dissatisfaction with the architectural decisions being made early on in the project. He was right, however what he didn't know, was he was right on a number of levels.

I've struggled for some time with not being able to intelligently articulate what Agile software delivery is exactly. I hear lots of people repeating the the same old mantras, but for me, they lacked depth. Reciting the Agile manifesto to my grandmother isn't going to help her understand what Agile software delivery is. I reason that if I don't feel Able to describe it t my grandmother, how could i ever manage to explain it to the upper echelons of the executive?

Its not Agile, its fragile.

So the team has a story wall, and swim lanes and cards. Pretty standard stuff (actually the wall is further segmented by geographical location, because the team is split across two sites). Estimation is done planning poker style and points are points, pink elephants if you like, not days, or elapsed time (although we have worked out how much a point cost).

The team knows how many points we can get through the production line, and is comfortable managing the loss in throughput from context switching or working on new technologies etc.

We have a team made up of DEV pairs, QAs and BAs, and they understand what Agile software delivery is. They understand why we pair program. The developers are using TDD and we automate as much as we can using Continuous Integration. We have stand ups, retrospectives, iteration kick off meetings and showcases. We have brown bags and we use collaboration tools. We as a software delivery team have made the paradigm shift. But its still all buggered, why? Effectively, all we have done is painted over the rust.
The rust in our case is, (unfortunately) the executive. They don't get it. Worse, yet i don't think the product owners don't get it, and in some cases i even wonder if the project managers get it. So what can we do?

We could shield the business from our internal workings. Let them treat us like a black box. They request function x, and we return a cost and an estimated delivery date. That way, they don't have to get it, they don't have to understand it, nothing changes for them, and the delivery team carries on irrespective. They then don't need to learn about points or swim lanes or story cards or any other aspect of Agile. But the rust is still there, under the shiny new Agile paint, and its not going to go away, and left untreated it could get worse.

In the real world, there are only two real ways to treat rust. Cut it out, or blast it away. You can then replace the resultant hole with shiny new metal.Once the new metal is in place, its a good idea to apply a liberal coating of rust inhibitor.

2010-02-24T22:04:00.002Z

DNS pre fetch is something i have been looking into, but so far as in-browser DNS pre fetch tags go, its still a dream...

<link rel="dns-prefetch" href="http://working-thought.blogger.com">

Mozilla and Chromium browsers should be able to perform DNS pre-fetching to reduce the overall page load time, however, testing this out doesn't match the published behaviour.

MSIE simply does not pre fetch, as expected, perhaps IE9 will?
Google Chrome does pre fetch and it does seem to do it asynchronously (as per published specs), not surprising, see below.
Safari does not pre fetch.
FireFox should be able to pre fetch (see here) but i couldn't make it go.

Meanwhile Google have re invented the DNS server with one that incorporates DNS pre fetch.

Normally when the ISPs DNS server goes back to the authoritative host to get the DNS record it just caches it without inspecting the TTL on the record. For our site that is 300 seconds, which is deliberately quite short. The next time a user requests the DNS record from their ISPs DNS server, only then does it inspect the TTL and if its expired then it will go back to the authoritative host.

Google found that their googlebot was spending more time looking up DNS records, than it was spidering pages. So they made their DNS server inspect the TTL of the record and before it expired fetch it again so the cache is kept hot.

If you give yourself a tight target like a "1 second homepage" then you may see 700ms wasted in DNS resolving. If that's the case give the google DNS server a go. its available on 8.8.8.8 and 8.8.4.4 its quite shocking the difference it can make.

Stuart

No liars please, we're testers.

2009-10-13T18:59:00.000+01:00

I’ve been prompted to write this after sifting through another load of dross CVs. I say another because I was recruiting heavily last year. A recruitment drive that took me twelve months to find just two test analysts.. hold that thought.

Its not that I didn’t get many applicants the first time round I did, hundreds in fact (quite literally) and I duly read and scored every CV personally and feedback too. Not only that, I also had two of my peers review the CVs too. You cant ask for more than that. If two out of three agreed, the decision was final.

The reason it took a year was down to the quality of the CVs. At least half of the CVs we read didn’t state the daily tasks that you would expect a career tester to be doing. Only if you were playing at being a tester would you make that mistake. Unfortunately sometimes the CV would tick all the boxes and we would invite the applicant in for interview. Imagine their horror when they were faced with a simple exercise to test their SQL skills even though on their CV they had waxed lyrical about how they practically spoke to their friends in SQL because they had used it so much they were fluent.

“so I have a table called customer, with the fields ID, Name, Address. How would I get all of the records form the table?” you should see some of the answers. Its often hard for me not to shout “just give up, I know you haven’t got a bloody clue despite what it says on your CV” they muddle on “GET ALL RECORDS FROM TABLE WHERE NAME == CUSTOMERS AND ID….”

Imagine also them recoiling in their seats when I ask them to draw out the V-model and label it.
“What’s that you say, you don’t know how? But here on your CV you state you are a senior test analyst and have and ISEB foundation certificate, how do you not know the V model?” they shuffle their feet and mumble how they studied at home, well not actually studied so much as bought a guide on how to pass that’s full of example questions.

I watch in wonder as their faces contort when I ask them “so what is the difference between white box testing and black box” I let them fumble through telling me how they have used both of those “methodologies” and I follow up with “can you give me some examples where you white box tested?”

Then come the questions on web testing (its what we do after all) “so whats a cookie I ask”, the smile, easy they think “it’s a virus you get from visiting sex sites” oh my! What should I do as a tester? “never ever accept cookies, they track all your movements, like little spys in your computer”.
I follow with a simple exercise about shopping carts and sessions to see if the candidate understands why a cookie maybe important here. “the system gets all that info from the cookie” but how did it get in the cookie? “from the internet” can I see it, I want to see my cookie “oh no you cant see them, they are secret”.

I have even had to terminate several interviews because it became apparent very quickly that the applicant in the chair didn’t actually know what was on their CV because they had just copied it off a (delete as appropriate) friend / colleague / LinkedIn profile.

It was so bad in the past that we setup an online quiz. It was very simple, multiple choice, some questions around testing, some around our domain. Some very easy questions “Which of the following is a search engine” with an obvious answer “google”. We discovered a side effect of an easy question like that is that we could see how fast the candidate answered the question when they knew it straight off the cuff (about 9 seconds for that question) and compare it to a testing related question that took 3 them minutes to answer (did they have to search for the answer?). The test was very easy for a career web tester, but not so easy for an IT support person or BA or even a developer who fancied a move into testing. Its only real purpose was to filter out the complete time wasters.

So here I am again, I’m hiring and I’m inundated with CVs and again 50% are pure wasted bandwidth (I don’t give them the luxury of printing them out). But this time I don’t have the online test, and I’m gnashing my teeth at some of the incredulous stuff in these CVS. Some of them read like horrible blog postings “on this job we had this challenge and so we had to X because of Y but then Z happened and so we used plan A…” blah blah blah “then the business wanted B but I wrote the very detailed spec of C” bletch grrr spit pfftt, its all I can do to stop myself posting these fetid monologues online for no other reason than ridicule and I hate myself for it.

So faced with the prospect of interviewing a load of (lets be blunt here) bullshit artists only to show them the door at the end of it isn’t a prospect I’m overjoyed with. I don’t want to spend two hours of my life demonstrating why the candidate is a liar. I don’t want to be associated with these bottom feeders in a professional sense either. I loathe them, and I loathe the arseholes that gave them a “consultant” badge at Logica (or any other faceless body shop), because now they think they are gods gift and we should roll out the red carpet for them.

I will continue to sift through the dross, the cream always floats and that’s what I’m after, the cream, the crème de la crème.

So if you are interviewing a tester that tells you that you gave them a much easier ride than in a previous interview they attended, you know I rejected them, and that you may want to make use of that probationary period.

But if you’re a testers who’s CV isn’t straight up and down, you may want to rethink applying for a job with me.

Oh and by the way, don’t put “I have a keen eye for attention to detail” then litter your CV with spelling mistakes poor grammar and mixed styling!

Hokey Cokey or Hocus Pocus

2009-10-12T23:27:00.002+01:00

Back in September 2007 we released a new version of our search application.
The new version was step change for us. At that time we were powering the core of our search offering with an Oracle database, and a Java application that returned flat html. It was all very web 1.0, and we had begun to see issues with the performance of the site and discovered that throwing 8 more servers into the oracle grid didn’t gives 8 x more power. We took the oracle database out of the mix and brought in Endeca search.

The Endeca API allowed us to show the visitors how many of the things they were searching for were available before they submitted the search from. For example, if you were searching for a BMW 5-Series, the fuel type drop down on the search form would list the number available next to the drop down [Petrol (5), LPG (2)]. So a big change to the “build your search and submit it and hope it returns results” model we had previously used. To be able to allow this feature to work we had to use Ajax, or more specifically JSon. So as the user changed their criterion the relevant drop down were updated without refreshing the form. So like I said a step change for the front end, the back end and user behaviour.

The new version was released in stages, inviting visitors to the site to try the new version. This tactic has its own associated problems (for example only a certain type of person will follow a “try our new X” link, so your new application doesn’t get exposure to a good representation of your audience), once the visitors had interacted with the new search form, we invited them to give us some feedback, so that we could improve on what we had we had done. Below is a selection of that feedback:

It crashed 5 times and slow.
Takes longer, too complicated, should be kept simple
Too slow!!
Not as easy to use
Very slow to bring up menus, Spent time waiting.
It doesn't work – my searches keep spontaneously disapearing (Cars)
is slow. maybe is my broadband problem.
I don't want to know how many cars in the uk, I just want to know how many in my local area
It's silly to have to click 'show your results' it was better on the previous version where it showed the results.
Too slow in uploads.
More criteria = more time.
Too many details to put in .
More options, as not 100% encyclopaedic knowledge of cars, the sub model option was difficult .

So, pretty damming stuff. But something didn’t make any sense. We had rigorously tested the performance of the system and were confident that it was faster than the old system. The Market leading browse back then was IE6 and given that we had engineered or built it for IE6 it positively flew in Opera or FireFox. So we were perplexed. That is until we did some usability testing (I wont discuss the fact that the usability testing was too late in the project to be really beneficial).

The usability testing did allow us to understand why we got so many “slow” comments on the feedback. Faced with all the feedback the new search form gave the users as they refined their search, they believed two things, 1. That they had to fill in all of the options and 2, that they couldn’t interact with the form until the animated counters stopped moving.

Manufacturer, Model, Variant, Trim, Colour, Fuel Type, Mileage, Age, Min Price, Max Price, Distance from the visitor. As the user slowly changed each of the drop down controls on the search form, some options would become unavailable (greyed out). This was because the back end had contracted them out of the possible results. If no Red BMWs were available, the colour Red would not be available for choice on the drop down control. So the user would change say model to 3-Series and find there wasn’t any Red available on the drop down, so they would back up and change 3-Series to 5-Series and so on. They didn’t realise you could just search for all the Red cars within 20 miles of their house, and drill down from there. To some extent they still don’t 2 years on.

It reminds me a little bit of when I was working on a project with BT and the then new System-X exchanges. The switches could support loads of (then) new features (things we take for granted today, like 1471 in the UK). Being a geek I was amazed at what I could do with a DTMF (touch tone) phone, and went out immediately and bought one. The next day I asked why BT hadn’t publicised any of the features and capabilities. Their response was immediate, dry and serious. “Our users won’t understand them”. I can still remember how I felt, almost like I had stumbled into some great conspiracy. BT wanted to keep people in the dark, and protect them from the nasty technology that might confuse them.
It was several years later that I received a booklet with my phone bill that explained the features and how to access them. Having used the features for sometime at hat point, I had great difficulty in understanding the booklet. Maybe BT were right, maybe it was all too confusing.

Fast forward to now, and my current project. Again another release and another step change. This time the look and feel of the site has been overhauled. The backend is still Endeca powered, but the Java app has been completely rewritten. And in the rewriting of the application we have taken the opportunity to bake testing in from the start. The JavaScript, cascading style sheets and html are all tested automatically. Regression should be a thing of the past (but that’s another blog post), the application has unit testing, functional and non functional testing applied at every check in. The functional testing has been expanded into “User Journey” testing, in which likely user scenarios are played out. All of this happens automatically before the application reaches QA. Then the QA team goes to town, with time to exercise their real skill, exploratory testing. So there you have it, never in the history of our company has a product been so well inspected. So we felt pretty confident when we were ready for Beta.

This time round, instead of us inviting user to try out our new site, we employed AB testing. 5% of our traffic was diverted to the new site, and once again, users were invited to leave feedback. I took the opportunity to setup a Google alert for spotting the use of the beta url in forum or blog posts, so I could keep track of what the community was saying.
Once again the feedback came in…

The used car search, the old one is much clearer to use and a lot better, . The new "improved" one is poor.
Preffered old site looked more proffesional and was easier to use.
The search criteria should be your main focus and keep that in a clear box format like your old site and allow people to search quickly but also as specifically as they want.
The old site is much better the new site is more complicated to use in the end I shut it down and went on to ebay please change back.
It looks much better than the previous website, but since I dont live in UK, I usually have to copy and paste the London postcode from the FAQ page. Unfortunately, I cannot find the page.
Bad design. Not as easy to use and selct options, not as clear and concise. the old one was perfect.

Erm, what? The old one was better? Perfect? Now we are confused.

So again we tackle the perceived issues of our users. We keep seeing comments of missing images, and we start pulling apart the application, the infrastructure and network. It turns out it was an Ad Blocker, that has decided that the way we format our image urls (cache busting) means they are adverts and blocks them.
People complain of slow loading time, so I begin to conduct some testing around that. I conclude they maybe right, so we engage with Gomez to find out for sure. Gomez shows something alarming. Users on a half decent (2Mb and above) broadband connection will get a decent experience. People on anything less are going to be pulling their hair out. The digital Britain report suggests that most of the UK has 3Mb broadband, so do our users just have slow connections? Regardless I have begun some work into improving the perceived page load times, and will roll those requirements into cross cutting requirements in the same way as we do for SEO and DDA compliance. We are going to lighten the page weight and strip out the heavy JQuery that is only used to titillate. We are going to build our own analytics into the front end that will allow us to see in real time what the users experience (current render times etc), we are moving some of the content so that it resides under a new host allowing it to be fetched in parallel by the browser. All of this should help the usrs with slow connections.

But what about the “it crashes my browser” comment? Our in page analytics trap JavaScript errors and reports them. And while our users suffer at the hands of errant JavaScript squirted into the page by 3rd party advert traffickers our own code is solid, so what’s this crash?

We contacted a customer who had left his details and asked him if he could walk us through the “crash” and we would follow along step by step in the office. At the point where he claimed his browser had crashed, we were watching the light box “melt away”, something we had designed in. His expectation was that the light box would work like a tab, and that he could tab between the photos and the detailed specification of the vehicle. Not melt away to the bottom of the screen. So now we will remove the animations on the light boxes (and other objects).

What have I learnt?

Three things:

1. Next project, I’m running the usability testing, with real scenarios and everything.
2. Perceived performance is more damaging than actual performance.
3. BT may have been right...

How will we preserve Twitter, Facebook or LinkedIn?

2009-09-18T09:52:00.003+01:00

On Tuesday i attended a talk by Doron Swade at the MOSI for the Computer Conservation Society.

Doron Swade is an engineer, historian, and museum professional, internationally recognized as the authority on the life and work of Charles Babbage, the 19th-century English mathematician and computer pioneer. He was Senior Curator of Computing at the Science Museum, London, for fourteen years and during this time he masterminded the eighteen-year construction of the first Babbage Calculating Engine built to original 19th-century designs. The Engine, was completed in 2002.

Doron was talking about the historical and cultural issues in the history of computing that he faced at the Computer History Museum in Silicon Valley, California. When something struck me.

The big kids on the block today are not the computers but the programmes they run, the software. There hasn't been any significant advancement in computing hardware for some time. However the internet is changing the way we communicate and socialise and somehow we will need to preserve it for historical interest.

But how on earth will we preserve software like Google, Twitter or FaceBook or MySpace? Because the software that powers these sites is only part of the puzzle, because with these sites its the content that makes these sites what they are. Terabytes of user generated content. How can we preserve that so that in 60 years we can look back with the same fondness we look back at the Manchester Baby, the UNIVAC or the IBM360??

Once you have wrapped your head round that task, who will test such a system? and how will the ensure that its a true representation of what those sites look like today?

While i sat there among some of the early pioneers of British computing who were gently dozing off i wondered if one day, i would be sat in that room, while tomorrows Doron tells me the problems faced with ranking the websites, facebook before twitter, or linkedin before Myspace?

2009-09-09T00:02:00.001+01:00

09:17 I'm attending BarCampManchester2 -- bcman2.eventbrite.com/?ref=estw #

Automatically shipped by LoudTwitter

If you are not thinking about perfomrance, you are not thinking

2009-09-07T21:59:00.005+01:00

Performance testing is a funny old thing, in that whenever the subject comes up, people get all hot and bothered about it. The thing that really tickles my fancy is when developers suddenly get righteous about testing!

Testers and developers have a totally different view of the world. The best testers i have worked with have a real need to dig into systems. Even with black box testing they find a way to work out what a systems does way beyond its simple inputs and outputs. They cant help themselves. It is almost like they cant pass go if they don't break the system, almost an addiction (or is that affliction).

Now that the developers find themselves writing unit tests, integration tests and acceptance tests they think that overnight they have learnt everything there is to know about testing, right? Wrong!

Yes, sure a developer can write a test, but they often struggle with the intent of the test an more so with non functional testing like performance testing. let me shake it down.

Ok, so the business wants to monetise their existing data by presenting it in a new way, for example "Email Alerts", you know the sort of thing. You create a search, and when your criteria are met you get sent an email.

The developer sits down to think about performance testing, and thinks about how the system works. In our example here, the system will fire the searches every night, when the database is relatively quiet so that we don't overload the system during peak hours.

So the developer thinks OK I'll create a load of these "alerts" using SQL inserts and fire up the system and see how fast it can work through them.

They do just that and get back some statistics like, number of threads, amount of memory the JVM consumed, how many connections to the DB were needed, how many searches were executed, how long it took to execute a search, that sort of thing. They call meetings and stroke their chins in a sage like way. The figures look about right.

But in real life the database would never have the alerts inserted into it in that way. Its probable that users would be inserting data at the same time as it was being read out. Also the product isn't likely to go live and have 100% take up over night. Its more probable that the take up would be slower, perhaps taking weeks or months never achieving 100% take up. Old alerts would be expiring and some users would renew those while new ones are being created while some other are being edited (change of email address etc).

The crux of the matter is mindset. The tester sits down and thinks, what could go wrong? What happens if the DB is unavailable at the time the batch job runs? What happens if the DB needs to be taken down for maintenance during a batch run, will the batch pick up where it left off? Can the batch job complete before the off peak period comes to an end? Can the mail server handle the number of emails to be sent? What happens to email that bounces? In other words the tester takes a step back and looks at the system holistically. Because a user doesn't give a damn if your search engine can execute a query in 33ms if they don't get the email until 12hours after it was relevant.

Now on the current project we have completely rewritten the platform. New version of Java, new styles of writing code, new infrastructure etc etc. The search engine technology is the same, however during the life of the project the API has been updated and a particular feature enabled. This feature allows us to search in a related way. Generally speaking it allows us to do the Amazon type thing; "People who search for X also search for Y", but it comes at a cost, it takes longer to return the result set (of course it would its got to do another search under the hood).

Again during "testing" the developers just threw as much load as they could muster at the application. But guess what, now its live the figures they got during testing don't match, not even close.

It isn't like hadn't been bleating on about performance for months. I even printed out posters that read "If you are not thinking about performance, you are not thinking" and stuck them above the urinals in the toilets.

Its only now that the team are in the spotlight (its so bright it burns) that they have come to us to ask for our help. Once again we are trying to polish a rough product instead of building the quality in from the start. Once again we cant.

It doesn't matter a damn that we went all out Agile, TDD and XP if the thing doesn't perform. The end user doesn't care that we have continuous integration, they know its damn slow and they vote with their feet (or keyboards/mouse).

Skills not Roles - Communities not Teams.

2009-09-04T22:09:00.007+01:00

When i decided to put this blog together, part of the impetus was using it as a historical repository. There are a great many of my posts on the internet today that date back to 1999. Ten years is a fairly long time in internet years, and looking back over those posts i can see how my understanding of different topics has evolved and how much I've learnt and grown. That said, its not going to work (a historical record) if i don't post anything is it!

The reason i haven't posted for quite some time is two fold. Simply put I've been too busy at work and too busy at home. That is to say, the blog has had to take a back seat. I'm sorry I'll try harder.

OK on to the post proper.

So this post will differ in that i'm not going to discuss technologies like Twist or Selenium or WebDriver.

I want to talk about something that is currently crippling the project I'm currently working on.

For a project to be truly Agile and Lean it needs to be able to respond to the challenges that we face daily in IT and overcome the challenges without waste. So why then do we have to hand off tasks like deployments to another non Agile team? Moreover the handover is done via an abhorrent "work-flow" tool that admonishes the receiving team from any aspect of quality or responsibility "I've done my bit mate, its with team X now".

I want to get rid of teams as we know and recognise them today and usher in communities. Yeah sure the name is a bit hippy-ish but then so is the ideal. Skills not roles. If someone within our delivery community has the necessary skills to deploy some code to a database or server then why do we have to interface with an external team? If we have the capability, and we are responsible, what's the problem?

OK, sure, the guys who look after the production systems want to achieve 99.999% uptime (26 seconds of downtime a month), and often they are targeted on this so they become averse to change. After all any change increases the risk of a failure. However, if we have tested the code, not once, not twice, but umpteen times and more importantly we only compiled the code only once, and all previous deployments have gone without incident. You could be forgiven for thinking that the deployment could be considered as being safe for deployment. A non event. We should be able to deploy the code at 17:30 on a Friday afternoon and skip off home safe in the knowledge that the site is up and running, humming along like a well oiled machine.

However, those teams have become so averse to change or risk as they perceive it, that they actually start to display behaviours reminiscent of the 1970 trade union shenanigans that plagued British industry "you can't do that mate, not your job. Not anyone can deploy code you know, oh no. where would we be if just any old tom, dick or harry could deploy code willy nilly".

As an Agile commune focused on delivery of our project we would share the skills and socialise ideas. We need to create innovative environments that promote people trying new things. This aids the members of the community which therefore benefits the business. So not anyone could deploy the code. Only those people that had the skills and were responsible in the execution of their duties.

The more i think about this, the clearer it all becomes. I suddenly find myself questioning my own role as a "people manager" within such a community. After all my role as in its current shape would be wasteful. I should not manage the team (to be honest, that's not my natural style) I should coach and mentor, not preach and target. I should lead by example, not autocratic rule. As Alan Keith of Genentech said "Leadership is ultimately about creating a way for people to contribute to making something extraordinary happen."

I've run this idea past my peers. Old peers agree with me, they see it as a way to empower individuals, and therefore the community they reside in. But younger, less wise peers are worried. "How will we administer pay grades" they ask, "how will we hire people, if we don't have recognised roles".

Its really quite simple. Individuals are rewarded for the skills they have not their ranking within a role. Why should an experienced tester with polygot skills and several years domain knowledge be paid less than a BA with flaky knowledge of your technology platform? What because business analysts traditionally earn more than testers? For that matter why should a developer be paid more than a business analyst if the BA can also test? Two skills vs one. The current game is rigged and is demotivational.

Hiring is also easy. You want titles for your people? Call them analysts. Then all you need to do is hire analysts with the appropriate skills for your domain and your platform. Other companies call their staff "consultants", and they hire consultants with the appropriate skills for the client they engage with.

Once you have a pool of multi skilled analysts, it would be easier to create a community that had the right skills required to deliver the project, instead of worrying about the interfaces to external teams or having a shortfall of a particular discipline within your community. You can select your community members based on their proven experience, their skills, their domain knowledge and the feedback they receive from the communities they have previously worked in. A community is unlikely to carry a lazy person who knows little about the domain and has few or poor skills.

Now i'm not saying we don't need SysAdmin or DBA or networks etc. We need all those teams, and what they do is invaluable to the delivery of the project. But do those external teams need to perform what amounts to mundane tasks for us? Shouldn't they concentrate on what's important to them, the stability and performance of their area. Because as it stands today at the point we interface with those external teams for the execution of a task that could be carried out in-team by an appropriately skilled and responsible analyst, they become a blocker and they become wasteful. We sit twiddling our thumbs while we wait for those teams to follow their internal processes and use the infernal work-flow tool (and its only real purpose is to provide the business with yet more meaningless statistics).

When i have approached the external teams, i have found that they harbour a fear of "Agile" and i think this fear is the real problem. They feel uncomfortable, anxious, or inadequate. They worry that by allowing us to be responsible for our actions then they will be allowing themselves to be exploited and by denying us it helps protect their rights as an individual.

The business feels the same way. Despite the corporate line being "we are Agile, lean, innovative..." they fear change to the point that they have implemented a change management process, and a change manager and recently said we have to use sharepoint (ffs). But what the business hasn't realised is that as a tester i am risk averse (no really its a curse), so we are actively baking the quality into our products and continuously inspecting them for that quality - through unit testing, integration testing, and acceptance testing.

It will be incredibly hard for the external teams let go, especially while the business is frozen with fear, but in the future they will have to. They will have to or we may as well pack up and go back to PrinceII - not while there is breath left in my body...

Life after ThoughtWorks?

2009-03-30T17:47:00.007+01:00

After working with ThoughtWorks I’m completely sold on Agile software development for the type of product we deliver. Which if you know me may be a bit of a surprise as I have a reputation for being a cynical old curmudgeon, and a staunch doubter to boot.

While I am still unsure if agile would work for large scale software projects (think air traffic control) I am certain that there are agile practices (lean for example) that could be pilfered and applied to those lumbering projects too. If you are reading this and you haven’t had any exposure to Agile, I guess the biggest thing I should tell you is that this isn’t some process you just pick up and run with, nor is it a methodology to apply. You can’t learn it from a book, or by a project template; it is quite simply a mindset and as such it presents the practitioner with quite a shift in paradigm. Simply put, you can’t flick a big agile switch and hey presto you are Agile; it is more subtle than that.

I think before I continue it’s important that I clarify something. The term agile is applied to a wide variety of processes, techniques, methods, tools, practices, projects, and phases of the development life cycle; it has become a buzzword used by people trying to paint their work in a new light (who doesn't want to be known as being agile?). It's important, therefore, to set out some basic definitions and context for the use of the term "agile," especially as i will use it constantly throughout this article.

Within the context of software development, the term "agile" (with a small "a") is meant to imply that the development team is nimble, flexible and responsive to the business needs, and that it is able to adopt new technologies and techniques that can improve software delivery. The term "Agile" (with a capital "A") refers to a very specific set of processes (and i use the term process as more of a place holder) applied to software development that have evolved over the past fifteen years or so; including some you have probably heard of, like eXtreme Programming (XP), Scrum, Feature-driven Development (FDD), Crystal, Dynamic Systems Development Method (DSDM) and Lean Software Development. A non-profit organisation The Agile Alliance was created by the ideas people behind most of the Agile processes. The Agile Alliance promotes a set of core values that a process must follow to be called Agile:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

So to be Agile then, a process must support these values (and more), albeit in diverse ways. Some, processes for example like Scrum, address team management, while others such as XP or DSDM, address development activities or other activities of the software development life cycle. It’s worth making a mental note that users of Agile processes do not have to follow all its Agile practices, and neither does the use of one process preclude the use of any other. One thing I learnt is that Agile supports such process change, if a particular way of working is not working then change. In fact many I found that many of the Agile practices are complimentary.

No matter your preference, all of the different flavours of Agile will deliver working functionality in short time-boxed iterations. They implement early and frequent testing. They require lots of involvement from the customer on a frequent if not full-time basis and they assume that the customers requirements will continually change.

So why all the fuss about big "A" little "a" when talking about Agile? There are two main reasons:

Companies who adopt Agile processes have to be prepared to completely change the way they not only develop software but how they think, and this change is so big it is very hard to do. However companies that have achieved this shift will appreciate the significant benefits they can reap in productivity, quality and value of the software that they deliver. Notice i didn't say it would be faster. These companies are Agile with a capital A.

Companies that are not able to embark on this level of change can still become more reactive and flexible in the ways that they build software. They become more agile and begin to realise the advantage that Agile can deliver. This toe dipping exercise can lead to a true Agile team, however the company must understand that it's not the prefferable way, and they would benefit from the gung-ho approach to Agile adoption.

To reiterate what i've said, at a minimum, every Agile process delivers working functionality in short, time-boxed iterations. Agile implements early and frequent testing, and it involves the customer on a frequent if not full-time basis; and assumes that requirements cannot be fully defined at the start of a project, and that the requirements will continually change. The ethos is simple that by using these practices, the development teams will be able to respond quickly to the ever changing customer priorities and feedback, and deliver value to the business. This is often missunderstood as being quicker, I would prefer to describe it as improved time to benefits.

As a tester I have become quite accustomed to being involved late in a project, often right before delivery. I have adapted and learnt how to cope with shortened time frames for testing, and receiving specifications and requirement documentation that don’t actually match the product delivered in QA. Typically the business (or for that matter the project team) usually has little interest in the test teams input, that is until they call us a bottle neck.

Feeling loved and appreciated.

Agile software development for a tester is radically different from the traditional PRINCE(2)/waterfall software development lifecycle (SDLC), because it throws QA right into the heart of project on day one. As Agile testers we suddenly found ourselves being involved in the analysis and design of the product. We became heavily involved in decision-making throughout the whole project and because the delivery of the software is incremental we found that we began testing at the very beginning of the project and that we had to maintain pace and keep step with development to prevent any delay. This is a far cry from waiting for a software deliverable (possibly unfit for purpose) to be thrown over the fence a few weeks before go live.

The paradigm shift I have mentioned is a large one, not just for QA but the whole project team, because our QA team now drives the entire software development process.

Quality assurance, with its focus on preventing defects, is translated into the agile practice of having committed QA resources on the development team that participate in decision-making on a daily basis, throughout the life cycle of the project.

Their input during elaboration and design helps developers write better code. More “what-if” scenarios are considered and planned for, as the collaboration between coders and testers gives the coders more insights than if they were to have planned the work on their own.
Likewise, the testers gain added insight into the expected functionality from the coders and the product owner, and are able to write more effective test cases for the product.
Quality control places its emphasis on finding defects that have already slipped into the system, and working with developers to eliminate those defects. This bug-checking is done within the iteration, using such techniques such as daily builds and smoke tests, automated regression testing, unit testing, functional and exploratory testing, and acceptance testing. Everyone participates – no one is exempt from the tasks of ensuring that the feature coded meets the customer’s expectations.

Your role morphs and evolves.

Through a methodology known as "Story Test Driven Development" the test requirements (aka the acceptance tests) are captured (we currently use Twist) in a test like format and they are then augmented to make them into automated tests. Nothing unusual in that, lots of test teams create automated tests, however here the automated tests are being executed by the development team not the test team and the tests exist before the development team have even created the code for the software the team is delivering. The real beauty of this method is that the development team can then integrate the automated tests into a Continuous Integration environment, where the newly checked in code is built and tested automatically. So the QA teams test suite is run against the code every time a change gets checked in. But wait, that’s not all. The delivery of code into QA can not happen until it has passed all of our tests, which means that we have built the quality in from the very start. Let me state that again, in case you missed it. The development team can not deliver code into QA until it has passed the QA teams tests. That’s a statement that should raise a lot of internal debate with any passionate tester who hasn’t worked in this way before, and it did with us. Its worth me making the distinction here that Test Driven Development (TDD) is not a testing methodology it’s a design methodology. Being testers we have a very different view of the world to a developer, its in very our nature. Working this way felt like we had harnessed our power and put it where it belongs, under the smelting pot of code.
I should also mention here that the developers were also working in a new way, because they worked in pairs (pair programming) and also used TDD at unit level. This means that a developer has to write a unit test before he writes the code for the unit being developed. That means then, that before its delivered to QA, our code has been tested twice. That’s two times tested.

So if development is running the QA teams tests, and the QA team are writing the automated tests (and its blurry line between coding and writing an automated test), and the code is tested before its delivered to QA, a whole load of question begin to bubble up.

Some of early questions we had:

Who tests the testers’ tests?
This is agile, so what happens if the requirements change?
So what do QA test if it’s already passed the QA tests?

On a typical PRINCE(2)/waterfall project the testing team plans as far in advance as it can. We normally try to follow the IEEE 829-1983 standard and document as much as we can.
The documentation covers how we approach the testing and detail the testing activities. These documents are usually created in isolation from the project team and may be published for approval. However in my experience the documents are rarely scrutinised and any feedback given only serves to pay lip service to the processes. Another checkbox checked, and another Gant chart milestone met.

Working in an agile testing environment also has a requirement to define which tools and methods will be used for writing, executing, and reporting tests and determine the best approach to testing, and the scope of that testing. The big difference is that the whole team is engaged in this deterministic process and we found that it was important to engage the developers in this definition, because they would be executing our tests and writing their own unit tests. Moreover thought had to be given to automating the regression testing, something that would happen as part of the continuous integration process.
The business stakeholders were also involved in this process (unfortunately only by proxy through the business analysts, as they are physically located in a different part of the country) as they would help to define and run the acceptance tests. In agile, we (the whole team) all test, but the business accepts.

In short within agile practices, everyone has a contributory part in defining, upholding, and improving the quality of the product.

One of the gotchas was that we found that we needed to become more technical. We had thought that we were already more technically skilled than your average tester, however as we endeavoured to automate our testing we found that we had to skill up and learn not just how to write code (Java for us), but compile the code and version control it. They were steep, steep learning curves, which now we have overcome them, have empowered us, giving us the tools for a brighter, faster and more accurate future. Having said that, it does appear that some of this coding effort could be considered a one off setup task, because we now have a framework of “tools” that cover all of the tasks we need to help us execute an automated test. Couldn’t we have asked the development team to do the tech tasks for us? We did, and they didn’t have any resource free to accommodate our needs as well as those of the agile project. Regardless, we have skills now that we have been able to share with the wider QA team, and they too are seeing the fruits of our initial labour.

Traditional Tools Solve Traditional Problems in Traditional Contexts. Agile Is Not Traditional.

Traditional, heavyweight, record-and-playback tools (like Quality Center) address the challenges faced by teams operating in a traditional context with specialisms. They address the challenge of having non-programmers automate tests by having record-and-playback features, a simplified editing environment, and a simplified programming language.

But Agile teams don’t need tools like these (optimised for non-programmers). What Agile test teams need are tools to solve an entirely different set of challenges that are related to collaborating, communicating, reducing the waste (Muda), and decrease the feedback loop. Ergo, traditional (long standing) test automation tools just don’t cut the mustard in an Agile context because they are designed to solve traditional problems, in traditional contexts and those really are quite different to the challenges faced by Agile test teams. To make it clear, QC & TD isn't going to cut it in Agile.

At the Google Tech Talks December 9, 2005 Elisabeth Hendrickson gave a talk on how as more teams are adopting Agile practices such as XP and Scrum, software testing teams are being asked to become "Agile" as well... View it here

Using Twist With Different Selenium Versions

2009-03-25T17:30:00.003Z

When you start using Twist you will find that the twist team have baked selenium into twist.
You will also find that the version of selenium that has been integrated has some quirks.

Here is a simple guide for using any Selenium version with Twist.

You can also use this guide to help you implement other drivers like WebDriver.

Step 1
Add the selenium-java-client-driver.jar and the selenium-server.jar of your preferred selenium release (I'm going to be using my old friend selenium.0.9.2) to the Twist project class path, make sure these jars are loaded first in the class path order.

Step 2
You need to create a factory class that will create the Selenium instance for your test suite.

For example, let’s create "SeleniumFactory.java" in the Twist source folder.

import java.io.FileInputStream; import java.util.Properties;
import org.apache.tools.ant.types.Commandline;
import org.openqa.selenium.server.RemoteControlConfiguration;
import org.openqa.selenium.server.SeleniumServer;
import org.openqa.selenium.server.cli.RemoteControlLauncher;
import com.thoughtworks.selenium.DefaultSelenium;
import com.thoughtworks.selenium.Selenium;

public class SeleniumFactory { private SeleniumServer server; private Selenium selenium; }

public void start() {
try {
Properties properties = new Properties();
properties.load(new FileInputStream(getClass().getClassLoader().getResource("twist.properties").getFile()));
}

String[] serverOptions = Commandline.translateCommandline((String) properties.get("selenium.server.options"));
RemoteControlConfiguration serverConfiguration = RemoteControlLauncher.parseLauncherOptions(serverOptions);
String browserLauncher = (String) properties.get("selenium.browserLauncher");
String browserURL = (String) properties.get("selenium.browserURL");

server = new SeleniumServer(serverConfiguration);
server.start();

selenium = new DefaultSelenium("localhost", serverConfiguration.getPort(), browserLauncher, browserURL);
selenium.start();

} catch (Exception e) {
throw new RuntimeException(e);
}

public void stop() {
try {
if (selenium != null) {
selenium.stop();
}
} finally {
if (server != null) {
server.stop();
}
}
}

public Selenium geSelenium() {
return selenium;
}

Step 3
We now need to remove any bean definitions with id="seleniumFactory" and id="selenium" from the applicationContext-suite.xml file.
You will have to manually edit, the "applicationContext-suite.xml" of the twist project, and add the following bean definitions.

Step 4
The workflows will now have to depend on the Selenium interface.

For example,
public NewWorkflow(Selenium selenium) {
this.selenium = selenium;
}

With the above changes, your scenarios will be now use the selenium server and driver jars included in “step a”.

Note:

The selenium options are not the same between releases. The beta-2 release of Selenium RC contains some significant changes you should be aware of. You will need to account for these and change these values in "twist.properties" as required.

For example:

selenium 0.9.2 config : selenium.browserLauncher = *firefox
selenium 1.o Beta 2 config : selenium.browserLauncher = *firefoxproxy

selenium 0.9.2 config : selenium.server.options = -port 4545 -avoidProxy
selenium 1.o Beta 2 config : selenium.server.options = -port 4545 -avoidProxy -honor-system-proxy -singleWindow

For a full list of the changes take a look at http://clearspace.openqa.org/community/selenium/blog/2009/01/13/selenium-rc-beta-2-goodies-and-gotchas

Choosing Twist - Part two

2009-03-23T18:11:00.011Z

The next Big Thing

Enter our next big project. We (the QA team) held a huddle to talk about the capability of Fit and FitNesse for the new project. We all agreed that we would sooner avoid it if at all possible, in fact the language used during the meeting was much stronger that that. It was around that time I was shown a demo install of Twist, and immediately asked the question to the guy showing me “what is the problem I have with FitNesse that Twist will resolve?”, he looked at me as if I was crazy then reeled off all of the pain points that we knew and then some.

It’s worth having a look I thought, and installed the 30 day trial. One of the guys who had been very instrumental in fighting the FitNesse issues also installed it, and we said we would take a look at it and then compare notes.

At first we didn’t get it, but some things are very apparent with Twist.

For those of you that don’t know, Twist is built on top of the eclipse IDE, I should mention here (for existing Eclipse users), that it’s not currently available as a plug-in for an existing eclipse install (however that feature is in the programme plan). That means it comes with the entire feature set that eclipse does, e.g. integration with source control (CVS or SVN for example) albeit via a plug-in. The team write their tests, run them locally to make sure they work, and commit.

Search
You can search across your code, files or project. Search and replace across an open file or the whole project.

Keyword completion.

This last feature is biggie for us. If you create a method call UserOpensUrl then the next time you type v-e-r and press Ctrl+Space it pops up a list of matching methods.

straight away people can see what methods that have the phrase “ver” in them have already been written.

If you have used eclipse before, then all the usual features and short cut keys you love are available (I’m still learning new ones every day).

But what about twist? What does that bring to the party?

The most striking thing is the WYSYWIG scenario editor.

If you have grappled with fitnesse then the ability to make text Bold and Italic with the click of a button is great.

Not having to remember any mark-up language allows you to concentrate on the task in hand, writing tests.

The scenarios are broken down into several areas. The first part is where you write your test prose, the purpose or intent of the test. The second part is where the test proper is written.

Tests are currently written using either bullet points or tables.

And just like fitnesse a passed test turns green.

The stats for all the tests are given in a separate panel

The scenario editor allows a scenario to have one or more tags associated with it. These can be tags such as “QA Complete” or “shopping cart” or “smoke” or all three or none. These tags come into their own when you only want to run particular types of test. E.g. if you only want to execute tests that are QA Complete and are for the Shopping Cart area of your site; this is especially useful for CI builds.

Tags can be applied at a scenario level (while you are editing a scenario) or then can be applied or removed en masse.

When you have written your tests they need to be instrumented.
This can be achieved in a number of ways. Either right click and select Quick Fix from the context menu

Or use the quick-key combination of Ctrl+1 and you are presented with another context menu

“Create method” simply creates an appropriately named empty method ready for you to work on.

Again this is great where you don’t have development skills within your team as your team can focus on writing the scenarios and the dev team work on instrumenting the empty methods.

Choosing Record from the context menu starts a selenium server (which only currently supports Firefox) and fires up firefox. The record system differs a liitle bit in that it first runs though any previous steps in test if you have written any, then begins recording at the new point in your test, and completes the method.

I have only touched on a few good points here, those which counter the main pain points we had with Fitnesse. One of the communities biggest gripes with Twist is the lack of good documentation. Which when you consider it is a paid for product is a little shameful.

i should also point out that while i was writing this post, ThoughtWorks Studios release the first GA version of Twist, which i have yet to try out....

Ferrograph SDX - scrolling LED sign

2009-03-23T16:34:00.003Z

One important aspect of being Agile is have short feedback loops. Wherever possible we aspire to reduce the feedback times, and one aid in decreasing feedback time is a feedback monitor. Having a some sort of visual indicator for the state of the project is obviously valuable. Its believed that if used correctly it can boost productivity and morale within the team considerably, however there is a certain amount of Kudos to be gained for having anything more than a VGA screen.

If you have a quick Google for Extreme Feedback Devices (no i'm not talking about a Marshall JCM800, 4x12" cab and a Gibson Les Paul) will turn up a large amount of hits for glowing orbs, VGA screens, lava lamps, lots of bespoke devices powered by Atmel AVRs, devices controlled by X10, water features, gummi bears and so on. One post by Dirk Ziegelmeier caught my eye, he had used a scrolling LED message board.

I picked up an old Ferrograph Aurora 64 SDX moving message / LED wall board, the type that is typically used in a call centre environment. In fact our call centres employ these boards to show the inbound call queue, and average call waiting times etc. for the Avaya Index ACD system.

The Ferrograph SDX boards turn up quite frequently on Ebay, and i had (somewhat naively) assumed that there would be enough info on the internet for me to be able to integrate the sign into our build monitor. Like i say, i had assumed, and as Eric Bogosian put it so eloquently in "Under Siege 2: Dark Territory (1995)" , "assumption is the mother of all f**ck ups".

The sign came with a short pig tail of cat5 cable terminated in an RJ45 plug. I have seen these signs listed on eBay as being "networked", this is an incorrect assumption being made by the seller as this is not a network cable but a serial cable. In my case the sign had been configured for RS422. Being a hardware geek more than a software geek, meant i had the sign opened up and running on RS232 in a matter of minutes.

Inside the sign is a main-board, at the end of which there is an RS422 header and RS232 header. Although there didn't seem to be a pin out that i could see printed on the PCB i traced back a little bit to work out which pin was which (a guesstimate based on where the pins ended up), i soldered up a 9pin D-connector to three wires (gnd, Rx, Tx) and stuffed it into the back of my PC. I took a guess at 9-n-1 (as its fairly standard HW setting) and sent "hello world" using the Alpha 1-byte protocol. The sign displayed nothing, nada, zip, bupkis, diddly-squat, zilch, bugger.

A long hard trawl of the internet turned up nothing. A few posts on various "Q&A" sites asking for info on these boards, but little else. A few lunchtimes of hunting and i came across a posting on makezine where Robert Coward said he had successfully reverse engineered the SDX signs.

I dropped Robert a line, and he very generously sent me the secret i needed to know for me to be able to get "hello world" onto the sign.

Because the Ferrograph SDX signs were designed for call centre ACD systems using the Avaya Index telephone system, it expect to be sent the phrase "SDX" at the start of every packet. It was that one nugget of information that allowed me to get going with the SDX wallboard but it was Roberts reverse engineering that really enabled me to push the sign to its limits - which took all of about 5 minutes!

There are many things wrong with the SDX sign, in fact possibly more defects than working features. I'm guessing that when Ferrograph tested the sign they only tested the features that would be needed for the sign to operate with the ACD system and not the features that are typical of the Betabrite or Alpha LED moving message led displays.

Anyhoo, long story short, Robert has created some new firmware ADF for the board, that does the following amazing things:

* Conforms to the publicly available Alpha protocol. It works with ready made applications, and it's easy to write your own applications for it.
* It has lots of fancy effects (snow, dissolve, drop down, cursor wipe and many more), and smooth continual scrolling, as well as all standard wipes/scrolls.
* Supports small/large normal and fancy character fonts; one or two line operation for small fonts. Double wide, flashing and colour shift flashing options available.
* Automatic word fit and centering to ensure a presentable display at all times.
* Many colour combinations, including three colour rainbow and two colour stripes.
* Full support for pictures and animations; configurable animation speed.
* Sophisticated & flexible memory management.
* Full real time and date display support in various formats.
* Ability to set messages to appear and disappear at certain times/days.
* Control of two optically isolated misc IO signals available on some Aurora 63 units, allowing automatic control of external buzzers, lights and even mains appliances (via suitable interface hardware) in synchronisation with message displays.
* Serial readback for message data and system/error status.
* Works on RS232 or RS485 interfaces.
* Serial timeout message option to display an error or blank message if host serial comms fails.
* Automatically detects Z8S180 CPUs on modern units and switches to high speed internally for maximum performance. Also automatically handles boards hard wired to double speed operation.

I'm sure you will agree that its some features list. Robert has turned what was previously a skippable board into a fantastic fully featured LED scrolling message board that would cost thousands of pounds.

A big feature here is that the with the ADF firmware the sign talks Alpha protocol, 1-byte, 2-byte and 3-byte. The Alpha protocol is publicly available and therefore there is much info available on the internet about it including sample code.

For example, in Linux i can:


echo "_01Z00_02A0hello world_04" > /dev/ttyS0

I contacted Robert again and purchased a copy of the firmware off him, which he supplied on an EEPROM ready to drop straight into the mainboard of the sign.
I powered up the sign and put and sent it "hello world" and basked in the 3 colour glow. within minutes a small crowd had gathered around my desk, Ooh-ing and Ahh-ing at the effects. You see if you sent the old SDX firmware "hello world", that's what you got however with Roberts ADF firmware, the sign centres the text and positions it in the middle of the two rows, then applies various effects to the text. The crowd were pleased.

Ok, so know i needed to make this sign work hard, i needed it to parse the cctray.xml that is generated by Cruise. Surely this is something i'd find ready made on the t'interweb? Well yes and no. Lots of people are parsing the xml that is output from Cruise Control, but we are using Cruise, and cctray.xml was what i wanted to parse (at first at least). But that as they say, is another story.

If you have a Ferrograph Aurora (Aurora 62, 63 or 64, new or old hardware type) SDX display / wallboard and you are interested in Roberts firmware you can contact him directly via email: robertcoward{at}gmail{dot}com

Choosing Twist - Part One

2009-02-27T14:57:00.013Z

Choosing Twist - Saying goodbye to Fitnesse

I wont go into the whys and how’s in this post, but my team and i found ourselves grappling with FitNesse. It was a huge undertaking for us. Something of a torrid affair, all love hate and very emotive. But now we have embarked on a new journey of discovery and left FitNesse behind moving to twist. Here in this post, I will hopefully capture why and how of the transition.

Before I go any further I should perhaps lay out my stall. We are a large website operating in the UK. When I say large, I’m talking in terms of the footfall on the site and the revenue generated. Around 500,000,000 page views a month genberated by 10% of the UK population. Traditionally all of our testing had been manual and we were a Prince II-ish development house; aka the waterfall approach.

All of our testers are career testers and should not be confused with TDD developers, or BA’s or UAT-ers. Our testers are plain old fashioned, passionate (ISEB accredited) functional and non functional software testers. To us, testing is our career not a stepping stone to project management.

Our scope was varied. We would test from just above Unit testing all the way up to and including UAT (that is most of the right hand side of the V model). Yes we are testers, but we are technical testers; in that we understand HTTP, and Ajax and CSS and load balancers, edge suites, 3DNS, PL-SQL, Jboss, Apache, J2EE etc etc. Experience has taught us that on the whole we understand a lot more than your average "web tester" . However we don’t do code, ever, never ever.

In the past we have used a number of what could be described as "de-facto" test tools, most notably the Microsoft Office suite (Excel spreadsheets, Word templates and so on). Of course there has been much talk and pitches from prospective Testing Tool vendors over the years, and when my team was first learning how to get Fit with FitNesse another department was seriously evaluating QTP and Axe, more on that later.

We were originally introduced to FitNesse through a project that we had decided we would run using Agile principles (note the big A, we went all out). We had some guidance from Agile partners ThoughtWorks who were helping us find our way in the Agile world, and they said we would need to seriously consider test automation, and suggested a few tools, one of which was FitNesse.

I have made a previous comment that the first time we used FitNesse, we felt like the Apes in the beginning of A Space Odyssey 2001. There we were staring up at this great monolith, with no idea what to do with it, but we knew it was important.

Through some heavy googling we settled on the Webtest fixture by Gojko Adzic. We soon became comfortable with FitNesse and Gojkos fixture. We rapidly begun to out grow the webtest fixture. Before we knew it, the thing we called the webtest fixture contained very little of Gojkos code and had code specific to some of the idiosyncrasies of our website; it had become our DSL. Yes, we were writing code. This was quite scary for us, because who tests the testers tests? However no sooner had we begun to feel very comfortable with writing the fixtures we needed to drive selenium or execute a stored procedure than we started to feel the pain of FitNesse. Here is a short (by no means exhaustive) in no particular order list of our pain points:

Version control
Version control in FitNesse or with FitNesse, is one thing that still sends shudders down my spine. I know it can be done (we achieved it to some extent) but we had little faith in it, and consequently the team would keep local revisions on their machines.

Duplicate code
The code was continually being replicated/duplicated and different testers would write different methods to do the same thing, e.g

The user navigates to <url>

The user Opens <url>

The URL <url> is opened

Open the Bowser and Types <url>

Despite our best efforts to keep an internal dictionary (and an ant task to generate html JavaDocs) we would still find duplicate (or very similar) methods.

Wiki Markup
The wiki mark-up language. I know its fairly straight forward, but when you have to escape words that you need to use in your test so that FitNesse doesn’t try and parse it, it gets overly complicated quickly. I want to write a test quickly, not have to remember the syntax for bold italics.

Metrics
Lack of metrics, seriously is this a test tool? Not having metrics available after the test had run was and still is the Achilles heel of FitNesse.

Setup and Teardown
Setup and Teardown – this seemed great at first, until we discovered that when we wanted to pass parameters around the tests based on an initial value that was discovered in a setup routine we had to jump through hoops. Things are further compounded when you have a large suite that needs multiple setups and teardowns to allow the tests to be independent.

Lack of Documentation
While there is some documentation out there a lot of it confusing. Even the FitNesse website is confusing running inside FitNesse it has pages that lead nowhere.

Tables
We found that because of the nature of the beast we needed to express all of our tests as tables. We learnt how to use Fitlibrary's DoFixture to help us get around that limitation. however, once we had gained a better understanding of the what we wanted to test we would end up back in a table. The downside of this is that we may have wanted a test to read:

"a registered user returns to the site and successfully logs in."

The tabular nature of the test would coerce us into using each row in the table as a step to achieve the goal.

The whole table thing was such a PITA that i looked at using David Peterson's Concordion, it seems like a good alternative because it doesn't have the same dependence on tables, but i didn't want to disappear Alice like down a the rabbit hole.

Throughout this painful period we just kept thinking it’s our fault because we don’t know how to use Fit and FitNesse is just exasperating that. However there became point in time when we realised that we did posses more FitNesse knowledge than we gave ourselves credit for and that realisation galvanised the thought that FitNesse was unsuitable for what we wanted. Ergo, there must be a better way.
In short, we had a real love hat relationship with FitNesse. While FitNesse seemed to present us with more challenges than benefits, the benefits outweighed the effort to make it all work, and to that end, I became very proactive, and begun selling FitNesse to the business at large.

Remember that evaluation of QTP I was talking about? This was partly driven by a new development manager who was eager to influence the department with his experiences of continuous integration; not having any existing automated tests was a non starter for him.
We spent a significant amount of money on some POC work to see if QTP and Odin’s AXE would work for us. It took 5 consultants 1 week to complete that proof if concept and demo their findings, however that work was quickly put to shame by a tester who had never used FitNesse or selenium, but had seen its potential on our project. In one evening at home he was able to create an end to end test for his product, whereas the consultancy team who conducted the POC work (hint: this was their product) were unable to automate more than one fifth of the application in that week, and even then they could only drive Microsoft Internet Explorer. In one fell swoop FitNesse and Selenium saved the department £40,000 and another team adopted FitNesse.

Meanwhile our life with FitNesse was coming to an end, or so we thought. Our toe dipping exercise into the world of Agile was hailed by all as a success, and so it was decreed that from now on it was the Agile way or no way.

End of Part One

Debian - Backup and restore instaled software

2009-02-11T10:05:00.003Z

- This is just a quick post so i can remember how i did something -

So I have taken an old Ferrograph SDX display and put new firmware on it ready to integrate it into an XFD (more on this in a later post).

I have decided to drive the display using Perl as there is a requirement to get RSS and xml and parse it appropriately build the packets for the sign and send them to the serial port of the host PC. I chose Perl because it has all of that stuff ready to go and i can develop on my local pc (windows) and port it to Linux fairly easily.

Anyhoo, i had already built an IRC server using an old junked desktop machine for the teams to use as a communication tool, and i built that on top of Debian etch. I couldn't use the same machine for the XFD because it doesn't have a serial port. I got hold of another junked desktop machine with a serial port and installed ebian etch on there too, and soon realised i couldn't remember what software i had installed on the IRC box. I wanted to keep them in sync.

Backup list of installed software

to find out what software i had installed i used the dpkg command to list installed software:


$ dpkg --get-selections

I then redirected the output into a file to store the list of installed software.


$ dpkg --get-selections > ~/installed-software.log

on the other box i used sftp to fetch the file, and used that to select the software i wanted to install

All i had to do is type following two commands:


# dpkg --set-selections < installed-software.log

Now my list is imported i need to use dselect to install the package.


# dselect

Select 'i' at the menu for install the software, answering Y to the question about additional space.

Job done!

going round the Twist

2009-02-04T16:01:00.002Z

I haven't posted for a while, and that's because we have been up to our ears in Twist.

Twist is a tool made by ThoughtWorks that take a lot of the good ideas from FitNesse and bolts them on top of eclipse.

I cant be too harsh on Twist because it is still in beta, however we have had a lot of pain to deal with.

For one thing, ThoughtWorks appear to have thrown the baby out with the bath water as the supplied selenium appears to be altogether different.

So this is where we are.

We have Twist installed on all the testers machines, and we have installed subclipse into Twist. This takes care of the version control issues we had with fitnesse.

One issue we did run into was getting the Twist project to play nice within the main project Trunk (more on this later).

The guys are writing the prose like business speak heavy tests quickly (the Twist Scenarios) however the instrumentation of those tests has become a little bottle neck, and some dev guys have come on board to help us with that.

What was interesting was seeing the dev guys pulling their hair out over Twist too.

I will post a fuller write up soon that explains how we run the twist tests (they are running as part of the CI build) and how we get reports from those tests.

Stuart.

Something new?

2008-12-11T21:15:00.002Z

I was left this tantalising teaser today "A language and framework agnostic vocabulary for testing web sites", and i've got to say that got my mind ticking.

You can follow Thistles progress over on the social code hub github.

I'll be watching with interest!

Fitnesse - Another DSL

2008-12-10T22:06:00.005Z

So, if you have followed this blog you will know that we we first started on our journey of discovery with FitNesse by trying anubhavas generic fixture but while this idea seems great it has its shortcomings. Now Anubhava did a pretty good job of telling people about his fixture (which is how i found it) he was also very approachable and gave help when i got stuck. So i was surprised to see that Keith Sterling has reinvented the wheel with FitNium.

Keith says he created FitNium as part of a learning exercise and also through the frustrations he is having with the Webtest Fixture.

While i can fully understand Keiths frustrations with the Webtest fixture (and this is not to say the webtest fixture isn't useful, it truly is), this is the very reason we are making the leap to Twist.

Helpfully FitNium is released under Ver 2.0 of the Apache License.

FitNesse is dead - Long live FitNesse

2008-12-02T09:27:00.004Z

Wow, we have been so busy for the last 7 weeks i haven't had time to update the blog at all, also it has been pointed out to me that the instructions for getting up and running with FitNesse wont work as is because i made some assumptions, so i will re hash that post soon and add a download that contains a ready-2-run FitNesse instance complete with working examples.

So what have we been up to?

ThoughtWorks are back, and this time people who didn't get exposure to the Agile way on the last project are running the project this time. We have also gone "all out" so we are doing the whole pair programming and BA and QA paring thing which meant we had to radically change our physical environment to accommodate such practices.

The TW test lead on the project had suggested we try their new tool "Twist" instead of FitNesse, and i thought, oh right another sales pitch, and asked them "what is the problem we have today that using Twist will solve". Answer? we have out grown FitNesse.

When i looked at the main fixture we are using for most of our tests (the webtest fixture) we had extended it that much, that 75% of the code in there was ours! So i had to evaluate why are we using FitNesse?? Answer, because it was a quick and easy way (not really that easy) to implement automated browser testing for our last project. Now however, we have a good understanding of driving selenium (born out of fixing the [apparent] inadequacies of webFixture [for our needs at least]).
So with our new skills (the ability to write tight reusable selenium tests), why would we want to preserve FitNesse?

Anyone who has read my ramblings will (hopefully) have read my post [i hate fitNesse / FitNesse Rocks], so you should be aware that we have a torrid love hate relationship with FitNesse already. There are lots of things wrong with FitNesse. A lack of historical test metrics is just one feature, but my main gripe is this:

FitNesse -> Fit -> Fixture -> Selenium

That is just too much overhead (read bloat). If we have the skills to write solid selenium tests directly then that's what we should do:

OurTests -> selenium

It is more important now because we want to run the tests as part of an automated build pipeline, and every millisecond counts. In the time it takes FitNesse to start up, i can run 3 raw selenium tests. ergo i need to take FitNesse and Fit out of the equation.

But wait that's not all...

As i've stated before, not all of my team posses the skills required to dive straight into the Java code and write selenium tests (nor do i want them to, they are testers and not developers after all). This is where FitNesse scores its points. It allows a non coder to write the tests in english prose (or near enough) and a coder and intrument the tests afterwards. However, when you begin to work like this versioning can become problematic very quickly. The problem becomes exasperated when a broken QA test causes the automated build to fail.

So what can Twist do that FitNesse cant?

Twist is built upon the Eclipse IDE with some important additions. Our developers already use Eclipse, and so we have been using Eclipse to maintain the webfixture. ergo, we are comfortable using Eclipse. It has CVS (SVN via an add on) integration built in, so that's versioning problem we had with FitNesse taken care of, but the real power of twist is its ability to allow non tech people (like BAs for example) to write prose like acceptance tests and save them, without writing any code.

A tester or developer can then check out the test on his Twist instance and instrument the test (Note: This can be achieved by recording the test in firefox via the selenium plug-in). So we still have the ability of writing tests in "English", but without having to learn any Wiki mark-up (bonus, also you don't have to be precious about what characters is in your test, as it wont be interpreted as wiki mark-up), and as soon as the test is instrumented it can be saved to the code repository and hey presto the version control is taken care of and the latest version of the test can be used in the automated build.

And finally the lack of historical test metrics. Because we will run the tests as part of the build, historical metrics are taken care of. The build server will maintain the metrics against each build, and the builds will be frequent.

However. Twist is still in Beta. Its a licensed product so we have to stump up some cash, and there will be another learning curve for us to tackle. We have had "fun" today trying to get it to launch selenium using our proxy with a list of hosts to ignore.

Our next challenge is to get the output of the build metrics displayed on a moving message LED display so that the culprit of a broken build will be named and shamed...

New look - Same Old Blog

2008-10-30T14:41:00.003Z

I was becoming frequently frustrated by the design constraints of the template i had chosen. Whenever i needed to share long lines of code (FitNesse Mark-up) they would wrap or truncate on screen.

I have solved this by using a very simple layout (all of the articles are still here) and i have implemented dp.SyntaxHighlighter by Alex Gorbatchev

If your not seeing pretty "HighLighted" code snips like this:

you are probably blocking the functionality with an AdBlocker or some such.

Proxy chaining with selenium

2008-10-29T17:33:00.005Z

Proxy chaining with selenium was the topic of a recent conversation with a colleague,he wanted to know more about proxy chaining and selenium, this has lead me to write this up. Again this was an issue we ran into early on with FitNesse and selenium , and its something we are currently tackling with Twist.

Selenium uses a proxy.pac files to configure the browser's proxy configuration.

An example selenium proxy.pac:


function FindProxyForURL(url, host) {
   if(shExpMatch(url, '/selenium-server/')) {
       return 'PROXY localhost:4444; DIRECT'
   }
}

In this example the browser will automatically forward any requests containing "/selenium-server/" to the selenium server however all other requests are un proxied and go DIRECT to the requested host.

We work behind a corporate proxy, so we need to be able to send request via the proxy, buy not for all the hosts.

We are using Selenium-RC and we specify some proxy settings on the command line at start-up:


java -Dhttp.proxyHost=proxy.ourdomain.dom -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*dev.ourdomain.dom*^|*qa.ourdomain.dom*^|*staging.ourdomain.dom

when you use start Selenium-RC in this way it generates a proxy.pac file like this:


function FindProxyForURL(url, host) {
       return 'PROXY localhost:4444; PROXY proxy.ourdomain.dom:8080';
}

The problem my colleague faced was that no matter what he put on the command line, the browser wasn't honouring his proxy configuration, or so he thought.

The selenium documentation suggest that the way we invoke selenium creates a proxy chain, but this isn't the case, if you look at the proxy.pac file selenium generated it just creates a fail-over chain, where the browser will try and use Selenium RC as the proxy and if it fails it will try and use the proxy you specified on the command line. bugger.

But fear not, there is an additional command line parameter that can be invoked (like a magic incantation) when starting Selenium-RC, its -avoidProxy:


java -Dhttp.proxyHost=proxy.ourdomain.dom -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*dev.ourdomain.dom*^|*qa.ourdomain.dom*^|*staging.ourdomain.dom -jar selenium-server.jar -avoidProxy

Adding the -avoidProxy flag, causes Selenium-RC to generate a proxy.pac file like this:


function FindProxyForURL(url, host) {
   if(shExpMatch(url, '*/selenium-server/*')) {
       return 'PROXY localhost:4444; PROXY proxy.ourdomain.dom:8080';
   } else if (shExpMatch(host, '*dev.ourdomain.dom*')) {
       return 'DIRECT';
   } else if (shExpMatch(host, '*qa.ourdomain.dom*')) {
       return 'DIRECT';
   } else if (shExpMatch(host, '*staging.ourdomain.dom*')) {
       return 'DIRECT';
   } else {
       return 'PROXY proxy.ourdomain.dom:8080';
   }
}

What this does is use selenium for anything that has /selenium-server/ in the url, else it uses the corporate proxy, unless the host is one of the ones specified in which case it goes direct to that host. Eureka!

Well almost. Enter the same origin policy.

Back to my colleague. He was using *chrome in his tests (note this has nothing to do with google chrome, its firefox but with more schwartz) and if you use one of these "experimental browsers" as Selenium calls them, (*chrome, *iehta) then you need to set your browser's proxy settings manually and just specify the path to your browser as if it were an unsupported browser.

For example, you can launch Firefox with a custom configuration like this:


*custom c:\Program Files\Mozilla Firefox\firefox.exe

When the browser is started like this you have to manually configure the proxy settings to use Selenium Server as a proxy. This just means opening the browser preferences and specifying "localhost:4444" as the HTTP proxy.

I have also used the experimental browser *pifirefox thats proxy inject firefox with good results.

FitNesse and User Authentication

2008-10-29T14:11:00.008Z

So i have a need to secure our FitNesse instance. FitNesse has built in security, and i have begun to look at that more closely, I'm really only posting this here to serve as an aide mémoire. I will update the post as i progress with the topic.

As described on http://fitnesse.org/FitNesse.StartingAndStoppingFitNesse

To start FitNesse with security enabled and get the username and password form a
text file called password.txt

To create a file called password.txt and add a user into the file.


java -cp fitnesse.jar  fitnesse.authentication.Password -f password.txt $FITNESSE_USERNAME

It is usual to use your login id as $FITNESSE_USERNAME.
This would allow you to map the environment variables $LOGNAME or $USER on UNIX or
%USERNAME% under Windows to the FitNesse username.

For stopping fitnesse from command-line when authentication has been enabled:


java -cp fitnesse.jar fitnesse.Shutdown [...more-options] -c $FITNESSE_USERNAME $PASSWORD

Getting started with FitNesse

2008-10-08T10:04:00.006+01:00

A few other colleagues have been trying FitNesse, and hitting the same problems we did.

Lets be frank, the documentation that comes with FitNesse inst great for a beginner, so i have written a brief "how to" (see below) that should get you up and running.

This isn't a step by step hold your hands tutorial (and i haven't found one on the internet either), it assumes you have some level of technical Schwartz and i assume you are using windows (if you are using linux, you should be able to translate without problem). I am alos talking about testing in java here, i wont be covering the .NET tests.

Some Handy Links:

The guy (gojko) who wrote WebTest and Dbfit fixtures, writes a blog which contains a wealth of info.

The Webtest Reference is a must read if you are going to use that fixture, but it is better to print out and pin up.

Steps to create your first test suite.

Installing:

Do you have java installed?
FitNesse is written in Java so you need to have Java installed on your PC to be able to use Fitnesse. If you don't know if you have Java installed, open a command prompt, and tytpe the following

java -version

If you have java installed you should see a message similar to


H:\>java -version
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Client VM (build 1.5.0_15-b04, mixed mode)

it doesn't matter too much if the message isn't exactly the same, but if you get the following message


'java' is not recognized as an internal or external command,
operable program or batch file.

then you don't have java installed (or you might have but its not in your path, either way fitnesse isn't going to run until java works for you).

The download of fitnesse will give you a zip file fitnesse.zip.
You should unpack it anywhere you like. But c:\ would be best. if you extract it with folder names it will create a fitnesse directory for you.
Go to that directory and double click on run.bat
You should get a message saying that FitNesse started.
You can then start up your browser and go to http://localhost

Note: when you clicked on the run.bat you have got a warning from windows firewall you should allow or unblock fitnesse as it runs a webserver so it needs to have access.

If you get an error, you may have a web server running on your machine already, using port 80. If this is the case you can specify a different port for fitnesse to run on, like 8080, you can achieve this by going to the fitnesse directory and typing:

java -cp fitnesse.jar fitnesse.Fitnesse -p 8080

Then navigate to it in your web browser. (http://127.0.0.1:8080/)

if this works for you, simple edit the run.bat file and add your port preference in there.

Beefing it up

We now need to install the webfixture. This is the thing that will save us from having to write java code, its taxonomy is also a little warmer than raw selenese.

visit http://www.fitnesse.info/webtest and download the java binaries only package. It will be a jar file.

once downloaded you need to navigate to your fitnesse folder, again i'm assuming its c:\fitnesse and create a new folder in there called lib

inside lib place the downloaded webfixture jar file.

Now get a copy of selenium from http://selenium-rc.seleniumhq.org/download.jsp

i should point out here that i still use version 092 because there are a few flaky things with the newer versions that could leave you pulling out your hair.

Again, place the jar ....

Okay, so i should mention here that if you are behind a proxy, you need to specify that in your selenium start up command.
Creating a test:

Now you have fitnesse up and running you can browse its pages. As you can see there is some content in there already.

To create a new test suite just type the name of the suite in CamelCase in the address bar your browser, e.g lets say i'm going to call my suite MyFirstSuite, i would type

http://127.0.0.1:8080/MyFirstSuite

into the address bar of my browser.

Fitnesse will tell you the page doesn’t exist and give you the option of creating it (the newer version just gives you the edit window, in which case just paste in the path info below).

Before we can write any tests we have to tell fitnesse where we will be storing our libraries by putting the path information into the suite page. In this example I assume you are putting all your jars (selenium, webtest, dbfit) into a folder called lib inside the fitnesse folder.

In the newly created page (type into the edit box) i would type:

!path lib\*.jar

Then i add the fitnesse wiki mark-up command to generate a contents list automatically (this is just me being lazy).

!contents -R

and now save the page (click the save button).

Note: If you want your SetUp and TearDown pages to be collapsed on your test pages you can define the variables:


!define COLLAPSE_SETUP {true}
!define COLLAPSE_TEARDOWN {true}

in the same page too. If you dont know what that means, just move on, dont worry.

Now we have a suite page we need to tell fitnesse that the page is to be treated as a suite. Click the properties button on the left menu. If the properties button is not available on the left menu, then type ?properties at the end of the url in your browser:

http://127.0.0.1:8080/MyFirstSuite?properties

From within the properties page make sure the "suite" checkbox is checked and uncheck the buttons you don’t want to appear in your left hand menu (I never use the files, refactor, search or versions button, so I turn these off but I enable edit if its not already).

Now we need to create the setup and teardown pages for our suite, again we can just type the names at the end of the url in the browser preceded with a '.' period:

http://127.0.0.1:8080/MyFirstSuite.SetUp

Within the SetUp page we declare the fixture we are using for the test, e.g the WebTest fixture

!|com.neuri.webfixture.WebTest|

The exclamation mark tells the wiki not to parse the text in the cell, otherwise is would look for a page called WebTest and think you want to create that page.

You can then put the startup command for selenium here, so for webtest (refer to the reference you printed out above) we use:


|Start Browser|firefox|With Selenium Console On|localhost|At Port|4444|And Scripts At|http://www.bbc.co.uk/|

Note: you would substitute the values for the selenium console (localhost) and its port (4444) depending on what machine you are driving and how you set selenium up. Here I'm assuming you are running the selenium server on your own machine, and it started on port 4444 (the default). But if you have a bit of savvy you should have just realise that you can drive any selenium instance remotely.

You can now save the SetUp page and create the TearDown page in the same way as we created the SetUp page, by typing the page name into the browser:

http://127.0.0.1:8080/MyFirstSuite.TearDown

We only need to put one thing in here, the instruction to shutdown the browser we opened in the SetUp page. again refer to the reference.

|shutdown browser|

Okay, now we have an empty suite.

Ok. Now lets create the test page, in the same way we created the other pages above, by typing the url:

http://127.0.0.1:8080/MyFirstSuite.TestOne

In here we will add a basic test using the WebTest reference to guide us:

Navigate to a URL during the test (change active page)

|user opens url|http://www.bbc.co.uk/radio2|

Check that the page loads fully within given time (use after user opens url)

|page reloads in less than|10|seconds|

Check current browser URL

|page url is|http://www.bbc.co.uk/radio2/|

Check that page contains some text (regardless of whether text is visible or not)

|element|title|contains text|BBC - Radio 2 - Home|

Save the page.

Edit the pages properties, either by the properties button on the left, or (if the button isn't available) by typing ?properties at the end of the url in the browsers address bar:

http://127.0.0.1:8080/MyFirstSuite.TestOne?properties

You can select the option you want, but make sure "Suite" is not checked, and "Test" is checked. Save the page.

You should now find that your test has the SetUp page and TearDown page included.

If selenium is running you should be able to run the test by clicking "test" on the menu.

There you have it, your first test suite. Try now to add TestTwo to your test suite.

Connecting to Oracle Grid with DbFit

2008-10-07T11:38:00.007+01:00

While there is a good deal of info on the DbFit website, there is little about connecting to Oracle databases. The examples given didn't help us at all because we used two nodes in our grid, and we have to give both of them in our connection string (if one node doesn't respond the test will use the other).

!|Connect|(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=node1.grid.domain.net)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=node2.grid.domain.net)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=WEB1_DEFAULT_SERVICE)))| UserName | PassWord |

As you can see you can end up with a very long row in your table, but it does connect.

Using WebTest & DbFit together

2008-10-07T09:42:00.004+01:00

If you are testing a web application, there is a good chance that you will need to manipulate or verify some data in a database as part of your testing of the front end application.

It would seem obvious to use DbFit and WebTest on the same test, however you should be aware of a few things.

WebTest is a DoFixture, and DbFit can work in two modes, flow or standalone, you will need to use the standalone mode of DbFit when using DbFit with another fixture..

When creating the SetUp page for the suite of tests add the fixtures like this:


!|com.neuri.webfixture.WebTest|
!|import|
|dbfit.fixture|
!|DatabaseEnvironment|sqlserver|
|Connect|server\SQLEXPRESS|FitNesseUser|Password|TestDB|

Note that there isn't a gap between DatabaseEnvironment and Connect, they must be in the same table

You can also connect to DbFit with the Connect table.

Also remember that since you're using standalone mode of DbFit, it wont rollback or commit, so you will need to use the following tables to commit or rollback transactions:


!|DatabaseEnvironment|
|Commit|


!|DatabaseEnvironment|
|Rollback|

We use the WebTest fixture and DbFit together and it works very well for us. However we have also extended WebTest to include some SQL features in WebTest that allow us to do simple SELECTS and use the output to verify what should be on screen without breaking the flow of the test.

I Hate FitNesse / Fitnesse Rocks!

2008-10-06T13:51:00.005+01:00

I want you to know that i have a love hate relationship with fitnesse.

I have been using it for several months now and sometimes i really hate it; i still use it and advocate its use, but i still hate it. Tim Ottinger summed up using FitNesse really well over on the FitNesse group (yahoo) . He said that Using Fitnesse is not unlike trying to teach a dog to sing. Its not that the dog is a bad singer, its simply a miracle that it can sing at all.

And its true. FitNesse is not a great wiki, but it does do wiki. It doesn't work very well in CI, but it can be integrated into CI. It has its own versioning that isn't great (millions of zip files), but it can be versioned traditionally (using SVN or CVS). It doesn't have any metrics (test execution over time, defects, test coverage), but you can get those through CI or integration with another tool (we are experimenting with integration into Quality Center [sic]).

There is also the big issue of user management. There several authentication methods supported by FitNesse:

No authentication (default), anybody can do any thing!
UserPassword: Defines one user and its password on fitnesse start: ... -a userA:passworA
PasswordFile: For multiple users, read once on fitnesse start, so adding new users requires a restart ffs.
LinuxPAM plugin: Use PAM to access Linux user passwords, requires a plugin.
Provide your own AuthenticatorClass as plugin.

Given a choice i'd opt for the PasswordFile mode but then administration becomes tiresome.

For more info on user management checkout:

What is a "story"?

2008-07-09T17:17:00.004+01:00

I got asked today by my Boss, "what is a story?" so i gave him a 5 minute overview, then i realised i could share this.

Simply put a story is a very short (should fit on an description of a piece of functionality.
A story serves several purposes.
It is used to estimate and identify areas of risk during initial release planning and ongoing through development.
A story is always written from the business perspective, and is normally based on an action (e.g. display search results, search stock).
A story is testable through the UI if application is UI based.
Our Stories are written using the form: "As an X I want to Y so I can Z"
Stories answer these questions:

Who is 'doing' the story (As an X).
What action is being taken (I want to Y).
What business need is the story hitting? (so I can Z).

They are at first very high level and i've found that initial story cards are almost always revised and rewritten. They are usually rewritten to lower levels of granularity so that at development time they are testable.

They are small enough that it only takes one iteration to complete. If they are too complex, they are expanded into narratives.
Stories can be either functional or non-functional, E.g. performance should be a story.

ThoughtWorks write the stories on ThoughtWorks branded index cards which are 147mm wide and 104mm high which are bigger than the index cards we could get from our stationers. (Who stocks these size cards? Viking have them on-line, but staples have them in store only. They are called 6x4 index cards by both)

They write on the cards to ensure the story is kept to a small size, this helps the customer "see" what they are saying. The cards are usually kept visible on a board. a Kanban board.

In Agile projects, it's common practice to share the project status via "big visible charts" on a wall in the project room (or elsewhere if a room is not available).

Kanban is a Japanese word which loosely translated, means card or sign. In a "Lean" production system, Kanban is a method which uses standard units or lot sizes (our story cards). The methodology dictates that a new card is only "pulled" into the system when the work represented by a card that is "in Play" is completed.

It works really well. I was away from the project for a couple of days to deal with recruitment. When i returned to the meeting i was able to jump straight in after reading the board for few minutes and attending the morning meeting.

Testers - Thou Shall Not Cut Code!

2008-07-02T17:34:00.008+01:00

okay, so its been a while since my last post, and we have some very frustrating times, i'll try to remember them now.

One of our 500lb gorillas is the fact that Fitnesse out of the box comes with limited functionality. By that i mean if you want to pull a value from a database and assert that it is being displayed in the users browser (via selenium) then you have to write your own Java class. As i said before we are a team of dedicated web testers, and while we are very technical testers, we are not Java programmers. So we looked round for some possible solutions to our dilemma.

First up is Anubhava’s Generic Fixture. Anubhava has had the same thoughts as me, but has the skills to actually put the idea into fruition. His generic fixture allows the fitnesse scriptee to write in DSL without writing a single line of Java, eureka!

i tried using the generic fixture for connecting to the DB. However we use Oracle 10G and our connection string details two nodes. In the examples Anubhava gives, the fixture accepts the usual server, port, dbname, user, password strings.

I thought i would share this with any other visitors who may use generic fixture and use similar JDBC connection strings to us. OracleDataSource will accept a url, so i amened Anubhavas example fitnesse tables to


| Generic Fixture | oracle.jdbc.pool.OracleDataSource |
| setDriverType | thin |
| setURL | jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST =(ADDRESS=(PROTOCOL=TCP)(HOST=node1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=node2)(PORT=1521)))(CONNECT_DATA =(SERVICE_NAME=WEB)))
| setUser | db_user |
| setPassword | db_user_password |
| conn=getConnection |

and we can now connect to our DB, and script without writing Java.

Next up is Gojko Adzic's DbFit another fixture for fitnesse, and comes with some examples for .net and Java and Oracle.

A little bit more of a learning curve with this one, but still within the realms of what we can manage as testers. We can use DbFit with the WebTest fixture and again accomplish a lot without writing any Java.

I guess i should point out here that its not just writing the Java, its where to store it (CVS), compiling the code, keeping it up to date, making sure we are using the right versions of drivers (ojdbc14), that we have the right environment variables set-up and so on. All of a sudden the line between the bounds of a tester and developer have become so blurred. Well okay, maybe i wouldn't just test the "happy path" :-)

Automate This

2008-06-18T08:45:00.006+01:00

A big bucket of rambley nonsense about automating web testing

Okay, so we have been trying to grapple with FitNesse and Selenium to produce automated "acceptance" tests for our continuous integration development.

Before we brought ThoughtWorks in i had already been looking at automated test tools. There are lots of tools out there, some very simple (yet effective) and other very complicated.

Some of these tools profess to be web testing tools, while others claim to be able to test anything. Having said that any tool that requires me to purchase an add-on or plug-in to achieve automated testing is going to get shot down straight away. Testing is core to what i do I'm a tester, so far as i'm concerned testing is not an add on.

I thought i would get my thoughts on automation down, so should the wheels of the fabled bus ever get chance to put my mothers under-cracker warnings to the test my ramblings can be copy and pasted into someone else' documentation.

During the usual testing activities we aim to undertake the following test activities:

1) Functionality Testing
2) Usability testing
3) GUI\Interface testing
4) Browser Compatibility testing
5) Performance testing
6) Security testing

The list isn't in any particular order and none is implied. But any seasoned tester will recognise that i'm talking about the upper right hand side of the V model, wheras FitNesse has traditionaly been used further down.

1) Functionality Testing:

It isn't immediately obvious what functional testing means when talking about web testing. If you are sat in front of a static html page, that is merely a funnel into a web application you are fairly limited in the amount of functional testing you can do. All web pages can be functionally tested for working includes (css, images, etc.) testing all the links, forms, images,JavaScript and cookies. All the individual elements that are used to make up a web page. Lets take a more in depth look at this.

Link checking:

Once aspect of functional testing of web pages, checks links for broken or malformed types of:

Outgoing links
Internal links.
named anchors.
links used to send email (mailto).

If its possible (its not always possible to get access the the disc) test for orphaned objects. Note that i didn't mention things like SEO within the context of functionality testing or links, because i see that as being non functional testing of a link. I also didn't mention that the link was tagged appropriately for disabled users, because again this is covered later on. What should be covered is that the link is tagged appropriately for whichever analytic packages is being used by the website.

Test forms:
Forms were previously essential for submitting user information, but thanks to web 2.0 they are starting to disappear. Nevertheless, they still need to be tested for

input validations (where applied).
default values of fields (where set).
constraints (sting length, numerical, alpha etc).

I haven't mentioned the error handling for the actual submission be cause that is the responsibility of the functionality within url the form points at, and not the form itself.

Cookies:
Testing Cookies, can be a right PITA. Thankfully firefox and its numerous wonderful plugins like web developer and firebug have made our life a lot easier. Cookie testing normally covers:

cookies enabled
cookies disabled
cookies deleted
cookies set
cookies read
cookies updated

We often visually inspect the cookie to make sure the developer hasn't done something incredibly stupid like store the username and password in the cookie in plain text.

Database testing:
Data consistency is very important in web application. Check for data integrity and errors while you edit, delete, modify the forms or do any DB related functionality.
Check if all the database queries are executing correctly, data is retrieved correctly and also updated correctly. More on database testing could be load on DB, we will address this in web load or performance testing below.

2) Usability Testing:

Test for navigation:
Navigation means how the user surfs the web pages, different controls like buttons, boxes or how user using the links on the pages to surf different pages.
Usability testing includes:
Web site should be easy to use. Instructions should be provided clearly. Check if the provided instructions are correct means whether they satisfy purpose.
Main menu should be provided on each page. It should be consistent.

Content checking:
Content should be logical and easy to understand. Check for spelling errors. Use of dark colors annoys users and should not be used in site theme. You can follow some standards that are used for web page and content building. These are common accepted standards like as I mentioned above about annoying colours, fonts, frames etc.
Content should be meaningful. All the anchor text links should be working properly. Images should be placed properly with proper sizes.
These are some basic standards that should be followed in web development. Your task is to validate all for UI testing

Other user information for user help:
Like search option, sitemap, help files etc. Sitemap should be present with all the links in web sites with proper tree view of navigation. Check for all links on the sitemap.
“Search in the site” option will help users to find content pages they are looking for easily and quickly. These are all optional items and if present should be validated.

3) Interface Testing:
The main interfaces are:

Web server and application server interface
Application server and Database server interface.

Check if all the interactions between these servers are executed properly. Errors are handled properly. If database or web server returns any error message for any query by application server then application server should catch and display these error messages appropriately to users. Check what happens if user interrupts any transaction in-between? Check what happens if connection to web server is reset in between?

4) Compatibility Testing:
Compatibility of your web site is very important testing aspect. See which compatibility test to be executed:

Browser compatibility
Operating system compatibility
Mobile browsing
Printing options

Browser compatibility:
In my web-testing career I have experienced this as most influencing part on web site testing.
Some applications are very dependent on browsers. Different browsers have different configurations and settings that your web page should be compatible with. Your web site coding should be cross browser platform compatible. If you are using java scripts or AJAX calls for UI functionality, performing security checks or validations then give more stress on browser compatibility testing of your web application.
Test web application on different browsers like Internet explorer, Firefox, Netscape navigator, AOL, Safari, Opera browsers with different versions.

OS compatibility:
Some functionality in your web application is may not be compatible with all operating systems.
We Test our web application on different operating systems like Windows, MAC, Linux with different OS flavors. What can work in MSIE7 on Windows XP with service pack three, will fail onMSIE7 Windows Vista Home, but work on MSIE7 with Windows Vista Office Premium.

Mobile browsing:
This is a tough one. We get thousands of visitors a day using mobile devices, and even games consoles to view our site.

Printing options:
If you are giving page-printing options then make sure fonts, page alignment, page graphics getting printed properly. Pages should be fit to paper size or as per the size mentioned in printing option.

5) Performance testing:
Web application should sustain to heavy load. Web performance testing should include:
Web Load Testing
Web Stress Testing

Test application performance on different internet connection speed.
In web load testing test if many users are accessing or requesting the same page. Can system sustain in peak load times? Site should handle many simultaneous user requests, large input data from users, Simultaneous connection to DB, heavy load on specific pages etc.

Stress testing: Generally stress means stretching the system beyond its specification limits. Web stress testing is performed to break the site by giving stress and checked how system reacts to stress and how system recovers from crashes.
Stress is generally given on input fields, login and sign up areas.

In web performance testing web site functionality on different operating systems, different hardware platforms is checked for software, hardware memory leakage errors,

6) Security Testing:

Following are some test cases for web security testing:

Test by pasting internal url directly into browser address bar without login. Internal pages should not open.
If you are logged in using username and password and browsing internal pages then try changing url options directly. I.e. If you are checking some publisher site statistics with publisher site ID= 123. Try directly changing the url site ID parameter to different site ID which is not related to logged in user. Access should denied for this user to view others stats.
Try some invalid inputs in input fields like login username, password, input text boxes. Check the system reaction on all invalid inputs.
Web directories or files should not be accessible directly unless given download option.
Test the CAPTCHA for automates scripts logins.
Test if SSL is used for security measures. If used proper message should get displayed when user switch from non-secure http:// pages to secure https:// pages and vice versa.
All transactions, error messages, security breach attempts should get logged in log files somewhere on web server.

Creating FitNesse tests

2008-06-12T09:26:00.003+01:00

Okay, so we are still having problems with CruiseControl (it falls over after n builds) but meanwhile i need to start creating tests in Fitnesse.

The ThoughtWorks consultant has already started, but i just reviewed the work and i'm less than impressed.

I want each page to contain some prose that describes what's being tested and why.

The structure of a requirement should follow a basic syntax. We should have a basic description of what is needed from a business perspective:

As a [role] I want [requirement] so that [business value].

And the tests should line up by defining an Acceptance criteria, basically we define what "done" looks like:

Given [context] when [action] then [outcome]

The Acceptance criteria should be written jointly by BA and QA, and the acceptance tests map directly to acceptance criteria.

So on each page (a test) i want a preamble (the story), the requirement and the acceptance criteria.

The Fit table (the actual test) can then sit directly under the prose for the acceptance criteria.

That way, each page (each test) is the requirement, the test and the test results.

Integrating Fitnesse with CruiseControl

2008-06-11T12:08:00.004+01:00

So like i said in earlier posts we are implementing FitNesse on our java development projects (well to be precise just one) with help from ThoughtWorks so we can run automated acceptance tests.

We are still at the bashing the stick on the ground and making a lot of noise stage (so far as my Space odyssey metaphor goes), and we are not yet 100% convinced of the value of FitNesse within our organisation. The way it has been set-up so far, makes it unmaintainable, hard to navigate and it feels a bit flaky. Sure FitNesse is great where the requirements are a little thin on the ground, but i think there is an absolute requirement for process and procedure when it comes to building a suite of tests in FitNesse if it is ever going to have longevity. That is not slur on FitNesse, but i have yet to see any one describe how they build a suite that has some structure.

So today we are trying to get FitNesse and CruiseControl working together.
The dev team have used CruiseControl in the past, but it doesnt work properly (some issue with forking) and depending on how we look at cobertura we either have 27% coverage or 35% coverage of Unit test so there is some way for them to go. I hope can integrate the FitNesse Tests with the current build process and gain an extra layer of testing.

One thing as a tester i dislike about FitNesse is that lacks a lot (does it have any) of the reporting features you see on other test tools. So i will spend this afternoon looking at some free (as in beer) reporting tools. I also cant see how you do test selection, or report on test coverage....

Fitnesse for dummies?

2008-06-06T12:31:00.007+01:00

We pride ourselves on our technical prowess, by we i mean my team and I and by team i mean the greater QA team at work.

As testers we are often thought of as not being technical and in some institutions it is actively discouraged. However being a technical tester is dependent on the domain you work in.

In our domain we need to be holistic on our approach to testing, we need to be multidisciplinary with skills in Linux, Apache, Jboss, J2EE and Oracle. We work for a large website so we all need to understand headers, HTTP, cookies, dynamic HTML/Json.

So it was with some surprise (read, it caught us out) that we found ourselves staring at FitNesse (the Acceptance Testing Framework) like the apes staring at the monolith at the start of 2001: A Space Odyssey (1968). We understand its importance, and we can see the benefits, but how do we turn it on? what makes it go?

Well a few hours later and to our horror we have installed JDK and eclipse and we are coding in java. But this doesn't really sit well at all, testers writing tests in java? who will test our tests?

Worse than this, we don't have Java coding skills across the team (nor do we want them) so what can we do?

A few googles later and we can see that we are not the only people that think this sucks. For every test case we write in Fitnesse we ill have to write a corresponding fixture (read some java code). Until i run across Anubhava's Tech Blog and his post Introducing Generic Fixture for FitNesse. Here he talks of the same problem of having write java code for every test. His solution is rather neat, the GenericFixture.

I followed his examples and hit a few dead ends, so i emailed Anubhava and after a few question from him discovered that what i believed to be the GenricFixture.jar was in fact a 700Kb web page (an error message from sourceforge). With the proper jar downloaded from sourceforge we are off and running, and can now write tests without cutting a single line of java, but wait that's not all...

Something was still unsettling about the whole deal, and that was the fact that FitNesse allows users to create tests using natural language.

here is a simple example in fitness

users opens the URL	http://www.google.co.uk
page has the title		Google
page has an element named	q		true
page has an element named	btnG		true
user types	chocolate	into	q	field
user clicks on the button named	btnG
page loads in less than	5	seconds
page has the title		chocolate - Google Search
user clicks on the link named	Chocolate - Wikipedia, the free encyclopedia
page loads in less than	5	seconds
page has the title		Chocolate - Wikipedia, the free encyclopedia

and here is the same example in selenease

open	http://www.google.co.uk
assertTitle	Google
type	q	chocolate
clickAtAndWait	btnG
assertTitle	chocolate - Google Search
clickAtAndWait	link=Chocolate - Wikipedia, the free encyclopedia
assertTitle	Chocolate - Wikipedia, the free encyclopedia

As you can see the FitNesse example is easily read and understood whereas the selenim version requires the user to understand the selenium syntax (selenease).

Our friend Anubhava (above) had hit upon the same problem, and implemented a Domain Specific Language (DLS) adepter for the GenricFixture, so problem solved, for now...

Running Selenium with proxy exceptions

2008-06-06T10:44:00.005+01:00

The way we have our QA environment set-up we use a DNS server this allows us to mimic the live environment (the customer facing server names are the same as in the production environment). It sounds ideal however it isn't without problems. We cant set-up a banner server in our environment (because we use a third party banner service) nor can we set-up a WA server (we use multiple WA vendors) so we have to use proxy exceptions that excludes everything from going through the proxy that exists in inside the QA environment.

This obviously means we can end up with a pretty big proxy exceptions list in our browser. Firefox isn't a problem as there are a couple of plug ins available that can manage multiple proxy settings. However IE requires us to use a hand crafted batch file to update registry settings. Not really a problem, and we are comfortable with how it works. It rarely catches us out.

Now when we cam to automate our tests with selenium we ran into a problem. We needed to tell selenium to use our corporate proxy so it can proxy request to the outside world, and when selenium starts the browser it tell the browser to use a particular PAC file that is generated at test run time. The PAC file is quite simple and tells the browser to use Selenium for anything that lies within the SUT else go external. sound's great except that it doesn't work.
We read a few forum posts, and blog posts and scratched our heads. Finally we opened the Selenium source code and found that we could in fact pass in a list of hosts to apply to the proxy exceptions list. We were feeling pretty jaded are spending hours of google time on it, why its not documented clearly anywhere i don't know.

Okay, so our startup command for selenium looked like this

java -Dhttp.proxyHost=proxy.ourdomain.co.uk -Dhttp.proxyPort=8080 -jar selenium-server.jar –avoidProxy

To add proxy avoidance you specify the hosts you want to avoid through the proxy and delimit with the pipe character |, but we then found we had to delimit the pipe character with the caret ^

-Dhttp.nonProxyHosts=*www.ourdomain.co.uk*^|*search.ourdomain.co.uk*

So our new selenium startup string looked like this

java -Dhttp.proxyHost=proxy.ourdomain.co.uk -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*www.ourdomain.co.uk*^|*search.ourdomain.co.uk* -jar selenium-server.jar –avoidProxy

This means that if the request is for anything other than www. or search . on our domain, selenium forwards it through our corporate proxy.

Of Agile, ThoughtWorks, Selenium and Fitnesse

2008-06-06T09:38:00.005+01:00

I have set up this blog to capture some of the problems we are bound to run into over the next 60 days.

I work for a well known (read busy) commercial website in the UK. We currently use a traditional waterfall approach to our software development life cycle, and we are under pressure to deliver more, and deliver it more frequently.

I am an ISEB certified Test Practitioner and I supported by two testers on my Team. Martin, a test analyst, who has achieved his ISEB/ISTQB Foundation certificate in software testing, and Robert, a junior test analyst who will be taking his foundation training later this year.

We are all aligned to thinking about software testing in the same way. That is, we have a clear set of requirements (be they, technical, business or design) and we test against those; simple no?

However in the real world it is never that cut and dried, and often requirements are thin, late or non existent, and (of course) liable to change at the drop of a hat. This is compounded further by the businesses increasing frustration at having to wait for a simple change to be implemented; we are only talking about a website here right? It’s not like we are trying to boil the ocean.

Cue Music – Enter ThoughtWorks.

Somewhere along the way the business got to hear of ThoughtWorks, a company that can offer (among other services) a helping hand for any software house wishing to become Agile. I don’t want to talk too much about ThoughtWorks and our relationship with them, because I only want to focus on the Testing aspect of becoming Agile, and because I could fill another blog with chatter about working with ThoughtWorks.

Anyhoo, we are a Linux, Apache, J2EE & Oracle house. So it felt right that we should use Selenium for our automated testing. ThoughtWorks originally created selenium so it felt right to use it with them. We have looked at Selenium in the past and I have used selenium scripts with the most excellent Reality QA product from the guys at GOMEZ (more on that later).

A couple of interesting points about selenium; Selenium is a chemical element with the atomic number 34 and is represented by the chemical symbol Se. Selenium can be found naturally in nuts (Brazil nuts are the richest ordinary dietary source), cereals, meat (such as kidney), fish (shellfish such as crab and lobster), and eggs.
Selenium is often touted as "The cure for mercurial poisoning" (being poisoned by Mercury), and any tester can tell you that Mercury produced (before being bought by HP) some highly priced automated test tools of the variety that "require plug-ins" to do anything more ordinary than record and playback (Test Director for instance). So I think the developers of selenium thought it would be funny to name their product Selenium. I think however that the last laugh may be on them, because selenium is not actually an antidote to mercury poisoning, it’s just a chelating agent for heavy metals including mercury and although selenium is an essential trace element, it is toxic if taken in excess. There is no known antidote to selenium poisoning.