Something new?

I was left this tantalising teaser today "A language and framework agnostic vocabulary for testing web sites", and i've got to say that got my mind ticking.

You can follow Thistles progress over on the social code hub github.

I'll be watching with interest!

Fitnesse - Another DSL

So, if you have followed this blog you will know that we we first started on our journey of discovery with FitNesse by trying anubhavas generic fixture but while this idea seems great it has its shortcomings. Now Anubhava did a pretty good job of telling people about his fixture (which is how i found it) he was also very approachable and gave help when i got stuck. So i was surprised to see that Keith Sterling has reinvented the wheel with FitNium.

Keith says he created FitNium as part of a learning exercise and also through the frustrations he is having with the Webtest Fixture.

While i can fully understand Keiths frustrations with the Webtest fixture (and this is not to say the webtest fixture isn't useful, it truly is), this is the very reason we are making the leap to Twist.

Helpfully FitNium is released under Ver 2.0 of the Apache License.

FitNesse is dead - Long live FitNesse

Wow, we have been so busy for the last 7 weeks i haven't had time to update the blog at all, also it has been pointed out to me that the instructions for getting up and running with FitNesse wont work as is because i made some assumptions, so i will re hash that post soon and add a download that contains a ready-2-run FitNesse instance complete with working examples.

So what have we been up to?

ThoughtWorks are back, and this time people who didn't get exposure to the Agile way on the last project are running the project this time. We have also gone "all out" so we are doing the whole pair programming and BA and QA paring thing which meant we had to radically change our physical environment to accommodate such practices.

The TW test lead on the project had suggested we try their new tool "Twist" instead of FitNesse, and i thought, oh right another sales pitch, and asked them "what is the problem we have today that using Twist will solve". Answer? we have out grown FitNesse.

When i looked at the main fixture we are using for most of our tests (the webtest fixture) we had extended it that much, that 75% of the code in there was ours! So i had to evaluate why are we using FitNesse?? Answer, because it was a quick and easy way (not really that easy) to implement automated browser testing for our last project. Now however, we have a good understanding of driving selenium (born out of fixing the [apparent] inadequacies of webFixture [for our needs at least]).
So with our new skills (the ability to write tight reusable selenium tests), why would we want to preserve FitNesse?

Anyone who has read my ramblings will (hopefully) have read my post [i hate fitNesse / FitNesse Rocks], so you should be aware that we have a torrid love hate relationship with FitNesse already. There are lots of things wrong with FitNesse. A lack of historical test metrics is just one feature, but my main gripe is this:

FitNesse -> Fit -> Fixture -> Selenium

That is just too much overhead (read bloat). If we have the skills to write solid selenium tests directly then that's what we should do:

OurTests -> selenium

It is more important now because we want to run the tests as part of an automated build pipeline, and every millisecond counts. In the time it takes FitNesse to start up, i can run 3 raw selenium tests. ergo i need to take FitNesse and Fit out of the equation.

But wait that's not all...

As i've stated before, not all of my team posses the skills required to dive straight into the Java code and write selenium tests (nor do i want them to, they are testers and not developers after all). This is where FitNesse scores its points. It allows a non coder to write the tests in english prose (or near enough) and a coder and intrument the tests afterwards. However, when you begin to work like this versioning can become problematic very quickly. The problem becomes exasperated when a broken QA test causes the automated build to fail.

So what can Twist do that FitNesse cant?

Twist is built upon the Eclipse IDE with some important additions. Our developers already use Eclipse, and so we have been using Eclipse to maintain the webfixture. ergo, we are comfortable using Eclipse. It has CVS (SVN via an add on) integration built in, so that's versioning problem we had with FitNesse taken care of, but the real power of twist is its ability to allow non tech people (like BAs for example) to write prose like acceptance tests and save them, without writing any code.

A tester or developer can then check out the test on his Twist instance and instrument the test (Note: This can be achieved by recording the test in firefox via the selenium plug-in). So we still have the ability of writing tests in "English", but without having to learn any Wiki mark-up (bonus, also you don't have to be precious about what characters is in your test, as it wont be interpreted as wiki mark-up), and as soon as the test is instrumented it can be saved to the code repository and hey presto the version control is taken care of and the latest version of the test can be used in the automated build.

And finally the lack of historical test metrics. Because we will run the tests as part of the build, historical metrics are taken care of. The build server will maintain the metrics against each build, and the builds will be frequent.

However. Twist is still in Beta. Its a licensed product so we have to stump up some cash, and there will be another learning curve for us to tackle. We have had "fun" today trying to get it to launch selenium using our proxy with a list of hosts to ignore.

Our next challenge is to get the output of the build metrics displayed on a moving message LED display so that the culprit of a broken build will be named and shamed...

New look - Same Old Blog

I was becoming frequently frustrated by the design constraints of the template i had chosen. Whenever i needed to share long lines of code (FitNesse Mark-up) they would wrap or truncate on screen.

I have solved this by using a very simple layout (all of the articles are still here) and i have implemented dp.SyntaxHighlighter by Alex Gorbatchev

If your not seeing pretty "HighLighted" code snips like this:



you are probably blocking the functionality with an AdBlocker or some such.

Proxy chaining with selenium

Proxy chaining with selenium was the topic of a recent conversation with a colleague,he wanted to know more about proxy chaining and selenium, this has lead me to write this up. Again this was an issue we ran into early on with FitNesse and selenium , and its something we are currently tackling with Twist.

Selenium uses a proxy.pac files to configure the browser's proxy configuration.

An example selenium proxy.pac:

function FindProxyForURL(url, host) {
   if(shExpMatch(url, '/selenium-server/')) {
       return 'PROXY localhost:4444; DIRECT'
   }
}

In this example the browser will automatically forward any requests containing "/selenium-server/" to the selenium server however all other requests are un proxied and go DIRECT to the requested host.

We work behind a corporate proxy, so we need to be able to send request via the proxy, buy not for all the hosts.

We are using Selenium-RC and we specify some proxy settings on the command line at start-up:

java -Dhttp.proxyHost=proxy.ourdomain.dom -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*dev.ourdomain.dom*^|*qa.ourdomain.dom*^|*staging.ourdomain.dom

when you use start Selenium-RC in this way it generates a proxy.pac file like this:

function FindProxyForURL(url, host) {
       return 'PROXY localhost:4444; PROXY proxy.ourdomain.dom:8080';
}

The problem my colleague faced was that no matter what he put on the command line, the browser wasn't honouring his proxy configuration, or so he thought.

The selenium documentation suggest that the way we invoke selenium creates a proxy chain, but this isn't the case, if you look at the proxy.pac file selenium generated it just creates a fail-over chain, where the browser will try and use Selenium RC as the proxy and if it fails it will try and use the proxy you specified on the command line. bugger.

But fear not, there is an additional command line parameter that can be invoked (like a magic incantation) when starting Selenium-RC, its -avoidProxy:

java -Dhttp.proxyHost=proxy.ourdomain.dom -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*dev.ourdomain.dom*^|*qa.ourdomain.dom*^|*staging.ourdomain.dom -jar selenium-server.jar -avoidProxy

Adding the -avoidProxy flag, causes Selenium-RC to generate a proxy.pac file like this:

function FindProxyForURL(url, host) {
   if(shExpMatch(url, '*/selenium-server/*')) {
       return 'PROXY localhost:4444; PROXY proxy.ourdomain.dom:8080';
   } else if (shExpMatch(host, '*dev.ourdomain.dom*')) {
       return 'DIRECT';
   } else if (shExpMatch(host, '*qa.ourdomain.dom*')) {
       return 'DIRECT';
   } else if (shExpMatch(host, '*staging.ourdomain.dom*')) {
       return 'DIRECT';
   } else {
       return 'PROXY proxy.ourdomain.dom:8080';
   }
}

What this does is use selenium for anything that has /selenium-server/ in the url, else it uses the corporate proxy, unless the host is one of the ones specified in which case it goes direct to that host. Eureka!

Well almost. Enter the same origin policy.

Back to my colleague. He was using *chrome in his tests (note this has nothing to do with google chrome, its firefox but with more schwartz) and if you use one of these "experimental browsers" as Selenium calls them, (*chrome, *iehta) then you need to set your browser's proxy settings manually and just specify the path to your browser as if it were an unsupported browser.

For example, you can launch Firefox with a custom configuration like this:

*custom c:\Program Files\Mozilla Firefox\firefox.exe

When the browser is started like this you have to manually configure the proxy settings to use Selenium Server as a proxy. This just means opening the browser preferences and specifying "localhost:4444" as the HTTP proxy.

I have also used the experimental browser *pifirefox thats proxy inject firefox with good results.

FitNesse and User Authentication

So i have a need to secure our FitNesse instance. FitNesse has built in security, and i have begun to look at that more closely, I'm really only posting this here to serve as an aide mémoire. I will update the post as i progress with the topic.

As described on http://fitnesse.org/FitNesse.StartingAndStoppingFitNesse

To start FitNesse with security enabled and get the username and password form a
text file called password.txt

<br />java -cp fitnesse.jar fitnesse.FitNesse -a password.txt [...more-options]<br />

To create a file called password.txt and add a user into the file.

java -cp fitnesse.jar  fitnesse.authentication.Password -f password.txt $FITNESSE_USERNAME

It is usual to use your login id as $FITNESSE_USERNAME.
This would allow you to map the environment variables $LOGNAME or $USER on UNIX or
%USERNAME% under Windows to the FitNesse username.

For stopping fitnesse from command-line when authentication has been enabled:

java -cp fitnesse.jar fitnesse.Shutdown [...more-options] -c $FITNESSE_USERNAME $PASSWORD

Getting started with FitNesse

A few other colleagues have been trying FitNesse, and hitting the same problems we did.

Lets be frank, the documentation that comes with FitNesse inst great for a beginner, so i have written a brief "how to" (see below) that should get you up and running.

This isn't a step by step hold your hands tutorial (and i haven't found one on the internet either), it assumes you have some level of technical Schwartz and i assume you are using windows (if you are using linux, you should be able to translate without problem). I am alos talking about testing in java here, i wont be covering the .NET tests.

Some Handy Links:

The guy (gojko) who wrote WebTest and Dbfit fixtures, writes a blog which contains a wealth of info.

The Webtest Reference is a must read if you are going to use that fixture, but it is better to print out and pin up.

Steps to create your first test suite.

Installing:

Do you have java installed?
FitNesse is written in Java so you need to have Java installed on your PC to be able to use Fitnesse. If you don't know if you have Java installed, open a command prompt, and tytpe the following

java -version

If you have java installed you should see a message similar to

H:\>java -version
java version "1.5.0_15"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
Java HotSpot(TM) Client VM (build 1.5.0_15-b04, mixed mode)

it doesn't matter too much if the message isn't exactly the same, but if you get the following message

'java' is not recognized as an internal or external command,
operable program or batch file.

then you don't have java installed (or you might have but its not in your path, either way fitnesse isn't going to run until java works for you).


The download of fitnesse will give you a zip file fitnesse.zip.
You should unpack it anywhere you like. But c:\ would be best. if you extract it with folder names it will create a fitnesse directory for you.
Go to that directory and double click on run.bat
You should get a message saying that FitNesse started.
You can then start up your browser and go to http://localhost

Note: when you clicked on the run.bat you have got a warning from windows firewall you should allow or unblock fitnesse as it runs a webserver so it needs to have access.

If you get an error, you may have a web server running on your machine already, using port 80. If this is the case you can specify a different port for fitnesse to run on, like 8080, you can achieve this by going to the fitnesse directory and typing:

java -cp fitnesse.jar fitnesse.Fitnesse -p 8080

Then navigate to it in your web browser. (http://127.0.0.1:8080/)

if this works for you, simple edit the run.bat file and add your port preference in there.

Beefing it up

We now need to install the webfixture. This is the thing that will save us from having to write java code, its taxonomy is also a little warmer than raw selenese.

visit http://www.fitnesse.info/webtest and download the java binaries only package. It will be a jar file.

once downloaded you need to navigate to your fitnesse folder, again i'm assuming its c:\fitnesse and create a new folder in there called lib

inside lib place the downloaded webfixture jar file.

Now get a copy of selenium from http://selenium-rc.seleniumhq.org/download.jsp

i should point out here that i still use version 092 because there are a few flaky things with the newer versions that could leave you pulling out your hair.

Again, place the jar ....

Okay, so i should mention here that if you are behind a proxy, you need to specify that in your selenium start up command.
Creating a test:

Now you have fitnesse up and running you can browse its pages. As you can see there is some content in there already.

To create a new test suite just type the name of the suite in CamelCase in the address bar your browser, e.g lets say i'm going to call my suite MyFirstSuite, i would type

http://127.0.0.1:8080/MyFirstSuite

into the address bar of my browser.

Fitnesse will tell you the page doesn’t exist and give you the option of creating it (the newer version just gives you the edit window, in which case just paste in the path info below).

Before we can write any tests we have to tell fitnesse where we will be storing our libraries by putting the path information into the suite page. In this example I assume you are putting all your jars (selenium, webtest, dbfit) into a folder called lib inside the fitnesse folder.

In the newly created page (type into the edit box) i would type:

!path lib\*.jar

Then i add the fitnesse wiki mark-up command to generate a contents list automatically (this is just me being lazy).

!contents -R

and now save the page (click the save button).

Note: If you want your SetUp and TearDown pages to be collapsed on your test pages you can define the variables:

!define COLLAPSE_SETUP {true}
!define COLLAPSE_TEARDOWN {true}

in the same page too. If you dont know what that means, just move on, dont worry.


Now we have a suite page we need to tell fitnesse that the page is to be treated as a suite. Click the properties button on the left menu. If the properties button is not available on the left menu, then type ?properties at the end of the url in your browser:

http://127.0.0.1:8080/MyFirstSuite?properties

From within the properties page make sure the "suite" checkbox is checked and uncheck the buttons you don’t want to appear in your left hand menu (I never use the files, refactor, search or versions button, so I turn these off but I enable edit if its not already).

Now we need to create the setup and teardown pages for our suite, again we can just type the names at the end of the url in the browser preceded with a '.' period:

http://127.0.0.1:8080/MyFirstSuite.SetUp

Within the SetUp page we declare the fixture we are using for the test, e.g the WebTest fixture

!|com.neuri.webfixture.WebTest|

The exclamation mark tells the wiki not to parse the text in the cell, otherwise is would look for a page called WebTest and think you want to create that page.

You can then put the startup command for selenium here, so for webtest (refer to the reference you printed out above) we use:

|Start Browser|firefox|With Selenium Console On|localhost|At Port|4444|And Scripts At|http://www.bbc.co.uk/|

Note: you would substitute the values for the selenium console (localhost) and its port (4444) depending on what machine you are driving and how you set selenium up. Here I'm assuming you are running the selenium server on your own machine, and it started on port 4444 (the default). But if you have a bit of savvy you should have just realise that you can drive any selenium instance remotely.

You can now save the SetUp page and create the TearDown page in the same way as we created the SetUp page, by typing the page name into the browser:

http://127.0.0.1:8080/MyFirstSuite.TearDown

We only need to put one thing in here, the instruction to shutdown the browser we opened in the SetUp page. again refer to the reference.

|shutdown browser|

Okay, now we have an empty suite.

Ok. Now lets create the test page, in the same way we created the other pages above, by typing the url:

http://127.0.0.1:8080/MyFirstSuite.TestOne

In here we will add a basic test using the WebTest reference to guide us:

Navigate to a URL during the test (change active page)

|user opens url|http://www.bbc.co.uk/radio2|

Check that the page loads fully within given time (use after user opens url)

|page reloads in less than|10|seconds|

Check current browser URL

|page url is|http://www.bbc.co.uk/radio2/|

Check that page contains some text (regardless of whether text is visible or not)

|element|title|contains text|BBC - Radio 2 - Home|

Save the page.


Edit the pages properties, either by the properties button on the left, or (if the button isn't available) by typing ?properties at the end of the url in the browsers address bar:

http://127.0.0.1:8080/MyFirstSuite.TestOne?properties

You can select the option you want, but make sure "Suite" is not checked, and "Test" is checked. Save the page.

You should now find that your test has the SetUp page and TearDown page included.

If selenium is running you should be able to run the test by clicking "test" on the menu.

There you have it, your first test suite. Try now to add TestTwo to your test suite.

Connecting to Oracle Grid with DbFit

While there is a good deal of info on the DbFit website, there is little about connecting to Oracle databases. The examples given didn't help us at all because we used two nodes in our grid, and we have to give both of them in our connection string (if one node doesn't respond the test will use the other).

!|Connect|(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=node1.grid.domain.net)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=node2.grid.domain.net)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=WEB1_DEFAULT_SERVICE)))| UserName | PassWord |

As you can see you can end up with a very long row in your table, but it does connect.

Using WebTest & DbFit together

If you are testing a web application, there is a good chance that you will need to manipulate or verify some data in a database as part of your testing of the front end application.

It would seem obvious to use DbFit and WebTest on the same test, however you should be aware of a few things.

WebTest is a DoFixture, and DbFit can work in two modes, flow or standalone, you will need to use the standalone mode of DbFit when using DbFit with another fixture..

When creating the SetUp page for the suite of tests add the fixtures like this:

!|com.neuri.webfixture.WebTest|
!|import|
|dbfit.fixture|
!|DatabaseEnvironment|sqlserver|
|Connect|server\SQLEXPRESS|FitNesseUser|Password|TestDB|

Note that there isn't a gap between DatabaseEnvironment and Connect, they must be in the same table

You can also connect to DbFit with the Connect table.

Also remember that since you're using standalone mode of DbFit, it wont rollback or commit, so you will need to use the following tables to commit or rollback transactions:

!|DatabaseEnvironment|
|Commit|

!|DatabaseEnvironment|
|Rollback|

We use the WebTest fixture and DbFit together and it works very well for us. However we have also extended WebTest to include some SQL features in WebTest that allow us to do simple SELECTS and use the output to verify what should be on screen without breaking the flow of the test.

I Hate FitNesse / Fitnesse Rocks!

I want you to know that i have a love hate relationship with fitnesse.

I have been using it for several months now and sometimes i really hate it; i still use it and advocate its use, but i still hate it. Tim Ottinger summed up using FitNesse really well over on the FitNesse group (yahoo) . He said that Using Fitnesse is not unlike trying to teach a dog to sing. Its not that the dog is a bad singer, its simply a miracle that it can sing at all.

And its true. FitNesse is not a great wiki, but it does do wiki. It doesn't work very well in CI, but it can be integrated into CI. It has its own versioning that isn't great (millions of zip files), but it can be versioned traditionally (using SVN or CVS). It doesn't have any metrics (test execution over time, defects, test coverage), but you can get those through CI or integration with another tool (we are experimenting with integration into Quality Center [sic]).

There is also the big issue of user management. There several authentication methods supported by FitNesse:

1. No authentication (default), anybody can do any thing!
2. UserPassword: Defines one user and its password on fitnesse start: ... -a userA:passworA
3. PasswordFile: For multiple users, read once on fitnesse start, so adding new users requires a restart ffs.
4. LinuxPAM plugin: Use PAM to access Linux user passwords, requires a plugin.
5. Provide your own AuthenticatorClass as plugin.

Given a choice i'd opt for the PasswordFile mode but then administration becomes tiresome.

For more info on user management checkout:

• http://fitnesse.org/FitNesse.StartingAndStoppingFitNesse
• http://fitnesse.org/FitNesse.CommandLineArguments
• http://fitnesse.org/FitNesse.PasswordFile
• http://fitnesse.org/FitNesse.PluginUsage
• http://fitnesse.org/PluginsPage.LinuxPamAuthenticator
  

What is a "story"?

I got asked today by my Boss, "what is a story?" so i gave him a 5 minute overview, then i realised i could share this.

Simply put a story is a very short (should fit on an description of a piece of functionality.
A story serves several purposes.
It is used to estimate and identify areas of risk during initial release planning and ongoing through development.
A story is always written from the business perspective, and is normally based on an action (e.g. display search results, search stock).
A story is testable through the UI if application is UI based.
Our Stories are written using the form: "As an X I want to Y so I can Z"
Stories answer these questions:

• Who is 'doing' the story (As an X).
• What action is being taken (I want to Y).
• What business need is the story hitting? (so I can Z).

They are at first very high level and i've found that initial story cards are almost always revised and rewritten. They are usually rewritten to lower levels of granularity so that at development time they are testable.

They are small enough that it only takes one iteration to complete. If they are too complex, they are expanded into narratives.
Stories can be either functional or non-functional, E.g. performance should be a story.

ThoughtWorks write the stories on ThoughtWorks branded index cards which are 147mm wide and 104mm high which are bigger than the index cards we could get from our stationers. (Who stocks these size cards? Viking have them on-line, but staples have them in store only. They are called 6x4 index cards by both)

They write on the cards to ensure the story is kept to a small size, this helps the customer "see" what they are saying. The cards are usually kept visible on a board. a Kanban board.

In Agile projects, it's common practice to share the project status via "big visible charts" on a wall in the project room (or elsewhere if a room is not available).

Kanban is a Japanese word which loosely translated, means card or sign. In a "Lean" production system, Kanban is a method which uses standard units or lot sizes (our story cards). The methodology dictates that a new card is only "pulled" into the system when the work represented by a card that is "in Play" is completed.

It works really well. I was away from the project for a couple of days to deal with recruitment. When i returned to the meeting i was able to jump straight in after reading the board for few minutes and attending the morning meeting.

Testers - Thou Shall Not Cut Code!

okay, so its been a while since my last post, and we have some very frustrating times, i'll try to remember them now.

One of our 500lb gorillas is the fact that Fitnesse out of the box comes with limited functionality. By that i mean if you want to pull a value from a database and assert that it is being displayed in the users browser (via selenium) then you have to write your own Java class. As i said before we are a team of dedicated web testers, and while we are very technical testers, we are not Java programmers. So we looked round for some possible solutions to our dilemma.

First up is Anubhava’s Generic Fixture. Anubhava has had the same thoughts as me, but has the skills to actually put the idea into fruition. His generic fixture allows the fitnesse scriptee to write in DSL without writing a single line of Java, eureka!

i tried using the generic fixture for connecting to the DB. However we use Oracle 10G and our connection string details two nodes. In the examples Anubhava gives, the fixture accepts the usual server, port, dbname, user, password strings.

I thought i would share this with any other visitors who may use generic fixture and use similar JDBC connection strings to us. OracleDataSource will accept a url, so i amened Anubhavas example fitnesse tables to

| Generic Fixture | oracle.jdbc.pool.OracleDataSource |
| setDriverType | thin |
| setURL | jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST =(ADDRESS=(PROTOCOL=TCP)(HOST=node1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=node2)(PORT=1521)))(CONNECT_DATA =(SERVICE_NAME=WEB)))
| setUser | db_user |
| setPassword | db_user_password |
| conn=getConnection |

and we can now connect to our DB, and script without writing Java.

Next up is Gojko Adzic's DbFit another fixture for fitnesse, and comes with some examples for .net and Java and Oracle.

A little bit more of a learning curve with this one, but still within the realms of what we can manage as testers. We can use DbFit with the WebTest fixture and again accomplish a lot without writing any Java.

I guess i should point out here that its not just writing the Java, its where to store it (CVS), compiling the code, keeping it up to date, making sure we are using the right versions of drivers (ojdbc14), that we have the right environment variables set-up and so on. All of a sudden the line between the bounds of a tester and developer have become so blurred. Well okay, maybe i wouldn't just test the "happy path" :-)

Automate This

A big bucket of rambley nonsense about automating web testing

Okay, so we have been trying to grapple with FitNesse and Selenium to produce automated "acceptance" tests for our continuous integration development.

Before we brought ThoughtWorks in i had already been looking at automated test tools. There are lots of tools out there, some very simple (yet effective) and other very complicated.

Some of these tools profess to be web testing tools, while others claim to be able to test anything. Having said that any tool that requires me to purchase an add-on or plug-in to achieve automated testing is going to get shot down straight away. Testing is core to what i do I'm a tester, so far as i'm concerned testing is not an add on.

I thought i would get my thoughts on automation down, so should the wheels of the fabled bus ever get chance to put my mothers under-cracker warnings to the test my ramblings can be copy and pasted into someone else' documentation.

During the usual testing activities we aim to undertake the following test activities:

1) Functionality Testing
2) Usability testing
3) GUI\Interface testing
4) Browser Compatibility testing
5) Performance testing
6) Security testing

The list isn't in any particular order and none is implied. But any seasoned tester will recognise that i'm talking about the upper right hand side of the V model, wheras FitNesse has traditionaly been used further down.

1) Functionality Testing:

It isn't immediately obvious what functional testing means when talking about web testing. If you are sat in front of a static html page, that is merely a funnel into a web application you are fairly limited in the amount of functional testing you can do. All web pages can be functionally tested for working includes (css, images, etc.) testing all the links, forms, images,JavaScript and cookies. All the individual elements that are used to make up a web page. Lets take a more in depth look at this.

Link checking:

Once aspect of functional testing of web pages, checks links for broken or malformed types of:

• Outgoing links
• Internal links.
• named anchors.
  
• links used to send email (mailto).

If its possible (its not always possible to get access the the disc) test for orphaned objects. Note that i didn't mention things like SEO within the context of functionality testing or links, because i see that as being non functional testing of a link. I also didn't mention that the link was tagged appropriately for disabled users, because again this is covered later on. What should be covered is that the link is tagged appropriately for whichever analytic packages is being used by the website.

Test forms:
Forms were previously essential for submitting user information, but thanks to web 2.0 they are starting to disappear. Nevertheless, they still need to be tested for

• input validations (where applied).
• default values of fields (where set).
• constraints (sting length, numerical, alpha etc).

I haven't mentioned the error handling for the actual submission be cause that is the responsibility of the functionality within url the form points at, and not the form itself.

Cookies:
Testing Cookies, can be a right PITA. Thankfully firefox and its numerous wonderful plugins like web developer and firebug have made our life a lot easier. Cookie testing normally covers:

• cookies enabled
• cookies disabled
• cookies deleted
• cookies set
• cookies read
• cookies updated
  

We often visually inspect the cookie to make sure the developer hasn't done something incredibly stupid like store the username and password in the cookie in plain text.

Database testing:
Data consistency is very important in web application. Check for data integrity and errors while you edit, delete, modify the forms or do any DB related functionality.
Check if all the database queries are executing correctly, data is retrieved correctly and also updated correctly. More on database testing could be load on DB, we will address this in web load or performance testing below.

2) Usability Testing:

Test for navigation:
Navigation means how the user surfs the web pages, different controls like buttons, boxes or how user using the links on the pages to surf different pages.
Usability testing includes:
Web site should be easy to use. Instructions should be provided clearly. Check if the provided instructions are correct means whether they satisfy purpose.
Main menu should be provided on each page. It should be consistent.

Content checking:
Content should be logical and easy to understand. Check for spelling errors. Use of dark colors annoys users and should not be used in site theme. You can follow some standards that are used for web page and content building. These are common accepted standards like as I mentioned above about annoying colours, fonts, frames etc.
Content should be meaningful. All the anchor text links should be working properly. Images should be placed properly with proper sizes.
These are some basic standards that should be followed in web development. Your task is to validate all for UI testing

Other user information for user help:
Like search option, sitemap, help files etc. Sitemap should be present with all the links in web sites with proper tree view of navigation. Check for all links on the sitemap.
“Search in the site” option will help users to find content pages they are looking for easily and quickly. These are all optional items and if present should be validated.

3) Interface Testing:
The main interfaces are:

Web server and application server interface
Application server and Database server interface.

Check if all the interactions between these servers are executed properly. Errors are handled properly. If database or web server returns any error message for any query by application server then application server should catch and display these error messages appropriately to users. Check what happens if user interrupts any transaction in-between? Check what happens if connection to web server is reset in between?

4) Compatibility Testing:
Compatibility of your web site is very important testing aspect. See which compatibility test to be executed:

• Browser compatibility
• Operating system compatibility
• Mobile browsing
• Printing options

Browser compatibility:
In my web-testing career I have experienced this as most influencing part on web site testing.
Some applications are very dependent on browsers. Different browsers have different configurations and settings that your web page should be compatible with. Your web site coding should be cross browser platform compatible. If you are using java scripts or AJAX calls for UI functionality, performing security checks or validations then give more stress on browser compatibility testing of your web application.
Test web application on different browsers like Internet explorer, Firefox, Netscape navigator, AOL, Safari, Opera browsers with different versions.

OS compatibility:
Some functionality in your web application is may not be compatible with all operating systems.
We Test our web application on different operating systems like Windows, MAC, Linux with different OS flavors. What can work in MSIE7 on Windows XP with service pack three, will fail onMSIE7 Windows Vista Home, but work on MSIE7 with Windows Vista Office Premium.

Mobile browsing:
This is a tough one. We get thousands of visitors a day using mobile devices, and even games consoles to view our site.

Printing options:
If you are giving page-printing options then make sure fonts, page alignment, page graphics getting printed properly. Pages should be fit to paper size or as per the size mentioned in printing option.

5) Performance testing:
Web application should sustain to heavy load. Web performance testing should include:
Web Load Testing
Web Stress Testing

Test application performance on different internet connection speed.
In web load testing test if many users are accessing or requesting the same page. Can system sustain in peak load times? Site should handle many simultaneous user requests, large input data from users, Simultaneous connection to DB, heavy load on specific pages etc.

Stress testing: Generally stress means stretching the system beyond its specification limits. Web stress testing is performed to break the site by giving stress and checked how system reacts to stress and how system recovers from crashes.
Stress is generally given on input fields, login and sign up areas.

In web performance testing web site functionality on different operating systems, different hardware platforms is checked for software, hardware memory leakage errors,

6) Security Testing:

Following are some test cases for web security testing:

• Test by pasting internal url directly into browser address bar without login. Internal pages should not open.
• If you are logged in using username and password and browsing internal pages then try changing url options directly. I.e. If you are checking some publisher site statistics with publisher site ID= 123. Try directly changing the url site ID parameter to different site ID which is not related to logged in user. Access should denied for this user to view others stats.
• Try some invalid inputs in input fields like login username, password, input text boxes. Check the system reaction on all invalid inputs.
• Web directories or files should not be accessible directly unless given download option.
• Test the CAPTCHA for automates scripts logins.
• Test if SSL is used for security measures. If used proper message should get displayed when user switch from non-secure http:// pages to secure https:// pages and vice versa.
• All transactions, error messages, security breach attempts should get logged in log files somewhere on web server.

Creating FitNesse tests

Okay, so we are still having problems with CruiseControl (it falls over after n builds) but meanwhile i need to start creating tests in Fitnesse.

The ThoughtWorks consultant has already started, but i just reviewed the work and i'm less than impressed.

I want each page to contain some prose that describes what's being tested and why.

The structure of a requirement should follow a basic syntax. We should have a basic description of what is needed from a business perspective:

As a [role] I want [requirement] so that [business value].

And the tests should line up by defining an Acceptance criteria, basically we define what "done" looks like:

Given [context] when [action] then [outcome]

The Acceptance criteria should be written jointly by BA and QA, and the acceptance tests map directly to acceptance criteria.

So on each page (a test) i want a preamble (the story), the requirement and the acceptance criteria.

The Fit table (the actual test) can then sit directly under the prose for the acceptance criteria.

That way, each page (each test) is the requirement, the test and the test results.

Integrating Fitnesse with CruiseControl

So like i said in earlier posts we are implementing FitNesse on our java development projects (well to be precise just one) with help from ThoughtWorks so we can run automated acceptance tests.

We are still at the bashing the stick on the ground and making a lot of noise stage (so far as my Space odyssey metaphor goes), and we are not yet 100% convinced of the value of FitNesse within our organisation. The way it has been set-up so far, makes it unmaintainable, hard to navigate and it feels a bit flaky. Sure FitNesse is great where the requirements are a little thin on the ground, but i think there is an absolute requirement for process and procedure when it comes to building a suite of tests in FitNesse if it is ever going to have longevity. That is not slur on FitNesse, but i have yet to see any one describe how they build a suite that has some structure.

So today we are trying to get FitNesse and CruiseControl working together.
The dev team have used CruiseControl in the past, but it doesnt work properly (some issue with forking) and depending on how we look at cobertura we either have 27% coverage or 35% coverage of Unit test so there is some way for them to go. I hope can integrate the FitNesse Tests with the current build process and gain an extra layer of testing.

One thing as a tester i dislike about FitNesse is that lacks a lot (does it have any) of the reporting features you see on other test tools. So i will spend this afternoon looking at some free (as in beer) reporting tools. I also cant see how you do test selection, or report on test coverage....

Fitnesse for dummies?

We pride ourselves on our technical prowess, by we i mean my team and I and by team i mean the greater QA team at work.

As testers we are often thought of as not being technical and in some institutions it is actively discouraged. However being a technical tester is dependent on the domain you work in.

In our domain we need to be holistic on our approach to testing, we need to be multidisciplinary with skills in Linux, Apache, Jboss, J2EE and Oracle. We work for a large website so we all need to understand headers, HTTP, cookies, dynamic HTML/Json.

So it was with some surprise (read, it caught us out) that we found ourselves staring at FitNesse (the Acceptance Testing Framework) like the apes staring at the monolith at the start of 2001: A Space Odyssey (1968). We understand its importance, and we can see the benefits, but how do we turn it on? what makes it go?

Well a few hours later and to our horror we have installed JDK and eclipse and we are coding in java. But this doesn't really sit well at all, testers writing tests in java? who will test our tests?

Worse than this, we don't have Java coding skills across the team (nor do we want them) so what can we do?

A few googles later and we can see that we are not the only people that think this sucks. For every test case we write in Fitnesse we ill have to write a corresponding fixture (read some java code). Until i run across Anubhava's Tech Blog and his post Introducing Generic Fixture for FitNesse. Here he talks of the same problem of having write java code for every test. His solution is rather neat, the GenericFixture.

I followed his examples and hit a few dead ends, so i emailed Anubhava and after a few question from him discovered that what i believed to be the GenricFixture.jar was in fact a 700Kb web page (an error message from sourceforge). With the proper jar downloaded from sourceforge we are off and running, and can now write tests without cutting a single line of java, but wait that's not all...

Something was still unsettling about the whole deal, and that was the fact that FitNesse allows users to create tests using natural language.

here is a simple example in fitness

users opens the URL	http://www.google.co.uk
page has the title		Google
page has an element named	q		true
page has an element named	btnG		true
user types	chocolate	into	q	field
user clicks on the button named	btnG
page loads in less than	5	seconds
page has the title		chocolate - Google Search
user clicks on the link named	Chocolate - Wikipedia, the free encyclopedia
page loads in less than	5	seconds
page has the title		Chocolate - Wikipedia, the free encyclopedia

and here is the same example in selenease

open	http://www.google.co.uk
assertTitle	Google
type	q	chocolate
clickAtAndWait	btnG
assertTitle	chocolate - Google Search
clickAtAndWait	link=Chocolate - Wikipedia, the free encyclopedia
assertTitle	Chocolate - Wikipedia, the free encyclopedia

As you can see the FitNesse example is easily read and understood whereas the selenim version requires the user to understand the selenium syntax (selenease).

Our friend Anubhava (above) had hit upon the same problem, and implemented a Domain Specific Language (DLS) adepter for the GenricFixture, so problem solved, for now...

Running Selenium with proxy exceptions

The way we have our QA environment set-up we use a DNS server this allows us to mimic the live environment (the customer facing server names are the same as in the production environment). It sounds ideal however it isn't without problems. We cant set-up a banner server in our environment (because we use a third party banner service) nor can we set-up a WA server (we use multiple WA vendors) so we have to use proxy exceptions that excludes everything from going through the proxy that exists in inside the QA environment.

This obviously means we can end up with a pretty big proxy exceptions list in our browser. Firefox isn't a problem as there are a couple of plug ins available that can manage multiple proxy settings. However IE requires us to use a hand crafted batch file to update registry settings. Not really a problem, and we are comfortable with how it works. It rarely catches us out.

Now when we cam to automate our tests with selenium we ran into a problem. We needed to tell selenium to use our corporate proxy so it can proxy request to the outside world, and when selenium starts the browser it tell the browser to use a particular PAC file that is generated at test run time. The PAC file is quite simple and tells the browser to use Selenium for anything that lies within the SUT else go external. sound's great except that it doesn't work.
We read a few forum posts, and blog posts and scratched our heads. Finally we opened the Selenium source code and found that we could in fact pass in a list of hosts to apply to the proxy exceptions list. We were feeling pretty jaded are spending hours of google time on it, why its not documented clearly anywhere i don't know.

Okay, so our startup command for selenium looked like this

java -Dhttp.proxyHost=proxy.ourdomain.co.uk -Dhttp.proxyPort=8080 -jar selenium-server.jar –avoidProxy

To add proxy avoidance you specify the hosts you want to avoid through the proxy and delimit with the pipe character |, but we then found we had to delimit the pipe character with the caret ^

-Dhttp.nonProxyHosts=*www.ourdomain.co.uk*^|*search.ourdomain.co.uk*

So our new selenium startup string looked like this

java -Dhttp.proxyHost=proxy.ourdomain.co.uk -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*www.ourdomain.co.uk*^|*search.ourdomain.co.uk* -jar selenium-server.jar –avoidProxy

This means that if the request is for anything other than www. or search . on our domain, selenium forwards it through our corporate proxy.

Of Agile, ThoughtWorks, Selenium and Fitnesse

I have set up this blog to capture some of the problems we are bound to run into over the next 60 days.

I work for a well known (read busy) commercial website in the UK. We currently use a traditional waterfall approach to our software development life cycle, and we are under pressure to deliver more, and deliver it more frequently.

I am an ISEB certified Test Practitioner and I supported by two testers on my Team. Martin, a test analyst, who has achieved his ISEB/ISTQB Foundation certificate in software testing, and Robert, a junior test analyst who will be taking his foundation training later this year.

We are all aligned to thinking about software testing in the same way. That is, we have a clear set of requirements (be they, technical, business or design) and we test against those; simple no?

However in the real world it is never that cut and dried, and often requirements are thin, late or non existent, and (of course) liable to change at the drop of a hat. This is compounded further by the businesses increasing frustration at having to wait for a simple change to be implemented; we are only talking about a website here right? It’s not like we are trying to boil the ocean.

Cue Music – Enter ThoughtWorks.

Somewhere along the way the business got to hear of ThoughtWorks, a company that can offer (among other services) a helping hand for any software house wishing to become Agile. I don’t want to talk too much about ThoughtWorks and our relationship with them, because I only want to focus on the Testing aspect of becoming Agile, and because I could fill another blog with chatter about working with ThoughtWorks.

Anyhoo, we are a Linux, Apache, J2EE & Oracle house. So it felt right that we should use Selenium for our automated testing. ThoughtWorks originally created selenium so it felt right to use it with them. We have looked at Selenium in the past and I have used selenium scripts with the most excellent Reality QA product from the guys at GOMEZ (more on that later).

A couple of interesting points about selenium; Selenium is a chemical element with the atomic number 34 and is represented by the chemical symbol Se. Selenium can be found naturally in nuts (Brazil nuts are the richest ordinary dietary source), cereals, meat (such as kidney), fish (shellfish such as crab and lobster), and eggs.
Selenium is often touted as "The cure for mercurial poisoning" (being poisoned by Mercury), and any tester can tell you that Mercury produced (before being bought by HP) some highly priced automated test tools of the variety that "require plug-ins" to do anything more ordinary than record and playback (Test Director for instance). So I think the developers of selenium thought it would be funny to name their product Selenium. I think however that the last laugh may be on them, because selenium is not actually an antidote to mercury poisoning, it’s just a chelating agent for heavy metals including mercury and although selenium is an essential trace element, it is toxic if taken in excess. There is no known antidote to selenium poisoning.