Acceptance Testing for Web App

I'd like share an approach we use at Qubell to ensure quality of our product. A set of practices that allows us to use Acceptance Testing in simple and efficient manner.

Note: this article is my personal opinion and can't be associated with Qubell's official view on the matter in any way.

Test Case

Let's try to express the test case with plain english first. A little about the product itself

  • the Qubell Platform simplifies deployment. You write requirements for your Application, hit the launch button - and your Application get delivered into production.

On the screenshot below you can see list of sample applications.

Application List

So, here's the test case:

  1. Create an Application (to make it more interesting we'll create it via REST API)
  2. Select our Application in the list of Application Items and click the Launch button.
  3. Type the name of new Instance of Application in the opened Dialog and finish the launch.
  4. Navigate to the newly created Instance of our Application and check it's ok.

You can see the record of test case run below (the slowed down version, usually it takes about 150ms to complete)

Now let's see how the code looks and examine each line in detail.

it('should launch application', function() {
  var applicationId = browser.post('/applications', {name: 'WordPress'}).id

  browser.navigate('/applications')

  var applicationItem = browser.eventually.applicationItem(applicationId)
  applicationItem.click('Launch actions...')
  applicationItem.eventually.menu().click('Advanced Launch...')

  var dialog = browser.eventually.dialog('Launch WordPress')
  dialog.type({name: 'My Blog'})
  dialog.click('Launch')

  var instance = browser.eventually.instance('my-blog-id')
  instance.should.have.text('My Blog')
  instance.should.have.text('Running')
  browser.path().should.contain('/instances/my-blog-id')
})

As you can see the code is compact and close to what we described in plain english before. There's no CSS gibberish, we used terms with meaning like applicationItem and instance. Now let's see how it works internally.

Meaningful names vs. CSS selectors

As I mentioned before the business logic should be defined in a simple way, that means that it should stay close to english and don't contain any gibberish.

Also, it should be clear what's going on by reading tests so it should use only those things that are visible on the screen to the user and not require any knowledge about the underlying implementation. So, we use click('Launch') not CSS or XPath stuff like click('.launch-button').

The elements are selected by the text they contain, so when you write click('Launch') - it will find the element that contain the Launch text. It ensures that there's only one such element and throw exception if none or many found instead.

How to deal with icons? If you look carefully at the click('Launch actions...') - you'll notice that there's no button with such text on the screen, instead the button contains a gear icon, without any text. The trick is to provide any icon button with the title attribute and extend the text search to look also at this attribute.

Composite vs. Ambiguity

Let's consider what happens when we want to click on the Launch button. As you can see on the picture below - there are multiple Application Items on the page, and each of it contains the Launch button.

Application List

So, if we just write browser.click('Launch') - there's an ambiguity and it will throw an error that multiple such elements found.

One way to solve it is to provide more detailed selector, like CSS or XPath - but as I told before it makes tests less readable, so we will use another approach. Similar to one used by humans naturally.

If we consider our application we notice that it have hierarchical, composite structure and consist from components. So, when we told to human to click on the Launch we leverage this knowledge and provide additional information, we say:

click the Launch button on the Application 'App 1'

Here we use same approach, when there's an ambiguity - we provide additional information and tell what part of screen or component we mean:

browser.applicationItem(id).click('Launch')

The good practice is to be consistent in CSS naming and mark every significant Component of Web Application with corresponding classes - like application-list, application-item and application-item-id (see more in maintenance section).

Eventuality

Sometimes there are pause between the action performed and the result became visible. For example in our test case there are couple of such moments. When we navigate to the list of applications it won't appear instantly, instead it perform asynchronous request to the server and only then render the list of applications.

One option is to explicitly wait for some time, like wait(100ms) - but, there's huge disadvantage - your test would be slow and fragile:

browser.navigate('/applications')
wait(100)
var applicationItem = browser.applicationItem(id)

Thankfully there's better approach - use special eventual expectations. It means that the result should be available right now or a bit later:

browser.navigate('/applications')
var applicationItem = browser.eventually.applicationItem(id)

It also has another, functional form (the previous one is just a shortcut of this full form):

browser.navigate('/applications')
var applicationItem = eventually(function(){return browser.applicationItem(id)})

Internally it's implemented as function trying every let's say 5ms to check if the condition is meet and throw the timeout exception if condition hasn't been meet for given amount of time (let's say for 2sec).

Unlike tests with the explicit wait the eventual tests are fast and robust.

API calls

Sometimes you need not only click on the buttons but also to talk to the server directly, it's also easy, in our case we use REST API, it is also possible to use other protocols (MessageQueues, Databases etc.).

var applicationId = browser.post('/applications', {name: 'WordPress'}).id

Synchronous Code

JavaScript is asynchronous and asynchronous code is much harder than synchronous, so we use Fibers to make it synchronous and add support for continuation (ability to pause the execution).

Language - JavaScript

There are adapters for different languages to control the Browser, so you can use for the tests practically any programming language you want. I prefer JavaScript because it's relatively simple and flexible, easy to start with, already known by many people and it's the language of the Web.

Also, you probably anyway would need the Unit Tests to test some of your JavaScript components more thoroughly - so it's easier to use the same language and infrastructure for both Unit and Acceptance Tests.

Maintenance

It is nice to use explicit names like applicationItem in the tests instead of CSS gibberish, but the downsize is that you need to define it first. Let's see how complex it is?

Below is the code for declaring names we used in our test case:

var proto = Browser.Context.prototype

proto.applicationItem = function(id){
  return this.context(id ? '.application-item-' + id : '.application-item')
}

proto.instance = function(id){
  return this.context(id ? '.instance-' + id : '.instance')
}

proto.menu = function(id){return this.context '.menu'}

proto.dialog = function(id){return this.context '.dialog'}

You may notice the context function - it returns the context - an object that has most of the methods of the Browser but all queries would be scoped to this context only (the Browser is also a Context scoped to the whole document).

If you maintain the CSS naming consistent, you can make it even simpler:

var addComponent = function(name){
  var proto = Browser.Context.prototype
  proto[name] = function(id){
    return this.context(id ? '.' + name + '-' + id : '.' + name)
  }
  proto[name + 'Item'] = function(id){
    return this.context(id ? '.' + name + '-item-' + id : '.' + name + '-item')
  }
}

addComponent('application')
addComponent('instance')
addComponent('menu')
addComponent('dialog')

So, the maintenance cost is very small (in our case, the tests for Qubell Platform - all such definitions take a page and half).

Q&A

Q: What about Unit and Integration Tests?

A: Done similarly, for more details about the QA in general and balance between Acceptance, Unit and Integration Tests please take look at Quality of Product.

Q: What tools can be used for Acceptance Testing?

A: You need to have the Browser you can control programmatically - to run the Web Application inside of it and performs different checks defined in the tests.

There are different tools available, we use WebDriver (with Selenium or Phantomjs), node.js and mocha.js.

Q: Would it be possible to make tests even simpler and use plain english?

A: Yes, it is possible to write Acceptance Test using plain english, but it would impose additional maintenance cost and not always reasonable. Maybe I'll write about such technic in the next article.