From zero to automated web application testing

UI automation is a difficult domain to work in, and there aren't any magic "best practices" other than one: use your brain.
-- Jim Holmes, VP of Testing at Falafel Software

Okay, that's not really helpful. Let's do something about it. We'll build a logical progression from "I'm lost" to "I know what tools to use and what code to write". We'll also skip over unnecessary steps and obsolete technologies so you'll end up using the current best practices (as of 2015) - much like many African nations have skipped landlines to go from no electronic communications infrastructure to wireless 4G.

This article will take you from very little knowledge about automated acceptance testing for web applications, to having a good understanding of the best practices as of 2015, and how to get started quickly. It includes a comparison of Node.js WebDriver client libraries, cloud-based cross-browser testing services, and test framework considerations.

Web app testing with BDD, Selenium WebDriver, and JavaScript

To build a reliable web app and prevent regression bugs, you need automated tests. These tests should cover the critical paths of your app as taken by a real user using a real browser. For example: after adding the latest feature, can the user still log in and check out their shopping cart? Other than that, you also need unit tests for your app's modules, but those are usually easier to reason about given the much more limited scope.

In terms of methodology, the most modern testing+development approach is BDD (Behavior-Driven-Development), which is the successor of TDD (Test Driven Development).

Since JavaScript is rapidly becoming the best language for building web apps (especially with the advent of ES2015), this post will focus on testing web applications built with JavaScript, using BDD.

The goal is to cut through the confusion and enable you to make a clear choice among the multitude of players in the automated web app testing arena: BDD, TDD, Jasmine, Cucumber, Gherkin, Mocha, Chai, QUnit, WebDriver, Selenium RC, webdriver-sync, PhantomJS, CasperJS, ZombieJS, SlimerJS, WebDriverIO, SauceLabs, BrowserStack, TestingBot, CrossBrowserTesting etc.

BDD vs. TDD

BDD is the next step after TDD, created with the goal of giving more structure to tests. If we aim for reducing the paradox of choice, then this is an easy one: BDD wins for making your app testing more future-proof and easier to collaborate across the organization. If your app is very small, you may not benefit much from BDD, but it's a useful skill to pick up.

Implementing BDD in JavaScript

BDD tests are written in a language similar to English, called Gherkin. What really happens is that a BDD test framework such as Cucumber or Yadda will parse the Gherkin files against regular expressions, then execute assertions.

Why you'd want to use Gherkin is explained very well in Why use user story based testing tools like Cucumber instead of other TDD/BDD tools?. Here's an example:

@Tag
Feature: One-line description of the feature

Anything can be used here to provide context for this specification.
You may want to use the story format below.

As an <actor>
I want <feature>
So that <benefit>

Background: The background is optional
  Given something that happens before every scenario
  And something else that happens before every scenario

@AnotherTag
Scenario: One-line description of the scenario
  Given some pre-condition with a parameter "myParam"
  And another pre-condition
  When an event is triggered
  Then an expected result happens
  But an unexpected result does not happen

Note that Cucumber is more than a testing tool; it's actually best used as a collaboration tool between business stakeholders, developers, and QA folks. A tool called Yadda is preferred by some over CucumberJS. Yadda's advantages aren't clearly presented in its README, beyond being "less invasive" and having a more "flexible syntax" than Gherkin, and "smarter conflict resolution". From Yadda's raison d'etre post:

The only true BDD tool I found during my search was CucumberJS. The most popular tools, Jasmine, Mocha, Vows, Chai, should.js are proclaimed as BDD, but they're not.
-- Stephen Cresswell, author of Yadda, an alternative to CucumberJS

Neither Cucumber nor Yadda provide an assertion library. Chai and Chai-as-promised are most frequently used with both.

Selenium vs. CasperJS, ZombieJS, SlimerJS and other headless browsers

The short answer is that Selenium is the slowest, but it uses ("drives") a real browser. If you want to ensure cross-browser compatibility, Selenium is the best choice. If you want to run tests fast, headless browsers are better. However, they may lack "real browser" features such as precise drag-and-drop or using Flash (if you use ZeroClipboard, this may be important, until they drop Flash support altogether), and they may use different JavaScript engines (e.g. PhantomJS uses QtWebKit and there are no plans to move away from it).

Ideally the tests you write should be applicable without change to a headless browser first, so you can quickly validate functionality as you develop, and then to a bunch of real browsers automated via Selenium to check cross-browser compatibility. This is possible thanks to WebDriver, an API that can "send tests" to various browser implementations, including Chrome, Firefox, PhantomJS, or some mobile browsers.

Selenium, WebDriver, Cucumber, Jasmine, and all that jazz

If you're new to testing, you might be confused by the choice of technologies. Here's the key info:

  1. To automate tests, you need browsers to automate. Ideally you would write test scripts and have them run on various browsers to test your app for cross-browser compatibility. You can do this locally if you have multiple combinations of OSes and browsers and mobile devices, or you can run testing in the cloud. There are at least four decent cloud-based cross-browser testing providers:

    • CrossBrowserTesting - less known but takes pride in using real devices. Free to use for manual testing of CodePen.io projects, so you can get a good idea of its capabilities.
    • SauceLabs - probably the best known; free for open source projects, with a nice badge. No real devices.
    • BrowserStack - boasts using real devices instead of emulators, but it's not ready for prime time yet (Apple real devices generate lots of random errors)
    • TestingBot - doesn't use real devices, and the set of browsers is somewhat reduced.

    Check the Slant link above for more pros and con about each.

  2. To automate browsers, there's pretty much only one solid choice (thankfully) - Selenium. Selenium is a set of tools that includes:

  3. For web applications developed in JavaScript, it makes sense to use a JavaScript Selenium WebDriver client, even though Selenium development has been traditionally done in Java. There are (at least) three categories of Node.JS libraries for using Selenium:

    • the original selenium-webdriver,
    • those that simplify its syntax (e.g. WebdriverIO) or make it synchronous (webdriver-sync), and
    • WebDriver clients that include a testing framework (e.g. Nightwatch, Intern).

When does it make sense to use each, and which are the most effective Selenium Node.JS clients?

WebDriver vs. testing

First off, let's note that WebDriver clients only implement the JSON Wire WebDriver Protocol (which is used to control a browser), and connect to a WebDriver server (either local, such as ChromeDriver, or remote). The communication happens over HTTP, via POST requests. For example, to click an element in the browser, the WebDriver client will send POST request to /session/:sessionId/element/:id/click.

In turn, the WebDriver server communicates with the browser, using apparently yet another protocol.

To automate tests, besides a WebDriver client and a WebDriver server, you'll also need a testing framework (for JavaScript, popular ones include Jasmine, Cucumber, Mocha, QUnit) and an assertion library (Chai for Mocha; other testing frameworks usually include the assertion library). Here's a piece of code that highlights each of these components:

var webdriver = require('selenium-webdriver');

var driver = new webdriver.Builder().
    withCapabilities(webdriver.Capabilities.chrome()).
    build();
// WebDriver initialization code above

describe('basic test', function () {  // test framework code - test organization
    it('should be on correct page', function (done) {  // test framework code - one test
        driver.get('http://google.com');  // WebDriver command
        driver.getTitle().then(function (title) {  // WebDriver command
            expect(title).toBe('Google');  // test assertion
        });
        done();  // test framework code signalling end of async test
        driver.quit();  // WebDriver command
    });
});

The test framework lines pertain to Jasmine. The expect line is an assertion. Jasmine includes assertions, but some test frameworks, such as Mocha, don't, and you can use any assertion library you want, such as Chai. The other lines of code are calls to the WebDriver Node.JS library, in this case selenium-webdriver.

Notice how the getTitle() call is asynchronous - it waits for the browser to fetch the page and then uses Promises to call the callback. This is an unfortunate artefact of JavaScript which makes tests scripts unnecessarily verbose, but there are workarounds for it.

Node.JS WebDriver client libraries compared

There are again at least 7 interesting WebDriver client libraries implemented in Node.js. Let's start to eliminate choices.

Since we want to use BDD for testing, we'll look for libraries that officially support Cucumber. That excludes Nightwatch despite it taking care of more boilerplate than WebdriverIO. It also excludes the fantastically well-developed Intern, until it implements Cucumber support.

We are left with five candidates, of which we can skip two early on:

Below are the final three.

selenium-webdriver

This is the official implementation, with a rather rubbish syntax/API. Documentation, wiki.

driver.get('http://www.google.com');
driver.findElement(webdriver.By.id('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.id('btnG')).click();

WebdriverIO

WebdriverIO (formerly WebdriverJS) has a much simpler syntax than selenium-webdriver:

client
  .url('http://google.com')
  .setValue('#q','webdriver')
  .click('#btnG')

Chaining commands is easy, thanks to WebdriverIO using Chainit, a mechanism similar to selenium-webdriver's ControlFlow. You don't have to end up in callback hell.

WebdriverIO also lets you use any testing framework you want and has examples for Cucumber, Jasmine and Mocha. One downside is that you have to launch Selenium yourself. selenium-standalone helps with that, but it's a bit slower than selenium-webdriver.

webdriver-sync

It has a synchronous syntax which is very nice:

driver.get("http://foo.html");
title = driver.getTitle();
link  = driver.findElement(By.id('i am a link'));
link.click();
assert(driver.getCurrentUrl().indexOf('foo title 2') > -1);
title.should.equal('foo title');
console.log(title);
driver.quit();

Yet it still uses the selenium-webdriver driver.findElement(By.... syntax.

Since there are efforts to implement sync in WebdriverIO, and you can already use yield, we can consider WebdriverIO as a winner. It's also actively developed, has a Gitter chatroom where the main developer (who works at SauceLabs) answers questions promptly, and has recently passed 1,000 GitHub stars.

Refinements

If you have WebDriver APIs in your test methods, You're Doing It Wrong.
-- Simon Stewart.

If your tests simply look for elements by ID or class name, then interact with them, and you change the underlying HTML code, you'll have to repeat that change in every test that accesses elements using those constants. The problem is essentially the same as repeating constants in your code instead of declaring a const.

To avoid this, a pattern called PageObject can be used. It abstracts interactions with pages from a user's perspective, hiding implementation details to reduce duplicated code and make tests easy to update when the UI changes.

PageObject diagram from Martin Fowler

Here's an example from the Moonraker test framework:

// tests/pages/home.js
var Page = require('moonraker').Page;

module.exports = new Page({

  url: { value: '/' },

  txtSearch: { get: function () { return this.element("input[id='txtSearch']"); } },
  btnSearch: { get: function () { return this.element('btn-primary', 'className'); } },

  searchFor: { value: function (query) {
    this.txtSearch.sendKeys(query);
    this.btnSearch.click();
  }}

});

Each page has a URL and any convenient methods that you may require. You can then use your page objects in your step definitions:

// tests/steps/home-search-steps.js
var homePage = require('../pages/home'),
    searchResults = require('../pages/search-results');

exports.define = function (steps) {

  steps.when("I search for '$query'", function (query) {
    homePage.txtSearch.sendKeys(query);
    homePage.btnSearch.click();
    // Or use homePage.searchFor(query);
  });

};

Note that "PageObject" doesn't mean you should have only one object per page. A more accurate name could be "PanelObject", but the term "PageObject" has been popularized by Selenium. Another benefit of using the PageObject pattern is that it makes the test code easier to understand because its logic is about the intention of the test and not cluttered by UI details.

Putting it all together

WebdriverIO itself doesn't yet have a best practice for using the Page Object pattern, and its Cucumber boilerplate actually uses Yadda. Moonraker uses selenium-webdriver with its complex syntax. Wouldn't it be nice if there were some package that integrated Selenium, WebdriverIO, CucumberJS and the Page Object pattern? Well, there is - ChimpJS.

Chimp integrates CucumberJS, Selenium, WebdriverIO and Chai, and has some really neat features:

  • synchronous style
  • built-in "widget framework" (an implementation of the PageObject pattern)
  • automatically downloads dependencies (ChromeDriver, PhantomJS etc.)
  • works with SauceLabs and BrowserStack (CrossBrowserTesting TBD)
  • automatically takes screenshots on failures
  • works on Windows in addition to Linux and OS X
  • Cucumber automatically outputs boilerplate code for undefined steps, which you can copy, paste and edit
  • file watcher reuses the browser sessions and automatically re-runs tests when you save
  • using @watch tags, you can run only the tests you tag, to maximize development speed
  • works with continuous integration services like Travis or CircleCI
  • is used by xolvio:cucumber, the preferred package for Velocity, which is the official test framework for the excellent Meteor platform.

Check out the Chimp video:

Next, go through the tutorial to get started.

That's it - use Chimp. I was surprised at how easy everything is with Chimp, and it's one of those little known (only 21 GitHub stars at the moment) diamonds in the rough that can really speed up your development - if you know about it.

Now you do.

Prior art

Here are the articles I have summarized and coalesced into this one. None of those mentioned Chimp, but they're what shows up when you Google for combinations of "WebdriverIO", "Cucumber" and the like.

See also

My tags:
 
Popular tags: