Posted on by & filed under django, programming, testing.

In the first post of this series I described how to write a Selenium test for a Django app which can run in an assortment of different browsers just by changing an environment variable.  This works fine as long as you only want to run tests on the OS you use for development, but out in the real world people are using web browsers on a variety of different operating systems.  In order to upgrade our tests from simulating “some plausible users of the site” to “almost any possible user of the site”, we need to get tests running on a variety of different operating systems.  And we’d like to do this without having to get our entire development environment working on each OS; we don’t really care if our server can run on every possible operating system, just that the browsers for each OS can access and use it properly.  This means having our test server and test code running on one machine while the automated browser is running on one or more other machines.

Selenium can be run as a “remote control” service which waits for instructions on how to drive any browser installed on that computer.  To utilize a remote control for our Django Selenium tests, we can update the setUp method of SeleniumTestCase as follows:

The actual configuration of the remote control driver is implemented like this:

Note that we’re using environment variables again to specify how the test is to be run, so the test code itself doesn’t have to make any assumptions about it.  You may also notice that this looks very similar to the code in the original setUp() method for Safari; the Safari driver for Selenium is relatively new and not yet explicitly supported in the Python bindings, so the original method assumed that when driving Safari we have a Selenium remote control instance running in the default location of localhost port 4444.

If you have an assortment of different computers serving as remote controls, you can either specify for each test which one to use or set up a Selenium Grid.  In a grid, one computer serves as the hub which receives all requests for browser automation and distributes them to a computer in the grid capable of performing them.  Each computer in the grid needs to have Java, Ant, and Selenium Grid installed.

So, great; now we can run tests on our machine, or on another machine on the network which is more suited to the task.  But this still means we need a lot of different machines on our network if we want to be able to run our tests in a good variety of different operating systems and browser versions (necessary because those pesky users of our site haven’t agreed to all use the same browser yet).  It takes time to set them all up, and unless we run tests all the time or have other uses for those computers they’re going to be sitting there idle wasting electricity most of the time.  Oh right, that’s what the cloud is for!  Sure enough, Selenium Grid has instructions specifically for setting up a grid on Amazon EC2, and it shouldn’t be too hard to use a different vendor if your preference lies elsewhere.

So now we can set up an on-demand grid…assuming we have an account with a cloud computing vendor, teach ourselves how to launch and shut down computing resources, figure out how to get everything set up, script the process so it doesn’t take too long each time, and wait for one or more virtual machines to boot up every time we want to run a test.  If you’re a system administrator, there’s a decent chance that sounds interesting and well worth pursuing; if you’re a programmer or a QA engineer, odds are it sounds a lot like that system administration stuff you’ve been trying not to deal with any more than absolutely necessary.  For those people, there’s a company called Sauce Labs that thinks it has a solution for you.

Sauce Labs is basically a pre-built cloud of Selenium remote control nodes that you can access through a simple API, and then check out the results of the tests online.  They keep some of the more common testing configurations standing by so you usually don’t have to wait for one to boot up; it’s already set to go.  They also have spiffy results pages that show the Selenium instructions which were sent, output from the browser’s JavaScript console if any, screenshots of each step, and a video of the whole test.  It isn’t a free service, so you have to do your own cost-benefit analysis of whether or not it’s worth the money for you, but for a lot of people it’s a convenient way to bypass a lot of effort in setting up their own local or cloud testing grid.  If we want to add explicit support for running tests at Sauce Labs in our all-purpose test superclass, we can do it as follows:

The actual configuration can be done as follows:

Note that this is pretty similar to configuring a remote control driver.  The main difference is that you need to specify a username and API key for the Sauce Labs service so they know who to bill.  Note that we also pass along the name of the test method which is about to be run; this makes it easier to distinguish our tests in the Sauce Labs web interface.  If we just do this much and run our tests, we’ll notice that although we can see test successes and failures indicated on our console, Sauce Labs can’t tell the difference; they don’t know when our test code throws an exception or fails an assertion.  So if we want to be able to tell at a glance at their UI which ones failed, we need to tell them:

If there was an error in the test or an exception was thrown due to an assertion failing, our tearDown() method recognizes this.  And whether the test passed or failed, if we’re running the test on Sauce Labs we send them a brief JSON message indicating that particular test result.

So now we have lots of different options on where to run our tests…but we haven’t really talked much about how to write the tests themselves.  I’ll take a look at that in part 3 of this series.


3 Responses to “Writing a Selenium Test Framework for a Django Site (Part 2)”


  1.  Writing a Selenium Test Framework for a Django Site (Part 3) | Digital publishing and technology posts from the team at Safari Books Online
  2.  Optimizing JavaScript in a Django Project: django-require « Safari Books Online: Publishing & Technology