Speeding Up a Suite of Web Application Automated End-to-End Checks

Speed is always an issue when running a full suite of automated end-to-end checks for web applications because they are inherently slow. Tests open a browser on a desired machine, navigate to a required application state (and hence often wait for the app to load completely), do an action on the system, validate results, and finally perform cleanup if necessary. A single check runs for some good number of seconds especially if it goes through several phases, and this means that a suite of tests can easily total to an hour or more. It’s easy to add tests. It’s difficult to have the full suite of tests run in under only a few minutes, especially when the only infrastructure available is only a local machine.

Some ideas on speeding up automated end-to-end checks:

  • Bundle related tests together in a group, running on top of each other, the resulting state of one test becomes the requirement of the next test. This is risky. What happens when the first test fails? The next test does not run anymore, even if the test probably passes. But we don’t know that because the test did not run. Ideal automated checks should be independent of each other, should be as atomic as possible.
  • Run tests in parallel. TestNG does this with Java, the parallel_tests gem can do it with Ruby. Jenkins (among other continuous integration tools) can run multiple jobs if fancied. Remember however that running automated checks in parallel is a processor-intensive task and has some tendency to fail tests because of system lag, timing, or application state sharing between tests, especially on low-end machines. Investing in dedicated infrastructure (whether physical computers or software-as-a-service systems on the cloud like SauceLabs) definitely helps, but the decision for that depends on the organization’s financial prioritization.
  • Minimize end-to-end checks as much as possible. This is viable for systems built with a healthy dose of existing unit and component tests within the application stack, usually teams who practice good test-driven development (TDD) or behavior-driven development (BDD) when building software. For a lot of teams without such unit tests to fall back to, reducing the number of working automated end-to-end checks will prove to be difficult.
  • Use headless browsers. I haven’t had much experience running checks on headless browsers but people do use them, often successfully using xvfb on Ubuntu.
  • Leverage REST calls. How does the browser load a page URL? How are users are redirected from one page to another? How are form values submitted properly when a call-to-action button is clicked? Using REST calls, one can log in on a system with the right credentials, retrieve data from where it is displayed, or send data to the application without opening a browser. We can use that for automation, either for simulating a desired application state or for testing responses to manipulated requests whether that’s data or the resulting page. This can help our tests run faster, the only catch is that the tester will have to learn how REST calls work and (more importantly) understand how the web application under test runs under the hood of the browser.

5 thoughts on “Speeding Up a Suite of Web Application Automated End-to-End Checks

  1. I can’t state how important that last point is. This helps not only with speed of test execution but also has a huge impact on stability.

    Anything that is not specifically being tested should not go through the browser UI. If the test needs to be in a specific state to run (on the correct page, logged in, items in basket etc.) then this should all be done with API calls as part of the test setup.

    Any automation tester should be testing at the service layer anyway so this is a really valuable skill to learn.

    • Yes, a new world opens up when testers learn how to re-create web app functionalities they know how to run manually via API/REST calls. And I wonder why testers in the community do not specifically talk about this skill, I have not read anything about it before in blogs I follow or in books I’ve read. I only got an idea about using it when I recently studied Burp Suite, when I thought about how nice it would be if I can automate those GET and POST requests via code instead of using the app.

  2. Also, on the point of headless browsers, I currently use PhantomJS through the Serenity framework. Even though it is headless it is still able to record screenshots which can be really useful.

    • I’m curious about Serenity, especially since Jeff Nyman talks about it in his blog. Maybe I’ll try it next year, and learn some Groovy and Geb and Spock. Do you have tutorials/guides you can share about how to set up that framework?

  3. Pingback: Testing Bits – 11/13/16 – 11/26/16 | Testing Curator Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s