Set Cucumber to Retry Tests after a Failure

This is probably old news, but I only recently found out that there is a way for Cucumber tests running in Ruby to automatically retry when they fail.

Here’s a sample failing test:

Apparently we can just add a --retry command to set the test to retry if it fails. And we can set how many times we want the test to retry. If we retry the failing test above to retry two times, with a --retry 2 command, we’ll get the following output:

It did retry the test twice, after the initial failure. Neat! 🙂

But what happens if we run the test, with the same retry command, and the test passes? Does the test run three times? Let’s see:

Good, the retry command gets ignored when the test passes.

Advertisements

Notes from David Bryant Copeland’s “Build Awesome Command-Line Applications in Ruby 2”

The experience of writing and running automated checks, as well as building some personal apps that run on the terminal, in recent years, has given me a keen sense of appreciation on how effective command-line applications can be as tools. I’ve grown fond of quick programming experiments (scraping data, playing with Excel files, among others), which are relatively easy to write, powerful, dependable, and maintainable. Tons of libraries online help interface well with the myriad of programs in our desktop or out in the web.

Choosing to read “Build Awesome Command-Line Applications in Ruby 2” is choosing to go on an adventure about writing better CLI apps, finding out how options are designed and built, understanding how they are configured and distributed, and learning how to actually test them.

Some notes from the book:

  • Graphical user interfaces (GUIs) are great for a lot of things; they are typically much kinder to newcomers than the stark glow of a cold, blinking cursor. This comes at a price: you can get only so proficient at a GUI before you have to learn its esoteric keyboard shortcuts. Even then, you will hit the limits of productivity and efficiency. GUIs are notoriously hard to script and automate, and when you can, your script tends not to be very portable.
  • An awesome command-line app has the following characteristics:
    • Easy to use. The command-line can be an unforgiving place to be, so the easier an app is to use, the better.
    • Helpful. Being easy to use isn’t enough; the user will need clear direction on how to use an app and how to fix things they might’ve done wrong.
    • Plays well with others. The more an app can interoperate with other apps and systems, the more useful it will be, and the fewer special customizations that will be needed.
    • Has sensible defaults but is configurable. Users appreciate apps that have a clear goal and opinion on how to do something. Apps that try to be all things to all people are confusing and difficult to master. Awesome apps, however, allow advanced users to tinker under the hood and use the app in ways not imagined by the author. Striking this balance is important.
    • Installs painlessly. Apps that can be installed with one command, on any environment, are more likely to be used.
    • Fails gracefully. Users will misuse apps, trying to make them do things they weren’t designed to do, in environments where they were never designed to run. Awesome apps take this in stride and give useful error messages without being destructive. This is because they’re developed with a comprehensive test suite.
    • Gets new features and bug fixes easily. Awesome command-line apps aren’t awesome just to use; they are awesome to hack on. An awesome app’s internal structure is geared around quickly fixing bugs and easily adding new features.
    • Delights users. Not all command-line apps have to output monochrome text. Color, formatting, and interactive input all have their place and can greatly contribute to the user experience of an awesome command-line app.
  • Three guiding principles for designing command-line applications:
    • Make common tasks easy to accomplish
    • Make uncommon tasks possible (but not easy)
    • Make default behavior nondestructive
  • Options come in two-forms: short and long. Short-form options allow frequent users who use the app on the command line to quickly specify things without a lot of typing. Long-form options allow maintainers of systems that use our app to easily understand what the options do without having to go to the documentation. The existence of a short-form option signals to the user that that option is common and encouraged. The absence of a short-form option signals the opposite— that using it is unusual and possibly dangerous. You might think that unusual or dangerous options should simply be omitted, but we want our application to be as flexible as is reasonable. We want to guide our users to do things safely and correctly, but we also want to respect that they know what they’re doing if they want to do something unusual or dangerous.
  • Thinking about which behavior of an app is destructive is a great way to differentiate the common things from the uncommon things and thus drive some of your design decisions. Any feature that does something destructive shouldn’t be a feature we make easy to use, but we should make it possible.
  • The future of development won’t just be manipulating buttons and toolbars and dragging and dropping icons to create code; the efficiency and productivity inherent to a command-line interface will always have a place in a good developer’s tool chest.

Post-Installation Notes after Setting-Up Cucumber-Ruby-Watir Automated Checks with Jenkins on a Windows 10 Amazon EC2 Server

Had a chance to set up cucumber tests to run via Jenkins on a Windows 10 Amazon EC2 server last week. It’s been a while since I installed everything from scratch, so that was good practice. I ran into some problems, which was a bit expected since my local setup was on Windows 7, so this post serves as a guide for possible future installations.

The process:

  • Install desired browsers where the tests will run.
  • Install Java and the latest JDK. Include the JDK path to Window’s system environment variables. This is necessary for running the selenium server.
  • Install the latest Ruby and DevKit versions, required for installing ruby gems and running the test suite.
  • Install Jenkins, which will take care of scheduling and test results reporting. Also install Jenkins as a service, to run automatically after the server boots up.
  • Install Git on the machine, and setup the credentials for retrieving the test code from the remote repository.
  • Check out the test code from the remote repository.
  • Set the selenium server to run automatically after the machine boots.
  • Install the required ruby gems using bundler.
  • Check if cucumber tests run properly by running a sample via the command line interface.
  • Create/Import Jenkins jobs for the cucumber tests. Install the Jenkins plugins necessary for test results reporting. Possibly set up Jenkins to pull the latest test code from the remote repository.
  • Run cucumber tests via Jenkins.

Some reminders for future installs:

  • Install the 32-bit version of Ruby’s DevKit instead of the 64-bit because the latter version creates problems with gem installs.
  • Windows 10 has trouble installing ruby gems (using the bundle install command) from the default HTTPS source. To fix this, replace the gems source list in the Gemfile from HTTPS to HTTP.
  • Install Jenkins using the MSI-based Windows installer on its own directory on the C drive, then install it as a service afterwards.
  • Do not install all suggested Jenkins plugins at first, especially the ones related to Git. These will break Jenkins on Windows so install only those that are actually needed for cucumber tests to run.
  • Via Jenkin’s Configure System module, set the shell executable to the location of the command line’s executable file (i.e., C:\Windows\system32\cmd.exe) so that Jenkins can run cucumber tests via the default Windows command line interface. In addition, each Jenkins job Build needs to be set as “Execute Windows Batch Command” instead of “Execute Shell”.
  • Remember to change the timezone of the machine to the proper timezone.
  • Test a sample basic cucumber project first to see if cucumber is properly running on the machine before retrieving the actual test suite from the Git repository.
  • In saving the machine’s generated public SSH identification (via Putty or the ssh keygen command using Cygwin), do not include a name for the file to be saved so that the system generates the default id_rsa.pub file.

We Refactor When We Learn Better Ways Of Doing

For the past month or so, inspired by the lessons I learned from Jeff Nyman’s test description language blog post seriesGerard Meszaros’ talk about test abstractions, and Codecademy’s online class about the Ruby programming language, I have been steadily performing good chunks of code refactoring of our existing cucumber-watir-ruby end-to-end functional checks, which included (but not limited to) the following:

  • Cucumber
    • Removing information from cucumber steps that do not directly affect what is being checked in tests
    • Explicitly describing what the cucumber check is for, instead of using vague Then statements
    • Using step commands for re-using repeated cucumber steps
    • Renaming variable, method, and class names used for assertions so they mean what they actually mean
  • Ruby
    • Applying double pipes to set default variable values when necessary
    • Using one-line ifs and unlesses whenever possible, for readability purposes
    • Converting simple multi-line if-else statements to one line whenever helpful using the ternary/conditional operator
    • Replacing + or << operators with string interpolation
    • Using symbols for hash keys instead of string names because they’re faster to process
  • Locators
    • Making xpath locators shorter/readable whenever possible using the // modifier

It’s all difficult work, possibly more strenuous work than writing new checks even though basically I’m just rewriting existing tests into a more readable and maintainable form. I’m breaking existing code because they’re not so easy to understand and then revising them in order for their intent to be more understandable in plain view, finally making sure they’re checking whatever they’re supposed to, so that they act as a better source of truth for knowing how our applications behave and deliver value.

Defining What Cucumber-Watir-Ruby Checks To Run by Passing Parameters

Like any other type of document, automated checks can build up to so many files. There can be many features we would like to run tests on and it can be hard to keep track of all them every time. Maybe we prefer writing varying groups of checks in one single file, which cucumber does not restrict us from doing. We may also like to run our checks on different test environments, browsers, or locations that does not necessarily require a change in a feature to test but rather only an adjustment in configuration. Fortunately, there are ways to perform all of this without remembering numerous commands to run our checks. Here are some of them:

  • Cucumber Tags (-t)
    Using Cucumber tags is an easy way of defining groups of tests in a feature file, so we can choose to only run a select group instead of running everything when we test a particular feature. Here is an example of a feature file (with sample filename of policies.feature) with tags (displayed in bold, characterized by the @ symbol):

      Feature: Booking Engine Rate Plan Policy Copies

       @prepay @partial
       Scenario: ShowRooms Prepayment Policy, DWH Partial
         When guest views the policies for a DWH property for a partial rate plan
         Then a copy of 'Only 10% prepayment is required to confirm your reservation' is displayed in the prepayment policy

       @prepay @full
       Scenario: ShowRooms Prepayment Policy, DWH Full Non-Refundable
         When guest views the policies for a DWH property for a full nonrefundable rate plan
         Then a copy of 'Full prepayment is required to confirm your reservation' is displayed in the prepayment policy

       @reservation
       Scenario: ShowRooms Reservation Policies, DWH Partial, Lead Time Early Modification
         When guest views the policies for a DWH property from 4 DAYS FROM NOW to 6 DAYS FROM NOW for a Public_Partial_LT rate plan
         Then a copy of 'We don't charge you a modification fee if you choose to modify before' is displayed in the modification policy

     
    If we want to run the prepayment policy copy tests (the first and second scenarios in the example), we can run those checks as:

      cucumber policies.feature -t @prepay

    And if we want to run only the single reservation policy copy check in the example feature, we can run that as:

      cucumber policies.feature -t @reservation

    More information about how to use cucumber tag are found in the Cucumber tags Wiki page.

  • Custom Environment Variables
    We can create custom variables we can use for specifying how we want our tests to run. For example, we might like to run the same feature test for a different browser, or on a mobile application instead of on a desktop web app.

    To define what custom variable we want to use for our checks (say, a browser type), we could write the following code in the env.rb file:

      def browser_type
        (ENV['BROWSER'] ||= 'chrome')
      end


    which means that we are stating a browser type variable of name ‘BROWSER’ we can use for describing which browser we would like to use during the test run.

    If we want to run the policies feature check on a firefox browser (assuming that functionality is built in our test code), we could say that as:

      cucumber policies.feature BROWSER=firefox

    And if we want to run the same checks on a chrome browser, we can either say that the browser type variable should equal to ‘chrome’:

      cucumber policies.feature BROWSER=chrome

    or just omit the browser type variable altogether, since the browser defaults to the value of ‘chrome’ based on the code above:

      cucumber policies.feature

  • Cucumber Profiles (-p)
    Eventually there will be a situation where we would need to use common cucumber commands, tags, or custom environment variables in running our checks, and typing all of those may prove tiring and unproductive. For example, would you feel good typing in

      cucumber policies.feature FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty -t ~@not_ready BROWSER=firefox

    to run the policy copy checks (not including the checks that are not yet ready) for the firefox browser in the staging test environment with test results display in the terminal displayed with color and formatted properly? I personally wouldn’t enjoy writing all of that, and this is where cucumber profiles help. To use profiles, we could state desired profiles in the cucumber.yml file, like so:

      default: FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty --tags ~@not_ready
      dev: FIG_NEWTON_FILE=dev.yml --no-source --color --format pretty --tags ~@not_ready
      uat: FIG_NEWTON_FILE=uat.yml --no-source --color --format pretty --tags ~@not_ready
      parallel: FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty --tags ~@not_ready --format html --out reports/parallel_.html

    where in the example above we can see that there are four profiles, namely: default, dev, uat, and parallel.

    When we want to run the same checks as before in the uat test environment, we would only need to type in the following:

      cucumber policies.feature -p uat BROWSER=firefox

    And when we want to run the same checks in staging (where we would usually run them), we can even omit writing the profile since the default profile runs whenever the profile is not defined:

      cucumber policies.feature BROWSER=firefox

    More information about how to use cucumber profiles are found in the Cucumber cucumber.yml Wiki page.

Steps In Starting A Cucumber-Watir-Ruby ATDD Automation Project using TestGen (Windows)

I said that I was going to lay low on automation this year. However, I was also still curious about implementing a Watir-Ruby counterpart of my existing Java-Webdriver automation framework, including an acceptance-test driven development (ATDD) or a behavior-driven development (BDD) functionality built-in, just so I can understand how different the two frameworks are, based on my own experience. Long story short, I learned how to do it, with help from Jeff Morgan’s “Cucumber & Cheese” ebook, and was surprised how easy it was to start, and eventually write code.

If you ever want to try building a Cucumber-Watir-Ruby project from scratch on a Windows machine, try this:

  1. Download the latest version of Ruby for your machine
  2. Run the installer, ticking the following options when prompted
    • Install Tcl/Tk support
    • Add Ruby executables to your PATH
    • Associate .rb and .rbw files with this Ruby installation
      CucumberWatirRuby_setup_1_rubyinstall
  3. Run the command prompt, a shortcut is by pressing the Window and R keyboard buttons at the same time, typing in cmd, and then pressing Enter
    CucumberWatirRuby_setup_2_cmd
  4. Test the Ruby installation by running ruby -v
    CucumberWatirRuby_setup_3_rubyversion
  5. Close the command prompt if the Ruby version is displayed in the screen. If not, try re-installing Ruby or checking if Ruby is set in your machine’s Path under system environment variables
  6. Download the Ruby Development Kit for your version of Ruby
  7. Create a new folder under C:\ directory called devkit
  8. Extract the Development Kit files to the newly created directory
    CucumberWatirRuby_setup_4_devkit
  9. Run the command prompt again
  10. In the command prompt, navigate to the devkit directory by running cd C:\devkit
  11. Once inside the directory, type in ruby dk.rb init and then press Enter
  12. After that, type in ruby dk.rb install and press Enter
    CucumberWatirRuby_setup_5_devkitinstall
  13. Then let’s install some gems by running this: gem install cucumber testgen rake bundler yard watir-webdriver page-object fig_newton
    CucumberWatirRuby_setup_6_gems
  14. When the gems finish installing, go back to the command prompt and run cd C:\Users\Your_Machine_Name or go to a desired directory where you want to house your automation project
  15. Still in the command prompt, run testgen project Project_Name –pageobject-driver=watir. This should create a new directory structure inside your project folder which cucumber will use to run your automated checks.
    CucumberWatirRuby_setup_7_testgen
  16. Create a test_filename.feature file (without anything in it) inside your project directory
  17. Test if the project works by going to the project directory and then running cucumber test_filename.feature in the command prompt
    CucumberWatirRuby_setup_8_cucumber

If you encounter problems with a ffi_c file, such as a “cannot load such file — ffi_c (LoadError)” error, doing the following may help:

  1. Run gem uninstall ffi
  2. Then run gem install ffi –platform ruby

If the ffi_c issues persists, uninstalling the latest version of Ruby, installing an older Ruby version (say v2.1.8) and redoing Steps 3-17 may fix the problem.

Note: For a Mac or an Ubuntu machine, the Ruby and Devkit installations may differ.