Thoughts After a Recent Software Feature Release

Having a feature released to the customers does not mean I won’t have to test that feature anymore.

When a feature is released months later than it’s initially estimated, it could mean that the project was just a complex thing. Or it could mean that we did not give the thing enough attention to have it broken up and released into possibly much smaller chunks.

It’s terribly challenging to find confidence in releasing a new massive functionality which took months to finish, especially when the feedback you have about its quality is limited. I am being reminded that a part of testing is solving the problem of having good enough test coverage of product risks. What are those risks? What tests do I want to perform in order to cover those risks?

Just like how meetings that take hours get extremely boring, it gets annoyingly tiring for teams to spend more than a month of time building a feature before publishing it in public.

If we don’t spend some of our time now tinkering with systems that could possibly help us accomplish things better in the future, we’ll end up doing the same things as before and that reflects into our ability to grow.

Just because a defect happens to quietly slipped its way into production code does not mean the team did not work their asses off in bringing the best work they could possibly do to the project, under the circumstances. And just because a team worked hard does not guarantee they will write code free of any bug. What matters is what people do when those flaws are found. And there will always be flaws.

Some people, even testers, are unnecessarily loud when they find problems in production after a feature release. This just means that they did not have any idea how that part of the feature was built in the first place and did not care to explore it either before the release. They only assumed that everything is ought to be working. They forgot that quality is a team responsibility.

It’s important that we understand why a problems exists in production code (and maybe who did it) but I’m not sure if it’s particularly helpful that we leave the problem fixing later to the person who made the defect when we have the time and the know-how to fix it ourselves now. What’s the boy scout rule again for programmers? Yes, leave the code better than you found it.

Just because a shiny new feature is released to the public does not mean it has actual value to customers. And if we don’t find ways of getting user feedback, we’ll never find out whether what we built amounts to something or not.

Feature releases are always short-lived, like the physical store opening day launch.  There’s always a celebration, an attraction, but it’s not what really matters in the long run. What matters are the lessons learned in the duration of building the feature, what we intend to do with those lessons, the people we worked with, and building things that our customers love.

Learning How To Write Better Cucumber Scenarios

Some of the important lessons in writing automated checks are found, not in the actual implementation of the check itself, but rather in the specification. What does a green check mean? What are we really trying to find out when we’re running this check? Will somebody from another team understand why this test was written? Does this check say what’s necessary in a feature or does the check only state a procedure without context? Too often we concentrate on syntax, frameworks, and required steps in building automation, but not so much on clearly expressing what’s being checked and why the check is recorded in the first place. I’ve made that mistake and now I am trying to learn how to write better checks.

Take, for example, the cucumber checks below. Would you say that it is clear what’s being verified in the test? What would a product owner probably say about these checks when they see them for the first time?

Scenario Outline: Validating the Rate Plan Policies
  Given I am in the ShowRoomsPage for a "<hotel_type>" property from "<arrival>" to "<departure>" for a "<rateplan>" rate plan for a "confirmed" reservation
  When I view the policies for the rate plan
  Then I know that these are the correct policies
  | hotel_type | arrival         | departure       | rateplan            |
  | DWH        | 4 DAYS FROM NOW | 6 DAYS FROM NOW | Public_Partial_LT   |
  | DWH        | TOMORROW        | 3 DAYS FROM NOW | Public_Full_YesR_LT |

Some of the questions that would probably pop up in their minds include the following: How are the examples in the grid necessary for the test? What does Public_Partial_LT or Public_Full_YesR_LT mean? Do we really need to know the arrival and departure date settings for the checks? What does the Given statement mean? Most importantly, how do I know that the policies are actually correct for these tests?

This was how I wrote my checks before when I started studying how to write Cucumber-Watir-Ruby checks. And a lot of my checks in the existing test suite still are. I am, however, trying to learn how to re-write them in a better way, in terms of readability and conciseness, guided by lessons so far learned from Jeff Nyman’s test description language blog posts, so that almost everybody in our team can recognize in a glance what a particular check does and why they are included in the feature.

Re-writing the example checks above, I now have these:

Scenario: ShowRooms Prepayment Policy, DWH Partial
  When guest views the policies for a DWH property for a partial rate plan
  Then a copy of "Only 10% prepayment is required to confirm your reservation" is displayed in the prepayment policy

Scenario: ShowRooms Prepayment Policy, DWH Full Non-Refundable
  When guest views the policies for a DWH property for a full nonrefundable rate plan
  Then a copy of "Full prepayment is required to confirm your reservation" is displayed in the prepayment policy

Not perfect, but I’d like to think that they are more effective than the ones I had before. Checks are more specific, unrelated information are not included in the test, and we understand what it means when the checks pass or fail.

What do you think?

Five People and Their Thoughts (Part 3)

There’s always something new to learn, and these days watching webinars and recorded conference videos are good ways of finding and sharing interesting content about what practices do other people think is important in software testing. Lately I am finding myself getting more curious about performance and security testing, as well as the idea of software (including test code) as specification.

Some engaging thoughts about those topics, if you’re interested:

Defining What Cucumber-Watir-Ruby Checks To Run by Passing Parameters

Like any other type of document, automated checks can build up to so many files. There can be many features we would like to run tests on and it can be hard to keep track of all them every time. Maybe we prefer writing varying groups of checks in one single file, which cucumber does not restrict us from doing. We may also like to run our checks on different test environments, browsers, or locations that does not necessarily require a change in a feature to test but rather only an adjustment in configuration. Fortunately, there are ways to perform all of this without remembering numerous commands to run our checks. Here are some of them:

  • Cucumber Tags (-t)
    Using Cucumber tags is an easy way of defining groups of tests in a feature file, so we can choose to only run a select group instead of running everything when we test a particular feature. Here is an example of a feature file (with sample filename of policies.feature) with tags (displayed in bold, characterized by the @ symbol):

      Feature: Booking Engine Rate Plan Policy Copies

       @prepay @partial
       Scenario: ShowRooms Prepayment Policy, DWH Partial
         When guest views the policies for a DWH property for a partial rate plan
         Then a copy of 'Only 10% prepayment is required to confirm your reservation' is displayed in the prepayment policy

       @prepay @full
       Scenario: ShowRooms Prepayment Policy, DWH Full Non-Refundable
         When guest views the policies for a DWH property for a full nonrefundable rate plan
         Then a copy of 'Full prepayment is required to confirm your reservation' is displayed in the prepayment policy

       Scenario: ShowRooms Reservation Policies, DWH Partial, Lead Time Early Modification
         When guest views the policies for a DWH property from 4 DAYS FROM NOW to 6 DAYS FROM NOW for a Public_Partial_LT rate plan
         Then a copy of 'We don't charge you a modification fee if you choose to modify before' is displayed in the modification policy

    If we want to run the prepayment policy copy tests (the first and second scenarios in the example), we can run those checks as:

      cucumber policies.feature -t @prepay

    And if we want to run only the single reservation policy copy check in the example feature, we can run that as:

      cucumber policies.feature -t @reservation

    More information about how to use cucumber tag are found in the Cucumber tags Wiki page.

  • Custom Environment Variables
    We can create custom variables we can use for specifying how we want our tests to run. For example, we might like to run the same feature test for a different browser, or on a mobile application instead of on a desktop web app.

    To define what custom variable we want to use for our checks (say, a browser type), we could write the following code in the env.rb file:

      def browser_type
        (ENV['BROWSER'] ||= 'chrome')

    which means that we are stating a browser type variable of name ‘BROWSER’ we can use for describing which browser we would like to use during the test run.

    If we want to run the policies feature check on a firefox browser (assuming that functionality is built in our test code), we could say that as:

      cucumber policies.feature BROWSER=firefox

    And if we want to run the same checks on a chrome browser, we can either say that the browser type variable should equal to ‘chrome’:

      cucumber policies.feature BROWSER=chrome

    or just omit the browser type variable altogether, since the browser defaults to the value of ‘chrome’ based on the code above:

      cucumber policies.feature

  • Cucumber Profiles (-p)
    Eventually there will be a situation where we would need to use common cucumber commands, tags, or custom environment variables in running our checks, and typing all of those may prove tiring and unproductive. For example, would you feel good typing in

      cucumber policies.feature FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty -t ~@not_ready BROWSER=firefox

    to run the policy copy checks (not including the checks that are not yet ready) for the firefox browser in the staging test environment with test results display in the terminal displayed with color and formatted properly? I personally wouldn’t enjoy writing all of that, and this is where cucumber profiles help. To use profiles, we could state desired profiles in the cucumber.yml file, like so:

      default: FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty --tags ~@not_ready
      dev: FIG_NEWTON_FILE=dev.yml --no-source --color --format pretty --tags ~@not_ready
      uat: FIG_NEWTON_FILE=uat.yml --no-source --color --format pretty --tags ~@not_ready
      parallel: FIG_NEWTON_FILE=staging.yml --no-source --color --format pretty --tags ~@not_ready --format html --out reports/parallel_.html

    where in the example above we can see that there are four profiles, namely: default, dev, uat, and parallel.

    When we want to run the same checks as before in the uat test environment, we would only need to type in the following:

      cucumber policies.feature -p uat BROWSER=firefox

    And when we want to run the same checks in staging (where we would usually run them), we can even omit writing the profile since the default profile runs whenever the profile is not defined:

      cucumber policies.feature BROWSER=firefox

    More information about how to use cucumber profiles are found in the Cucumber cucumber.yml Wiki page.

About failing Automated Watir checks on Firefox v.48 and successfully running them on Google Chrome v.52

The recent updates on the Mozilla Firefox browser starting from version 46 onwards broke my automated end-to-end checks. The browser loads when started but does not go to a test page or do anything until it eventually fails, because the latest versions of Firefox supposedly doesn’t use the FirefoxDriver anymore for automation and instead makes use of a new driver implementation in what is called Marionette. In order to get my checks running again in Firefox, I had to resort (as what many others in the community also did) to using the Extended Support Release version of Firefox v. 45, a temporary measure until I finally get my Cucumber-Watir checks running properly on the latest Firefox version.

At the moment I am stumped by an error when running automated checks on Firefox v. 48.0.1 using Marionette (which seems to be a problem on permissions on my local Windows machine, and not on watir or selenium, although port 4444 points to the port which selenium grid is connected to):

Permission denied - bind(2) for "::1" port 4444 (Errno::EACCES)
C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.4/lib/selenium/webdriver/firefox/service.rb:103:in `stop_process': undefined method `poll_for_exit' for nil:NilClass (NoMethodError)
from C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.4/lib/selenium/webdriver/firefox/service.rb:83:in `stop'
from C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.4/lib/selenium/webdriver/firefox/service.rb:64:in `block in start'
from C:/Ruby/lib/ruby/gems/2.2.0/gems/selenium-webdriver-2.53.4/lib/selenium/webdriver/common/platform.rb:161:in `block in exit_hook'

So, while I’m trying to sort out that problem (anybody experienced this?), I decided to move my checks to run on the latest version of Google Chrome by default instead of running them on an old version of Firefox. To do that, I needed to update my browser capabilities:

if browser_type == :chrome
    arguments = "--ignore-certificate-errors" // Add more desired arguments
    capabilities = "chromeOptions" => {"args" => [ arguments ]}
elsif browser_type == :firefox
    capabilities = Selenium::WebDriver::Remote::Capabilities.firefox
browser = :remote, :url => url, :desired_capabilities => capabilities

And all that is left is to see how the checks behave on Google Chrome.

Some findings:

  • Button or link elements in the browser (especially when they are only span or heading elements that are being used functionally as buttons) that are not immediately visible in the Chrome viewport fail to be clicked properly. To fix this, focus on the element first (or alternatively call a scroll method in the browser using javascript to a position where the element becomes visible in the viewport) before running the click step.
  • Some wait methods that didn’t necessarily have to be written (when checks were run in Firefox) need to be explicitly stated when running checks on Chrome.

Takeaways from Jeff Nyman’s “Modern Testing” blog posts series

Jeff Nyman recently wrote a series of blog posts about modern testing, describing what he thinks it could be, while also introducing his idea of a lucid approach in testing. The posts themselves are not easy to read but they contain well-formed thoughts about artifacts, acceptance criteria, testing, software engineering, among other things, which deserve a look through. Thinking about his ideas on manual tests, acceptance criteria, user interfaces, software development constraints, and sources of truth was refreshing and insightful and I wonder how he would implement these ideas in a specific tool.

Quoting some of the words that I liked from the series of posts:

  • Tickets – in systems like JIRA or whatever you use – are entirely transient. They are there simply to allow us to track and measure various work (tasks, bugs, etc). They are not there to store acceptance criteria or to frame epics that have numerous tickets associated to them. Acceptance criteria should live as close as possible to the code, be versioned with the code, capable of being elaborated upon by the code pushing out relevant information. The elaboration of that acceptance criteria should be the code.
  • Each development project is a journey of discovery. And the real constraint isn’t really time or budget or some metric of man-hours for programmers or testers. The real constraint is the lack of knowledge about what needs to be built and how it should be built.
  • The fact is that manual test and an automated test are really just checks. What differs is what is doing the execution: human or machine. The testing part comes in when we have the conversations and collaborations to decide what in fact should ultimately be checked. Test tooling comes in with how that information is encoded as code and what kind of framework will support that encoding. Tests should be reflective of the work that was done by development. This means that the testing framework – whatever that happens to be – must be good enough to quickly encode the business understanding and push the business intent.
  • The closer the tests sit to the code, the closer the tests are aligned to the business criteria, the less artifacts you have to use to manage everything. When you have less artifacts, being lucid becomes quite a bit easier because your sight lines of what quality means are not obfuscated by an intermediating set of artifacts stored in a variety of tools, only some of which are directly executable.
  • Take testing out of the acceptance criteria writing business. Take out the focus on the right way to generate acceptance criteria. Don’t (necessarily) focus on Given/When/Then for acceptance criteria.
  • You want to enable fast and useful feedback which in turn allow you to fail-fast. When you can fail fast, it means you have safe-to-fail changes. When your changes are safe-to-fail, that means you can worry less about failure and more about experimenting with ideas, creating working solutions quickly, and even abandoning work that is going down an unfruitful path.
  • Going fast is not possible unless the quality is under control at all times, so we need a testing strategy that says testing is a design activity and automation is a core part of the strategy. Going fast on the long term is not possible unless the quality of the design and of the understanding of the business domain is under control at all times, so we need a executable source of truth strategy so that it’s possible to reason about the system at all times.
  • Any application is the end product at any given moment in time. That end product is a reflection of a set of features working together. The value of the application is delivered through its behavior, as perceived through its user interface. Behavior and user interface are critical there. That’s what your users are going to base their view of quality in. And, to be sure, user interface can mean browser, mobile app, desktop app, service (i.e., API), and so on. It’s whatever someone can use to consume some aspect of behavior of the application. So everything comes down to the behavior and the interfaces and the consumers of those interfaces.
  • Any act of testing acts as a means to aid communications, to drive a clean, simple design, to understand the domain, to focus on and deliver the business value, and to give everyone involved clarity and confidence that the value is being delivered.

A Basic Guide to Creating XML API Automated Checks

Checking out the Agoda YCS5 API

API testing is interesting. They’re very different from functional user interface testing because they do not require any UI, although programmers can provide testers some user interface for checking them if needed. They’re actually pretty straightforward to perform, they’re relatively easy to code, and they also provide fast feedback, unlike UI tests which often breaks easily, complicated to code, and are generally slow. I’ve been focusing on writing automated API checks for some apps in recent weeks and found out that the process was simple.

And here’s a basic guide of what that process looks like, in three steps:

  • Build the request content
    A basic XML request document source looks like this:

    <?xml version="1.0"?>
       <element attribute="value">
       possibly a chain of more elements will show here

    In creating checks, this request source is just treated as a String data type, and then converted later to XML if necessary. I usually strip the source into three parts, namely header, footer, and body, and then just combine the parts. Usually it’s just the body part of the request that changes for scenario checks. How the actual request looks like, of course, is often found in the documentation of the API that you desire to test.

    For our example:

    header = <request>
    body = <element attribute="value"></element>
    footer = </request>

    The body can be further stripped down into many parts if desired, as it happens that some requests can contain multiple elements and sub-elements.

    In building a request, I just create a method that I can call anytime:

    def build_request(request_type)
       return get_header(request_type) + get_body(request_type) + get_footer(request_type)

  • Send the request to the API endpoint, and retrieve the response
  • In API testing, what we check is the response of the API when we send a particular type of request to it. And to be able to generate a valid response from the API, we need to send our request to a valid endpoint. Sometimes that endpoint needs access credentials, sometimes it is the request themselves which contain said credentials. These details are often requested from the people who manage the API, or are sometimes found in your account information if the API requires you to create an account.

    As for how automation can send a request and generate a response, I often use the rest-client gem this way inside a method:

    def get_response(endpoint, request)
         response = endpoint, request, {:content_type => :xml}
       rescue => e
         response = e.response
       return response

  • Validate the API response
  • Checks are of course not complete if there are no actual comparison between actual and expected results. This defines whether the check passes or fails, and by default any check will pass if there are no conditions about response failure (except for failure of the test to reach the API endpoint).

    For the example request, this might be one of the ways I want to validate the API response using the nokogiri and rspec gems (assuming that a successful response contains a result element with a count attribute of value 1 inside it):

    def validate_response(response)
       expect(response.xpath(//result).attr('count').to_s).to eq('1'), "Result count should be 1"

    In the end, the implementation of how to properly validate responses vary. There are scenarios where tests need to check for error elements in the response, and there are some tests where data in the response also needs to be verified.