Notes from Jeff Nyman’s “Test Description Language” blog post series

What is a test description language? How do we use that language to effectively write tests? In a series of blog posts, Jeff Nyman talks about what TDL is and how to write better tests with them. The posts are mostly written in the sense of how Cucumber checks are being used for writing tests for automation, but the lessons can also be applied in writing test cases if ever there is a need to write them for clients.

Some notes from the series of posts:

  • It’s very easy to confuse the essence of what you’re doing with the tools that you use.
  • Just because I’m using TestLink, Quality Center, QTP, Cucumber, Selenium, and so forth: that’s not testing. Those are tools that are letting me express, manage, and execute some of my tests. The “testing” part is the thinking I did ahead of time in discerning what tests to think about, what tests to write about, and so forth.
  • Testable means that it specifically calls out what needs to be tested.
  • When examples are the prime drivers, you are using the technique of specification by example and thus example-driven testing. When those examples cover business workflows, you are using the technique of scenario-based testing.
  • Requirements, tests, examples — they all talk about the same thing. They talk about how a system will behave once it’s in the hands of users.
  • The goal of a TDL (test description language) is to put a little structure and a little rigor around effective test writing, wherein “effective test writing” means tests that communicate intent (which correspond to your scenario title) and describe behavior (the steps of your scenario)
  • It is now, and always has been, imperative that we can express what we test in English (or whatever your common language is). This is a key skill of testers and this is a necessity regardless of the test tool solution that you use to manage and/or automate your testing.
  • So many tests, and thus so much development, falters due to a failure to do this simple thing: make sure the intent is clear and expressive for a very defined unit of functionality.
  • Software development teams are not used to the following ideas: The idea of testing as a design activity. The idea of test specifications that are drivers to development. The idea of effective unit tests that catch bugs at one of the most responsible times: when the code is being written.
Advertisements

Learning How To Write Better Cucumber Scenarios

Some of the important lessons in writing automated checks are found, not in the actual implementation of the check itself, but rather in the specification. What does a green check mean? What are we really trying to find out when we’re running this check? Will somebody from another team understand why this test was written? Does this check say what’s necessary in a feature or does the check only state a procedure without context? Too often we concentrate on syntax, frameworks, and required steps in building automation, but not so much on clearly expressing what’s being checked and why the check is recorded in the first place. I’ve made that mistake and now I am trying to learn how to write better checks.

Take, for example, the cucumber checks below. Would you say that it is clear what’s being verified in the test? What would a product owner probably say about these checks when they see them for the first time?

Scenario Outline: Validating the Rate Plan Policies
  Given I am in the ShowRoomsPage for a "<hotel_type>" property from "<arrival>" to "<departure>" for a "<rateplan>" rate plan for a "confirmed" reservation
  When I view the policies for the rate plan
  Then I know that these are the correct policies
  
  Examples:
  | hotel_type | arrival         | departure       | rateplan            |
  | DWH        | 4 DAYS FROM NOW | 6 DAYS FROM NOW | Public_Partial_LT   |
  | DWH        | TOMORROW        | 3 DAYS FROM NOW | Public_Full_YesR_LT |

Some of the questions that would probably pop up in their minds include the following: How are the examples in the grid necessary for the test? What does Public_Partial_LT or Public_Full_YesR_LT mean? Do we really need to know the arrival and departure date settings for the checks? What does the Given statement mean? Most importantly, how do I know that the policies are actually correct for these tests?

This was how I wrote my checks before when I started studying how to write Cucumber-Watir-Ruby checks. And a lot of my checks in the existing test suite still are. I am, however, trying to learn how to re-write them in a better way, in terms of readability and conciseness, guided by lessons so far learned from Jeff Nyman’s test description language blog posts, so that almost everybody in our team can recognize in a glance what a particular check does and why they are included in the feature.

Re-writing the example checks above, I now have these:

Scenario: ShowRooms Prepayment Policy, DWH Partial
  When guest views the policies for a DWH property for a partial rate plan
  Then a copy of "Only 10% prepayment is required to confirm your reservation" is displayed in the prepayment policy

Scenario: ShowRooms Prepayment Policy, DWH Full Non-Refundable
  When guest views the policies for a DWH property for a full nonrefundable rate plan
  Then a copy of "Full prepayment is required to confirm your reservation" is displayed in the prepayment policy

Not perfect, but I’d like to think that they are more effective than the ones I had before. Checks are more specific, unrelated information are not included in the test, and we understand what it means when the checks pass or fail.

What do you think?

Takeaways From Rikard Edgren’s “The Little Black Book On Test Design”

Rikard Edgren’s test design e-book has been online since 2012, free for download and consumption, but I only knew about the book recently. It is a must read for beginner software testers. The book is short, only 32 pages long including the bibliography. Here are a few takeaways:

  • Learn a lot, because it is more likely that testers (more often) will know the important stuff when they do.
  • Testers who devise their own methods for testing learn far more than those who blindly follow scripted test cases written by somebody else.
  • Complete testing is a myth. Even if a software tester is able to execute all possible application functions and unit tests, some things will remain unknown to her and will never be replicated like user needs and feelings and data. This means that testing is a sampling problem, and test design means finding out test ideas that covers the most important test samples when executed.
  • Always ask what problem testing is trying to solve. Challenge the requirements.
  • A test that is testworthy is  a test that we think is likely to deliver information of some value about the app in test, even if the approach is risky, even if the requirement is not explicitly defined, or even if the process is time-consuming.