An Experiment on Executable Specifications

What: Create executable specifications for an ongoing project sprint. Executable specifications are written examples of a business need that can be run anytime and acts as a source of truth for how applications behave. It is a living documentation of what our software does, and it helps us focus more on solving the unusual and noteworthy problems instead of wasting time with the ones that we know we shouldn’t have been worrying about.

How:

  1. Join a software development team for one sprint duration.
  2. Discuss project tasks for that sprint with everyone.
  3. Ask about requirements examples and create executable specifications for those tasks as the application is being written.
  4. Refine the specifications when necessary.
  5. Continuously update the specifications until they pass.
  6. Keep the specifications running on a preferred schedule.

Why: To see if writing executable specifications alongside development is feasible during the duration of a sprint.

Limitations: Tester writes the executable specifications, programmers work as is.

Experiment Realizations:

  • Writing executable specifications can be done during actual app development, provided that tester is experienced with the tools for implementing it and understands well why the need for such specifications.
  • It is of course more beneficial if everyone in the team learn how executable specifications work and write/run the specifications, and why the need to implement it.
  • It will take quite a long while before executable specifications becomes a norm in the existing software development process, if ever. This is a function of whether everyone believes that such specifications are actually useful in practice, and then building the skills and habit of including these specifications in the team’s definition of done.

Lessons from Gojko Adzic’s “Specification By Example”

Automated checking is not a new concept. Gojko Adzic, however, provides us a way to make better integration of it in our software development processes. In his book titled “Specification by Example”, he talks about executable specifications that double as a living documentation. These are examples which continuously exercise business rules, they help teams collaborate, and, along with software code, they’re supposed to be the source of truth for understanding how our applications work. He builds a strong case about the benefits of writing specifications by example by presenting case studies and testimonials of teams who have actually used it in their projects, and I think that it is a great way of moving forward, of baking quality in.

Some favorite takeaways from the book:

  • Tests are specifications; specifications are tests.
  • “If I cannot have the documentation in an automated fashion, I don’t trust it. It’s not exercised.” -Tim Andersen
  • Beginners think that there is no documentation in agile, which is not true. It’s about choosing the types of documentation that are useful. There is still documentation in an agile process, and that’s not a two-feet-high pile of paper, but something lighter, bound to the real code. When you ask, “does your system have this feature?” you don’t have a Word document that claims that something is done; you have something executable that proves that the system really does what you want. That’s real documentation.
  • Fred Brooks quote: In The Mythical Man-Month 4 he wrote, “The hardest single part of building a software system is deciding precisely what to build.” Albert Einstein himself said that “the formulation of a problem is often more essential than its solution.”
  • We don’t really want to bother with estimating stories. If you start estimating stories, with Fibonacci numbers for example, you soon realize that anything eight or higher is too big to deliver in an iteration, so we’ll make it one, two, three, and five. Then you go to the next level and say five is really big. Now that everything is one, two, and three, they’re now really the same thing. We can just break that down into stories of that size and forget about that part of estimating, and then just measure the cycle time to when it is actually delivered.
  • Sometimes people still struggle with explaining what the value of a given feature would be (even when asking them for an example). As a further step, I ask them to give an example and say what they would need to do differently (work around) if the system would not provide this feature. Usually this helps them then to express the value of a given feature.
  • QA doesn’t write [acceptance] tests for developers; they work together. The QA person owns the specification, which is expressed through the test plan, and continues to own that until we ship the feature. Developers write the feature files [specifications] with the QA involved to advise what should be covered. QA finds the holes in the feature files, points out things that are not covered, and also produces test scripts for manual testing.
  • If we don’t have enough information to design good test cases, we definitely don’t have enough information to build the system.
  • Postponing automation is just a local optimization. You might get through the stories quicker from the initial development perspective, but they’ll come back for fixing down the road. David Evans often illustrates this with an analogy of a city bus: A bus can go a lot faster if it doesn’t have to stop to pick up passengers, but it isn’t really doing its job then.
  • Workflow and session rules can often be checked only against the user interface layer. But that doesn’t mean that the only option to automate those checks is to launch a browser. Instead of automating the specifications through a browser, several teams developing web applications saved a lot of time and effort going right below the skin of the application—to the HTTP layer.
  • Automating executable specifications forces developers to experience what it’s like to use their own system, because they have to use the interfaces designed for clients. If executable specifications are hard to automate, this means that the client APIs aren’t easy to use, which means it’s time to start simplifying the APIs.
  • Automation itself isn’t a goal. It’s a tool to exercise the business processes.
  • Effective delivery with short iterations or in constant flow requires removing as many expected obstacles as possible so that unexpected issues can be addressed. Adam Geras puts this more eloquently: “Quality is about being prepared for the usual so you have time to tackle the unusual.” Living documentation simply makes common problems go away.
  • Find the most annoying thing and fix it, then something else will pop up, and after that something else will pop up. Eventually, if you keep doing this, you will create a stable system that will be really useful.

Contemplating In-Office Knowledge-Sharing Sessions

In the past I tend to prepare presentation slides if I want to share something to tester colleagues at work, often clippings of interesting articles which I felt could be useful for our knowledge-sharing sessions. It worked, but after some time the sessions felt monotonous and tedious. Probably because I always have to explain in detail the ideas and the lessons behind those clippings. I try to make my presentations interesting, but I think that after a while hearing the same voice over and over can get old.

These days I’m sharing videos instead. The videos are usually recorded conference talks or tutorials I have watched and learned from in recent years, and I have taken care in listing the the ones that are insightful, fun, and relatively short. It’s like I’m inviting officemates to watch a short movie for free. The big change: I don’t take a lot of time talking during the knowledge-sharing anymore. There are of course still bits of discussions before, after, or during the showing of a video, whenever necessary, for explaining why I have taken a liking to the talk or to ask them about what they understood. We take turns telling stories about our experiences related to the ideas shared by the speaker, which is nice. And compared to the powerpoint presentations I did before, I felt that because the speaker is someone from outside it makes the ideas shared in the talks and tutorials feel more fresh and real than when I’m merely showing them quoted paragraphs from blogs. That makes it easier for my colleagues to get curious and actually learn something, which is exactly the point of the activity.

 

Lessons from Bernadette Jiwa’s “Difference”

Bernadette Jiwa does not specifically write about software testing or about software development but her writings always talk about how we should perform our work or how to run our businesses, general concepts that seem applicable to what we do too. Reading “Difference” was a treat, and the lessons she shares about what marketing really is are a bonus.

Some favorite lines from the book:

  • Do you need to have a brand new idea or invent something radically different in order to create difference? No, not necessarily. Starbucks didn’t invent coffee, and Apple didn’t invent the smartphone; these companies simply created new experiences of them, which in turn created a whole new set of meanings that we attached to what were once commodities.
  • Today the shortcut to more is to matter. It isn’t the person with the best idea who wins; it’s the person who has the greatest understanding of what really matters to people.
  • Perhaps it’s time to get comfortable with the fact that if we want to change the world, we need to stop being afraid to tell better true stories. Maybe the time has come to stop trying to convince people and to invest time instead in showing them what we stand for, so that they can choose the stories and brands they want to believe in.
  • The truth is that people don’t fall in love with ideas at all. They fall in love with how those ideas, products, services and places make them feel.
  • When you go the extra mile, people will know, and that knowing changes everything about how they feel about what you do. Emotional points of difference, the things that are less obvious and sometimes not even articulated, matter.
  • When return on investment is measured by delight instead of sales or conversions, there’s a lot more freedom to be creative, to be bold, or maybe even to be creative and bold.
  • You might not be able to change how the world values your profession, but you can change how you are valued by doing work that matters. Work that changes how people feel, not just what they think. We have two choices. We can stand around looking at the train wreck of what was, or we can design our own futures.
  • Love. It turns out that what applies to creating great drama, good storytelling, and life also applies to business. We sometimes forget that.
  • We are living in a unique moment in time. An age where we can bring things to the world without having to own a factory or an office building. A digital age that gives us more opportunities to really listen and to learn how to see what people are longing for.
  • If you want to make something new, start with understanding. Understanding what’s already present, and understanding the opportunities in what’s not. Most of all, understanding how it all fits together.
  • The truth is that what really moves us is feelings, not facts. There are a thousand ways for you to get noticed, but there’s only one way to really touch someone. And that’s to give them a reason to care, a story they can believe in. Great stories are woven, not told. They come to us in whispers, as goosebumps. A wry smile playing on our lips, a vigorous nod, a feeling we can’t explain. Or as a number scribbled on the back of a napkin, a shared link accompanied by a message that says, ‘you gotta see this’, which really means ‘I want you to experience this with me’.

Practicing Awareness

It’s easy to learn something when you’re genuinely curious about it, no need to look for any other external motivation. On the other hand, it’s terribly difficult (often a waste of time) to teach something to someone who is not interested in the subject at that given point in time.

Software tester and programmer pairing in bug fixing or feature-writing situations is supposed to be a fun experience for both parties. Either of them getting frustrated at what they’re doing means something is blocking them from being able to perform well as a duo. Maybe they don’t know enough about the needs of the other person or how to fulfill those needs, maybe they don’t understand enough about the thing they’re building or how to build it in the first place.

If the people using the software have qualms regarding a feature update, maybe there happened to be a miscommunication between clients and developers about what feature the customers really want built. Solving that communication problem matters, maybe more than writing feature code quickly, definitely more than pointing fingers and blaming other people after the update’s been released.

Testing with deep understanding of why the application exists the way it is and how features actually work in both user interface and code helps us perform our testing more effectively, especially in situations with time constraints.

Pair testing happens more often than I initially thought it does, only not as explicit as how I imagine it happening.

For whatever you wish to be master of, it pays to have mentors, people you follow and respect and trust and provide feedback. If there aren’t any in your place of work, find them online.

Thoughts After a Recent Software Feature Release

Having a feature released to the customers does not mean I won’t have to test that feature anymore.

When a feature is released months later than it’s initially estimated, it could mean that the project was just a complex thing. Or it could mean that we did not give the thing enough attention to have it broken up and released into possibly much smaller chunks.

It’s terribly challenging to find confidence in releasing a new massive functionality which took months to finish, especially when the feedback you have about its quality is limited. I am being reminded that a part of testing is solving the problem of having good enough test coverage of product risks. What are those risks? What tests do I want to perform in order to cover those risks?

Just like how meetings that take hours get extremely boring, it gets annoyingly tiring for teams to spend more than a month of time building a feature before publishing it in public.

If we don’t spend some of our time now tinkering with systems that could possibly help us accomplish things better in the future, we’ll end up doing the same things as before and that reflects into our ability to grow.

Just because a defect happens to quietly slipped its way into production code does not mean the team did not work their asses off in bringing the best work they could possibly do to the project, under the circumstances. And just because a team worked hard does not guarantee they will write code free of any bug. What matters is what people do when those flaws are found. And there will always be flaws.

Some people, even testers, are unnecessarily loud when they find problems in production after a feature release. This just means that they did not have any idea how that part of the feature was built in the first place and did not care to explore it either before the release. They only assumed that everything is ought to be working. They forgot that quality is a team responsibility.

It’s important that we understand why a problems exists in production code (and maybe who did it) but I’m not sure if it’s particularly helpful that we leave the problem fixing later to the person who made the defect when we have the time and the know-how to fix it ourselves now. What’s the boy scout rule again for programmers? Yes, leave the code better than you found it.

Just because a shiny new feature is released to the public does not mean it has actual value to customers. And if we don’t find ways of getting user feedback, we’ll never find out whether what we built amounts to something or not.

Feature releases are always short-lived, like the physical store opening day launch.  There’s always a celebration, an attraction, but it’s not what really matters in the long run. What matters are the lessons learned in the duration of building the feature, what we intend to do with those lessons, the people we worked with, and building things that our customers love.

Learning How To Write Better Cucumber Scenarios

Some of the important lessons in writing automated checks are found, not in the actual implementation of the check itself, but rather in the specification. What does a green check mean? What are we really trying to find out when we’re running this check? Will somebody from another team understand why this test was written? Does this check say what’s necessary in a feature or does the check only state a procedure without context? Too often we concentrate on syntax, frameworks, and required steps in building automation, but not so much on clearly expressing what’s being checked and why the check is recorded in the first place. I’ve made that mistake and now I am trying to learn how to write better checks.

Take, for example, the cucumber checks below. Would you say that it is clear what’s being verified in the test? What would a product owner probably say about these checks when they see them for the first time?

Scenario Outline: Validating the Rate Plan Policies
  Given I am in the ShowRoomsPage for a "<hotel_type>" property from "<arrival>" to "<departure>" for a "<rateplan>" rate plan for a "confirmed" reservation
  When I view the policies for the rate plan
  Then I know that these are the correct policies
  
  Examples:
  | hotel_type | arrival         | departure       | rateplan            |
  | DWH        | 4 DAYS FROM NOW | 6 DAYS FROM NOW | Public_Partial_LT   |
  | DWH        | TOMORROW        | 3 DAYS FROM NOW | Public_Full_YesR_LT |

Some of the questions that would probably pop up in their minds include the following: How are the examples in the grid necessary for the test? What does Public_Partial_LT or Public_Full_YesR_LT mean? Do we really need to know the arrival and departure date settings for the checks? What does the Given statement mean? Most importantly, how do I know that the policies are actually correct for these tests?

This was how I wrote my checks before when I started studying how to write Cucumber-Watir-Ruby checks. And a lot of my checks in the existing test suite still are. I am, however, trying to learn how to re-write them in a better way, in terms of readability and conciseness, guided by lessons so far learned from Jeff Nyman’s test description language blog posts, so that almost everybody in our team can recognize in a glance what a particular check does and why they are included in the feature.

Re-writing the example checks above, I now have these:

Scenario: ShowRooms Prepayment Policy, DWH Partial
  When guest views the policies for a DWH property for a partial rate plan
  Then a copy of "Only 10% prepayment is required to confirm your reservation" is displayed in the prepayment policy

Scenario: ShowRooms Prepayment Policy, DWH Full Non-Refundable
  When guest views the policies for a DWH property for a full nonrefundable rate plan
  Then a copy of "Full prepayment is required to confirm your reservation" is displayed in the prepayment policy

Not perfect, but I’d like to think that they are more effective than the ones I had before. Checks are more specific, unrelated information are not included in the test, and we understand what it means when the checks pass or fail.

What do you think?