Automating the Windows Desktop Calculator’s UI using Winium

Ever since I learned how to use Selenium to automate browsers and web applications a few years back, I’ve wondered from time to time whether I can use the same process or technology to automate Windows desktop applications via its user interface. There’s no particular use case at work considering our apps are all in the web, but I challenged myself to find an answer only for curiosity’s sake. It didn’t turn out to be a thought-provoking experiment but at least it was somehow amusing.

Here’s what I got from the quick study:

Click the image to view the GIF full-size on another browser tab 🙂

And a few notes:

  • We can’t use Selenium to automate Windows desktop applications.
  • A quick search about automating Windows apps tells us that there’s actually a number of tools we can use for such purpose, some paid, others free. Other than Winium, I found out that we can also use Sikuli (or its successor SikuliX), WinTask, autoIT, TestStack.White, Automa, MacroSchedulerCobra, or Pywinauto. There’s probably more tools out there. I just chose to use Winium because writing tests and automating the app with it is similar to how Selenium works, which means very little learning curve for me.
  • Source code for this short experiment can be found here: Win-Calculator. It uses Maven for handling installation of project dependencies while the test code is written in Java. The tests run locally, I’m not sure whether there’s a way to manipulate a Windows app on another machine using Winium. I have not researched that far.
  • Winium will actually take control of the machine’s mouse to move to the element locations set in the tests. This means that interrupting the position of the mouse while the tests are running would likely fail the test. This is annoying, and I’m not sure whether the other Windows app automation tools behave otherwise or the same.

Uploading Images in Watir 6.0.2

Even when most of the automated checks in our test server are running via the HTTP layer instead of using the browser, I have kept the automated UI versions of those tests for review purposes. They’re not very practical for continuous integration but still fun to watch sometimes, at least for me. It is also good practice to fix them when they break, to see whether I’m still updated with how Selenium or Watir runs.

And recently I found out that I’m unable to upload images in Watir 6.0.2 using the usual way, giving me an error:

element located, but timed out after 30 seconds, waiting for true condition on <element location> (Watir::Exception::UnknownObjectException)

The test code in question was this, written using the page-object gem

self.photo = IMAGE_FILEPATH

where photo is just a given name for a file field in the page where an image is to be uploaded. In basic Watir, this can also be written as

@browser.file_field(:id, "LOCATOR_PATH").set(IMAGE_FILEPATH)

Both test code are timing out, and the test failing because of the exception error. When manually checked, there are no issues in uploading images whatsoever.

Apparently, this an open issue in Watir and the workaround at the moment is to properly set the values of Watir’s relaxed_locate before and after the actual image upload code, like so

Watir.relaxed_locate = false
self.photo = IMAGE_FILEPATH
Watir.relaxed_locate = true

or

Watir.relaxed_locate = false
@browser.file_field(:id, "LOCATOR_PATH").set(IMAGE_FILEPATH)
Watir.relaxed_locate = true

And that makes the image upload work again. Both work for Google Chrome and Mozilla Firefox. 🙂

An Experiment on Executable Specifications

What: Create executable specifications for an ongoing project sprint. Executable specifications are written examples of a business need that can be run anytime and acts as a source of truth for how applications behave. It is a living documentation of what our software does, and it helps us focus more on solving the unusual and noteworthy problems instead of wasting time with the ones that we know we shouldn’t have been worrying about.

How:

  1. Join a software development team for one sprint duration.
  2. Discuss project tasks for that sprint with everyone.
  3. Ask about requirements examples and create executable specifications for those tasks as the application is being written.
  4. Refine the specifications when necessary.
  5. Continuously update the specifications until they pass.
  6. Keep the specifications running on a preferred schedule.

Why: To see if writing executable specifications alongside development is feasible during the duration of a sprint.

Limitations: Tester writes the executable specifications, programmers work as is.

Experiment Realizations:

  • Writing executable specifications can be done during actual app development, provided that tester is experienced with the tools for implementing it and understands well why the need for such specifications.
  • It is of course more beneficial if everyone in the team learn how executable specifications work and write/run the specifications, and why the need to implement it.
  • It will take quite a long while before executable specifications becomes a norm in the existing software development process, if ever. This is a function of whether everyone believes that such specifications are actually useful in practice, and then building the skills and habit of including these specifications in the team’s definition of done.

Lessons from Gojko Adzic’s “Specification By Example”

Automated checking is not a new concept. Gojko Adzic, however, provides us a way to make better integration of it in our software development processes. In his book titled “Specification by Example”, he talks about executable specifications that double as a living documentation. These are examples which continuously exercise business rules, they help teams collaborate, and, along with software code, they’re supposed to be the source of truth for understanding how our applications work. He builds a strong case about the benefits of writing specifications by example by presenting case studies and testimonials of teams who have actually used it in their projects, and I think that it is a great way of moving forward, of baking quality in.

Some favorite takeaways from the book:

  • Tests are specifications; specifications are tests.
  • “If I cannot have the documentation in an automated fashion, I don’t trust it. It’s not exercised.” -Tim Andersen
  • Beginners think that there is no documentation in agile, which is not true. It’s about choosing the types of documentation that are useful. There is still documentation in an agile process, and that’s not a two-feet-high pile of paper, but something lighter, bound to the real code. When you ask, “does your system have this feature?” you don’t have a Word document that claims that something is done; you have something executable that proves that the system really does what you want. That’s real documentation.
  • Fred Brooks quote: In The Mythical Man-Month 4 he wrote, “The hardest single part of building a software system is deciding precisely what to build.” Albert Einstein himself said that “the formulation of a problem is often more essential than its solution.”
  • We don’t really want to bother with estimating stories. If you start estimating stories, with Fibonacci numbers for example, you soon realize that anything eight or higher is too big to deliver in an iteration, so we’ll make it one, two, three, and five. Then you go to the next level and say five is really big. Now that everything is one, two, and three, they’re now really the same thing. We can just break that down into stories of that size and forget about that part of estimating, and then just measure the cycle time to when it is actually delivered.
  • Sometimes people still struggle with explaining what the value of a given feature would be (even when asking them for an example). As a further step, I ask them to give an example and say what they would need to do differently (work around) if the system would not provide this feature. Usually this helps them then to express the value of a given feature.
  • QA doesn’t write [acceptance] tests for developers; they work together. The QA person owns the specification, which is expressed through the test plan, and continues to own that until we ship the feature. Developers write the feature files [specifications] with the QA involved to advise what should be covered. QA finds the holes in the feature files, points out things that are not covered, and also produces test scripts for manual testing.
  • If we don’t have enough information to design good test cases, we definitely don’t have enough information to build the system.
  • Postponing automation is just a local optimization. You might get through the stories quicker from the initial development perspective, but they’ll come back for fixing down the road. David Evans often illustrates this with an analogy of a city bus: A bus can go a lot faster if it doesn’t have to stop to pick up passengers, but it isn’t really doing its job then.
  • Workflow and session rules can often be checked only against the user interface layer. But that doesn’t mean that the only option to automate those checks is to launch a browser. Instead of automating the specifications through a browser, several teams developing web applications saved a lot of time and effort going right below the skin of the application—to the HTTP layer.
  • Automating executable specifications forces developers to experience what it’s like to use their own system, because they have to use the interfaces designed for clients. If executable specifications are hard to automate, this means that the client APIs aren’t easy to use, which means it’s time to start simplifying the APIs.
  • Automation itself isn’t a goal. It’s a tool to exercise the business processes.
  • Effective delivery with short iterations or in constant flow requires removing as many expected obstacles as possible so that unexpected issues can be addressed. Adam Geras puts this more eloquently: “Quality is about being prepared for the usual so you have time to tackle the unusual.” Living documentation simply makes common problems go away.
  • Find the most annoying thing and fix it, then something else will pop up, and after that something else will pop up. Eventually, if you keep doing this, you will create a stable system that will be really useful.

An Encounter with the ‘cucumber.yml was found, but could not be parsed’ Error

One day last week I merged a colleague’s new feature tests to the existing test repository. Routinely after that, I logged in to our dedicated test server, updated the existing running test suite by retrieving the latest merged code, and ran a ‘bundle update’ because the updated code required the latest versions of the dependencies. The updates finished without hitches as expected after a minute or so. Task’s done, I just need to run a sample test to be certain everything’s well and good.

An error blows up in my face:

cucumber.yml was found, but cannot be parsed. Please refer to cucumber’s documentation on correct profile usage.

Before all these, I had run a simulation of the gem updates on my local machine including running all tests to make sure there were no surprises when updating the test server. But I was still caught off-guard. I didn’t see that coming.

What? That’s always the first question. I squinted at the screen and looked at the error message again; it says something about the cucumber.yml file unable to be read by the machine, like it was corrupted. I opened the file to check (which contains various cucumber profile configuration set up for running tests on different test environments), was able to view its contents in the text editor, inspected it for unfamiliar code and found nothing. It looks okay, but the machine says otherwise. Why?

I tried an initial Google search for the error message and saw a fix that was written five years ago, saying that the error occurs because of a particular ‘rerun.txt’ file. The post tells me that the problem will go away if I delete that file.

Except that I don’t have that file in my code repository. What now?

Maybe there’s really something going on with the test code. Let’s see what happens if I delete a particular profile in the cucumber.yml file. Done. No changes in behavior, error still exist. What if I delete everything? Done. Error didn’t go away. Hmm.. that’s odd. It seems that the file contents are not the problem, is valid YAML code too according to online checkers, and the file is not damaged in any way I can see.

I went to back to looking at the search results for possible solutions People keep telling me its about the rerun.txt file. Others say I need to edit a cucumber library file in order to see what test code specifically causes the error for cucumber’s runner. No more other clues. Now this is difficult.

I kept researching for plausible fixes online for a few hours, I thought that there may still be something that can help me but I missed. No such luck. Okay, let’s try editing that library file and see what happens. It was a first time viewing library code, because I didn’t have any reason to do it before, and told myself that maybe I should actually check it out more often.

I found the command-line interface profile_loader file and the particular code which loads the cucumber.yml  file in question:

Found you, cucumber's profile loader!

Found you, cucumber’s profile loader!

Commented out some few lines of code as suggested:

Now let's see what your problem really is

Now let’s see what your problem really is

Then ran the sample cucumber test again:

A problem with the Psych module! What's that? :O

A problem with the Psych module! What’s that? :O

Okay. It says that a missing private method named ‘load’ is being called for a particular Psych module. No wonder cucumber is failing. Bummer, I don’t have a single idea about a Psych module that cucumber runs. All I can do is another Google search for the new error message and maybe find a workaround.

I am reminded that problems in building systems for automated test suites are not limited to writing test code and the application under test. Just like any other software, they can sometimes break in areas we do not know anything.

Eventually I found this enlightening post on a Github repository:

Interestingly, a very short solution. It was a rubygems bug after all, and what I needed to do was to run a ‘gem update –system’ command to get our cucumber tests back up and running.

 

Interesting Talks from Google Test Automation Conference 2016

There’s a lot of stuff going on in the software testing community at the moment, specifically in the field of automation, because of how software is now being deployed into various other platforms besides personal computers. Google needs to worry about testing their eyeglasses, virtual reality headsets, and cars. Others care about testing robots and televisions. This is why it is fun to watch talks from conferences, like the Selenium Conference or the recently concluded Google Test Automation Conference: I get to find out what problems they’re facing and see how they try to solve them, and maybe learn a thing or two. Sometimes I get to pick up a new tool to try for my own testing too, a great bonus.

Some favorite talks from the conference are:

Notes from Alister Scott’s “Pride and Paradev: A Collection of Agile Software Testing Contradictions”

I’ve stumbled over Alister Scott‘s WatirMelon blog some years back, probably looking for an answer to a particular question about automation, and found it to be a site worth following. There he talks about flaky tests, raising bugs you don’t know how to reproduce, junior QA professional development, the craziest bug he’s ever seen, writing code, and the classic minesweeper game. He was part of the Watir project in the past, but is now an excellence wrangler over at Automattic (which takes care of WordPress). He has also written an intriguing book, titled “Pride and Paradev“, which talks about several of the contradictions that we have over in the field of software testing. In a nutshell, it explains why there are no best practices, only practices that work well under a certain context.

Here are a number of takeaways from the book:

  • A paradev is anyone on a software team that doesn’t just do programming.
  • Agile software development is all about delivering business value sooner. That’s why we work in short iterations, seek regular business feedback, are accountable for our work and change course before it’s too hard.
  • Agile software development is all about breaking things down.
  • Agile software development is all about communication and flexibility. You must be extremely flexible to work well on an agile team. You can’t be hung up about your role’s title. Constantly delivering business value means doing what is needed, and a team of people with diverse skills thrives as they constantly adapt to get things done. Most importantly flexibility means variety which is fun!
  • Delivering software everyday is easy. Delivering working software everyday is hard. The only way an agile team can deliver working software daily is to have a solid suite of automated tests that tells us it’s still working. The only way to have reliable, up-to-date automated tests is to develop them alongside your software application and run them against every build.
  • You’re testing software day in and day out, so it makes sense to have an idea about the internals of how that software works. That requires a deep technical understanding of the application. The better your understanding of the application is, the better the bugs you raise will be.
  • Hiring testers with technical skills over having a testing mindset is a common mistake. A tester who primarily spends his/her time writing automated tests will spend more time getting his/her own code working instead of testing the functionality that your customers will use.
  • What technical skills a tester lacks can be made up for with intelligence and curiosity. Even if a tester has no deep underlying knowledge of a system, they can still be very effective at finding bugs through skilled exploratory and story testing. Often non technical testers have better shoshin: a lack of preconceptions, when testing a system. A technical tester may take technical limitations into consideration but a non technical can be better at questioning why things are they way they are and rejecting technical complacency. Often non-technical testers will have a better understanding of the subject matter and be able to communicate with business representatives more effectively about issues.
  • You can be very effective as a non-technical tester, but it’s harder work and you’ll need to develop strong collaboration skills with the development team to provide support and guidance for more technical tasks such as automated testing and test data discovery or creation.
  • Whilst you think you may determine the quality of the system, it’s actually the development team as a whole that does that. Programmers are the ones who write the good/poor quality code. Whilst you can provide information and suggestions about problems: the business can and should overrule you: it’s their product for their business that you’re building: you can’t always get what you consider to be important as business decisions often trump technical ones.
  • A tester should never be measured on how many bugs they have raised. Doing so encourages testers to game the system by raising insignificant bugs and splitting bugs which is a waste of everyone’s time. And this further widens the tester vs programmer divide. Once a tester realizes their job isn’t to record bugs but instead deliver bug free stories: they will be a lot more comfortable not raising and tracking bugs. The only true measurement of the quality of testing performed is bugs missed, which aren’t recorded anyway.
  • Everything in life is contextual. What is okay in one context, makes no sense in another. I can swear to my mates, but never my Mum. Realizing the value of context will get you a long way.
  • Probably the best thing I have ever learned in life is that no matter what life throws at you, no matter what people do to you or how they treat you, the only thing you can truly control is your response.