Automating the Windows 10 Desktop Calculator using PyAutoGui

Last year, I’ve tried automating a Windows 7 desktop calculator using a Java library called Winium. A year later, someone from the comments told me that the example didn’t work on Windows 10. I ignored the comment for a while because I didn’t have a Windows 10 machine to perform a test back then, but now that I’ve recently upgraded my home PC I decided to try it out.

What I found:

Running a Windows 7 desktop calculator automation example (using Winium) on a Windows 10 machine

The Windows 10 calculator opened up but the tests didn’t run properly. The error log told me that the program was unable to find the calculator elements it was supposed to click. Bummer. Maybe the names of the calculator elements were different, Windows 10 versus Windows 7? It wasn’t; the element names were still the same according to UI Spy. Perhaps there’s something from Winium that can point me to a clue? Oh, it seems that the library has not been updated in recent years.

If I can’t use Winium, how then can I automate the Windows 10 calculator? A Google search pointed me to PyAutoGui. Instead of Java, it says that I’ll need Python (and Pip) for this tool to work. And yes, being Windows, I also need to properly set the environment variables so I can use the Python and Pip commands on a terminal.

Let’s install PyAutoGui:

Installing PyAutoGui

And after writing some code, let’s see if we can automate the Windows 10 calculator with it:

Automating the Windows 10 desktop calculator with PyAutoGui. Click the image to view the GIF full-size on another browser tab 🙂

It works!

But here are some catches:

• I had to rely on PyAutoGui’s keyboard control functions for performing the calculator actions, instead of finding elements via the user interface. Well, from what I’ve seen so far from the docs is that the only way to locate a UI element is by locating elements using screenshots. I tried that the first time and it was very flaky, so I opted for using the keyboard control functions instead.
• The code introduces a time variable to wait for the calculator to appear on screen. The code also introduces a time variable to pause a portion of a second in-between each keyboard action so the actions don’t happen too fast for a person’s eye.
• There are no assertions in the example code, because I couldn’t find any assertion functions I could use from the PyAutoGui docs. It is not a tool built for testing, only for automating desktop apps.

Source code for this experiment can be found on: Win-Calculator-PyAutoGui.

Automating the Windows 7 Desktop Calculator’s UI using Winium

Ever since I learned how to use Selenium to automate browsers and web applications a few years back, I’ve wondered from time to time whether I can use the same process or technology to automate Windows desktop applications via its user interface. There’s no particular use case at work considering our apps are all in the web, but I challenged myself to find an answer only for curiosity’s sake. It didn’t turn out to be a thought-provoking experiment but at least it was somehow amusing.

Here’s what I got from the quick study:

Click the image to view the GIF full-size on another browser tab 🙂

And a few notes:

• We can’t use Selenium to automate Windows desktop applications.
• A quick search about automating Windows apps tells us that there’s actually a number of tools we can use for such purpose, some paid, others free. Other than Winium, I found out that we can also use Sikuli (or its successor SikuliX), WinTask, autoIT, TestStack.White, Automa, MacroScheduler, Cobra, or Pywinauto. There’s probably more tools out there. I just chose to use Winium because writing tests and automating the app with it is similar to how Selenium works, which means very little learning curve for me.
• Source code for this short experiment can be found here: Win-Calculator. It uses Maven for handling installation of project dependencies while the test code is written in Java. The tests run locally, I’m not sure whether there’s a way to manipulate a Windows app on another machine using Winium. I have not researched that far.
• Winium will actually take control of the machine’s mouse to move to the element locations set in the tests. This means that interrupting the position of the mouse while the tests are running would likely fail the test. This is annoying, and I’m not sure whether the other Windows app automation tools behave otherwise or the same.