The Lessons I Learned This Year In Software Testing

It was only several years ago when I started writing Selenium tests, first with Selenium IDE, then in Java with Webdriver, then in Ruby with Watir. Now I don’t write a lot of Selenium tests anymore, ever since I found out that it is often better (faster and more stable) to write automated checks for application features through the API. Or through unit tests. Selenium has its place in checking user flows or automating the UI, but only if I have to, if its value exceeds that of its costs. There lies an important lesson in automation: there is not a single tool that does it all. It is us who decides which tool to use for a particular test, and it helps to understand if a tool fits the specific use case.

And I think I’ve familiarized myself with a number of tools this year: Postman for API testing, Winium for automating Windows applications, BackstopJS for open-source visual testing, Cloud9 for cloud-based software development, Phonegap for HTML-based mobile app development, Docker for building shareable self-contained images of applications for development or testing, PHP testing tools (PHPUnit, Guzzle, Behat), and source code linting tools like Rubocop for Ruby. I’m not a master of these tools, but I know enough to be able to decide whether I need them (or not) for a particular thing I want to achieve. They’re in my tool belt.

Needless to say, I have outgrown the hype of automation. It is programming and tooling at its core. It helps us perform repeatable tasks without breaking a sweat, not limited to testing apps, if done with care. It is not easy. It can be rewarding. It all starts with a deep understanding of what definite task or problem actually needs solving.

And this year’s experiences has lead me to better grasp the nuances of software development, which is actually a problem of people, of teams and their habits and biases. Our team certainly has its defaults, some of which are not helping us get better at what we do. And as such from here on I’d like to contribute in key areas I believe our programmers have not had much time to think about because of project deadlines and resource constraints – dockerized application environments and shift-left testing – solutions for providing testing feedback earlier in the software development cycle, which in turn can help us build better apps and release faster.

Advertisements

On Setting Up A Local Version Of Applications For Testing

So.. after about a week or so since I asked permission for read-write access to our application code repository I’m glad to say that I’m almost done with setting up a version of our apps locally on my machine. It is necessary because I first need to check my changes if they work locally before committing those changes. No code commits yet until said local apps have the same stability as our apps in Staging.

But there are no unit tests. How would I know if everything works after cloning the app repository and running the local settings? Only one option: I had to run local versions of my Staging application API tests. They’re slower than unit tests but at least they let me know if the apps work on some good enough level.

Running tests on local applications!

We’re in business! 🙂

Most of the problems that I encountered whilst setting up were database problems. That’s because no one was maintaining a small-but-updated version of the database. As told, I had to manually match which queries to run according to what problems my tests found. Not pretty, though I could say that in retrospect looking at the errors and finding the DB fix on my own was good exercise. Not elegant, but it helped me get familiarized a little bit with our applications as code.

There was also no documentation about the application and how to run them on various machines. Guides are important but README files were mostly left blank. I had to rely on programmer friends for clues about what to do next whenever I got stuck.

Such problems took time and patience to solve. I had to take notes about updating certain pieces too. Sometimes, I had to make changes to the code itself in order for some features to not fail locally. And yes, I need to remember not to accidentally commit those changes to the remote repository.

It would be nice if we can just go to some private repository and download an environment image or two which runs smoothly when integrated with the app repository. Update the code on a local machine and the environment updates automatically. Set up would have been done in a matter of minutes, not days. But, alas, that’s a problem worth solving for another day.

Extending The Avenues Of Performing Testing

Last Wednesday afternoon I anxiously asked my boss for permission to make changes on our application code repository. I said I wanted to try fixing some of the reported bugs listed on our tracking system, if there are no other resources available to pass them to. I made a case about myself not posing any problems because of the code review process built into our repository management tool, that there’s no reason for me to merge any changes without getting feedback from a senior developer first.

He smiled at me and gleefully said “Go ahead. I’m not going to stop you.“, to which I beamed and heartily replied “Thanks, boss!”

This is a turning point in my software testing career, to be able to work on the application code directly as needed. It is actually one of my biggest frustrations – to not be able to find out for myself where the bug lives in the code and fix them if necessary. It’s always a pain to be able to do nothing but wait for a fix, and for a fix to be dependent on the resources available. In my head I think that I’m available and maybe I can do something, but I don’t explicitly have access to the application itself and the code that runs it so I can’t do anything until I have the rights to do so. That’s how it always been. Software testers are often not expected to fiddle with code, at least in my experience, especially in the past where automation was not yet known to be useful as a testing tool. Now that I have the skills and the permission to work on the application repository, I feel that my reach for making an impact on application quality has now expanded remarkably well.

Now bug-fixing is not software testing work in the traditional sense. But I figured there’s no harm in trying to fix bugs and learning the nitty-gritty details of how our legacy applications actually run deep in the code. I believe that learning technical stuff helps me communicate better with programmers. It helps me test applications in a more efficient manner too. Of course I have to consistently remind myself that I am a software-tester-first-programmer-second guy and have to be careful not to fill my days playing with code and forgetting to explore our applications themselves. That said, there are ideas I really want to experiment within our software development process, towards the goal of improving code quality and feedback, and I can only tinker with those ideas inside the application repository itself. Dockerized testing environments, code linting, and unit tests are three things I want to start building for our team, ideas that I consider to be very helpful in writing better code but has not been given enough priority through the years.

I think I’m still testing software, just extending the knowledge and practice of the various ways I perform testing.

Being Reminded of All the Phases I’ve So Far Had in Writing Automated Checks

I’m currently in the midst of a test code overhaul, a re-writing project of sorts. It started about a week ago and so far I’ve made considerable progress on what I’ve wanted to achieve with the rewrite, which is basically cleaner and more maintainable code, mostly in the sense of test data management and test description language. The number of tests running everyday in our Jenkins system has grown noticeably and I’ve felt that it’s been difficult to add certain tests because of how I structured the test data in the past, which I have not upgraded since then. The two possible avenues for running tests – on the UI and HTTP layers – also adds a bit of complexity and it’d be nice if I can integrate the two smoothly. It’s an interesting development because I did not plan on any re-writing to be done anytime soon but I guess at the back of my mind I knew it’ll happen eventually. And so I decided to take a step back from writing more tests and do some cleanup before it gets tougher to change things. I plan to finish everything in about a month or so.

At the moment, I’m reminded of the phases I’ve gone through in learning to code and writing automated checks in the past few years:

  • Early 2014. It all begins with Selenium IDE, with giving the self some time to study the basic Selenese commands for writing automated checks and (more importantly) understand how to properly retrieve the page elements you want to manipulate.
  • Mid 2014. Test management in Selenium IDE becomes difficult as the number of tests grow, hence the decision to switch to Selenium WebDriver. The only programming language background I had back then was C++, which was limited to only functions and logical/conditional operators, so I chose Java to work with to lessen the learning curve.
  • Late 2014. Familiarized myself with Git, which hooked me on making daily commits and appreciating version control. Along the way I learned the concepts of classes and objects.
  • All of 2015 up to Early 2016. I was in a trance, writing code daily and pushing myself to create all the automated checks that I wanted to run for our apps before every release. Tests run on the Eclipse IDE using TestNG and I was happy with what I had, except that those end-to-end tests are really slow. Running everything took overnight to finish, which was okay for my employer but annoying for me personally.
  • Mid 2016. Re-writing existing tests in Ruby with Cucumber integration started off (when I found Jeff Morgan’s “Cucumber & Cheese” book online) as a side project for fun and testing my skill level in programming. And I did have buckets of fun! The experiment told me that there’s still a lot I need to practice on if I want to write better code, and it also told me that I can be more productive if I switch programming languages. There’s a bit less code to type in when writing code in Ruby than Java and I liked that, plus all the interesting libraries I can use. I switched to Sublime Text and used both Jenkins and the command-line interface more extensively too.
  • Late 2016. As I was looking for ways to speed up end-to-end tests total execution, which by then takes about 4 hours to complete, I ended up exploring testing apps in the HTTP layer instead of in the UI. That took a lot of studying of how our apps actually behave under the hood, what data are being passed around, how images are actually sent, how to view pages without a browser, how redirections work, among other things. After years of testing apps via the user interface, this was such a refreshing and valuable period, and I completely wondered why I never knew such a thing existed until then. It wasn’t being taught extensively to testers, perhaps because it all depends on how the app was structured to run through an API.

And these phases brings me to now, where there’s a healthy dose of API and UI layer tests all checking major app features. It’s all good, just several pieces needing a cleanup, a little parallelization, better test description language, and great documentation. It’s all good, because the lessons in both programming and testing keep piling. The two practices differ in mindset but I think they complement each other, and I think that there’s no reason anyone can’t do both.

The Lessons I Learned This Year About Software Testing

Last year I told myself I would take a step back from automation because I’ve already built tests with a working knowledge of Java and TestNG. Apparently, with the way things went this year, I was not satisfied after all with what I had. I went on to study Ruby with Watir and Cucumber for about half of 2016, and I learned how to run web applications without using browsers after that. And for a second year in a row I feel I overdid studying again, and maybe I’m getting too comfortable about writing test code. In any case, programming has become such a useful tool in my workflow – performing checks automatically that otherwise would have eaten up a chunk of my time and enabling me to test other things.

Some other realizations during the year:

  • Re-writing test code using Ruby with Watir and Cucumber and other useful ruby gems pointed out how messy the checks I wrote last year using Java was. It helps that the Ruby language has less boilerplate syntax but my earlier mistakes are more about refactoring and design rather than about the programming language itself. I still need more deliberate practice at that.
  • I’ve found out that I actually do security testing in my normal day-to-day exploratory work, but I’m not especially great at it because nobody has taught me how to perform good web security testing before. It seems that getting comfortable reading and manipulating API calls is a step towards becoming better at it.
  • Teams who develop software with agility does not necessarily mean they implement scrum or kanban. If there’s good communication and teamwork, teams can actually forego many of the rituals and focus on the actual work.
  • Writing automated checks with healthy test description language is an important skill.
  • People only learn to solve problems when they feel that those problems are worth solving. And problems that are worth solving for me does not mean they’re worth solving for other people too, so I should never stop using perspectives.
  • We can leverage REST calls with web automation to make them run a lot faster and more stable.
  • Exploratory testing is a prerequisite to automation.
  • Building a great team is difficult. There are a lot of factors to consider – team composition, empathy, attitude, member skill set, strengths and weaknesses, to name a few. It takes some time too.

Teaching Software Testing

Work, so far in the recent weeks, has been a little different this year than the previous years because I have been setting aside some regular time for teaching software testing to new hires. It feels great, like some of the presentations I’ve made in the past, but more demanding, not that I complain. After all, there are some things one can only learn by indulging in the experience.

Like understanding (and gauging) how much do I really know about software testing, about how software exploration happens, about what automation can and cannot do, about finding risks, and about what matters in testing.

Like how there is still a need for thinking time and note taking and other preparations, in order to deeply grasp what to discuss with the students and what lessons do I want to share, even if I already have the ideas about how to proceed in my head.

Like finding out what teaching techniques work well, and which one’s do not.

Like acknowledging that roles switch some of the time, that teachers are also students, and students are also teachers.

Like how fun it is to challenge colleagues, and see them learning and growing as time passes into remarkable people we expect them to be.

If You Want To Change

Then start small. Really, small.

Sure, write your big goals on (physical or digital) paper, and dream about all the great things you can do when you’re slimmer and healthier, more skillful in your art, more financialy secure, or more social, but please don’t expect those changes to just magically happen because you listed and thought about them. They’re necessary steps for the new you that you’re excited to be, but incomplete. All artists want to draw better, all musicians dream of playing music in the grandest stage, all writers think about being successful authors, all of us want to be fit, want to be less socially awkward, and want to travel more. We all want to make a living where we both have fun and generate enough income for our families. Big goals are fun to think of, but they can be overwhelming. Vague goals are easy to list but they are difficult to implement in our lives.

So start small. Pick the most important goal that you have in your list, and if it is big, split, slice, break it down to incredibly tiny, specific, chewable pieces. Instead of drawing better, you could re-write your goal to draw one portrait (or landscape or idea) for an hour every Saturdays of January in your favorite place. If you want to lose some pounds, you could start by religiously running every morning of every other day of this month just for 30 minutes in a park somewhere near, before everybody else wakes up. If you want to write a novel, you could timebox yourself to writing something about it every day for one month just for an hour right after you wake up, before going to work. If you want to learn a new skill, give yourself 30 minutes to research and practice it everyday at work. Test yourself, experiment for a month without excuses, see if you can do these small things and like your results. If you do, then do them again. If you don’t, ask yourself why. The thing to remember here is that in order to make big stuff come true, you need to make small changes first, no shortcuts. You should make few but bold decisions, you need to re-build your systems of working, you must review and replace habits that don’t work with ones that do, because that’s where it all really starts.