Data-Driven Testing

For new apps and software features, it’s perfectly normal to exhaustively test the matrix of all possible scenarios with equal importance before release – the happy path, the confused path, the forgetful or greedy paths, etcetera – whenever the tester can. After all, no one knows yet how exactly the new application will behave in Production, and it is always better (and cheaper) to find problems and fix them early instead of reacting as they occur.

For small software that are already in the wild and which do not change often, it is still a good rule of thumb to test everything with equal priority, if the budget allows for it. It’s still nice to catch bugs before the next version release rather than have a client surprise us at odd times with a system failure.

For complex legacy systems which frequently change and usually do not automatically test themselves, it is still a great idea to test features in full-scale, if we can, if it’s possible to perform all those complicated use cases within hours before the deadline. That’s a noble but a hugely difficult task for testers, if not impossible to accomplish, and something that might be a complete waste of time and creativity, or a foolish thing to do, even if it is very important for teams to have confidence in the stability of their applications. For such systems it might be better to re-evaluate what goals does our testing need to reach, re-think what sort of stuff do we want to find, re-learn what services do we want to keep providing to our customers as well as how effectively do we want to provide those services, and go from there. It might be better to spend some of our time on data monitoring and analysis, on understanding how our clients actually use our products and focus our testing those usage scenarios. It might be better for us to amplify testing and developing the features that our clients love. And it might be more worth it to regularly identify the riskiest parts of the system and improve their design and testability, rather than trying to keep up with the matrix we know we couldn’t possibly deal with everytime.

Apps evolve, and I think that the testing that we perform needs to change too according to our current contexts.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s