We are back to the traps, so prepare for another moral story of how we can be better. Again, let us start with the rhyme:
“The Vigilant Watchers, lured astray
Miss the dangers, hidden away
Temptress´ trails you follow, effort low
But it won´t show you what you need to know”
So, what does it mean?
Well, in short: This is about the lure of the golden path.
We can call it positive tests, happy path, default path, basic flow or whatever we want, there are many names. Basically, it is about checking that the system does what we want it to do, when given proper input and all dependencies are working perfectly.
In my experience: This is not how reality works.
I´ve noticed that a vast majority of the tests that testers around me seem to do are of a “positive nature”, while I believe I see way more bugs found by the negative tests. Then a few years back I read a really interesting dissertation about TDD that mirrored those observations. In their project they had a ratio of 70/30 percentage of positive vs negative test scenarios created. At the same time, they had a reverse ratio where the negative tests found 70% of the bugs. I can´t determine from my own experience and one dissertation if this is a pattern but I find it very interesting and I believe there is truth in it.
And, when looking for data to support a hypothesis, of course you find it! (Oh, the irony of me referencing confirmation bias here)
Why? Well, first of all because it´s usually more defined what the system should do than what it should not. It is more… tangible. Looking at the different types of “requirements” I´ve seen come and go over the last 20 years it is obvious to me that we spend a lot more time on defining positive routes.
And of course, it´s way easier. For every single happy path there are endless alternative routes, some more obscure than others. With a lot of inexperienced testers out there, lack of proper training for testers (there is awesome education out there but so many sadly get none or bad training) and less and less time set aside for testing, it is natural that the easy testing is prioritized.
And I also believe people start here because they feel that if the golden path does not work there is no point in looking at the rest. And then perhaps time and/or energy runs out.
But if there is truth to the ratio, should we not do the opposite? If 30% of our tests find 70% of our bugs, why are we spending 70% of our time on something else? (Notice the likeness with the Pareto principle? I sure did!)
Food for thought.
Moral of the story: Try to challenge yourself and your hypotheses and consider if a few negative tests might bring more value.
There are almost too many examples to choose from here!
One team that I was indirectly working with released a new way of uploading files. It was a great improvement from the old version. It removed a lot of dependencies to external parties while at the same time shortening the waiting time for the user from hours to “real time” (or as close as you can get). We had a of of old test documentation to draw from, so it seemed like a very simple thing to test. However, in the end it turned out that the testing was almost exclusively of a positive nature, meaning of course the first time a user uploaded an unexpectedly large file it all came crashing down, with pretty big consequences for the users.
A quick code fix and a few negative tests (file sizes, file formats, pulling down integrations) later the problem was solved, and everyone involved had learned something. They learned the importance of testing more than the bare minimum and I learned not to assume that others know what I know.
In another project we were implementing a new web service. We had a lot of issues with it so at one point I was sitting next to two developers who were discussing the flow from start to end. I was listening while doing something else, but something grabbed my attention and, almost without thinking, I asked “So, what will happen with the transaction if System X does not respond?”
The senior dev stopped talking. Looked at me for a while, sighed and then they re-wrote the flow again. The difference of course was in out approaches to the problem. He was trying to confirm that his solution solved the problem. I was trying to falsify it. Not break it, mind you. But I find challenging my hypotheses is way quicker than trying to prove that it is 100% bullet proof.
Quote of the day
“Positive test bias is a critical concernLaura Marie Leventhal, Barbee Eve Teasley, Diane Schertler Rohlman
in software testing and may have
a seriously detrimental effect
on the quality of testing”
How-abouts and what-ifs – QA Hiccupps
Taking the negativity out of Negative testing – Sticky Minds, Jessica Lavoie
Analyses of factors related to positive test bias in software testing – Laura Marie Leventhal, Barbee Eve Teasley, Diane Schertler Rohlman
The Test case as a scientific experiment – Sticky Minds, David Coutts
Quality of test design in TDD – Adnan Čaušević
Negative Testing: Is it important – BoTree Technologies, Sejal Vala