Another week means another trap. This one is a common problem in immature teams or teams where testing is not seen as a valuable activity. I can rant all day but by now you know it all starts with a rhyme:
“Oh sailor, the shallow waters beware
The surface is lovely, yet danger lurks there
Hidden rocks and monsters abound
Might suddenly run your ship aground”
So, what does it mean?
I am guessing this is one of the more obvious cards in the deck.
Shallow testing is fast, requires less skill and finds obvious bugs. There is a place and value to shallow testing – but it is not enough.
If you find yourself stressed for time, doing ad-hoc testing without purpose, scanning the application for anything obvious – you are probably doing shallow testing. A lot of unskilled testers stop there, missing out on anything below the surface.
Only testing the happy paths is one obvious way of testing too shallow but it is perfectly possible to do shallow negative testing as well.
Say you are testing maximum input in a text field. You check that the system stops you from entering more than 5 characters manually, but you forget to try to copy-paste. Or you test that you can add a new user and it looks ok, but you forget to check that it is _actually_ saved.
(Yes, this card interconnects with both The Temptress’ Trails and The Facade and a bunch more).
Checking your web page against the WCAG requirements but not doing anything more is shallow accessibility testing (in my opinion, others might disagree)
A skilled tester might start out shallow, letting their focus drift in the stream until something grabs their attention. Then they dive. They dig, prod, pull and push. Deeper and deeper until they are satisfied (or out of time). A skilled tester switches between shallow and deep, focused and unfocused.
Again: Shallow testing is not useless. It has value and a place.
It is great for quickly finding anything big and obvious.
Alas, too many people out there only do shallow testing. Either due to lack of skill, lack of time or as a choice.
If this is a conscious decision, say the risks associated with buggy software is very low in the context, go ahead!
Just know that shallow tests will never find the interesting or complex issues, unless stumbled upon by accident.
So, for the story this time I am going to talk about the application we created for our recruitment assignment: Bling-R-Us. This software was created to be used as a tool for assessing candidates in a recruitment process. It is a generic web shop with a bunch of bugs built into it.
The candidates where asked to read a specification and then got some sort of testing assignment around the application. Looking at the result you could see some interesting things, main point for this blog post is that unskilled testers stopped at a certain point and never ventured past that. They found the shallow bugs but more importantly – they were so focused on the shallow bugs they never found anything more interesting. And they felt “done” much sooner, pleased with having found so many issues.
The interesting candidates, even the junior ones, could ignore the obvious ones (although you could see that they noticed them of course) and grab hold of the hints of deeper issues and try to figure them out. An example is a calculation that was… slightly off. If you just had a quick look it looked ok but if you did some more complex tests, it became worse and worse.
These candidates also, without fault, felt they had a lot more tests to do when the time was up.
I’ve written more about this here.
Quote of the day
“Shallow testing find shallow bugs”Maaret Pyhäjärvi
Null – QA Hiccups
Deep exploratory testing – A Seasoned Tester’s Crystal Ball
Deeper testing – Developsense
Software Quality Approaches: Testing, Verification, and Validation– Michael Haug, Eric W. Olsen, Luisa Consolini