Blog Image

When live gives you lemons – don’t let Bobby Drop-Tables steal them

Testing Posted on 2019-07-04 19:25:58

So my recent Power Hour at Ministry of Testing got me thinking.
To start: I’m sorry, tomorrow-morning-me, apparently I had to write this now, while on the rush of that experience.
Why am I feeling this rush? From a Q&A? Well. I’m doing a workshop about security testing and OWASP Juice Shop this fall and I’ve been worrying about it.
– Maybe it’s too easy and scripted?
– Will people learn anything?
– What if people who know more than me show up and get disappointed?
– What if someone gets stuck and I can’t help them?

Well the questions and reactions on the power hour (and discussions outside of that) have led me to realize a few things:
1. A lot of people are completely intimidated by the concept of security testing
2. A lot of the material out there expects a certain level of basic knowledge
3. And a lot of people can’t make it over that first hurdle
4. I’ve got this.

Yes. A few people will find it too easy. Some might even find it too hard. But you know what? Others will find it exactly right. And then it’s a win.

So. Let us back it up. I need to tell you a story.

My first direct contact with security testing was almost a decade ago. It was my first web project and in a security audit I learned about the concept of SQL injections for the first time. I was intrigued! I learned a bit, enough to make me able to check that it had been fixed, and added another tool to my tool belt without thinking about it.
Enough to understand Bobby Drop Table-jokes but not much more.
(Link to the comic: XKCD: Exploits of a mom)

In that same project I learned about testing SOAP and how to manipulate calls; and by doing that I made myself a little more versatile, again without realizing.

Next up was sessions and query string. I learned how to manipulate those and just how easily that could be exploited.
I think up until that point it felt like something only “true hackers” would do, and I didn’t really put much weight on those types of testing but with variables it became so easy! I mean if you put something glaringly obvious in your URL, someone is bound to be curious and try to change it! Someone who might not even mean to do harm.

Along the years I’ve picked up small things, piece by piece, to put on my pearl necklace of security testing and for me a big change came a few years back. In that year I had been reading a lot of audit reports, due to my role at that time, and after the third or fourth I was confused and annoyed.
– Are they just sending in the same reports with switched title pages? Because they sure say the same thing.
– This is not rocket science. I understand this. I know how to check for these things.
– We are making the exact same errors as in that first web project of mine.

Which led to:
– WHY… oh…. damn. Why am I not pushing for this? Why have I not made sure we test this?

Because if I can understand the reports I can figure out how to check it. If I can figure out how to check it I can find someone who knows how to fix it. If I can learn how to fix it and to check for it I can teach others. If I teach others they will grow and learn and somewhere along that line we might just stop introducing those errors over and over again.

And that is where my mission started. I want to find a way to make the bar low enough for anyone with basic software knowledge to take that first step. Learn that one thing. And then the next. And the next. I want people to understand that yes, it is a VERY complex area, and the difference between a TRUE expert and someone like me is huge, but that doesn’t mean I can’t make sure to check that the basic things that frankly all applications should make sure to protect against, are fixed, tested and in the end: never introduced in the first place. And THEN we can add some real penetration testing on top of that, by experts.
The workshop was born from that. And I hope after the workshop we can refine it, make it even better, keep building upon it and hopefully release it into the wild!

Security should be on our mind from step one and all the way to the end.
– It should be a factor in all requirements (such as: evil user stories, evil personas, threat modeling, just thinking “how could someone exploit this”)
– It should be a factor in our coding (such as: programming guidelines, SonarQube, Whitesource bolt, ZAP, just thinking “how could someone exploit this”)
– It should be a factor in our testing (such as: Bug Magnet, OWASP ZAP, Chrome DevTools security audit, Postman, just thinking “how could someone exploit this”)
– It should be a factor in our operations (such as: monitoring and analyzing behaviour, configurations, audits, logging… I can keep going forever and ever)

Build it into your pipeline. Build it into your culture. Learn it. Preach it. Teach it.

Testing vs. Checking – separate entities or part of a whole?

Testing Posted on 2019-03-12 12:42:51

There has been a lot of
discussion, heated and calm, about the concept of testing. What it is. What it
is not. Arguments about real testing vs. what a lot of people imagine when they
hear the word. Twitter and blog posts talking
about “real testing”, “the future of test”, “no testing” and so on, seemingly

To be honest, I’ve had a hard
time grasping a lot of it and some distinctions don’t make sense to me. But. I’ve
also had revelations and that is what this blog post will be about.

So, let’s start. What is
testing actually?

To me, testing = Checking + Exploration. Not
one or the other, both. (Shoutout to Elisabeth Hendrickson and her awesome book Explore it!)

Let’s begin with checking. The
stuff that some will say is not “real testing”.

My opinion? Well. If you _know_ something is
true/false, black/white, start/stop, by all means – check it. Go ahead, it’s
fine. Yes, I promise! If you _know_, from a reliable source, that you are
supposed to be able to login to the site with credentials XYZ from device A
then do it.

  • Is it important enough that you want to check it
    all the time? Automate it.
  • Is it time consuming enough that you notice
    yourself avoiding checking it manually? Automate it.
  • Is it something that would help the team to get
    instant feedback on when it’s not working? Automate it.

Checking, to me, has an
important part to play in what I call testing.

  • Important because knowing that I’ve checked the
    “obvious” gives me confidence to go further and deeper.
  • Important because knowing stuff works in the
    basic ways frees up headspace and time for more creative work.
  • Important because testing things in multiple
    different ways is more valuable than doing one way perfectly.
  • And let’s be honest: Important because it still
    find a lot of issues quickly.

Exploration, to me, is all the
other parts. The stuff that might be hard to explain to someone else.

To me, exploration is about
framing a hypothesis, inventing an experiment to try to _falsify_ it (yes,
falsify and not prove), executing the experiment and then observe, analyze,
learn, adapt.

  • It can lead to broken assumptions, a feeling of
    knowing less than before, Learning almost nothing useful. Which is ok, it
    will make me a better tester tomorrow.
  • It can lead to new stuff to check next time.
    Possibly/probably checks to automate. Which is awesome, it will allow me
    to do even more deep-diving next time.
  • It can lead to us having to re-think the entire
    solution. Which is GOOD, it will make the solution better.
  • It will find things checks can’t possibly do
    because we didn’t know about it and therefore couldn’t check it. It will
    find risks, loopholes, cracks in the ice and give us information that can
    let us make better decisions.
  • Some will say it’s more creative and fun, some
    will say it’s “real testing”, some will say it’s a waste of

I say it’s the stuff that
makes me a very valuable asset in any development team. It also makes me worry
too much. Imagine being unable to shut off the “hm. I wonder what
could possibly go wrong with this scenario” at any point of time in your
life, say for example while trying to look forward to travelling…

So, where am I going with
this. Well I am trying to explain why I think we have such a big gap between
the “Automate everything”-people and the “That’s not real

I confess I struggle with
models like the DevOps infinity-loop. It puts “TEST” in bold letters
in an isolated part of the loop and if that is your view of the world, how can
you ever take someone like me seriously? I find it seems impossible to find
common ground without a long discussion about “what is test really?”
and how do I explain that when I say test I don’t mean executing a particular
scenario in an application, I mean basically most of the things I do from the
time I hear a user express a need (yes, I am horrible to talk to) to the time I
stop working on a particular system.

  • I test the importance of that need by challenging
    it. Usually by talking to people.
  • I test the proposed solution by challenging it.
    Usually by asking questions or making explicit assumptions.
  • I test the process by challenging it. Usually by
    observing where people tend to take shortcuts or get flustered from not
    knowing how to proceed.
  • I test the solution under creation by challenging
    it. Usually by writing some sort of description of how I will test it and
    making as many people as possible tell me what I did wrong.
  • I test the solution when it’s “done” by
    challenging it. Usually by a combination of checks (the stuff I _know_)
    and exploration (the stuff I _believe_).
  • I test the delivery by challenging it. Usually by
    analyzing data, either from an analytics tool or from actually _talking_
    to people. (Did you know customer service tend to categorize calls? Do you
    know how much information you can get by just checking in with them after
    a release? It’s awesome!)

So, if that is what I mean
with testing and the person I talk to see it as writing some sort of test
descriptions, executing them and then report the result… it will clash. And
it does. Repeatedly.

Some will argue exploratory
testing encompasses all the aspects I’ve mentioned and won’t approve of me
separating them. They will argue exploratory testing is more a way of thinking
of testing and I totally get it. I really do!

But. What I’ve come to realize is
that to me, checking and exploration are two totally different mind sets that
make me act and think differently. Yes, call them what you want but I need to
do context-switching to change between them, just as I need to switch to
another way of thinking when building something. And I do. I switch. And I need
to do that switching in order to be the best possible tester I can be. So, to
me they are all tools I use, for solving different tasks.

I’ll stop there for now. In the next blog post I’ll share an example from a
previous assignment.

Cute kitten to calm any upset feelings:

Photo by Romane Van Troost on Unsplash