Would Heu-risk it? Part 12: The Twins

Another weapon for you to weild! I have some funny stories of how wrong things can end up when this fails. First as usual, rhyme-time:

”Your process might work like a charm with one client
Make sure you´re not on a single queue reliant
What happens the day more than one person try
And simultaneously demand the same thing to buy”

So, what does it mean?

Have you ever been to a web shop, browsing through. You see something you are interested in buying and you put it in your cart. After some more shopping you try to check out – only to be told that the item is no longer available! Can you remember how annoying that was?
Well, this card is about that, sort of.

See, threading is something that can mess up a lot of things.
In the example above, it might be a concious decision from the store to not reserve your item – more of a first come – first serve approach. Which can be ok, of course, but if it gets too common it might cost you users.

So, imagine each task someone tries to do in your system as one of the clients above. They each have something they want to buy, and they all think their request is the most important one. Your application’s job is to make sure you don’t mess up their orders, while still serve everyone as quickly as possible.

Imagine threading as queues in the department store. Every cashier is a separate thread and adding more cashiers let you serve more clients at the same time, but you still have to make sure everyone’s groceries stay groups together and separated.

In applications you also have the added complexity that suddenly the client might leave in the middle of checkout, halfway through, to go find something else. When they return, they expect all of their stuff to be intact, but you can of course not wait for them. You need to reverse their check-out, serve the next client, and be ready when client 1 returns. Without messing stuff up.

Oh, the ways this can go wrong!

There are a bunch of ways to solve this, they all have benefits and drawbacks. If you want to learn more, I recommend reading up on it. It is a great programming complexity to know about as a tester. Good search words: Multi-threading, concurrency, race conditions

Long story short – whenever you have threading in your application, make sure to test that multi-threading works! Depending on the expected load this can be a simple thing like 2 testers trying to do the exact same thing at the exact same time, or it might require complex load testing. Don’t forget it though!

Story time

So, once upon a time we worked with an application that had been around for a long time. It had a lot of complex batch jobs and archaic architecture.

We decided to replace a certain part with a web service, which would allow ”instantaneus” response to the user instead of having to wait for the next batch. It also allowed us to outsource the actual web service to another company, reducing the amount of code we had to maintain in-house.

The service was built, delivered and tested. Everything looked fine. We even had some load testing in place.

Release day came. Everything still looked fine.
Next release day came. Everything came crashing down. Oh my.
It turned out, the service could not handle multiple threads without messing up id:s.
In the next release, we had introduced another big improvement to our users, where we split a request into multiple requests. This meant the number of requests multiplied by… a lot.
Result? The service started mixing up id:s left and right, leaving us with a big pile of transactions ending up in the wrong place, everything going haywire.

So, what had we missed? Well, our load tests didn’t look for the important thing. They looked for a status – which was ’ok’. It never checked for the multi-threading issue, because in modern software development that is usually built into whatever framework ised.
Back in my programming days, this was something you had to spend a lot of time building yourself, but today we take it for granted.

Don’t. Please.

Quote of the day

”The possibility of incorrect results in the presence of unlucky timing is so important in concurrent programming that it has a name: a race condition. A race condition occurs when the correctness of a computation depends on the relative timing or interleaving of multiple threads by the runtime; in other words, when getting the right answer relies on lucky timing.”

Brian Goetz, Java Concurrency in Practice

Reading suggestions

Thread – Wikipedia
Concurrent computing – Wikipedia
Concurrency – Wikipedia
Race condition – Wikipedia
Race condition – Stack overflow
Testing multi-threaded and asynchronous code – Jonathan Halterman
Thread testing – Guru99
Testing for race conditions – Owasp
Testing concurrent systems – Donald Firesmith, Software engineering institute

Previous posts in the series