This weapon is one of our most well-known ”non functional” aspects I believe. There is a lot to unpack so let´s check the rhyme first:
”Just because something works with one single ball
Does not mean additional balls might not fall
Make sure you test with a much bigger load
Before you declare success and put the show on the road”
So, what does it mean?
There are multiple ways I can think of to interpret this card. Hopefully you can think of even more!
In The Twins I wrote about multi-threading and concurrency. I will touch upon that again today but dive into a few more ways load can show quality issues.
When testing concurrency we are testing that the software can handle multiple users doing the same thing. Say, two people from customer service trying to resolve the same customer complaint at the same time. Or say two people trying to buy the last item at the same time.
But there are other things that can go wrong with threading and with increased load. You can run into trouble such as race conditions, different kinds of database locks and variants of issues related to running things synchronous (waiting for the answer before proceeding) or asynchronous (not waiting for an answer before moving on).
The links above are more technical explanations of the concepts, I’m going to try to explain them with metaphors.
Imagine being at the store, trying to check-out your grocery shopping. Instead of one cashier per customer, they have decided that each cashier can handle multiple customers at the same time. One way would be having one cashier in the middle and one conveyor belt on each side of the cashier. Another could be that multiple customers put their items on the same conveyor belt, in a specific order or with some clear way of identifying which customer it belongs to.
It works really well most of the time but sometimes, an item ends up in the wrong customer’s bag or the cost of one item ends up on the wrong receipt. Or the cashier can’t handle the pressure and breaks down completely!
You could run into a race condition, where the cashier is expecting items to come in a certain order but they don’t so the result is not what was expected. Say the expectation was that items were delivered as one item per customer in the same order each time, but one customer threw in two items. This would result in the wrong items in the wrong bag or confusion and chaos.
Or say the cashier handles one customer on each side. They have a very good flow, matching the two moving lines in a steady tempo. Suddenly an item they don’t recognize shows up and they have to stop and check the price. Both lines will now have to wait until the check is ready, the transaction/database is locked until the check is done.
To symbolize asynchronous and synchronous imagine this:
One item is not recognized by the scanner. The cashier has to send a runner to check the price. Our choices are: Do we wait for the result before moving on with all customers (synchronous handling of the transactions) or do we carry on with the other customer’s items (asynchronous handling of the transactions)?
To top it off: there are way more things that can only be found with load.
- If we create too large transactions and hold everything until done, this might work fine with a small load but overflow buffers of eat memory with larger loads.
- Or we might not close/kill our transactions properly, which will also only show with load (or over time)
- Or we might read/write too much to the database which will result in slower response times, which might not show with a small load but will be worse and worse with load.
- Or we might update/reload the UI too often (or too much of it), which might also result in a decreasing user experience as load grows.
- As databases grow, indexing becomes more important and increased load to those tables will show performance issues that might not be obvious with just a few transactions.
Hardware might not hold up.
Your web hosting might have restrictions for how many calls they allow per day/week/month that you did not think of.
Or a zillion other things probably! To quote Dr. Seuss: ”Oh the things you can think up if only you try!”
Story time
One system I worked on had been around for a ”few years” (as in: many). It was built with a ”safety first” perspective, meaning that for each transaction we checked, double-checked and triple-checked before deciding to accept it.
This worked pretty well, it made sense in the context and served its purpose. But, alas, time changes faster than software and at this point in time it had ended up a two-pronged problem.
One problem was that the database grew and at the time it had millions of records to check. The complexity also grew and indexing everything is not really a viable option.
Another problem was that we wanted to move away from the ”guilty until proven innocent”-perspective and into an ”accept if we have no reason to believe otherwise”-perspective, which in our case meant that we added a lot of business rules to bypass manual steps. Which basically went against the architecture of that module.
The result of these two problems was that we ran way (WAY) too many and too complex business rules, each running against an ever-growing database which no-longer could serve up the data as fast as needed.
The transaction time grew, and grew, and grew. This in turn caused other issues and it was very complicated to fix without rewriting it all.
I don’t know if we would (could?) have built it differently at the time but simulating a large load would probably have shown it and given us more time to prepare a better solution. To be frank, having a thorough discussion about how the particular module could be (mis)used and how it would act over time would likely have been the best investment and saved lots of hours (and angry customers) later.
At the least it would have given us ideas on what, and how, to monitor to know when we needed to start preparing for refactoring.
Quote of the day
”Only conducting performance testing at the conclusion of system or functional testing is like conducting a diagnostic blood test on a patient who is already dead”
Scott Barber
Reading suggestions
Performance testing vs. load testing – Blazemeter
The dangers of race conditions – Ivan Mushketyk
Load testing, why should we do it – Strong QA ltd
Managing application performance throughout the lifecycle – Scott Barber