The Pattern beyond the pixels – Fit for purpose
So far, in what is quickly becoming a full blown small book, we have looked at the project and the problem, the danger of focusing too much on reactive work, changing the seemingly unchangeable and metrics and the dangers of measuring the wrong thing. This post is about a closely related problem that took me a long time to fully understand: what happens when you try to use one test suite to serve everyone — and why it almost inevitably serves nobody particularly well.
Everyone wanted something different
Once I started having proper conversations with the different people involved — developers, testers, product owners, managers — something became clear quite quickly. They all had completely legitimate needs from the automation. And those needs were not just different. In some cases they were actively in conflict with each other.
Developers wanted ultrafast feedback. The ability to make changes with confidence. For a developer applying a test-first approach, a red test is expected — even wanted. It means the feature isn’t done yet. A failing test is not a crisis; it’s the system working. For them, a report full of failures from ongoing work is mostly noise.
Testers wanted something completely different: feedback on the stability of a release candidate, and confidence that nothing previously working had broken. They needed to be able to test different release candidates in parallel, with different sets of data. For them, a failing test means a problem found that needs immediate attention. A bad report is not noise — it’s a blocker.
Product owners and scrum masters had a single question: is this safe to ship? A failing test means either we don’t release, or we have to make a difficult judgement call about severity. They needed a clear, trustworthy signal — not a noisy report they had to interpret.
Managers wanted strategic information. Are we spending money in the right places? What are the trends over time? Where are the root causes when things go wrong? They needed a different cadence and a different level of aggregation entirely.
All of these are completely reasonable things to want from a test setup. And we were trying to meet all of them with the same suite, running on the same schedule, producing the same report. Which meant it wasn’t really serving any of them particularly well — and sometimes the needs were pulling in directly opposite directions. Which of these people are you? Which ones are you trying to serve? (And yes, the answer can be more than one. Or all of them. Or maybe different ones.)
Tests (should) have layers
You don’t have to use one tool and one set of tests for everything. In fact, you shouldn’t.
Tests have layers — and different layers serve different purposes. Unit tests give developers fast, granular feedback at the component level. Integration tests check that pieces work together. End-to-end tests verify that whole user flows work as expected. On top of those you might have smoke tests for quick sanity checks, contract tests to catch breaking changes between services, performance tests, security tests, regulatory tests, chaos testing. The layers vary by organisation and product. (And anyone who tells you there is one right answer to how many layers you need is probably selling something.) The point is that each type answers different questions, runs at a different cadence, and costs different things to build and maintain.
What we had was primarily end-to-end, GUI-level functional tests run nightly. That tells you something specific and valuable — whether certain flows work from a user’s perspective at a given point in time. It does not tell you whether individual components are well-built, whether performance is acceptable, whether a change in one service is quietly breaking a contract with another, or whether a developer’s branch is in a shippable state right now.
Treating that one layer as the measure of overall quality is like trying to use a single instrument to play an entire orchestra. You get a sound. You don’t get the music.
The question to ask is: what do we actually need to know, and when? Then work backwards from that to figure out which layers to invest in. Layers of depth. Layers of cadence. Layers of focus.
Which bring me to the first glimpse of my last epiphany: This story was never about automation. It was about learning about human motivation, change management and having an impact. Because whatever it might look like – It’s always a people problem.
”The 2nd Law of Consulting: No matter how it looks at first, it’s always a people problem”
– Gerald Weinberg, The secrets of consulting, 1985
The sunk cost trap
One thing I found that I genuinely thought would be straightforward to fix turned out to be much harder than I anticipated. Not technically. Emotionally.
I’m talking about deleting tests. Retiring approaches that had stopped making sense. Letting go of things that had been built with real effort and real pride.
People hate getting rid of things. At the individual level: ”it might find something next time.” At the organisational level: ”we invested a lot into this.” The emotional barrier to removing something that people have built and are proud of — and that the organisation has spent real money on — is enormous. And completely understandable. But it causes real damage if it goes unchallenged.
Because every test run is a cost. And with automation – that cost is not as visible as when it is connected to a human pay slip. But in my opinion – A test should only be run if it has a value greater than its cost.
Cost can be visible, in the form of maintenance cost (the human resources needed), licenses, hardware, energy and other computing costs. Or it could be less obvious, in the form of complexity, execution time, slowing down time to market instead of speeding it up, context switching and mental energy needed, trust and confidence (in the software or your team),speed of feedback . That cost is ongoing, every single run, whether or not the test is still earning its place.
The question to ask is not ”has this test been valuable in the past.” That investment is gone regardless. The question is: does it make sense to keep running this from today forward?
The answer is sometimes yes. Sometimes no. Sometimes ”yes, but we should rebuild it differently.” The only way to know is to actually look at it honestly — which means tracking what things cost as well as what they return, and being willing to have the uncomfortable conversation when the maths stops mathing.
Of course there’s no simple threshold for when something starts costing too much — it depends entirely on your context. But measuring total run time and total maintenance time over time, and noticing when those start rising without a corresponding increase in value, is a simple place to start. The most valuable I’ve always found to be hard to measure – how confident and happy are your people? Do they feel joy working with it? Not a metric upper management will love, but one that builds trust and confidence in different ways.
In part six — the final post — we’ll look at the thing that underpins all of this and why none of the technical work above actually lands without understanding the human side of it. Change management and humans.
