Skip to main content

The Importance of Technical System Knowledge

How knowing your system helps you automate it better.

In the world of test automation, we often talk about strategies, tools, flakiness, frameworks, coverage, pipelines, and environments. But there’s a topic that’s frequently overlooked, despite, in my opinion, being the backbone of effective test automation: technical system knowledge.

You can’t effectively automate what you don’t understand. As discussed in my previous articles in this series, there are many different types of automated tests we can do, but how do you choose the most appropriate? The most common approach is to treat the system as a black box, poke and probe it solely through the user interface (UI). Yet, we know that doesn’t work by the plethora of stories we hear about test flakiness and slow execution. It’s not that those tests are bad. If the risk is on the UI layer and they’re implemented cleanly, they provide value. But, if the UI is the only layer you are automating on, there may be better ways.

I know this because I’ve been that engineer, adding hundreds of UI tests a sprint, yet still struggling to get the value we were hoping for. However, in the last ten years I’ve changed tactics, as have many others, and started to get much more value from our automation efforts, and that’s by having a strong grasp of the system I’m working on. By this I mean, its architecture, the implementation, the data structures, the layers, the components, all the pieces of the puzzle that make that UI show what it does. The most valuable automation efforts I’ve seen are where this knowledge is used to create targeted automated tests.

In this article, I’m going to explore why technical system knowledge matters. We’ll look at how it improves the reliability of automated tests, their effectiveness and helps us avoid common pitfalls, and how it can help us better target our testing efforts. I’m going to help you trim your automated tests by helping you make them Targeted, Reliable, Informative and Maintainable. Which in turn will help you speed up your test automation, and how quickly your teams get the valuable insight they provide.

What Do We Mean by Technical System Knowledge?

Let’s start with the first word, technical. I’m referring to things like the architecture, data flow, the technologies used and system integrations. I’m not talking about writing the code, although the code could be a good place to learn about the technical system, it’s one amongst many. The system is the system under test, and specifically the technical knowledge of that system. While you may have experience of other systems that use the same architecture or toolings, it’s imperative you explore how they are used in this system.

The technical knowledge I always seek:

  • The technologies that make up the system
  • The system architecture, what are the main building blocks and how many of each exists, such as DBs / APIs.
  • How those building blocks communicate with each other
  • Where data is stored, and how it’s retrieved
  • How interactions on the UI translate to low system activity and back up again
  • What integrations do we have with other system, including our own and third parties
  • What configuration options exist

I’m not saying you need to know every single line of code or config, but instead gather enough technical knowledge so that you could explain the flow of data to somebody. For example, If a user fills in a form, that sends data to the API, which converts it to a type, sends it to the messaging queue, and eventually saves it in the DB. We need to know enough to make informed decisions about where the risks we are trying to mitigate are.

Put another way, if someone asked you to draw a map of the system, you’d have no issue knocking up a high level diagram with enough specific information.

How a Lack of System Knowledge Hinders Automation Efforts

Let’s explore some common situations teams find themselves in, which I believe could be down to a lack of system knowledge.

Large Tests

If you lack the understanding of how the system works under the UI, you won’t see alternatives approaches to automated tests. If you don’t know there are APIs, you’ll never think to create some API tests. Sure you’ll also need to know how to create API tests, but knowing APIs exist in the system comes first. That lack of understanding often results in large UI tests that navigate multiple screens, and exercise the full system from ‘end to end’. Another commonly misunderstood term, but it takes two definitions for me. End-to-end of the journey you are automating, e.g. shopping cart to order confirmation. As well as end-to-end of the system, e.g. from UI to Database, and typically traversing that multiple times.

These tests often take longer to create, maintain and execute, and due to traversing the system multiple times, are prone to constant change and flakiness.

Flaky Test

Flakiness is directly related to a lack of system and tool knowledge in my opinion. Often engineers point the blame at the tools/frameworks being used, but they are rarely at fault, the majority of the time it’s the author of the test.

Our automated tests will only ever follow the instructions they are given, if our instructions are lacking, so will the resulting test, garbage in, garbage out. Often our instructions are lacking because we aren’t fully aware of how the system works, even on the UI layer. Are you locators changing all the time? Why is it, what framework is being used? Pages sometimes load in a different order then expected, why is this? Inconsistent behaviour on the UI, what is causing this? We need to be answering these questions by expanding our system knowledge, a flaky test is an invitation to explore the system.

Over or Assumed Testing

If we don’t understand where logic lives in the system, we don’t know if we’re testing it at all. On the other hand, we may be testing the same logic multiple times across different layers without realising. Or we have a very large test just to test something on the final page. This can create a false sense of coverage, providing the illusion of confidence, while missing critical paths entirely.

System knowledge helps us target the right layer for the right test.

Gathering This Knowledge

You may be thinking at this point, that you agree, you’ve had these issues, and maybe, just maybe, more technical system knowledge may help you. But you don’t know how to gather that information, where do we start? Building technical system knowledge requires intentional effort, but the rewards specifically for test automation are huge, and justify the investment.

Here’s how to get started:

1. Read the Code (Even if You Don’t Write It)

I repeat, even if you don’t write code yourself, try reading it. There's a huge amount of insight to be gained from exploring the code base. Start by exploring the folder structure, this will help you orient yourself in the codebase. Then dig into specific files. Don’t worry if you don’t fully understand what’s in front of you, formulate questions and ask a colleague, or even AI if allowed. Look for conditional logic such as if statements, switch cases and understand what data is being used and how the paths alter.

Get involved with code reviews/pull requests (PRs). Often it’s easier to build up your code knowledge by looking at contextual changes. As you are usually aware of stories relating to these tickets, which provide framing.

Explore package management files such as package.json or .nuget files. They list all the packages that are utilised in the code base and can often direct further discovery.

2. Pair with Developers

Pairing is such a great way to gain technical knowledge. Code reviews, PRs, or implementing a story. You’ll be able to ask contextual questions, explore your curiosity and in turn it will help the developer cement their decisions and knowledge.

3. Trace a Flow

Pick a simple user behaviour, like logging in or submitting a form and trace what happens from click to database. What APIs are called? What happens in the backend? What logging is created? Is a message published? Where does the data end up?

Visualise it,  you’ll probably discover unknowns, those are opportunities to learn.

4. Root Cause Analysis

Every bug or live incident is an opportunity to learn. When a defect is found, don’t just rush to get it fixed, explore why it happened and where in the system it originated. This builds your system mental model and helps avoid similar gaps in the future.

5. Documentation / Specs

It’s common these days to have solid documentation and specs such as OpenAPI specs. These are a great source of technical knowledge, often framed with more digestible words than the code itself. Create for people who aren’t coders, but are comfortable reading code.

TRIM(S) Automated Tests

When it comes to thinking about automated tests and how we can best use all this technical system knowledge, I use my TRIM(s) mnemonic. This mnemonic helps me trim down existing tests, and in turn helps me create smaller tests from the off. Every time you create/review an automated test, ask yourself, is it TRIM enough?

Targeted

We need to think about risk and implementation. When it comes to risk we need to be selective about what we automate, because we can’t automate everything, as some things make no sense to automate. We should only automate 100% of the things that should be automated. One way to decide if something should be automated is to focus on the risk that the test would be mitigating. Is it a risk that matters, does the business care about this risk? If the test failed would the business care enough to take action?

The second part of targeted focuses on the implementation of the test. How close can we get to the risk with the testability we have? The risk could be mitigated on the unit layer of the system, we know this because we’ve invested in learning the technical implementation, but in this context developers refuse to write unit tests. You could pick that fight up, or you could choose to automate on the API layer instead, because the automation engineers are comfortable doing that. Or the risk is on the API layer, but the team doesn't have skills in that area. You could choose to invest in those skills, and pick a tool, or you could opt to automate the test on the UI.

In an ideal context, we’d also automate directly on the layer we believe the risk can be mitigated, but it’s simply not always possible, and that’s OK, every context is different, and an experienced automation engineer would adapt. We are aiming to be as targeted as the context allows us.

Reliable

If we are going to delegate some of our work to robots we need them to do that work reliably and consistently, we need the tests to be deterministic. We’ve discussed flakiness already, and its root causes, we have to avoid flakiness. If our tests aren’t reliable their value can be quickly diminished and that impacts our team's ability to make key decisions. When our tests are reliable, we create rapid feedback loops that we trust, and significantly speed up team decisions on quality. Fully understanding the technical implementation of our system will help us write more reliable tests.

Informative

Automated tests pass, and automated tests fail, or as I prefer to say, automated tests detect change. They are change detectors. If your team is moving at a good pace, they should be detecting changes frequently, and that's a win for us, we want that. However, when they do detect change, we don’t know if that change is good or bad, until someone starts investigating.

We can significantly speed up that investigation by utilising our technical system knowledge. We can name our tests appropriate to different parts of the system. We can automatically pull in log/observability data if we know it exists. We can map our understanding from pull requests to the failing test to connect the dots between the changes in the code and the detected change.

Maintainable

This constant change means that the tests need continuous maintenance, and we need this to be as easy and quick as possible. If we utilise our technical knowledge, we should end up with much smaller tests, therefore there is less code to maintain when these changes are detected. Having a solid grasp on the technical implementation of the system under test can also help you predict which tests are going to need changing during planning, and therefore include that effort in the story/feature.

Speedy

The bonus letter is for Speedy. Our teams want to move at an ever increasing pace, and we need to ensure our automation is not slowing them down, it should be enabling them. With layered tests based on technical system knowledge, which are quicker to create, quicker to maintain, quicker to execute, we should be facilitating this.

System Knowledge Isn’t Optional
Anymore

Technical system knowledge is the silent force behind truly effective test automation. It’s what transforms our tests from slow, flaky, large tests into fast, focused, and reliable  rapid feedback mechanisms that support delivery. By understanding how our systems are built and behave, from architecture to data, to integrations, we put ourselves in a far better position to build automation that’s effective. It enables us to create tests that are Targeted, Reliable, Informative, Maintainable and Speedy (TRIMS).

So, if you want your automation to keep up with your team’s pace and provide meaningful insight, don’t just learn the tool, learn the system. Because technical system knowledge isn’t a, it’s a core skill for modern testing.

About Automated Testing series

You're at the fourth part of our Automated Testing series - a collaboration with Richard Bradshaw. The series will explore many facets of Testing Automation and its implementations.

  • In the first part, we explored the Manual and Automated Testing paradox. Check it out here.
  • In the second part, we explored the different types of UI Automated Testing. Check it out here.
  • In the third part, we looked at the essential types of Automated API Testing. Check it out here.

The next part, coming out on 8 July 2025, will explore different types of API automated tests and their best use cases. Stay tuned!

Comments