In this article I will discuss the software tests that can be carried out at the system level. These tests are an essential link in the continuous integration chain, they may or may not be functional and have very specific characteristics. I will discuss best practices to be followed and implementation issues, based on the example of system tests of an API. So, let’s see what the test levels are and what we call a system test.
The test levels
In accordance with the recommendations of the ISTQB (International Software Testing Qualification Board), software tests are classified into different levels. Here are the traditional test levels, which are found in the classic test pyramid:
- Unit tests : these tests aim to validate each code unit by isolating it (using stubs, mocks, or fakes…)
- Integration tests : these tests focus on communications or message exchanges between different code units
- System tests : these tests verify that the different aspects of a complete software system meet the specifications
- Acceptance tests : these tests aim to obtain acceptance of the system as a whole by the customer, user or representative.
At AT Internet, as our software solution is very extensive and complex, we have chosen to add two levels of testing:
- Systems integration tests : these tests focus on the exchanges between different systems that make up our solution
- Solution tests : these tests validate the proper functioning, as a whole, of our solution, which is composed of different systems
The system tests
These “system” tests are the first level of the so-called “black box” tests, i.e. they are defined on the basis of the product specifications (business rules, performance requirements, etc.) without any access to the source code. The so-called “white box” tests, on the other hand, are defined on the basis of the internal structure of the product (architecture, code, code portions, etc.).
These tests
must be completely independent of the internal structure of the system
(language and technologies used, implementation details etc…). They must
simply focus on the inputs and outputs of the system, observing it completely
objectively. This is what makes them powerful and useful: they then make it
possible to validate a profession or behaviours that are directly perceptible
to the customer.
System
tests are essential when modifying the internal structure of the system
(refactoring, replacing internal components, adding functionalities, etc.):
they ensure that these modifications do not lead to regressions for customers.
In Agility, we have no other option than to automate these tests, otherwise continuous integration is compromised by longer delivery times due to manual testing phases that can be endless. I will therefore only deal here with automated system tests.
While unit
and integration tests are essential and must be provided as much as possible,
it is essential to invest in tests at the system level to ensure that the
desired business is delivered to the customer.
Each
business brick can indeed correctly do what it is designed for, but their
assembly can sometimes lead to peculiar behaviour! On the other hand, the job
can be respected while parts of the system are sometimes in poor condition,
which is still rarer.
System tests are the first tests directly related to the customer’s acceptance criteria in the test process. These tests, if automated, will make it possible to detect possible regressions as early as possible, otherwise there is a risk of unpleasant surprises in later phases of the project: when the system is integrated into the complete solution or worse, at the customer’s or one of their representatives’ premises (product owner for example).
It is then
much more expensive to correct the problems detected and the risk of delaying
the delivery of the project is much greater.
It is not always easy to find the resources to set up and automate this type of test: it requires multiple skills, both in test design and automation development and we are not always lucky enough to have a test developer on the team. In some cases, product developers will be called upon to carry out some of these implementations.
Testing is surely part of the development activities and the responsibility for delivering a product that works is that of the entire team, isn’t it?
What’s the approach to
testing a system?
To be able to properly test a system, you must first understand various aspects of our system:
The system entries
These are
all triggers for system behaviour. We often think of user actions as obvious
triggers, but there may be many others. Here are some examples:
- A user action
- Receipt of notifications
- A change in system status
- The time that passes (yes, it can trigger actions and it even happens very often: synchronisation mechanisms, task scheduling, etc.)
The system outputs
We often
think here of the answer provided to the user but there are other very common
outputs that we do not always think about. For example:
- The
answer to the user - Log
writing (logs) - Writing
data to a database - The
issuance of a notification
Expected behaviours (or
business rules)
We now know
how to activate the system by playing on its different inputs. We also know
what we can observe by looking at the different outings. We then need to know
the relationships between inputs and outputs. These relationships are described
as system business rules and can take different forms: from the acceptance
criterion of a user story in Agility to more detailed specifications in other
contexts.
These more or less formally described behaviours will be used directly to describe the different test cases that will take on the role of system tests.
Execution contexts
Some system
behaviours depend not only on the inputs received but also on the context in
which the system is located at the time of the test.
Let’s take
the example of a call to an API to record new information in the system. We
then have two cases, with different imaginable behaviours:
- The information is already present in the system:
- We store both pieces of information
- We update existing information
- We store the history of the information values
- The information is not yet available
- We record the information received
- We don’t do any treatment
In both
situations, the execution context has a direct influence on the expected
behaviour of the system. This is where
the test dataset takes on its importance: we will generate the context in which
we want to be before each test to be able to validate the different behaviours
of our system. We then have a number of combinations of all these elements that
will constitute our test cases:
- A
test data set - One
or more inputs to “activate” - Expected
behaviour - One
or more outputs to be controlled
System
tests can then be implemented. This will require the implementation of a test
mechanism that must be able to play on the system inputs and control the
validity of its behaviour by observing its different outputs:
Depending
on the case, this test mechanism is directly made available in tools on the
market, depending on the system we want to test:
- API tests: SOAP UI, supertest, Postman, etc.
- Interface tests: Selenium, Cypress, NightwatchJS, etc.
In other
cases, it will be necessary to implement a mechanism adapted to your needs,
which must allow you to play with the different inputs and outputs of the system
under test:
- Reading/writing
in a Kafka topic - Sending
/ retrieving notifications - Insertion
of data into databases - Receiving
emails - etc.
It can be
seen here that the test tool is not linked to the technology used in the system
to be tested. It is often very different and is not related to the information
processing needs of the system but rather to the constraints imposed on the
client(s) of the system.
Non-functional aspects such as performance or resilience can also be tested in system tests, with different techniques and tools depending on the needs.
Testing an API
Let’s take
the concrete example of an API as a system to be tested. The main classics of
PLC validation are, in the order in which the system must check them:
- The rights of use of this API
- An unauthorised call can be rejected immediately, regardless of its validity
- Parameter validation
- The absence of mandatory parameters
- The validity of the received parameter combination
- Some parameters may sometimes be incompatible and should not be passed in the same call
- The format of each parameter
- Type, pattern, permitted values,…
- The consistency of the values received for different parameters
- For example, receiving a sort request on an unsolicited field can sometimes be refused
- Relevance related to the business of the system
- This involves validating the business rules of the system itself. For example, an information request (GET) on a non-existent property can trigger an error, the insertion of a data in the database via the API triggers the receipt of its identifier in return….
These
operations are increasingly costly for the system, which must try to return an
error as soon as possible if it is relevant, without initiating the next
steps, otherwise generating unnecessary load on the servers and thereby
altering non-functional aspects of the system, or even its security.
To design
these different tests, different design techniques can be used, including limit
value analysis, equivalence classes, state transitions, decision tables, etc.
Performance
or safety tests can be considered with different tools such as LOAD UI,
Gatling, JMeter, Neoload, for example. In this type of test, we find well-known
techniques such as:
- Injection tests
- The fuzz testing
- The hammering
- The gradual increase in load
- …
Conclusion
I hope I
have been able to enlighten you on this level of testing, its importance and
how to get the most out of it. It is also important to keep in mind that
testing, design and implementation always depend on your context, and “best
practices” must be continuously analysed, redesigned and interpreted to be
applied to your projects in the best possible way.
It is in
this spirit that we at AT Internet invest in the implementation of automated
system tests, according to the needs of each project and taking into account
the context of each team. We focus our investments in accordance with our test
strategy with the objective of ensuring the highest quality of our products for
the benefit of our customers.
In the next article, I will discuss 10 pitfalls to avoid when setting up system tests.
Featured photo credits: Markus Spiske
