Automated SLO-based testing can transform your delivery pipeline;
By Andreas Grabner, DevOps Activit, Dynatrace.
More and more organisations should be embracing SLIs (Service Level Indicators) and SLOs (Service Level Objectives) as part of their testing strategy and delivery pipeline. Not only does automating SLO-based testing make the process easier, it enables software testers to deliver feedback faster and highlight issues earlier in the delivery cycle. Furthermore, it arguably ensures a more structured and better quality of testing.
What are SLIs and SLOs?
An SLI refers to the metrics or indicators of a particular tool that are interesting to the tester. Meanwhile, an SLO means the criteria which measures how good or bad an application is, based on the indicators specified.
Why use these metrics?
As well as making the testing process more efficient, which can help to control the cost associated with identifying and fixing an issue at a later stage in development, this approach can eradicate a common problem within the classic delivery pipeline associated with an application build.
This problem concerns the deployment and analysis of the tests that are conducted at different stages throughout a project. Typically, there is a stage of manual approval which involves someone looking at the testing results and manually comparing the information to determine if there is a problem and/or whether the build is better or not. This time-consuming task is made even more complicated when it involves unstructured data.
While organisations are monitoring and collecting more data nowadays, it is often unstructured and lacks context, making it difficult for software testers to analyse and identify the source of the data. Did it come from a test and if so, which test and at what stage? In turn, this makes it very hard to determine whether the build is performing as it should be or has improved.
This is where the benefit of SLIs and SLOs come in. By using SLI and SLO-based quality gates (milestones located at phases throughout the process which are dependent on the outcome of a previous phase), it is possible to automate & approve this process.
The software can also incorporate new objectives or adapt current ones through the life cycle of a project. If it is an intentional change, you can update the SLOs within your build. Alternatively, if it is unintentional – for example, you did not expect a particular outcome – it will make you aware of it. You can also track where the change happened because it will be automatically applied for the testing of subsequent builds.
What does this approach involve?
When you run functional, performance or API tests, you need to tag additional contextual information to allow the monitoring tool know that these results are coming from a particular test.
Using SLI or SLO-based quality gates, you can specify the metrics that are important to you – what you are looking for and which thresholds tell you that a build is good or bad. Open source projects, such as Keptn (an event-driven control plane which can orchestrate the execution of any type of test and evaluate SLOs against multiple SLI data sources as a self-service) not only automates this process, it also visualizes the results.
By comparing the metrics and thresholds you have set, it can show which of these elements are green (good), yellow (needs attention) and red (bad). It can also produce an overall score on the build, showing how good or bad it is. This visualization can highlight the problematic areas within the architecture of your application, showing where they are located – the front end, back end or both.
How to make the method effective?
There are a lot of factors to consider for incorporating this approach within your delivery pipeline. When do we run it? How much hardware is needed and who will run the infrastructure? How do we enable different workloads? To where do we stream the metrics and how do we analyse the results?
Thankfully, there are a lot of great do-it-yourself approaches which involve open source tools, as well as cloud-based solutions. The key thing to remember is that it will take time to get it right and it is important to test often.
It could be argued that deploying performance testing as early as possible within the process should be more of a priority for organisations – usually, it tends to be an afterthought. With the help of SLIs and SLOs, it is possible to embed it in the approach from the start and arguably increase the quality of your deliverables.
By looking at indicators early on, this approach enables you to work towards set goals, which can be validated with every single build, reinforcing a quality mindset and ultimately streamlining the way you test.
If you are interested in software testing and want to engage with like-minded individuals from across the globe, check out this year′s Quest for Quality conference. For more information and to reserve a spot, click here!
Have amazing topics worth sharing? Drop them on our Q4Q Knowledge hub, click here!
…and don’t be left out;