Pipeline-embedded load testing with Locust
Earlier in 2021 I’ve published a blog post on how to implement Locust via Terraform to conduct globally distributed load tests in Azure with Locust.io on Azure Container Instances (ACI). ACI in combination with Locust is a powerful solution to temporarily spin up infrastructure to run load tests from various locations around the world.
This blog post has raised a lot of interest and also the ask to conduct load tests as part of a deployment pipeline instead of standalone. This is the topic of today’s blog post.
The reason why customers want to run load tests embedded into their pipelines is to on one hand generate load in development and integration environments to identify issues early and before changes hit a production environment and on the other hand to do an active baselining to compare metrics across builds, deployments and code changes.
In this blog post we’ll not dive deeper into the architecture and deployment details of Locust and Azure Container Instances — this was already covered in my previous post — here we’ll focus on how to embed this into existing pipelines and processes.
The locust infrastructure
First of all, in my setup I’ve implemented Locust in a modular way into my Azure DevOps pipeline that allows me to re-use as much code as possible. This applies to the Terraform HCL code as well as to the YAML pipeline templates for Azure DevOps.
In the snippet below you can see my locals.tf file. It contains an environment_variables_common and environment_variables_master block containing the general environment variables needed for my Locust deployment. When deploying it in headless mode, we need some additional environment variables specified in environment_variables_headless block.
Important! Some of these environment variables, especially for the worker nodes, are very specific to my workload and might not be needed in other scenarios.
In addition to that, I’ve specified a variable in my Terraform definition called locust_headless which defaults to false. And only when this variable is set to true, the headless env vars will be used, otherwise not.
In line 22 in the gist above, you can see that environment_variables_common is merged with environment_variables_master and environment_variables_headless when var.locust_headless is set to true.
Important! One of the main differences between standalone and headless is that headless automatically runs tests based on pre-defined parameters and does not have a web ui.
This might be a bit confusing in the first place, but the result is that we can use the same Terraform HCL definition for our Locust (on ACI) deployment switching between embedded (headless) and standalone by using the locust_headless variable when using terraform apply/plan.
Pipeline integration
In my example I’m using Azure DevOps Pipelines. To deploy Locust I have a stage template YAML that can be used for standalone as well as for embedded deployments. This can be controled via its parameters and defaults to a standalone deployment.
The following diagram is supposed to visualize the stage template a bit to make it easier to read and understand:
The top part of the template (in blue) is applicable to both deployment types, the bottom part (in orange) is only used when parameter locustHeadless is set to true. This results in a couple of additional steps being executed like waiting till the test is finished (based on the parameter testDurationSeconds) and downloads the logs and stats from the Locust containers before the infrastructure is getting deleted (aka destroyed).
Analyze results
The previous steps are already sufficient to run an embedded load test, but to really add value we need to analyze the results and compare them against a predefined baseline. And if needed to fail our pipeline in case the metrics differ too much from our predefined baseline.
This is done in the “Parse logs and stats” step mentioned in the previous diagram. This step calls another template (keeping reusability in mind):
It calls a PowerShell Core script that compares our results against the baseline file stored in ./.ado/pipelines/config/loadtest-baseline.json.
The loadtest-baseline.json gist above contains a simple JSON structure that allows us to define thresholds for individual tests. The comparison is done as part of our pipeline run:
As you can see in the screenshot above, some tests have failed. This results in a warning in our pipeline:
We can now either adjust our baseline file or we can analyze the recent changes to figure out why the load test results were not within the scope defined in our baseline.
This is a simple way to implement load testing in your pipeline — I’m pretty sure that there are other soltions out there. Please let me know if you’ve questions or other ideas how to implement that.
