In the past load testing of a system would often be delayed until the end of a project, normally a few weeks before the system was expected to go live. Experience and time have taught the software development industry that this approach is fraught with problems, not least of all the cost and complexity of fixing issues at such a late stage.
These days it is much more widely accepted that load testing should start on the system as early as is feasibly possible, even when it is still in the early stages of development. It is therefore palpable that load testing should be built into the CI or CD solution, facilitating the execution of a load test every time there is a new version or build of the application created.
In this article, we will examine a few of the criteria to be aware of when it comes to integrating performance and load testing into your build pipeline.
Test Environment
If you are using an on-premises test environment to run your load tests against, it is important to ensure that the environment is in the same state each time you run a test. If the state is fluctuating between test executions, then this renders the test results as significantly less meaningful and makes it difficult to accurately spot performance trends.
It may be preferable to configure a test environment in the cloud, such that the environment is created dynamically by the CI tool before each build and subsequent load test run. If you opt to go this route, beware that in the cloud you’re often sharing a physical server with other applications. This has the potential to influence and skew your load testing results.
Test data
Next, you will need to identify the test data that your load test scenario is going to use and ensure that it is available in the build pipeline. You may need to provide test data for the load test scenario execution in the form of physical data files, such as CSV, TSV or JSON files.
Wherever possible, it is preferable to have test data dynamically created on an “as needed” basis by the load testing tool. Gatling offers powerful and flexible test data creation through Feeders, making it an excellent choice for load testing in a CI or CD environment.
Load Generation tool
This is the part where you configure the tool that will be used to generate load against the target application as part of the load test. If you are in the early stages of system development, and just want to run a relatively small and short load test against your target application, then a single on-premises load injector will likely suffice. Gatling was designed to be able to efficiently generate large quantities of load and throughput from a single instance, making it an ideal choice for execution as part of the CI process at this stage.
If your software development process is more mature, or you are looking to execute a large-scale test distributed amongst multiple injectors and locations, then a distributed load testing solution is necessary. Gatling FrontLine is an excellent solution to this requirement, as it easily integrates with a wide range of CI tools and facilitates the execution of distributed load tests in the cloud.
Monitoring Tools
To measure the performance and resource utilisation of the application’s backend systems, external monitoring tools need to be set up and configured to instrument the test environment’s infrastructure.
Beyond this, it is also useful to be able to monitor the progress of load test execution in real-time. The open-source version of Gatling will print out messages to the console every few seconds that update on the status and progress of the load test. Gatling FrontLine offers a far greater level of analysis and insight through customisable, extensive real-time test reporting.
Automate Build Pass / Fail Status
Before a load test is executed through the CI build, criteria should be defined whereby the build will fail if certain thresholds are not met. Examples of criteria that might be specified include the number of errors or the overall response time of transactions in the system.
Gatling has the capability to define such criteria directly in the scripts through assertions. If one or more of the assertions does not pass successfully Gatling will fail the CI build. This clearly indicates to all other users that there was an issue encountered with the load testing step of the build that is not acceptable against the defined non-functional requirements.
Analyse Test Results
Regardless of the build status, it is necessary to be able to analyse and interpret the results of the load test right away. Engineers will look for data such as transactional response times, the throughput of the system (i.e. requests per second) and counts of any errors that occurred. The trend in performance test results against previous builds and subsequent load tests can also be analysed.
Gatling open source offers a visual reporting plugin with Jenkins that facilitates all of the above. Additional plugins for other CI tools such as Bamboo and TeamCity are offered through Gatling FrontLine.
Cleanup
If you are running load tests in the cloud, then the cleanup process is a relatively straightforward procedure in that the only thing required is tearing down the test environment.
When running your load tests on-premises, it is a good practice to ensure that the test environment is restored to the state that it was in at the start of the load test execution. The CI tool that you use should have the capability to automate this functionality for you.
Summary
Those are the main factors to be aware of in regards to integrating load testing as part of your build pipeline in a continuous integration environment. With load tests as code, Gatling integrates seamlessly with a range of CI tools, making it the natural choice for adding load testing to your build pipeline.
Find out the best solution to shift-left your testing
Learn more about Gatling, the best developer tool to load test your applications, at gatling.io
The Gatling team
Related resources
You might also be interested in…
Peter
-
Customer Success Manager