In the past, load testing of a system occurred at the end of a project, usually a few weeks before the system's go-live date. Experience and time have taught the software development industry that this approach is fraught with problems, not least the cost and complexity of fixing issues at such a late stage.
These days, it is increasingly common for load testing to start as early as is feasibly possible, even when it is still in the early stages of development. Thus, load testing should be built into the CI or CD solution, facilitating the execution of a load test with each new version or build.
In this article, we will examine a few of the criteria to be aware of when integrating performance and load testing into your build pipeline.
Test Environment
If you are using an on-premises test environment to run your load tests against, ensuring that the environment is in the same state each time you run a test is vital. If the state fluctuates between test executions, the test results are significantly less meaningful, making it harder to spot performance trends accurately.
Configuring a test environment in the cloud may be preferable, such that the CI tool creates the environment dynamically before each build and subsequent load test run. If you opt to go this route, beware that in the cloud, you’re often sharing a physical server with other applications, which has the potential to influence and skew your load testing results.
Test data
Next, you must identify the test data your load test scenario will use and ensure it is available in the build pipeline. You may need to provide test data for the load test scenario execution in physical data files, such as CSV, TSV, or JSON.
Wherever possible, it is preferable to have test data dynamically created on an “as needed” basis by the load testing tool. Gatling offers powerful and flexible test data creation through Feeders, making it an excellent choice for load testing in a CI or CD environment.
Load Generation tool
Here, we will discuss configuring the tool to generate a load against the target application. Suppose you are in the early stages of system development and want to run a relatively small and short load test against your target application. In that case, a single on-premises load injector will likely suffice. Gatling generates large quantities of load and throughput from a single instance, making it an ideal choice for execution as part of the CI process at this stage.
Suppose your software development process is more mature, or you want to execute a large-scale test distributed amongst multiple injectors and locations. In that case, a distributed load-testing solution is necessary. Gatling Enterprise is an excellent solution for this requirement, as it easily integrates with a wide range of CI tools and facilitates the execution of distributed load tests in the cloud.
Monitoring Tools
External monitoring tools need to be set up and configured, instrumenting the test environment’s infrastructure to measure the performance and resource utilization of the application’s backend systems.
Beyond this, it is also useful to monitor the progress of load test execution in real time. The open-source version of Gatling will print out messages to the console every few seconds that update the status and progress of the load test. Gatling Enterprise offers greater analysis and insight through customizable, extensive, real-time test reporting.
Automate Build Pass / Fail Status
Before executing a load test through the CI build, define criteria whereby the build will fail if certain thresholds are not met. Examples of criteria include the number of errors or the overall response time of transactions in the system.
Gatling can define such criteria directly in the scripts through assertions. Gatling will fail the CI build if one or more assertions do not pass successfully, indicating to all other users that an issue encountered with the load testing step of the build is not acceptable against the defined non-functional requirements.
Analyze Test Results
Regardless of the build status, it is necessary to be able to analyze and interpret the results of the load test right away. Engineers will look for data such as transactional response times, the system's throughput (i.e., requests per second), and counts of any errors. Analysis of the trends in performance test results compared to previous builds and subsequent load tests is also interesting.
Gatling open source offers a visual reporting plugin with Jenkins that facilitates all of the above. Additional plugins for other CI tools, such as Bamboo and TeamCity, are available through Gatling Enterprise.
Cleanup
If you run load tests in the cloud, the cleanup process is relatively straightforward in that the only thing required is tearing down the test environment.
Restoring the test environment to the starting state is a good practice when running your load tests on-premises. The CI tool you use should be able to automate this functionality.
Summary
Those are the main factors regarding integrating load testing as part of your build pipeline in a continuous integration environment. With load tests as code, Gatling integrates seamlessly with various CI tools, making it the natural choice for adding load testing to your build pipeline.
