Continuous delivery (the CD in CI/CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. Each cycle begins with a developer committing code to the main repository. On each commit, the code is run through a series of automated tests. If all tests pass, the code is automatically compiled and deployed live to the production environment. If any of the tests fail, the results of the testing are reported back to the developer and they can make changes to the code and try again until all the tests are passed.
The most common way to implement CD is using an automated tool like Jenkins, TeamCity, CircleCI, or Bamboo that builds software whenever a developer checks in code. The build process can run unit tests, code style checks, static analysis tools, load tests, linting tools, and so forth. Once this process is completed successfully, it triggers an automated deployment process that copies your application onto servers so that customers can access it via a URL.
We’re Gatling, of course, we mentioned load testing! In a continuous delivery strategy, load tests are used to ensure the application can handle the expected traffic. The purpose of load testing is to identify potential bottlenecks, performance issues, and bugs before they happen in production. It’s important to understand that the goal of load testing in the continuous delivery pipeline is not always to simulate production traffic.
Some load tests are important to run in your continuous delivery strategy, however, some are best left as scheduled tests. It also really depends on the type of application you’re building, the traffic you’re expecting, and which tests are most applicable to you. Below, we’ll give you a few dos and don’ts for the different types of load tests you should be running in your continuous delivery pipeline.
Let’s not make this too confusing...
"Load Testing" encompasses all of the different types of load tests. "Load test" is a specific type of load testing. Got it? Clear? No worries; as a quick definition a load test is a test designed to replicate your expected traffic. This is a test you’ll want to run in your suite of automated tests each time you make a commit. Why? Obviously, you want to make sure anything that’s moving to production has been tested and can withstand the general amount of traffic you can expect without encountering any performance issues.
A soak test involves simulating users over a long period of time and is designed to detect any performance degradations that might occur as a result of consistent, long-term traffic. This is a very real-world situation and one of the most useful load tests in your arsenal, so why not run it in your Continuous Delivery process? By nature, a soak test is long. Ideally, the tests in your Continuous delivery process should be short.
Picture this: you’ve got a new commit you’ve just made and you’re hoping to get it live. You must wait for a 4-hour soak test to complete before anything goes live. Now, imagine for some reason this test, or any other test, fails. You receive the notification, make your changes, and have to go through another 4-hour wait.
So, what is your alternative for a soak test? That’s easy: schedule your soak tests to run nightly, or weekly. That way, you can be sure to get all the benefits of the test but not have to wait on them for deployments.
Does your application often experience spikes in traffic? Or could it? If this is a situation that you expect, running a spike test in your suite of continuous delivery tests is a must. It will keep your DevOps team happy knowing that you’ve prepared for the inevitable. If you generally have steady traffic it’s still not a bad idea to be running a spike test with every commit though; that way if you do get a sudden surge in traffic you can be confident that your application will be up to the task.
A capacity test is a load test designed to steadily add users to a scenario. It helps you find what your application’s breaking point is.
A typical scenario will add hundreds, or even thousands of users every minute or so. It will let you see how many users your system can handle before it starts experiencing issues and requests start failing. You can then decide to add additional servers if you expect this kind of traffic. However, as you might expect, a capacity test can take a lot of time. So, in the same vein as the soak test mentioned above, you can instead choose to schedule a capacity test, nightly, weekly, or monthly as suitable for your needs.
That being said, you could also decide to set up a capacity test with each commit. For instance, if you’re making changes for the specific purpose of increasing your application's capacity. That way, you will make sure the changes you’re pushing are positively improving your app.
In order to know whether your test is passed or failed, you’re going to need to set up an assertion in your scenario. Assertions are a way to get a binary answer to whether what you’ve pushed has met your base-level requirements or not. For example: “Did any of the request-response times exceed 500ms”. To figure this out you can use Gatling’s ability to set up assertions when you’re building your scenarios.
An assertion exists of 4 simple parts broken down below:
You can find detailed information on how to set up assertions in your tests in each of Gatling’s DSLs in our documentation here.
Great question! If you don’t want to go through that whole post though here it is in 4 steps.
I hope this has helped! Be sure to check back in often as we’ll cover other load testing tips in the coming weeks.