If you’re brand new to Gatling and want to learn about our Open-Source reports Constance Armitage has written a great article you can explore to learn about the features, what they mean, and how they can help you measure the performance of your websites or applications.
When we talk about Gatling Enterprise Cloud one of the key points is that you get advanced reporting including load generator monitoring, TCP connections, DNS resolutions, and more. What are these features though and more importantly how can they benefit your load testing strategy? In this post, we’re going to dig into the differences between Gatling Enterprise’s and Gatling open source’s reporting, learn a few best practices, and answer your questions.
When you run a test in Gatling Open Source, the report is generated as an HTML file. The report is static which means that the charts and graphs you get can’t be filtered but you do get a very useful set of results including:
I’m getting there! Gatling Open Source reports while powerful and a great start are stored in separate HTML files. For Gatling Enterprise Cloud one of the first and most important changes is how reports are kept. In order to compare two reports with Gatling Open Source you need to open two reports separately and compare the results manually. However, with Gatling Enterprise Cloud you can directly and quickly compare any two reports with the click of a button.
When you upload a script to Gatling Enterprise Cloud and set up a simulation it is saved for you in your simulation dashboard. Part of a great load testing strategy is to run the same set of simulations time after time as you make changes and your application evolves to make sure it’s still reacting well to the traffic you’re expecting. By using Gatling Enterprise Cloud’s Run History you can get a quick snapshot of how your changes have affected your application or do a more detailed dive into the difference in response times and error rates in each request in your simulation.
Once you enter your Run History for a simulation you’ll be able to see and access each report for each run.
If you’d like to make a quick comparison between the request response times and error rates in each run select any two and click the “Compare” button.
This allows you a simple way to see the results of your changes, especially if you’re using Gatling Enterprise Cloud in your CI/CD process.
On the Run History page for a simulation, you also get access to 3 Trends graphs that give you a quick snapshot of the Requests and Responses, Response Time Percentiles, and Throughput over the history of a test you’ve run.
You can view your application's general performance over a long period of time easily or do a quick comparison to see if any changes you’ve made between runs have caused any improvements or regression in terms of performance.
Each of these trends can be filtered by scenario (if you have multiple scenarios in a simulation), Group (if you’ve set up groups), or Request so that you can drill into details over the history of your simulation.
That’s a quick overview of our Run History and Trends for Gatling Enterprise Cloud. If you have any questions or thoughts let us know by emailing contact@gatling.io and we’ll update this post in the future with FAQs and answers for you.
Let’s get into the full reports now.
As mentioned before Gatling Open Source reports are static html files that can’t be changed or modified. Gatling Enterprise Cloud reports are designed for collaboration.
To start with you have your run bar:
You can use the run bar to focus on specific sections of your simulation results. You can also collaborate with teammates by creating and sharing a public link, exporting to a PDF, or having Gatling Users in your organization leave comments on specific sections of a report.
This allows you to work with a full set of team members collaboratively on your load testing and also to get specific about which areas of your simulation you’d like to highlight.
We’re now going to dive into the tabs of a report and give you a little more information about each one, what they do, and how it can be helpful for you. Starting with…
The requests tab in Gatling Enterprise Cloud allows you to see the response times and error ratios for each individual request. In the default view you also can get global overviews of your errors per second, responses per second (with status), response time distributions, and response time percentile distributions:
If you’d like a closer look at individual requests you can switch from the chart view to the summary view to get a look at the response times and error ratios for individual requests.
That should be pretty obvious. This is where you can see the performance of your application based on the scenario you’ve created. You can look at your overall performance to see where you might be experiencing issues or drill into individual requests to see what might be causing a bottleneck in terms of performance.
When designing a scenario, you can create a group of requests to organize and model processes on a page. You can even nest your groups to get a little more organized. Here’s a look at how to create a group in your scenario using our Java DSL:
group("foo").on(
exec(http("name").get("/"))
);So, if you’ve created any groups in your scenario the Groups tab will give you the same information as the Requests tab but the results will be based on the groups of requests you’ve designated rather than individual requests.
As with the requests tab you can switch to a summary view as well to view response times and error ratios based on the group.
The groups tab helps you organize your requests and lets you see if specific processes will experience any issues under load when running your tests.
Load testing is all about seeing how your application reacts to traffic or simulated users. The “Users” tab lets you see how many users you have arriving, leaving (terminating), and at a given time (concurrently) in your simulation.
On its own, it’s a great way to visualize the users arriving in your test. In terms of testing though you can use it in conjunction with your requests page and the run bar to see what number of users is starting to cause problems in your application. By right-clicking anywhere on a chart you can add a marker that will carry across all charts. As an example below I have added a marker on the response time chart when load times start spiking so I can see on the “Users” tab how many concurrent users I have when the issues begin.
Here, you can find information about the transport layer of your simulation. This tab is full of tons of data including connection open and closing rates, information about the TCP connections, and the TLS handshakes:
Looking at the connections and their open and closing rates allows you to see if the test is running as expected. Looking at the TLS handshakes can help to see if the encryption process is a bottleneck in the response time of your website or application. TCP connections can help you understand the traffic that is coming to your application and help show you if accepting new clients on your application is working successfully or causing unwanted issues.
The DNS tab is incredibly helpful in determining, you guessed it, everything you need to know about the DNS for your website or application. Gatling Enterprise Cloud gives you the option of examining the DNS resolutions per second, DNS percentiles, DNS duration distribution, and percentiles for each hostname your website or application is using.
If you’re using a custom DNS you can use the information in the DNS tab to answer the below questions:
By answering these questions you can correctly configure your DNS in the best way to serve your clients faster and more efficiently.
In the load generators tab, you can see how much of each load generator you use. You’ll generally notice a spike in the CPU usage at the start of your simulation, this is the JVM warming up. Afterward, you’ll see how much load generator power you use to run your simulation.
When creating or editing your simulation on Gatling Enterprise Cloud you can go to the “Time Window” screen and set a ramp-up and ramp-down time to make your simulation cleaner and more accurate as both your application and Gatling’s load generators may need some warm-up time.
Once you’ve done this you can get a good idea of how many virtual users you can have in your simulation using the load generators you’ve selected and reconfigure your script to add more if you choose to.
When discussing Gatling Enterprise Cloud the most common question we’re asked is “How many virtual users can I generate with one load generator?” The answer from our end is “It depends on the scenario you’re trying to test. The theoretical limit is 35,000 - 40,000 users can be generated with one load generator or 300,000 requests per second. But, it depends on your scenario and the requests and API calls it contains. By using the load generators tab you can determine how much you’re stressing the load generator and get a real expectation of the load you’ll be able to generate for your specific scenario.
The load generators tab can let you know if performance issues in your simulation are a result of your application or whether you’re actually stressing the load generator too much with the load you’re trying to generate. You can also see how much load generator power you’re using and decide to add more users or load generators if necessary. This can help you choose your Gatling Enterprise Cloud subscription based on how many load generators you’ll need and how many credits you’ll use in your testing as well.
That’s our quick run-through of Gatling Enterprise Cloud’s reports. While this is a general overview it’s very likely that you have more questions. That’s great! We’re here to help! In order to help the most people possible please send us an email with any questions you have and we’ll publish an FAQ with all questions asked and their answers in the coming months.
If you’ve learned what you need to know and are ready to start click here to get your free trial of Gatling Enterprise Cloud.