MQTT Performance Testing using Gatling

The rest of this blog post explores the MQTT protocol and how to load-test it with Gatling.

Introduction

MQTT (Message Queuing Telemetry Transport) is a lightweight protocol that is designed for use in machine-to-machine (M2M) and Internet of Things (IoT) communications.  It’s a simple and efficient protocol built on top of TCP/IP that allows devices to send and receive data messages over a network using the publish-subscribe model.

The architecture consists of three main components:

• Brokers
• Clients
• Topics

The broker is the central server that receives and routes data messages between clients. These clients are the devices or applications that connect to the broker and send or receive messages. Topics are the channels or subjects to which data messages are published or subscribed.

MQTT also includes the concept of Quality of Service (QoS) levels, which determine the level of reliability for message delivery. QoS levels range from 0 to 2, with 0 being the lowest level of reliability and 2 being the highest.

Higher levels of QoS are necessary in mission-critical applications where data accuracy and timeliness are paramount. Low levels of quality of service also help reduce message overhead, which can benefit applications with limited bandwidth or messages that are not mission-critical.

There are open-source brokers, such as Mosquito, alongside closed-source commercial message brokers, such as HiveMQ, VerneMQ, and Bevywise.

Performance test the broker

Performance evaluation of MQTT brokers is necessary to ensure that the system can handle the expected load and usage patterns of data in real-time. Performance testing is critical to identify bottlenecks and potential scalability issues while ensuring the broker can handle unexpected spikes in traffic from clients or devices in an IoT system.

Gatling as a performance-testing solution

Gatling is an excellent choice for MQTT performance testing of your IoT system. 

The most common use of Gatling is to simulate large volumes of traffic to load test websites. Still, load testing is also vital in low bandwidth applications, where the definition of large volumes can differ dramatically. 

You can test applications that utilize the MQTT protocol with Gatling’s plugin. This plugin enables Gatling to establish connections against brokers and transmit continuous messages from a minimal load-testing infrastructure. The plugin also allows you to configure different client QoS levels and evaluate the response time.

For the performance evaluation of a broker to be realistic, it is vital to simulate tens of thousands of connections to the broker. Most load test tools are implemented on top of standard MQTT clients, using libraries designed to implement a single connection. These libraries don’t scale well enough to support the thousands of connections required to simulate realistic performance tests.

The Gatling MQTT plugin is different from other stress testing tools. The protocol is built on top of Netty codecs and network framework, enabling it to scale effectively and emulate the tens of thousands of connections needed for accurate performance testing.

Stress tests must also be executed regularly in a repeatable and comparable pattern to pinpoint performance issues in your broker. 

Gatling Enterprise allows you to execute performance tests on demand from either the Cloud UI or your CI chain. The UI also has a run comparison feature, enabling you to track performance improvements or regressions over time.

Configuring Gatling for Performance Testing

In the rest of this article, we’ll go over setting up Gatling for performance evaluation of an MQTT broker in an IoT system.

Gatling simulates connections from numerous clients and transmits data messages to the broker in real time while measuring the response time and providing other key metrics to measure performance.

Let’s walk through the steps to start performance testing with Gatling from scratch.

Create Gatling Project from Maven

The easiest way to start is to clone one of our MQTT demo projects from the GitHub repository. For the rest of this post, we will use the Maven demo project written in Java. 

If you already have a Maven project you prefer, you can check the Gatling dependencies imported in the pom.xml of the same project and add those to your Java project.

Add Dependencies

Gatling does not, by default, import the plugin dependencies unless you start from the demo project. If you are working from an existing project, add the following code to the top of your Gatling script: 

import io.gatling.javaapi.mqtt.*;
import static io.gatling.javaapi.mqtt.MqttDsl.*;

Write a Gatling Script

We’ll use the MQTT SDK to write our Gatling script with the MQTT dependencies added. Our MQTT performance test script will use two scenarios connecting to the broker. One will subscribe to a topic and continue listening for messages, while the other will publish messages on the topic.

For this project, we will use the publicly accessible MQTT broker hosted at https://test.mosquitto.org. Our load test will simulate a single client connection from a subscriber alongside 1000 different client connections from publishers, ramping for 30 seconds.

Our Gatling code for the performance test looks like this:

public class MqttSimulation extends Simulation {
 MqttProtocolBuilder mqttProtocol = mqtt
  .broker("test.mosquitto.org", 1883)
  .qosAtLeastOnce();

 ChainBuilder connect = exec(mqtt("Connecting").connect());

 ScenarioBuilder subscriberScenario = scenario("MQTT Subscriber")
  .exec(connect)
  .exec(mqtt("Subscribing_Topic")
   .subscribe("gatling/demo/topic")
    .qosExactlyOnce() // Override the default QoS level
    .expect(Duration.ofMillis(500)));

 ScenarioBuilder publisherScenario = scenario("MQTT Publisher")
  .exec(connect)
  .exec(mqtt("Publishing_Topic")
   .publish("gatling/demo/topic")
    .message(StringBody("Gatling test message"))
    .expect(Duration.ofMillis(500))
     .check(jsonPath("$.error").notExists()));

 {
  setUp(
   subscriberScenario.injectOpen(atOnceUsers(1)),
   publisherScenario.injectOpen(rampUsers(1000).during(30))
  ).protocols(mqttProtocol);
 }

}

Execute a Performance Test on Gatling Enterprise

We can now upload the simulation to Gatling Enterprise and execute a load test while watching the real-time results.

Remember that the MQTT protocol differs from the HTTP protocol commonly evaluated in Gatling and other performance test tools. HTTP is a request-response protocol where a client waits for a response from the server. The MQTT protocol, on the other hand, is a publish-subscribe protocol that doesn’t block threads waiting for a response. Thus, Gatling does not record response times unless a check is configured to block and await a message response.

Relevant metrics to monitor for MQTT testing, such as TCP connections, can be found in the Connections tab of the Gatling Enterprise test execution page.

Summary and Next Steps for MQTT Performance Testing

In this article, we’ve seen how Java, Scala, and Kotlin developers can start utilizing Gatling Enterprise to conduct performance tests of MQTT brokers at scale while enduring minimal cost or development time.

The ability to simulate realistic MQTT performance test scenarios with high levels of traffic using the expressive Gatling SDK, combined with widely distributed geographic load test infrastructure deployable on demand in the cloud with a single click, makes Gatling Enterprise an outstanding choice for MQTT performance evaluation of your IoT application.

To learn more about Gatling and how it can help transform your performance testing efforts, contact our team to schedule a demo or discussion.