MQTT Performance Testing using Gatling

5 min read
Feb 23, 2023 10:06:00 AM

MQTT (Message Queuing Telemetry Transport) is a lightweight protocol that is designed for use in machine-to-machine (M2M) and Internet of Things (IoT) communications.  It’s a simple and efficient protocol that allows devices to send and receive data messages over a network using the publish-subscribe model. 

The architecture of MQTT consists of three main components:

  • MQTT brokers
  • MQTT clients
  • Topics

The MQTT broker is the central server that receives and routes data messages between MQTT clients. These clients are the devices or applications that connect to the MQTT broker and send or receive messages. Topics are the channels or subjects that data messages are published to or subscribed to.

MQTT also includes the concept of Quality of Service (QoS) levels, which determine the level of reliability for message delivery. QoS levels range from 0 to 2, with 0 being the lowest level of reliability and 2 being the highest. 

Higher levels of QoS are necessary in mission-critical applications where data accuracy and timeliness are of utmost importance. Low levels of quality of service also help reduce message overhead, which can benefit applications with limited bandwidth or messages that are not mission-critical.

There are open-source MQTT brokers available, such as Mosquito, alongside closed-source commercial MQTT message brokers, such as HiveMQ, VerneMQ, and Bevywise MQTT.

 

Why performance test the MQTT broker?

Performance evaluation of MQTT brokers is necessary to ensure that the system can handle the expected load and usage patterns of data in real-time. Performance testing is critical to identify bottlenecks and potential scalability issues whilst ensuring that the MQTT broker can handle unexpected spikes in traffic from MQTT clients or devices in an IoT system.

 

Why is Gatling an excellent tool for MQTT performance testing?

Gatling is an excellent tool choice for MQTT performance testing of your IoT system. 

Whilst a typical use of Gatling is to simulate huge levels of traffic to test website traffic load, it also offers a specialist MQTT protocol. This protocol enables Gatling to establish connections against MQTT brokers and transmit continuous MQTT messages from minimal load testing infrastructure. Gatling users can also configure different QoS levels from MQTT clients and evaluate their response time.

For the performance evaluation of an MQTT broker to be realistic, it is vital to simulate tens of thousands of connections to the broker. Most MQTT load test tools are implemented on top of standard MQTT clients, using libraries that are designed to implement one single connection. These libraries don’t scale well to support the thousands of different connections required to simulate realistic MQTT performance tests.

The Gatling MQTT protocol is different from the ones offered by other stress testing tools. The protocol is built on top of Netty’s MQTT codecs and network framework, enabling it to scale effectively and emulate the required tens of thousands of connections that are needed for accurate MQTT performance testing.

To pinpoint performance bottlenecks in your MQTT broker, stress tests must also be executed regularly in a repeatable and comparable pattern. Not only does Gatling Enterprise allow you to execute MQTT performance tests on demand in the cloud with just a few clicks, but you can also compare the details of test execution results side by side.

MQTT simulation Gatling

Comparing MQTT performance runs Gatling

Configuring Gatling for MQTT Performance Testing

In the rest of this article, we’ll go over setting up Gatling for performance evaluation of an MQTT broker in an IoT system.

Gatling will simulate connections from numerous MQTT clients and transmit MQTT data messages to the MQTT broker in real-time whilst measuring the response time and providing other key metrics to conduct MQTT performance evaluation.

Note that MQTT performance testing is only supported in Gatling Enterprise. The open-source version of Gatling can be used to develop and compile MQTT performance testing scripts, but the script will only execute from Gatling Enterprise infrastructure.

Let’s walk through the steps to start MQTT performance testing with Gatling from scratch.

 

Create Gatling Project from Maven

The easiest way to start a new MQTT performance testing Gatling project is to clone the Gatling Maven Java Plugin Demo project from the Gatling GitHub repository. 

If you already have an existing Maven project that you would prefer to use, you can check the Gatling dependencies imported in the pom.xml of the same project and add those to your own Java project.

 

Add MQTT Dependencies

The dependencies for using the Gatling Enterprise MQTT DSL are not imported by default, so they need to be added at the top of your Gatling script as follows:

import io.gatling.javaapi.mqtt.*;

import static io.gatling.javaapi.mqtt.MqttDsl.*;

Write MQTT Gatling Script

With the MQTT dependencies added, we’ll use the MQTT DSL to write our Gatling script. Our MQTT performance test script will use two different scenarios that will both connect to the MQTT broker. One will subscribe to a topic and continue listening for messages, whilst the other will publish messages on the topic.

For this project, we will use the publicly accessible MQTT broker hosted at https://test.mosquitto.org. Our load test will simulate a single MQTT client connection from a subscriber alongside 1000 different MQTT client connections from publishers, ramping over a period of 30 seconds.

Our Gatling code for the MQTT performance test looks like this:

public class MqttSimulation extends Simulation {

    MqttProtocolBuilder mqttProtocol = mqtt
            .broker("test.mosquitto.org", 1883)
            .qosAtLeastOnce();

    ChainBuilder connect = exec(mqtt("Connecting").connect());

    ScenarioBuilder subscriberScenario = scenario("MQTT Subscriber")
            .exec(connect)
            .exec(mqtt("Subscribing_Topic")
                .subscribe("gatling/demo/topic")
                    .qosExactlyOnce() // Override the default QoS level
                    .expect(Duration.ofMillis(500)));

    ScenarioBuilder publisherScenario = scenario("MQTT Publisher")
            .exec(connect)
            .exec(mqtt("Publishing_Topic")
                .publish("gatling/demo/topic")
                    .message(StringBody("Gatling test message"))
.expect(Duration.ofMillis(500))
                        .check(jsonPath("$.error").notExists()));

    {
        setUp(
                subscriberScenario.injectOpen(atOnceUsers(1)),
                publisherScenario.injectOpen(rampUsers(1000).during(30))
        ).protocols(mqttProtocol);
    }

}

 

Execute MQTT Performance Test on Gatling Enterprise

We can now upload this Gatling MQTT performance test script to our Gatling Enterprise environment and execute an MQTT performance test while watching the results generated in real-time.

Remember that the MQTT protocol is different from the HTTP protocol that is commonly evaluated in Gatling and other performance test tools. HTTP is a request-response protocol where a client waits for a response from the server. The MQTT protocol, on the other hand, is a publish-subscribe protocol that isn’t blocking threads waiting for a response. This means that a response time will not be recorded in the Gatling load test report unless we specifically configure an MQTT check in Gatling to block and await a response of an MQTT message.

Relevant metrics to monitor relating to MQTT testing, such as TCP connections, can be found in the Connections tab of the Gatling Enterprise test execution page.

Gatling Enterprise MQTT simulation

Summary and Next Steps for MQTT Performance Testing

In this article, we’ve seen how Java developers can start utilizing Gatling Enterprise to conduct performance evaluations of MQTT brokers at a massive scale whilst enduring minimal cost or development time.

The ability to simulate realistic MQTT performance test scenarios with enormous levels of traffic using the expressive Gatling MQTT DSL, combined with widely distributed geographic load test infrastructure deployable on demand in the cloud with a single click, makes Gatling Enterprise an outstanding choice for MQTT performance evaluation of your IoT system.

For further information on MQTT performance testing with Gatling, refer to the Gatling MQTT Protocol documentation.

To learn more about Gatling and how it can help transform your MQTT performance testing efforts, refer to the Gatling Enterprise site and sign up for a free trial of either the Self-Hosted or Cloud versions today.