Analyzing gRPC Performance with Gatling on Qdrant Free Tier

Introduction

In the landscape of software development, gRPC has emerged as a framework for building efficient and scalable applications. As more developers adopt gRPC for its performance benefits, the need for load testing becomes critical. To address this need, we hosted a webinar with Sabrina Aquino from Qdrant.

Qdrant is an open-source, high-performance vector search engine written in Rust. It enables fast, accurate similarity searches across high-dimensional vector spaces, supporting features like filtered search, payload storage, and hybrid retrieval.

In this article, we'll provide a step-by-step guide on how to use Gatling to easily load test gRPC application with Qdrant's free tier as an example. You can also access the replay of our webinar on the topic.

Why should I load test a GRPC application

With more and more gRPC applications, load testing this type of application is important:

• Concurrency Handling: gRPC applications need to manage simultaneous connections. Load testing helps verify that your application maintains performance and stability under concurrency.
• Latency and Throughput: gRPC is designed for low-latency communication, and load testing ensures that your application meets expected latency and throughput requirements.
• Scalability: Load testing helps determine how well your gRPC application scales with increasing user load, ensuring its ability to handle growth without compromising performance.

Step-by-step guide to load test your GRPC application

Setting Up the Project

First, ensure you have Gatling installed and set up. If you haven't already, download it from our official documentation and create an account on the Qdrant Cloud.

You can now clone the project from GitHub, open the webinars/Qdrant folder, and create a free cluster on Qdrant. If you need guidance, please refer to their documentation.

Creating a free cluster

We need multiple free clusters to run our various simulations (collection, points, and queries). Since Qdrant provides only one free cluster per account, we will need to create and delete a cluster for each scenario.

Now that Qdrant is set up, let's proceed with load testing its gRPC API.

Gatling Enterprise

Package the simulation

Our simulation is currently written in Java, but we need to package it to run in Gatling Enterprise.

To do this, simply navigate to the webinars/Qdrant folder and execute the following command:

./mvnw gatling:enterprisePackage

Once the command is complete, you will find a JAR file in the target folder. This is the file we will upload to Gatling Enterprise.

Create the simulation

First, we need to upload the package we created earlier. Let's connect to Gatling Enterprise and click on Packages in the left sidebar. Then, click on Create name the package QdrantPackage select a team (in my case, the default team), and upload the JAR file we created earlier. Finally, click Save.

Select Simulation from the left sidebar and click on Create a simulation.Choose Test as code and, in the Package dropdown, select the package we created earlier.

Give your simulation a name. In the Simulation dropdown, choose the simulation you want to launch; in this case, select the Collection simulation. In the Locationfield, select one or multiple locations.

Click on the arrow next to Load Generators Parameters to expand it. Now, add the following three Java System Properties with their respective values:

1. The URL of your cluster.
2. The API key of your cluster.
3. Specify notopti or opti to indicate the type of collection you want to create.

Now, click Save to create the simulation.

Collection Simulation

Qdrant aimed to compare the performance differences between creating collections with and without optimization (e.g., HSNW, On Disk) using the gRPC API.

Since Gatling employs a load test as code approach, customizing our tests was straightforward. In the collection simulation, we create collections (optimized or not) until Qdrant returns an error status code, we also verify whether Qdrant successfully created each collection.

Now, let's examine the results and the differences between the two scenarios using Gatling Enterprise's compare run feature.

Thanks to Gatling, we observed that creating optimized collections takes less time, with a 5.92% reduction for the 99.99th percentile. Additionally, we can send 0.27% more requests per second.

Point Simulation

Now that we have observed the performance of inserting collections using Gatling, we want to examine the performance of inserting points into the two types of collections (optimized and not optimized).

In the Point simulation, we create a list of points with payloads and upload them to Qdrant until we receive an error status code. We also verify whether Qdrant has successfully uploaded each list of points into our collection.

Using Gatling's run comparison feature, we can see that the 99.99th percentile is reduced by 21.92% for approximately the same number of points. Additionally, we can send nearly 50% more requests per second.

Query Simulation

Now that we have added points to our optimized and non-optimized collections, we need to determine how quickly we can query them. This is the purpose of the QuerySimulation.

In the Query simulation, we first search for a list of points without a filter and then with a filter to assess whether using a filter impacts the response time or other parameters.

We can directly observe that queries are faster with collection optimization than without. Thanks to Gatling, we can easily determine if our query system scales well with an increasing number of users.

Conclusion

By using Gatling, Qdrant was able to assess the performance of its vector database under various scenarios and optimizations.

This guide underscores the necessity of load testing gRPC applications to ensure they meet performance expectations and can scale efficiently as user load increases. Remember to include the following in your load test strategy:

• Regularly update your test scenarios to reflect changes in your application.
• Carefully analyze Gatling reports to identify and address performance issues.
• Continuously refine your testing strategy based on insights gained from each test run.

For a more in-depth look at this topic, we recommend watching the replay of our webinar. In it, we provide a live walkthrough of this step-by-step guide.