API load testing with Gatling: From local scripts to enterprise-scale performance

70% of API performance issues aren't found until production. Most teams test functionally, not operationally: they confirm an endpoint returns the right JSON, not what happens to it under 5,000 concurrent users.

API load testing closes that gap. It simulates real traffic, measures latency and error rate under stress, and catches regressions before they reach production. This guide covers how to do that with Gatling: the core concepts, a complete worked example with real code and results, how to read what a load test tells you, and the parts of the topic most guides skip entirely, like testing AI and LLM-backed APIs.

What is API load testing?

API load testing simulates concurrent traffic against your API to see how it behaves under stress. It answers three questions:

• Can your API handle peak demand?
• How fast are responses at the 95th or 99th percentile, not just on average?
• Where does it break first: the database, a downstream service, connection pooling?

You measure three baseline numbers: response time (how long a request takes, end to end), throughput (requests per second), and error rate (percent of requests that fail under load).

Collecting those numbers is the easy part. Reading them correctly is where most guides fall short, and it's covered in Reading your results below.

Test types, matched to what you're trying to learn

Each test type answers a different question and needs a different load shape. Picking the wrong one gets you a green checkmark that doesn't mean anything.

In Gatling, these map directly to injection profiles: constantUsersPerSec for load testing, rampUsersPerSec for stress testing, stressPeakUsers for a spike, and a long during() duration for a soak test. Same DSL, different shape.

A worked example: load testing an API with Gatling

Here's a complete simulation in Java for an orders API: log in, then browse orders, with a ramp from 1 to 50 requests per second over 10 minutes.

public class OrdersApiSimulation extends Simulation {

  HttpProtocolBuilder httpProtocol = http
      .baseUrl("https://api.example.com")
      .header("Content-Type", "application/json");

  FeederBuilder<String> users = csv("users.csv").circular();

  ScenarioBuilder scn = scenario("Browse and order")
      .feed(users)
      .exec(http("Login")
          .post("/auth/login")
          .body(StringBody("{\"user\":\"#{username}\",\"password\":\"#{password}\"}"))
          .check(status().is(200))
          .check(jmesPath("token").saveAs("authToken")))
      .pause(2)
      .exec(http("List orders")
          .get("/orders")
          .header("Authorization", "Bearer #{authToken}")
          .check(status().is(200))
          .check(responseTimeInMillis().lte(800)));

  {
    setUp(
        scn.injectOpen(
            rampUsersPerSec(1).to(50).during(Duration.ofMinutes(10))
        )
    ).protocols(httpProtocol)
     .assertions(
         global().responseTime().percentile(95).lt(500),
         global().failedRequests().percent().lt(1.0)
     );
  }
}

‍

Be mindful of this, though:

• feed(users) pulls fresh credentials from users.csv for every virtual user, so every request isn't hitting the same cached session (more on why that matters below).
• jmesPath("token").saveAs("authToken") extracts the login token and stores it in the session, so the next request can use it. This is how you chain requests into a realistic flow instead of testing endpoints in isolation.
• rampUsersPerSec is an open workload model: Gatling controls the arrival rate, and the system's own concurrency emerges from how it handles that rate. This matches how real internet traffic behaves, users keep arriving even if your API is struggling. A closed model (a fixed pool of virtual users looping) would slow its own request rate down as your API slows down, hiding the exact problem you're trying to find. Don't use a closed model to test a system that's genuinely open to the internet.
• The assertions block is what turns "the test ran" into "the test passed or failed." No manual chart-reading required: if p95 response time exceeds 500ms or more than 1% of requests fail, the run fails, and in a CI pipeline (see below), so does the build.

A run against this scenario might report:

This passes the assertion (p95 is under 500ms), but the p99 number is worth noting anyway. One request in a hundred is taking nearly a full second, more than twice the p95. That gap between p95 and p99 is usually the first place a real production incident shows up: it's small enough to pass a threshold test today and large enough to become the norm once traffic grows.

Reading your results: percentiles, not averages

Gatling's own documentation is explicit about this: mean and standard deviation are not reliable metrics for load testing analysis. They only make clean sense on a normal, symmetric distribution, and real-world response times almost never look like that. They're typically skewed, with a long tail of slow outliers. Two very differently shaped distributions can share the same mean and standard deviation while one has a fine p99 and the other has a catastrophic one. Averaging hides exactly the thing you're trying to catch.

Percentiles fix this because they describe the shape of the distribution instead of collapsing it into one number:

• p50 (median): what a typical request experiences
• p95: what your slower-than-average users experience, worth setting as an SLO target for most consumer-facing APIs
• p99: where cascading failures and downstream timeouts start, because these are the requests that are slow enough to trigger a client retry, which adds more load, which makes more requests slow

One caveat: tail percentiles need volume to be stable. A two-minute test with a few hundred requests will show a p99 that bounces around wildly from run to run. Run long enough, or often enough, that the tail settles before you gate a release on it.

Managing test data at scale

If every virtual user sends an identical request, you're not testing your API, you're testing your cache. Real traffic has real variation: different user IDs, different search terms, different payload sizes. Gatling's feeders inject that variation from external data.

username,password
alice,hunter2example
bob,correcthorse
carol,batteryStaple

FeederBuilder<String> users = csv("users.csv").circular();

circular() means the feeder loops back to the start once it runs out of rows, so a long-running test doesn't crash when it exhausts a small file. Other strategies: queue (no duplicates, crashes if exhausted, the default), shuffle (randomized order, no duplicates), and random (duplicates allowed, infinite). Feeders also support JSON, and JDBC or Redis for pulling from a live data source instead of a static file.

This avoids two failure modes. The first is data collisions: if every virtual user tries to update the same record, you'll see contention that has nothing to do with your API's real capacity. The second is false cache hits: if every request uses the same search term or the same cached token, response times look artificially fast. Vary the input and you'll see the numbers your real users get instead.

Wiring load tests into CI/CD

The assertions block from the worked example above isn't just for local runs. When a Gatling simulation's assertions fail, the build fails with a non-zero exit code, which means a load test can gate a merge exactly like a unit test does.

name: Performance gate
on: [pull_request]
jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '21'
      - name: Run Gatling simulation
        run: mvn gatling:test

That last step is the whole mechanism: if the run breaches a threshold, mvn gatling:test exits non-zero, the step fails, and the pull request can't merge. Gatling has out-of-the-box integrations for GitHub Actions, GitLab CI, Jenkins, and Azure DevOps, so this same pattern works whichever pipeline you're already running. Teams typically run a small smoke-level version on every pull request and a heavier soak or stress test on a nightly schedule, since a full 10-minute ramp on every commit gets expensive fast.

Testing AI and LLM-backed APIs

This is the part most API load testing guides don't cover, and it's also where a load test is most likely to lie to you.

An LLM endpoint has the same HTTP shape as any other API, but the internals break the assumptions a normal load test relies on:

• Non-determinism: the same input doesn't produce the same amount of work twice. Response time is a distribution, not a fixed property of the endpoint. Test with one canned prompt and you've measured one narrow slice, not the real spread.
• Latency scales with output length, not just input: generation happens token by token, so a 500-token answer takes roughly ten times as long as a 50-token one. This splits into two separate metrics: time to first token (TTFT), how long until the user sees anything, and total generation time, how long until the answer finishes. A short prompt with a long answer feels snappy to start and drags at the end; a long prompt with a short answer is the reverse. If your test only uses short prompts, your TTFT numbers don't reflect production.
• Concurrency is bounded and expensive: inference runs against a finite pool of GPU slots. Once you exceed it, requests don't fail, they queue, and queuing on top of multi-second work produces a much sharper latency cliff than a CPU-bound web tier ever would.
• Every request has a variable cost: you pay per token, and output tokens cost several times more than input tokens. Slow requests and expensive requests are usually the same requests, which means your throughput and budget problems are coupled.

A load test that ignores all of this will pass cleanly and still miss the failure that shows up two weeks later: a retry storm, where rising latency triggers client timeouts, the timed-out request keeps running on the GPU slot anyway, the retry competes for that same scarce slot, and the whole thing cascades. A test with no modeled client timeout or retry behavior will never reproduce this, and it's one of the most common real AI outages.

Four things to build into the test instead:

1. Drive it from a real prompt corpus, bucketed by size (short, medium, long) and fed randomly in a mix that matches production. This alone surfaces the output-length variance a single fixed prompt hides. It also avoids an artifact in the other direction: if every virtual user sends an identical prompt, you may be measuring a cheap prompt-cache hit rather than a real inference cost.
2. Use an open workload model, the same principle as the API example above, with think time between a user's turns. A closed model self-limits its own concurrency as the endpoint slows down, which hides the exact queue buildup you're trying to catch.
3. Model client timeout and retry behavior explicitly. This is where the retry-storm cascade lives. A test client that waits forever engineers the most dangerous failure mode out of the test entirely.
4. Test against a real model, with cost controls. Mocking is fine for CI smoke tests, but variance and rate limits are the point of an AI load test. Use a cheaper model tier for volume runs, cap max output tokens, and set a hard spend ceiling on the test itself.

And measure four dimensions, not one:

1. Latency, TTFT and total time tracked separately, always as percentiles.
2. Throughput in tokens per second, not requests per second. Ten requests a second of one-line answers and ten requests a second of long generations are not the same load.
3. Cost per successful outcome. A response that got truncated mid-answer and still cost you money is pure waste; track cost per request that succeeded, not per request you sent, and consider a budget assertion the same way you'd assert on latency.
4. Graceful degradation as the pass criterion, not zero failures. At capacity, a healthy system sheds load cleanly, a fast 503 or a cost circuit breaker tripping, rather than melting into a multi-minute latency cascade. "Degrades gracefully under load" is a more honest target than "never fails," because the second one is fiction at high enough scale.

Popular use cases

E-commerce and retail. Flash sales, product browsing, checkout flows. Validate before a traffic spike hits, not after.

Financial services. Payment, trading, and fraud-detection APIs where millisecond-level latency and security constraints (mTLS, HMAC signing, JWT validation) both matter.

SaaS and microservices. Confirm services scale with the customer base and don't produce cascading failures when one dependency slows down.

AI and LLM inference APIs. Covered in detail above: validate an LLM or recommendation endpoint under concurrent, varied, realistic traffic, not a single canned prompt.

Running this at enterprise scale

Everything above works the same whether you're running it from a laptop or across a fleet of load generators, because Gatling's engine is fully asynchronous and event-driven rather than thread-per-user. That distinction is what determines how far a single generator can go before you need more machines.

A thread-per-user tool spins up one OS thread per virtual user, so CPU and memory climb linearly with concurrency and you hit ceilings well before you reach realistic traffic levels. Gatling's Netty-based engine doesn't block a thread per user, and pairs that with BoringSSL for efficient TLS handshakes and persistent connections to avoid the 60-second TCP port reuse delay Linux enforces on closed connections. The result: a single load generator can realistically sustain up to 60,000 concurrent virtual users or 300,000 requests per second, depending on protocol complexity. Gatling Enterprise customers reach 5M+ concurrent virtual users across a 20-generator fleet, and some run 100+ generators simultaneously across 200+ parallel automated tests.

That scale needs to be deployable somewhere your security team will actually sign off on. Gatling Enterprise Edition supports:

• SSO via SAML or OIDC, and RBAC for team-level access control
• Private load generators inside your own VPC, so test traffic and any sensitive data never leave your network
• Hybrid deployment, mixing Gatling-managed infrastructure with self-managed generators on AWS, Azure, GCP, or Kubernetes

This is the combination that makes load testing viable in finance, healthcare, and telecom, where a purely SaaS-hosted tool would fail a compliance review outright.

What this looks like in practice

Intuit runs Gatling Enterprise Edition across 1,000+ engineering teams and 8,000+ engineers, with performance test scaffolding shipped automatically in every new service repository. The numbers: 80,000+ load tests run annually, 97% performance test coverage across critical services, and 100% CI/CD performance integration, all without a single specialist team acting as a bottleneck.

"Any sort of outage means you're potentially disrupting someone's livelihood. On the small business side, an outage might mean payrolls are not going through, payments are not going through. We have no choice, reliability is not an option. It's a feature."

Chaitanya Bhatt, Principal Engineer, Intuit

On the throughput side, Attentive tuned gRPC and TCP connection pooling in their Gatling setup to go from roughly 6,000 RPS to 160,000 RPS per node in private test environments, with zero errors under load, and used the same tests to trace a gateway issue where only 40-45K of an intended 100K RPS was actually reaching the service.

Load testing your APIs is no longer optional

83% of internet traffic is API-based. Mobile apps, e-commerce, AI inference, IoT: APIs are the layer underneath nearly all of it, and that layer breaks under pressure unless someone has tested it under pressure first.

The practical shift is treating this as a continuous check wired into CI, not a one-time pre-launch event: test-as-code you can version and review like any other code, real-time results instead of a report generated after the fact, and the ability to scale from a laptop run to an enterprise fleet without rewriting anything.

{{author-bio}}