How Coincheck turns exchange load testing into a 25-minute pass or fail verdict

Coincheck runs a crypto asset exchange in Japan where many customers trade through the API with automated strategies. Its engineers now start Gatling Enterprise runs through the API, and every standard exchange load test ends with a pass or fail report judged against six SLOs.

The challenge: every load test depended on who ran it

Many Coincheck customers trade automatically through the API. A small change around the API or order processing can shift performance under that kind of load, so the team has to measure before a change reaches customers.

Before 2021, load tests ran project by project. The people involved prepared the environment and tools each time. The SRE team had shared guidelines for good load tests, but there was no common environment and no common process.

In 2021, growing database load and plans for more trading pairs and features made the gap clear. The SRE team and the trading systems team built a shared performance testing environment. They compared open-source tools (JMeter, Gatling, Locust) with several SaaS options, and chose a load testing SaaS that could run load generators inside Coincheck’s own cloud environment. The scenarios were written in open-source Gatling.

Two problems remained.

1. Purpose and criteria: In 2021, a load test became a prerequisite for adding new trading pairs. It was not clear to everyone why the test was needed or what result was good enough. The team stepped back to agree on the purpose and the pass criteria first.
2. Manual operation and manual judgment: Each test meant launching load generating instances by hand in Coincheck’s cloud account, attaching IP addresses, waiting for them to be ready and uploading scenario files. A typical test then meant a full day of stress load patterns and about half a day of reviewing the results by hand. How a test was run and judged depended on the engineer running it.

In mid-2024, the SaaS was discontinued at the end of June. The team had about a month to find a replacement. They evaluated Gatling Enterprise and one other commercial tool, and moved to Gatling Enterprise in July 2024.

The tech stack added its own constraints: Ruby on Rails, MySQL on Amazon Aurora and Sidekiq for background jobs on AWS, monitored with Datadog, with scenarios kept as code in a dedicated Git repository.

The solution: existing scenarios, Private Locations, and an API to automate everything

Gatling Enterprise let Coincheck keep the Gatling scenarios it had written since 2021 and run them from Private Locations inside its own cloud environment. No rewrite, and no load generators running outside Coincheck’s network. The team started on Gatling Enterprise in July 2024.

Three capabilities carry the setup:

• Private Locations: Coincheck uses Private Locations only. The team picks a location and a number of load generators, and Gatling Enterprise starts them. The manual cycle of launching instances, attaching IP addresses and waiting for readiness is gone.
• Maven plugin: Scenarios are packaged straight from the Git repository and can be tested locally first.
• Gatling Enterprise API: Test runs start on demand, are monitored and have their results collected without anyone operating the web UI. Coincheck does not run load tests in its CI pipeline. It drives them through the API instead.

On top of these building blocks, the SRE team built its own automation: templated exchange scenarios with automated SLO checks. A standard run covers a normal and a stress load level, then returns a pass or fail verdict as a report.

The templates did what workshops could not. The team ran hands-on workshops for development teams, but people forgot what they learned if they did not run tests themselves. With templates, someone running a test for the first time uses the same scenarios and judges results against the same criteria as everyone else.

The main exchange scenarios are API-based tests that place and cancel orders at high volume. They reproduce how automated traders use the exchange API and show how order processing behaves under that load.

Load tests are now part of shipping rather than a calendar item. Teams run them before major releases, new features, and infrastructure or database changes. On the exchange team, even small fixes around the exchange API get a templated run. Each product team owns its own scenarios, and engineers and their AI coding assistants can run a test and read the results from the same development workflow.

The move also changed how Coincheck pays. It sized the plan from past usage and switched from pay-per-use billing to an annual subscription, which made the cost predictable. Because several product teams share the platform, the platform team manages the budget centrally.

Results: one standard, comparable answer for every engineer

About 25 minutes from launch to verdict

A standard check now runs at a normal and a stress load level, judges the results against six SLOs and produces a report in about 25 minutes, test run included. Before, a typical test meant a full day of stress load patterns followed by about half a day of manual review. Engineers now review a clear pass or fail result instead of starting from raw metrics. Quick checks are practical even for small changes, and the longer exploratory tests are saved for when they are needed.

Six SLOs on every standard run

Success criteria used to be defined test by test and varied with the person running it. Every standard exchange load test is now judged against the same six service level objectives:

• Order API latency
• Matching job execution time
• Matching loop cycle time
• Background job delay
• Database load
• Error rate

Because each run covers both load levels and returns the same format, results can be compared from one run to the next. Exploratory tests that locate bottlenecks are designed separately for each investigation.

“The first person to run a load test now gets the same answer as the most experienced engineer on the team.”

Masashi Inoue, Backend Engineer, Exchange Core Platform, Coincheck, Inc.

About 4x maximum order throughput

This result predates Gatling Enterprise and came from the open-source Gatling scenarios. In 2023, Coincheck redesigned its matching engine to process orders in batches. The team ran the same scenarios before and after each change, found the next bottleneck at every step, and confirmed that maximum order throughput in its load tests rose about fourfold.

About 10 configurations validated before the resilience upgrade

In late 2024, the team used Gatling Enterprise to test around ten candidate infrastructure configurations at production-level peak load before upgrading the exchange for higher resilience. That showed them the first bottleneck, and let them scale it out before rollout.

“We don’t guess anymore. Before any major infrastructure change, we let production-level load show us where our system reaches its limits.”

Load testing is no longer a specialist event

Engineers on the exchange team run the templated scenarios themselves before release and get a standardized report with an SLO verdict. Because results look the same every time, the whole team can review them and make the release decision together. Engineers have also started using the same scenarios to check larger exchange changes early in development.

“Load testing is no longer an event for us. It’s simply part of shipping a change.”

For an exchange, staying stable when markets are volatile and trading spikes is tied directly to customer trust. Load testing lets Coincheck ship features and infrastructure improvements without putting customers’ trading at risk.

What Coincheck would tell another team

• ‍Start with the question each test answers: Checking a release for regressions and hunting for a bottleneck are different tests, so Coincheck runs them differently. The standard run answers “did this change hurt us?”. Exploratory runs are designed per investigation.‍
• Templates beat training: Workshops alone did not make load testing stick. People got comfortable once they ran tests themselves, and templates made that possible for anyone.‍
• Keep scenarios as code: Scenarios live in Git next to the system they test. When the team builds a feature, it adds the scenarios alongside it, and both engineers and their AI coding assistants run them the same way. Load generators start on demand in Coincheck’s own cloud, so large tests do not require rebuilding infrastructure.