Performance testing with k6 and Grafana: turn load into evidence
How to model representative traffic with Grafana k6, define checks and thresholds, avoid coordinated overload, and correlate latency and errors with application behaviour in Grafana.
Begin with a performance question
A load test needs a decision to support. Can the account API sustain the expected weekday peak? Does latency recover after a burst? Which resource reaches its limit first? “Run 100 users” is a test configuration, not a useful question.
Record the environment, data volume, deployed version and external dependencies. A result from an undersized test environment is still useful when its purpose is explicit, but it should not be presented as production capacity.
Agree success criteria before the run. Otherwise a dashboard full of response times invites a favourable interpretation after the result is known.
k6 scripts describe behaviour and workload separately
A k6 default function describes one virtual-user iteration. Options and scenarios describe how iterations are scheduled. Keeping the user flow separate from the load model lets the same behaviour support a smoke, average-load or stress run.
Parameterise environment-specific base URLs and credentials without embedding secrets in source. Validate that the target is an authorised test environment before generating load.
Keep setup work outside the timed interaction where it does not represent user activity. Large shared setup can also create unrealistic data contention if every virtual user acts on the same record.
import http from "k6/http"
import { check, sleep } from "k6"
const baseUrl = __ENV.BASE_URL
export default function () {
const response = http.get(`${baseUrl}/api/accounts`, {
tags: { operation: "list-accounts" },
})
check(response, {
"list succeeds": (result) => result.status === 200,
})
sleep(1)
}Choose an open or closed workload deliberately
A closed model keeps a number of virtual users looping; when the system slows, each user completes fewer iterations and the offered request rate falls. This resembles users who wait before taking their next action.
An arrival-rate model starts iterations at a configured rate independently of response duration. It is useful when incoming demand does not slow merely because the service is struggling. It also requires enough preallocated and maximum virtual users to sustain the rate.
The two models answer different questions. State which one represents the production traffic and watch dropped iterations in arrival-rate scenarios; they indicate that the generator could not start all scheduled work.
export const options = {
scenarios: {
account_reads: {
executor: "ramping-arrival-rate",
startRate: 5,
timeUnit: "1s",
preAllocatedVUs: 20,
maxVUs: 100,
stages: [
{ target: 20, duration: "2m" },
{ target: 20, duration: "10m" },
{ target: 0, duration: "1m" },
],
},
},
}Checks protect correctness; thresholds decide pass or fail
A fast 500 response is not a successful performance result. Checks record whether responses have the expected status or content while the test continues.
A failed check does not by itself make k6 exit unsuccessfully. Thresholds define pass-or-fail criteria over metrics, including the checks rate, error rate and latency percentiles. This distinction matters in CI.
Use criteria connected to a service objective or accepted baseline. An average hides slow tail behaviour, so percentiles such as p(95) and p(99) are usually more useful alongside error rate and throughput.
export const options = {
thresholds: {
http_req_failed: ["rate<0.01"],
http_req_duration: ["p(95)<400", "p(99)<800"],
checks: ["rate>0.99"],
"http_req_duration{operation:list-accounts}": ["p(95)<300"],
},
}Representative traffic needs representative data
A test that repeatedly reads one cached account may measure a best case. A test that creates unlimited new data may measure database growth rather than the normal workload. Model the mix of reads, writes, identities and hot data observed or expected in production.
Generate unique data where writes must not conflict, and decide what cleanup occurs after the run. Avoid making setup itself the unmeasured bottleneck.
Think time should represent user pacing when modelling user behaviour. Removing every pause can turn a modest virtual-user count into an accidental maximum-throughput test.
Tags and custom metrics keep results attributable
Aggregate HTTP duration across every endpoint can hide one failing operation behind many fast health checks. Tag requests by stable operation name and apply thresholds to the important sub-metrics.
Use custom counters, rates or trends for application outcomes that built-in HTTP metrics cannot express. Keep tag values bounded; tagging every request with a unique identifier creates high-cardinality data that is costly to store and difficult to query.
Useful starting metrics include request count, HTTP failure rate and request duration. Add iteration and application metrics only when they answer the performance question.
Grafana connects load symptoms to system behaviour
Stream or store k6 metrics in a supported backend and display them in Grafana alongside application and infrastructure telemetry. A latency increase becomes more useful when it can be aligned with database time, error logs, CPU, memory or connection use.
Use a shared run identifier, deployment version and environment label so test and application data can be correlated. Annotate the ramp stages or retain the scenario rate on the same time axis.
Dashboards explain a run; thresholds still provide the automated decision. A visually interesting graph should not replace explicit criteria in CI.
- Offered and completed iteration rate
- HTTP error rate and check failures
- p(90), p(95) and p(99) latency by operation
- Dropped iterations and active virtual users
- Application errors and request correlation
- Database, runtime and host saturation signals
Protect the system and the test generator
Only run load against systems you own or have explicit permission to test. Coordinate production-shaped runs, set abort conditions and understand which third-party APIs, emails or payments the journey could trigger.
The load generator can become the bottleneck. Monitor its CPU, memory and network, especially at high rates or when response bodies are large. Distributed load adds clock, network and coordination considerations rather than automatically producing a truer result.
Warm-up can reduce one-off startup noise, but cold behaviour may itself be a requirement. Decide whether startup, steady state, burst recovery or saturation is the subject instead of discarding inconvenient samples.
Different tests reveal different failure modes
A smoke run verifies the script and environment with minimal load. An average-load test checks expected traffic, a stress test explores the capacity boundary, a spike test examines sudden demand and a soak test looks for degradation over time.
Do not run every profile on every commit. A small thresholded check can guard an important regression in CI, while longer or higher-volume tests run on an appropriate schedule or before risky releases.
Compare like with like: same environment, data scale, configuration and workload. Keep the script and thresholds versioned with the application so a historical result can be explained.
Performance work ends with a diagnosis or decision
When a threshold fails, identify where time and errors accumulated before changing code. A slow API can be blocked on a database query, connection pool, downstream service, execution context or resource limit.
Change one material factor, rerun the same workload and compare distributions rather than a single average. Preserve both successful and failed runs; only storing the best result destroys the baseline.
A useful performance test produces a defensible statement about one workload in one environment, together with enough correlated evidence to explain the limiting behaviour.