Understanding P50, P95, and P99 Latency Metrics#

The Problem with Averages#

Imagine an API receives 100 requests.

Most requests complete in around 100–200 ms, but a few requests take several seconds due to database locks, network issues, or resource contention.

Suppose the response times look like this:

1
100ms
2
120ms
3
130ms
4
140ms
5
...
6
180ms
7
200ms
8
250ms
9
5000ms

The average latency might still be relatively low.

An engineer looking only at the average could conclude:

“The system is performing well.”

However, users who experience the 5-second delay would likely disagree.

This is where percentile metrics become valuable.

What Is a Percentile?#

A percentile indicates the value below which a certain percentage of observations fall.

For example:

• P50 means 50% of requests are faster than this value.
• P95 means 95% of requests are faster than this value.
• P99 means 99% of requests are faster than this value.

Percentiles help us understand not only the typical user experience but also the experience of users affected by slower requests.

P50: The Median User Experience#

P50 is the 50th percentile, also known as the median.

Suppose:

1
P50 = 120ms

This means:

50% of requests completed in 120 ms or less.

P50 represents the experience of a typical user and is useful for understanding general system responsiveness.

However, it does not tell us much about performance spikes or slow requests.

P95: The Experience of Most Users#

P95 is the 95th percentile.

Suppose:

1
P95 = 500ms

This means:

95% of requests completed in 500 ms or less.

Only 5% of requests were slower than 500 ms.

P95 is often considered one of the most important latency metrics because it captures the experience of the vast majority of users while still exposing performance issues.

P99: Tail Latency#

P99 is the 99th percentile.

Suppose:

1
P99 = 3000ms

This means:

99% of requests completed within 3 seconds.

Only 1% of requests were slower.

Although that sounds insignificant, systems processing millions of requests per day can generate thousands of slow user experiences.

P99 is often referred to as tail latency because it measures the slowest portion of the latency distribution.

A Real-World Example#

Consider a login service with the following metrics:

At first glance:

1
Average = 200ms

The service appears healthy.

However:

• A typical user experiences around 150 ms latency.
• Most users experience less than 800 ms latency.
• Some users wait up to 5 seconds.

This illustrates why averages can be misleading.

Averages smooth out outliers, while percentiles reveal them.

Visualizing Percentiles#

Think of 100 users standing in a line from fastest response time to slowest.

1
|----50%----|----------------45%----------------|----4%----|--1%--|
2
     P50                     P95                  P99

• P50 marks the middle user.
• P95 marks the user at position 95.
• P99 marks the user at position 99.

Everything beyond P99 represents the slowest requests in the system.

Percentiles in Prometheus and Grafana#

When using Prometheus, percentile calculations are commonly derived from histograms.

For example:

1
histogram_quantile(0.50, ...)
2
histogram_quantile(0.95, ...)
3
histogram_quantile(0.99, ...)

These correspond to:

1
0.50 → P50
2
0.95 → P95
3
0.99 → P99

Grafana dashboards often display these metrics together to provide a complete view of system performance.

Which Metric Should You Monitor?#

Different percentiles serve different purposes.

As systems scale, engineers typically pay more attention to P95 and P99 than to averages.

These metrics reveal the hidden performance issues that users actually notice.

Key Takeaways#

• P50 shows the experience of a typical user.
• P95 shows the experience of almost all users.
• P99 reveals tail latency and performance outliers.
• Average latency can hide serious problems.
• Monitoring P95 and P99 helps identify bottlenecks before they impact users at scale.

A simple way to remember them is:

P50: What a normal user experiences P95: What a slightly unlucky user experiences P99: What the unluckiest users experience

In modern distributed systems, understanding tail latency is often more valuable than knowing the average response time. The users who encounter slow requests are usually the ones opening support tickets, leaving negative feedback, or abandoning your application altogether.