Measure p99 from request-duration observations over a defined time window, then use labels, traces, or request IDs to find which requests make up the slow tail. In Prometheus, a histogram is usually the right instrument when you need to calculate a service-wide percentile across replicas: aggregate its bucket rates first, preserving the classic histogram’s le label, and then apply histogram_quantile().
What p99 latency means
P99 is the latency boundary at or below which 99% of requests in a specified population and period completed. If p99 is 800 ms, approximately 99% of those measured requests took no more than 800 ms; the slowest 1% took at least that long. It is not the single slowest request, and it does not identify the cause of delay. Google Cloud Spanner’s latency guidance also cautions that p50 and p99 are poor indicators of overall performance when request volume is small.
As an Amazon Associate I earn from qualifying purchases.
Read p99 alongside p50, p90 or p95, and request count or rate. A stable p50 with a rising p99 is a clue that a smaller share of requests is slowing down; if the percentiles rise together, the impact may be broader. These patterns help direct investigation, but do not prove a cause. Amazon DynamoDB’s latency troubleshooting guidance similarly recommends percentiles to understand the distribution rather than relying only on an average.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Calculate p99 in Prometheus
For a classic histogram named http_request_duration_seconds, this query calculates p99 by service from observations in a five-minute rolling window:
#1 Best Overall
histogram_quantile(
0.99,
sum by (service, le) (
rate(http_request_duration_seconds_bucket[5m])
)
)rate() turns the bucket counters into observation rates over the selected window. The aggregation combines the bucket observations across the series being summed, and histogram_quantile() estimates the quantile from them. Keep le in the aggregation: it records each classic bucket’s upper bound. Replace the metric name and service label with those used by your instrumentation; each retained label combination produces a separate result. See Prometheus’s query-function documentation for the function syntax and histogram behavior.
For a native histogram, the query does not use an le label:
histogram_quantile(
0.99,
sum by (service) (
rate(http_request_duration_seconds[5m])
)
)Prometheus supports aggregation of histogram observations before quantile calculation; the classic and native forms differ in how bucket information is represented. Confirm which histogram type your metric uses before choosing the query. Prometheus’s histograms and summaries guide explains their respective aggregation properties.
Free tools Windows power users keep installed
One-click scans. No signup required.
Understand the estimate and choose buckets
A histogram records counts in buckets, not every request’s exact duration. Prometheus estimates a quantile by interpolating within the bucket containing it, so the result depends on the bucket boundaries and where p99 falls. Buckets that are too wide around the tail provide less resolution. An unbounded top bucket is particularly uninformative about how far beyond its lower bound observations extend.
Rank #3
- Compact Design: The Throwing Star LAN Tap features compact design that makes it incredibly portable. This passive Ethernet tap J1 J2 seamlessly integrates into your network without requiring power, allowing for easy installation and monitoring. By simply connecting it with Ethernet cables, users can obtain network traffic effectively, making it an essential tool for network monitoring.
- Efficient Monitoring: With dedicated monitoring ports, J3 and J4, the Throwing Star LAN Tap focuses on specific traffic directions, providing accurate and detailed insights. This targeted approach ensures that no vital network data is lost. It's suitable for users aiming to monitor IPTV source connections or obtain network packets efficiently.
- User Friendly Setup: Designed for convenience, this tap allows easy connection to existing network setups without complicated configurations. Simply attach the device to a network segment to start capturing data packets with your preferred software like tcpdump or . Its adaptable nature makes it suitable for both novices and experienced users looking to improve their network monitoring capabilities.
- Reliable Construction: Housed in a plastic shell, the Throwing Star LAN Tap is built to withstand the rigors of frequent use. The robust design ensures longevity and reliable performance in diverse environments, making it a trusted module for net monitoring.
- Versatile Compatibility: Compatible with various network equipment, making it a versatile tool for different monitoring scenarios. It operates seamlessly with a variety of Ethernet standards and configurations, accommodating users' unique needs. Whether assessing network traffic or establishing connectivity, this device consistently delivers excellent performance and flexibility.
- Set bucket boundaries close enough together around the latency range and tail that matter to your service.
- Make the highest finite boundary high enough to cover the observed range, while recognizing that extreme values beyond it are not precisely located by that boundary.
- Review whether the bucket layout supports the question you need to answer; a percentile estimate is not an exact record of an individual request.
For managed metrics, how the backend stores histogram data and estimates quantiles also affects resolution. Amazon CloudWatch describes histogram storage and the limitations of estimates from bucketed data in its histogram metrics documentation.
Choose histogram or summary instrumentation
Use a histogram when percentiles need to be calculated across replicas or when you may want to change the queried percentile or time window later. Histograms expose bucket counts that can be aggregated before the server calculates a quantile. Summary quantiles are calculated in the instrumented application, so their quantile values generally cannot be combined across replicas to produce a valid service-wide percentile.
Compare the options against the operational need, the instrumentation library, and the cost of recording and storing the data:
Recommended Free Tools
| Consideration | Histogram | Summary |
|---|---|---|
| Aggregate across replicas | Bucket observations can be aggregated before calculating a quantile. | Quantile values calculated in each application instance generally cannot be aggregated into a valid service-wide quantile. |
| Change percentile or window later | Query bucket observations for a different percentile or time window. | Exposed quantiles are calculated in-process; changing the calculation can require instrumentation changes. |
| Resolution and extremes | Estimate depends on bucket layout and the bucket containing the quantile. | Quantile calculation is performed in the application; behavior depends on the implementation and configured objectives. |
| Instrumentation and storage cost | Depends on the number of buckets and label combinations. | Depends on the summary implementation and exported series. |
For CloudWatch percentile statistics, raw datapoints are generally needed, with documented exceptions for particular statistic sets. Check the data shape before expecting a percentile statistic to be available. See CloudWatch’s statistics definitions.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Find the requests behind a high p99
- Set the scope. Choose the time window and request population, then check p50, p95, p99, and request volume together. A percentile from a small number of requests may not be stable.
- Break down the metric. Filter or group by labels available in the instrumentation, such as service, route, method, resource, instance, or operation. This narrows the part of the system contributing to the aggregate. For example, Google Kubernetes Engine’s control-plane metrics guidance uses labels such as
verbandresourceto investigate API-server latency and provides separate webhook-latency queries. - Check the measurement boundary. Establish what start and end points the duration covers. A service-side metric, a requester or edge view, total server request duration, and an SLI that excludes queue or webhook time are different measurements. AWS X-Ray notes that service-recorded latency does not include network latency between requester and service; its latency histogram documentation describes the distinct perspectives. Amazon EKS’s Kubernetes SLO examples also distinguish total request latency from an SLI that excludes webhook execution or waiting in a queue.
- Inspect individual slow requests. Use distributed traces or request IDs to identify requests that crossed a threshold, then inspect their spans and dependency calls. DynamoDB recommends logging request IDs for slow-request support investigations in its latency troubleshooting guidance.
- Test likely contributors. Follow the trace through queues, webhooks, downstream services, database calls, client resources, and network behavior. In the specific context of Kubernetes API-server latency, GKE lists webhook duration, large LIST responses, client CPU limits, slow client networks, and clients exiting while connections remain open as possible contributors; these are examples for that context, not universal explanations.
- Compare after a change. Recheck the same request population and measurement boundary over a comparable window. Otherwise a change in traffic volume or route mix can look like an improvement or regression even when request behavior has not changed.
Why p99 alone is not enough
A high p99 tells you that the tail deserves attention, not which request is slow or where its time was spent. A service-side duration may omit requester-to-service network time, while a total request measure may include queueing or webhook execution that a service-level indicator deliberately excludes. Choose the view that matches the user or reliability question, then use traces or request IDs to locate the slow spans within that same boundary.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
