Use tail sampling in Grafana Alloy to retain every error trace, traces that exceed a service-appropriate latency threshold, and traces carrying explicitly important attributes. Apply a small probabilistic policy only to the traffic that does not match those rules, and put that policy last because tail-sampling evaluation stops at the first match. There is no universal “best” percentage: the correct baseline depends on trace volume, storage and egress limits, service-level objectives (SLOs), and how much diagnostic detail your team needs.
Keep span metrics and service-graph data on an unsampled branch of the pipeline. If those signals are generated after sampling, request rate, error rate, latency, dependency relationships, and alerts describe only retained traces rather than all requests.
As an Amazon Associate I earn from qualifying purchases.
What Tempo sampling decides
Sampling is the process of determining which traces are stored in Tempo (or Grafana Cloud Traces) and which are discarded. A sampling decision is made for a complete trace, not for an individual log line or metric data point. Reducing the number of stored traces lowers storage and network use, but an overly aggressive policy can remove the exact evidence needed to diagnose an incident.
Recommended Free Tools
The two broad strategies make their decision at different times:
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
| Strategy | When it decides | What it can recognize | Main trade-off |
|---|---|---|---|
| Head sampling | At the start of a trace, before most spans exist | Inputs already known at trace start, such as a fixed ratio or selected instrumentation attributes | Low collector overhead, but it may discard an error or slow trace before that outcome is known |
| Tail sampling | After all or most spans for a trace have arrived | Status, latency, attributes, and combinations of evidence from the completed trace | Better critical-trace recall, with buffering, memory use, and decision-delay costs |
Tail sampling is therefore the usual choice when the requirement is “keep all failures and outliers, then sample normal traffic.” It cannot guarantee that every late-arriving span will be available: the collector must wait for a decision period, and traces that run longer than that period can be split into separate decisions.
Define what “critical” means for your services
Write the retention rules before choosing a percentage. A useful policy usually has four layers, in this order.
1. Keep traces with an error status
Match the OpenTelemetry span status code that your instrumentation sets for failed operations. Keep every trace whose relevant span reports an error. Confirm that libraries actually set status on failures; an exception recorded as an event without an error status will not satisfy a status-code policy.
Free tools Windows power users keep installed
One-click scans. No signup required.
2. Keep latency outliers
Choose a threshold from the service’s SLO or normal latency distribution. Grafana documentation uses 5,000 milliseconds as an illustrative latency-policy value; it is a configuration example, not a universal target. A checkout API with a 1-second SLO needs a different threshold from a batch job that normally runs for minutes.
3. Keep explicitly important attributes
Match stable, low-cardinality attributes that identify business or incident context, for example payment, checkout, authentication, or an incident-debug flag. Make the attribute names and values part of your instrumentation contract. Avoid matching unbounded values such as user IDs unless you have a deliberate privacy and cardinality design.
4. Apply a probabilistic baseline to everything else
Use a fixed ratio for ordinary, successful traffic after the critical policies. Grafana examples show 10% and 15% probabilistic values, and some guidance discusses a baseline of 1% or less. These figures illustrate configuration choices; they are not measured optima. Start with a value your storage budget can sustain, then adjust using observed volume and diagnostic misses.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
Tail-sampling policies are processed sequentially. Once any policy condition is satisfied, evaluation stops and later policies are not checked. Consequently, a broad probabilistic rule placed before the error or latency rules can discard critical traces at random.
Build the policy order in Grafana Alloy
In a self-managed Alloy or OpenTelemetry Collector deployment, place the tail-sampling processor after trace receivers and before the exporter that sends retained traces to Tempo. The exact component and field names vary by Alloy and collector release, so validate the configuration against the reference for the version you run. The logical structure is consistent:
receive OTLP traces
-> tail-sampling processor
1. status-code policy: ERROR
2. latency policy: service SLO threshold
3. attribute policies: payment, checkout, authentication, incident flag
4. probabilistic policy: baseline percentage
-> export sampled traces to TempoConfigure the policies as separate entries, not as one large expression whose precedence is hard to audit. Give each entry a descriptive name and record the reason for the chosen threshold in version-controlled configuration. A reviewer should be able to see immediately that errors and latency are evaluated before the baseline.
Set the decision period deliberately
The decision period (often exposed as a decision-wait setting) is how long the processor buffers spans by trace ID before evaluating policies. Set it longer than the common duration of your traces, including normal queueing and network delay. If it is shorter than many traces, one trace can produce multiple decisions and the stored result may lack spans that arrived later.
A longer period improves the chance that a complete trace is present, but it keeps more trace data in memory and delays export. Measure the 95th and 99th percentile trace durations and collector arrival delay, then choose a period with headroom rather than copying a value from another workload.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Use sampled and non-sampled decision caches carefully
Decision caches let the processor remember an earlier sampled or non-sampled result for a trace ID. When later spans arrive, the cache can apply the same decision instead of waiting for a new evaluation. This reduces inconsistent outcomes across a trace, but caches consume memory and can prevent a later span from satisfying a policy that was not visible at the first decision. Increasing cache capacity improves continuity at the cost of memory; size it from measured concurrent traces and retention needs.
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
Illustrative policy values
| Policy layer | Illustrative setting | How to tune it |
|---|---|---|
| Error status | Keep all spans/traces with status ERROR |
Verify instrumentation sets status consistently for failed requests. |
| Latency | 5,000 ms in a Grafana example | Replace with a threshold tied to the service SLO; the example is not a recommendation for every service. |
| Explicit attributes | Values such as payment, checkout, authentication, or an incident flag | Use controlled, documented values and avoid unbounded cardinality. |
| Probabilistic baseline | 10% or 15% in Grafana examples; a 1%-or-less baseline is also discussed | Choose from volume, storage budget, and the diagnostic value of normal traffic. These are illustrative values, not performance statistics. |
Keep RED metrics and service graphs accurate
Sampling traces and calculating service health are different operations. Request rate, error rate, latency, and dependency relationships should be derived from all spans whenever possible. If the tail-sampling processor runs before the span-metrics or service-graph connectors, those connectors see only retained traces and undercount traffic.
When sampling occurs in the collector before export to Tempo, fork the pipeline:
- Receive all spans from your applications.
- Send one branch to the span-metrics connector so request, error, and duration measurements include sampled and non-sampled requests.
- Send another branch to the service-graph connector so dependency edges and relationship rates represent all observed traffic.
- Pass the trace branch through tail sampling.
- Export the sampled trace branch to Tempo, while exporting connector output to your metrics backend as configured.
This placement keeps dashboards and alerts independent of the storage-saving decision. It also lets you investigate an alert with a representative trace sample without mistaking a reduced trace count for a reduced request rate.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Fixed-ratio sampling and metric scaling in Tempo
A fixed-ratio sampler makes a percentage decision without examining the eventual trace outcome. When the ratio is recorded with the trace data, Tempo can scale metric counts to estimate the unsampled population. Record the ratio used by the sampler and keep it synchronized with the configuration; changing the ratio without updating the recorded value makes scaled counts misleading.
Ratio scaling is distinct from policy-based tail sampling. A tail policy that keeps all errors and a small percentage of normal traces does not represent one uniform probability, so a single ratio cannot correctly scale every category. Treat policy-retained traces as a diagnostic dataset and use the unsampled metrics branch for authoritative RED totals.
Choose a baseline percentage from workload data
No authoritative universal optimum for Tempo sampling percentage has been established. Select a starting point with a simple capacity calculation:
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
- Measure incoming spans and traces per second during normal periods and peaks.
- Estimate the average retained trace size and the storage and egress cost of retaining 100% of traffic.
- Reserve retention for all error, latency, and attribute-matched traces.
- Apply the candidate baseline only to the remaining traffic.
- Check collector CPU, memory, queue depth, export failures, and Tempo ingestion after the change.
If the baseline produces too much data, lower it gradually or narrow the normal-traffic scope. If engineers cannot reconstruct common successful requests, increase it or add a targeted attribute policy. Do not lower the error or latency policies to solve a capacity problem; address collector capacity, retention duration, or the baseline first.
Estimate the memory impact of tail sampling
Tail sampling buffers spans until a decision is made. Memory pressure grows with incoming spans per second, average spans per trace, trace duration, decision period, and cache sizes. Peak traffic matters more than a daily average. Leave headroom for bursts, garbage collection, and other collector processors. Monitor resident memory and the collector’s out-of-memory protection; an evicted or restarted processor can lose buffered traces.
Validate that critical traces are not being lost
Test each policy with synthetic or controlled requests before relying on it during an incident. Use a test service or a maintenance window where possible.
- Generate a request that sets an error status and verify its complete trace arrives in Tempo.
- Generate a request slower than the latency threshold and confirm it is retained.
- Set each explicit business or incident attribute and verify the matching policy retains the trace.
- Generate ordinary successful requests and compare retained volume with the configured baseline over a large enough sample.
- Send a trace longer than the decision period and inspect whether it is split; then test again after increasing the period.
- Compare span-metrics request totals with application or load-balancer totals to detect undercounting.
- Inspect service-graph edge rates against known traffic before and after enabling sampling.
Record the policy name or sampling reason as an attribute when your collector version supports it. That makes it possible to answer “why was this trace kept?” without inferring the decision from the trace contents.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot common failure modes
Errors are missing
Check that the instrumentation sets the span status to ERROR, that the status-code policy appears before the probabilistic policy, and that the trace reaches the tail-sampling processor rather than being head-sampled earlier. Also check exporter queues and Tempo ingestion for drops after the sampling decision.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Slow traces are missing
Confirm the latency unit and threshold, then compare the threshold with actual trace duration rather than one individual span. Increase the decision period if slow traces routinely finish after the current wait. A latency policy cannot match evidence that has not arrived before the decision.
One trace appears in pieces
The trace duration or arrival delay likely exceeds the decision period, or the trace spans multiple collectors that do not share decisions. Increase the period, improve routing so a trace ID reaches the same sampling instance, and size memory for the longer buffer. Decision caches can help later spans follow an earlier result but do not reconstruct spans that were already discarded.
Collector memory or CPU is too high
Reduce the number of concurrent traces per instance by scaling collectors, shorten the decision period only after checking completeness, and right-size cache capacity. A lower baseline percentage reduces exported volume but does not eliminate the buffering cost of traces that must be evaluated for policy matches.
RED dashboards dropped after sampling was enabled
Move span-metrics and service-graph connectors to the all-spans branch before tail sampling. If fixed-ratio sampling is used, ensure the ratio is recorded so Tempo can scale counts. Policy-based retention cannot be corrected by applying one global ratio.
Self-managed Alloy versus managed Adaptive Traces
Self-managed Alloy or Collector tail sampling gives you direct control over policy order, thresholds, decision periods, caches, and capacity. You also own upgrades, scaling, memory sizing, configuration review, and failure recovery.
Grafana Cloud Adaptive Traces is a managed tail-sampling capability. Grafana describes it as providing recommendations for errors, high latency, and representative sampling. It can reduce the operational work of maintaining sampling rules, but you still need to validate that its recommendations match your SLOs, data-governance requirements, and retention budget. Confirm current availability and commercial terms for your Grafana Cloud region and plan before adopting it.
| Consideration | Self-managed tail sampling | Grafana Cloud Adaptive Traces |
|---|---|---|
| Critical-trace recall | Explicitly defined by your ordered policies | Managed recommendations for errors, latency, and representative traffic; validate behavior for your workload |
| Buffering and memory | You size decision periods, caches, and collector instances | Operational capacity is managed, while policy outcomes still need review |
| Metrics and service graphs | You must fork the pipeline before sampling | Verify how your configured telemetry pipeline separates unsampled metrics from sampled traces |
| Operational burden | Run, monitor, upgrade, and troubleshoot Alloy or Collector | Less sampling infrastructure to operate, with service and plan dependency |
| Cost control | Control compute, storage, and egress directly | Evaluate managed-service pricing and data-transfer implications for your region |
A practical rollout sequence
- Inventory current head-sampling and collector stages so you know where traces may already be discarded.
- Instrument and verify error status, latency, and the small set of explicit attributes that define critical workflows.
- Route all spans to span-metrics and service-graph connectors before any sampling processor.
- Enable tail sampling with error, latency, and attribute policies, followed by the probabilistic baseline.
- Choose a decision period from measured trace-duration percentiles and size memory for peak concurrency.
- Enable decision caches only after estimating their memory cost and deciding how to handle late policy matches.
- Run the validation tests for errors, latency, attributes, ordinary traffic, long traces, RED metrics, and service graphs.
- Watch collector resource use, retained-trace volume, Tempo ingestion, and on-call diagnostic success through at least a normal peak period.
- Adjust the baseline percentage or thresholds in small, reviewed changes. Keep the policy order unchanged unless you have a deliberate reason to alter precedence.
Operational checklist
- Error status policy is first and retains every matching trace.
- Latency threshold is tied to the service SLO, not copied blindly from an example.
- Business and incident attributes are documented, bounded, and tested.
- Probabilistic sampling is last and its percentage is treated as an initial setting.
- Decision period covers normal trace duration and network delay without exhausting memory.
- Sampled and non-sampled caches are sized and their late-span trade-offs are understood.
- Span metrics and service graphs receive all spans before tail sampling.
- Any fixed-ratio sampler records its ratio for Tempo metric scaling.
- Collector memory, CPU, queues, export errors, and restarts are monitored.
- Policy behavior is re-tested after instrumentation, Alloy, Collector, or Tempo upgrades.
The Bottom Line
Start with ordered tail-sampling rules that retain errors, SLO-breaking latency, and explicitly important workflows, then sample the remaining traffic probabilistically. Tune the baseline and decision period from measured volume and trace duration, while generating RED metrics and service graphs before sampling so cost savings do not corrupt your view of system health.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
