Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteUber now discovers service dependencies from production metrics emitted for every request, rather than waiting for rare failures to appear in sampled distributed traces. Its system builds caller-to-callee call trees, joins each downstream error to the request that triggered it, and classifies each edge as fail-close, fail-open or unknown using explicit probability thresholds.
The approach gives Uber a continuously updated view of which service-and-endpoint relationships can break an API, while retaining static analysis for relationships that runtime traffic cannot expose.
As an Amazon Associate I earn from qualifying purchases.
What Uber’s dependency model captures
Call trees made of services, endpoints and datastores
Uber represents an API’s dependencies as a call tree. Each node is a service-and-endpoint pair or a datastore, and each edge records a caller invoking a callee. This endpoint-level detail matters because one procedure on a service can be critical while another is optional; a service-only graph would merge those different failure behaviors.
Fail-close and fail-open dependencies
A fail-close, or hard, dependency is one whose failure consistently causes its caller to fail. A fail-open, or soft, dependency can fail while the caller still completes successfully, typically because the caller has a fallback, treats the data as optional or otherwise absorbs the error. Uber also keeps an unknown category for relationships whose observed behavior falls between those two extremes.
Why sampled traces are too slow for rare failures
Distributed traces provide detailed views of individual requests, but sampling can make a rare dependency failure take hours or days to observe. Uber’s example assumes an API handling 500 requests per second, 99.9% availability and a distributed-trace sampling rate of 0.01%.
| Measure | Value in Uber’s example | What it means |
|---|---|---|
| Throughput | 500 requests per second | Traffic used for the example calculation |
| Availability | 99.9% | One tenth of one percent of requests are failures under the assumption |
| Trace sampling | 0.01% | Only one request in 10,000 is retained as a trace |
| Expected time to sample one failed request | 5 hours, 33 minutes, 20 seconds | Estimated under those stated assumptions |
| Expected time to observe ten such failures | 2 days, 7 hours, 55 minutes, 31 seconds | Uber’s estimate under the same assumptions |
Metrics reverse that trade-off: Uber emits a compact record for every request and aggregates those records by dependency edge. The system gives up the complete visual story of each individual failure in exchange for faster convergence on the failure relationship itself. As Uber Engineering authors Deepanshu Mehndiratta, Alok Srivastava and Shivam Jindal put it, “We’d trade off visibility into individual failures for rapid convergence, but gain it by observing every failure in production.”
How the instrumentation joins callers to callees
Inbound middleware starts the request record
Uber’s Muttley on-host proxy already routes service-to-service traffic and emits success or failure call metrics. It does not, however, know the caller’s endpoint context. Uber adds that context in YARPC middleware. Inbound middleware creates a unique request ID, places it in the request context and starts an IngressLog.
Rank #2
Context propagation follows the request
The application passes the context to downstream clients. Because the request ID travels with the call, every outbound operation can be associated with the original inbound request instead of appearing as an unrelated service-level event.
Outbound middleware records each edge
Outbound middleware appends an EgressLog entry containing the callee service, the procedure invoked and the error state. A shared in-memory RequestTracker stores the caller, callee, procedure and error information while the request is in progress.
Completion emits caller-and-callee metrics
When the inbound request finishes, the middleware records whether the caller itself errored. It then emits caller/callee metrics for every outbound call. Aggregation can therefore answer a specific question: when this callee fails, does this caller endpoint fail in the same request?
How Uber decides whether a dependency is hard or soft
For each caller-to-callee edge, Uber computes:
Pc = (number of requests where the callee failed and the caller failed) / (number of requests where the callee failed)
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Pc is conditional on a callee failure. It is not the caller’s overall error rate and should not be interpreted as the probability that the callee will fail.
| Pc range | Classification | Operational interpretation |
|---|---|---|
Pc ≥ 0.8 |
Fail-close (hard) | The caller fails with the callee in at least 80% of observed callee failures |
Pc ≤ 0.2 |
Fail-open (soft) | The caller usually continues when the callee fails |
0.2 < Pc < 0.8 |
Unknown | The observed relationship is neither consistently hard nor consistently soft |
Uber describes 0.8 and 0.2 as heuristics, not universal reliability laws. A team applying the method would need to validate thresholds against its own error semantics, traffic volume and fallback behavior.
Why retry order changes the result
Retries can otherwise make a dependency look healthier or more confusing than it is. Uber disallows application-layer retries and places outbound retry middleware after dependency analysis in the request pipeline. Dependency analysis therefore sees the post-retry outcome: a call is evaluated after the configured retry behavior has completed, rather than counting an intermediate attempt as the final result.
This ordering is specific to Uber’s middleware design. Systems with retries embedded in clients, application code or proxies would need an equivalent decision about whether dependency classification should describe individual attempts or the final operation visible to the caller.
What Uber uses the dependency graph for
- Cataloging dependencies of critical APIs so owners can see which downstream edges are structurally important.
- Indexing most fail-close anomalies, making hard dependency failures searchable instead of buried in individual traces.
- Adding regression-safety guardrails for Failover Architecture 2.0, so changes intended to improve resilience can be checked against dependency behavior.
- Supporting operational work such as alerting, retry-budget decisions, call-path optimization and failover analysis.
Metrics versus distributed traces
| Dimension | Sampled distributed traces | Uber’s per-request dependency metrics |
|---|---|---|
| Evidence source | A sampled subset of requests | A metric emitted for every request, then aggregated |
| Granularity | Detailed individual request spans | Service-and-endpoint dependency edges |
| Decision method | Interpretation of trace examples | Explicit Pc thresholds |
| Time to detect rare failures | Potentially hours or days at low sampling rates | Faster statistical convergence because failures are all observed |
| Best operational use | Explaining a particular request’s path | Inventory, classification, alerting and regression gates |
The two methods answer different questions. A dependency metric can rapidly show that an endpoint is fail-close, while a trace remains better for reconstructing the exact sequence, payload and timing of one failed request.
Best Value
Limits and the role of static analysis
Uber’s description does not publish fleet-wide dependency counts, measured precision or recall, or a controlled benchmark against tracing. The reported thresholds are heuristics, and the observed graph depends on traffic: an edge that is rarely exercised may take longer to classify or remain unknown.
Runtime evidence also cannot reveal every possible relationship before traffic reaches it. RPCShield adds a static-analysis path for dependency relations, complementing the production metrics rather than replacing them. Static analysis can identify declared or reachable relationships, while runtime aggregation shows how those relationships behave during real failures.
Can another service mesh use the same pattern?
- Capture a request identifier at the inbound boundary and propagate it through every downstream client.
- Record caller endpoint, callee endpoint and error state at each outbound edge.
- At request completion, attach the caller’s final success or failure to every recorded callee outcome.
- Aggregate the resulting events by edge and calculate a conditional failure metric such as
Pc. - Choose and validate hard, soft and unknown thresholds using representative traffic, then account explicitly for retries and fallbacks.
- Pair the runtime graph with static dependency analysis so unexercised paths are not mistaken for nonexistent dependencies.
The central design choice is not a particular proxy or RPC framework. It is joining every downstream result to the final outcome of the request that initiated it, then making the classification rule explicit enough to automate inventory, alerts and resilience safeguards.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
