Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsA single interrupt that never reached its target left one gVisor subprocess waiting indefinitely. That wait sat on the sandbox teardown path, so a kill request for one Kubernetes pod never finished, and every later request queued behind the same stuck lock. The upstream fix resends the interrupt, dumps diagnostics when the subprocess stays silent, and kills only the stuck subprocess rather than the whole sandbox.
What the alert reported, and what was actually wrong
Nahum Litvin, technical lead at Wix, describes a production sandbox that runs untrusted backend JavaScript under gVisor on Amazon EKS. The alert reported 559 pods stuck in Terminating. Litvin’s account says that number counted failed kill events, not distinct pods. At the time, the real problem was one wedged pod. The kubelet’s kill request received DeadlineExceeded every two minutes, and a person intervened about 90 minutes after the alert.
As an Amazon Associate I earn from qualifying purchases.
The same account describes an earlier episode, about a week before, in which eight pods on four nodes stayed stuck for days. The two episodes were not the same failure at the same scale, and the figures below should be read with that in mind.
| Figure | What it measures | Scope and attribution |
|---|---|---|
| 559 | Failed kill events counted by the alert | Wix incident alert, as reported by Litvin, 2026. Not 559 distinct pods. |
| 1 | Pods actually wedged during the main incident | Wix production incident, as reported by Litvin, 2026. |
| 8 pods on 4 nodes | Pods stuck for days in an episode about a week earlier | Wix, as reported by Litvin, 2026. |
| About 45,000 waiting threads over five days | Accumulation in a related incident involving another team | Reported by Litvin, 2026. Attributed to repeated cAdvisor Stats requests blocked behind a hung kill lock. |
| Roughly one blocked request every 10 seconds | Rate at which the blocked requests accumulated | Matches the cAdvisor scrape interval, per Litvin, 2026. |
| About 600 MB | Shim memory in that same related incident | Reported by Litvin, 2026, as the approximate size reached after repeated blocked requests. |
| About 3 load-average points per hour, with CPU near 30% | Behavior on one affected Wix node | A specific incident observation, not a general gVisor characteristic. |
How systrap waits for a stub
gVisor’s systrap mode uses a component called the sentry to coordinate application threads. Those threads run inside stub processes. When the sentry needs a stub to stop, it sends that stub an interrupt signal and then waits for it to park.
#1 Best Overall
In the failure Litvin describes, the sentry sent that interrupt once. When the stub missed it, the sentry kept waiting and never retried. A 30-second deadline existed, but when it passed, the only action was a warning in the log. Nothing re-sent the signal, and nothing ended the wait.
Litvin’s summary of this design is blunt: “A timeout that only logs a warning is not a timeout. It is a diary.”
Why one wait blocked the whole pod’s termination
Killing a sandbox proceeds in order. The sandbox first freezes its work, then waits for every worker thread to park. A single stuck worker therefore holds up the entire sequence. In this chain, that stall propagated upward:
Free tools Windows power users keep installed
One-click scans. No signup required.
- the sentry waited on the stub that missed its interrupt;
runsc kill, the gVisor call, did not return;- the gVisor shim call that wraps it did not return;
- the kubelet’s kill request timed out, and the pod stayed in Terminating.
Meanwhile, cAdvisor’s Stats requests for the same sandbox needed a lock that the hung kill was holding. The account reports that caller timeouts did not cancel goroutines already waiting for that mutex. The timed-out requests kept existing as waiting goroutines, so the problem grew with every poll.
Where 45,000 waiting threads came from
The 45,000 figure does not mean 45,000 sandboxes failed. Litvin attributes it to one stuck kill and a steady stream of Stats requests queued behind it. The arithmetic is consistent with that account: five days at one blocked request every 10 seconds comes to about 43,000, close to the reported number. That estimate is ours, not a count the author published, but it shows why a single polled resource can produce a large pile of waiters.
The same accumulation pushed shim memory to roughly 600 MB in the related incident. Litvin also makes a broader point about how such incidents get written up: “The half-report you are embarrassed to file is someone else’s missing half.” In this case, the sandbox-side failure and the monitoring-side pile-up were two halves of one problem.
Rank #3
Why timeouts and retries did not recover it
Every layer between the kubelet and the sandbox had some form of timeout, and none of them recovered the stuck subprocess. The table compares each layer on two questions: whether its wait is bounded, and whether it can recover without the component that is stuck.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →| Layer | Is the wait bounded? | Can it recover without the stuck component? | Status in the account |
|---|---|---|---|
| Kubelet kill request | Yes, per request. Each call fails with DeadlineExceeded after its timeout. |
No. A retry sends the same request into the same stall. | Unchanged. The kubelet kept retrying every two minutes. |
| containerd | Calls time out, but the account describes no escalation after repeated kill timeouts. | No. | Escalation after repeated kill RPC timeouts is listed as remaining work. |
| gVisor shim (Kill, Stats, Status) | Not bounded in the reported code. Caller timeouts did not cancel waiting goroutines. | No. | Work to bound these waits is described as in progress at publication. |
| Sentry and stub (systrap) | The 30-second deadline existed, but it only logged. | No, before the fix. The signal was never re-sent. | Fixed upstream, as described below. |
The pattern is that each timeout moved the failure one layer up. Only the layer that owned the missed signal could break the chain, and it did not do so.
The upstream fix, in three stages
The merged change, as Litvin describes it, works in three stages:
- Resend the interrupt at each five-second checkup wake-up, instead of sending it once.
- If the stub is still unresponsive past the 30-second deadline, dump internal stack traces to the log so the stall can be diagnosed.
- Kill only the stuck subprocess, using the existing path that handles a stub that has already died naturally. This frees the blocked task and lets teardown proceed, while healthy subprocesses are left running.
The scope is the important design choice. The fix does not try to make the sandbox kill itself faster or wider. It retries the signal first, collects evidence second, and terminates the smallest unit that is actually stuck. Litvin credits gVisor maintainer Konstantin Bogomolov with pushing for that narrower termination action. Bogomolov also identified a race in which a context that had just recovered could still be killed, which the change accounts for.
Litvin states that the fix shipped in gVisor release-20260831.0. That label comes from the author’s account. Check the upstream gVisor release notes for the exact contents of that release, and the issue and pull-request numbers the account cites, before relying on the version string.
What remains open
The fix addresses the sentry-side wait. It does not, by the account’s own description, close the whole deletion chain. Two items were still outstanding:
- a gVisor change to bound the shim’s waits on Kill, Stats, and Status;
- containerd escalation after repeated kill RPC timeouts.
The shim work was described as in progress when the account was published. Its status after that date is not confirmed by the account, so do not assume every path from kubelet to sandbox now has a bounded wait. Litvin’s operational advice is worth taking literally: “If the answer to the second bottoms out at ‘a human with SSH’, write that down, because that is your actual design.” Until the containerd escalation exists, a stuck kill in your fleet may still end in manual intervention.
Sources and limits
This is one firsthand account by Nahum Litvin. It was published on catchkill9.dev on 2026-09-29 and reposted on DEV Community on 2026-10-01. The incident figures, the description of the fix, and the release label are the author’s statements. They are not independent measurements, and the account does not give a complete reproduction. Treat the 559, 45,000, and 600 MB figures as reported observations from specific incidents, not as properties of gVisor in general.
- Verify the release label, and the upstream issue and pull-request details, against gVisor’s own repository and release notes.
- Check the status of the shim and containerd work independently, since the account predates its current state.
The lesson the account supports is narrow and practical. A timeout must have a recovery action that can work when the stuck component is the one that failed to respond. Here, the working action was re-sending the signal and killing only the stuck subprocess.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
