Free tools Windows power users keep installed
One-click scans. No signup required.
A Kubernetes cluster can slow down while every Pod reports healthy, because finished work still costs the control plane something to store and list. In an incident write-up published on DEV Community by Sergey Shinder, the cause was that directly created Jobs and their Pods were never removed after they finished. The author reports about 340,000 accumulated Jobs, a similar number of Pods, and an etcd database of 6.4 GB. Slow list calls and scheduler resyncs followed, and a rollout tool began timing out while listing Pods. The fix combined batched cleanup with a rule that stops the accumulation from returning.
The figures below are the author’s own and have not been independently verified. The publication year of the write-up could not be confirmed, so treat the numbers as one team’s case rather than a benchmark.
As an Amazon Associate I earn from qualifying purchases.
Why healthy Pods do not mean a healthy control plane
Health checks answer a narrow question: is this Pod running and passing its probes? They say nothing about the cost of the objects around it. Every finished Job is an API object, and each of its Pods is another one. Each object is written to etcd, returned by list requests, and watched by controllers and the scheduler. When the count grows into the hundreds of thousands, the work does not show up as a failing workload. It shows up as slower responses from the API server and as a steadily larger database.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →The author links the sequence this way: the import controller created roughly 900 Jobs per day starting in early 2024, the finished ones were never deleted, and the object count and etcd size grew until list calls and scheduler resyncs slowed. The rollout tool then timed out while listing Pods before it began watching them. That chain is the author’s causal account. The write-up does not include independent telemetry from the cluster, so the reader should judge it as a plausible diagnosis that fits the symptoms described.
#1 Best Overall
- DeskPi Super6C is the Pi cluster board a standard size mini-ITX board to be put in a case with up to 6 RPI CM4 Compute Modules. For Case Kit, please refer to ASIN B0BGWZH6MP
- 6 RPI CM4 supported 1 Gbps RJ45 x2
- Onboard ON/OFF and Reset button
- 12V FAN Header x3
- DC 19v~24V or ATX 12V
Directly created Jobs versus CronJob-managed Jobs
The account separates two cases. Jobs created directly through the API are entirely the responsibility of whatever creates them. A CronJob, by contrast, creates Jobs on a schedule and manages how many finished ones it keeps. Its successfulJobsHistoryLimit and failedJobsHistoryLimit fields control that retention. A team that moves from ad hoc Job creation to a CronJob therefore gets a retention setting it did not have before, while a team that keeps creating Jobs directly must supply its own cleanup.
The incident came from the second case: the cluster had not set ttlSecondsAfterFinished on the directly created Jobs, so nothing removed them.
The cleanup sequence the author used
The author describes a two-part recovery. The order matters because each step changes the load on the next one.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →-
Delete old finished Jobs in batches. The author removed them in batches of 500 with pauses between batches, spread over two days. Deleting hundreds of thousands of objects at once would itself add pressure to the API server, which is the reason for batching.
-
Compact and defragment etcd, one member at a time. Deleting objects does not shrink the database file on its own. Compaction discards old revisions, and defragmentation returns the freed space to the filesystem. Processing one member at a time keeps the cluster available while each member is worked on.
-
Give new Jobs a time-to-live. The author set a one-hour TTL on new Jobs. The one-hour value is the author’s choice, not an official recommendation.
Rank #3
Cluster Case for Raspberry Pi5/Pi4/Pi3 and Other Single Board Computers | Cloudlet Case - Black Lime- For Raspberry Pi5, 4B, 3B+, 3B, 2B, and B+ (not included). Other single board computers must adhere to the RPi mounting hole pattern and port configuration.
- Eight bays hold Raspberry Pi and MOST single-board computers or 2.5" hard drives (RPi 5 & 4B compatible).Room for most 8-port switches (maximum size of 4 1/2″ x 8 3/4″ x 1 5/8″)
- Multiple Cloudlet Cases can be bolted together, either vertically or horizontally for modular clusters.
- Plates click securely into place for fast removal without bolts. Made with double thick acrylic for high durability
- Designed and Crafted in Tacoma, WA, USA.
-
Enforce the rule with admission control. An admission policy rejects Jobs that do not set a TTL, so the rule cannot be skipped by a future manifest.
Recommended Free Tools
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Alert on object counts. The author added an alert for resource-type counts above 5,000 in a namespace. That threshold reflects this cluster’s history and should be calibrated to your own baseline.
Setting a TTL on a Job
Kubernetes provides the TTL-after-finished mechanism for Jobs. Setting ttlSecondsAfterFinished in the Job spec tells the TTL-after-finished controller to delete the Job, and through garbage collection its Pods, once the specified number of seconds has passed after the Job completes or fails. Availability of this field depends on your Kubernetes version, so check the Job API reference for the release your cluster runs before relying on it.
apiVersion: batch/v1
kind: Job
metadata:
name: import-batch-example
spec:
ttlSecondsAfterFinished: 3600
template:
spec:
restartPolicy: Never
containers:
- name: importer
image: example/importer:1.0The value 3600 corresponds to the author’s one-hour choice. Any other value is a decision about how long finished work remains visible in the cluster.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choosing a retention interval
A TTL trades cluster state for history. Once the Job is deleted, its status and the logs that were only reachable through its Pods go with it. The right interval depends on how long your team needs to investigate a failed run and on whether logs and events are kept somewhere else.
| Approach | What remains in the cluster | Main risk | Fits when |
|---|---|---|---|
| No TTL (the default for directly created Jobs) | Every finished Job and Pod until someone deletes them | Object count and etcd size grow without limit; list calls slow | Rarely; only when an external job removes finished work on a schedule |
| Short TTL, such as the author’s one hour | Finished Jobs for about an hour | Investigations that start later find no Job or Pod to inspect | High-volume workloads where logs are already shipped off-cluster |
| Longer TTL combined with log export | Finished Jobs for the chosen window, with logs stored elsewhere | More objects exist during the window; export must work reliably | Teams that need in-cluster history for troubleshooting or review |
Whatever interval you choose, decide before deletion what evidence must survive it. Export logs from the Pods, and record Job completion status where your audit process requires it. Events also expire on their own schedule, governed by the API server’s event TTL, so do not assume they outlive the Job.
Checks before you change anything
- Count finished Jobs per namespace, for example with
kubectl get jobs -A, and compare the total with the Job creation rate of your controllers. - Confirm which Jobs are created directly and which come from CronJobs, since the retention controls differ.
- Check the etcd database size and whether compaction and defragmentation run on a schedule in your environment.
- Verify that your Kubernetes version supports the TTL field and that your admission tooling can validate it.
- Confirm that logs and any required audit records are exported before the TTL deletes the Job.
7
What to take from the account
The incident is a case study of a single cluster, reported by its author. Its specific values, including the batch size, the one-hour TTL, the 5,000-object threshold, and the two-day remediation window, are the author’s choices for that environment and should be treated as starting points for your own measurements. What transfers is the structure: anything that creates objects at a steady rate needs a removal rule, and that rule needs to be part of the workload definition rather than a memory of the team.
The Bottom Line
Finished Jobs are cluster state, and state accumulates unless something removes it. Set a retention rule on every Job when it is created, enforce that rule, and measure object counts so growth is visible before it slows the API server. As Sergey Shinder puts it in the account: “Anything in your system that creates objects at a rate needs a rule for removing them, written on the same day, because the platform will keep them faithfully until it cannot.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
