The reports failed because the PostgreSQL container had a 64 MB /dev/shm, while the month-end queries needed shared-memory segments hundreds of megabytes in size. The database volume was only 40% full, so freeing disk space could not solve it. The former virtual machine had an 8 GB tmpfs at the same path; moving PostgreSQL into Kubernetes silently reduced that capacity.
Daily queries used only a few megabytes and passed. Month-end reports joined three years of ledger lines with parallel hash joins, causing the container to hit its shared-memory ceiling about one time in three. Disabling parallel query restored correct results at lower speed. The lasting fix was a 2 GB memory-backed Kubernetes emptyDir mounted at /dev/shm, together with an equivalent increase in the pod’s memory limit.
As an Amazon Associate I earn from qualifying purchases.
What actually failed
The incident involved a PostgreSQL 15 reporting database moved into a Kubernetes StatefulSet in June. On the first working day of July, month-end finance reports began returning:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minutecould not resize shared memory segment, No space left on deviceIn this message, “No space left on device” does not necessarily mean the database’s persistent disk is full. PostgreSQL was unable to enlarge a shared-memory segment in /dev/shm, the temporary shared-memory filesystem exposed inside the container.
#1 Best Overall
| Resource | Observed value | Why it mattered |
|---|---|---|
Container /dev/shm |
64 MB | Too small for the peak parallel-reporting workload |
Former virtual machine /dev/shm |
8 GB tmpfs |
Made the same workload appear to have ample headroom |
| Persistent data volume | 40% full when failures occurred | Disk capacity was not the exhausted resource |
| Failure frequency | About one in three month-end reports | Whether a query failed depended on its plan and concurrent allocations |
| Peak query scope | Three years of ledger lines | Large joins and parallel workers required much more shared memory than daily queries |
These figures describe this 2026 incident report, not a reliability rate for PostgreSQL or Kubernetes generally.
Why Kubernetes changed the behavior
A container has its own view of several operating-system resources. A directory can have the same name as it did on a virtual machine but a very different size. Container runtimes commonly provide a small shared-memory mount unless the workload specification replaces it. In this case, the runtime supplied only 64 MB, compared with 8 GB on the old VM.
PostgreSQL uses shared memory for coordination between server processes. Parallel query plans add more demand because workers must exchange data and coordinate operations. A parallel hash join can create shared structures that grow as the join processes large inputs. When PostgreSQL asks the kernel to resize one of those segments and the /dev/shm mount has no free capacity, the kernel reports an out-of-space condition even though the persistent database volume has plenty of room.
The exact amount required varies with PostgreSQL version, query plan, row widths, worker count, join cardinality and concurrency. The important distinction is that /dev/shm is a separate capacity pool; checking only the data volume or the node’s free disk space does not measure it.
Why only month-end reports failed
Normal daily traffic stayed below the ceiling
Routine reports consumed a few megabytes of shared memory. They never approached the 64 MB limit, so migration testing based on ordinary daily activity appeared successful.
Month-end plans were materially larger
The finance reports scanned and joined three years of ledger history. PostgreSQL selected parallel hash joins for these heavier statements. Each worker and the leader needed access to shared structures, and several reports could run at the same time. A query that is safe on a quiet day can therefore exceed a small /dev/shm mount during a reporting peak.
Intermittent failure is expected at a hard capacity boundary
“About one in three” failures does not mean the database randomly lost data. It indicates that some executions fit and others did not, depending on the selected plan, active workers and simultaneous reports. Every failed statement returned an error rather than silently producing an incorrect result.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How to confirm that /dev/shm is the bottleneck
- Check the mount from inside the PostgreSQL container. Run
kubectl exec POD_NAME -- df -h /dev/shm. The size and available columns show the capacity the database process can actually use; node-level values are not a substitute. - Confirm the mount type and options. Run
kubectl exec POD_NAME -- mount | grep ' /dev/shm '. This helps distinguish the container’s shared-memory filesystem from another temporary directory. - Inspect the pod’s memory limit. Use
kubectl get pod POD_NAME -o yamland reviewresources.limits.memory. A memory-backed replacement for/dev/shmconsumes pod memory, so the limit must leave room for it and for PostgreSQL itself. - Match failures to the workload. Compare database logs and report schedules with month-end executions. Look for the shared-memory error during parallel reporting, not merely for high persistent-disk usage.
- Review the query plan. Use
EXPLAINorEXPLAIN (ANALYZE, BUFFERS)in a safe test environment to determine whether the statement uses parallel workers and hash joins. Do not runANALYZEon production queries that are too expensive for your maintenance window.
If /dev/shm reports roughly 64 MB and the error appears only on large parallel statements, the symptoms align with this incident. If the mount has ample space, investigate other limits such as the pod’s total memory, process limits or file-descriptor limits instead of changing the data volume.
Rank #3
Immediate service restoration: turn off parallel query
The short-term workaround is to prevent the affected session from using parallel workers:
SET LOCAL max_parallel_workers_per_gather = 0;For a reporting role that must remain serial until capacity is fixed, an administrator can apply a role setting:
ALTER ROLE reporting_user SET max_parallel_workers_per_gather = 0;Use the smallest scope that restores service. A session-level setting limits the blast radius; a role-level setting affects every connection for that reporting user. Verify the result with SHOW max_parallel_workers_per_gather; and confirm that the report completes.
Serial execution avoids the parallel shared-memory demand, but it generally takes longer and may increase CPU time or the reporting window. It is a service-restoration measure, not evidence that the container is correctly sized. Remove the override only after the enlarged mount has been deployed and peak queries have been tested.
Durable fix: mount a larger memory-backed emptyDir
The incident was resolved with a 2 GB memory-backed emptyDir mounted over /dev/shm. The relevant StatefulSet template pattern is:
spec:
template:
spec:
containers:
- name: postgres
volumeMounts:
- name: postgres-shm
mountPath: /dev/shm
resources:
limits:
memory: 10Gi # illustrative: set this to the previous limit plus 2Gi
volumes:
- name: postgres-shm
emptyDir:
medium: Memory
sizeLimit: 2GiThe 10Gi value is illustrative. Replace it with the pod’s existing memory limit plus the 2 GB allocated to the new shared-memory filesystem, while also accounting for PostgreSQL’s normal working set. Do not copy that number unless it matches your workload.
Why both settings are required
medium: Memory stores the emptyDir contents in RAM. Those pages count against the container’s memory limit. Increasing only sizeLimit can turn a PostgreSQL shared-memory error into an OOM kill: the kernel may terminate the container when the cgroup limit is reached before the filesystem’s 2 GB allowance is useful.
Free tools Windows power users keep installed
One-click scans. No signup required.
Conversely, increasing the pod memory limit without mounting a larger /dev/shm leaves the original 64 MB filesystem in place. The two changes solve different constraints and must be rolled out together.
Deploy and verify the change
- Update the StatefulSet template with the volume and mount. Ensure the volume name is unique and that the PostgreSQL container, rather than a sidecar, receives the mount.
- Apply the manifest and wait for the StatefulSet rollout to complete. A pod restart is normally required because the volume is part of the pod specification.
- Run
kubectl exec POD_NAME -- df -h /dev/shmin the new pod and verify that the reported capacity is close to 2 GB. - Check
kubectl describe pod POD_NAMEfor memory limits and recent OOM events. A successful mount with repeated OOM kills means the cgroup limit or PostgreSQL working-set budget is still too small. - Re-enable parallel reporting only after the month-end replay described below completes successfully.
Choosing between the two practical responses
| Response | Report correctness and speed | Shared-memory capacity | Pod memory headroom | Operational complexity | Peak month-end concurrency reproduced? |
|---|---|---|---|---|---|
| Disable parallel query | Correct results, generally slower execution | Leaves the 64 MB mount unchanged; avoids parallel demand | No additional allocation, but serial work can consume more time and CPU | Low; apply a session or role setting | No; it avoids the workload rather than validating capacity |
2 GB memory-backed emptyDir plus a 2 GB memory-limit increase |
Preserves parallel plans and their expected performance, subject to testing | Raises the mounted capacity to 2 GB in this incident | Requires an equivalent increase and budget review because pages count toward the limit | Moderate; edit, roll out and monitor the StatefulSet | Yes, when the actual month-end queries and concurrency are replayed |
The report describes no benchmark timings, so the speed difference should be measured in your environment rather than assumed. The serial setting is useful while a deployment is being prepared; the enlarged mount addresses the underlying capacity mismatch.
Size shared memory from the peak workload, not the average day
A 2 GB mount was sufficient for this incident’s revised deployment, but it is not a universal PostgreSQL recommendation. Size the mount from observed peak demand:
- Capture the largest month-end statements, including their production parameters and query plans.
- Replay the same statements concurrently, not one at a time. Shared-memory pressure is affected by the number of reports active together.
- Record
/dev/shmusage, pod memory usage, worker counts and whether any process is OOM-killed. - Leave headroom for PostgreSQL background processes, connection overhead, caches and ordinary application activity. A mount that exactly matches one successful run may fail when two runs overlap.
- Recheck sizing after schema, statistics, PostgreSQL-version or reporting changes. A plan that adds workers or widens rows can increase the requirement without any change to disk size.
The sizeLimit is a ceiling, not a reservation of free RAM. Kubernetes and the container runtime still need enough node and cgroup capacity for the pages that are actually used.
Test the migration the way finance uses it
The original migration test replayed one week of normal daily queries, so it never exercised the resource boundary that failed in production. A meaningful acceptance test must replay the actual previous month-end queries, with realistic parameters and concurrency.
- Export the SQL statements or application workload from the previous month-end period, removing sensitive values where necessary without changing row counts or join behavior.
- Run the replay against a production-like PostgreSQL 15 image, schema, statistics and data volume.
- Start the same number of report sessions that finance uses during the closing window.
- Verify that
/dev/shmremains below its limit, the pod stays below its memory limit, and no OOM or shared-memory errors appear. - Measure completion time and compare it with the reporting deadline. Correctness alone does not prove that the enlarged configuration is operationally adequate.
- Repeat after changing worker settings, indexes, PostgreSQL versions or Kubernetes resource limits.
Migration checklist for PostgreSQL containers
Include these host-resource checks in every database migration review:
/dev/shm: record its size inside the container and compare it with the largest parallel-query demand.- Memory limits: account for memory-backed volumes in the cgroup limit, not only for PostgreSQL’s configured buffers.
- Open-file limits: verify that the container’s file-descriptor ceiling supports the connection and storage workload.
- Huge pages: determine whether the database configuration expects them and whether the Kubernetes deployment exposes them correctly.
- Process limits: check PID limits and other process quotas that may be lower than on the source VM.
- Peak replay: test month-end or other quarterly workloads, including concurrency, rather than sampling only ordinary days.
- Monitoring: alert on shared-memory utilization, pod memory pressure, OOM kills and the PostgreSQL error text so that a capacity trend is visible before a close deadline.
What not to conclude from this incident
A full database volume was not the cause, and adding persistent disk would not enlarge /dev/shm. Raising only PostgreSQL parallel-worker settings would increase demand without adding capacity. Raising only the emptyDir limit could instead expose the pod to OOM kills. Finally, the fact that one report failed about one-third of the time does not establish a general failure probability for other clusters; it is a workload-specific observation from this migration.
“A container gives you a filesystem that looks like a machine’s, and some of its directories are much smaller than the machine’s were.” — Sergey Shinder, 2026
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.Quick Recap
SaleBestseller No. 3SaleBestseller No. 4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
