A dependable Go job queue needs more than a goroutine loop: PostgreSQL must persist jobs and coordinate claims, workers must have bounded concurrency, and the design must define what happens when a handler fails or a process disappears. This guide lays out those decisions and an illustrative implementation pattern. A queue design alone does not prove production readiness; that requires workload-specific load and failure testing.
Decide what the queue guarantees
Start by defining what it means for a job to be accepted, completed, retried, or abandoned. A process-local channel can hand work to goroutines, but jobs waiting only in memory can disappear when the process exits. Persisting jobs in PostgreSQL lets work survive process restarts, but durability depends on when the enqueue transaction commits and how workers recover interrupted jobs.
As an Amazon Associate I earn from qualifying purchases.
- Acceptance: Is a job accepted only after its database transaction commits?
- Completion: Does success mean the handler returned without error, or must a downstream action also be confirmed?
- Failure: Which errors are retried, how many attempts are allowed, and when does a job stop being eligible?
- Recovery: How does a worker make abandoned work eligible again after a crash?
- Duplicates: Can a handler safely run more than once, and how are duplicate side effects prevented?
These are contract decisions, not details to leave implicit. The pgq project, for example, documents scheduled execution, retry and backoff behavior, completion retention, and enqueueing within an application transaction. Those are useful design examples, not evidence that every queue implements them.
Keep the architecture boundaries clear
Clean architecture helps keep database mechanics separate from the application behavior that gives a job meaning. One practical division is:
#1 Best Overall
- Job contract: A job type and validated payload describe the work, without embedding SQL details in the handler.
- Application orchestration: The application enqueues jobs and invokes handlers through interfaces.
- Storage port: An interface exposes operations such as enqueue, claim, acknowledge success, and record failure.
- PostgreSQL adapter: This layer owns SQL, transactions, row locking, and persistence mapping.
- Worker runtime: This component polls or waits for work, dispatches handlers, observes cancellation, and coordinates shutdown.
For example, a handler should receive a job and a context, perform the job-specific work, and return an error. It should not know that another worker used SKIP LOCKED to claim that row. Conversely, the PostgreSQL adapter should not contain business rules for sending an email or charging an account.
Represent job state and eligibility explicitly
A table needs enough information to answer two separate questions: which jobs are eligible to run, and what happened to each job. The following is an illustrative starting point, not a schema verified for a particular implementation:
CREATE TABLE jobs (
id bigserial PRIMARY KEY,
job_type text NOT NULL,
payload jsonb NOT NULL,
status text NOT NULL,
priority integer NOT NULL DEFAULT 0,
run_at timestamptz NOT NULL DEFAULT now(),
attempts integer NOT NULL DEFAULT 0,
lease_until timestamptz,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz
);
CREATE INDEX jobs_pending_claim_idx
ON jobs (priority DESC, run_at, id)
WHERE status = 'pending';The partial index is an example aligned with a query that selects pending jobs in priority and scheduled-time order. If claiming also scans expired running jobs, that recovery path may need its own index. Choose indexes from the actual predicates and ordering, then verify them against representative data and query plans. The state values and columns should match the policy you actually implement; for example, a system may represent exhausted work as dead rather than retaining it as a failed row.
Recommended Free Tools
Claim jobs atomically with PostgreSQL
Concurrent workers must not both claim the same available row. A common pattern is to select one eligible row under a row lock, skip rows already locked by another claimant, update the selected row, and return it in one transaction. PostgreSQL documents FOR UPDATE ... SKIP LOCKED for queue-like consumers, while warning that skipped locked rows make the result an inconsistent view unsuitable for general-purpose reads. That is an intentional tradeoff for distributing work, not a general query optimization (PostgreSQL 18 SELECT documentation).
WITH candidate AS (
SELECT id
FROM jobs
WHERE (status = 'pending' AND run_at <= now())
OR (status = 'running' AND lease_until <= now())
ORDER BY priority DESC, run_at, id
LIMIT 1
FOR UPDATE SKIP LOCKED
)
UPDATE jobs AS j
SET status = 'running',
lease_until = now() + interval '60 seconds',
attempts = j.attempts + 1
FROM candidate
WHERE j.id = candidate.id
RETURNING j.id, j.job_type, j.payload, j.attempts, j.lease_until;This example makes the selection order explicit: higher priority first, then earlier schedule time, then stable ID order. A different ordering means a different scheduling policy. Strict priority can delay low-priority jobs if high-priority work keeps arriving, so fairness requirements may call for a different policy.
Run the claim statement inside a transaction and commit promptly after the claim is persisted; do not hold the row lock while the handler does network or CPU work. The lease is a recovery mechanism: if a process dies after claiming, another worker can reclaim the row after expiry. But if a handler continues longer than its lease, a second worker may also start it. Choose a lease policy that fits handler duration, and consider lease renewal for long-running work. Neither a lock nor a lease can guarantee exactly-once external side effects.
Bound worker and database concurrency together
Goroutines make it straightforward to run independent handlers concurrently, but worker count is not a throughput guarantee. The Go FAQ notes that concurrency enables parallelism only when the work is intrinsically parallel, and synchronization or communication overhead can make additional concurrency slower (Go FAQ, concurrency section).
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Go’s sql.DB is safe for concurrent use and manages a connection pool. Its maximum-open-connection setting can make operations wait once connections are occupied; the Go database guide warns that this waiting can contribute to deadlocks when application code holds resources while waiting for another connection (Go: Managing connections).
- Set a bounded worker count rather than starting one goroutine per available job.
- Size the database pool with all database users in mind, not just queue workers; HTTP request handlers and other application operations may share it.
- Track pool statistics and wait behavior, and test under representative concurrent load.
- Review transaction and resource lifetimes so a worker does not hold a connection or lock while waiting on work that itself needs that resource.
Keep claim transactions short and avoid doing handler work while holding them. More workers than useful database or downstream capacity can increase contention and queueing rather than improve service time.
Define success, retry, and crash recovery
A worker should acknowledge success only after the handler has completed the work the queue contract defines as successful. If it crashes after a downstream side effect but before recording completion, the lease can expire and the job can run again. Design handlers to be idempotent where possible—for example, by using a stable job identifier as a downstream idempotency key or by recording the effect in a transaction shared with the job state when both live in the same database.
On handler error, record enough information to understand the failure, decide whether it is retryable, and schedule the next eligible time rather than retrying continuously. A typical backoff policy increases the delay after successive attempts and caps it; the actual formula, maximum attempts, and treatment of permanent errors are product decisions. The pgq documentation describes retry/backoff and scheduled jobs as examples. It also notes that whole-queue backoff can be appropriate when a downstream service is broadly failing.
Make the transitions explicit: pending to running on claim; running to completed on success; running back to pending with a later run_at after a retryable error; and running to a terminal state after retry exhaustion or a permanent failure. Expired running leases need a defined reclaim path. If dead-letter or failed jobs are retained, provide a way to inspect and, where appropriate, redrive them without silently resetting useful error history.
Rank #4
Enqueue atomically with the application change when needed
Suppose an HTTP request updates an order and must also schedule a follow-up job. If the application commits the order change and then separately inserts the job, a failure between those actions can leave the order updated but the job missing. When both writes use the same PostgreSQL database, inserting the job through the application’s transaction lets the data change and enqueue succeed or roll back together. The pgq project documents this transaction-coupled enqueue pattern as an example (pgq README).
This pattern couples reliable job creation to the database transaction; it does not make external effects transactional. A worker still needs duplicate-safe handling if it calls another service, and an application that uses separate databases or a broker needs a different coordination design.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Make operations and shutdown observable
A queue that stores work but cannot show whether it is progressing is difficult to operate. Define the signals that tell an operator whether workers are keeping up, failing repeatedly, or stuck behind a dependency.
- Backlog: Count eligible jobs and track the age of the oldest pending job, not just total rows.
- Execution: Track active workers, handler duration, successes, failures, and lease expirations.
- Retries: Expose attempt counts, next scheduled times, terminal failures, and representative error details without leaking sensitive payload data.
- Database health: Observe pool usage and wait time alongside query latency and lock contention.
- Retention: Decide how long completed and failed records remain, and how cleanup avoids interfering with active workers or investigations.
For graceful shutdown, stop claiming new jobs when cancellation begins, let active handlers finish within a bounded shutdown window, and ensure unfinished work remains recoverable through its persisted state and lease policy. If a handler cannot finish before that window closes, the system needs a deliberate policy for cancellation and later retry; returning from the process does not itself make an in-flight external side effect safe to repeat.
Best Value
Know when PostgreSQL is the right queue
A PostgreSQL-backed queue can be attractive when the application already operates PostgreSQL and benefits from committing business data and job creation together. That convenience comes with database work: claim queries, indexes, connection-pool sizing, cleanup, and lock behavior must be designed and monitored.
The goforj/queue documentation describes SQL-backed queues as operationally convenient while noting a throughput tradeoff at higher concurrency and recommending broker-backed drivers for higher-throughput workloads (goforj/queue README). Treat that as the project’s own characterization, not a universal benchmark or a capacity threshold. The right choice depends on measured workload, existing infrastructure, transaction needs, and the administrative and recovery features the system requires.
What makes the design production-ready?
Architecture is a plan; readiness depends on evidence from the implementation under expected and failure conditions. Before relying on the queue, validate the behaviors its contract promises.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Test multiple concurrent claimers and verify that a live claim is not handed to two workers at once.
- Kill a worker after claim, during handler execution, and after a side effect but before acknowledgement; confirm recovery and duplicate handling behave as designed.
- Exercise retry exhaustion, scheduled jobs, invalid payloads, and a failing downstream dependency.
- Load test realistic job sizes, handler durations, worker counts, database pool limits, and competing application traffic; record methodology and results rather than assuming throughput.
- Verify indexes and query behavior as the queue grows, and test cleanup and retention policies against active work.
- Test shutdown and restart behavior, including jobs whose handlers exceed the configured lease or shutdown window.
Without implementation-specific tests and measurements, no throughput figure or production-readiness claim is established. A PostgreSQL queue is a set of explicit guarantees and recovery mechanisms that must be verified against the workload it will serve.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
