Use a buffered channel for pending jobs and a fixed pool of worker goroutines to process them. The buffer limits how many jobs can wait; the worker count limits how many can run at once. This design uses only Go’s standard library, but it is an in-memory queue: pending and active work is lost if the process stops.
What this queue guarantees—and what it does not
A channel-backed queue is a good fit when one Go process needs to defer work, apply backpressure, and cap concurrency. It does not provide persistence, coordinate workers across multiple processes, or recover jobs interrupted by a crash. If jobs must survive a restart or be shared by several application instances, use a durable store or broker with explicit recovery logic. Redis’s official Go queue example illustrates the extra pending and processing states, completion and failure handling, worker recovery, and retry decisions that such a design requires: Redis job queue example for Go.
As an Amazon Associate I earn from qualifying purchases.
Choose a job type and handler
Keep the job representation typed and small. This example passes a value containing an ID and payload; replace the fields with the data your work needs. The handler returns an error so the worker can make failures visible. The queue itself does not decide whether a failed job should be retried, recorded, or discarded.
type Job struct {
ID string
Payload []byte
}
type Handler func(context.Context, Job) errorIf a job contains pointers, maps, slices, or other mutable data, establish who owns them after enqueueing. A channel transfers the value, not exclusive ownership of the referenced memory. Avoid changing shared data concurrently unless it is protected appropriately.
#1 Best Overall
Implement a bounded queue with fixed workers
The following implementation uses a positive-capacity buffered channel for pending jobs and starts exactly the configured number of workers. A send to a full buffer waits until a worker receives a job, providing backpressure rather than silently creating more processing goroutines.
package queue
import (
"context"
"errors"
"sync"
)
type Job struct {
ID string
Payload []byte
}
type Handler func(context.Context, Job) error
type Queue struct {
jobs chan Job
handler Handler
ctx context.Context
cancel context.CancelFunc
workers sync.WaitGroup
}
func New(parent context.Context, capacity, workerCount int, handler Handler) (*Queue, error) {
if capacity <= 0 {
return nil, errors.New("capacity must be positive")
}
if workerCount <= 0 {
return nil, errors.New("worker count must be positive")
}
if handler == nil {
return nil, errors.New("handler must not be nil")
}
ctx, cancel := context.WithCancel(parent)
q := &Queue{
jobs: make(chan Job, capacity),
handler: handler,
ctx: ctx,
cancel: cancel,
}
q.workers.Add(workerCount)
for i := 0; i < workerCount; i++ {
go q.worker()
}
return q, nil
}
func (q *Queue) worker() {
defer q.workers.Done()
for job := range q.jobs {
if err := q.handler(q.ctx, job); err != nil {
// Replace this with logging, error reporting, or an explicit
// bounded retry policy appropriate to the application.
}
}
}
func (q *Queue) Enqueue(job Job) error {
select {
case <-q.ctx.Done():
return q.ctx.Err()
case q.jobs <- job:
return nil
}
}
func (q *Queue) Close() {
close(q.jobs)
q.workers.Wait()
q.cancel()
}Save this as a package file and import it where needed, or adapt the queue type to your application. The example leaves handler errors for the application to route; silently ignoring them in a production system would hide failed work. In a real implementation, report errors through a logger, callback, result channel, or other deliberate mechanism.
What capacity and worker count control
- Capacity: the maximum number of jobs waiting in the channel. It does not include jobs already received by workers.
- Worker count: the maximum number of handler calls running concurrently, assuming each worker calls the handler once at a time.
- Full queue: the example’s
Enqueueblocks until a worker receives a job or the queue context is canceled. This naturally slows producers when processing falls behind.
For an API that must never wait, use a non-blocking send and return a queue-full error. For a caller-controlled wait limit, make enqueueing context-aware and select between sending, the caller’s context, and queue shutdown. Choose one behavior intentionally so callers can handle overload predictably.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Make shutdown and channel ownership explicit
Closing a channel tells workers that no more jobs will arrive. Workers ranging over the channel finish jobs already buffered and then exit. Only the component that owns all sends should close it: sending to a closed channel panics, and concurrent enqueueing while another goroutine closes the channel creates a lifecycle race.
The simple Close method above assumes the application has already stopped every producer before calling it and that it calls Close only once. It drains queued work and waits for workers; it does not impose a timeout. If producers cannot be stopped first, add synchronization around enqueueing and closure, or expose a single lifecycle owner that stops producers before closing the jobs channel.
The handler receives the queue context, allowing cooperative cancellation. In the shown drain-first shutdown, that context is canceled only after workers finish, so handlers are not canceled merely because the channel was closed. If shutdown must stop slow handlers, define a separate cancellation path and a deadline; decide whether remaining queued jobs should be abandoned or drained. A context cannot forcibly stop a handler that ignores cancellation.
Rank #4
Decide how failures and retries work
A handler error is not automatically a failed delivery protocol. The in-memory example has no retry or dead-letter mechanism. Record or return failures in a way the application can act on, then set a bounded retry policy if retries are appropriate. Avoid immediate unlimited retries, which can consume worker capacity without making progress.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRetries can repeat side effects. A handler that charges an account, sends a message, or changes another system may execute the side effect more than once if work is retried after an uncertain failure. Make such operations idempotent where possible, or use an idempotency key and a retry limit suited to the operation. Redis’s queue guidance also highlights idempotency keys and lower retry counts for non-idempotent actions: Redis guidance on job retries and idempotency.
Best Value
When a channel queue is no longer enough
Use this pattern for bounded, process-local background work—not as a substitute for durable delivery. A broker or persistent job store becomes necessary when jobs must be recovered after a process crash, coordinated across instances, or delivered with explicit at-least-once semantics. Those guarantees require persisted state transitions and recovery handling, not just a larger channel.
The Go project’s Effective Go concurrency guidance describes channels as a way for goroutines to communicate. A channel is a coordination mechanism; durability and retry guarantees must come from the surrounding system you build.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
