How do I make concurrent requests in Ruby? Async, threads, limits, and failure handling

For several independent HTTP calls, run one Async task per request inside a parent Async block. Ruby’s Fiber Scheduler lets scheduler-aware I/O suspend one fiber while another request uses the thread, so network waits overlap instead of running strictly one after another.

require "async"
require "net/http"
require "uri"

urls = [
  "https://example.com/one",
  "https://example.com/two",
  "https://example.com/three"
]

Async do
  tasks = urls.map do |url|
    Async do
      Net::HTTP.get(URI(url))
    end
  end

  responses = tasks.map(&:wait)
  puts responses.map(&:length)
end

This overlaps supported I/O; it does not make CPU-heavy code parallel, and a library that blocks its thread can pause other fibers. Confirm compatibility with the Ruby and gem versions you deploy.

#	Preview	Product	Price
1		The Ruby Programming Language: Everything You Need to Know	$17.76	Buy on Amazon
2		Programming Ruby 3.3: The Pragmatic Programmers' Guide (Pragmatic Programmers; Facets of Ruby)	$58.31	Buy on Amazon
3		The Little Book Of Ruby Programming: Learn To Program Ruby For Beginners (Little Programming Books)	$18.99	Buy on Amazon
4		The Well Grounded Rubyist	$44.55	Buy on Amazon
5		Eloquent Ruby (Addison-Wesley Professional Ruby Series)	$19.25	Buy on Amazon

How do I make concurrent requests in Ruby?

Install the async gem, require it with net/http, and create a child task for each independent URL. The parent task waits for every child and then processes the collected results.

gem install async

The parent Async call starts the scheduler event loop. Each nested Async call is a fiber-backed task. When a scheduler-compatible operation waits for the network, the scheduler can suspend that fiber and run another task.

#1 Best Overall

Sale

The Ruby Programming Language: Everything You Need to Know

• Used Book in Good Condition

$17.76

Buy on Amazon

A production-shaped example

require "async"
require "net/http"
require "uri"
require "json"

URLS = [
  "https://api.example.test/users/1",
  "https://api.example.test/users/2",
  "https://api.example.test/users/3"
].freeze

def fetch(url)
  uri = URI(url)
  response = Net::HTTP.get_response(uri)

  unless response.is_a?(Net::HTTPSuccess)
    raise "#{url} returned HTTP #{response.code}"
  end

  JSON.parse(response.body)
end

Async do
  tasks = URLS.map do |url|
    Async do
      begin
        { url: url, value: fetch(url), error: nil }
      rescue StandardError => e
        { url: url, value: nil, error: e }
      end
    end
  end

  results = tasks.map(&:wait)

  results.each do |result|
    if result[:error]
      warn "#{result[:url]} failed: #{result[:error].message}"
    else
      puts result[:value].inspect
    end
  end
end

Net::HTTP.get_response lets you inspect status instead of treating every body as success. The example records failures per URL, allowing partial results; an application that requires all-or-nothing behavior can re-raise after collecting or cancel the remaining tasks according to its policy.

What Fiber Scheduler changes—and what it does not

Ruby’s Fiber Scheduler interface allows a scheduler to intercept supported blocking operations and suspend and resume fibers. Async supplies the event loop and task API. This behavior is cooperative: the call must use hooks that the scheduler understands.

• Good fit: many network-bound operations whose clients cooperate with the active scheduler.
• Not a CPU speedup: parsing, encryption, image processing, and other processor-bound work still consumes the thread. Move genuinely parallel CPU work to an approach designed for it.
• Blocking dependencies matter: an extension or library that blocks the thread can prevent every other fiber on that thread from progressing.
• Thread-local assumptions matter: code written around thread-local state may need isolation or a different execution model.

Ruby’s 3.0 release announcement introduced the Async and Net::HTTP pattern. Current scheduler APIs and Async behavior should be checked against the exact Ruby release and gem version installed in your application; current master documentation can describe development versions.

Fan-out patterns and result collection

Independent requests

Fan out only work that is logically independent. Mapping URLs to tasks and calling wait preserves a simple result array corresponding to the task order.

Async do
  tasks = urls.map { |url| Async { Net::HTTP.get(URI(url)) } }
  bodies = tasks.map(&:wait)
end

Dependent requests

If request B needs an identifier returned by request A, keep that dependency sequential. Starting B before A has produced its value is incorrect concurrency, not an optimization.

Rank #2

Sale

Programming Ruby 3.3: The Pragmatic Programmers' Guide (Pragmatic Programmers; Facets of Ruby)

$58.31

Buy on Amazon

Async do
  user = Async { fetch_user }.wait
  orders = Async { fetch_orders(user.fetch("id")) }.wait
  puts orders
end

Preserving identity

For larger jobs, return a structure containing the input, output, status, and exception. This prevents a failed or reordered task from becoming an untraceable body string.

Bound concurrency instead of launching everything

Creating one task for thousands of URLs can overload the remote service, your file descriptors, DNS, memory, or connection capacity. Async’s guidance describes coordinating work beneath a parent task that can act as a semaphore or barrier. The sources do not establish one universally safe number, so choose a limit from the API’s documented quota, your deployment capacity, and measured behavior.

• Start with a deliberately conservative limit.
• Respect server rate limits and Retry-After responses.
• Measure latency, error rate, open connections, and memory while increasing the limit.
• Use cancellation and timeouts so a stalled request cannot hold a slot forever.

Use a semaphore or another Async-compatible limiter supplied by the version you install; verify its exact API in that version’s documentation rather than copying an example for a different release.

Timeouts, retries, cancellation, and partial failure

Concurrency mechanics do not choose an application policy for failures. Decide explicitly:

• Timeouts: set connection and response deadlines appropriate to the service. A timeout should release the task’s capacity.
• Retries: retry transient network errors and selected 5xx responses with bounded attempts and backoff. Do not blindly retry non-idempotent operations.
• Cancellation: if one result is essential, cancel or stop scheduling work when it fails; otherwise collect per-task errors and continue.
• HTTP status: treat redirects, authentication failures, rate limits, and server errors according to the endpoint’s contract.
• Observability: log URL or request ID, elapsed time, status, attempt number, and the final exception without exposing credentials.

Keep request bodies, authorization headers, and cookies out of logs. Validate response size before parsing untrusted JSON or other content.

Rank #3

The Little Book Of Ruby Programming: Learn To Program Ruby For Beginners (Little Programming Books)

$18.99

Buy on Amazon

Net::HTTP sessions and connection reuse

Net::HTTP.get is convenient for one request and manages that request’s session. For repeated calls to one host, Net::HTTP.start with a block starts a session and closes it when the block exits:

uri = URI("https://example.com")

Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https") do |http|
  ["/one", "/two"].each do |path|
    response = http.get(path)
    puts response.code
  end
end

A session can carry multiple requests and reuse a connection. That is different from proving that one mutable session object is safe for simultaneous use by concurrent fibers. Keep client or session ownership clear—one task per session when necessary—and verify behavior for your Ruby version or chosen HTTP client before sharing objects.

When threads or a thread pool fit better

Use threads when existing blocking libraries cannot cooperate with the Fiber Scheduler, when an operation should run outside the fiber event loop, or when the workload is naturally organized as worker jobs. Async’s guidance shows a background-thread fallback for otherwise unsafe code. Concurrent Ruby also provides thread pools and synchronization primitives.

• Bound the pool; an unbounded thread-per-request design simply moves overload elsewhere.
• Protect shared mutable state with appropriate synchronization.
• Keep lock acquisition order consistent; careless locking can deadlock.
• Prefer message passing or immutable values when practical.
• Check whether the HTTP client is thread-safe before sharing a client instance.

Threads do not automatically make CPU work parallel in every Ruby implementation, and they add stack and scheduling overhead. Choose them for compatibility and isolation needs, not because they are universally faster.

Ractors: a specialized alternative

Ractors isolate object sharing and impose more restrictions than fibers or ordinary threads. Ruby 3.0 described Ractor as experimental at introduction. That isolation can suit selected parallel workloads, but it is usually more complexity than needed for ordinary HTTP fan-out.

Rank #4

Sale

The Well Grounded Rubyist

$44.55

Buy on Amazon

Common failures and fixes

Everything still runs one at a time

The HTTP client or another call may block the thread instead of yielding to the scheduler. Confirm scheduler support, look for synchronous extensions, and move incompatible work to a background thread or use a thread pool.

A task raises before results are collected

wait propagates that task’s exception. Wrap each child in a result structure when partial success is acceptable, or rescue around the collection when the whole operation should fail together.

Too many open connections or rate-limit responses

Your fan-out is not bounded for the service or host. Add a semaphore or barrier, reduce the limit, honor server-provided retry timing, and reuse connections only where the client documents safe ownership.

Requests hang indefinitely

Configure connect and read deadlines, ensure cancellation is wired into the task tree, and investigate DNS, TLS, proxy, and remote-server latency separately.

Results are inconsistent between Ruby versions

Compare the deployed Ruby, Async gem, and HTTP-client versions with the documentation you used. Scheduler hooks and library compatibility can change; test the exact production bundle.

Best Value

Sale

Eloquent Ruby (Addison-Wesley Professional Ruby Series)

$19.25

Buy on Amazon

Or skip the browser setup

If your concurrent job is collecting website screenshots rather than API bodies, ScreenshotNeo provides a single HTTP endpoint instead of maintaining browser workers. Before capture it accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result.

Ruby call:

require "net/http"
require "uri"

uri = URI("https://api.screenshotneo.com/v1/shot")
uri.query = URI.encode_www_form(
  access_key: "YOUR_API_KEY",
  url: "https://stripe.com"
)

response = Net::HTTP.get_response(uri)
raise "HTTP #{response.code}" unless response.is_a?(Net::HTTPSuccess)
File.binwrite("shot.webp", response.body)

See the ScreenshotNeo API documentation for options and response details. The same endpoint accepts PNG, JPEG, WebP, or PDF output and supports full-page capture, lazy-image loading, CSS-selector elements, device presets, retina scale, dark mode, custom CSS and JavaScript, clicks, waits, blocked resources, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed links, asynchronous webhooks, bulk capture of up to 100 URLs per call, usage reporting, and an OpenAPI specification. It also offers an MCP server with take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.

cURL:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python:

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

There is a free allowance of 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is included on every plan. Create a free ScreenshotNeo account.

Frequently Asked Questions

Does Async make every Ruby HTTP library non-blocking?

No. Only operations that cooperate with the active Fiber Scheduler yield as expected; incompatible calls can block the thread.

Can I share one Net::HTTP session across concurrent tasks?

The documented session lifecycle explains reuse and cleanup but does not establish that simultaneous use of one mutable session object is safe. Keep ownership separate unless your client documents concurrency safety.

What concurrency limit should I use?

There is no universal safe number. Base it on the remote service’s limits, local resources, and measurements, then enforce it with a semaphore or equivalent coordination.

When should I choose a thread pool?

Choose threads when blocking or scheduler-incompatible dependencies dominate, or when worker isolation suits the workload better than fibers.

Quick Recap

SaleBestseller No. 1

The Ruby Programming Language: Everything You Need to Know

Used Book in Good Condition

$17.76

SaleBestseller No. 2

Programming Ruby 3.3: The Pragmatic Programmers' Guide (Pragmatic Programmers; Facets of Ruby)

$58.31

Bestseller No. 3

The Little Book Of Ruby Programming: Learn To Program Ruby For Beginners (Little Programming Books)

$18.99

SaleBestseller No. 4

The Well Grounded Rubyist

$44.55

SaleBestseller No. 5

Eloquent Ruby (Addison-Wesley Professional Ruby Series)

$19.25

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.