Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesTwo days was the time Baptiste Laget says his team spent assembling Milliseconds.ai—not building its models or an entire serving platform from scratch. The models and much of the supporting infrastructure had already been developed over months for CloudRaker’s Paperwork product. The short build focused on a different request path for fast inference: scheduling, admission checks and routing requests to GPU runners.
What took two days?
In his case study, Baptiste Laget describes assembling a product from existing models and infrastructure. The new work concentrated on the parts that mattered for short decision requests. Paperwork already handled large PDFs, signature workflows and redaction jobs, but its gateway had been designed for longer-running document processing.
As an Amazon Associate I earn from qualifying purchases.
That gateway’s authentication, authorization, tenant context, logging, tracing, metering and network hops were acceptable for lengthy jobs. For an inference call lasting only a few milliseconds, however, those steps could outweigh the model work. Laget says the team benchmarked the decision routes and reviewed Dash0 traces; in the worst case they observed, gateway overhead took twelve times as long as inference. That figure is the team’s report about its own system, not an independently validated or generally applicable benchmark.
As Laget put it, “The title leaves out months of work on Paperwork.” The two-day timeline is best understood as a focused assembly and redesign effort, made possible by a substantial base that already existed.
Why give fast decisions a separate request path?
The key design choice was to separate a fast decision API from a gateway built for document jobs. When inference itself takes milliseconds, reducing work around the model can matter as much as optimizing the model call. The team’s goal was to keep authentication and admission controls while avoiding unnecessary gateway cost on the critical path.
Every request still needed authentication, an organization usage check, an available inference slot and a later billing record. The design split those responsibilities across a lightweight front end, organization state and a scheduler rather than sending short calls through the existing long-job path.
How did a request reach a model?
The API front end was a Cloudflare Worker running Hono. Its bindings connected to D1 for hashed API keys, Analytics Engine for request metrics, and the metering service. Durable Objects handled organization state and regional inference scheduling. The GPU runners were separate services, reached through a Cloudflare tunnel and Workers VPC binding.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Authenticate and admit: The Worker used an API key to identify an organization, check cached admission information and obtain a billing verdict through the metering service.
- Lease capacity: The regional scheduler selected an available inference slot, preferring GPU capacity and falling back to CPU slots when GPUs were full.
- Send the inference request: The Worker routed work to a runner through the tunnel rather than to a public endpoint on each GPU machine.
- Release and record: After a response, the slot lease was released and usage could be recorded for billing.
This arrangement put request handling and scheduling at the edge while leaving the GPU execution environment outside Cloudflare. The case study does not name the cloud provider, GPU model, instance type or regions.
How did scheduling handle a full GPU fleet?
A Durable Object per GPU region tracked inference slots in memory. Requests leased slots; when all were busy, new work queued. If a slot did not become available in time, the API returned HTTP 529 with a retry hint. Laget says a failed slot was skipped for thirty seconds so that a retry could be directed to a different GPU host.
Rank #2
- [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
- [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
- [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
- [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
- [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.
The scheduler also adjusted a spot-GPU fleet as demand changed. Its pool was temporary: an in-flight lease kept the Durable Object alive, and a cold start rebuilt the pool instead of restoring persisted scheduler state. That is the design described for this system, not a general recommendation; a different workload may need durable scheduling state or different queue and retry behavior.
How did organization admission and billing work?
API keys used a namespace containing the organization ID, allowing the Worker to find the relevant Durable Object without a database lookup. Namespace objects mirrored keys from the database, maintained token buckets for requests per minute and input tokens per minute, and kept a usage ledger in fifteen-minute buckets. Metering ran through a Worker service binding to Schematic, which deducted credits and returned a billing verdict.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallThe Worker cached each key’s admission verdict and rate-limit headers for sixty seconds at each location that saw the key. Because usage was recorded after the response, a cached verdict could admit a burst that exceeded a limit before a block took effect. Laget says the team accepted that enforcement trade-off to remove most admission checks from the critical path. This is a concrete latency-versus-limit-enforcement choice, not a guarantee of strict real-time rate limiting.
How were the GPU runners connected?
The GPUs ran outside Cloudflare on spot instances in managed instance groups across several regions. Each VM ran an inference runner and cloudflared. Workers reached those runners through a Workers VPC binding, avoiding a public endpoint for each runner. Laget says that when a VM was preempted, its connector dropped while the tunnel continued through remaining instances.
The account does not specify hardware, instance types, provider, named regions, traffic volume, availability or cost savings. It therefore describes a connection pattern, not a hardware shopping list or evidence that the same setup will meet another team’s capacity or reliability needs.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What did image inference require?
Callers sent images as base64 in the request body; the API did not accept image URLs. Laget says the choice avoided fetching external images and adding latency outside the team’s control. The service resized the longest image edge to one of three detail tiers:
| Detail tier | Longest edge | Token cost | Reported image-decision latency |
|---|---|---|---|
| Low | 512 pixels | Fixed cost; amount not stated | Part of the reported 45–120 ms range; per-tier latency not stated |
| Medium | 768 pixels | Fixed cost; amount not stated | Part of the reported 45–120 ms range; per-tier latency not stated |
| High | 1024 pixels | Fixed cost; amount not stated | Part of the reported 45–120 ms range; per-tier latency not stated |
The 45–120 ms figure is Laget’s reported range across detail tiers, not an independent benchmark; the case study does not give a separate latency for each tier. He also says images were processed in memory at the runner and were not written to disk or included in logs. That describes the implementation as reported by its author, not an independent security audit.
What existing work made the schedule plausible?
The team reused its Worker template and configuration conventions, three environments, CI, an API-spec-generated typed client, shared secret vault, release pipeline and admin access policies. Laget says these established controls also supported the company’s SOC 2 Type II setup. The case study does not include an audit report or independent verification of the controls.
That reuse explains why the two-day effort could focus on changed request handling rather than rebuilding deployment, access and operational foundations. For another team, the timeline depends heavily on whether comparable models, services and delivery practices already exist.
When does this architecture fit?
The case study suggests a set of questions to answer before separating a fast inference path from an existing gateway. They are decision criteria drawn from Laget’s account, not results from a comparison of competing products.
- How long does the work take? A millisecond-scale decision call is more sensitive to gateway overhead than a long document job.
- How large is the overhead relative to inference? Measure the whole request path; the reported twelve-to-one worst case belongs to this team’s benchmarks and traces only.
- What happens when GPUs are full? Decide whether to queue, fall back to CPU, reject with a retry hint, or combine those behaviors, and set retry rules that avoid repeatedly targeting a failed host.
- How strict must admission enforcement be? Caching can lower critical-path latency, but recording usage after a response may permit bursts beyond a limit.
- What foundations already exist? Reusable models, credentials, CI, deployment, observability and access policies make a short assembly effort more plausible than starting from zero.
Laget’s account is a first-person case study. Its timeline, measurements and operational details are author-reported; they do not establish that another team can rebuild a complete inference stack in two days.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
