October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run ScanOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk4 min

2.2 ms, 19.9 ms, 529 ms: What an A/B Testing Runtime Benchmark Really Measures

ABTestly’s 2.2 ms, 19.9 ms and 529 ms figures measure variation-to-DOM timing in three different cache and navigation conditions—not universal runtime speed.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The figures 2.2 ms, 19.9 ms and 529 ms are not three scores for one universal A/B testing runtime speed. They are ABTestly’s reported 75th-percentile times for one event—the moment a variation lands in the DOM—in three different navigation and cache conditions. The measurements are useful as a description of that test, but they do not establish how quickly every experiment will appear to real visitors.

What the three figures measure

In a September 26, 2026 post, ABTestly reported timing 200 page loads per condition in headless Chromium. Its endpoint was the time from navigation start until a variation landed in the DOM. The test used the built runtime and a single variation that rewrote an above-the-fold heading.

As an Amazon Associate I earn from qualifying purchases.

Condition ABTestly-reported p75 What the condition means
Single-page app route change 2.2 ms A route changed within a single-page application.
Repeat view, warm cache 19.9 ms A repeat page view with cached resources.
First visit, empty cache, throttled 529 ms A first view with an empty cache, tested at 1.6 Mbps downstream and a 150 ms round trip.

These are the vendor’s observations, not independently replicated results. The large spread reflects distinct test conditions; it is not a like-for-like ranking of three runtimes or three versions of the same page load. ABTestly’s post describes the setup and results.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

DOM arrival is not the same as what visitors see

The benchmark’s endpoint is a DOM change, not a measurement of when a visitor first sees the variation. In the throttled first-visit condition, ABTestly says the original heading appeared on screen before replacement in all 200 loads, with a median delay of 326 ms before it was replaced. On cached repeat views, the original was absent from every painted frame in 199 of 200 loads; on route changes, it was absent in all loads, according to the post.

That distinction matters: a variation can land in the DOM quickly after navigation and still follow an initial paint of the original content. The reported median is specific to this heading rewrite and throttled condition; it does not establish the amount of flicker on other pages or with more complex variations.

Why 529 ms is not a real-world first-visit estimate

ABTestly says the benchmark ran against a local origin. The stated network throttle applied, but the test did not include real first-visit DNS lookup, TLS setup or edge latency. The processor was not throttled either. ABTestly describes 529 ms as a floor rather than a field result and says a first visit on a mid-range phone would be slower. The post does not provide evidence quantifying how much slower.

The scope is also narrow: one browser setup, one heading rewrite and one local-origin benchmark. The result does not establish timing across other browsers, devices, network conditions, page structures or variation complexity. Nor is this endpoint a page-load metric.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use LCP as complementary context, not a substitute

Largest Contentful Paint (LCP) measures when the largest visible image, text block or video is rendered relative to navigation. It is different from timing when a variation enters the DOM. web.dev’s LCP guidance recommends evaluating the 75th percentile of page loads separately for mobile and desktop, with 2.5 seconds or less considered a good LCP target. It also notes that connection setup, redirects and time to first byte can materially affect field measurements.

That guidance helps put page experience in context; it does not validate or contradict ABTestly’s runtime figures. A runtime benchmark and field LCP answer different questions and should not be presented as interchangeable scores.

The anti-flicker tradeoff

ABTestly says its runtime loads as a dynamic script and does not block the page parser. Its optional anti-flicker mechanism hides the page using an opacity rule until variants apply or a two-second timeout expires; the setting is unchecked by default.

The vendor’s rationale is that page-wide hiding affects visitors who are not assigned to an experiment and could leave a page visibly empty while configuration is slow, whereas flicker is limited to pages actually changed by a variant. This is ABTestly’s design rationale, not an independently tested comparison showing which approach produces a better visitor experience.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What the speed guardrail can—and cannot—tell you

ABTestly’s dashboard flags a variation as slower when its p75 LCP is at least 400 ms above the control. A row is evaluated only after passing these checks, stopping at the first failure:

  • Capture rate is not above 100%.
  • Each arm has at least 100 page loads.
  • The capture-rate gap between arms is no more than 20 percentage points.
  • Capture is at least 50% in each arm.

ABTestly explicitly characterizes this as a descriptive, not inferential, measure: “There is no confidence interval, no bootstrap, and no significance test. The panel is a descriptive guardrail, not an inferential one.” The vendor also warns that a 400 ms difference based on 100 loads per arm is not equivalent evidence to the same difference based on 100,000. Although the panel shows load counts, it does not weight the evidence for the reader.

The panel makes no correction across devices or variations. In ABTestly’s example, four variants across two devices create six comparisons, each judged against its own threshold. A result marked “not flagged” therefore means there was no visible threshold crossing at the displayed volume—not that the variation has been proved safe or that the difference is statistically reliable.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Questions to ask when evaluating a runtime benchmark

When a vendor shares a speed figure, ask for the measurement details that determine what it means:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Endpoint: Is the clock stopping when a variant enters the DOM, when it is painted, or at a page-level metric such as LCP?
  • Navigation and cache: Are route changes, warm-cache repeat views and uncached first visits reported separately?
  • Test environment: Was the origin local or deployed? Were network and processor both constrained? Which browser, device and page were used?
  • Visible behavior: How often did original content paint before the variation, and for how long?
  • Evidence behind flags: What are the load counts and capture rates for each arm? Does the tool report uncertainty, and how does it handle multiple device and variation comparisons?

These questions make it possible to distinguish a useful, bounded benchmark from a claim about general visitor experience. The specific figures here should be read within ABTestly’s stated test conditions and limitations.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Outdated Drivers Are Slowing You DownFree scan - exact matches

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.