Companies should publish direct, reproducible evaluations of their products against named competitors—and show where they lose. Gorgias’s ecommerce AI-agent benchmark makes the case: its October 2026 page ranks Gorgias first overall, but third in shopping and slower than named rivals on shopping responses. That transparency makes the comparison more useful, not less. It does not make a vendor-run benchmark independent: Gorgias publishes the test and competes in it.
Why publish direct competitive evaluations?
Product demos and vendor claims rarely let a buyer compare products on the same task. A useful competitive evaluation puts live products through comparable work, names the competitors, explains how results are scored, and makes the losing categories visible. Jason Lemkin, writing for SaaStr in September 2026, put the principle plainly: “Run them on live products, against named competitors, and include the categories where you lose.”
As an Amazon Associate I earn from qualifying purchases.
That standard is particularly relevant to AI agents. A polished demo does not establish how an agent behaves across ordinary customer questions, store data, or the moments when it should resolve a request rather than hand it to a person. Evaluating live behavior can reveal trade-offs that a single headline score conceals.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhat the Gorgias benchmark tests
Gorgias’s benchmark covers two ecommerce jobs: a shopping assistant that helps a shopper find and buy a product, and a support agent expected to resolve questions about shipping, returns, and policies without a human. Vendors receive the same questions, adapted to each store’s catalog. The benchmark page, marked refreshed October 2026, reports 9,226 conversations captured, 9,220 blind-judged, 18 vendors, and 224 live stores. Those are counts for that changing benchmark snapshot, not permanent or market-wide totals.
#1 Best Overall
How the test is run
- Cold sessions: each run begins in a cold browser session.
- Agent-initiated handoff: the auditor does not ask to speak to a human, so a handoff counted in the test is initiated by the agent.
- Blind judging: vendor names are hidden from judges.
- Transcript evidence: claims count only when they are quoted from the conversation transcript. The page describes quality scoring as binary, evidence-forced checks rather than a judge simply selecting a score.
- Minimum sample: a vendor needs at least 15 judged conversations in a job to qualify for a head-to-head rank.
Gorgias says the benchmark is rerun weekly. Repeated runs and a minimum sample can make results more informative than a handful of curated examples, but they do not by themselves prove that the scoring rules are neutral or that every store will see the same outcome.
How to read the scores and rankings
The benchmark measures automation, answer quality, and speed. Automation is the share of conversations resolved without a human; quality is a blind score from 0 to 100; speed is the time to a complete answer. Gorgias combines those measures with different weights for each job:
| Job | Automation | Answer quality | Speed |
|---|---|---|---|
| Shopping assistant | 40% | 35% | 25% |
| Support agent | 50% | 40% | 10% |
These are Gorgias’s stated composite weights, not a universal definition of the best AI agent. A business that prizes containment may prefer the support weighting; one that values quick shopping answers may want speed to count more. A rank can change when the weights change even if the underlying conversation results do not.
In the October 2026 display, Gorgias is #1 overall, #2 in support, and #3 in shopping. Gorgias reports support answer quality of 74/100 and says its shopping answer quality is the highest in the field. It also identifies speed as a weakness: its shopping answers take about 18 seconds, compared with about 8 seconds for Envive and 10 seconds for Sierra; its support answers take about 14 seconds. These are figures and claims from Gorgias’s benchmark, not independently audited measurements.
The central lesson is that automation, quality, and speed are separate axes. An agent can answer accurately but slowly, or resolve more conversations while giving weaker answers. The composite is useful only when the reader can see the constituent measures and the weighting choices.
Why the date and reporting window matter
The SaaStr case study and the live Gorgias page describe different snapshots. Lemkin’s September 26, 2026 article reported 8,356 live conversations, 18 vendors, and more than 212 storefronts. The October 2026 benchmark page reports 9,226 conversations captured, 9,220 judged, 18 vendors, and 224 live stores. These figures should not be combined as if they were collected for one fixed reporting period.
The earlier SaaStr article also reported an Envive pre-sale composite of 72 against Gorgias at 65; Gorgias answer quality of 76; shopping response times of 18.4 seconds for Gorgias and 7.9 seconds for Envive; and 28% of Gorgias shopping answers taking longer than 20 seconds. Lemkin said that applying the support weights instead produced a recalculated Gorgias score of 74.3. Those are the article’s September snapshot and calculation, not the October live page’s current score descriptions. In particular, a comparison is hard to interpret unless it specifies the job, metric, weighting, and date.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →What the case study does—and does not—establish
Lemkin’s article describes Gorgias as a roughly $100 million ARR company and says about 80% of its revenue came from AI support for ecommerce brands. Those company figures are claims reported by the September 2026 SaaStr article; they are not independently verified by the benchmark figures described here. The benchmark itself is evidence about measured performance under its stated setup, not proof of a company’s revenue, broad customer outcomes, or superiority for every merchant.
There is also a clear relationship to disclose. SaaStrFund led Gorgias’s seed round, according to Lemkin, and he says SaaStr encouraged Gorgias to publish the evaluation. That context does not invalidate the benchmark, but readers should know it when weighing the case study.
Gorgias is both a vendor in the comparison and the benchmark publisher. Its page says it applies the same blind rubric to itself and competitors, and says a former rule that excluded Gorgias alone was removed in July 2026. The result is a vendor-published comparison, not independent certification. The benchmark itself cautions that no vendor leads automation, quality, and speed all at once; vendors can perform differently across stores; configuration matters; and nearly a third of detected “AI chat” widgets did not produce a real conversation. Those cautions are findings as reported by Gorgias, not independently audited market statistics.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical standard for publishing your own eval
A company considering a competitive evaluation should publish enough information for a skeptical reader to understand what was tested and what the headline rank means. The Gorgias example points to a concrete checklist:
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute- Define the job: distinguish tasks such as shopping assistance from support resolution instead of treating “AI agent” as one uniform category.
- Name the competitors and test live products: make it possible to understand which products were compared and what they actually did.
- Hold conditions as constant as practical: use comparable questions and disclose how store catalogs, configuration, and session conditions affect each run.
- Publish the scoring rubric: explain the measures, how evidence is judged, and how any composite is calculated.
- Show weights and underlying measures: report automation, answer quality, and completion time separately so readers can apply their own priorities.
- Disclose sample size, thresholds, and dates: say how many conversations were captured and judged, what minimum qualifies a vendor, and when results were collected or refreshed.
- Report losses and trade-offs: do not bury a weak category beneath a favorable overall rank.
- Disclose interests and make the method inspectable: state the publisher’s stake. Gorgias says its harness is open source, and its public GitHub repository lets readers inspect project files; code availability alone does not show that every reader has rerun or validated the benchmark.
A buyer comparing two agents can use the same framework: check automation or containment, answer quality and transcript evidence, time to a complete answer, task type, store setup, sample size and reporting window, and composite weights. A vendor’s overall position is only one part of that decision.
Best Value
- 100 Two-Part Carbonless Sets in One Book – Each service call log book includes 100 preprinted 2-part carbonless forms Write once and produce a duplicate copy instantly without separate carbon sheets Ideal for service call logs work order records and daily business documentation
- Compact Size for Convenient Daily Use – The 5 5/8 x 8 1/2 inch layout offers a comfortable writing area while fitting neatly on desks service counters and clipboards The portable size makes this service call log book easy to carry for both office staff and field technicians ensuring quick and efficient documentation anywhere
- Includes Writing Shield for Clean Copies – Each book comes with a sturdy backing board that prevents ink bleed-through to other sets and provides a firm writing surface making it convenient for field service technicians and office front desk use
- Durable Spiral Binding for Smooth Use – Strong metal spiral binding keeps all sets secure and allows pages to flip easily and lay flat while writing Sheets tear off cleanly for customer or office copies supporting mobile and on-site communication needs
- Versatile for Service and Office Applications – Suitable for HVAC repairs plumbing electrical maintenance appliance service and more Also functions as a phone call log book or invoice receipt book for small businesses ensuring professional job tracking and customer messaging
Where to be cautious
A disclosed rubric improves scrutiny, but it cannot eliminate every source of bias. The publisher chooses the tasks, stores, conditions, metrics, and weights; it may also have commercial incentives. Blind judging can reduce the influence of vendor identity on scoring, while consistent tasks can improve comparability, but neither makes the test independent of those choices.
Results also depend on setup and context. Gorgias notes that the same vendor may behave differently across stores and that configuration matters. Its report that almost a third of detected AI chat widgets did not produce a real conversation is a reminder that a widget’s presence is not evidence of a functioning agent. Readers should treat the benchmark as directional evidence about the tested runs and ask whether their own catalog, policies, and support needs resemble those conditions.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
