Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix Now×
Skip to content
World desk6 min

How Grab Redesigned Counter Service Storage for Roughly 50% Lower P99 Latency

Grab’s Counter Service redesign grouped bucket counts into timestamp-keyed maps and used staged shadow reads during its Aerospike migration. Grab reports roughly 50% better production p99 read latency, with schema changes driving much of the result.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Grab says a redesign of its Counter Service storage model and a migration to Aerospike improved production p99 read latency by roughly 50% across the service’s read paths. The result is Grab’s own production report, not an independently reproduced benchmark or a general promise about Aerospike. Grab attributes much of the gain to changing how counters were stored: instead of keeping each time bucket in its own row, the chosen design groups timestamped counts into maps.

Why Grab changed a high-volume counter service

Grab’s Counter Service supports anti-fraud rules that need time-windowed counts—for example, the number of recent ride requests from a user or failed payment attempts on a card. A request might ask for a count for a key over the last 90 minutes. The service divides the requested window into the smallest necessary combination of 15-minute, hourly and daily buckets, retrieves those counts and adds them together.

As an Amazon Associate I earn from qualifying purchases.

Grab Tech reported that the service handled tens of thousands of queries per second and about one billion requests per day. Because fraud rules use the results in real time, the service has demanding latency and reliability requirements. It had been using a wide-column database; an infrastructure review led the team to evaluate alternatives, and Grab judged Aerospike a fit for this workload. The migration also gave the team a chance to separate storage concerns from business logic and reconsider the data model.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What changed in the data model

In the original design, each time granularity had a separate table. Processing an incoming event meant issuing three parallel SELECT operations, incrementing the relevant counts in memory, then batch-writing the results. That added four network round-trips per event. The team then tried three Aerospike writer designs.

Design How it handles buckets Read and storage implications reported by Grab
Original wide-column design Separate tables for the three time granularities; rows hold individual buckets. Three parallel SELECTs and a batch write meant four network round-trips per event. Reads involved many small paginated queries.
Aerospike, row per bucket with secondary index Continues to store one row per bucket and uses a secondary index to find buckets in a range. Grab says Aerospike’s primary index uses 64 bytes per record in memory, making the high record count a concern at billions of records. The secondary index also added overhead and operational complexity.
Aerospike, row per bucket with client-side BatchGet Continues to store one row per bucket, with the client using BatchGet to retrieve the records needed for a range. It avoided relying on the secondary index for range lookup, but retained the row-per-bucket cardinality. Grab did not report a specific round-trip count or footprint for this trial.
Selected Aerospike map design Stores all bucket counts for a counter at one granularity in one record, in a sorted map keyed by bucket timestamp. Reads fetch a record and filter map entries to the requested window. Grab reported more than an order of magnitude fewer records in testing after adopting this design.

For the selected design, Aerospike’s atomic MapIncrementOp updates a bucket count, while MapRemoveByKeyRangeOp prunes stale map entries. Record-level expiry acts as a safety net for counters that stop receiving writes. The map design addresses the workload’s repeated bucket lookups without creating a separate record for every bucket.

How reads and writes work after the redesign

Writes update and prune map entries

Rather than incrementing separate bucket rows, a write increments the entry for the relevant timestamp in the map for that counter and granularity. The map operation is atomic, and stale entries can be removed by key range. Expiry on the enclosing record covers counters that no longer receive events.

Reads batch work by granularity

The old wide-column backend made many small, paginated queries to satisfy a range request. In Aerospike, the client groups subqueries by granularity and issues one BatchOperate per granularity. Grab says a user request therefore takes no more than three network round-trips, regardless of the number of subqueries; the client filters the returned map to the requested time range and sums the counts.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

How Grab separated storage from service logic

The reader service was written in Rust, and its original implementation combined business logic with session creation, query construction, fan-out and storage-specific types. Grab extracted storage into backend modules and put a concrete facade with enum dispatch in front of them. The team chose this structure because backend selection happened at runtime; in its account, trait objects with boxed futures would add a heap allocation per query at the service’s volume.

Three configuration-driven modes supported validation and cutover:

  • Single: one backend serves the request.
  • WithShadow: the primary backend supplies the response while a secondary runs asynchronously for parity comparison.
  • WithSplit: a deterministic share of requests is served by each backend during cutover.

Grab describes increasing shadow reads from 5% to 20%, 50% and then 100%, checking parity through metrics. It then shifted live traffic in stages—for example, 5%, 30%, 70% and 100%. The response returned when the primary completed; shadow work ran in the background. This sequence let the team compare results before progressively routing live traffic to the new backend.

What Grab reported after migration

Grab Tech’s 3 July 2026 account reported roughly 50% improvement in production p99 read latency across read paths. It also reported a smaller dataset footprint: around 1 TB on disk on the Aerospike setup versus around 3 TB on the original setup. Grab says the reduced disk footprint was primarily attributable to the map schema, so these figures should not be read as a controlled measurement of the database change alone.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For the Aerospike deployment, Grab reported about 50 GB of primary index using roughly 100 GB of usable memory per node. It also reported roughly 45–50% lower cost per node. These are case-specific production results from Grab’s account; they are not universal performance or cost expectations for other deployments.

Grab also reduced the replication factor from 3 to 2, which it said saved roughly a third of storage and primary-index memory. The team described the resulting setup as tolerating a single-node loss. A simultaneous failure across multiple availability zones remained a risk, which Grab considered acceptable for this workload: counter increments continue from the source event stream, and missing counter data can self-heal as events arrive. Grab reported that the migration finished with zero downtime and no data-integrity issues.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Why index placement and client behavior mattered

Keeping the index in memory

Grab ran Aerospike in Hybrid Memory Architecture, with the primary index in RAM and data on SSD. It tested placing the index on local NVMe, but saw p99 spikes associated with I/O activity on hot keys. More capable nodes improved the issue slightly but did not remove it, so the team returned to an in-memory index on a memory-optimized instance type. The account illustrates that index placement and hot-key access can affect tail latency even when the storage model is otherwise a better fit.

Rust client changes during rollout

At the time of the rollout, the Aerospike Rust client available to Grab was synchronous, so batch reads needed spawn_blocking. After an official async client became available, Grab removed that bridge and observed improvements in p50 and p99 latency. The team also encountered a hostname-resolution issue: the client initially did not re-resolve seed hostnames after a topology refresh, and Grab says a fix shipped in a subsequent release. These are historical implementation details from the migration account, not evidence that current Aerospike Rust clients have the same limitations.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What the case study shows—and what it does not

  • Schema fit mattered: Grab explicitly credits the map-based schema as a major source of the gains, particularly through reduced record cardinality and a smaller disk footprint.
  • Cutover was measured in stages: shadow comparisons and parity metrics preceded staged traffic shifts, making the migration observable rather than an all-at-once switch.
  • Tail latency needs workload-specific testing: the NVMe-index trial exposed p99 sensitivity to I/O on hot keys, despite modest improvement on more capable nodes.
  • The figures describe one production system: Grab reported its own latency, storage and cost outcomes for this workload and architecture; the account does not establish a controlled, database-only comparison or guarantee similar results elsewhere.

The case is best understood as a storage-model redesign carried out alongside a database migration. Aerospike provided the selected backend, but the largest conceptual change was consolidating bucket counts into timestamp-keyed maps and validating the replacement through shadow traffic before shifting production reads.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Outdated Drivers Are Slowing You DownFree scan - exact matches
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.