Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Excess Separability is a white-box method for looking for benchmark contamination in a model’s residual-stream activations, including models that have undergone reinforcement learning (RL) post-training. Its design addresses a real weakness in conventional detectors: brief Group Relative Policy Optimization (GRPO) can make contaminated and uncontaminated examples look similar in outputs and likelihoods. However, the method has not yet demonstrated a reliable contamination verdict on real transformers. In Florian Braun’s revised 2026 preprint, all four real-transformer audit arms were inconclusive; a nominal p=0.0075 became p=0.0745 after the instability of the placebo baseline was included. The defensible conclusion is that activation-level probing is a promising audit direction, not that it has solved post-RL contamination detection.
What is benchmark contamination, and why does it matter?
Benchmark contamination occurs when evaluation questions, answers, or close variants appear in a model’s training data. A contaminated model can then score well because it has memorized or repeatedly encountered the material, rather than because it learned capabilities that generalize to new tasks. That distinction affects leaderboard integrity, comparisons between systems, and claims about progress.
As an Amazon Associate I earn from qualifying purchases.
Free tools Windows power users keep installed
One-click scans. No signup required.
Contamination does not require an exact copy of every test item. Near-duplicates, generated solutions, explanations, or chain-of-thought (CoT) traces can provide useful exposure. A benchmark score therefore becomes difficult to interpret when the training boundary, data filtering, and release dates are unclear. The ICLR 2026 study by Wang, Li, Ko and Zhang treats this as a fairness problem: models with hidden exposure can receive an advantage over models evaluated on genuinely unseen material.
Why can reinforcement learning hide contamination signals?
RL changes the model after supervised fine-tuning (SFT), and that extra optimization can erase the statistical fingerprints that detectors usually measure. In controlled experiments, even brief GRPO reduced the separability between contaminated “members” and matched non-members used by output- and likelihood-based tests.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Two contamination points were tested
- Stage I: benchmark data was introduced during SFT while a base model was being turned into a reasoning model, followed by clean GRPO.
- Stage II: CoT benchmark contamination was applied to an already capable reasoning model.
The study links the collapse in detector performance to PPO-style importance sampling and clipping. Those mechanisms can alter how strongly individual examples influence later updates, making a model that was trained on contaminated material appear less like a straightforward memorizer.
The measured effects show why a clean-looking detector result is not proof of clean training. In the reported setup, clean SFT raised pass@1 by 11.30%. Adding contaminated data during SFT produced another 8.82% average pass@1 inflation across six benchmarks. After GRPO, the contaminated model still retained 7.14% average inflation, even as many detectors lost the ability to distinguish members from non-members. In the advanced-LRM CoT-contamination setting, LiRA averaged only 58.74% AUROC, close to random discrimination rather than a dependable audit signal.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How does Excess Separability inspect a model’s internal state?
Transformers pass information through a residual stream at every layer. Instead of asking only what text the model generates or what probability it assigns, Excess Separability asks whether benchmark members and carefully matched non-members become unusually separable in that internal representation.
The protocol’s four core components
- Residual-stream activations: collect intermediate vectors from selected transformer layers for suspect benchmark items and a clean comparison set.
- Depth-profile probing: measure how well a probe separates the two groups at different layer depths. The resulting profile captures where separability appears and how it changes through the network.
- Level-matched placebo baseline: compare the suspect result with a control constructed to match nuisance conditions rather than assuming that every separation is caused by memorization.
- Label-permutation null: repeatedly shuffle member and non-member labels to estimate how much separation can arise by chance.
The reference set is twice the size of the suspect set. That choice gives the baseline more observations and is intended to stabilize comparisons. The protocol also recentres the depth-profile score against the level-matched placebo, reducing the chance that a probe is reacting to domain, length, formatting, or other surface differences.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
What “excess” means in this context
Raw separability is not enough. A benchmark item may be easier to classify because its wording, token length, formatting, or subject matter differs from the comparison material. Excess Separability is the residual signal left after subtracting a matched nuisance-control profile. A result is interesting only if it exceeds both that placebo behavior and the distribution produced by label permutations.
Did the method really survive RL post-training?
It was designed to target the post-RL gap, but the current evidence does not establish that it survived RL in the sense of delivering a validated contamination finding. Braun’s revised preprint audited four contaminated real-transformer checkpoint arms. In every arm, the placebo baseline was more variable than the permutation null: its standard deviation was 1.30 to 1.56 times larger.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesThat variance matters because an apparently unusual score can look significant when the control itself is treated as stable. Once the placebo uncertainty was propagated into the test, the nominal p=0.0075 result changed to p=0.0745. The revised abstract states, “No arm shows contamination,” and the paper issues no contamination verdict.
This is a statistical correction, not a minor editorial downgrade. It means the method identified a promising pattern under an optimistic variance assumption, then failed to clear the revised significance threshold when the control’s sampling instability was acknowledged. The result does not prove that the checkpoints were clean; it shows that this protocol could not distinguish contamination from controlled variation in those audits.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
How does Excess Separability compare with conventional detectors?
The main distinction is where each method obtains its signal and what assumptions it makes about post-training behavior.
| Method family | Primary signal | Effect of RL in the reported studies | Access requirement | Current evidentiary status |
|---|---|---|---|---|
| Likelihood or loss tests | Token probabilities, losses, or perplexity on suspected items | Separability can shrink after GRPO and related optimization | Token-level probabilities; unavailable from many output-only APIs | Useful screening signal, not sufficient by itself after RL |
| Generation-based detectors | Answer choices, completions, stylistic behavior, or consistency across prompts | Many representative detectors moved toward random performance after brief GRPO; CoT contamination was especially difficult | Ability to generate repeated outputs and control prompts | Vulnerable when the model’s policy is changed after contamination |
| Reference-model methods such as LiRA | Comparison with models or distributions trained under known conditions | LiRA averaged 58.74% AUROC in the advanced-LRM CoT-contamination experiment | Reference models, calibrated data, and substantial compute | Can quantify exposure in controlled settings but is not robust to every post-RL case |
| Excess Separability | Residual-stream activation geometry across layers, corrected by placebo and permutation controls | Specifically proposed for signals that output tests lose after SFT or RL | White-box or intermediate-activation access; output-only APIs cannot run it as described | Four real-transformer audits produced no corrected contamination verdict |
The comparison is not a claim that internal probes are inherently superior. Activation geometry can provide information that outputs conceal, but it also introduces a new statistical burden: the null and placebo must be stable enough to support a calibrated decision.
What would an auditor need to run the protocol?
Excess Separability is practical only for organizations that can instrument the model or obtain saved activations. A normal chat endpoint that returns text, and perhaps aggregate usage statistics, does not expose the residual stream needed for the analysis.
Operational checklist
- Model access: obtain intermediate activations at defined layers, with a fixed tokenizer and inference configuration.
- Suspect set: identify benchmark items or a documented sample whose possible exposure is being tested.
- Matched reference set: build a comparison pool twice as large as the suspect set, matching relevant domain, length, formatting, and difficulty variables.
- Checkpoint history: retain models from before SFT, after SFT, after GRPO, and after any later RL stage when possible. A final checkpoint alone cannot show when a signal appeared or disappeared.
- Independent splits: repeat the analysis over multiple split seeds rather than relying on one convenient partition.
- Null construction: run label permutations and estimate the placebo baseline separately, then propagate the placebo’s observed variance into the final uncertainty.
- Pre-registered decision rule: define the threshold, layer aggregation, and number of permutations before inspecting the result.
A defensible execution sequence
- Freeze the benchmark sample and document any known overlap or deduplication procedure.
- Extract residual-stream activations at the same layers and token positions for suspect and reference examples.
- Train or fit the depth-profile probe without allowing the probe to learn the benchmark labels from formatting artifacts.
- Construct the level-matched placebo and recentre the observed profile against it.
- Generate the label-permutation null with enough repetitions to estimate its tails, recording the random seeds.
- Repeat every stage across independent split seeds and checkpoints, then compare the placebo and null variances before interpreting a p-value.
- Report a verdict only if the corrected statistic clears the pre-specified threshold and the effect is stable across splits and checkpoints.
What are the method’s main limitations?
It is white-box by design
Closed commercial models that expose only text cannot be audited with this protocol unless the provider supplies intermediate activations or an equivalent diagnostic interface. Prompting the model more times does not recreate residual-stream access.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
A significant-looking score can be a control artifact
The revised preprint demonstrates that the placebo baseline can fluctuate more than the permutation null. Any implementation that treats the placebo as fixed risks overstating evidence, even when the model and benchmark are held constant.
Contamination is broader than exact memorization
Near-duplicate questions, solution traces, and CoT data can affect generalization differently. A probe calibrated on exact item exposure may not detect every form of related training exposure, and failure to detect a signal is not proof that no overlap exists.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Real-world replication is still limited
The four reported real-transformer audit arms did not yield a corrected positive finding. That leaves open questions about model scale, architecture, contamination volume, layer choice, probe capacity, and the behavior of models trained with RL procedures other than the reported GRPO setup.
Secondary coverage can overstate the result
A Tech Times article published on 14 August 2026 presented the method as reading activation geometry without training-corpus access or reference-model retraining. The arXiv record was revised on 16 August 2026, and the revised primary account is more cautious: it withdraws the nominally significant claim after variance correction. The preprint, not the earlier headline, should control the interpretation.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
What should organizations do about contamination now?
Evaluation design is more dependable than assuming a detector will recover a hidden training history after the fact. Teams building or auditing models should use private holdouts, rolling release cutoffs, and documented deduplication, while treating benchmark scores as provisional when training-data disclosure is incomplete.
For open-weight model auditors
- Use activation probes as one component of a multi-signal audit, not as a standalone certificate.
- Publish checkpoint-by-checkpoint results, split seeds, nuisance features, placebo variance, and permutation settings.
- Test whether a signal survives changes in layer range, token pooling, benchmark subset, and matched-reference construction.
- Separate “no evidence detected” from “evidence of no contamination” in the report.
For closed-model evaluators
- Keep sensitive test items private and rotate them before public release.
- Use fresh, task-balanced holdouts whose publication dates postdate the model’s documented data cutoff.
- Request intermediate checkpoints or contamination disclosures from model providers when high-stakes claims depend on benchmark scores.
- Evaluate performance on newly authored and adversarially paraphrased tasks, while recognizing that these tests reduce but do not eliminate exposure risk.
What is the most accurate takeaway?
Excess Separability addresses an important technical question: can internal activation geometry reveal contamination after output-level clues have been altered by RL? Its nuisance controls, doubled reference set, and permutation null are a thoughtful framework for investigating that question. Yet the revised real-transformer audits did not support a contamination verdict, and the only nominally significant result disappeared when placebo variance was included.
The ICLR 2026 RL-fragility study supplies the broader warning: brief GRPO can markedly conceal signals used by many existing detectors, while CoT contamination in advanced reasoning models may leave little detectable evidence. Until activation-level methods are independently replicated with stable, variance-aware controls, the safer practice is to prevent contamination through private evaluation design and transparent checkpoint reporting rather than treating any detector as definitive.
Frequently Asked Questions
Can an output-only API run Excess Separability?
No. The protocol needs residual-stream activations from intermediate layers, which ordinary text-generation endpoints do not expose. A provider would have to release activation hooks, saved vectors, or an equivalent white-box interface.
Does a failed detector prove that a benchmark was not in training?
No. RL can suppress observable member/non-member differences, and contamination may involve paraphrases or reasoning traces rather than exact strings. “No signal detected” should be reported as an inconclusive audit result unless the test has demonstrated calibrated sensitivity for the specific exposure pattern.
Why use a reference set twice as large as the suspect set?
The larger comparison pool gives the nuisance-matched placebo more observations and can reduce ordinary sampling noise. It does not solve instability automatically; the revised audits still found placebo variation 1.30–1.56 times the permutation-null variation.
Recommended Free Tools
What evidence would make an activation probe convincing?
Independent teams would need to reproduce the effect across split seeds, checkpoints, model families, and contamination mechanisms, while publishing the complete placebo and permutation distributions. A positive result should remain significant after the placebo baseline’s uncertainty is propagated and should be distinguished from performance changes caused by domain or formatting differences.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
