RA-Bench finds that the evaluated deepfake detectors do not generalize consistently across the benchmark’s generator sources, and that simulated dissemination can make generated crisis videos harder to detect. The findings describe specific benchmark evaluations—not every detector, video, or social platform.
What RA-Bench tested
Shuo Liang and co-authors introduced RA-Bench in a paper dated August 14, 2026, to evaluate detection of AI-generated videos depicting real-world crisis events. Rather than treating generated clips as an unrelated collection, the benchmark anchors them to real crisis footage and tests detection across multiple generator sources, human judgments, and simulated sharing conditions.
As an Amazon Associate I earn from qualifying purchases.
The benchmark contains 17,886 videos: 1,830 real-video anchors and 16,056 generated clips. The generated material comes from four open-source and five closed-source generators and covers ten social-risk categories. These are benchmark counts, not estimates of how common generated crisis videos are outside the benchmark.
Recommended Free Tools
How the evaluations are organized
Three tracks address different challenges
- Main source-generalization evaluation: tests detection across the benchmark’s generation sources.
- HumanProof: focuses on generated clips that all five assigned reviewers judged to be real.
- LastMile: tests detector robustness across six sequential dissemination conditions described by the project repository.
These tracks separate questions that can otherwise get blurred together: whether a detector works across generator sources, whether people can be fooled by a clip, and whether detection holds up after a video passes through simulated sharing conditions.
#1 Best Overall
Detector families and metrics
The paper evaluates seven traditional detectors, ten zero-shot multimodal models under three review settings, and two fine-tuned multimodal large language models. For generated videos, the repository describes source-matched evaluation: each source is tested against real anchors from which its clips were generated. Results are reported separately for each of the nine generation sources, then averaged arithmetically so each source contributes equally, regardless of how many clips it contains.
Reported measures include AUC and TPR@5%FPR for continuous fake scores, and balanced accuracy, Macro-F1, and FakeR for discrete decisions. The published summary figures below are benchmark-specific headline results; they should not be treated as per-detector scores or as a complete account of every metric.
Rank #2
Why detector performance does not carry over reliably
The paper’s abstract states: “Across these methods, none of the three detector families generalizes consistently across RA-Bench instances.” In practical terms, a result from one evaluation setting is not enough to establish that a detector will work equally well on videos from another generator or under other benchmark conditions.
The authors also analyze video quality, conditioning information, and sampling seeds. They report that generation properties affect detector families differently, while source-level detection patterns remain stable across seeds. That is a more qualified finding than saying that a particular visual flaw explains detection failures: the summary does not establish a single cause, nor does it provide a universal rule for how every detector responds to every clip.
Human-deceptive clips create another test
The RA-Bench project page reports that 633 generated clips were judged “Real” by all five reviewers assigned to each clip. On this challenge subset, traditional detectors averaged 47.5% AUC, according to the project page. This result illustrates that clips capable of fooling the assigned reviewers can also be difficult for the evaluated traditional detectors; it does not show that all people or all detectors would make the same judgment.
HumanProof is a benchmark subset, not a population survey. Its reviewer judgments should therefore be read as part of the evaluation design rather than as a measure of how often viewers are fooled by crisis videos in everyday social-media use.
What the simulated sharing result does—and does not—show
The project page reports that fine-tuned multimodal LLMs’ mean FakeR changed from 46.0% to 1.4% across the displayed LastMile dissemination chain. The repository describes that chain as six sequential dissemination conditions. Together, these findings support the benchmark’s conclusion that dissemination can weaken detection signals in the tested setup.
Those percentages are reported outcomes for the fine-tuned MLLMs and the displayed benchmark chain; they are not a prediction for every detector or a measurement of what happens on every real platform. RA-Bench simulates sequential dissemination conditions. The available summary does not establish that the study tested each platform’s actual processing pipeline or that all platforms apply the same transformations.
Best Value
How to read a detector result about a crisis video
- Check the generator coverage: ask whether the relevant source was represented in the evaluation or whether performance was tested across sources.
- Check the evaluation design: source-matched results, pooled results, and results from a held-out source answer different questions.
- Check the detector family and review setting: traditional detectors, zero-shot multimodal models, and fine-tuned multimodal models are not interchangeable.
- Check what happened to the video: a result on an unaltered benchmark clip does not establish robustness after dissemination.
- Do not treat one score as proof of authenticity: RA-Bench reports inconsistent generalization across its evaluated instances, so a detector score alone cannot settle whether a crisis video is genuine.
For aggregate comparisons in RA-Bench, the equal-source arithmetic mean is important: it gives each of the nine generation sources the same weight instead of letting sources with more clips dominate the average. The benchmark also reports metrics that capture different aspects of detector performance, so a single headline score should not replace the underlying evaluation context.
What the benchmark supports—and what remains unestablished
RA-Bench supports a bounded conclusion: in its evaluations, detection performance varied across benchmark instances, human-deceptive generated clips challenged traditional detectors, and the tested simulated dissemination chain weakened the reported signal for fine-tuned MLLMs. The findings argue for evaluating detectors across generator sources and sharing conditions rather than relying on a result from one standardized setting.
The available paper abstract and project and repository summaries do not provide the complete per-detector tables and methods. They do not support exact per-detector comparisons, statistical-significance claims, a causal account of every failure, or platform-specific generalizations. The results also do not prove that detection is futile; they show why a detector’s benchmark performance needs to be interpreted within the conditions in which it was measured.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Using RA-Bench responsibly
The repository says its rights inventory records release mode, source URL, rights basis, and attribution requirements for real-video anchors. Following record-level rights review, 511 of the 1,830 real anchors are represented by source URLs rather than redistributed media; generated media are released in full. Anyone using the materials should preserve source attribution and follow the terms attached to each source. The repository describes the benchmark as intended for research on detection, provenance, robustness, and responsible media analysis—not for creating or distributing deceptive content.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
