An Australian Securities and Investments Commission (ASIC) proof of concept found that employee-written summaries outscored summaries generated by Llama 2 70B on a specific document-analysis task. The aggregate scores were 61 out of 75 (81%) for human summaries and 35 out of 75 (47%) for AI summaries. The result describes one model, one task and a short 2024 test—not AI performance across jobs or current systems generally.
What ASIC tested
ASIC worked with Amazon Web Services Professional Services on the proof of concept from January 15 to February 16, 2024. The task was to summarize public submissions to a parliamentary inquiry into ethical and professional-accountability challenges in the audit, assurance and consultancy industry. The summaries were meant to identify material relevant to ASIC and include references and page numbers.
As an Amazon Associate I earn from qualifying purchases.
The AI system used Llama 2 70B, and ASIC employees also wrote summaries. Five evaluators read the source documents and assessed the outputs. Futurism reports that the summaries were labeled A and B for blind assessment, although three evaluators later said they suspected which outputs were AI-generated.
Free tools Windows power users keep installed
One-click scans. No signup required.
How the scores compare
| Summaries | Aggregate score | Meaning |
|---|---|---|
| Employee-written | 61 of 75 points (81%) | Aggregate result against the proof of concept’s assessment rubric, as reported in ASIC’s answer reproduced by Going Concern. |
| Llama 2 70B-generated | 35 of 75 points (47%) | Aggregate result against the same rubric, as reported in ASIC’s answer reproduced by Going Concern. |
These percentages are rubric scores, not the share of summaries that were correct or a measure of overall workplace productivity. ASIC’s reproduced answer says the AI summaries scored lower on every criterion. It also noted that outputs could require additional fact-checking or be less useful than the original material, partly because the model struggled to capture the nuance and context needed to analyze submissions.
#1 Best Overall
Futurism’s account adds that the system failed to provide requested page numbers and that some summaries were wordy, irrelevant or redundant. Those details come from the news report; the official parliamentary document was not independently accessible for verification here.
What the result does—and does not—show
The test indicates that this implementation performed poorly on a document-heavy assignment where contextual judgment and traceable references mattered. If checking and correcting a generated summary takes substantial effort, automation may not save time or improve the work.
Rank #2
But the experiment was a proof of concept, not an AI system deployed in ASIC’s regulatory work. ASIC’s reproduced answer explicitly limits the finding to one model at one point in time and one use case. It does not establish that AI generally performs worse than employees, that every model would score similarly, or that the result applies to other tasks. The test also was not designed to compare AI with human workers across occupations.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What ASIC said about improving results
ASIC’s reproduced observations said generic prompts produced lower-quality outputs than specific or targeted instructions. They also emphasized experimentation and iteration, monitoring outcomes, and active feedback between data scientists and subject-matter experts. The answer noted that the proof of concept’s short duration limited opportunities to optimize the setup.
ASIC expressed the expectation at the time that AI capabilities would improve as models changed. That was a contemporary expectation, not evidence from this test about how later models perform.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Bottom line for readers
In ASIC’s early-2024 proof of concept, human summaries beat Llama 2 70B summaries on the trial’s rubric for analyzing parliamentary submissions. The gap is a warning about using generated summaries for evidence-sensitive work without verification—not a universal verdict on AI or a forecast of how today’s models would perform.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
