
Two companies test the same detector against the same kind of rewritten text. One reports that it catches 91.8% of it. The other says 35.95%.
That isn’t a hypothetical. GPTZero’s own February 2026 benchmark put its recall on bypasser text at 91.80%, while Originality.ai’s November 2026 launch post scored a GPTZero model at 35.95% on humanized text. Both are vendor tests. Both can be internally honest. They simply made different choices about what to test, how to score it and which version to run.
That’s the thing most people miss about benchmarks. The headline number is the last step of a long chain of decisions, and every link in that chain can move the result. If you know what those decisions are, you can read any benchmark, from any company, with far more confidence.
What Goes Into a Humanizer Benchmark
A benchmark for AI rewriting tools has four moving parts: the texts going in, the tools doing the rewriting, the detectors doing the judging, and the rules for turning detector output into a score. Change one and the result changes with it.
The input texts
This is where most of the bias hides.
Fresh texts vs real-world texts
Some tests write new AI texts specifically for the benchmark. Others pull from public datasets or real user submissions. Fresh texts avoid the risk that a detector has already seen them in training. Real-world texts show how a tool performs on the messy writing people actually submit.
Phrasly’s November 2026 Benchmarks uses both. Its main set is 60 fresh AI texts written by six chatbots, 10 each, across 10 writing types and ranging from 147 to 1,171 words. Before rewriting, all four detectors flagged all 60, at an average human score of 0.9%. A second set of 876 real-world texts was not filtered by detectors at all, and averaged 16.4% human before any rewriting. Phrasly sells the two models it tested, Ultra and Lite, so read the numbers with that in mind. As a design, though, the two-set approach is worth copying.
Why the source model matters
Detectors don’t treat every chatbot equally. A Cambridge-led study called DACTYL tested detectors on roughly 639,000 texts from 11 models across six domains, and found the Gemini 1.5 and Llama families the hardest to detect (DACTYL, 2025). A benchmark built entirely on one chatbot’s output can’t tell you much about the others.
Why the starting point matters
A 2026 benchmark called ARB found something even more striking. Detectors that did well on raw AI text collapsed on human writing that had been revised by an AI. FastDetectGPT fell from 91.2% to 30.8% detection at a 1% false positive rate, and Binoculars from 93.5% to 15.1% (ARB, arXiv 2607.29539, 2026).
@@IMG@@
Detection rate at a 1% false positive rate on raw AI text vs human text revised by an AI. Source: ARB benchmark, arXiv 2607.29539 (2026).
Same detectors, different inputs, wildly different verdicts. That’s why the input set deserves more scrutiny than anything else on a results page.
The tools and their settings
Most humanizers offer modes or strength levels. A fair test says which one it used and sticks to it. The November edition ran Ultra in its Balanced mode on default settings. It also ran the Strong and Max modes but didn’t report them. Lite has only one mode. Without that detail, nobody could repeat the test, and you couldn’t tell whether a result came from the gentlest setting or the most aggressive one.
One attempt per text
The single most important rule is also the simplest: one rewrite per text, with no retries and no picking the best draft. The November test follows it. Any benchmark that regenerates until something passes is measuring persistence, not quality.
The detectors
Benchmarks should name each detector and, ideally, its model version. Detectors change constantly. GPTZero alone shipped 12 base or multilingual model versions in 2026, from Model 4.1b in January to Model 4.11b in October (GPTZero release notes). Originality.ai launched version 5.0 on November 9, 2026. A result from last spring may describe software that no longer exists.
The November edition used four detectors through their APIs: Pangram, GPTZero, Originality.ai and ZeroGPT. It left out Turnitin because Turnitin has no public API, a gap the page states openly. It also calls its results “a dated snapshot,” as of November 17, 2026.
How Raw Detector Output Becomes a Score
Defining “human”
Detectors don’t speak the same language. One reports the probability that a whole document is human. Another reports what share of the text looks human. A good benchmark spells out how it converts each into one comparable number.
The November page does exactly that. Pangram’s score is the share of text read as human. GPTZero’s is the probability that the whole text is human. For Originality.ai it’s 100 minus its AI score, and for ZeroGPT it’s 100 minus the percentage it reads as AI. The “all four” figure is a plain average of the four detector averages.
Averages and pass rates tell different stories
An average hides spread. A model can average 90% while failing one text in five. That’s why stronger benchmarks report a second number: how many texts cleared every detector. In November, Ultra was judged human by all four on 51 of 60 fresh rewrites, against 15 for Lite. Those counts say more about reliability than either average alone.
Confidence intervals
Sixty texts is a small sample, and small samples wobble. The fix is a confidence interval, usually built with the bootstrap, a resampling method introduced by statistician Bradley Efron in 1979. You resample the results thousands of times and see how much the average moves. ARB, for instance, used 5,000 block-bootstrap resamples to report 95% intervals.
Phrasly’s technical write-up, how the Ultra and Lite models were measured, uses the same technique. It reports Ultra at 94.5% (interval 91.7 to 96.9) and Lite at 77.9% (73.5 to 82.0). Many published humanizer comparisons give no intervals at all, which makes their rankings look far more certain than they are.
Quality alongside detection
Passing a detector means little if the meaning is lost. Better benchmarks score quality too. The November edition used an automated grader across the 876 real-world texts. It reported grammar of 9.20 out of 10 for Ultra and 9.34 for Lite, and meaning retained of 7.25 and 7.19. Those last two figures are a useful reminder that even good rewrites drift from the original, and that a separate quality check is worth having.
A Checklist for Reading Any Benchmark
Before trusting a humanizer ranking, check for these:
- Where the input texts came from, how many there were, and how long they ran
- Which chatbots produced them
- Tool modes and settings, stated plainly
- One attempt per text, with no cherry-picking
- Detector names, versions and test dates
- A written definition of the score
- Pass counts, not just averages
- Confidence intervals
- A quality or meaning measure
- Who ran it, and whether they sell one of the products
A test that ticks most of these boxes deserves attention, even when a vendor runs it. A test that ticks two is a press release.
The Number Is Only as Good as the Method Behind It
Every humanizer benchmark is really a stack of choices. Which texts, which settings, which detectors, which scoring rules, how many samples, and how much uncertainty gets admitted. Two honest teams can make different choices and arrive at results 50 points apart, as GPTZero’s and Originality.ai’s own numbers show.
So read the method first and the headline last. Look for fresh and real-world inputs, one attempt per text, named detector versions, clear score definitions and confidence intervals. When those pieces are in place, the final number starts to mean something. When they’re missing, no percentage, however impressive, can make up for it.

