So we stopped looking for one, and built something that catches it when it's wrong. This page is the long version — including the parts that didn't work.
Someone asked the drive how many drops of bleach to use for a gallon of water. It found a real document, quoted it correctly — two drops per quart — and then did this:
"Since there are four quarts in a gallon, you would need eight drops for one gallon."
The arithmetic is right. The answer is wrong. The EPA lists common household bleach at two drops per quart and six drops per gallon, because published dose tables are rounded for safety and don't scale in a straight line. The AI had no way to know that, and it sounded completely confident. It also left out the bleach strength the whole figure depends on.
Nothing about that answer looks wrong on the page. That's the problem worth solving.
We wrote a large set of questions designed specifically to cause that kind of failure — not general knowledge quizzes, but traps. Questions that ask for a unit the source doesn't use. Questions where the right answer depends on a strength or a contact time that's easy to drop. Questions where two manuals on the drive disagree. Questions that ask two things when the library only covers one. Questions whose answer simply isn't in the library at all, to see whether the AI admits it.
Then we ran every model we could get through all of them, using the same search the app really uses, and graded the answers automatically so our opinion didn't come into it.
Every model failed somewhere. Including the good ones. Including the big ones. Size predicted almost nothing — one of the smallest models we tested beat models several times its size, and the largest model we tested was not the most accurate.
They failed in different places. This turned out to be the useful part. A model that got every single weight-based dose right would then botch every temperature conversion. Another was flawless on ratios and poor on doses. There was no "just pick the good one" available to us.
The obvious fix is to have a second AI check the first one's answer. We tried it properly, including with an open-source model built by researchers specifically for that job.
It didn't work. On our questions, the purpose-built checker caught fewer than one in five of the bad answers — and it missed the dropped-condition failures almost entirely. A general-purpose model did a little better at catching mistakes but raised so many false alarms on correct answers that it would have trained you to ignore the warnings. Asking a model to grade itself was worst of all: it approved everything.
The reason is worth understanding, because it explains the whole design. Those checkers are built to catch an answer that contradicts its source or makes something up. But "8 drops per gallon" doesn't contradict a document that says "6% bleach, 8 drops per gallon." It's perfectly faithful to the source. It's just incomplete — and incompleteness is exactly what hurts the person holding a different bottle.
Six checks that run on every Library answer. These are ordinary code, not AI — they can't be talked around, they can't have an off day, and they add nothing to how long an answer takes.
Every number in the answer is matched against the documents it cited. A figure that appears in none of them is flagged, whether it's written "8" or "eight".
Catches the bleach failure above — an answer that worked a figure out by converting units instead of quoting the one that's printed.
A dose is only right with its conditions. If the source states a strength, a water condition or a contact time and the answer leaves it out, you're told.
When two manuals give different figures and the answer quietly reports one, you get both — the disagreement is the thing you most need to know.
Ask two things and the library only covers one, and the answer will often address the covered half as though it were the whole. It quotes your own words back and tells you which part nothing covered.
Sometimes the AI says the library has nothing when the library plainly has something. If the search found related pages, you get them anyway — judge for yourself.
When a check trips you get a Check this notice above the answer, in plain English, saying what to look at and why. Above, not below — a warning printed under a dose is a warning read too late.
The checks catch the overwhelming majority of the bad answers in our testing, and they almost never fire on a good one — which matters just as much, because a warning that cries wolf is a warning you'll learn to skip.
But they are not a guarantee, and here is what they don't do:
No AI is going to be perfect — even online. That's worth saying plainly, because the assumption behind "well, ChatGPT would have got it right" is wrong. The big cloud models make these same mistakes: they convert doses that don't convert, and drop the condition that made the number correct. Being connected doesn't fix it. What being connected gets you is a confident paragraph with nothing to check it against.
So we could put a number here. We're not going to, and you should be a little suspicious of anyone who does.
Our test questions were written to be nasty — deliberately built to bait the exact mistakes described above. A model's score against them is much worse than its behaviour on ordinary questions, so publishing that figure would understate the drive. Quoting a friendlier number from an easier test would overstate it. Either way you'd be reading a number about a test you can't see, chosen by the people selling you the drive.
What we'll say instead is what's true: the AI gets things wrong. We tested hard to find out where, we ship the models that held up best, we check every Library answer for the failures we know about, and we tell you when something looks off. That is the honest shape of it.
It is not a replacement for a doctor, a pharmacist, an electrician, or your own judgement. If an answer contradicts what you already know — trust yourself.