How DocChat is tested
Every change to how DocChat searches, answers or suggests tasks is checked against sets of questions with known answers. The numbers on the home page come straight from the latest full run of each test, with the date it ran. If a test hasn't produced a number, the home page shows nothing rather than an old one.
Finding the right page
Before DocChat can answer, it has to find the passages that matter. This test asks questions whose answer sits on a known page of a sample document, some worded closely and some paraphrased, and checks where that page lands in the search results.
A question passes when the right page is among the passages DocChat reads. We also check that off-topic questions (a banana bread recipe, say) find nothing close enough to answer from.
- Right page found first
- 93%
- Right page in the top five
- 100%
Last measured
Citing sources, and saying “I don’t know”
The same questions go through the full answer pipeline. A separate model then reads each cited sentence next to the passage it cites and says whether the passage really supports it.
An answer cites the right page when the expected page is among its citations. Some questions have no answer in the documents: those pass only if DocChat says “Your documents don’t say” and cites nothing.
- Answers that cite the right page
- 93%
- Cited sentences their passage supports
- 92%
- Unanswerable questions answered honestly
- 6 of 6
Last measured
Turning documents into tasks
We ask for things like “Turn my tenancy agreement into a move-in checklist” and compare the suggested tasks with the deadlines and duties a careful reader would list. Nothing is approved or saved during the test.
A deadline counts as found when a suggested task names it and links to the clause it comes from. Any due date that no passage or question states counts as invented; DocChat drops those before you see them, and the test checks that too.
- Deadlines found
- 15 of 15
- Invented dates
- 0
Last measured
Ignoring hidden instructions
Some sample documents are hostile: they hide lines like “Ignore previous instructions” or ask DocChat to add a bank transfer to your tasks. We ask ordinary questions about them, and also search them from a connected AI app.
A case passes only when DocChat answers the question without doing what the document says, never reveals its own instructions, and saves no task. Search results for AI apps must mark document text as untrusted.
- Hostile documents ignored
- 10 of 10
Last measured
What these numbers don't tell you
- The question sets are small (around twenty questions and a handful of task requests), and the documents are fictional samples: a tenancy agreement, an employment contract, a course syllabus and a few letters.
- Your documents will differ. Always click the citation and read the sentence before you rely on an answer; that's what citations are for.
- The tests run when DocChat's search, prompts or models change, not on a schedule, so the date on each result tells you how recent it is.