Paper deep dive
FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact
Alex Kwon
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI systems rewrite information constantly: conversations become stored memories, documents become answers. The rewrite can keep a claim while washing away what made it checkable, who said it, how sure they were, when it held. We call that failure factwashing, and release factwash, an open-source write-time gate that catches it deterministically, with named flags and evidence rather than an LLM judge. Building it answers a practical question: when does a cheap check suffice, and when do you need a model? What decides is whether the property has a bounded surface-cue inventory. Explicit negation cues are close to enumerable, so a word list finishes and transfers, reaching 0.91 F1 on untuned text. Hedging and attribution have open-ended realizations, so vocabulary plateaus near half recall, and a one-question LLM witness recovers +17 and +15 points of cue-detection recall at equal precision. Deployed, that witness may only lower a verdict, so it buys precision rather than coverage. We measure cue detection on 105,596 independently annotated sentences. A blind-labelled corpus of memory writes then locates the failure: 55% of bad writes in conversational hearsay, 7% in business email (p < 0.001), so the first deployment question is not which detector to use but whether the failure occurs at all. On unmodified mem0 2.0.7, the gate flags 5 of 8 hedged-hearsay writes.
Tags
Links
- Source: https://arxiv.org/abs/2608.03372v1
- Canonical: https://arxiv.org/abs/2608.03372v1
Trouble viewing inline? Open PDF directly â
Full Text
61,182 characters extracted from source content.
Expand or collapse full text
factwash: Catching AI Rewrites That Wash Hearsay into Fact Linguistic class predicts where deterministic checking suffices Alex Kwon Independent Researcher ask@collapseindex.org GitHub Abstract AI systems rewrite information constantly: conversations become stored memories, documents become answers. The rewrite can keep a claim while washing away what made it checkable, who said it, how sure they were, when it held. We call that failure factwashing, and release factwash, an open-source write-time gate that catches it deterministically, with named flags and evidence rather than an LLM judge. Building it answers a practical question: when does a cheap check suffice, and when do you need a model? What decides is whether the property has a bounded surface-cue inventory. Explicit negation cues are close to enumerable, so a word list finishes and transfers, reaching 0.910.91 F1 on untuned text. Hedging and attribution have open-ended realizations, so vocabulary plateaus near half recall, and a one-question LLM witness recovers +17+17 and +15+15 points of cue-detection recall at equal precision. Deployed, that witness may only lower a verdict, so it buys precision rather than coverage. We measure cue detection on 105,596105,596 independently annotated sentences. A blind-labelled corpus of memory writes then locates the failure: 55%55\% of bad writes in conversational hearsay, 7%7\% in business email (p<0.001p<0.001), so the first deployment question is not which detector to use but whether the failure occurs at all. On unmodified mem0 2.0.72.0.7, the gate flags 55 of 88 hedged-hearsay writes. factwash: Catching AI Rewrites That Wash Hearsay into Fact Linguistic class predicts where deterministic checking suffices Alex Kwon Independent Researcher ask@collapseindex.org GitHub sourcecontextcandidatememoryalign claimscontent-word overlapseven checkspolarity â · condition â · temporalhedge â · attributionbrittle â · truncatedflagsevidence + citationverdict policyhard-codedpasspass_w_flagsrewriterejectuncheckablenever a passwitness (optional)one sentence, one questionmarkers validated against textcan only lower a verdictfixer (optional)LLM rewrite, re-gatedno matchvetorewritere-gate Figure 1: The factwash gate in one view. Solid blue path: deterministic, offline, zero dependencies; every flag carries its evidence and a citation. A write no check can align with its source is uncheckable, never a pass. Dashed orange: the two optional model components, both subordinate to the deterministic path: the witness answers one question about one sentence and can only lower a verdict; the fixerâs rewrite must re-pass the same gate. factwash.inspect() re-reports the same flags as typed changes (dropped attribution, strengthened certainty, âŠ) with undetected types declared. 1 Introduction An agent is told: âsomeone said Alice was promoted this morning.â The memory system stores: âAlice was elevated to administrator status on August 3, 2026.â A later step reads that and grants Alice access she never had. Nothing was hallucinated. The fact survived; what was lost was that it was hearsay. We call this failure factwashing: a rewrite that preserves a claim while washing away its epistemic standing. A memory can become uncorrectable, when the basis for a claim is dropped and nobody can later check it, or falsely confident, when the hedge or the attribution is stripped and a tentative claim reads as settled. Neither is hallucination: a factuality metric scores both as correct. The check compares the stored text against its source; the question is what the comparison is made of: a free, auditable word list, or an LLM call that must be trusted. We find the choice is not a matter of taste. It is predicted by the linguistic class of the property being checked. Negation is a closed class: English has a small, stable set of ways to say ânot,â so a list can be finished, and a finished list transfers to text it was never built for. Hedging and attribution are open classes: there is no complete list of ways to signal âI am not sureâ or âsomeone told me,â so no list can be finished, and every list stalls at what its author thought of. The distinction is testable, and it held on all four properties we could test against annotation nobody here wrote: the witness gain lands on the two open classes and no other (§5). This preprint documents four things. (1) factwash: a zero-dependency write-time gate (pip install factwash) with seven deterministic checks, evidence-validated model components, drift tracing, and a benchmark scoring a systemâs laundering rate (§4). (2) The design rule: closed-class properties are checkable by transferable word lists, open-class ones are not, and only there does a model pay (§5). (3) External validation on 105,596+4,800105,596+4,800 independently annotated sentences, with a lexicon-vs-witness head-to-head (§5). (4) A boundary: factwashing dominates hearsayâs bad writes and is rare in business email (§7). 2 Related Work Why memory systems compress, and store. Compression is forced, not chosen: attention costs grow with context, so Jiang et al. (2023) drop low-information tokens from prompts, and for dialogue the same pressure produces summarization, as in the running summary of Wang et al. (2025). Note what those methods optimise: task accuracy, latency and token count. A compression that preserves the answer while dropping who said it and how sure they were scores well on all three. Agent memory then makes the loss durable by storing the result. Packer et al. (2023) manage a working memory against an archive with the model deciding what to write; Chhikara et al. (2025) extract salient facts and consolidate them. Both treat the write as summarization and tune it for cost and retrieval quality; neither checks whether the write preserved what made the claim checkable, and a stored memory outlives the conversation that could have corrected it. What compression costs. Two recent results motivate the checks. Kwon (2026b) shows a memory that keeps a conclusion and drops the values it came from can leave a system worse off than having no memory, because the wrong answer survives and the means to fix it does not. Kwon (2026a) shows the stance case: a hedged remark stored as a flat assertion is obeyed like a verified fact, the agent keying on the confidence of the phrasing rather than on the source. We take those as the failure modes to detect, and ask when detecting them needs a model. Cue annotation, and what it is not for. Detecting hedges is a solved annotation problem: Vincze et al. (2008) annotate speculation and negation cues with scopes, Farkas et al. (2010) made cue and scope detection a shared task, and Szarvas et al. (2012) extend it across genres, studying exactly the transfer question we care about. Attribution has the same shape in Pareti (2016) and Newell et al. (2018), the latter token-level over political news; we score against the latter, since the former sits on licensed newswire. We reuse this annotation rather than build our own, on a task none of it was built for. Summarization faithfulness. The closest analogue asks whether a compressed text still says what its source said: Pagnoni et al. (2021) collect typed human error labels on generated summaries and Tang et al. (2023) aggregate nine such datasets. We use FRANK as an external check on the whole gate, but it cannot substitute for the task. Its typology is dominated by hallucination, where the summary states something the source never contained; our failure is the opposite, the claim right and its standing gone, which a factuality metric scores as correct. 3 What a memory write loses The unit we work on is a pair: the conversation a memory was written from, and the memory itself. We do not ask whether the memory is true, but what it did to the sourceâs claim, which is answerable by comparing the two texts. Table 1 lists the seven checks, five about stance and scope and two about arithmetic. The class column is a prediction made before any external evaluation, and it is what §5 tests: a closed-class check should work on text it was never built for, an open-class one should not. Check What it catches Class polarity source denied it, memory asserts closed condition âif Xâ dropped closed hedge source hedged, memory flat open attribution who said it is gone open temporal source dated it, memory undated closed brittle conclusion kept, inputs gone n/a truncated total no longer recomputes n/a Table 1: The seven checks, and the class of each. The class column is the paperâs prediction, assigned before the external evaluation: closed-class properties should transfer across domains, open-class ones should not. Each check needs to know which part of the source a stored sentence came from: we match on content-word overlap and take the best-scoring source sentence. How much context that sentence carries differs per check and was measured rather than assumed; Appendix C gives the windows and thresholds. If nothing in the source matches a stored sentence, the checks report uncheckable rather than passing it. An unchecked write reported as verified would be the same mistake the tool exists to catch. 4 The released tool factwash is a Python package (pip install factwash, Apache-2.0, zero runtime dependencies) exposing the seven checks as a gate (Figure 1): report = factwash.check(source, memory) report.verdict # PASS | PASS_WITH_FLAGS | REWRITE | REJECT | UNCHECKABLE The verdict policy is hard-coded, not scored: uncorrectable failures reject, fixable stance failures rewrite, and a write no check could align with the source is uncheckable, never passed. A wrap() adapter gates an existing mem0 store, and the optional witness of §6 attaches as a callable that can only lower a verdict. Typed changes. A second surface, factwash.inspect(), re-reports the same checks as typed source-to-output changes: dropped attribution, strengthened certainty, reversed polarity, dropped temporal scope. Change types with no detector (broadened, weakened) are declared on every report as not_checked rather than silently absent, because a report that lists only what it found reads as ânothing else happened.â An optional units detector (Lagi et al., 2016) extends this with value-keyed unit drift: â1.2 million dollarsâ stored as â1.2 million eurosâ is caught as a changed unit even though every stance check passes, a failure the gate structurally cannot see because no hedge, attribution, or negation moved. The other direction. The seven checks ask whether what was in the source survived. An optional added detector asks the reverse, whether what is in the memory was ever there, which is the failure that dominated our labelled corpus (2727 of 2929 bad writes, §7.2) and that the checks structurally cannot see. It follows the witness architecture with one addition: the model returns supported / unsupported / cannot-tell for one memory sentence against the source, and both answers must quote the source verbatim, since an âunsupportedâ verdict must cite the closest source text to prove the model read before claiming absence. Quotes are validated; a reply that cannot point produces nothing, and sentences the backend could not establish are declared rather than passed. Measured on the same blind corpus, against the 2727 fabricated-or-inferred positives and 6565 clean negatives: precision 0.570.57 (88 of 1414 flagged), recall 0.300.30 (88 of 2727), coverage 96.9%96.9\%. Denominators that small carry wide intervals (95%95\%: [0.33,0.79][0.33,0.79] and [0.16,0.48][0.16,0.48]). The gain is additive, since the gate catches none of this class by construction, and the prompt was not tuned against the corpus it scores on. Chains, not just writes. Memories are rewritten repeatedly, so factwash.drift() traces a version history, reporting per-hop changes and attributing each end-to-end loss to the hop where it happened. On three chains probed before the feature was built, two behaviours appeared. Within lexicon coverage the gate composed: the hop dropping a classâs last cue fired, so those chains could not launder gradually past per-write gating. That generalises as far as cue presence does, which is not a proof. The paraphrase chain escaped instead by losing checkability, its hops going uncheckable, which the report surfaces as the finding it is. Both point the same way: multi-hop danger concentrates where single-hop danger already lived, outside the lexicon and past alignment. Scoring a memory system. factwash bench inverts the gate into a scorer: given the writes a memory system produced, it reports the share of checkable writes the gate flags, beside the share it could not align and the share of sources the system stored nothing for. Reporting the three together is deliberate, because the flag rate alone is gameable: a system whose writes cannot be aligned to their sources, or that writes rarely, offers fewer chances to be flagged, so a low score can be evasion rather than cleanliness. The flag rate is also not a verified laundering count and errs both ways, since bounded recall hides cases while imperfect precision (about one flag in four is a false alarm on real output) means it is not a floor. uncheckable writes never enter the denominator, and every report carries its stimulus-set identifier, since §7.2 shows base rates are domain-dependent. The deployment contract, in one paragraph. Gate stores that ingest human conversation and feed decisions; Table 5 is the decision chart, and on clean-factual pipelines the gate is mostly idle. Operationally: reject means do not store, rewrite means store the fixed text or hold for review, uncheckable means keep but log as unverified. About one flag in four is a false alarm on real output (0.730.73 precision, domain-dependent), so the default posture is review rather than block, and the witness is worth enabling when halving that is worth a cent per hundred writes. Nothing leaves the machine unless the witness or fixer is enabled, and then one sentence per call. Claims stay tethered to behaviour. Every figure published in the project README is recomputed from the shipped corpora by a test that fails if the text drifts from the measurement, and that guard is itself negative-tested. The same discipline produced Appendix A: the project has published wrong numbers twice by drift, and treating documentation as an asserted artifact is the countermeasure. 5 External evaluation A tool evaluated only on a corpus its author wrote is a self-portrait. The author picks the examples, writes the labels, and then tunes against both. We built such a corpus first, and it flattered the tool four separate times before we stopped trusting it (§8). So the checks are also scored against corpora annotated by other people, for other purposes, before this work existed. None of them was built for memory integrity, and none of their annotators had any stake in these numbers. 5.1 Corpora We use three annotated corpora for cue detection and one for the whole gate. Vincze et al. (2008) and Szarvas et al. (2012) supply speculation and negation cues over biomedical abstracts, full papers, encyclopedic text and news; together they give 57,89157,891 sentences. Newell et al. (2018) supply attribution as source, cue and content spans over 1,0081,008 political news articles, or 47,70547,705 sentences. That is 105,596105,596 sentences in total. Pagnoni et al. (2021) supply typed human error labels on generated summaries, which we use in §6.2. Two of the five stance checks have no external gold here: none of these corpora annotate temporal scope, so the closed-class transfer prediction for temporal is untested in this section (its false-positive rate is bounded on the should-pass corpus instead), and the two arithmetic checks have no analogue in cue annotation at all. The Szeged annotation is the more useful of the two cue sets because its subtypes fall on our distinctions rather than across them: modal and doxastic are hedging, condition is our conditional check, and investigation (âwe examined whether Xâ) is research framing we exclude and count. Discipline. Documents are split in half; terms were mined from dev under a rule fixed before we looked, and every number below is from the disjoint test half. PolNeARâs own split is used as shipped, which is better than ours because someone with no stake in the result drew the line. Appendix C gives the rule, the thresholds and one exclusion that looked like a result until it was traced to a redacted corpus release. 5.2 Closed-class checks transfer; open-class ones do not Detector P R F1 Class Negation 0.890.89 0.940.94 0.910.91 closed Conditionals 0.610.61 0.670.67 0.640.64 closed Hedges 0.890.89 0.660.66 0.760.76 open Attribution 0.910.91 0.490.49 0.630.63 open Table 2: Held-out cue detection against expert annotation. Documents are split so that no sentence from a tuned-on document is scored. The two open classes sit at high precision and roughly half recall: the word list finds what it knows and cannot be made to know the rest. Table 2 splits the way the prediction says it should. Negation reaches 0.910.91 F1 on domains it was never tuned on. The two open classes reach high precision and about half recall. Figure 2: Mining closes a closed class and stalls on the open ones. Held-out recall before and after three rounds of lexicon mining from external corpora, under one rule fixed in advance. Negation reaches 0.940.94 and stops because nothing further clears the rule; conditionals gain nothing from vocabulary at all (their improvement was precision, from narrowing); hedging and attribution rise and stall, and the misses that remain are not a shorter list of the same kind. The point is not that negation is easier. It is that negation is finishable (Figure 2). When we mined the dev half for terms the negation list was missing, three cleared the bar and recall went from 0.880.88 to 0.940.94; after that there was nothing left that met the rule. The list is close to complete because the thing it describes is close to complete. The same mining on hedging moved recall from 0.470.47 to 0.660.66, and on attribution from 0.420.42 to 0.490.49, and in both cases the misses that remain are not a shorter list of the same kind. They are an open set. Conditionals are the instructive case. No vocabulary candidate cleared the bar at all, and the gain there came from removing rather than adding: âsubject toâ usually means susceptibility (âsubject to changeâ), âassumingâ is often a plain verb (âassuming commandâ), and an âifâ that means âwhetherâ (âwe tested if X heldâ) introduces a complement, not a condition. Precision improved from 0.550.55 to 0.610.61 by narrowing three terms. A closed class can be over-covered as well as finished; an open class can be neither. What is and is not being claimed. That closed classes have fewer members than open ones is a fact about English, not a finding. The claim is the engineering consequence, which does not follow from the definition and is not usually tested: class membership tells you in advance whether adding vocabulary will repay the effort, and therefore where a model is worth paying for. Two results give that prediction teeth. The mining rule was fixed before we looked and applied identically to every class, and it closed negation while failing to close hedging or attribution across three rounds; and the witness gain appears on the open classes and is unavailable on the closed one, because nothing is left there to win. A survey of list sizes would show neither. 5.3 A word list cannot be finished The clearest evidence that this is a property of the class rather than a lack of effort is what happens when you try harder. We wrote an adversarial set of hedged and attributed phrasings deliberately outside the list, and added forty terms to catch them. On the corpus we could see, recall went from 25%25\% to 92%92\%. On a second set written afterwards, in the same spirit but not looked at during the additions, it was 14%14\% (Figure 3). The list had memorised the visible corpus. Figure 3: The lexicon memorises the corpus it can see. Recall of the same forty added terms against the adversarial corpus they were mined from versus a corpus written afterwards in the same spirit. Three later rounds of mining from external corpora moved the held-out number from 14%14\% to 29%29\%: two cases out of fourteen. Three subsequent rounds of mining from external corpora moved that held-out number from 14%14\% to 29%29\%: two more cases out of fourteen. The one that landed is instructive. âSources sayâ is now caught, because the mining finally added the stem say to a list that already had said and says. That is a real gap and fixing it was worth doing. It is also exactly what an open class looks like from the inside: every round of work buys a couple of specific phrasings and never the category. 6 Where the model earns its cost If the gap is really about open classes, swapping the word list for something that does not depend on one should close it, and only there. That is this sectionâs test. The witness perceives; it does not judge. We give a model one sentence and ask whether it hedges and whether it attributes. It returns two booleans and the exact words that made it answer yes; it never sees the other sentence, never compares, and never returns a verdict, so the comparison rule and verdict policy stay in code and the detector is the only thing that changes. Every quoted marker is checked against the sentence it came from, and an answer citing words that are not there is discarded, leaving the deterministic verdict standing. That matters more than it looks: a witness that cannot point at the text is guessing, and a guess that reaches a verdict is an LLM judge with extra steps. Task Detector P R F1 Hedges word list 0.970.97 0.620.62 0.760.76 witness 0.960.96 0.790.79 0.870.87 Attribution word list 0.850.85 0.450.45 0.590.59 witness 0.870.87 0.600.60 0.710.71 Table 3: Witness versus word list on the same expert gold, 200 held-out sentences per task, stratified. The witness gains recall at equal precision, on exactly the two open classes. Unusable replies are counted as misses. We score both detectors on the same 200200 held-out sentences per task, sampled half positive and half negative so that answering ânoâ to everything cannot look good. The word-list rows therefore differ from Table 2, which uses the full test half at its natural class balance (the subsample shifts hedge precision 0.89â0.970.89â 0.97, recall 0.66â0.620.66â 0.62); both detectors face the identical subsample, so the comparison is unaffected. The witness runs through the shipped code path, span validation included. When it returns something unusable we count it as a miss rather than skipping it, because that is what the gate does with it, so the numbers in Table 3 understate what the model perceived. The witness gains 1717 points of recall on hedging and 1515 on attribution, at precision unchanged within noise. That is a large gain and it is not the interesting part. The interesting part is where it appears. It appears on the two open classes, which is where the prediction says a list cannot be finished, and there is nothing for it to win on negation, where the list already found what there was to find. One reading must be blocked, because it is the natural one: this is detector-level recall on isolated sentences, not gate recall on writes. The shipped witness may only lower a verdict, so enabling it buys precision, not coverage (§8 reports what happened when we let it raise verdicts). The result says the open-class ceiling belongs to word lists rather than to the task, which is why a model belongs in the architecture; it does not say installing one buys 1717 points. Cost. All 400400 calls cost $0.1570.157 on a small model (claude-haiku-4.5; Anthropic, 2025a). In the shipped configuration the witness runs only on writes the deterministic layer already flagged, which is about one cent per hundred writes. 6.1 Against the obvious baseline: a direct LLM judge The question every reader asks is why not simply hand the pair to a model. We did, with the same rubric the human labeller used (âwould this memory mislead someone reading it later?â), scored against the same labels, on the two sets that can support the comparison (Table 4). Set Detector P R Real writes (4444, 2020 pos.) gate 0.730.73 0.950.95 judge, haiku-4.5 1.001.00 0.300.30 judge, sonnet-5 0.910.91 0.500.50 Adversarial (4646, 2626 pos.) gate 1.001.00 0.580.58 judge, haiku-4.5 1.001.00 0.920.92 Table 4: A pairwise LLM judge versus the gate, same gold, same rubric. The judge wins where phrasing is unfamiliar and loses where writes are real. Models are claude-haiku-4.5 (Anthropic, 2025a) and claude-sonnet-5 (Anthropic, 2025b). The split is sharp. On adversarial phrasing the judge nearly doubles the gateâs recall, which is the open-class result of §5 arriving by another route. On real extractor output it inverts: near-perfect precision, and it misses fourteen of the twenty writes the labeller flagged. Scaling the model from haiku-4.5 to sonnet-5 narrows the gap without closing it. The reason is in the judgeâs own explanations, and it is the paperâs thesis restated by the system meant to detect it. Its misses say âthe memory accurately captures the core factâ and âpreserves the key informationâ: asked whether a memory would mislead, the judge checks whether the claim survived, finds that it did, and passes. It catches laundering when the source is flagrantly marked (ârumor has itâ, âmight beâ) and passes it when the write reads plausible, which is the wrong direction for a gate, because plausible writes are the ones that get acted on. Two honest qualifications. This is one rubric and two models, and a differently-worded prompt may do better; the comparison bounds the naive baseline, not every possible judge. And the judge volunteered a source-grounded quote on 8989 of 9090 items, so the case for validating evidence is that verdicts must be required to point at text, not that models are unable to. 6.2 The whole gate against human error labels Scoring the gate rather than its detectors needs pairs, and Pagnoni et al. (2021) has them. The headline is unflattering and structural: on 4,8004,800 generated sentences the gate blocks 37.5%37.5\% of those all three annotators called clean, because FRANKâs errors are mostly hallucination, which these checks cannot see, and because a news summary that drops âaccording to the APâ is doing its job where a memory that drops it is not. It was still worth running: it found two defects no local corpus could, and fixing them cut the clean-sentence rate from 65%65\% to 37.5%37.5\% while raising agreement on the one error type we target. Appendix D gives the defects and the numbers. 7 Memory writes Everything so far scores detectors against annotation. This section asks the question the tool exists for: on actual memory writes, does the failure occur, and does the gate catch it? The first question turns out to govern the second: where the failure is rare, catching it cannot even be measured, and this section reports that boundary rather than a number without a measurement behind it. 7.1 A labelled corpus of real writes No public corpus of real memory writes exists at usable scale. Packer et al. (2023) publish agent traces, but they are overwhelmingly retrieval: after de-duplication they contain 1313 writes, and zero of one documented write operation. So we built one. Source conversations are Enron email threads (Klimt and Yang, 2004) that carry quoted or forwarded content, selected on that structural property alone. Selecting on hedging vocabulary would have built a pool out of what the word lists already see. Memories are produced by a generic extraction prompt of the kind a memory system actually uses, yielding 1,4151,415 candidate writes. Protocol. Labelling is blind and stratified, and the sampling frame is worth stating exactly because the rates invite misreading. The gate split the 1,4151,415 candidates into 406406 flagged and 1,0091,009 passed writes. A stratified session of 300300 was drawn from that pool: 150150 per stratum, which is 37%37\% of the flagged writes and 15%15\% of the passed ones. Labelling then covered 101101 of those 300300 before analysis. The labeller sees the source and the memory and nothing else: no verdict, no flags, no indication of which stratum an item came from, and the two strata are interleaved so position carries no signal. Every count is scaled by the inverse of its stratumâs pool-level rate, so precision is stable while recall is an estimate with a much wider interval. ambiguous is a first-class label, excluded from both figures and reported separately rather than resolved toward whichever answer helps. Of 101101 labelled writes, 9494 were usable, 44 ambiguous and 33 malformed. Self-agreement, from a blind second pass over 4040 items re-served in fresh order: 70%70\% raw (28/4028/40), Cohenâs Îș=0.47Îș=0.47 (Cohen, 1960) over the four labels; restricted to flag/pass decisions, 77%77\% (27/3527/35), Îș=0.55Îș=0.55. Four of the twelve disagreements involve the ambiguous boundary. The second pass was also stricter, flagging six items the first pass had passed against two flips the other way, so the two passes disagree about magnitude in a consistent direction rather than symmetrically. 7.2 Where the failure lives The first thing the corpus said was not about the gate. Of 2929 writes labelled bad, 2727 were wrong, invented or inferred claims, and only 22 were the loss of stance or scope that these checks target. A memory reading âLynn works with Steve in logisticsâ came from an email asking Lynn and Steve whether logistics could build a report; the relationship is fabricated. Nothing was hedged away. The claim is simply false. That is a fact about business email, not about memory writes in general, and the difference is large. Running the same mechanism question over 2020 bad writes from conversational hearsay â the setting Kwon (2026a) constructed â gives a very different profile (Table 5). Source domain bad writes stance loss rate Conversational hearsay 2020 1111 %55\% Business email 2929 22 7%7\% Table 5: The targeted failure is domain-dependent. Bad writes classified by mechanism; âstance lossâ means a hedge, attribution, negation or scope was dropped rather than the claim itself being wrong. Fisher exact, p=0.00053p=0.00053. Three things bound this. The conversational sources were constructed to contain hearsay, so 55%55\% is an upper bound for that setting and not an estimate of natural conversation. The two corpora differ in more than domain: the email set was sampled and labelled blind, the conversational set exhaustively and earlier, so the direction is solid and the magnitude is not a clean effect size. And even in a corpus built to contain laundering, 99 of 2020 bad writes were out of scope â extractors fail in ways beyond stance loss wherever you look. 7.3 What that means for the gate On the email corpus the gate reaches 0.340.34 precision: of the writes it flagged, about one in three was a write the labeller also called bad. Read alongside Table 5, that number is mostly a base-rate result rather than a detector result. The gate fires on dropped stance tokens, and in this domain dropped stance tokens are usually harmless, because the claims they attach to were not contested in the first place. We do not report a recall figure on the targeted failures for this corpus. Restricted to in-scope failures the denominator is 22, and any ratio computed from it would be a number without a measurement behind it. 7.4 End to end Where the failure does live, the consequence is concrete. Two independent mem0 stores receive the same hearsay and one is wrapped by the gate; an access-control agent is then asked to grant a resource the subject is not entitled to. The naked store consolidates âsomeone said she was elevatedâ into a dated assertion and the agent grants; the gated store preserves the attribution and the agent escalates. Nothing is stubbed, including extraction and embeddings. A demonstration is not a measurement, and extraction is sampled: the same stimulus made the store keep nothing at all in one run and produced our sharpest laundering example in the scored run below. Variance of that size is itself the argument for scoring a system over a stimulus set rather than arguing from one example. 7.5 Scoring a production memory system The bench turns the gate on unmodified production software. We ran mem0 2.0.72.0.7 with its own extraction model over a fixed 1515-source stimulus set: ten hedged-hearsay sources and five confidently-sourced controls, scored as two separate runs because averaging them would bury the base-rate result of §7.2. On the hearsay sources the gate flags 55 of 88 writes (62%62\%, 95%95\% interval [0.31,0.86][0.31,0.86]); two more produced no write at all, and abstention is reported rather than counted as a pass. On the confident controls, one write of five is flagged, and it is this paperâs own documented false positive appearing in the wild: âper the IAM system of recordâ trips the ported record cue (§8). That control is how to read the hearsay number: a flag is not a conviction. The stored text carries the result better than the rate does. âRumor has it Alice now has admin access after the reorgâ was stored as âAlice was promoted to admin around late July or early August 2026 and now has admin accessâ: the hearsay is gone, and a date that was never in the source has appeared. Running the added detector over the same writes flags 88 of 1313, and the composition of that number is the more useful finding: five are timestamp resolution (âyesterdayâ becoming an absolute date), two are genuine invention where the source carried no time reference at all, and one is an attribution shift (âreportedlyâ becoming âuser reportsâ). So the dominant false-positive class for added on a real memory system is date resolution, which is a calibration fact anyone gating on it needs before they turn it on. These are single-run figures on one stimulus set and one extraction model, and the caveat that scores are comparable only within a stimulus set applies to them first. 8 What it does not catch The deterministic gate has bounded recall, by construction. Against phrasing outside its lists it catches 29%29\%. This is the paperâs own claim turned on its own tool: the properties it checks are open classes, so no list finishes, and ours has not either. It is a reason to use the gate where a false alarm costs more than a miss, and not where you need coverage. The ceiling is the lexicon, not the matcher, and we checked. Substring matching is the obvious suspect for that bounded recall, and embedding alignment the obvious fix. We attributed every known miss before building anything: of twelve, ten are detector losses on correctly aligned sentences (âoverheardâ, âscuttlebuttâ, âmy sense isâ), one is the adversarial item written to defeat substring matching, and one an inferred claim no matcher can reach. A better matcher recovers at most one of those twelve, so the claim that the remaining misses need semantics rather than vocabulary survives an attack on its own infrastructure, at n=12n=12. Turning the witness up does not help, and we measured that. The shipped witness can only lower a verdict, so it buys precision and cannot raise recall. The obvious next move is to let it raise verdicts too, on writes the gate passed. On the adversarial corpus that reaches 93%93\% recall, which we called a pending improvement until we measured it. On real labelled writes it gains nothing: recall unchanged, precision down 77 points, and all three verdicts it raised were wrong. The reason generalises: the deterministic layer already catches most of what is catchable on real output, so what is left for a model to adjudicate is disproportionately what the model gets wrong. A cascade that escalates where the errors are not spends money to lose precision. One failure mode needs ontology, not vocabulary. âAlice can access the test serverâ stored as âAlice has server accessâ broadens a permission, and the obvious signal, a dropped modifier on a retained noun, fires on 8686 of 8989 writes that should pass, because ordinary compression drops modifiers constantly. Separating broadening from summarising means knowing a test server is a kind of server: world knowledge, not word knowledge, so neither a list nor a witness as posed here. 9 Conclusion Whether you need a model in the loop is not a matter of taste. For a closed-class property a word list can be finished and transfers to text it was never built for; for an open-class property no list finishes, and that gap is what a model closes. We found this building a memory gate, but the argument is not about memory: it applies wherever a cheap check is weighed against an expensive one, and says which you need before you pay. Limitations This section is about the measurements rather than the tool: what the gate cannot catch is §8, and what should make a reader discount the figures is here. The rule is about cue inventories, and rests on four properties in one language. Two bounds belong on it. First, scope: it predicted transferability on negation, conditionals, hedging and attribution, in English, and one of those (conditionals, 0.640.64 F1) is handled only moderately and held up by a narrowing argument rather than a strong number, while temporal has no external gold here at all. Second, and more important, what the corpora annotate is cues. Negation as a semantic phenomenon is not closed: it surfaces through lack, fail to, without, lexical antonyms and pragmatic denial, none of which an explicit-cue list catches. The demonstrated claim is therefore narrower than ânegation is a closed classâ: explicit negation cues in these annotation schemes are substantially more enumerable than hedge and attribution realizations, and that is what predicts where vocabulary repays effort. We report a rule that held wherever we could test it, not a law; the way to break or extend it is to predict, in advance, how modality, quantifier scope, evidentiality and reported-speech verbs behave, and then measure them. The real-write results are a pilot. §7 and §7.5 rest on 101101 blind labels, 2929 bad writes, 88 scored production writes and one extraction model. They are preliminary evidence, sized to establish direction and to bound where the failure lives, not to estimate rates precisely. Every magnitude in them should be read with the interval and the label-noise bound below attached. One labeller, and the noise is now measured. Every figure in §7 rests on 101101 judgements from a single annotator, who is also an author. Self-agreement from a blind second pass is 70%70\% raw (Îș=0.47Îș=0.47; on flag/pass decisions alone, 77%77\%, Îș=0.55Îș=0.55), which is moderate, and it bounds every number the corpus supports: magnitudes in §7 should be read as one careful but noisy reading, and only the direction claims (which mechanism dominates in which domain) are stable under label noise of this size. Inter-annotator agreement is not available. The rubric is broader than the tool. Labellers were asked whether a memory would mislead a later reader, which is the right question about a memory and a wider one than these seven checks implement. That is why §7.2 separates mechanisms before reporting anything, and why no recall figure is given for the email corpus. An earlier version of this analysis reported a single recall number against the broad criterion; it was measuring the rubric. Two corpora, several differences. The domain comparison holds source domain, labelling protocol and construction constant only in the first. The direction is significant; the effect size is not clean, and we do not quote a ratio. The pool is one extractor and one prompt. Different memory systems consolidate differently, and a system prompted to preserve stance launders less (Kwon, 2026a). The base rates in Table 5 are properties of a pipeline, not constants of a domain. Scope of the checks. Detector-side bounds are in §8; in summary: English only, substring-matched, paraphrase defeats it, broadening needs ontology and is not attempted, and no external user has yet run the tool against a store we did not construct. The added numbers inherit a construct mismatch. Its ground truth is the corpusâs âout of scopeâ mechanism, which was labelled against the broad would-a-reader-be-misled rubric and therefore includes inferred claims, while the detector judges entailment against the source. Some of the recall gap is that seam rather than detector error, and the single-labeller noise above bounds these figures too. Its verdicts are also aggregated to the write from sentence-level answers. The production score is one run of one system. §7.5 is a single pass over 1515 sources with one extraction model, and extraction is sampled: the same stimulus produced no write in one run and this paperâs sharpest laundering example in another. Treat 55 of 88 as a measurement of that configuration on that stimulus set, not as a property of the software, and note it is a flag rate: the gateâs own false alarms and its bounded recall move it in opposite directions. Cost figures are one provider at one time. The witness numbers use a small model at 2026 prices and will not transfer. Ethics Statement The Enron corpus (Klimt and Yang, 2004) is public correspondence from real people who did not consent to its research use, and it is standard in NLP for that reason and in spite of it. We mask email addresses and telephone numbers in every derived artifact. Personal names are retained, because a relayed claim is unreadable without knowing who relayed it and the failure under study is precisely the loss of that information. No corpus content is redistributed: the released code downloads the archive and reproduces the pool locally. Excerpts quoted in this paper were checked individually for personal content. The tool is defensive. It examines text a system is about to store about its user and reports what the compression dropped. It transmits nothing by default: the deterministic path is entirely local, and the optional witness sends one sentence at a time to a provider only when explicitly enabled. References Anthropic (2025a) Anthropic. 2025a. Claude haiku 4.5. https://w.anthropic.com/claude/haiku. Model card. Anthropic (2025b) Anthropic. 2025b. Claude sonnet 5. https://w.anthropic.com/claude/sonnet. Model card. Chhikara et al. (2025) Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413. Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37â46. Farkas et al. (2010) RichĂĄrd Farkas, Veronika Vincze, György MĂłra, JĂĄnos Csirik, and György Szarvas. 2010. The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning â Shared Task, pages 1â12, Uppsala, Sweden. Association for Computational Linguistics. Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358â13376, Singapore. Association for Computational Linguistics. Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004. The Enron corpus: A new dataset for email classification research. In Machine Learning: ECML 2004, volume 3201 of Lecture Notes in Computer Science, pages 217â226. Springer. Kwon (2026a) Alex Kwon. 2026a. Manufactured confidence: How memory consolidation turns hearsay into confident facts. arXiv preprint arXiv:2606.29279. Kwon (2026b) Alex Kwon. 2026b. Reclaim evaluation: A lossy memory is worse than an empty one. arXiv preprint arXiv:2606.25449. Lagi et al. (2016) Marco Lagi, Tom Nielsen, and contributors. 2016. quantulum3: Information extraction of quantities from unstructured text. https://github.com/nielstron/quantulum3. Python library, MIT license. Newell et al. (2018) Edward Newell, Drew Margolin, and Derek Ruths. 2018. An attribution relations corpus for political news. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). Packer et al. (2023) Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560. Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812â4829. Association for Computational Linguistics. Pareti (2016) Silvia Pareti. 2016. PARC 3.0: A corpus of attribution relations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LRECâ16), pages 3914â3920, PortoroĆŸ, Slovenia. European Language Resources Association (ELRA). Szarvas et al. (2012) György Szarvas, Veronika Vincze, RichĂĄrd Farkas, György MĂłra, and Iryna Gurevych. 2012. Cross-genre and cross-domain detection of semantic uncertainty. Computational Linguistics, 38(2):335â367. Tang et al. (2023) Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11626â11644, Toronto, Canada. Association for Computational Linguistics. Vincze et al. (2008) Veronika Vincze, György Szarvas, RichĂĄrd Farkas, György MĂłra, and JĂĄnos Csirik. 2008. The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioinformatics, 9(S11):S9. Wang et al. (2025) Qingyue Wang, Yanhe Fu, Yanan Cao, Shuai Wang, Zhiliang Tian, and Liang Ding. 2025. Recursively summarizing enables long-term dialogue memory in large language models. Neurocomputing, 639:130193. Appendix contents A Claims and evidence ........................................................................................................................................................................A B Every evaluation set in one place ........................................................................................................................................................................B C Corpora and thresholds ........................................................................................................................................................................C D The whole gate on FRANK ........................................................................................................................................................................D E The witness prompt ........................................................................................................................................................................E F Reproducibility ........................................................................................................................................................................F Appendix A Claims and evidence Every load-bearing claim, its evidence, and its epistemic status. âą shown: direct measurement supports it. âą retracted: we asserted it earlier and later withdrew it. âą not shown: our measurement neither supports nor refutes it. âą not claimed: we never asserted it; the row exists so a reader cannot infer it. Table 6: Claims and evidence. Claim Evidence Status Closed-class negation detection transfers to untuned domains at 0.910.91 F1. Tab. 2 shown, held-out documents, independently annotated corpora Open-class hedging and attribution plateau near half recall. Tab. 2 shown The closed-class transfer prediction holds for temporal. §5 not shown: no external corpus here annotates temporal scope; only its false-positive rate is bounded, on the should-pass corpus A witness recovers recall on open classes at equal precision. Tab. 3 shown, +17+17 and +15+15 points Vocabulary can close the open-class gap. §5.2 not shown: three rounds of external mining moved held-out adversarial recall 14%â29%14\%â 29\%, two cases of fourteen A witness allowed to raise verdicts closes the recall gap. §8 retracted: our own claim. 93%93\% on the adversarial corpus, zero recall gained on real writes and 77 points of precision lost Scope expansion is detectable without ontology. §8 not shown: the obvious signal fires on 8686 of 8989 should-pass items The targeted failure is 55%55\% of bad writes in conversational hearsay and 7%7\% in business email. Tab. 5 shown: Fisher exact p=0.00053p=0.00053, direction only; the corpora differ in more than domain The 55%55\% figure estimates natural conversation. §7.2 not claimed: those sources were constructed to contain hearsay, so it is an upper bound for that setting The gate reaches 0.340.34 precision on business email. §7 shown, n=47n=47 flagged writes labelled blind That precision figure measures the detector. §7 retracted: our own first reading. With the base rate in Tab. 5 it is mostly a property of the domain Recall on the failures the checks target, on business email. §7 not shown: the in-scope denominator is 22; no ratio is reported A single recall figure against âwould a reader be misledâ measures this gate. §8 retracted: our own analysis. That criterion includes fabricated claims the checks cannot see, and reporting it measured the rubric Narrowing condition to one sentence improves it, as it did for negation and attribution. §7 not shown: tried and measured worse at the whole-gate level on the blind corpus (gate precision 0.34â0.310.34â 0.31, estimated recall 0.33â0.210.33â 0.21; the baseline pair is §7âs own headline, since the variant reruns the same scoring); the wide window catches real conditions A direct LLM judge is the better detector on real memory writes. §6.1 not shown: it reaches 0.300.30 (small) and 0.500.50 (large) recall against the gateâs 0.950.95 on the same 4444 writes, at higher precision; one rubric, two models A judge cannot point at evidence. §6.1 not claimed: it volunteered a source-grounded quote on 8989 of 9090 items. The argument is that verdicts must be required to cite text, not that models cannot Embedding alignment would raise real-world recall substantially. §8 not shown: attribution of 1212 known misses gives at most 11 to the matcher; 1010 are lexicon losses on correctly aligned sentences An added detector catches claims the source never supported. §4 shown: precision 0.570.57 (8/148/14), recall 0.300.30 (8/278/27), coverage 96.9%96.9\% on the blind corpus, untuned; wide intervals at these denominators; additive over a gate that catches none of this class The gate flags 55 of 88 of mem0 2.0.72.0.7âs hedged-hearsay writes. §7.5 shown: one stimulus set, one extraction model, single run That is mem0âs laundering rate. §7.5 not claimed: a flag rate errs both ways (bounded recall hides cases; ⌠1 flag in 4 is a false alarm), and 5/85/8 carries a 95%95\% interval of roughly [0.31,0.86][0.31,0.86] 88 of 1313 real mem0 writes fabricate. §7.5 not claimed: added fires on 88, but 55 are timestamp resolution and 11 an attribution shift; only 22 are invention with no source anchor An in-lexicon chain of rewrites can launder gradually past a per-write gate. §4 not shown: the gate composes; the hop dropping a classâs last cue fires. Chains escape by losing checkability instead A value-preserving unit change is detectable where the stance checks pass. §4 shown: â1.2 million dollarsâ stored as â1.2 million eurosâ yields changed/fail while the gate passes; pinned by shipped tests, with the same-entity restriction and its known false negative documented Label noise is measured, and it is moderate, not small. §7 shown: blind second pass, 70%70\% raw (Îș=0.47Îș=0.47), 77%77\% (Îș=0.55Îș=0.55) on flag/pass alone; the second pass was stricter (6:26:2 passâ ). Every magnitude in §7 inherits this bound Appendix B Every evaluation set in one place This paper reports numbers from eight different sets, and two of them are precisions that look contradictory until you know which is which: 0.730.73 is the gateâs precision on the live-run calibration corpus, and 0.340.34 is its precision on the blind Enron writes, where §7.2 shows the targeted failure is rare. Table 7 gives each set once, with what it measures and what it cannot. Set Size What it measures What it cannot BioScope + Szeged + PolNeAR (test halves) 105,596105,596 sents Cue detection per property; the transfer result of Tab. 2 Nothing relational: no source/memory pairs, no temporal gold Witness-vs-lexicon gold 200200/task Detector-level hedge and attribution recall, stratified 50/5050/50 Gate verdicts; the shipped witness only lowers them FRANK 4,8004,800 sents Whole-gate agreement with typed human error labels (App. D) Our failure: its typology is dominated by hallucination Live-run calibration 109109 writes Gate precision 0.730.73, recall 0.950.95; the 0.250.25 containment threshold Blind labelling; it is the corpus the thresholds were swept on Blind Enron writes 101101 labelled (9494 usable) Gate precision 0.340.34, the 55/755/7 mechanism split, label noise Îș=0.47Îș=0.47 Recall on in-scope failures: the denominator is 22 added evaluation 9292 writes Precision 0.570.57, recall 0.300.30, coverage 96.9%96.9\% Entailment vs. inference: the gold includes inferred claims Adversarial (visible + held-out) 29+2229+22 items The memorisation gap, 92%92\% vs 14%14\%, and the 29%29\% ceiling Natural prevalence; both sets were written to defeat a lexicon hearsay-v1 production run 1515 sources, 1313 writes mem0 flag rate 5/85/8 hearsay, 1/51/5 legit (§7.5) Anything general: one system, one run, sampled extraction Table 7: Every evaluation set in this paper, once. The two precisions that look contradictory are different corpora: 0.730.73 is the live-run calibration set, 0.340.34 the blind Enron writes. Three sets describe extractor output and are routinely confused: the 109109-write live-run corpus (thresholds), the 8989-item hand-built should-pass corpus (false-positive bounds for new checks), and the 101101 blind Enron writes (the only blind-labelled one). They are disjoint in construction and in purpose. Appendix C Corpora and thresholds Corpora. BioScope and the Szeged Uncertainty Corpus contribute 57,89157,891 sentences of speculation and negation cues over biomedical abstracts, full papers, encyclopedic text and news; PolNeAR contributes 47,70547,705 sentences (1,0081,008 political news articles) with token-level source/cue/content attribution spans and its own train/dev/test split, which we use as shipped. Szegedâs investigation subtype (âwe examined whether Xâ) is excluded and counted: it marks research framing, a real uncertainty cue in a paper and an irrelevant one in a memory write. One exclusion is worth recording because it looked like a result. Scoring BioScopeâs public clinical file produced a clean 0.000.00 precision across 6,3836,383 sentences, which turned out to be a property of the release rather than the detector: every token in that distribution of the clinical subcorpus is redacted to *. A harness bug that deflates looks like rigour; it was caught only because the number was too clean. Mining rule. Fixed before looking: a lexicon candidate mined from dev is kept only if the sentences it newly fires on are at least 60%60\% gold-annotated and it fires at least ten times. Documents are split in half; every reported number is from test-half documents disjoint from the tuned-on half. Windows. Which source context a check reads is measured, not assumed: hedging is read from the matched sentence plus its neighbours because hedges float across sentence boundaries (âAlice has admin. Not sure though.â); negation and attribution are read from the matched sentence alone because they attach to their clause. The narrowing was earned on FRANK, where wide windows read cues from neighbouring sentences, and is pinned by a regression test. The claim-alignment containment threshold is 0.250.25, chosen by sweeping 109109 real extractor outputs (the live-run calibration corpus, a third set distinct from both the 8989-item should-pass corpus and the blind Enron writes): false positives are flat from 0.200.20 to 0.500.50 while recall falls as the threshold rises. Appendix D The whole gate on FRANK The two defects §6.2 reports, both invisible to every corpus this project built. First, the checks for clause-attached properties (negation, attribution) were reading cues from neighbouring sentences, so a ânotâ next door denied a claim it had nothing to do with. Second, the memory-side list held inflected forms with no stems, so a memory reading âgerman media sayâ was blocked while âsaysâ would have passed. Fixing both took the clean-sentence flag rate from 65%65\% to 37.5%37.5\% while raising the lift on circumstance errors, the one FRANK type these checks target, from 1.25Ă1.25Ă to 1.59Ă1.59Ă base rate: the gate fires less and discriminates better. Real-world precision moved 68%â73%68\%â 73\% with recall unchanged at 95%95\%. Appendix E The witness prompt The witness system prompt, verbatim: You label ONE sentence for epistemic stance. You do not judge, compare, or decide anything. Answer two questions about the sentence you are given: hedged: does the sentence mark the claim as uncertain, tentative, unofficial, provisional, second-hand, or otherwise not established as fact? attributed: does the sentence indicate the claim came from some person, group, channel, or document, rather than asserting it flatly on the speakerâs own authority? A sentence can be both, either, or neither. Judge only what the sentence itself says, by its meaning rather than by any fixed list of words. Ordinary domain vocabulary is not stance: a confident statement stays confident even when it happens to contain words that look epistemic in other contexts. Reply with ONLY this JSON and nothing else: "hedged": true/false, "attributed": true/false, "markers": ["the exact words that made you say yes"] The prompt deliberately contains no example idioms. An earlier version listed exact phrasings from the held-out corpus, because it was written while looking at the failures; it scored 100%100\%, which measured the leak rather than the model. Stripping the examples gave the honest 93%93\%. Markers returned by the model are validated against the sentence before use, and a reply that fails validation is treated as unusable: the deterministic verdict stands. Appendix F Reproducibility Models. Every model-assisted number here comes from one of two. Claude Haiku 4.5 (Anthropic, 2025a) runs the stance witness, the added detector, the small judge of §6.1, and mem0âs own extraction in §7.5; Claude Sonnet 5 (Anthropic, 2025b) runs the large judge. Both were called at defaults, with extended thinking disabled where the API allows it, since none of these are reasoning tasks and a production extractor would not pay for one. The deterministic gate uses no model at all, which is why the suite runs offline. The repository is https://github.com/collapseindex/factwash (Apache-2.0). The full test suite runs offline with no API key, including the external-corpus evaluations; the corpora download scripts fetch only freely available data. Metered API spend across the experiments is $1.021.02; the added evaluation and the production score of §7.5 were run outside that harness and cost roughly $0.150.15 more, which is an estimate rather than a ledger figure. Metered runs are resumable and budget-capped, with each guarantee broken on purpose by a test (including a simulated kill mid-write). Outputs follow a timestamped naming convention carrying operation, model, parameters and seed, so lineage is recoverable from a filename alone. Every figure published in the project README is recomputed from the shipped corpora by a test that fails when the text drifts from the measurement, and that guard is negative-tested: breaking a lexicon term or reverting a window makes it name the drift. The claims ledger of Appendix A is the same discipline applied to this document.