Paper deep dive

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs

Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, Srinivasan Iyer

Year: 2021Venue: arXiv preprintArea: Model EditingType: EmpiricalEmbeddings: 80

Models: BART-base, RoBERTa-base

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 7:48:52 PM

Summary

This paper investigates whether language models (LMs) possess beliefs, proposing methods to detect, update, and visualize these beliefs. The authors introduce the SLAG (Sequential, Local, and Generalizing) training objective to improve learned optimizers for model editing, and they propose the 'belief graph' as a tool to visualize interdependencies between model beliefs. Experiments demonstrate that while LMs exhibit limited belief-like qualities, the SLAG objective significantly improves the consistency and success rate of belief updates compared to existing baselines.

Entities (5)

Language Models · technology · 100%Daniel Dennett · person · 99%SLAG · methodology · 98%Belief Graph · interface · 95%KnowledgeEditor · algorithm · 95%

Relation Signals (3)

SLAG → improves → KnowledgeEditor

confidence 95% · We propose a Sequential, Local, and Generalizing belief update objective (SLAG) that substantially improves the performance of the KNOWLEDGEEDITOR method

Belief Graph → visualizes → Language Models

confidence 90% · the introduction of the belief graph, which is a new form of interface with language models

Language Models → exhibit → Belief-like qualities

confidence 85% · Our experiments suggest that models possess belief-like qualities to only a limited extent

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks. Our main contributions include: (1) new metrics for evaluating belief-updating methods that focus on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing model updates (SLAG) that improves the performance of learned optimizers, and (3) the introduction of the belief graph, which is a new form of interface with language models that shows the interdependencies between model beliefs. Our experiments suggest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work. Code is available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

79,466 characters extracted from source content.

Expand or collapse full text

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs Peter Hase 1,2 Mona Diab 1 Asli Celikyilmaz 1 Xian Li 1 Zornitsa Kozareva 1 Veselin Stoyanov 1 Mohit Bansal 2 Srinivasan Iyer 1 1 Meta AI 2 UNC Chapel Hill peter, mbansal@cs.unc.edu mdiab, aslic, xianl, zori, ves, sviyer@fb.com Abstract Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decou- pled from any motivational state. In this paper, we discuss approaches to detecting when mod- els have beliefs about the world, and we im- prove on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks. Our main contributions include: (1) new met- rics for evaluating belief-updating methods that focus on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing model updates (SLAG) that improves the performance of learned optimiz- ers, and (3) the introduction of thebelief graph, which is a new form of interface with language models that shows the interdependencies be- tween model beliefs. Our experiments sug- gest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the- shelf optimizers are surprisingly strong belief- updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work. 1 1 Introduction Language models (LMs) may not have beliefs in the same sense that people do, but there are a few reasons to analyze LMs in terms of the beliefs they may possess. The first is that this is a useful way to understand and speak about how LMs behave. When discussing whether animals have beliefs (rac- coons, in particular), philosopher Daniel Dennett (1995) writes: You might as well call the state of the raccoon a belief, since if you call it a “registration” or 1 All supporting code for experiments in this pa- per is publicly available athttps://github.com/ peterbhase/SLAG-Belief-Updating. a “data-structure” in the “environmental infor- mation store” or some other technical term, the logic you use to draw inferences about the ani- mal’s behavior, given its internal states, will be the standard, “intentionalistic” logic of belief. Dennett bases this conclusion in the fact that we can and do draw accurate inferences about animal behavior by first understanding their beliefs. We are drawn to speak about the beliefs of LMs in the same “maximally bland (but maximally useful!)” sense. To the extent that these neural networks act intelligently in response to stimuli, we may form more accurate theories of how they work by understanding their beliefs. The second reason for ascribing beliefs to lan- guage models is that many of the stricter definitions of belief incidentally exclude many real beliefs held by real people. Following Dennett (1995), Newen and Starzak (2020) define a belief asan informa- tional state decoupled from any motivational state, and they outline a few additional properties of be- liefs, namely that they should (1) be recombinable with motivational states and other informational states and (2) have some minimal structural orga- nization. Further, an entity with beliefs should (1) be sensitive to new information, (2) categorize new beliefs as they develop, and (3) display some kind of logical consistency. These are all proper- ties that come in degrees, and setting the bar too high will exclude many of the statements that peo- ple earnestly express to others in their everyday lives. Meanwhile, animals and neural networks alike store information in accordance with these properties to at least some extent. We also note that we use the termbeliefrather thanknowledgeas in related work (Zhu et al., 2020; De Cao et al., 2021) because we want to analyze beliefsoflanguage models rather than knowledge inthem. LMs may contain a great deal of knowl- edgeto us, but in a traditional view of knowledge as Justified True Belief, it is relatively more diffi- arXiv:2111.13654v1 [cs.CL] 26 Nov 2021 cult to say that an LM knows something rather than that it believes it (Schwitzgebel, 2019). In the remainder of this paper, we turn our at- tention to three practical endeavors:detecting,up- dating, andvisualizingbeliefs in LMs. We build on work on editing models after training, which is an exciting recent direction of research with many potentially valuable use cases (Sinitsin et al., 2020; Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2021). For LMs, uses include correcting factually inaccurate outputs and preventing otherwise un- wanted outputs from models (e.g. toxic generated text) without expensive data curation and retraining efforts. These are important applications given that LMs (1) struggle with future data when trained on data from the past (Lazaridou et al., 2021; Dhingra et al., 2021), (2) generate morally undesirable text in many situations (Gehman et al., 2020; Bender et al., 2021), and (3) simply give inaccurate outputs for tasks like question answering (Lin et al., 2021). Notably, there is good evidence that scaling models to larger sizes will not fix these particular problems or may even exacerbate them in cases like imitative falsehoods in QA, so we will likely need an alterna- tive solution (Lazaridou et al., 2021; Gehman et al., 2020; Lin et al., 2021). We next outline a few key contributions of the paper. Figure 1 represents the core ideas behind these contributions. Detecting beliefs.We measure the degree to which LMs exhibit several properties of belief-possessing systems, using models finetuned on fact verifica- tion and question answering tasks. Beyond simply checking individual model responses, we want to assess the structural properties of model outputs: Are they consistent under paraphrase? Are they logically consistent? Does changing one belief correctly change other entailed beliefs? Does it erroneously change other unrelated beliefs? Past work has focused primarily on consistency under paraphrase (Elazar et al., 2021; De Cao et al., 2021; Mitchell et al., 2021). Here, we adapt data from Talmor et al. (2020) to measure consistency under entailment (including for contrapositives), and we use the Wikidata5m dataset (Wang et al., 2021b) to construct logically neutral belief pairs for checking that models do treat the beliefs as independent. Updating beliefs. We propose a Sequential, Local, and Generalizing belief update objective (SLAG) that substantially improves the performance of the KNOWLEDGEEDITORmethod from De Cao et al. (2021).KNOWLEDGEEDITORis a learned op- timizer that edits a model’s weights in order to change its prediction on an input while satisfying other desiderata, like consistency under paraphrase. Principally, we use more difficult training data for the learned optimizer, and we also train the net- work to apply multiple small edits rather than just one edit. These changes markedly improve the overall update success rate and lower the rate at which other beliefs are corrupted. Moreover, we find thatKNOWLEDGEEDITORalmost totally fails when updating multiple beliefs in a row as opposed to a changing a single belief. In this setting, off- the-shelf optimizers are far preferable methods. However, by explicitly training the optimizer to update multiple beliefs sequentially, we are able to once again outperform off-the-shelf optimizers. Lastly, we advocate that these methods be eval- uated for their ability to correct false or morally undesirable model beliefs, rather than to arbitrarily adjust model beliefs to plausible alternatives as in past work (Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2021). Visualizing belief graphs.We explore a new form of interface with LMs, thebelief graph. Given a set of beliefs, we construct belief graphs by chang- ing each model belief and checking what other beliefs are sensitive to those changes. Each belief becomes a node, and directed edges between nodes show that updating one belief changes the other. We discuss graph metrics that help summarize the dependencies between model beliefs. We summarize our main conclusions as follows: 1.∼100M parameter models exhibit limited belief- like qualities, as paraphrase consistency scores are under 70%, and models show mixed levels of consistency under entailment (Sec. 5.1). 2.Off-the-shelf optimizers are surprisingly effec- tive baselines for updating model beliefs, and they generally outperform learned optimizers when updating a single belief (Sec. 5.2). 3.When updating multiple beliefs in a row, method performance greatly declines (especially for learned optimizers). By using SLAG, we can improve learned optimizers’ performance beyond what baselines can reach (Sec. 5.2). 4.Belief graphs reveal many nonsensical depen- dencies between model beliefs. We find that (1) updates are mostly likely to change already incorrect model beliefs and (2) there are highly connected beliefs which influence a large frac- tion of all model beliefs (Sec. 6.3). SLAG: Sequential, Local, and Generalizing Model Updates (Main Input) (Entailed Data) (Local Neutral Data) (Paraphase Data) (Random Data) A viper is a vertebrate. A viper has a brain. A viper is venemous. Chile is a country. Vipers are vertebrates. Figure 1: Relying only on a single Main InputM i , we want to make a targeted update to a language model that (1) changes the output for inputM i to a desired labely ∗ i (e.g. True/False, or an answer to a question), (2) changes the output for equivalent paraphrases ofM i , (3) appropriately changes outputs for data entailed by the tuple (M i ,y ∗ i ), and (4) doesnotchange outputs for other logically neutral data, even if it is similar (local) in some way. 2 Related Work Detecting beliefs in language models. Much past work has explored how information is stored and represented in pretrained language models (Rogers et al., 2020), though few discuss what qual- ifies information as a model belief. Petroni et al. (2019) provide evidence that LMs store relational information between entities, and Roberts et al. (2020) show that LMs can answer open-ended ques- tions. Subsequent work has explored how much knowledge is stored in LMs (Heinzerling and Inui, 2021), approaches to querying models for knowl- edge (Hewitt and Liang, 2019; Jiang et al., 2020; Voita and Titov, 2020; West et al., 2021), and meth- ods for learning more knowledge during pretraining (Wang et al., 2021b,a). Most relevant to our work are studies from Talmor et al. (2020) and Elazar et al. (2021). Talmor et al. (2020) train LMs to perform True/False classification of factual claims, and they measure how a model’s belief in one fact correlates with its belief in an entailed fact. We use their LeapOfThought dataset to measure model consistency under entailment before and after up- dating the up-stream beliefs in models. Meanwhile, Elazar et al. (2021) measure the consistency of model predictions for sets of paraphrased inputs. We adopt their metric for paraphrase consistency as a measure of belief. In concurrent work, Kassner et al. (2021) discuss consistency under entailment and paraphrase as conditions for belief, and they measure consistency under entailment with a new dataset, BeliefBank. Updating beliefs in language models.Ap- proaches to making targeted updates to model be- liefs vary along a few dimensions. First is whether the methods alter model training or operate in a post-training setting. Sinitsin et al. (2020) use a meta-learning objective during training to encour- age ease of editing afterwards, though the memory requirements of their approach limit its scalability beyond 100M parameter models. A larger family of methods make post-training updates to models, differing in how they formalize the update prob- lem: Dai et al. (2021) propose a hand-crafted algo- rithm for updating model weights, while Zhu et al. (2020) use projected gradient descent for batches of points. De Cao et al. (2021) and Mitchell et al. (2021) frame the problem as a machine learning problem and train hypernetworks (learned optimiz- ers) that process model gradients in order to pro- duce a new model that (1) gives the desired output for the edited point, while (2) incorporating other objectives like minimizing the changes in predic- tions for other data. Here, we build directly upon the method from De Cao et al. (2021), showing where it fails and providing an improved training objective (SLAG). In particular, we find that the method struggles with updating multiple beliefs sequentially. This setting bears some commonality to the continual learning problem, though continual learning methods generally aim to learn new tasks or datasets rather than make targeted updates to specific model beliefs (Parisi et al., 2019). Not all methods edit model weights. Kassner et al. (2021) update model beliefs by adding in relevant information to the input at test time (to im- prove consistency under entailment). But as with retrieval-based methods, this approach does not change the model weights and hence does not influ- ence model outputs on all other potentially relevant inputs (Lewis et al., 2020; Hase and Bansal, 2021). DatasetData TypeInputLabel(s) zsRE Main InputPlayer Ali Kanaan plays for what team? Sporting Al Riyadi Beirut ParaphraseWhat team is Ali Kanaan associated with? Wikidata5m Main InputMary Good has relation ‘award received’ toGarvan-Olin Medal; Arkansas Women’s Hall of Fame; etc.ParaphraseMary Lowe Good has relation ‘winner of’ to Local NeutralMary Good has relation ‘educated at’ toThe University of Arkansas; U Arkansas; etc. FEVER Main InputTardigrades are also known as space bears.True Main InputThe Lion belongs to the genus Vulpes.False LeapOfThought Main InputA viper is a vertebrate.True Entailed DataA viper has a brain.True Table 1: Example datapoint from each dataset, and auxiliary data that accompanies the Main Input. We catalogue examples of noise and other shortcomings for each dataset in Appendix C. 3 Updating Beliefs in Language Models Following De Cao et al. (2021), we approach the problem of updating model beliefs as a machine learning problem and train a learned optimizer to perform desired model updates. We discuss metrics for detecting beliefs in Sec. 5.1 and our approach to visualizing belief graphs in Sec. 6.3. The core ideas of our approach are outlined in Fig. 1. Problem statement and metrics. We suppose we have a modelf θ =p θ (y|x)parametrized by θ. For an inputx i that has some undesired model outputˆy i = arg max y p θ (y|x), we wish to obtain a new modelθ ∗ that produces a desired outputy ∗ i forx i . This new modelθ ∗ should also fulfill a few other desiderata. As in past work (De Cao et al., 2021; Mitchell et al., 2021), we operational- ize these desiderata in the following metrics: 1. Update Success Rate (Main Input): The rate at which the updated model gives the desired outputy ∗ i for the Main Inputx i . 2.Update Success Rate (Paraphrase): The rate at which the updated model gives the same new prediction forx i as it does for para- phrases ofx i , which are inputs with the same meaning but different surface form. 3.Retain Rate (All Data): The rate at which the updated model’s predictions are unchanged for all other data besides the Main Input. 4.∆-Acc (All Data): The change in accuracy for the updated model on all other data besides the Main Input. In practice, Retain Rate (All Data) and∆-Acc are computed with random subsets of a dataset, since these must be computed after every belief update. We add two metrics to those used in past work: 5.Update Success Rate (Entailed Data): The rate at which the updated model makes predic- tions that are logically entailed by the model’s prediction for the Main Input. 6.Retain Rate (Local Neutral): The rate at which the updated model’s predictions are un- changed for data that is similar to the Main Input but still logically neutral. We use Update Success Rate (Entailed Data) to measure logical consistency for an updated model, since changing one belief will entail changes in logically entailed beliefs. We also split “retain accuracy" into two cases, one for randomly sam- pled data as in past work (All Data) and the other for specially constructedLocal Neutraldata. Un- like randomly sampled data, Local Neutral data is guaranteed to be logically independent of the Main Input, while still being similar (local) to it. To- gether, these six metrics better cover the criteria for belief outlined by Newen and Starzak (2020). We compute the metrics using data of the kind shown in Table 1. For a glossary of terms used for these metrics across papers, see Appendix Table 13. Evaluation data. To date, methods have been evaluated on the basis of their ability to change model predictions for all data points, including correctly and incorrectly predicted points. More- over, the desired labelsy ∗ i n i=1 on sequence pre- diction tasks have each been selected from the beam search which produced the original model prediction (De Cao et al., 2021; Mitchell et al., 2021). We propose for method evaluation to focus on a more valuable use case: changing the predic- tions on incorrect points to be correct. In Sec. 5, we show that this is a harder task than simply changing predictions to other similar outputs, so the effec- tiveness of past methods has been overestimated. Sequential updates.The default evaluation pro- cedure in past work on learned optimizers is to up- date a single model belief, evaluate the new model, then rollback the update before repeating the pro- cess for each test point. In Sec. 5, we show that it is much harder to update multiple beliefs in a row before evaluating the new model. This is notable because in practice, it is likely that model develop- ers will want to update many beliefs of a trained model, possibly over long timescales, meaning se- quential updating is a more realistic application of update methods. We obtain sequential versions of all our metrics by applyingrmodel updates in a row before checking the metrics, meaning there are floor(n/r)measurements for a test set ofnpoints. Belief updating method.We use theKNOWL- EDGEEDITORarchitecture from De Cao et al. (2021) with our training objective, SLAG. For the details of this architecture, we refer readers to Ap- pendix A. Let it suffice for now to observe that a new model is given as a differentiable function θ ∗ =θ+g φ (x i ,ˆy i ,y ∗ i ,θ) using the learned optimizerg φ , current LM weights θ, Main Inputx i , current predictionˆy i , and desired model outputy ∗ i . In this paper, we generalize the update step to occur in a loop. If we package the above update asθ (k+1) =θ (k) +g φ (x i ,ˆy i ,y ∗ i ,θ (k) ), then we can obtain new model parameters as θ ∗ =θ (k) + K−1 ∑ j=0 g φ (x i ,ˆy i ,y ∗ i ,θ (k+j) ) =Update(x i ,ˆy i ,y ∗ i ,θ (k) ;φ,K) for a number of stepsKfrom the initial parameters θ (k) . In fact, De Cao et al. (2021) use such a loop at test time; we incorporate the loop into training to align the train and test-time distributions. Learned optimizer training.The training objec- tive forKNOWLEDGEEDITORincludes differen- tiable terms corresponding to Update Success for the Main Input and paraphrases, as well as Re- tain Rate for all other data. We also include terms for Update Success on entailed data and the Local Neutral Retain Rate, when this is possible given available data. The overall objective requires sev- eral kinds of additional data for each point, which we denote byD R for other random data,D LN for local neutral data,D E for entailed data, andD P for paraphrases ofx i . For a data pointx i with desired predictiony ∗ i , the full objective is then: L(φ;x i ,ˆy i ,y ∗ i ,θ) =λ 1 L Task (f θ ∗ (x i ),y ∗ i ) +λ 2 1 |D P | ∑ x P ∈D P L Task (f θ ∗ (x P ),y ∗ i ) +λ 3 1 |D E | ∑ x E ,y E ∈D E L Task (f θ ∗ (x E ),y E ) +λ 4 1 |D LN | ∑ x LN ∈D LN KL(f θ ∗ (x LN )||f θ (x LN )) +λ 5 1 |D R | ∑ x R ∈D R KL(f θ ∗ (x R )||f θ (x R )) (1) whereL Task is the loss used to get gradients forf θ . We use the Cross Entropy loss for binary classifica- tion and sequence-to-sequence tasks. We optimize this objective w.r.t.φusing AdamW (Loshchilov and Hutter, 2019). To obtain update labelsy ∗ i n i=1 , we always use the oppo- site class in binary classification. For sequence-to- sequence tasks, we use the correct label whenˆy i is incorrect, and whenˆy i is correct, we randomly select another label from the training data. This choice is in contrast to De Cao et al. (2021) and Mitchell et al. (2021), who use samples from the model beam search as update labels for all points. SLAG objective. To better prepare the update method for evaluation in a sequential-update set- ting, we consider trainingg φ to update multiple datapoints in a row. Using the per-datapoint loss in Eq. 1, we obtain our Sequential, Local, and Gen- eralizing (SLAG) loss for a set ofrMain Inputs D=x i ,ˆy i ,y ∗ i r i=1 as L Sequential (φ;D,θ t ) = r ∑ i=1 L(φ;x i ,ˆy i ,y ∗ i ,θ t+i )(2) whereθ t+i =Update(x i ,ˆy i ,y ∗ i ,θ t+i−1 ;φ,K) are the model parameters obtained from updating on the firstipoints inD(starting fromθ t ). This objec- tive allows us to traing φ to update multiple beliefs in a row. To ensure training with this objective is still efficient, we limit how far back through the LM history we backpropagate when computing the gra- dient w.r.t.φfor each term in the RHS sum of Eq. 2. Each parameter vectorθ t depends onφandθ t−1 . We always apply the stop-gradient function to the most recent vectorθ t−1 to prevent backpropagating through it (visualized in Appendix Fig. 4). This choice allows our memory use to remain constant inr(see Appendix Fig. 5). 4 Experiment Setup 4.1 Datasets We run experiments with four datasets (example data shown in Appendix Table 15). (1) FEVER in- cludes 115,409 True/False factual claims (Thorne et al., 2018). We use the original test set of 10,444 points, and we randomly split the training data into 94,469 train points and 10,496 dev points. (2) zsRE includes 151,631 questions based on relational knowledge from Wikipedia, which we randomly shuffle into train/dev/test splits with 80/10/10% of the data (Levy et al., 2017). 32.8% of zsRE ques- tions in each split include paraphrases, and we mea- sure Update Success Rate (Paraphrase) for only these points. Talmor et al. (2020) introduce (3) the LeapOfThought dataset, consisting of 33,484 factual claims that are entailed to be true or false depending on a fact and distractor statements pro- vided as context. We drop the distractors from each input and filter the data so that the facts are unique, then shuffle the resulting 14,939 points into train/dev/test splits with 60/10/30% of the data. We also construct (4) a sequence prediction task using data from Wikidata5m, which is a relational knowledge base with over 20 million triplets (Wang et al., 2021b). We build this dataset in order to get Local Neutral data. Each input consists of an entity e 1 and relationr, and the label is another entity e 2 that completes the triplet. All inputs come in pairs that share the same entitye 1 but use different relations with different labels. The relations are always one of ten relations that apply to people (see Appendix Table 11). In general, the completione 2 to the Main Input triplet (e 1 ,r 1 ,e 2 ) has no logical consequences for its paired input, (e 1 ,r 2 , ?). This means that changing the model belief for the Main Input should not change its belief for its neutral paired input. The paired points are also local to the Main Input, i.e. they pertain to the same entity e 1 as the Main Input. We obtain four paraphrases for each Main Input using different aliases for the entity and synonyms of the relation. We construct a train set of 150k points and dev and test sets of 10k points each. See Appendix B for further details. 4.2 Methods Evaluated Models. We train five models with different ran- dom seeds for each dataset, using RoBERTa-base for binary tasks and BART-base for sequence-to- sequence tasks (accuracies in Appendix Table 14). For each of the five models, we train one learned Belief Consistency↑ DatasetParaphraseEntailedContrapos. LeapOfThought-85.6 (1.1)16.5 (2.7) zsRE69.5 (1.1)-- Wikidata5m25.8 (0.5)-- Table 2: Belief metric results across datasets. Paraphrase Consistency↑ DatasetModel IncorrectModel Correct zsRE61.39 (1.33)91.82 (1.17) Wikidata5m24.55 (0.48)37.20 (2.06) Table 3: Paraphrase consistency by the correctness of the model prediction on the Main Input. optimizer using SLAG and one with the objective from De Cao et al. (2021), which we list as KE in tables below. Our model selection criterion is the mean of: the average Update Success Rate (across data types), Retain Rate (only for Local Neutral data), and∆-Acc for All Data. We tune the choice of SLAG objective terms for each task separately (see Appendix Table 10 for final selections; results discussed in Sec. 5.3). Other hyperparameters are given in Appendix B and memory use is shown in Appendix Fig. 5. To summarize the differences between SLAG andKNOWLEDGEEDITOR: (1) we useK train =K test rather thanK train = 1; (2) we adopt training labels using real data labels rather than alternatives from the model’s beam search; and (3) our objective terms differ following tuning. Baselines.We use off-the-shelf optimizers as base- lines. We tune the baseline hyperparameters sep- arately for each dataset, selecting among several kinds of optimizers, learning rates, and the num- ber of update steps. The selection criterion is the same as the criterion outlined for learned optimiz- ers above. The resulting baselines are surprisingly strong (see Appendix Table 12 for final selections). Hypothesis testing.We obtain 95% confidence intervals and perform hypothesis tests via block bootstrap, resampling model seeds and data points (Efron and Tibshirani, 1994). For ablation experi- ments, we run only one model seed per condition. 5 Experiment Results 5.1 Do LMs have beliefs about the world? We measure Paraphrase Consistency, Entailment Acc, and Contrapositive Acc for our finetuned task models. Paraphrase Consistency is the fraction of paraphrase pairs for which a model produces the same output (Elazar et al., 2021). Entailment Acc Single-Update SettingUpdate Success RateRetain Rate∆-Acc DatasetMethodMain InputParaphrasesEntailed DataLocal NeutralAll DataAll Data FEVER AdamW100 (0.0)---98.80(0.2)0.22(0.1) KE99.98 (<0.1)---98.28 (0.3)-0.24 (0.1) SLAG99.99 (<0.1)---98.41 (0.2)-0.20 (0.1) LeapOfThought SGD100 (0.0)-72.48 (4.6)-95.52 (0.4)1.23(0.8) KE99.78 (0.4)-74.48 (4.4)-93.50 (1.3)-1.33 (1.1) SLAG100 (0.0)-75.50 (4.3)-94.92 (1.4)-1.31 (1.2) zsRE SGD99.36(0.1)94.44 (0.6)--74.73 (0.4)-0.43 (0.1) KE84.73 (1.4)89.26 (1.8)--71.55 (2.4)-2.19 (0.4) SLAG94.29 (0.4)94.71 (0.5)--80.48(1.3)-0.29 (0.1) Wikidata5m SGD98.05(0.3)68.78 (0.8)-41.46 (1.0)58.62 (0.6)-1.97 (0.3) KE74.57 (2.9)58.05 (2.2)-40.84 (1.8)53.58 (2.2)-3.03 (0.5) SLAG87.59 (0.6)80.70(0.9)-47.85(1.0)63.51(1.3)-1.71 (0.3) Table 4: Belief update metrics for off-the-shelf optimizers, KNOWLEDGEEDITOR(KE) from De Cao et al. (2021), and SLAG, withr test = 1. Bolded numbers are the best in their group at a statistical significance threshold of p < .05(or lower). Our SLAG objective improves over KE, but off-the-shelf optimizers perform surprisingly well. is the accuracy of a model on data that is entailed by the Main Input. For LeapOfThought (see Table 1), “Main Inputx i is true" implies “entailed inputx E has labely E ," but the inverse (¬A⇒¬B) does not necessarily hold. Therefore, we compute Entail- ment Acc only where the Main Input prediction is correct. We do know that the contrapositive holds: “Entailed inputx E does not have labely E " implies that “Main Inputx i is false." So for Contrapositive Acc, we measure how often the model follows this rule, when the antecedent holds of its prediction. Belief measurement results.Table 2 shows the belief metrics for each dataset.We find that ∼100M parameter models show limited evidence of having beliefs about the world. Paraphrase con- sistency is 69.50% (±1.09) for zsRE and much lower for Wikidata5m (25.84%±0.53). While entailment accuracy is high for LeapOfThought (85.63%±1.08), the model is consistent under the contrapositive only 16.51% (±2.71) of the time. One might reasonably set the bar for qualifying as a “belief" higher than these scores. But since belief- likeness comes in degrees, we continue to refer to model beliefs for the rest of the paper. Interest- ingly, the metrics are much higher when the model prediction on the Main Input is correct (Table 3). 5.2 Can we update beliefs in LMs? First, we compare two evaluation procedures for sequence prediction tasks: correcting model be- liefs versus changing them to an alternative from the model’s beam search. We do so for zsRE us- ing SLAG. Next, we compare belief update met- rics across datasets usingKNOWLEDGEEDITOR, SLAG, and off-the-shelf optimizers as baselines. Update Success Rate↑∆-Acc↑ Desired LabelMain InputParaphraseAll Data Beam Label97.41 (0.3)97.03 (0.4)-0.30 (0.1) Correct Label94.46 (0.7)94.45 (0.7)-0.24 (0.1) Table 5: Evaluation difficulty by desired model output, for a learned optimizer trained with SLAG on zsRE. We report results in single-update (r test = 1) and sequential-update (r test = 10) settings. See Ap- pendix Fig. 6 for an ablation acrossr test . Correcting beliefs vs. changing beliefs.Given the results in Table 5, we find that correcting model outputs is harder than simply changing them to a plausible alternative. Update Success can rise by a full 2.96 (±0.48;p<1e−4) points for Main Inputs and 2.58 (±0.81;p<1e−4) for Paraphrases, while ∆-Acc is virtually unchanged. This suggests that that past work has overestimated the efficacy of belief update methods for actually fixing models. Henceforth we evaluate methods according to their ability to update model beliefs to be true. Update method results (single update). Table 4 shows the results in a single-update setting. First, we find that off-the-shelf optimizers are very effec- tive across the board. The baselines show Main Input Update Success Rates of 100% for binary tasks with positive∆-Acc scores. 2 On sequence prediction tasks, SGD achieves 98%+ Main Input Update Success with competitive∆-Acc scores. When strongly tuned, these baselines outperform learned optimizers on most metrics here. 2 Positive∆-Acc values are possibly due to distribution shift in the test split. In FEVER, for instance, the train and dev data are 73% True, while test data is 50% True. On the dev split, AdamW achieves a negative∆-Acc, -0.18 (±0.11). Sequential-Update SettingUpdate Success RateRetain Rate∆-Acc DatasetMethodMain InputParaphrasesEntailed DataLocal NeutralAll DataAll Data FEVER AdamW92.81 (1.3)---91.86(1.4)1.16(0.6) SLAG 1 74.13 (1.8)---39.86 (0.7)-27.13 (1.3) SLAG 10 91.27 (2.9)---70.30 (5.8)-11.96 (4.5) LeapOfThought SGD100 (0.0)-61.34(5.0)-82.62(0.8)-4.93(1.0) SLAG 1 96.14 (2.3)-49.27 (6.0)-72.45 (0.9)-15.03 (1.0) SLAG 10 100 (0.0)-50.46 (5.5)-74.02 (1.1)-13.03 (1.3) zsRE SGD82.71 (0.6)90.81 (0.7)--40.49 (0.6)-2.38 (0.3) SLAG 1 0.10 (<0.1)36.55 (1.4)--0.05 (<0.1)-20.98 (0.7) SLAG 10 87.57(0.6)92.20(0.7)--47.19(0.7)-1.74(0.3) Wikidata5m SGD56.82 (0.8)54.49 (0.7)-6.40 (0.4)26.37 (0.6)-3.96 (0.4) SLAG 1 0 (0.0)40.84 (0.9)-0 (0.0)0 (0.0)-10.05 (0.6) SLAG 10 58.27(1.0)65.51(0.9)-7.36(0.5)27.76(0.7)-3.62 (0.4) Table 6: Belief update results when a model is sequentially updatedr test =10times. SLAG R usesr train =R. On sequence prediction tasks in this setting, SLAG can outperform the off-the-shelf optimizers across metrics. However, SLAG does surpass the baselines in a few places. All Data Retain Rate on zsRE rises by 5.77 points (±1.43;p<1e−4), and on Wiki- data5m we improve Paraphrase Update Success by 11.92 points (±1.20;p<1e−4) and the Local Neu- tral Retain Rate by 6.40 (±1.41;p<1e−4) points. The gain on Entailed Data Update Success is 3.02 points, but it is not significant (±6.26;p=.345). The SLAG objective also greatly improves perfor- mance over KE for sequence prediction tasks. Update method results (sequential updates). We give results for a sequential update setting (r test =10) in Table 6. Immediately we see this is a much more difficult setting for updating model beliefs, as update metrics are generally much lower for each dataset. Next, we observe that learned optimizers with SLAG 10 (r train =10) now outper- form baselines on sequence prediction tasks. On zsRE, we improve Update Success for Main In- puts by 4.86 (±0.83;p=1e−4) and for Paraphrases by 1.39 (±0.93;p=.004), with better∆-Acc by 0.64 (±0.35;p=.0005). Improvements trend in the same direction for Wikidata5m and are all sta- tistically significant except for the gain in∆-Acc. The jump on Paraphrases in particular is very large (11.02±1.17;p<1e−4). In comparison, using a non-sequential (r train = 1) training objective leads to drastic drops in performance. Learned optimizers still struggle with the binary datasets compared to the off-the-shelf optimizers. The baselines achieve high update update success with much better∆-Acc scores, by 13.12 (±4.51; p=1e−4) on FEVER and 8.16 (±1.63;p=1e−4) on LeapOfThought. Also on LeapOfThought, the baseline’s update success with entailed data is over 10 points higher (±7.38;p=.004). 5.3 How does the learned optimizer objective influence performance? Here, we discuss ablations with respect to the terms in the training objective, Eq. 1. We show the effect ofK train in Appendix Fig. 9 and the choice of optimizer training labels in Appendix Table 16. Training objective ablation.We give objective ablation results in Appendix Table 17. Surpris- ingly, we do not always see that the objective terms help for the data they are intended to help with. First, we obtain mixed results for the paraphrase objective. On zsRE, the objective term seems to hinder performance, with update success dropping on Main Inputs by 0.71 (±0.60;p=.021) and∆- Acc dropping by 0.18 (±0.19;p=.069), while the paraphrase Update Success Rate itself is unaffected. With Wikidata5m, however, the paraphrase term improves paraphrase update success by a large mar- gin of 16.94 (±1.03;p<1e−4) points. Adding the Local Neutral (LN) term with the paraphrase term greatly improves the LN Retain Rate for Wiki- data5m, by 9.71 points (±1.44;p<1e−4), though both of these terms come at a cost to Main Input Update Success, similar to zsRE. Lastly, we do not find that the entailment objective improves Entailed Data Update Success; in fact, this metric falls by 4.56 (±7.22;p=.213) points with the objective. 6 Analysis 6.1 Belief updates improve consistency In Table 7, we show belief metrics before and af- ter model updates using SLAG withr test =1. We observe that belief updates greatly improve para- MetricBefore UpdateAfter Update Entailment Acc58.30 (5.7)*75.50 (4.3) Para. Cons (zsRE)61.39 (1.3)94.53 (0.6) Para. Cons (Wiki)24.69 (0.5)84.56 (0.9) Table 7: Entailment Acc and Paraphrase Consistency before and after model updates to incorrect points. *All Main Inputs in this subset are wrongly predicted as false, so the entailment does not actually hold. phrase consistency and entailment accuracy for updated data. Paraphrase consistency rises by 33.14±1.46 on zsRE and 59.87±1.09 on Wiki- data5m, while Entailment Acc rises by 17.20±7.10 points. To see if these improvements depend on pre- update consistency, we plot paraphrase consistency before and after updating in Fig. 2. For zsRE, con- sistency rises irrespective of pre-update consistency. There is a noticeable trend for Wikidata5m para- phrases, where post-update consistency is 90.1% when pre-update consistency is maxed out, versus 77.1% for totally inconsistent pre-update beliefs. We conclude that learned optimizers can induce a fairly consistent model belief even where there is no consistent belief to begin with. 6.2 Which beliefs are hard to retain when updating other beliefs? We find that the Retain Rate depends heavily on whether the predictions on that data are correct to begin with. On zsRE for instance, the retain rate on correct inputs is about 96%, while for incorrect predictions, it is about 75%. So it appears that in- correct predictions are the most sensitive to model updates, and these points merely change from one incorrect prediction to another. On FEVER, incor- rect beliefs change around 4% of the time, while correct beliefs change only 2.5% of the time. We also find that Local Neutral beliefs are much harder to avoid changing than simply random data. For Wikidata5m in Table 4, the Retain Rate on All Data is 61.51±1.33, while for Local Neutral data it is a full 15.66 points lower, at 47.85±0.96. 6.3 Belief Graphs We now constructbelief graphsfor the purpose of better understanding the connections between model beliefs. We form the graphs from a set of dat- apoints by updating each prediction and checking what other predictions change. We represent each datapoint as its own node in a belief graph. When- ever updating a datapointuchanges the model pre- diction for pointv, we draw a directed edge from zsRE Wikidata5m 0.000.250.500.751.00 0.7 0.8 0.9 1.0 0.7 0.8 0.9 1.0 Pre−Update Consistency Post−Update Consistency Updates Improve Consistency Everywhere Figure 2: Post-update consistency under paraphrase is high even for points with low pre-update consistency. utov. Following our results in Sec. 5.2, we use off-the-shelf optimizers to change the model output to the opposite of its original prediction for every datapoint. The resulting graphs have up ton 2 −n edges (no self edges). For FEVER we obtain a graph of 10,444 nodes, and for LeapOfThought we obtain a graph with 8642 nodes, which is double the original test set size because we include both Main Inputs and Entailed Data as their own nodes. We visualize part of a belief graph in Fig. 3. This figure shows a non-random subgraph intended to give a representative view of the data (we give three random subgraphs of 20 nodes in Appendix E). On inspection, we see no reason that beliefs are connected or not connected. Whether or not chang- ing one belief changes another appears essentially random. We come to same conclusion looking at other random subgraphs (see Appendix Figures 10, 11, 12). However, we do know of some aggregate trends from earlier results. Sec. 6.2 suggests that a model’s incorrect beliefs are most likely to change after model updates, and following Sec. 5, we have reason to believe that Local beliefs are more likely than others to change with model updates. We highlight a few summary statistics here from Table 8 for a broader view of the graphs. First, % Edgeless is the proportion of nodes which have no in or out edges. Since this is 0 for both datasets, every belief can be changed by editing the right belief. # In Edges is the number of in edges at the 95 th percentile, meaning 5% of beliefs have more in edges than this value, and the same holds of # Out Edges. These values grow to a rather large fraction of the overall datasets, suggesting that (1) some beliefs are sensitive to changes in a large fraction of all beliefs, and (2) some beliefs are influential to hundreds of other beliefs when changed. # Cor- rupted is the number of correct predictions changed Middle-earth is a real place. [y: false] Hot Right Now is mistakenly attributed to DJ Fresh. [y: false] There are no musical or creative works in existence that have been created by Phillip Glass. [y: false] The Daily Show is incapable of focusing on recent news stories. [y: false] The Chrysler Building was always the world's shortest building. [y: false] Shane McMahon officially retired on the first day of 2010. [y: false] Bessie Smith died on April 26, 1937. [y: false] Despicable Me 2 was written by Cinco Paul. [y: true] Hot Right Now is from Nextlevelism. [y: true] Figure 3: A non-random subgraph of the belief graph for a model trained on FEVER. Directed edges fromuto vindicate that changing the model belief inucauses the belief invto change. The ground-truth label is given in brackets for each point, and node color shows the model’s accuracy before any updates (green=correct). Dataset MetricFEVERLeapOfThought # Nodes10,4448,642 % Edgeless0.00.0 # Edges Total1.88m9.71m # In Edges (95 th perc.)1,0885,347 # Out Edges (95 th perc.)3903,087 # Corrupted (95 th perc.)2112,752 % Update-Transitivity66.6424.38* Table 8: Belief graph summary statistics. *We compute Update-Transitivity for LeapOfThought withn= 4000 points due to computational cost. to be incorrect following a model update. For 5% of the data, model updates cause at least 211 points to become incorrectly predicted on FEVER, and 2,752 points for LeapOfThought. Lastly, % Update- Transitivity represents the answer to the question: if updating belief A changes belief B, and updating belief B changes belief C, what proportion of the time does updating A change C? For these datasets, a logically consistent model should display 100% Update-Transitivity (see Appendix D for a caveat on this metric). We find that belief updates often yield intransitive results for both datasets. 7 Discussion and Conclusion Degrees of commitment to beliefs. The data we use comes in the form of declarative statements and answers to questions. These utterances take what is called a veridical stance toward a proposition, dis- playing a “full commitment" to that proposition’s truthfulness (Giannakidou and Mari, 2020). It will be valuable for future work to explore two dimen- sions of uncertainty in beliefs: (1) expression of uncertainty in language, via partial or trivial com- mitments (like “X might be Y") and (2) expression of uncertainty mathematically, via probabilities as- signed by a model to utterances or True/False val- ues. In this paper we treat a belief as “updated" when the model output changes, but this ignores any underlying change in the distributionp θ (y|x) that could occur even if its mode does not change. Ethics and dual use concerns. Belief update methods may be used to either correct undesired beliefs or induce problematic beliefs in LMs, and it is not clear whether these capabilities could be separated. We propose to evaluate methods only on the basis of their ability to correct mistaken model beliefs, but the malicious use case remains. We are uncertain about how a bad belief would influ- ence the general behavior of a model (e.g. answers to many questions), but it is possible that a belief update method could instill bad beliefs in a gen- erally capable LM with far-reaching implications for model behavior. That said, we hope that these methods will instead be used to update undesirable moral, social, and factual beliefs in large LMs. Conclusion.We first discuss criteria for detect- ing when LMs havebeliefsabout the world. Next, we argue for evaluating belief update methods by their ability to correct mistaken beliefs, which is harder than the evaluation done in past work. We show that strongly tuned off-the-shelf optimizers make for surprisingly good belief update methods, even surpassing specialized learned optimizers in several settings. But with a new training objective (SLAG), we are able to outperform these baselines on sequence prediction tasks when updating mul- tiple beliefs one after another. Finally, we intro- ducebelief graphsas a means of understanding the connections between model beliefs. We find that model beliefs are highly interconnected, with some beliefs influencing hundreds of other beliefs. While it is hard to point to concrete reasons for individual connections between beliefs, we identify several patterns in the dependencies between beliefs. References Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610–623. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021. Knowledge neurons in pretrained trans- formers.arXiv preprint arXiv:2104.08696. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In EMNLP, pages 6491–6506. Association for Compu- tational Linguistics. Daniel Dennett. 1995. Do animals have beliefs?Com- parative approaches to cognitive science, 111. Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021.Time-aware language models as temporal knowledge bases.arXiv preprint arXiv:2106.15110. Bradley Efron and Robert J Tibshirani. 1994.An Intro- duction to the Bootstrap. CRC press. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.Measuring and im- proving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. InFindings of EMNLP. Anastasia Giannakidou and Alda Mari. 2020. A lin- guistic framework for knowledge, belief, and veridi- cality judgement.HAL. Peter Hase and Mohit Bansal. 2021. When can mod- els learn from explanations? a formal framework for understanding the roles of explanation data.arXiv preprint arXiv:2102.02201. Benjamin Heinzerling and Kentaro Inui. 2021. Lan- guage models as knowledge bases:On entity representations, storage capacity, and paraphrased queries. InProceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, pages 1772– 1791, Online. Association for Computational Lin- guistics. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. InEMNLP. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know?Transactions of the Association for Computational Linguistics, 8:423–438. Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021. Beliefbank: Adding memory to a pre-trained language model for a systematic notion of belief.arXiv preprint arXiv:2109.14723. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gri- bovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Sebastian Ruder, Dani Yogatama, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. InNeurIPS. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancou- ver, Canada. Association for Computational Linguis- tics. Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.Retrieval-augmented generation for knowledge-intensive NLP tasks. InNeurIPS. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods.arXiv preprint arXiv:2109.07958. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. InICLR. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale.arXiv preprint arXiv:2110.11309. Albert Newen and Tobias Starzak. 2020. How to as- cribe beliefs to animals.Mind & Language. German I Parisi, Ronald Kemker, Jose L Part, Christo- pher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases?InProceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 2463–2473, Hong Kong, China. As- sociation for Computational Linguistics. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works.Transactions of the Associ- ation for Computational Linguistics, 8:842–866. Eric Schwitzgebel. 2019. Belief. In Edward N. Zalta, editor,The Stanford Encyclopedia of Philosophy, Fall 2019 edition. Metaphysics Research Lab, Stan- ford University. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. InICLR. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Gold- berg, and Jonathan Berant. 2020. Leap-of-thought: Teaching pre-trained models to systematically rea- son over implicit knowledge. InNeurIPS. JamesThorne,AndreasVlachos,Christos Christodoulopoulos,andArpitMittal.2018. FEVER: a large-scale dataset for fact extraction and VERification.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics. Elena Voita and Ivan Titov. 2020.Information- theoretic probing with minimum description length. InEMNLP. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2021a. K-adapter: Infusing knowledge into pre-trained models with adapters. InFindings of ACL. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b.Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021.Sym- bolic knowledge distillation: from general language models to commonsense models.arXiv preprint arXiv:2110.07178. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Sri- nadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models.arXiv preprint arXiv:2012.00363. A Learned Optimizer Details Architecture. KNOWLEDGEEDITORis a learned optimizerg:X ×Y ×Y ×Θ→Θthat produces new model weights by applying an adjusted gra- dient step to a model. For reference, we give a glossary of symbols used here in Table 9. For ad- ditional details beyond what is presented here, we refer readers to De Cao et al. (2021). Symbol Glossary f θ Language Model g φ Learned optimizer x i Main Input ˆy i LM output onx i y ∗ i Desired output ∇ θ L(x i ,y ∗ i )Gradient of LM Update(x i ,ˆy i ,y ∗ i ,θ)Update one LM belief L(φ;x i ,ˆy i ,y ∗ i ,θ)Belief update objective forx i L Sequential (φ;D,θ t )Sequential objective (SLAG) K# gradient steps in Update(·) r# beliefs updated inL Sequential Table 9: Symbol descriptions for the learned optimizer. At a high level,g φ first encodes an inputx i and requested prediction change into a vectorh, then processeshinto two low-rank matricesAandB that are used to transform the model gradient onx i , ∇ θ L(x i ,y ∗ i ) . For Transformer models, the method edits only attention and feed-forward weights, so all model gradients match the shape of an associ- ated weight matrix of shaped 1 ×d 2 . Formally, a new modelθ ∗ is obtained using a learned optimizer g φ as follows: h=LSTM([x; ˆy;y ∗ ]) u,v,γ,δ=MLP i (h) 4 i=1 A=softmax(u)v T B=softmax(γ)δ T η=σ(MLP(h)) θ ∗ =θ+η(A◦∇ θ L(x i ,y ∗ i ) +B) whereφconsists of all LSTM and MLP parameters. Training Algorithm.The learned optimizer ob- jective is optimized w.r.t.φwith AdamW through a standard procedure of randomly sampling mini- batches without replacement (Loshchilov and Hut- ter, 2019). Within each batch, one datapoint is randomly selected as the Main Input, and the re- maining points are used asD R . To obtain update labelsy ∗ i n i=1 , we always use the opposite class in binary classification. For sequence-to-sequence tasks, we use the correct label whenˆy i is incorrect, and whenˆy i is correct, we randomly select another label from the training data. This choice is in con- trast to De Cao et al. (2021) and Mitchell et al. (2021), who use samples from the model beam search as update labels for all points. Task Model Optimizer Backprop Stop Gradient Sequential Backprop Graph Figure 4: The backpropagation graph for sequential model updates. B Additional Training Details B.1 Compute Costs. Learned optimizer memory.The hypernetwork has 92m trainable parameters for RoBERTa-base (which is 125m parameters), and 105m param- eters for BART-base (which is 139m parame- ters). To increase training efficiency, we limit how far into the task model history we backprop- agate. As shown in Fig. 4, when backpropagat- ing through task model parametersθ t =θ t−1 + Update(x i ,ˆy i ,y ∗ i ,θ t−1 ;φ), we continue backprop- agating throughUpdate(x i ,ˆy i ,y ∗ i ,θ t−1 )butnot θ t−1 , which is also dependent onφ. That is, we ap- ply a stop-gradient function toθ t−1 . This way, we compute the derivative∇ φ Update(x i ,ˆy i ,y ∗ i ,θ t ;φ). only once for eacht, rather than recomputing these gradients for all subsequent time steps. These choices allow the memory use of our training algo- rithm to remain constant inr. We make the same choice for ourKlooped steps in a single applica- tion of the Update function, so the gradient for the update at stepkdepends only ong φ (x i ,ˆy i ,y ∗ i ,θ (k) ) and notθ (k−1) . See Fig. 5 for a graph of memory use depending onrandk. Experiment runtimes. We now give runtimes for experiments in the paper. Building the belief graphs takes 25 hours for FEVER (n= 10,444) and 17.5 hours for LeapOfThought (n= 8642) on an NVIDIA RTX 2080 GPU. Computing sum- mary statistics for graphs takes 3 hours on FEVER and 3 hours for LeapOfThought for statistics be- sides Update-Transitivity. We compute Update- Transitivity for LeapOfThought with a subset of 4000 points, which takes 45 hours. All other experiments are run on a NVIDIA V100 32GB GPU. Training the task models takes 7 minutes for LeapOfThought, 45 minutes for FEVER, 4 hours for zsRE, and 10 hours for Wiki- data5m. Training the learned optimizer withr= 1 takes 2.3 hours for LeapOfThought, 5 hours for FEVER, 9.5 hours for zsRE, and 16 hours for Wikidata5m. Training the learned optimizer with r= 10takes 53 minutes for LeapOfThought, 2.9 hours for FEVER, 7 hours for zsRE, and 12.5 hours for Wikidata5m. Computing update statistics with the off-the-shelf optimizers withr= 1takes 4 min- utes for LeapOfThought, 30 minutes for FEVER, 2.3 hours for zsRE, and 3.9 hours for Wikidata5m. Withr= 10, the baselines require 1 minute for LeapOfThought, 15 minutes for FEVER, 54 min- utes for zsRE, and 1.8 hours for Wikidata5m. Total runtimes for each experiment should take into ac- count multiple conditions and multiple seeds of each model being run. B.2 Hyperparameters and Objective Terms. Training hyperparameters. We fit our RoBERTa- base and BART-base task models to their respec- tive datasets with the following hyperparameters: We train for 10 epochs on the binary tasks, and 20 for the sequence-to-sequence tasks. When pre- dicting with BART-base, we use a beam search with width 5. In each case, we use AdamW from torch.optimwith a LR of 1e-5 and weight de- cay of 1e-4. We select the best model according to the best dev set accuracy, checkpointing after each training epoch. The learned optimizers are optimized with AdamW, using a learning rate of 3e-4 and weight decay of 0. We train the learned optimizer for 5 epochs on each dataset except for LeapOfThought, which we train for 10 epochs given its smaller size. The learned optimizers are also selected based on dev set performance, with checkpointing after each training epoch. Their se- lection criterion is a raw average of Update Success Rate (averaged over each kind of data), Retain Rate (Local Neutral) and∆-Acc, with terms dropped Datasetr test kObjective FEVER 15Main 101Main LeapOfThought 15Main 101Main zsRE 15Main 105Main Wikidata5m 15Main+Para 105Main+Para Table 10: Final hyperparameters and objective terms of the learned optimizer for each task. Relation% Test Data Place of Birth11.00 Award Received11.00 Cause of Death5.66 Place of Death11.00 Place of Burial8.33 Educated At11.00 Child11.00 Occupation11.00 Spouse11.00 Sibling9.01 Table 11: Wikidata relations and their proportion of the test data. when they cannot be computed given the available data. Note that dev epochs with zsRE and Wiki- data5m are fairly slow, so in order to speed up our experiments we compute dev epochs with a subset of 4000 dev points. Learned optimizer. We give the final hyperparam- eter and objective terms used in each experiment in Table 10. Our objective ablation is given in 17, and we select the best performing condition for each dataset according to dev set performance, using the same selection criterion outlined previously. We keep all weight coefficientsλ i equal rather than tuning them. Main refers to the first term in Eq. 1, plus the KL term with random data. We use K train ≤5for all experiments. For results across Kvalues on zsRE, see Fig. 9. Baseline update method. We tune a baseline off- the-shelf optimizer separately for each dataset, us- ingr test = 1. Our performance criterion is the same as with learned optimizers, a raw average of Update Success Rate (averaged over each kind of data), Retain Rate (Local Neutral) and∆-Acc. The grid search is over the following parameters: The off-the-shelf optimizers are fromtorch.optim and include AdamW, SGD, and RMSProp with default arguments (except for the learning rate). We consider a number of maximum steps in 5, 10, 100. The learning rates we consider depend DatasetOptimizerLRNum. Steps FEVERAdamW1e-6100 LeapOfThoughtSGD1e-2100 zsRESGD1e-110 Wikidata5mSGD1e-110 Table 12: Final hyperparameters of the baseline update method for each task. on the optimizer: 1e-4, 1e-5, 1e-6 for AdamW, 1e-4, 1e-5, 1e-6 for RMSProp, and 1e-1, 1e-2, 1e-3 for SGD. The LR ranges were selected af- ter some initial manual exploration of the space. Our final hyperparameter values are shown in Ta- ble 12 for each dataset. For comparison, De Cao et al. (2021) use RMSProp with 100 update steps. The LR for zsRE and Wikidata5m may seem quite high, but this is the condition that actually does the least damage to the model’s accuracy on other data, ∆-Acc. The baseline optimizes all of the train- able parameters in the language model, unlike the learned optimizer which optimizes only attention and feedforward weights for purposes of parameter efficiency. B.3 Wikidata5m Additional Details. We construct four paraphrases per Main Input by selecting from a set of alternative phrasings for the entity and relation in the Main Input. The syntax for each paraphrase follows the same simple tem- plate as the Main Input, in contrast to zsRE where syntax differs between paraphrases. A couple de- tails remain. Some relations are one-to-many, and therefore we accumulate valid completing entities from the data as possible answers; later we com- pute accuracy as an exact match with any possible answer. All 10 relations appear in each split of the data. Only 33.80% and 37.18% of the entities in the dev and test splits are seen in the training data, though we do not find that models perform better on entities seen in training. B.4 LeapOfThought Additional Details The LeapOfThought dataset consists of a fact and a claim for each datapoint, where the truth of the fact implies that the claim has labely i (True/False). All of the facts in the data are true, while half of the claims are true and half are false. When training the learned optimizer, we treat the the facts as the Main Input when training the learned optimizer and claims as entailed data. When training the True/False classifier, we fit to the claims, for which test accuracy is 83.65 (±1.05). This seems to K K 0 10 20 30 1246810 r Memory Used (GB) Memory Usage byr Figure 5: Training memory usage in terms ofKandrhyperparameters in our implementation, for a learned optimizer trained for a BART-base model on zsRE, using a batch size of 16. For comparison, the orange dashed line shows the memory use of training the BART-base model on zsRE, using the same batch size. Our use of the stop-gradient function limits the growth of runtime and memory w.r.t. bothKandr. By accumulating gradients across points, memory w.r.t.ris kept constant. The same trick could be applied to theKlooped gradient steps inside the Update function, at the trade-off of backpropagatingKtimes per point rather than one time. OursDe Cao et al. (2021)Mitchell et al. (2021) Update Success Rate (Main Input)Success rateEdit success Update Success Rate (Paraphrase)Equivalence accuracyEdit success Update Success Rate (Entailed Data)-- Retain Rate (Local Neutral)-- Retain Rate (All Data)Retain accuracy- ∆-Acc (All Data)Performance deteriorationDrawdown Table 13: A glossary of terms used in work on model update methods. Note metrics are not always calculated in exactly the same way. For instance, Performance deterioration is a ratio in accuracies rather than difference in accuracies, and edit success from Mitchell et al. (2021) combines two metrics in our case. The performance metric in Zhu et al. (2020) is an average of Update Success Rate (Main Input) and∆-Acc. generalize well to the facts, as test accuracy here is 93.66 (±0.87), although as the low contrapositive accuracy suggests (Table 3), the model seems to be too prone to predicting true for this data. Since very few of the Main Inputs are predicted as false, we run into a small dilemma when fit- ting the learned optimizer with the use of the en- tailed data objective term. The entailment between fact and claim only holds when the fact is true, so we can only compute the objective when updat- ing a point from false to true. This ends up being less than 10% of the training data. We ultimately choose to oversample points that fit this descrip- tion during training of the learned optimizer, which allows the learned optimizer to fully fit to the en- tailed data. Also note that during learned optimizer training, we include Entailed Datafrom other data points besides the Main Inputin the KL term in Eq. 1, and we measure∆-Acc using both Main Inputs and Entailed Data. C Noise in Datasets We briefly document some shortcomings of each dataset, with reference to examples in Table 15. FEVER. Some claims are slightly vague or am- biguous when taken on their own. For instance “Doug Ducey was the CEO of Cold Stone Cream- ery and offered many opportunities to new hires" is rated True, though this will depend heavily on what one thinks “many opportunities" means. Sim- ilar whether or not “L.A. Guns is a tattoo shop" depends on which “L.A. Guns" one is referring to, the tattoo shop or metal band. Of course, this is a generic issue of language, and not unique to this dataset. Some inputs seem to be a matter of person opinion: “Los Angeles is known for its food" is rated False. LeapOfThought.Many examples use an “is a" relation, producing sentences like “A sunlight is a good health." This could be more false than true, but it’s a fairly nonsensical statement to begin with. There are also other nonsensical or vague examples in the data: ”A friar is the opposite of mineral" is labeled False. “A detective desires equal opportu- nity." is labeled True. It is not immediately clear what conditions would make these statements true or false. zsRE. Some questions invoke potentially one-to- many or temporally dependent relations, though there is only one ground-truth answer per ques- tion in this dataset. For instance, a paraphrase of the question about Gifford Pinchot in Table 15 is: DatasetModelAccParaphrase Cons↑Entailment Acc↑Contrapositive Acc↑ FEVERRoBERTa-base78.29 (0.86)--- LeapOfThoughtRoBERTa-base93.66 (0.87)-85.63 (1.08)16.51 (2.71) zsREBART-base21.01 (0.64)69.50 (1.09)-- Wikidata5mBART-base10.21 (0.59)25.84 (0.53)-- Table 14: Model accuracy and belief metric results and for four datasets. DatasetData TypeInputLabel(s) zsRE Main InputWhat did Gifford Pinchot die of? Leukemia ParaphraseHow did Gifford Pinchot die? Main InputPlayer Ali Kanaan plays for what team? Sporting Al Riyadi Beirut ParaphraseWhat team is Ali Kanaan associated with? Wikidata5m Main Input Margarita Nolasco Armas has relation ‘place of birth’ to Orizaba, Veracruz; Orizaba; etc. Paraphrase SusunW/Margarita Nolasco Armas has rela- tion ‘born at’ to Local NeutralMargarita Nolasco Armas has relation ‘place of death’ to Mexico City; Ciudad de Mexico; etc. Main InputMary Good has relation ‘award received’ toGarvan-Olin Medal; Arkansas Women’s Hall of Fame; etc.ParaphraseMary Lowe Good has relation ‘winner of’ to Local NeutralMary Good has relation ‘educated at’ toThe University of Arkansas; U Arkansas; etc. FEVER Main InputTardigrades are also known as space bears.True Main InputThe Lion belongs to the genus Vulpes.False LeapOfThought Main InputA viper is a vertebrate.True Entailed DataA viper has a brain.True Main InputA amaranth is a herb.True Entailed DataA amaranth has a nose.False Table 15: Example datapoint from each dataset, and auxiliary data that accompanies the Main Input. "What disease did Gifford Pinchot have?" A per- son might have had many diseases over their life which could all be valid responses. The answer is especially ambiguous for spatial relations, where a valid answer might refer to a city, region, country, province, or continent. Wikidata.Aliases sometimes vary greatly even as they refer to the same person, or they are sim- ply noisy. For example, as shown in Table 15, “SusunW" appears in an entity name, but this is actually a username of someone who contributed to the Wikipedia article for Margarita Nolasco Ar- mas. Meanwhile, other aliases for J.R.R Tolkien include “Tolkienian" and “Mabel Suffield," his mother. Rephrasings of relations might also create confusing inputs, e.g. switching “child" with “has kids," “daughter", or “son." Similar to zsRE, some relations are also one-to-many and temporally de- pendent (like occupation), though we hope that by using many valid answers we circumvent this issue to some extent when calculating prediction correctness. D Metric Computation and Bootstrap Details Metric computation.The only computationally difficult metric to calculate is∆-Acc, which re- quires computing the updated language model’s accuracy on other data after every single belief up- date. We randomly sample other data after every update for this purpose, usingn= 30points for zsRE and Wikidata5m andn= 200points for FEVER and LeapOfThought. We ensure that all evaluation data is used at some point during this sampling by preferentially selecting data that has been infrequently selected before. We note that paraphrase consistency is easy to evaluate for a small number of paraphrases per datapoint, as we have for both zsRE and Wikidata5m. Additionally, on LeapOfThought, we compute∆-Acc using both Main Inputs and Entailed Data. Update-Transitivity caveat. The % Update- Transitivity metric represents the answer to the question: if updating belief A changes belief B, and updating belief B changes belief C, what pro- portion of the time does updating A change C? FEVERZSRE 12468101246810 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 r test Update Success Rate Method by r train Baseline r=1 r matches test Ablation by r Figure 6: Ablation across values ofrfor training and testing. On zsRE, our method outperforms the baseline when r test = 10, and the gap is likely to increase asr test rises further. When using a non-sequential objective from past work, performance declines drastically asr test rises. We would treat this as a normative metric that we hope to maximize, except we do not know in gen- eral whether there is a confounding belief D that determines the relationship between B and C. If changing A also changed a confounding belief D, then we might not be able to expect that C should change too. That said, when we have no reason to think there are such confounding beliefs, we would expect a logically consistent model to display 100% Update-Transitivity of their beliefs. In Fig. 3, for instance, we see no reason to suspect there are con- founding beliefs for the relationship between the date Bessie Smith died and the writer of Despicable Me 2, and therefore we would expect that updat- ing the belief about what album Hot Right Now is on would change the belief in Despicable Me 2’s authorship (which it does). Bootstrap computation.We account for sample and seed variance by block bootstrap (Efron and Tibshirani, 1994). When there is a single statistic per data point, like Main Input Update Success, we form a matrix of shapen×sforndata points and smodel seeds (where the seed was used for both task model training and learned optimizer train- ing). We then resample rows and columns of this matrix 10,000 times, which was sufficient for con- vergence. When we perform hypothesis tests for the difference in statistics between conditions, we pair the data points by using the same rows of this matrix at each step of the bootstrap (i.e. we conduct paired tests). For metrics involving multiple data points per Main Input, like paraphrases or other random data, we make a simplifying assumption where we do not resample the multiple data points but just compute the average metric for those data points and treat that as the ground-truth statistics Update Success Rate∆-Acc Desired LabelMain InputParaphrasesAll Data Beam Label91.19 (0.5)92.07 (0.8)-0.39 (0.1) Hard Label94.46 (0.7)94.45 (0.7)-0.24 (0.1) Table 16: Update metrics by optimizer training labels. for the Main Input. We explored using a full 3- dimensional bootstrap, where we resample among these extra datapoints by constructing a matrix of shapen×s×n, but it was quite slow and gave similar results to the block bootstrap. E Additional Results Ablation across num. sequential steps.Fig. 6 shows the results for an ablation acrossr test using two kinds of learned optimizers: SLAG 1 , wherer train = 1, and a SLAG condition where r train =r test . It is critical to the success of learned optimizers to train them to update points sequen- tially when this is a desired application. Further, sequential updating with sequence prediction tasks is the only setting where we see learned optimizers outperform baselines across all relevant metrics. Choosing training labels for learned optimizers. In early experiments, we found that it is beneficial to use all data points (including correctly predicted points) as Main Inputsduring training, rather than restricting training to only incorrectly predicted points. We still focus on correcting wrong outputs at test time. But so we must select what label to use during optimizer training. To get a Hard Label, we use the correct label for incorrectly predicted points, and for correctly predicted points, we sim- ply draw a label randomly from the labels in the Objective Term AblationUpdate Success RateRetain Predictions∆Acc DatasetObjectiveMain InputParaphrasesEntailed DataLocal NeutralAll DataAll Data FEVER Main100 (0.0)---98.27 (0.1)-0.15 (0.1) (no KL)100 (0.0)---40.42 (0.6)-27.19 (1.2) LeapOfThought Main100 (0.0)-76.43 (5.3)-96.84 (0.3)-1.22 (0.8) +Ent100 (0.0)-71.87 (5.3)-96.52 (0.3)-0.40 (0.8) zsRE Main94.46 (0.4)94.44 (0.7)--81.96 (0.4)-0.24 (0.1) +Para93.75 (0.4)94.41 (0.7)--75.24 (0.5)-0.42 (0.2) Wikidata5m Main88.67 (0.7)64.12 (0.7)-49.78 (1.0)71.04 (0.5)-1.54 (0.3) +Para87.46 (0.7)81.06 (0.7)-47.15 (1.0)63.02 (0.6)-1.55 (0.3) +LN87.73 (0.7)59.75 (0.7)-60.49 (1.0)72.69 (0.6)-1.57 (0.3) +Para+LN87.02 (0.7)81.18 (0.7)-56.86 (1.0)68.42 (0.6)-1.65 (0.3) Table 17: Belief update results by the objective terms used for the learned optimizer. We do not bold any numbers based on statistical significance. For tuning purposes we select whichever condition achieves the higher selection criterion without testing for statistical significance. training data. The alternative Beam Label condi- tion uses a sample from the model’s beam search for a data point, as done in past work (De Cao et al., 2021; Mitchell et al., 2021). We show up- date metrics for zsRE split by the desired label in Table 16. If one’s goal is to fix wrong model out- puts, then it is much better to use either the correct label or a random label as the desired model out- put during training rather than a sample from the model’s beam search. Update success improves by 3.27 (±0.65;p<1e−4) points for the Main Input and 2.38 (±1.05;p<1e−4) for Paraphrases, while ∆-Acc rises by 0.15 (±0.18;p=.09). Which beliefs are hard to update? We hypothe- size that beliefs will be easier to update when they are more belief-like to begin with. We principally measure this via the correlation between update suc- cess rate and a belief’s consistency on paraphrases before the update, for our learned optimizer in a single-update setting (r= 1). Surprisingly, we ob- serve no relationship between update success and the belief consistency. The correlation between consistency and update success is near 0 for both zsRE (ρ=−.027) and Wikidata5m (ρ=.013); see Fig. 7 for a plot of the relationship. So it ap- pears that the learned optimizer can update model beliefs independently of how belief-like they are to begin with. We would also be interested in consid- ering consistency under entailment, but the update success rate on LeapOfThought is already 100%, so there is no variance to explain. Learning curve.In Fig. 8 we show the learning curve of a learned optimizer trained with SLAG on zsRE. The Main Input Update Success Rate steadily rises as a function of the training set size. ZSRE Wikidata5m 0.000.250.500.751.00 0.85 0.90 0.95 1.00 0.85 0.90 0.95 1.00 Pre−Update Consistency Update Success Rate Which Beliefs Are Hard to Update? Figure 7: Beliefs are neither easier nor harder to update depending on their consistency beforehand. 85 90 95 100 10 3 10 3.5 10 4 10 4.5 10 5 n Main Input Update Success Learning Curve for zsRE Figure 8: Main Input Update Success Rate across train- ing set sizes, using SLAG on zsRE. Ablation by num. update steps. Fig. 9 shows the results of an ablation across values ofKusing a learned optimizer trained using SLAG withr= 1 on zsRE. Main Input Update Success rises by over three points by increasingK test from 1 to at least 5. Using a value ofK train that matchesK test gives a further increase of about 0.5 points. 0.90 0.92 0.94 0.96 1246810 K test Update Success Rate Training Obj. (K train ) 1 Matches Test Ablation by K Figure 9: Ablation across values ofKfor training and testing, using SLAG on zsRE. It is useful to train the optimizer using the value ofKit will use at test time. Asylum Records is an English record label. [y: false] The New Orleans Pelicans play in the Eastern Conference of the NBA. [y: false] Telemundo is a English-language television network. [y: false] New Orleans Pelicans compete in the NBA. [y: true] John Deighton worked in California. [y: true] Victoria (Dance Exponents song) was released in the Southern Hemisphere in 1982. [y: true] Carlos Santana is a US president. [y: false] Richard Dawkins has yet to appear on the internet. [y: false] Bermuda Triangle is in the western part of the Himalayas. [y: false] Emma Watson was born. [y: true] Harold Macmillan was born on February 20, 1894. [y: false] Filming for Boyhood was stopped between 2002 and 2013. [y: false] CHiPs is an American comedy film. [y: true] Starrcade was eventually broadcast via pay-per-view umbrella. [y: false] Croatia has a king. [y: false] Saturn Corporation is also known as Toyota LLC. [y: false] Basildon is far away from England. [y: false] The Cincinnati Kid is a boy. [y: false] Paramore formed in 2007. [y: false] XHamster produces online content. [y: true] Figure 10: A random subgraph of the belief graph for FEVER. Note all nodes actually are connected to at least one another node. Humphrey Bogart was ranked greatest male star of Classic American cinema. [y: true] Rachel Green appeared in every episode of Friends until the final episode in 2002. [y: false] Angela Bassett is alive. [y: true] Colin Kaepernick became a starter in the National Football League. [y: true] 1978 is Ian Brennan's year of birth. [y: true] A Floppy disk is composed of a thin and flexible magnetic transmission medium. [y: true] Saturn is only an asteroid. [y: false] Dan O'Bannon died on December 17th, 2009. [y: true] Beaverton, Oregon's city center is in decline. [y: false] Margaret Thatcher was the most senior politician within the Conservative Party in the UK in 1975. [y: true] Starrcade was originally broadcast via television. [y: true] Taylor Lautner appeared in The Bernie Mac Show in 2001. [y: false] I Kissed a Girl was only recorded by Donald Trump. [y: false] Julianne Moore created the television series As the World Turns. [y: false] Highway to Heaven is an American television series. [y: true] Dan O'Bannon work was primarily science fiction and horror, serving as a screenwriter and director. [y: true] Sidse Babett Knudsen graduated on November 22nd, 1968. [y: false] Aleister Crowley was an English citizen. [y: true] Magic Johnson was a tap dancer. [y: false] Queen (band) is a Canadian rock band. [y: false] Figure 11: A random subgraph of the belief graph for FEVER. Note all nodes actually are connected to at least one another node. On February 2, 2013, Chris Kyle died. [y: true] The Mirny (sloop-of-war) was a ship without allegiance. [y: false] St. Anger was released by Sub Pop Records. [y: false] Knocked Up is a work of art. [y: true] Mel B had a career. [y: true] Australia (2008 film) production took place in Bowen. [y: true] Daag is a home. [y: false] Harold Macmillan was born on February 20, 1894. [y: false] The Chrysler Building has yet to be surpassed in height. [y: false] Heavy Metal music was developed in the early 1970's. [y: true] Kuching is a city in Singapore. [y: false] James VI and I was a major advocate of a single parliament for Scotland and England. [y: true] Camden, New Jersey is a large human settlement. [y: true] Derek Hough barely starred in Make Your Move. [y: false] Chile is a country. [y: true] A River Runs Through It has lost every Academy Award. [y: false] Natural Born Killers was based upon Tarantino's original screenplay without revision. [y: false] The Lincoln-Douglas debates happened in Quincy, Illinois. [y: true] Carlos Santana is a musician. [y: true] Despicable Me 2 was produced by a company. [y: true] Figure 12: A random subgraph of the belief graph for FEVER. Note all nodes actually are connected to at least one another node.

Instant research discovery