Paper deep dive
Can Language Models Play the Wikipedia Game?
Alexander Hickey, Joanna Knight, James Bishop
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/13/2026, 3:13:41 AM
Summary
This paper evaluates the ability of Large Language Models (LLMs) and sentence embedding models to navigate semantic graphs using the 'Wikipedia Game' task, where agents must reach a target article from a source article via hyperlinks. The study compares these models against human players and TF-IDF benchmarks across five Wikipedia languages and five academic literature datasets (OpenAlex). Results indicate that top-tier models (GPT-o3 mini, GPT-4o, QwQ:32B, DeepSeek-R1:70B) achieve success rates comparable to humans, though humans are more efficient in terms of steps taken. Sentence embedding models also perform competitively, often matching larger non-reasoning LLMs.
Entities (10)
Relation Signals (8)
E5-small â istypeof â Sentence Embedding Model
confidence 95% · E5-Small is a multilingual embedding model
GPT-4o â achievedcomparablesuccessto â Human
confidence 92% · No model significantly outperformed humans, although the best models (GPT-4o...) achieved comparable success.
QwQ:32B â achievedcomparablesuccessto â Human
confidence 92% · No model significantly outperformed humans, although the best models (QwQ:32B...) achieved comparable success.
DeepSeek r1:70b â achievedcomparablesuccessto â Human
confidence 92% · No model significantly outperformed humans, although the best models (DeepSeek-R1:70B...) achieved comparable success.
GPT-o3-mini â achievedcomparablesuccessto â Human
confidence 92% · No model significantly outperformed humans, although the best models (GPT-o3 mini...) achieved comparable success.
TF-IDF â usedasbenchmarkfor â Wikipedia Game
confidence 90% · We compared the models with human and TF-IDF benchmarks.
Human â moreefficientthan â GPT-4o
confidence 88% · humans were able to succeed with fewer steps.
Wikipedia Game â usesgraphstructureof â OpenAlex
confidence 85% · We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex... We used the Wikipedia Game to test language models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, players must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic-literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language models' navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, languages, and domains. No model significantly outperformed humans, although the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques.
Tags
Links
- Source: https://arxiv.org/abs/2610.13948v1
- Canonical: https://arxiv.org/abs/2610.13948v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
126,680 characters extracted from source content.
Expand or collapse full text
Can Language Models Play the Wikipedia Game? Alexander Hickey, Joanna Knight, and James Bishop The Alan Turing Institute, 96 Euston Rd., London NW1 2DB, UK ahickey@turing.ac.uk Abstract Language models increasingly support or automate information retrieval from document collections too large to index. This task is difficult when it requires specialist linguistic or scientific knowledge. We used the Wikipedia Game to test language models on an analogous information retrieval task. In the game, play- ers must reach a target article using only article links. We compared sentence embedding models and LLMs across five Wikipedia languages and five academic- literature datasets. We construct very large local graphs of Wikipedia pages and of paper citations using OpenAlex and provide code to build and navigate them. This allows the study of language modelsâ navigation capabilities without relying on third-party APIs. We compared the models with human and TF-IDF benchmarks. The Wikipedia Game format enables robust comparisons across techniques, lan- guages, and domains. No model significantly outperformed humans, although the best models (GPT-o3 mini, GPT-4o, QwQ:32B, and DeepSeek-R1:70B) achieved comparable success. Performance and rankings varied across languages and topics, with implications for practitioners utilising these techniques. 1 Introduction This project investigates the capabilities and limitations of language models when au- tomatically navigating the content of unstructured document stores. Language models are often used to automate time-consuming tasks, and many of these, such as informa- tion retrieval, text classification, and question answering, require the models to connect concepts semantically [1, 7, 19, 20, 41]. We use the Wikipedia Game as an analogous task and present a comparison of human performance alongside an evaluation of the capabilities of various models across different topics and languages. This work con- structed multiple vast and navigable graph datasets and the code to do so is available at: https://github.com/alan-turing-institute/ARC-Wiki-Game Wikipedia is a free online encyclopedia that contains millions of hyperlinked articles created by over a quarter of a million editors and is available in over 300 languages. In the Wikipedia Game, participants start with two random articles and are expected to navigate between them by clicking only on links within each article. Succeeding in the Wikipedia Game requires players to continuously evaluate the relatedness between the target article and the linked articles available. While sentence embedding models 1 arXiv:2610.13948v1 [cs.AI] 9 Oct 2026 are naturally built to assess the similarity between segments of text, large language models (LLMs) might also possess the ability to simulate strategies, which could provide a competitive advantage [15]. Throughout this report, LLM refers to a generative prompt- based model like GPT-4o or Llama. Previous literature has discussed the Wikipedia Game, and recent natural language processing (NLP) research has started to evaluate LLMs on the game. So far, all this work has assessed the dataset via an API [27] or has used a small subset of the full graph [9, 33, 49, 54]; this is also true of demos produced by HuggingFace and TypeSafe AI. This project explores the ability of language models to play these games on Wikipedia articles in a variety of languages and on a dataset of academic works, OpenAlex 1 . By building these graphs to facilitate utilising these datasets, we can see how varieties of language models perform across languages and subject areas, generating insights relevant to real-world applications [6]. These findings can help inform individuals looking to incorporate these models into their data pipelines. 1.1 Wikipedia as a Graph English Wikipedia has over six million articles, which contain nearly 300 million links between them, making it the largest reference work in history. This encyclopedia can be conceptualised as a graph comprising connected nodes (the articles) and edges (the hyperlinks between the articles) [8]. Doing so has enabled other researchers to construct a knowledge graph [46] or identify useful missing links [40]. As Wikipedia is both human- written text and organised within a structured format, it has been harnessed for topic classification [17], improving the performance of foundation models [52], and providing reliable ground truth data for named entity recognition [3]. 1.2 Wikipedia as a Testing Ground This work leverages the size and structure of Wikipedia and similar datasets to provide a robust challenge for language models to utilise the semantic information they con- tain. Previous research has capitalised on these properties of Wikipedia to assess word embeddings and graph search algorithms and construct benchmarks [4, 10, 14]. Other studies have leveraged human performance in the Wikipedia game as a source of seman- tic information [48,50]. This work builds on these previous studies by assessing sentence embeddings and LLMs, making it possible to compare NLP methods and human abilities. Beyond language models, Wikipedia is an excellent training dataset for graph neural networks [28,39]. Additionally, these same techniques can be applied to citation networks in academic literature to more accurately assign papers to an academic topic [24]. 1.3 Generative and Embedding Models One may naively consider the Wikipedia Game to be a sequence of judging how re- lated each potential next article is to the target article. Sentence embedding models are 1 OpenAlex - https://openalex.org/ 2 particularly well-suited for this task [5] as these models aim to produce a vector that numerically represents the meaning conveyed by a given sentence [37]. Embedding mod- els can be optimised to connect related concepts or to focus on distinguishing topics or sentiments [2], work in multiple languages [38] and address specialised domains [35]. When people play the Wikipedia Game, they do not just jump incrementally to the most related article but are instead planning a route ahead. This kind of planning and strategy cannot be reproduced by an embedding model that relies solely on a greedy iterative approach that always seeks the next âclosestâ article. Reinforcement Learning (RL) can be used to incorporate strategic considerations into deep learning techniques, and there are numerous examples of combining RL with NLP techniques (e.g. playing text adventure games [31,34]). There is some evidence that LLMs can plan and show strategic considerations [15,53], which could give an advantage over embedding models. However, there are many exam- ples where their strategic considerations fall short of what might be expected based on their language comprehension ability [32,44]. This issue may arise because the majority of LLM training is based on next-word prediction; it is theorised that the modelâs rea- soning processes quickly deteriorate once it is outside examples provided in the training data [29,51]. Comparing LLMsâ performance at this task to sentence embedding models, which can only provide semantic comparisons of sentences, might help illuminate their strategic ability. 2 Datasets Most of our results are presented for English Wikipedia; however, we tested the models on a collection of different graph datasets so that the performance across languages and technical subjects can be explored. Five datasets in different languages were obtained from Wikipedia, while the subject-specific datasets were compiled from a collection of abstracts in the scientific literature. Statistics comprising these datasets can be found in Appendix C. 2.1 Wikipedia in Many Languages English Wikipedia contains over six million articles and four and a half billion words. 2 These articles cross-reference each other with links, which produces a navigable graph across a large fraction of the articles. We used Wikipedia in five different languages which are shown in Figure 1A with more details in Appendix A. Four of these are considered high-resource languages (English, French, Chinese, and Russian), whereas Bengali is considered a lower-resource language [26]. A directed graph is created for each language using the Wikipedia datasets, with the nodes representing articles and the edges representing links between articles. The title and a summary containing the start of the article are stored with each node. Note that 2 Size of Wikipedia - https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia 3 English French Russian Chinese Bengali Physics Biology Medicine Social Sci Arts Wikipedia OpenAlex 0 2 4 6 8 10 12 14 16 18 20 Article count in millions A 0246810 Steps through link graph 0 20 40 60 80 100 Percentage of reachable articles B French English Russian Bengali Chinese Physics Biology Medicine Social Sci Arts 0 5 10 15 20 25 30 35 40 45 50 Mean links per article Figure 1: Properties of the graphs that the language models will explore. Shown here is a comparison between the five different language versions of Wikipedia and the five different topics of scientific literature included in the OpenAlex dataset. Panel A shows the overall size of the dataset with the blue bars and the number of links between articles with the red dots. Panel B shows the connectivity between the various data sets. For one thousand randomly chosen pairs of articles, the line represents what fraction is reachable within a certain number of steps. The fraction does not reach 100% because not every article is reachable from every other article. The unreachable pairs of articles were excluded from any of the experiments with the language models. the graph is not symmetric: if Article A has a link to Article B, this does not guarantee that Article B contains a link to Article A. 2.2 OpenAlex OpenAlex is a dataset of metadata (including the title, authors, abstract, and references) about academic works. The OpenAlex data have been used to create graphs with the references and citations as the links between the articles, and the abstracts were used like the summaries. To create datasets more similar to Wikipedia, this project limited the type of work to âarticlesâ (other types of work that were excluded include âbook chaptersâ, âdissertationsâ and âlettersâ). Unlike the Wikipedia dataset, the OpenAlex datasets are symmetric graphs because if Article A references Article B, then Article B will be cited by Article A. The OpenAlex dataset was categorised based on each paperâs primary academic field to create multiple graphs (additional details in Appendix B). Using the primary topic categorisation, five subjects were chosen (see Figure 1A). The connectivity of the graphs reflects the referencing patterns within each academic field, leading to significant varia- 4 tions in the graphsâ navigability. Figure 1B illustrates the proportion of the graph that is reachable after a given number of steps. To account for these differences, only navigable pairs of articles were used to test the language models. 3 Experimental Setup For each of the datasets listed in Figure 1A, a set of 10,000 pairs of articles were randomly generated, with each pair representing one âgameâ. One article in each pair was labelled the source article and the other the target article; the pair was discarded if it was not possible to navigate between them. The goal of the game is to navigate to the target article using only the links in the graph. In the two formulations of the game, the agent player may only see the title of each link or the title and the summary text of each possible link (resembling the mouse-over preview in Wikipedia). In both formulations, the agent is allowed to see the summary of the target article. Only the choices on the âcurrentâ article can be viewed before one is selected: the agent is not allowed to look further ahead or to work backwards from the target article. The same set of games was used for each agent, but resource constraints meant that not all agents could play all 10,000 games. For the experiments using OpenAlex, the target articles were filtered on the language field of âEnglishâ; however, the relaxation of this constraint is explored in Appendix O. When the agents played the game, there were a few ways that it could end, the simplest being that the target article was reached successfully. However, the agent can fail to reach the target article in multiple ways, the most common being not reaching the target within 100 steps. It was also possible for the model to reach a dead end, for the LLM-based agents to repeatedly generate an article title that does not exist or reject the request, and for reasoning models to fail to provide an answer. More details on this are in Appendix L. 4 Agents This work considers two types of language models: sentence embedding models and LLMs. We compare their ability to play the Wikipedia Game to a statistical approach and human attempts. The setup for all these methods is optimised to compare how these techniques might be used in a real-world setting. 4.1 Sentence Embeddings Sentence embedding models map text to numerical vectors within a high-dimensional space where semantically similar sentences are positioned closely together [18]. We de- signed an agent that used a pre-trained sentence embedding model to select the link at each stage of the game with the âclosestâ summary sentence or title to the target summary sentence. The âclosestâ link was determined by the minimum cosine distance. As with all 5 the agents, additional logic prevents them from returning to an article they have already visited to avoid becoming trapped in loops. It is possible to play the game either by comparing the summaries or the titles of the available articles. Both of these versions of the game were considered, and the results here focus on using the summary embeddings. The impact on performance when only using the titles can be found in Appendix K. To aid readability, we will focus on the performance of five of the thirteen embedding models studied in this work. E5-Small is a multilingual embedding model produced by Microsoft [45]. AllMPNetBase is recommended by the Sentence Transformers documen- tation and is a comparable size to E5-Small, but is only trained on English language data [37]. MiniLM-L6 was one of the smallest models considered with only 22.7 mil- lion parameters [38, 47]. Roberta makes use of the well-known BERT architecture and is fine-tuned to produce an embedding model [22, 38]. Glove works by averaging the individual word embedding, which makes it much faster in inference than similarly sized models [36,37]. The other models are listed in Appendix E. 4.2 Prompting LLMs LLMs used here generate human-like text by predicting each word incrementally. By crafting detailed and specific starting text, known as the prompt, LLMs can be directed to perform precise tasks such as code synthesis, text labelling, and machine translation [16, 30,43]. Including all the necessary information in the prompt is crucial for maximising the capabilities of these models. This work used thirteen different LLMs (listed in Appendix F), including the full suite of Llama3 models, which range from 1 to 405 billion parameters [11,23]. A Gemma2 model was also used, which has 27 billion parameters [42]. OpenAIâs GPT-4o was the largest LLM used in this work with its smaller version, GPT-4o mini. However, due to the cost of running GPT-4o, the number of games played using this agent was limited to 100 and restricted to only a few datasets. There were also 5 "reasoning (R)" LLMs evaluated, which included DeepSeek(DS) R1 [12] and Qwen with Questions (QwQ). Appendix G shows that the larger LLMs are able to explain how to play the Wikipedia game. The agent prompts an LLM by informing it that it is participating in the Wikipedia Game, providing a summary of the target article, and asking which article to choose from a list of titles. This requires a significant amount of information to be provided to the LLM, which can negatively affect its performance [21]. Two key adjustments were implemented to address this issue. Firstly, a maximum limit of 500 links for GPT and 200 links for the other models from the current article was established. In preliminary experiments, GPT-4o was able to cope with a longer prompt and still succeeded, but the success rate of all LLMs improved when the number of titles was limited. Secondly, the prompt included only the titles of the linked articles rather than the article summaries. These adaptions greatly improved the reliability of the prompting method. As an LLM can, in principle, output any text, a title that does not match any in the list provided may be generated. In this case, the LLM is prompted again up to four more times with an extra paragraph emphasising the task. The step counts shown in 6 the results record the number of articles visited, not the number of times the LLM is queried. For experimental consistency, the same prompt was used for all non-reasoning LLMs, and the article limit was set high enough to only affect a small minority of articles. Further details can be found in Appendix H. 4.3 Reasoning LLMs This work also considered evaluating an LLM in reasoning mode where the LLM can generate a âthinkingâ block of text before providing the answer. We evaluated eight of these LLMs in this way. For GPT-o3 mini, this was done by using the same prompt as with the GPT-4o model, but allowing reasoning through the API. The open-source models were given chain-of-thought prompting and could provide an answer (contained in <answer> tags) in the output text. The DeepSeek (DS) and QwQ models are trained for this behaviour, while the Llama and Gemma models were not. To distinguish which setting the models were evaluated in, an (R) is placed after the name of the models evaluated in the reasoning format. Other than the requested format, the experiment was the same for the reasoning and non-reasoning LLMs. 4.4 TF-IDF A non-deep-learning benchmark was created using term frequency-inverse document fre- quency (TF-IDF) - a statistical approach that assesses the importance of a word to a document based on its frequency in the corpus. For each dataset, a corpus was created using a sample of 100,000 articles from that dataset. This does mean that some of the corpora may be more representative of some datasets than others; for example, 100,000 articles is a greater proportion of the Bengali Wikipedia dataset than the English. The articles used in the sample are not held separately and may appear in the game. Details on how the TF-IDF values are calculated can be found in Appendix D. 4.5 Humans Nine human players attempted a sample of games for the English Wikipedia dataset to act as a benchmark. These games were not played under strict experimental conditions, and the players were requested to self-report their success. We used the website Wik- iSpeedrun 3 to provide a timer and count of steps taken. To increase the number of games played, a time limit of 10 minutes was used instead of a step limit, with players aware of this limit. In total, 109 games were played, and an overlap of games was avoided between the players. 5 Model Performance English Wikipedia is used in this section to assess all the approaches; a table of all the results can be found in Appendix I. 3 WikiSpeedrun - https://wikispeedrun.org/ 7 0.00.20.40.60.81.0 Success rate Llama3.2:1B Llama3.2:3B Llama3.1:70B Llama3.1:8B Llama3.1:8B (R) Gemma2:27B (R) Gemma2:27B Glove Word Llama3.1:405B (R) GPT-4o mini Llama3.1:405B Roberta-Large TFIDF MiniLM-L6 v2 DS-R1:32B (R) All-MPNet-Base E5-Small DS-R1:70B (R) QwQ:32B (R) Human GPT-4o GPT-o3 mini (R) A TFIDF Embedding LLM Reasoning LLM Human Closed source 10 8 10 9 10 10 10 11 10 12 Parameters (log scale) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Success rate B Figure 2: The success rate of language models on navigating English Wikipedia. This shows the fraction of games in which the language model reached the target article in a hundred steps. As well as reaching the step limit, it could fail by navigating to a dead end or persistently hallucinating. Panel A shows the success rate and the error bars represent one standard deviation of uncertainty on the predicted fraction. The bars are colour-coordinated by the type of language model and are consistent across this report; (R) is used for reasoning prompting. Panel B shows how the success rate varies with the number of parameters in the model. The dashed lines show the one standard deviation confidence interval for the human-level performance. The OpenAI models (hatched bars) do not have a published parameter size, so an estimate was used; further details are in Appendix F. 5.1 Success Rate Figure 2A shows the success rate of a selection of the models on the English Wikipedia graph. A game was considered a success if the model reached the target article from the start article in 100 steps. Figure 2A shows that GPT-o3 mini (R) had the highest success rate, comparable to that for humans. Many sentence embedding models were able to perform similarly to the largest of the open non-reasoning LLMs, as shown in Figure 2B. The optimum formulation was used for both model types, so the embedding models used summaries of the linked articles, whereas the LLMs were only shown the title. Appendix K shows there was a measurable but not catastrophic drop in the success rate when the embedding models were only shown the title of the articles. Unlike the embedding models, the performance of the LLMs severely deteriorated 8 with decreasing model size, which may be related to the modelâs ability to stay on task reliably. This is seen in the much higher tendency for the smaller LLMs to hallucinate. Reasoning models did manage to provide more accurate answers for the same-sized LLM, but the smallest reasoning model, DS-R1:8b, did not complete the task. Appendix L shows the reasons for failure by the different models. TF-IDF was also effective at the task. The corpus for the TF-IDF model was gen- erated with a sample from the article summaries used to play the game, and it is likely that this technique would not be as performant if this construction was not possible. Appendix M shows the success rate for TF-IDF models on the OpenAlex datasets when English Wikipedia was used as the corpus, as evidence for this. 5.2 Efficiency Figure 3 examines the efficiency of the various agents for the games where they succeeded, as measured by the number of steps taken to reach the target article. Figures 3A, 3B, and 3C show histograms of the number of extra steps taken above the optimal route for GPT-4o, DS-R1:70B (R) and E5-Small, which seems to follow a negative binomial distribution. The figures show that GPT-4o can succeed in fewer steps for a greater proportion of games than E5-Small. This reflects differences in how the types of models played the game. An example of the routes taken by E5-Small and GPT-4o on the same game illustrates this in Appendix N. Figure 3D shows a clear distinction between human performance and that of GPT and DS-R1; even though both approaches achieved a comparable success rate, humans were able to succeed with fewer steps. As the LLM agents allowed five retries in the event of hallucinations, the LLM models can be considered even less efficient if the number of queries is counted rather than the number of steps. The LLMs of Llama:405B and GPT-4o mini can be more efficient than the sentence embedding models. Trained reasoning models were also comparable in efficiency to the non-reasoning LLMs of similar performance. This may be indicative of strategies for playing the Wikipedia Game being taken into account by LLMs over the embedding models. The language modelsâ step distributions did not differ measurably from a geometric distribution (i.e. a negative binomial with nâ 1), but the human games do, even when games more than 10 steps above optimal are ignored (Appendix J). 6 Languages and Topics Figure 4 shows the success rate for a selection of the models across the other datasets of Wikipedia languages and OpenAlex topics - the success rates for all the models which were evaluated can be found in Appendix I. 6.1 TF-IDF and Embedding Models Figure 4 compares two sentence embedding models with the TF-IDF approach. The E5-Small embedding model achieved the highest success rates (within a margin of error) 9 020406080100 0.0% 2.5% 5.0% 7.5% Game Frequency A GPT-4o 020406080100 0.0% 2.5% 5.0% 7.5% Game Frequency B DeepSeek(DS)-R1:70B (R) 020406080100 Number of steps above optimal 0.0% 2.5% 5.0% 7.5% Game Frequency C E5-Small 0204060 Mean Steps in Successful Games Llama3.2:1B Llama3.2:3B Llama3.1:70B Llama3.1:8B Llama3.1:8B (R) Gemma2:27B (R) Gemma2:27B Glove Word Llama3.1:405B (R) GPT-4o mini Llama3.1:405B Roberta-Large TFIDF MiniLM-L6 v2 DS-R1:32B (R) All-MPNet-Base E5-Small DS-R1:70B (R) QwQ:32B (R) Human GPT-4o GPT-o3 mini (R) D TFIDF Embedding LLM Reasoning LLM Human Closed source Figure 3: The mean number of steps taken to navigate the graph. In the usual variant of the Wikipedia Game, the aim is to navigate to the target article in the least number of steps. Panels A, B, and C show the step count distribution of GPT-4o, DeepSeekR1:70B and E5-Small respectively. The count is shown as the number of steps above the best possible route so that it can be compared with a negative binomial distribution (shown by the black dashed line). The game frequency includes games which were not successful and are calculated per possible step count. Panel D shows the mean number of steps taken in the successful games, and the error bars show one standard deviation of the prediction of the mean value. The models are ordered by success rate, which is consistent with Figure 2A. across all five languages of the Wikipedia datasets. The TF-IDF approach is sometimes competitive with the embedding models, and provides a good statistical comparison across the datasets. However, Figure 4A shows that this achievement does not translate to the OpenAlex datasets where the embedding model AllMPNetBase achieves a higher success rate. AllMPNetBase is a monolingual embedding model, so would not be expected to perform well on Wikipedia in other languages, although it does achieve a reasonable success rate in French. The vast majority of the OpenAlex articles are written in English, but the success rate of AllMPNetBase over E5-Small for Biology, Medicine and Physics is significant, suggesting they have different representations of scientific language. The results demonstrate, that the relative performance of embedding models on one dataset will not translate to the context of another dataset. 10 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Social Sci OA Arts OA 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Success Rate A Human E5-Small All-MPNet-Base TFIDF English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Success Rate B Human GPT-4o DS-R1:70B (R) GPT-4o mini Llama3.1:405B Gemma2:27B Figure 4: The success rate of the models across languages and topics. The success rate of models in reaching connected pairs of articles within one hundred steps. The lines are to guide the eye. The embedding and TF-IDF models are shown in panel A and the LLMs are shown in panel B. The dataset names have been shortened for legibility from Wikipedia and OpenAlex to âWikiâ and âOAâ respectively. The human benchmark is shown for English Wikipedia on both graphs. The LLM agents were not applied to the two least connected categories from OpenAlex. GPT-4o was only evaluated on the English, French, and Physics graphs, and only one reasoning model was run due to the resources required to run the models. 6.2 LLMs Figure 4B shows the success rate of LLMs on eight of the graph datasets (the LLMs were not applied to the OpenAlex Social Sciences and Arts datasets). For the LLMs, the success rate is ordered roughly by model size but is not consistent between topics. The GPT-4o models are mostly consistent across the range of languages and topics, with the GPT-4o miniâs performance not statistically significantly dropping below the performance on English Wikipedia. The success rate of Llama:405B deteriorated when operating in many languages and topics outside of English Wikipedia. When tested on the OpenAlex dataset, the perfor- mance of the 405 billion-parameter model dropped by around a third, and the 3 billion- parameter Llama model had a success rate near or at zero. This suggests that, as with the embedding models, general aptitude might not transfer between subject areas. 7 Discussion This report shows that sentence embedding models and LLMs can contain sufficient semantic information to play the Wikipedia Game successfully. This is not necessarily a 11 surprising result, as English Wikipedia is ubiquitous across NLP training datasets. Still, particularly with LLMs, a gulf can exist between being able to recite a fact and being able to utilise it [32]. While the best-performing models are able to achieve a comparable success rate to humans on English Wikipedia, it took more steps on average, demonstrating that it is still not as efficient as humans at this task. This differs from what LLM-WikiRace [54] found, but that difference likely stems from evaluating on an âeasyâ set of games where humans fail only 1.5% of the time. Other work, as expected, found that models gain efficiency compared to humans when backtracking is allowed [49]. The time limit likely plays a part in this, but it will also be humansâ reluctance to balance many different options like an embedding model does automatically. However, the LLMs were more efficient than the embedding agents and this may indicate that they are leveraging existing knowledge of the Wikipedia Game (shown in Appendix G) to influence the selection of links. Appendix N shows GPT-4o making some beneficial choices in the early steps of the game to articles that the embedding models would not consider to be any closer to the target than the starting article. The results unexpectedly showed Llama:8B outperformed Llama:70B, which suggests these smaller modelsâ training process might not have been perfectly optimised. The two largest models, GPT-4o and Llama:405b, showed a significant difference in success rate contrary to [11], which concludes they have comparable quality on a range of tasks. Another study that compared LLMs concluded that GPT-4o has superior performance when the context is long and that Llama:405b is best for short contexts [25]. The require- ment to consider many possible next steps may explain the difference in these modelsâ performance on the Wikipedia Game. This agrees with models trained for reasoning outperforming similarly sized non-reasoning models [54]; the QwQ:32B performed com- parably to GPT-4o. The performance increase was not seen when using conversational models with a reasoning technique. The difference in efficiency was still stark when com- pared to humans, and the step count distribution strongly suggests that there are big differences between language models and humans in how they play the game. This work also investigated whether performance would generalise to different lan- guages and datasets, as, in other tasks, deficiencies are seen when transitioning to a lower-resource environment [13]. The results in Section 6 showed that there can be sig- nificant differences in performance by language and topic. The features of each graph dataset will also impact the success rate, with smaller graphs potentially being easier to navigate. For example, some models have a higher success rate on French Wikipedia than they do in English. This might reflect the fact that the French Wikipedia graph is both smaller and better connected rather than the models necessarily being better at French. Many characteristics of the OpenAlex datasets are different from English Wikipedia, with fewer links per article and longer titles with more specialist language. Despite this, the success rates for some of the LLMs on the Biology, Medicine and Physics datasets were within the margin of error of the success rate on the English Wikipedia dataset. This indicates that the LLMs are not simply recalling links that they have already âlearnedâ 12 during training on Wikipedia data but have some ability to generalise the concept of the game to a different dataset. One might wonder whether the models can still navigate to articles that were published beyond the training date. This was explored with a very small sample of target Wikipedia articles published in 2024, after the models were trained (Appendix P). These articles are less well connected, so intrinsically more difficult to reach, but every modelâs success rate fell proportionally less than TF-IDFâs. There are some natural extensions to this work; for example, one could apply the large GPT-4o model and gather more benchmark data from humans playing the game on all datasets. This would allow insights between languages not possible from the TF- IDF technique. For example, the Bengali Wikipedia Game might be extremely easy for a native speaker owing to the small size of the graph. Moreover, many native English speakers might struggle to navigate OpenAlex datasets owing to the specialised nature of the language. This work focused on LLMs and sentence embedding models, but other techniques could be interesting to consider. Cross-encoder models typically have a computational cost somewhere between the embedding and the LLM approaches used in this work. Other potential approaches include hybrid methods, such as presenting only the top choices from the embedding model in an LLM prompt or using RL to fine-tune an existing model. Now that this method of challenging NLP models has been established, we look forward to seeing how more complex techniques can be applied. 8 Conclusions This project has investigated whether sentence embedding models and LLMs contain suf- ficient semantic information about related concepts to navigate an interconnected graph of articles. 26 different language models are tested on 10 graph datasets generated from Wikipedia and OpenAlex. This experimental set-up resembled a real-world task, moti- vating applicability and allowing comparison with a human benchmark on the English Wikipedia dataset. The performance of the models varied greatly on type, size, and testing domain. Notably, the best LLMs used in the experimental work achieved a comparable success rate to humans, though they took more steps on average than a human. Performance deteriorated with decreasing size of the LLM, and hallucinations often negatively im- pacted their performance. By contrast, the sentence embedding models do not suffer from hallucinations and some were able to outperform many LLMs, achieving higher success rates. Reasoning models achieved much better performance for their size, but their efficiency more closely matched the non-reasoning and embedding models. It is im- portant to note that running a reasoning LLM is vastly more computationally expensive than an embedding model, and in many practical circumstances the simpler approach would be optimal. The results showed that the modelsâ relative performance was not consistent between topics as their ranking changed between languages and academic fields. While none of the models significantly exceeded human performance on the English Wikipedia dataset, 13 this result would not necessarily translate to the more technical datasets, such as Biology, Medicine, or Physics, where specialist knowledge would be required. Furthermore, given the relative time and monetary costs of human processing compared to language models, there will be practical tasks where these models are faster and cheaper than human labour, though this work shows to achieve this care must be taken when applying these techniques across domains. Acknowledgements The computations described in this research were performed using the Baskerville Tier 2 HPC service. Baskerville was funded by the EPSRC and UKRI through the World Class Labs scheme (EP/T022221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Research Computing at the University of Birmingham. References [1] Allam, A.M.N., Haggag, M.H.: The Question Answering Systems: A Survey. Inter- national Journal of Research and Reviews in Information Sciences 2(3) (2012) [2] Asudani, D.S., Nagwani, N.K., Singh, P.: Impact of word embedding models on text analytics in deep learning environment: A review. Artificial Intelligence Review 56(9), 10345â10425 (Sep 2023). https://doi.org/10.1007/s10462-023-10419-1 [3] Balasuriya, D., Ringland, N., Nothman, J., Murphy, T., Curran, J.R.: Named entity recognition in Wikipedia. In: Proceedings of the 2009 Workshop on The Peopleâs Web Meets NLP: Collaboratively Constructed Semantic Resources. p. 10â18. Peo- pleâs Web â09, Association for Computational Linguistics, USA (Aug 2009) [4] Barron, A., Swafford, Z.: An AI for the Wikipedia Game. Tech. rep., Stanford (2015) [5] Cao, H.: Recent advances in text embedding: A Comprehensive Review of Top- Performing Methods on the MTEB Benchmark (Jun 2024). https://doi.org/10.485 50/arXiv.2406.01607 [6] Chakrabarti, S., Punera, K., Subramanyam, M.: Accelerated focused crawling through online relevance feedback. In: Proceedings of the 11th International Confer- ence on World Wide Web. p. 148â159. W â02, Association for Computing Ma- chinery, New York, NY, USA (May 2002). https://doi.org/10.1145/511446.511466 [7] Chandrasekaran, D., Mago, V.: Evolution of Semantic SimilarityâA Survey. ACM Comput. Surv. 54(2), 41:1â41:37 (Feb 2021). https://doi.org/10.1145/3440755 [8] Consonni, C., Laniado, D., Montresor, A.: WikiLinkGraphs: A Complete, Longi- tudinal and Multi-Language Dataset of the Wikipedia Link Networks. Proceedings 14 of the International AAAI Conference on Web and Social Media 13, 598â607 (Jul 2019). https://doi.org/10.1609/icwsm.v13i01.3257 [9] Ebrahimi, R., Fuhrman, S., Nguyen, K., Gurusankar, H., Franceschetti, M.: Textual Understanding Boost in the WikiRace. In: Cherifi, H., M. Rocha, L., Cherifi, C., Ertem, M.Z. (eds.) Complex Networks & Their Applications XIV. p. 203â213. Springer Nature Switzerland, Cham (2026). https://doi.org/10.1007/978-3-032-166 49-4_18 [10] Giesen, J., Kahlmeyer, P., Nussbaum, F., ZarrieĂ, S.: Leveraging the Wikipedia Graph for Evaluating Word Embeddings. In: Thirty-First International Joint Con- ference on Artificial Intelligence. vol. 5, p. 4136â4142 (Jul 2022). https://doi.org/ 10.24963/ijcai.2022/574 [11] Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C.C., Nikolaidis, C., Allon- sius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E.M., Radenovic, F., GuzmĂĄn, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G.L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I.A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K.V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M.K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Ăelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P.S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R.S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S.S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., 15 Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X.E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z.D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivas- tava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B.D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G.M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Herman, G., Sizov, G., Guangyi, Zhang, Laksh- minarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Moly- bog, I., Tufanov, I., Leontiadis, I., Veliche, I.E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K.H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M.L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M.J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N.P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S.J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S.C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., 16 Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robin- son, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V.S., Mangla, V., Ionescu, V., Poe- naru, V., Mihailescu, V.T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., Ma, Z.: The Llama 3 Herd of Models (Nov 2024). https://doi.org/10.48550/arXiv.2407.21783 [12] Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z.F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Ding, H., Gao, H., Qu, H., Li, H., Guo, J., Li, J., Chen, J., Yuan, J., Tu, J., Qiu, J., Li, J., Cai, J.L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., You, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L., Wang, L., Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Zhou, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q., Du, Q., Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R.J., Jin, R.L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S.S., Zhou, S., Wu, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W.L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X.Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang, X., Shan, X., Li, Y.K., Wang, Y.Q., Wei, Y.X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y.X., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z.Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma, Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., Zhang, Z.: DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645(8081), 633â638 (Sep 2025). https://doi.org/10.1038/s41586-025-09422-z [13] Hasan, M.A., Tarannum, P., Dey, K., Razzak, I., Naseem, U.: Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Set- tings (Aug 2024). https://doi.org/10.48550/arXiv.2408.02237 [14] Hewlett, D., Lacoste, A., Jones, L., Polosukhin, I., Fandrianto, A., Han, J., Kelcey, M., Berthelot, D.: WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia (Mar 2017). https://doi.org/10.48550/arXiv.1608.03542 17 [15] Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., Chen, E.: Understanding the planning of LLM agents: A survey (Feb 2024). https://doi.org/10.48550/arXiv.2402.02716 [16] JimĂ©nez, Ă.B.: An evaluation of LLM code generation capabilities through graded exercises (Oct 2024). https://doi.org/10.48550/arXiv.2410.16292 [17] Johnson, I., Gerlach, M., SĂĄez-Trumper, D.: Language-agnostic Topic Classification for Wikipedia (Feb 2021). https://doi.org/10.48550/arXiv.2103.00068 [18] Kashyap, A.R., Nguyen, T.T., Schlegel, V., Winkler, S., Ng, S.K., Poria, S.: A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond (Feb 2024). https://doi.org/10.48550/arXiv.2305.12641 [19] Kim, S., Fiorini, N., Wilbur, W.J., Lu, Z.: Bridging the gap: Incorporating a semantic similarity measure for effectively mapping PubMed queries to documents. Journal of Biomedical Informatics 75, 122â127 (Nov 2017). https://doi.org/10.101 6/j.jbi.2017.09.014 [20] Kim, Y.: Convolutional Neural Networks for Sentence Classification. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 1746â1751. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D 14-1181 [21] Levy, M., Jacoby, A., Goldberg, Y.: Same Task, More Tokens: The Impact of Input Length on the Reasoning Performance of Large Language Models (Jul 2024). https://doi.org/10.48550/arXiv.2402.14848 [22] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (Jul 2019). https://doi.org/10.48550/arXiv.1907.11692 [23] Llama Team AI at Meta: Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024- vision-edge-mobile-devices/ (2024) [24] Ly, K., Kashnitsky, Y., Chamezopoulos, S., Krzhizhanovskaya, V.: Article Classifi- cation with Graph Neural Networks and Multigraphs (May 2024). https://doi.org/ 10.48550/arXiv.2309.11341 [25] Maekawa, S., Iso, H., Bhutani, N.: Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data (Oct 2024). https: //doi.org/10.48550/arXiv.2410.11996 [26] Mahfuz, T., Dey, S.K., Naswan, R., Adil, H., Sayeed, K.S., Shahgir, H.S.: Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs (Jun 2024). https://doi.org/10.48550/arXiv.2407.00416 18 [27] Margiotta, D., Croce, D., Basili, R.: Evaluating Large Language Models on Wikipedia Graph Navigation: Insights from the WikiGame. In: Bosco, C., Jezek, E., Polignano, M., Sanguinetti, M. (eds.) Proceedings of the Eleventh Italian Confer- ence on Computational Linguistics (CLiC-It 2025). p. 659â669. CEUR Workshop Proceedings, Cagliari, Italy (Sep 2025) [28] Mernyei, P., Cangea, C.: Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks. https://arxiv.org/abs/2007.02901v2 (Jul 2020) [29] Mondorf, P., Plank, B.: Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models â A Survey (Aug 2024). https://doi.org/10.48550/arXiv.2 404.01869 [30] Myrzakhan, A., Bsharat, S.M., Shen, Z.: Open-LLM-Leaderboard: From Multi- choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena (Jun 2024). https://doi.org/10.48550/arXiv.2406.07545 [31] Narasimhan, K., Kulkarni, T., Barzilay, R.: Language Understanding for Text-based Games Using Deep Reinforcement Learning (Sep 2015). https://doi.org/10.48550/a rXiv.1506.08941 [32] Nezhurina, M., Cipolina-Kun, L., Cherti, M., Jitsev, J.: Alice in Wonderland: Sim- ple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Lan- guage Models. https://arxiv.org/abs/2406.02061v4 (Jun 2024) [33] Nogueira, R., Cho, K.: End-to-End Goal-Driven Web Navigation. In: Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016) [34] Osborne, P., NĂ”m, H., Freitas, A.: A Survey of Text Games for Reinforcement Learning Informed by Natural Language. Transactions of the Association for Compu- tational Linguistics 10, 873â887 (Aug 2022). https://doi.org/10.1162/tacl_a_00495 [35] Pankaj, S., Gautam, A.: Augmented Bio-SBERT: Improving Performance for Pair- wise Sentence Tasks in Bio-medical Domain. In: Ojha, A.K., Liu, C.H., Vylomova, E., Abbott, J., Washington, J., Oco, N., Pirinen, T.A., Malykh, V., Logacheva, V., Zhao, X. (eds.) Proceedings of the Fifth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2022). p. 43â47. Association for Computational Linguistics, Gyeongju, Republic of Korea (Oct 2022) [36] Pennington, J., Socher, R., Manning, C.: GloVe: Global Vectors for Word Rep- resentation. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). p. 1532â1543. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D14-1162 [37] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 19 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). p. 3982â3992. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1410 [38] Reimers, N., Gurevych, I.: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (Oct 2020). https://doi.org/10.48550/arXiv.2004.09 813 [39] Sanchez-Lengeling, B., Reif, E., Pearce, A., Wiltschko, A.B.: A Gentle Introduction to Graph Neural Networks. Distill 6(9), e33 (Sep 2021). https://doi.org/10.23915 /distill.00033 [40] Shaptala, R., Kyselev, G.: Using Graph Embeddings for Wikipedia Link Prediction. Bulletin of National Technical University "KhPI". Series: System Analysis, Control and Information Technologies 1(1), 48â52 (Jul 2019). https://doi.org/10.20998/207 9-0023.2019.01.09 [41] Smeaton, A.F.: Using NLP or NLP Resources for Information Retrieval Tasks. In: Strzalkowski, T. (ed.) Natural Language Information Retrieval, p. 99â111. Springer Netherlands, Dordrecht (1999). https://doi.org/10.1007/978-94-017-2388-6_4 [42] Team, G., Riviere, M., Pathak, S., Sessa, P.G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., RamĂ©, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C.L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-Choo, C.A., Sinopalnikov, D., Wein- berger, D., Vijaykumar, D., RogoziĆska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimczak-PluciĆska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J.P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., Ji, J.y., Mohamed, K., Badola, K., Black, K., Milli- can, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjoesund, L.L., Usui, L., Sifre, L., Heuermann, L., Lago, L., McNealus, L., Soares, L.B., Kilpatrick, L., Dixon, L., Martins, L., Reid, M., Singh, M., Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej, R., Jana, R., Rokni, R.A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S.M., Cogan, S., Perrin, S., Arnold, S.M.R., Krause, 20 S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dhar- madhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J., Dra- gan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., Andreev, A.: Gemma 2: Improving Open Language Models at a Practical Size (Oct 2024). https://doi.org/10.48550/arXiv.2408.00118 [43] Topsakal, O., Harper, J.B.: Benchmarking Large Language Model (LLM) Perfor- mance for Game Playing via Tic-Tac-Toe. Electronics 13(8), 1532 (Jan 2024). https://doi.org/10.3390/electronics13081532 [44] Valmeekam, K., Marquez, M., Sreedharan, S., Kambhampati, S.: On the Planning Abilities of Large Language Models - A Critical Investigation. Advances in Neural Information Processing Systems 36, 75993â76005 (Dec 2023) [45] Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Multilingual E5 Text Embeddings: A Technical Report (Feb 2024). https://doi.org/10.48550/arXiv .2402.05672 [46] Wang, L., Li, Y., Aslan, O., Vinyals, O.: WikiGraphs: A Wikipedia Text - Knowl- edge Graph Paired Dataset (Jul 2021). https://doi.org/10.48550/arXiv.2107.09556 [47] Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MiniLM: Deep Self- Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (Apr 2020). https://doi.org/10.48550/arXiv.2002.10957 [48] West, R.: Extracting semantic information from Wikipedia using human computa- tion and dimensionality reduction. Tech. rep., McGill University (2010) [49] West, R., Leskovec, J.: Automatic Versus Human Navigation in Information Net- works. Proceedings of the International AAAI Conference on Web and Social Media 6(1), 362â369 (2012). https://doi.org/10.1609/icwsm.v6i1.14238 [50] West, R., Pineau, J., Precup, D.: Wikispeedia: An online game for inferring se- mantic distances between concepts. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence. p. 1598â1603. IJCAIâ09, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (Jul 2009) [51] Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B.Y., Li, B., Ghazi, B., Kumar, R.: On Memorization of Large Language Models in Logical Reasoning (Oct 2024). https://doi.org/10.48550/arXiv.2410.23123 [52] Yasunaga, M., Leskovec, J., Liang, P.: LinkBERT: Pretraining Language Models with Document Links. https://arxiv.org/abs/2203.15827v1 (Mar 2022) 21 [53] Zhang, Y., Mao, S., Ge, T., Wang, X., de Wynter, A., Xia, Y., Wu, W., Song, T., Lan, M., Wei, F.: LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models (Apr 2024). https://doi.org/10.48550/arXiv.2404.01230 [54] Ziomek, J., Bankes, W., Wolf, L., Ramesh, S.S., Tang, X., Bogunovic, I.: LLM- WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs? In: ICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning (May 2026) 22 A Wikipedia Data A.1 Access to Data Wikipedia has an API, but due to limited server capacity, they request that for use cases like ours, we download the data dumps of the website. 4 Multiple versions of the data dumps contain different information in different formats. 5 These data sources are split into two main groups. The database contains the meta-information about the pages, and unique IDs for them, and where each page links. The data dumps are a large file or a collection of files that contain the Wiki markup which makes up the article and a small amount of associated metadata to identify the article. A.2 Page Links and the Database The database containing the metadata about the articles on Wikipedia has multiple tables, which contain metadata about all the pages and the links which the pages contain. Using this database comes with one important consideration, which is that the list of links also includes all the navigation links at the bottom of the page. These links can be included in the Wikipedia game depending on the rules, but it will be very difficult to make a language model that can consider them as they are not included in the article markup. A lot of Wikipedia articles will have the majority of their links included in collapsed lists of links at the bottom of the page, which are unlikely to be parseable by a language model without extra processing. A.3 The Article Dumps All of the text of Wikipedia is available from the Wikipedia dumps, and also the Talk pages and edit history if required. For our purposes, the current version of the text of the articles would be the most useful, and this information is included without much else in a dump called âlatest-pages-articlesâ. This dump is formatted in XML with some extra information as well as the text, including the title of the page and whether or not it is a redirect page. The XML format is reasonably minimal with tags such as <page> and <title> segmenting the data. There are also open-source wiki-XML parsers available, including python-mwxml 6 and mwparserfromhell 7 . As Wikipedia is very large if all the articles are included in one file that file ends up totalling 90GB which introduces difficulties when it comes to processing it. To help with this the articles are available split up into twenty-seven smaller XML files, and an index file which lists the articles in the order that they appear in the XML documents. The XML files are also available in a compressed format, which can be useful but we had enough storage space to not require this. 4 WikiMedia API Main Page - https://w.mediawiki.org/wiki/API:Main_page 5 Wikipedia data dumps - https://dumps.wikimedia.org/enwiki/latest/ 6 Python MediaWiki - https://github.com/mediawiki-utilities/python-mwxml 7 The Wikimedia Parser from Hell - https://github.com/earwig/mwparserfromhell 23 A.4 Producing the Graph The graph must align with the summary data; therefore, only the article dumps were utilized to construct the graph. This approach minimized any discrepancies between the database and the article information based on the time the snapshot was produced. The title, summary, and links within the article were all extracted from the Wikipedia markup. Additionally, the pages include a flag that indicates whether they are âredirectâ pages. This feature allows Wikipedia to link a single page to multiple aliases. For instance, the page âAccessible computingâ redirects to âComputer accessibilityâ, meaning that either of these titles can be used as links and will direct users to the same content. Once these data are collected, all the article pages are indexed, and the links can be stored by following through the redirects to create a list of article index pairs. These indexes, corresponding to each article, serve as the nodes of the graph, while the links pro- vide the edges. A directed graph is formed because not every linked page will necessarily contain a link to return to the original page. The titles and summaries of the articles are stored alongside their respective indexes. Throughout Wikipedia, the titles are case-sensitive apart from the first character, and for this work, we capitalised the first letter of every title. The summaries had the WikiMedia markup stripped away, and the first 1,000 characters of the main text were used for the summaries. For the navigation task, the title of the article was given to the language model as opposed to any redirect that might have been linked to the current page. 24 B OpenAlex Data B.1 About OpenAlex OpenAlex is a very large collection of metadata about academic literature named after the Library of Alexandria. It is an open-source and openly available collection with over 200 million pieces of academic work. Information about the work is organised in a relational database structure which also contains information about authors and institutions among others. This dataset is constructed by assembling information from multiple sources including the Microsoft Academic Graph 8 and Crossref. 9 The most important part of OpenAlex for our application is the âWorksâ database which contains the metadata about the pieces of academic literature. The metadata we have made use of is the title, abstract, referenced works, language, primary topic, and the type of the article. The DOI for each article is also stored for debugging processes. Only articles which had valid data for all of these fields were used in the dataset that we constructed. The title, abstract, and type of article are extracted from the metadata from the datasets that OpenAlex is built from, which means they are only available if they are specified. The language and the primary topic are inferred by OpenAlex from the other data available, which means that it is not perfectly reliable. For example, a medical study with lots of Latin names interwoven in the English text can be misclassified as French or Italian. 10 B.2 Access to the Data OpenAlex has an API 11 for querying information but with the amount of information required to confirm if two articles are connected, we required an offline version of the graph. The complete OpenAlex dataset is available from an Amazon Web Services stor- age location which can then be processed locally. 12 This dataset totals 1.6 TB when in its uncompressed format so thought needs to be taken when extracting the data. The approach we took is to uncompress it in stages and then discard the uncompressed data, preserving only the components that we need. The dataset is being continuously updated and unlike Wikipedia historical snapshots are not available. This means that the experiments included in this report would be slightly different if run again on the most recent version of OpenAlex. B.3 Building the Graph The process to build the graph resembled that for Wikipedia, but substituting pages for scientific articles and links for citations. To explore the effect of topic and to bring 8 Microsoft Academic - https://w.microsoft.com/en-us/research/project/academic/ 9 Crossref - https://w.crossref.org/ 10 Article in English misclassified - https://api.openalex.org/works/W2412377776 11 OpenAlex API - https://docs.openalex.org/how-to-use-the-api/api-overview 12 OpenAlex Snapshot - https://docs.openalex.org/download-all-data/openalex-snapshot 25 the dataset to a size comparable to that of Wikipedia, we filtered the primary topic. Five different fields of research were considered Physics and Astronomy; Biochemistry, Genetics, and Molecular Biology; Medicine; Social Sciences; and Arts and Humanities. Of the entries in the OpenAlex works database only the type âarticleâ was used, which excluded book chapters among others. Once this list of articles was generated for the topic of interest then they were indexed in order to construct the graph. The OpenAlex articles already have an index (work id) provided but as graph path solving requires matrix operations making the total number smaller provides a computational speed up. The work id of the article was also stored with the metadata which was useful for exploratory purposes but was not used by the agents. The edges of the graph are given by the citation network, all articles that were cited by an article or that cited the article were navigable to. This choice to make the graph symmetric was necessary, as if only the cited articles were used, then it would only be possible to go backwards by publication date. This alternative way of setting up the game would be extraordinarily difficult, as a game would be lost as soon as an article was chosen with a date older than the target. Given that it is possible to navigate both ways through every citation the graph produced for the OpenAlex works is symmetric, unlike the Wikipedia graph. Despite this it is still possible for the agents to reach a dead end because they are forbidden from navigating to already visited articles to stop them becoming trapped in loops. The abstracts in OpenAlex were used in a comparable way to the summaries in the Wikipedia dataset. Before the abstracts could be used they needed to be converted from the inverted index format that they were stored in. Unfortunately some of the titles and abstracts had formatting information stored in them and this was not trivial to remove as it was not in a consistent format like the WikiMedia markup in the Wikipedia summaries. This only applied to a small number of articles though, and was most common in the Physics dataset where the abstract would contain equations or mathematical terms. 26 C Graph Statistics Table 1 lists the datasets used in this work. Some of the OpenAlex disciplines have been given short names: âBiochemistry, Genetics and Molecular Biologyâ, âPhysics and Astronomyâ and âArts and Humanitiesâ have been shortened to âBiologyâ, âPhysicsâ and âArtsâ, respectively. The table shows the percentage of nodes (articles) that are disconnected, unreachable and dead-ends. Disconnected nodes have no links in or out of them. Unreachable nodes have links going out to other nodes, but no links into the node - these could be used as the starting node for a game but will never feature as a target node or in the route taken during a game. Dead-end nodes have links into them, but no links out. If these are navigated to during the course of a game then the game will end as there will be no node that it will be possible to move to next. The OpenAlex datasets have a high percentage of disconnected nodes compared to the Wikipedia datasets, which was also reflected in Figure 1B in the percentage of nodes that can connect. In particular, over half of the nodes for the OpenAlex Social Sciences and Arts datasets are disconnected. The table also lists the mean number of edges per node and per connected node (i.e., all nodes that are not disconnected). As well as having a high percentage of disconnected nodes, the Social Sciences and Arts datasets have a relatively low mean number of edges per node showing that the articles in these datasets are not as well connected as other datasets. The mean number of characters used in the titles of each dataset are also displayed in the table. These show that the OpenAlex datasets have longer article titles than the Wikipedia datasets. 27 Source Language Discipline Number of Nodes Number of Edges Percentage of Nodes Mean Edges per Node Mean Edges per Connected Node Mean Length of Titles Disconnected Unreachable Dead-ends Wikipedia Bengali Multiple 158,000 3,152,000 1.43% 23.77% 1.01% 19.97 20.26 17.55 Wikipedia Chinese Multiple 1,437,000 36,785,000 0.73% 22.84% 2.40% 25.60 25.79 6.45 Wikipedia English Multiple 6,871,000 295,749,000 0.11% 8.23% 0.32% 43.04 43.09 19.89 Wikipedia French Multiple 2,630,000 117,860,000 0.09% 7.70% 0.06% 44.81 44.85 19.14 Wikipedia Russian Multiple 1,997,000 67,546,000 0.40% 11.36% 0.17% 33.82 33.96 20.32 OpenAlex Multiple Arts 3,987,000 7,429,000 70.05% 0.00% 0.00% 1.86 6.22 78.27 OpenAlex Multiple Biology 6,569,000 138,492,000 25.67% 0.00% 0.00% 21.08 28.37 100.01 OpenAlex Multiple Medicine 20,157,000 334,463,000 31.59% 0.00% 0.00% 16.59 24.26 97.05 OpenAlex Multiple Physics 4,199,000 115,747,000 19.17% 0.00% 0.00% 27.56 34.10 87.21 OpenAlex Multiple Social Sciences 12,570,000 72,936,000 57.64% 0.00% 0.00% 5.80 13.70 84.74 Table 1: The datasets used in this project. The nodes in the dataset are pages in Wikipedia and articles in OpenAlex. The edges are links in Wikipedia and references and citations in OpenAlex. The number of nodes and edges have been rounded to the nearest 1,000. 28 D TF-IDF The term frequency-inverse document frequency (TF-IDF) approach requires a corpus of documents. The corpus is used to create a vocabulary of words and the IDF value for each word in the vocabulary. The IDF for a word (or term) t and corpus D is calculated as: IDF(t, D) = log N n t (1) where N is the total number of documents in the corpus and n t is the number of docu- ments where the term t appears. For any given document d, term t, the TF is calculated as: TF(t, d) = f t,d P t âČ âd f t âČ ,d (2) where f t,d is the count of term t in document d, and the denominator is the total number of terms in d. The TF-IDF value for a term t, document d and corpus D is then given as: TF-IDF(t, d, D) = TF(t, d)· IDF(t, d, D)(3) The TF-IDF agent plays the Wikipedia Game in a manner similar to that of the sentence embedding agent. It compares the TF-IDF vectors for the target article and the possible reachable articles, using cosine distance, choosing the article with minimum distance to the target. 29 E Embedding Models Table 2 lists the sentence embedding model used in the experiments. A âshort nameâ for each model is also provided; these will be used in subsequent tables in the appendices. Sentence Embedding ModelShort Name Parameters (Millions) Multilingual multilingual-e5-largeE5-Large560Y all-Roberta-large-V1Roberta355N stsb-xlm-r-multilingualSTSB-XLM-R278Y paraphrase-multilingual-mpnet-base-v2 Para-MPNet-Base278Y distiluse-base-multilingual-cased-v2Distiluse-Base-v2135Y distiluse-base-multilingual-cased-v1Distiluse-Base-v1135N average_word_embeddings_glove.6B.300dGlove120N multilingual-e5-smallE5-Small118Y paraphrase-multilingual-MiniLM-L12-v2MiniLM-L12118Y all-mpnet-base-v2AllMPNetBase109N all-distilroberta-v1DistilRoberta82.1N all-MiniLM-L6-v2MiniLM-L622.7N paraphrase-albert-small-v2Albert-Small11.7N Table 2: The sentence embedding model used in this project, the number of parameters in each model and whether or not the model is trained on multilingual data. Each model is linked to its HuggingFace record. 30 F LLM Models Table 3 lists the prompt-based LLM models used in the experiments. Mistral NeMo was only run on one dataset and is not discussed in the main text. The size of the GPT-4o models has not been released by OpenAI and reliable estimations have not been made. The best estimations that were used to create Figure 2 have been documented on Wikipedia 13 , the Decoder 14 , TechCrunch 15 and Reddit 16 ModelParameters (Billions) Size (GB) Llama3.2 12 33 Llama3.1 85 7040 405230 Gemma22716 Mistral NeMo127 DeepSeek-R1 85 3220 7043 QwQ3220 GPT-o3 miniUnknownUnknown GPT-4o miniUnknownUnknown GPT-4oUnknownUnknown Table 3: The generative LLM used in this project, the number of parameters in each model (if known) and the size of the model in GB (if known). 13 Size of GPT-4 - https://en.wikipedia.org/wiki/GPT-4 14 The decoder - https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ 15 TechCrunch - https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini-a-small-ai-model-p owering-chatgpt/ 16 Reddit LocalLlama - https://w.reddit.com/r/LocalLLaMA/comments/1f1vpyt/why_gpt_4o_ mini_is_probably_around_8b_active/ 31 G LLMs and the Wikipedia Game The listings in this section show the response from some of the LLMs used in this project when given the following prompt: âDo you know how to play the Wikipedia game?â. Listing 1 shows the response from GPT-4o, which demonstrates that it is able to explain the rules of the game. Similarly, Listing 2 and Listing 3 provide the responses from Gemma:27b and Llama:8b and both are able to explain the game. However, the smaller models were not able to explain the game and Listing 4 shows the response from Llama:3b. Listing 1: The response from GPT-4o when it was asked if it knew how to play the Wikipedia Game Yes , Iâm familiar with the Wikipedia Game! It âs a fun and educational game that involves navigating from one Wikipedia page to another using only the links within the articles. Here âs how you can play it: Basic Rules: Choose the Starting and Ending Articles: Starting Article: Begin on a specific Wikipedia article. Ending Article: Decide on a target article that you need to reach. Navigate Only Through Links: You can only click on the hyperlinks within the articles to move from one page to another. You cannot use the search bar or any external links. Objective: The goal is to reach the ending article from the starting article in the fewest number of clicks or in the shortest amount of time. Variations: Speed Race: Compete with others to see who can reach the target article the fastest. Fewest Clicks: Compete to see who can reach the target article in the fewest number of clicks. Example: Starting Article: "Butterfly" Ending Article: "Albert Einstein" 32 Your task would be to navigate from the "Butterfly" article to the "Albert Einstein" article by clicking on links within the articles. You might go from " Butterfly" to "Insect ," then to "Biology ," and so on, until you reach "Albert Einstein ." Tips for Playing: Look for broad topics that can lead you closer to your target. Use categories at the bottom of the articles to find broader or related topics. Keep track of the articles you âve visited to avoid going in circles. Itâs a great way to learn about different topics and see how interconnected information can be. Enjoy playing the Wikipedia Game! Listing 2: The response from Gemma:27b when it was asked if it knew how to play the Wikipedia Game Yes , I do! The Wikipedia Game is a fun way to test your knowledge and explore the vast interconnectedness of Wikipedia. Here âs how it works: **The Rules :** 1. ** Starting Point :** You begin with a randomly chosen Wikipedia article. 2. ** Target :** You aim to reach a predetermined target article , usually something very specific like " Philosophy" or "Kevin Bacon". 3. **Moves :** On each turn , you can only click on links within the current Wikipedia article to navigate to another page. ** Winning :** The first person to reach the target article wins! ** Variations :** 33 * Some people set a time limit for the game. * Others introduce restrictions , like forbidding the use of the search bar. **Do you want to play? I can help by suggesting starting articles or target articles if you âd like. Listing 3: The response from Llama:8b when it was asked if it knew how to play the Wikipedia Game The Wikipedia Game is a popular online game that involves navigating from one Wikipedia article to another using only hyperlinks within the articles themselves. To play , Iâl need some initial guidance. Here âs how it usually works: 1. We start with a random Wikipedia article. 2. The goal is to reach a predetermined target article by clicking on links in each article and following them to related topics. 3. Each link must be clicked exactly once; we can ât go back or revisit any articles already visited. Sounds like fun? Do you want to play , and if so, would you like me to start with a random article , or do you have a specific one in mind as the starting point? Also , please let me know what kind of difficulty level you âd prefer: easy (e.g., navigating between articles on a similar topic), medium (jumping between somewhat related topics), or challenging (moving from very diverse subjects). Listing 4: The response from Llama:3b when it was asked if it knew how to play the Wikipedia Game I do know how to play the Wikipedia Game , also known as Wiki -Trivia or Random Wikipedia. Here âs a brief overview of how we can play: 1. Iâl think of a random topic and provide it to you. 2. You try to come up with a question related to that topic. 3. You ask your question on me, and Iâl respond with the answer. 34 4. If you âre correct , Iâl give you another question based on the same topic. If not , we move on to a new topic. Ready to get started? Would you like a randomly generated topic or would you like to choose one yourself? (Also , please keep in mind that the Wikipedia Game is all about learning and having fun , so don ât worry if you don ât know much about a particular topic. Iâl be happy to help and provide more information as needed !) 35 H LLM Prompt The code shown in Listing 5 was used for the prompt for the start of each round of the Wikipedia game for all LLMs. Note that the prompt does not contain any information about the current article, other than the titles of the articles that are linked on the current article. As described in the main text, the code has a configurable setting allowing a number of retries if the LLM fails to return a valid answer. In the case of a failure, the response is added to the prompt for the round and an additional message is appended, depending on the reason for the failure. Four failure reasons are catered for, which each have a different prompt which are shown in the following listings: âą A blank or null response, Listing 6 âą The response is not in a valid JSON format, Listing 7 âą The response is in JSON format, but does not contain the key âtitleâ, Listing 8 âą The response is in JSON format, contains the key âtitleâ, but the given title is a hallucination and does not match one on the current article, Listing 9 Listing 5: The prompt given at the beginning of each round (step) of the game where <tar- get_title> <target_summary> and <list_of_titles> will be replaced by the values for the given round of the game. "role": "system", "content ": "You are a helpful assistant that helps me to play the Wikipedia Game! ", "role": "user", "content ": "If I give you the title and a summary of the target page that I am trying to navigate to, can you help me navigate to the target page? ", "role": "assistant", "content ": "Of course! What is your target page?", "role": "user", "content ": "âtitle â: â<target_title >â, âsummary â: â< target_summary >â, "role": "assistant", "content ": "Sure! I can help you navigate to that page , what titles are there on your current page ?" "role": "user", 36 "content ": "Here is a list of titles , which one should I pick? Please provide your answer in JSON format with the selected title given with key âtitle â and a justification given with key âreason â. The selected title must be from the following list: < list_of_titles >" Listing 6: The additional message added to the prompt if a blank output is generated by the LLM (this might happen if a content filter prevents a response) "role": "system", "content ": "âtitle â: âUnknown title â, âreason â: âI did not know which title to select .â" "role": "user", "content ": "But âUnknown title â is not in the list that was provided! You must select a title from the list of titles. Please try again and provide your answer in JSON format with the selected title given with key âtitle â and a justification given with key âreason â. The JSON should only include the two fields âtitle â and âreason â, and the title should only include the title text. You are tying to select the best page title to reach the target page â<target_title >â." Listing 7: The additional message added to the prompt if the output generated by the LLM is not in JSON format "role": "user", "content ": "You did not provide your response in JSON format - please try again! The JSON should only include the two fields âtitle â and âreason â, and the title should only include the title text. You are tying to select the best page title to reach the target page â<target_title >â." Listing 8: The additional message added to the prompt if the output generated by the LLM is in JSON format but does not contain the key âtitleâ "role": "user", "content ": "You must provide a title to navigate to. Please try again and provide your answer in JSON format with the selected title given with key âtitle â and a justification given with key âreason â. The JSON should only include the two fields âtitle â and â 37 reason â, and the title should only include the title text. You are tying to select the best page title to reach the target page â<target_title >â" Listing 9: The additional message added to the prompt if the output generated by the LLM is in JSON format but contains a hallucination in the âtitleâ field "role": "user", "content ": "But â<title >â is not in the list that was provided! You must select a title from the list of titles. Please try again and provide your answer in JSON format with the selected title given with key â title â and a justification given with key âreason â. To jog your memory , 10 example titles from the list are: <list_of_10_random_titles >. The JSON should only include the two fields âtitle â and âreason â, and the title should only include the title text. You are tying to select the best page title to reach the target page â<target_title >â." 38 I Model Performance Table 4 lists the success rate for the TF-IDF, embedding and LLM prompt models used in the experiments. The highest success rate for each of the datasets (within the bounds of error) are highlighted in bold. Table 5 lists the mean number of steps above optimal for each of the models. Where the success rate is less than or equal to 0.05, the mean number of steps is not shown. The lowest number of steps (within the bounds of error) for each dataset is highlighted in bold. For both tables, the errors are shown as one standard deviation of uncertainty on the predicted fraction. The error bounds will be larger when fewer games have been played with a model. 39 Wikipedia OpenAlex Model Bengali Chinese English French Russian Arts Biology Medicine Physics Social Sciences TF-IDF 0.48 ± 0.02 0.47 ± 0.02 0.47 ± 0.02 0.56 ± 0.02 0.40 ± 0.02 0.04 ± 0.01 0.43 ± 0.02 0.34 ± 0.02 0.33 ± 0.01 0.15 ± 0.01 AllMPNetBase 0.05 ± 0.00 0.18 ± 0.01 0.50 ± 0.00 0.46 ± 0.02 0.06 ± 0.00 0.09 ± 0.00 0.64 ± 0.00 0.61 ± 0.00 0.63 ± 0.00 0.27 ± 0.00 Distiluse-Base-v1 0.05 ± 0.00 0.34 ± 0.00 0.31 ± 0.00 0.34 ± 0.00 0.24 ± 0.00 0.04 ± 0.00 0.29 ± 0.00 0.25 ± 0.00 0.26 ± 0.00 0.17 ± 0.00 Distiluse-Base-v2 0.03 ± 0.00 0.33 ± 0.00 0.33 ± 0.00 0.41 ± 0.00 0.30 ± 0.00 0.04 ± 0.00 0.30 ± 0.00 0.24 ± 0.00 0.26 ± 0.00 0.17 ± 0.00 E5-Large 0.50 ± 0.00 0.53 ± 0.01 0.53 ± 0.00 0.61 ± 0.00 0.53 ± 0.00 0.06 ± 0.00 0.61 ± 0.00 0.54 ± 0.00 0.52 ± 0.00 0.25 ± 0.00 E5-Small 0.49 ± 0.00 0.55 ± 0.00 0.53 ± 0.00 0.59 ± 0.00 0.52 ± 0.00 0.05 ± 0.00 0.55 ± 0.00 0.48 ± 0.00 0.41 ± 0.00 0.23 ± 0.00 MiniLM-L6 0.04 ± 0.00 0.11 ± 0.00 0.47 ± 0.00 0.40 ± 0.00 0.05 ± 0.00 0.07 ± 0.00 0.60 ± 0.00 0.56 ± 0.00 0.57 ± 0.00 0.26 ± 0.00 MiniLM-L12 0.08 ± 0.00 0.27 ± 0.00 0.28 ± 0.00 0.34 ± 0.00 0.25 ± 0.00 0.03 ± 0.00 0.37 ± 0.00 0.33 ± 0.00 0.29 ± 0.00 0.15 ± 0.00 Para-MPNet-Base 0.26 ± 0.00 0.26 ± 0.00 0.30 ± 0.00 0.37 ± 0.00 0.26 ± 0.00 0.03 ± 0.00 0.35 ± 0.00 0.31 ± 0.00 0.29 ± 0.00 0.17 ± 0.00 STSB-XLM-R 0.22 ± 0.00 0.20 ± 0.00 0.29 ± 0.00 0.35 ± 0.00 0.24 ± 0.00 0.02 ± 0.00 0.15 ± 0.00 0.10 ± 0.00 0.13 ± 0.00 0.12 ± 0.00 Albert-Small - - 0.28 ± 0.00 - - 0.04 ± 0.00 0.44 ± 0.00 0.37 ± 0.00 0.35 ± 0.00 0.18 ± 0.00 DistilRoberta - - 0.44 ± 0.00 - - 0.07 ± 0.00 0.60 ± 0.00 0.57 ± 0.01 0.58 ± 0.00 0.26 ± 0.00 Glove - - 0.27 ± 0.00 - - 0.03 ± 0.00 0.25 ± 0.00 0.17 ± 0.00 0.20 ± 0.00 0.11 ± 0.00 Roberta - - 0.47 ± 0.00 - - 0.08 ± 0.00 0.64 ± 0.00 0.58 ± 0.00 0.61 ± 0.00 0.28 ± 0.00 GPT-4o mini 0.34 ± 0.05 0.33 ± 0.05 0.36 ± 0.05 0.52 ± 0.05 0.35 ± 0.05 - 0.45 ± 0.05 0.40 ± 0.05 0.28 ± 0.04 - GPT-4o - - 0.71 ± 0.05 0.69 ± 0.05 - - - - 0.64 ± 0.05 - DS-R1:70B (R) 0.46 ± 0.05 0.37 ± 0.05 0.64 ± 0.03 0.56 ± 0.05 0.50 ± 0.05 - 0.44 ± 0.05 0.45 ± 0.05 0.40 ± 0.05 - Gemma:27b 0.24 ± 0.04 0.16 ± 0.04 0.16 ± 0.02 0.25 ± 0.04 0.16 ± 0.04 - 0.14 ± 0.03 0.16 ± 0.04 0.13 ± 0.03 - Llama:1b - - 0.01 ± 0.00 - - - - - - - Llama:3b 0.09 ± 0.03 0.02 ± 0.01 0.08 ± 0.01 0.07 ± 0.03 0.02 ± 0.01 - 0.00 ± 0.02 0.00 ± 0.02 0.01 ± 0.00 - Llama:8b - - 0.10 ± 0.02 - - - - - - - Llama:70b - - 0.08 ± 0.01 - - - - - - - Llama-405b 0.09 ± 0.03 0.20 ± 0.04 0.47 ± 0.03 0.47 ± 0.05 0.19 ± 0.04 - 0.28 ± 0.04 0.34 ± 0.05 0.32 ± 0.04 - Mistral-Nemo - - 0.03 ± 0.01 - - - - - - - Table 4: The success rate by model and dataset 40 Wikipedia OpenAlex Model Bengali Chinese English French Russian Arts Biology Medicine Physics Social Sciences TF-IDF 25.11 ± 0.95 25.21 ± 1.45 32.88 ± 1.13 30.24 ± 1.02 36.51 ± 1.31 28.50 ± 3.08 37.43 ± 1.22 42.61 ± 1.23 45.28 ± 1.44 41.14 ± 2.14 AllMPNetBase x 43.25 ± 2.15 32.97 ± 0.36 33.42 ± 1.17 43.20 ± 1.10 23.82 ± 0.55 25.79 ± 0.25 27.70 ± 0.26 31.58 ± 0.28 24.13 ± 0.35 Distiluse-Base-v1 x 31.47 ± 0.42 36.44 ± 0.47 36.62 ± 0.45 38.73 ± 0.55 x 44.57 ± 0.47 47.93 ± 0.50 47.74 ± 0.51 38.02 ± 0.58 Distiluse-Base-v2 x 34.23 ± 0.45 36.38 ± 0.46 35.67 ± 0.42 37.55 ± 0.48 x 43.87 ± 0.46 47.90 ± 0.50 48.47 ± 0.51 37.73 ± 0.57 E5-Large 27.54 ± 0.33 29.48 ± 0.32 35.78 ± 0.35 32.17 ± 0.32 34.17 ± 0.35 30.04 ± 0.81 31.57 ± 0.28 36.55 ± 0.31 39.25 ± 0.34 30.73 ± 0.41 E5-Small 27.67 ± 0.32 28.50 ± 0.31 35.53 ± 0.35 32.38 ± 0.33 32.99 ± 0.35 x 35.60 ± 0.31 39.77 ± 0.34 43.12 ± 0.39 34.26 ± 0.47 MiniLM-L6 x 41.88 ± 0.81 33.91 ± 0.37 34.64 ± 0.40 45.75 ± 1.29 27.38 ± 0.68 29.62 ± 0.29 32.40 ± 0.29 35.84 ± 0.31 27.54 ± 0.39 MiniLM-L12 36.47 ± 0.95 32.89 ± 0.49 38.87 ± 0.50 36.39 ± 0.46 36.22 ± 0.54 x 41.34 ± 0.41 44.59 ± 0.43 46.08 ± 0.47 34.24 ± 0.56 Para-MPNet-Base 33.34 ± 0.50 33.57 ± 0.50 38.66 ± 0.49 35.75 ± 0.43 37.03 ± 0.51 x 41.33 ± 0.42 42.89 ± 0.44 46.52 ± 0.49 32.73 ± 0.54 STSB-XLM-R 35.66 ± 0.55 34.53 ± 0.57 40.39 ± 0.50 37.77 ± 0.45 38.26 ± 0.55 x 48.62 ± 0.66 52.62 ± 0.79 52.52 ± 0.72 40.13 ± 0.73 Albert-Small - - 39.74 ± 0.52 - - x 38.80 ± 0.36 42.22 ± 0.39 46.12 ± 0.43 35.57 ± 0.55 DistilRoberta - - 34.92 ± 0.39 - - 26.81 ± 0.69 29.41 ± 0.28 32.35 ± 0.31 34.72 ± 0.31 26.37 ± 0.40 Glove - - 37.25 ± 0.52 - - x 44.04 ± 0.50 50.68 ± 0.65 48.58 ± 0.57 40.27 ± 0.71 Roberta - - 35.40 ± 0.38 - - 25.35 ± 0.60 27.23 ± 0.26 30.12 ± 0.27 33.64 ± 0.30 23.98 ± 0.36 GPT-4o mini 19.38 ± 2.18 27.36 ± 3.08 26.47 ± 3.44 29.65 ± 3.02 25.26 ± 4.16 - 44.84 ± 3.54 50.55 ± 4.00 42.43 ± 3.56 - GPT-4o - - 21.55 ± 2.22 18.93 ± 2.08 - - - - 27.05 ± 2.17 - DS-R1:70B (R) 21.87 ± 3.05 19.84 ± 3.11 21.37 ± 1.71 26.09 ± 3.20 21.50 ± 2.83 - 32.09 ± 3.28 36.76 ± 3.46 37.78 ± 3.56 - Gemma2:27B 27.29 ± 4.18 45.00 ± 7.15 39.54 ± 2.88 52.08 ± 5.49 43.81 ± 6.08 - 45.50 ± 6.45 67.63 ± 5.53 54.85 ± 7.47 - Llama:1b - - x - - - - - - - Llama:3b 31.78 ± 5.22 x 35.21 ± 3.18 x 40.67 ± 23.73 - x x x - Llama:8b - - 36.21 ± 3.94 - - - - - - - Llama:70b - - 41.55 ± 3.12 - - - - - - - Llama:405b 29.10 ± 9.76 36.10 ± 5.74 27.61 ± 2.29 26.74 ± 3.49 36.40 ± 5.88 - 57.46 ± 4.80 53.06 ± 3.48 48.00 ± 3.74 - Mistral-Nemo - - x - - - - - - - Table 5: The mean number of steps above optimal for successful games by model and dataset. When the success rate is less than or equal to 0.05, the values are not shown and are marked with an âxâ. 41 J Distribution Fitting To understand the behaviour of the agents the distribution of the step count was studied. The form which most closely matched the pattern seen is a negative binomial distribution. This can be thought of as a count of the number of failed attempts to do something with probability p before it happens n times. If the total number of failures is given by x then the probability function follows Pr(x) = x + nâ 1 x (1â p) x p n .(4) One way to imagine the Wikipedia game is to think of a path that needs to be followed to the target article. If each step along that path could happen with probability p and if the misstep made no positive or negative progress, then the number of extra steps taken by the agents would follow the negative binomial distribution. The optimum number of steps for each of the pairs of pages is known, so it is possible to plot these extra steps to see if the distribution is a good fit. Figure 5 through to Figure 12 show histograms of the number of steps above optimal for eight models on the English Wikipedia dataset. What is interesting is that while the negative binomial distribution seems to be a good fit for these models, the value of n is strange. For all of the models, the expected optimum from the fit seems to be statistically indistinguishable from 1. This is interesting as it implies that there is one major step that seems to be dominating the performance of the agents. This behaviour is also not seen when humans play games which have an n value around 3 (when fitting to all games), much closer to the real optimum value (see Figure 13). This suggests fundamental differences between how humans and language models deal with Wikipedia games and speaks against tempting anthropomorphising comparisons seen between the success rate of humans and GPT-4o. For the human games, we repeated the fit, treating games more than 10 steps above optimal as unknown, since the 10-minute time limit is most likely to cut short longer games (Figure 14). With n free, the fit yields n = 4.45± 2.15, while fixing n = 1 (a geometric distribution) increases the negative log-likelihood by 6.58 with one fewer parameter (likelihood-ratio test, pâ 3Ă10 â4 ) and implies an unphysical eventual success rate of 110%. Constraining that rate to at most 100% could only worsen the n = 1 fit, so the test is conservative. 42 Figure 5: Histogram of number of steps above optimal for the TF-IDF model on the English Wikipedia dataset Figure 6: Histogram of number of steps above optimal for the E5-Small model on the English Wikipedia dataset Figure 7: Histogram of number of steps above optimal for the AllMPNetBase model on the English Wikipedia dataset 43 Figure 8: Histogram of number of steps above optimal for the GPT-4o model on the English Wikipedia dataset Figure 9: Histogram of number of steps above optimal for the GPT-4o mini model on the English Wikipedia dataset Figure 10: Histogram of number of steps above optimal for the Llama:405b model on the English Wikipedia dataset 44 Figure 11: Histogram of number of steps above optimal for the GPT-o3 Mini Reasoning (R) model on the English Wikipedia dataset Figure 12: Histogram of number of steps above optimal for the QwQ:32B Reasoning (R) model on the English Wikipedia dataset Figure 13: Histogram of number of steps above optimal for the successful games that the humans played. 45 Figure 14: Negative binomial fits to the human games on English Wikipedia, with n free (dashed) and with n fixed at 1 (dash-dot). Games more than 10 steps above optimal (grey bars) are treated as unknown to reduce the effect of the time limit. 46 K Model Performance - Link Titles Table 6 shows the success rate on English Wikipedia and OpenAlex Physics dataset for a sample of sentence embedding models. The table shows the success rate in two scenarios. The first is when the summary for each link in an article is provided and compared to the summary of the target article (these are the same values as shown previously in Table 4). The second is the success rate when only the titles of the linked articles are provided and are compared to the summary of the target article. The use of the titles only makes the information given to the model comparable to that given to the LLM. For English Wikipedia, the results show a significant drop in success rate for all the embedding models. For the OpenAlex Physics dataset, a significant drop in success rate also occurs for all models other than E5-Large and E5-Small. The E5-Large model has a more modest drop of 0.02 and the E5-Small model achieves an increase in success rate when only the titles are used. Wikipedia: EnglishOpenAlex: Physics Summaries Titles Summaries Titles AllMPNetBase 0.50± 0.00 0.34± 0.05 0.63± 0.00 0.34± 0.00 DistilRoberta0.44± 0.00 0.30± 0.05 0.58± 0.00 0.27± 0.00 E5-Large0.53± 0.00 0.29± 0.05 0.52± 0.00 0.50± 0.00 E5-Small0.53± 0.00 0.17± 0.04 0.41± 0.00 0.49± 0.00 MiniLM-L60.47± 0.00 0.35± 0.05 0.57± 0.00 0.33± 0.00 Roberta0.47± 0.00 0.42± 0.05 0.61± 0.00 0.26± 0.00 Table 6: The success rate on English Wikipedia and OpenAlex Physics dataset for some sentence embedding models when only the titles of the link articles are used compared to when the summaries of the link articles are used. 47 L Failure Reasons The games played with the sentence embedding and TF-IDF agents can have three possible outcomes: âą Success: The agent navigates to the target article. âą Failure - Step Limit Reached: The agent fails to reach the target article within a set number of steps (configured to 100 in our games). âą Failure - Dead End Reached: The agent fails to reach the target article as it either reaches an article without any outward links. Since most agents are deter- ministic, articles that had already been visited were excluded from the options to avoid the agents becoming trapped in loops, but this means that if an agent reaches an article with no unseen articles, then this is a dead-end. There are three more possible failure reasons when playing the game with the prompt agents: âą Failure - Hallucination: The LLM hallucinates in its response by either not providing the response in valid JSON format, or by making up or not providing an article title to navigate to next. âą Failure - Bad Request: The API rejects the request, which might be what is suspected to be explicit or harmful content in the prompt. âą Failure - No Response: This may occur due to a timeout by the API. On top of these was also one extra failure mode when using the reasoning prompt: âą Failure - No Answer: The result the model produced did not contain an answer block. The outcomes of the games for the English Wikipedia dataset and a selection of models are shown in Figure 15. This shows that reaching the step limit is the most common failure reason for the majority of the models. One exception is Llama:3b, which has a high proportion of games failing in hallucinations. It is notable that the proportion of games failing due to hallucinations is significantly higher for Llama:3b than for Llama:1b. This could be a result of the Llama:1b model selecting article titles from the given list randomly, without any reasoning about how to play the game. The GPT models are the only ones that fail due to bad API requests owing to the safeguards that are in place on Azureâs OpenAI service. The outcomes for games using sentence embedding models across all the datasets are shown in Figure 16. These show the high proportion of games fail for the Arts and Social Sciences datasets because they reach a dead end in the graph. This is not a surprising result given the poor connectivity of the graphs, as shown in Appendix C. Figure 17 shows the outcomes by dataset for a selection of LLMs. These show the high proportion of games that fail due to hallucinations for Llama:3b; although hallucinations 48 0.00.20.40.60.81.0 Fraction of Game Outcomes DS-R1:8B (R) Llama3.2:1B Llama3.2:3B Llama3.1:70B Llama3.1:8B Llama3.1:8B (R) Gemma2:27B (R) Gemma2:27B Glove Word Llama3.1:405B (R) GPT-4o mini Llama3.1:405B Roberta-Large TFIDF MiniLM-L6 v2 DS-R1:32B (R) All-MPNet-Base E5-Small DS-R1:70B (R) QwQ:32B (R) GPT-4o GPT-o3 mini (R) Success Step limit Dead ends Hallucination Bad requests No response No answer Figure 15: The outcome of the games played on English Wikipedia. There are a few different ways that the game could end, the aim being for the model to successfully reach the target article. The model could reach the limit of 100 steps through the graph, or reach a dead- end in the graph. The LLM also could hallucinate by either requesting a title not in the possible next steps or not returning the answer in the appropriate format. It was also possible for an API to timeout or deny a request but this happened only in a small fraction of games. occur across all datasets, the proportion of hallucinations on the OpenAlex datasets is particularly high. 49 0.00.20.40.60.81.0 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Social Sci OA Arts OA E5-Small 0.00.20.40.60.81.0 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Social Sci OA Arts OA All-MPNet-Base 0.00.20.40.60.81.0 Fraction of Game Outcomes English Wiki Physics OA Biology OA Medicine OA Social Sci OA Arts OA Glove Word Success Step limit Dead ends Hallucination Bad requests No response No answer Figure 16: Sentence embedding models game outcomes for a range of topics and languages. These show the outcome of the games played for three sentence embedding models. 50 0.00.20.40.60.81.0 English Wiki French Wiki Physics OA GPT-4o 0.00.20.40.60.81.0 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA DS-R1:70B (R) 0.00.20.40.60.81.0 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Llama3.1:405B 0.00.20.40.60.81.0 English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Gemma2:27B 0.00.20.40.60.81.0 Fraction of Game Outcomes English Wiki French Wiki Russian Wiki Chinese Wiki Bengali Wiki Physics OA Biology OA Medicine OA Llama3.2:3B Success Step limit Dead ends Hallucination Bad requests No response No answer Figure 17: LLM game outcomes for a range of topics and languages. This figure shows how the games ended for four LLMs. 51 M TF-IDF Performance by Corpus Table 7 shows the success rate for the TF-IDF model on the OpenAlex datasets in two scenarios. In the first scenario, a sample of 100,000 records from the target dataset is used to create the corpus for the TF-IDF model. In the second scenario, a sample of 100,000 records from the English Wikipedia dataset is used to create the corpus. The results show a drop in the success rate for the Biology, Medicine and Physics datasets when the English Wikipedia dataset is used for the corpus. However, the success rate for Social Sciences increases when the English Wikipedia dataset is used. This suggests that there is a bigger overlap between the vocabulary of the Social Sciences dataset with English Wikipedia than for Biology, Medicine and Physics. OpenAlex CorpusArts and Humanities Biology Medicine Physics Social Sciences Target Dataset0.04± 0.010.43± 0.02 0.34± 0.02 0.33± 0.010.15± 0.01 English Wikipedia0.04± 0.010.14± 0.01 0.27± 0.01 0.25± 0.010.35± 0.02 Table 7: The success rate of TF-IDF by corpus and dataset 52 N Example Routes This section shows some examples of routes taken by some of the models. These demon- strate how they can behave differently and some circumstances where one model might succeed and another fails. N.1 âAfloqualoneâ to âGaurotes pictiventrisâ Listing 10 shows the route taken (with reasoning) by GPT-4o from the article titled Afloqualone to Gaurotes pictiventris (a species of beetle) from the English Wikipedia graph. In the example shown, GPT-4o was able to navigate the target article in 13 steps. None of the sentence embedding model were able to navigate to the target article within 100 steps. At the time of writing, the summary for the target article contains limited relevant information: Gaurotes pictiventris is a species of beetle in the family Cerambycidae. It was described by Pesarini and Sabbadini in 1997.Bezark, Larry G. A Pho- tographic Catalog of the Cerambycidae of the World. Retrieved on 22 May 2012. References Category:Lepturinae Category:Beetles described in 1997 As well has having limited relevant semantic information in the article summary, the article has only two links into the article which will make it more difficult to navigate to. Listing 10: Example of a route taken by GPT-4o when navigating to âGaurotes pictiventrisâ with the reasoning given by the model at each step 0 : Afloqualone - Start page 1 : Dermatitis - Dermatitis is a condition that can be caused by various factors , including insect bites. This could potentially lead to a link about insects or beetles , which might help us navigate towards the target page about the beetle species Gaurotes pictiventris. 2 : Bacteria - Since Gaurotes pictiventris is a species of beetle , it belongs to the biological classification of organisms. Bacteria are also a type of organism , and navigating to a page about bacteria might provide links to other biological classifications , including insects and beetles , which could help us get closer to the target page . 3 : Bacteriology - Bacteriology is the study of bacteria , which includes the classification and description of bacterial species. Since Gaurotes pictiventris is a species of beetle , it is likely that the page on Bacteriology will have links to pages related to taxonomy and species classification , which can help navigate towards the target page about a specific beetle species. 53 4 : Taxonomy (biology) - The target page âGaurotes pictiventris â is about a species of beetle , which falls under the study of taxonomy in biology. Taxonomy deals with the classification and naming of organisms , which is relevant to finding information about a specific species of beetle. 5 : Family (biology) - Since Gaurotes pictiventris is a species of beetle in the family Cerambycidae , navigating to the âFamily (biology)â page will provide a direct link to the concept of biological families , which is closely related to the target page. From there , you can likely find links to specific families of beetles , including Cerambycidae. 6 : Subfamily - Gaurotes pictiventris is a species of beetle in the family Cerambycidae. To navigate to the target page , it is logical to start with a more specific taxonomic rank related to beetles. âSubfamily â is a relevant choice as it is a taxonomic rank within the family , which can lead us closer to the specific species. 7 : Zoology - Gaurotes pictiventris is a species of beetle , which falls under the study of animals. Zoology is the branch of biology that deals with the animal kingdom , so it is the most relevant title to navigate towards the target page. 8 : Entomology - Entomology is the scientific study of insects , and since Gaurotes pictiventris is a species of beetle , which is an insect , this link is likely to lead you closer to information about beetles and potentially to the family Cerambycidae. 9 : Beetle - Gaurotes pictiventris is a species of beetle in the family Cerambycidae. Selecting âBeetle â will likely lead to more specific information about beetles , which can help narrow down to the specific species. 10 : Longhorn beetle - Gaurotes pictiventris is a species of beetle in the family Cerambycidae , which is commonly known as longhorn beetles. Navigating to the âLonghorn beetle â page will likely provide links or information related to the Cerambycidae family , helping you get closer to the target page. 11 : Lepturinae - Gaurotes pictiventris belongs to the family Cerambycidae , and Lepturinae is a subfamily within Cerambycidae. This link is likely to lead you closer to the target page as it is more specific to the 54 classification of the beetle. 12 : Gaurotes - Gaurotes pictiventris is a species of beetle in the family Cerambycidae , and âGaurotes â is the genus to which this species belongs. Navigating to the genus page will likely provide relevant links or information to help you reach the specific species page. 13 : Gaurotes pictiventris - The title âGaurotes pictiventris â directly matches the target page you are trying to navigate to. This is the exact species of beetle described in the summary you provided. N.2 âMagic starâ to âChoir of St Georgeâs Chapel, Windsor Castleâ The TF-IDF, E5-Small and GPT-4o models were all capable of navigating from Magic star to Choir of St Georgeâs Chapel, Windsor Castle. The summary of this target article is longer than the previous example: right thumb The Quire of St Georgeâs Chapel, by Charles Wild, from W.H. Pyneâs Royal Residences, 1818. The Choir of St Georgeâs Chapel at Windsor Castle exists to sing services in St Georgeâs Chapel at Windsor Castle. It has been in existence since 1348 and, with the exception of the Commonwealth period (1649â1660), has sung services in the Chapel continuously ever since. The choir today The choir comprises up to 23 choristers (the youngest of whom are probationers or training choristers) and 12 professional Lay Clerks, who sing alto, tenor and bass. The Choir sings at Evensong each day (except Wednesdays) and at Mattins and Eucharist on Sunday mornings during the three terms of the College year. The choristers are educated at St Georgeâs School, Windsor Castle which is situated in the Castle grounds. The Lay Clerks live in the Horseshoe Cloister and on Dentonâs Commons. The choir sings regularly in the presence of the King and other members of the Royal Family. Concerts are als There is some text in this summary that is irrelevant to the article and is left over from the markup (such as âright thumbâ at the beginning of the article). The summary text also stops mid-sentence, when a limit on the text length has been reached. Both features are common across the Wikipedia articles. Listing 11 shows the route taken by the TF-IDF model, which only needs 12 steps to reach the target article (the minimum possible number of steps is five). The cosine similarity values between the TF-IDF vectors of the linked article and the target article are also shown to two decimal places. Listing 12 shows the route taken by the E5-Small embedding model. This takes 41 steps to reach the target. Listing 13 shows the route taken by the GPT-4o model with the reasoning given by the model at each step. GPT-4o is able to reach the target article in ten steps; however, 55 the model failed to pick the direct link to the target when it appeared in the article at step eight, so it is still not immune to errors. The model is able to reason that stepping to a general article like âReligionâ is likely to lead to relevant links. There may be parallels in the choice of this step with how humans play the game, where participants often navigate to a general article likely to have many links, before narrowing down the search to the specifics of the target article. Table 8 lists the article names in the route taken by GPT-4o and shows the cosine similarity between those article summaries and the target article summary for the TF-IDF and E5-Small models. This shows that the article titled âReligionâ is not considered similar to the target by either of these models, making both less likely to select the article. In contrast to GPT-4o, Listing 14 shows the route taken by Llama:3b, which was not able to navigate to the target article. The output shows inaccurate reasoning from the second step where the reasoning states that the word âdodecahedronâ is contained on the target article: it is not in the provided summary or anywhere on the Wikipedia article (at the time of writing). The model carries on selecting valid links, with many hallucinations in the reasoning, until it is unable to select a valid link at step 22. Listing 11: Example of a route taken by a TF-IDF model that used English Wikipedia to create a corpus when navigating to âChoir of St Georgeâs Chapel Windsor Castleâ - the cosine similarity scores are shown between the current and target articles at each step 0 : Magic star : 0.01 1 : Magic hexagram : 0.00 2 : Hexagram : 0.01 3 : Ulster Banner : 0.03 4 : Dublin Castle : 0.12 5 : Chapel Royal , Dublin : 0.21 6 : Glenstal Abbey : 0.06 7 : Faith of Our Fathers (album) : 0.09 8 : Cor na nog : 0.27 9 : Choir : 0.32 10 : Choir of King âs College , Cambridge : 0.47 11 : James Adcock : 0.44 12 : Choir of St George âs Chapel , Windsor Castle : 1.00 Listing 12: Example of a route taken by E5-Small when navigating to âChoir of St Georgeâs Chapel Windsor Castleâ - the cosine similarity scores are shown between the current and target articles at each step 0 : Magic star : 0.73 1 : Magic constant : 0.75 2 : Magic series : 0.76 3 : Richard Schroeppel : 0.77 4 : Notices of the American Mathematical Society : 0.79 5 : English language : 0.76 6 : Court of Chancery : 0.82 56 7 : Master of the Rolls : 0.81 8 : Domus Conversorum : 0.82 9 : Public Record Office : 0.81 10 : Cyril Flower (historian) : 0.82 11 : St Edward âs School , Oxford : 0.81 12 : Woodstock Road , Oxford : 0.82 13 : Observatory Street : 0.81 14 : Witney : 0.82 15 : George Wilkinson (architect) : 0.83 16 : Woodstock , Oxfordshire : 0.81 17 : Shipton -on-Cherwell : 0.81 18 : Bladon : 0.81 19 : Kidlington : 0.81 20 : River Cherwell : 0.80 21 : Bodicote : 0.82 22 : Adderbury : 0.81 23 : Ascot Racecourse : 0.82 24 : Windsor Castle Stakes : 0.82 25 : Harry Thomson Jones : 0.80 26 : St Leger Stakes : 0.81 27 : George Stanhope , 6th Earl of Chesterfield : 0.83 28 : George Stanhope , 7th Earl of Chesterfield : 0.82 29 : Cecil Weld -Forester , 1st Baron Forester : 0.82 30 : Orlando Weld -Forester , 4th Baron Forester : 0.82 31 : Baron Forester : 0.82 32 : Captain of the Honourable Corps of Gentlemen -at-Arms : 0.82 33 : Ivor Windsor -Clive , 2nd Earl of Plymouth : 0.82 34 : Robert Windsor -Clive (MP) : 0.82 35 : George Windsor -Clive (politician , born 1835) : 0.82 36 : George Windsor -Clive (politician , born 1878) : 0.82 37 : Parliament of the United Kingdom : 0.80 38 : Bishop of Winchester : 0.81 39 : William Day (bishop) : 0.83 40 : St George âs Chapel , Windsor Castle : 0.86 41 : Choir of St George âs Chapel , Windsor Castle : 1.00 Listing 13: Example of a route taken by GPT-4o when navigating to âChoir of St Georgeâs Chapel Windsor Castleâ with the reasoning given by the model at each step 0 : Magic star - Start page 1 : Star polygon - The Choir of St George âs Chapel , Windsor Castle is related to a historical and architectural context. âStar polygon â might lead to topics related to 57 geometry and design , which could eventually connect to architectural elements of historical buildings like Windsor Castle. 2 : Religion - The Choir of St George âs Chapel , Windsor Castle is closely associated with religious services and the Royal Family. âReligion â is the most relevant title from the list that could lead to information about religious institutions , services , and possibly specific chapels or choirs , including St George âs Chapel. 3 : Religious music - The Choir of St George âs Chapel , Windsor Castle is primarily known for its musical contributions to religious services. Starting from a page related to religious music could provide relevant links to choirs , chapels , or specific religious institutions , which can help navigate towards the target page. 4 : Anglican church music - The Choir of St George âs Chapel , Windsor Castle is part of the Anglican tradition , and Anglican church music is directly related to the type of music performed by the choir. This link is likely to lead to more specific information about choirs within the Anglican Church , which can then guide you to the specific choir at St George âs Chapel. 5 : Choir - The target page is about the Choir of St George âs Chapel , Windsor Castle , and the title âChoir â is directly related to the concept of a choir , which is central to the target page âs content. This should provide relevant links to navigate closer to the specific choir at Windsor Castle. 6 : Boys â choir - The Choir of St George âs Chapel , Windsor Castle , includes choristers who are boys , making âBoys â choir â a relevant and direct link to explore more about the specific type of choir that includes young male singers. 7 : List of choir schools - The target page is about the Choir of St George âs Chapel , Windsor Castle , which includes information about the choir âs education at St George âs School , Windsor Castle. âList of choir schools â is likely to have relevant links or information that can lead us closer to the target page. 8 : St George âs School , Windsor Castle - The Choir of St George âs Chapel , Windsor Castle is closely associated with St George âs School , Windsor Castle , as the choristers are educated there. This connection makes it the most 58 relevant link to navigate towards the target page. 9 : St George âs Chapel , Windsor Castle - The Choir of St George âs Chapel , Windsor Castle is directly related to St George âs Chapel , Windsor Castle. This title will likely provide a direct link or relevant information leading to the target page. 10 : Choir of St George âs Chapel , Windsor Castle - The title âChoir of St George âs Chapel , Windsor Castle â directly matches the target page you are trying to navigate to. Selecting this title will take you directly to the target page. Cosine Similarity Article TitleE5-Small TF-IDF Magic star0.730.01 Star polygon0.730.00 Religion0.730.00 Religious music0.770.00 Anglican church music0.800.11 Choir0.820.32 Boysâ choir0.830.29 List of choir schools0.830.25 St Georgeâs School, Windsor Castle0.910.62 St Georgeâs Chapel, Windsor Castle0.860.39 Choir of St Georgeâs Chapel, Windsor Castle1.001.00 Table 8: The cosine similarity between the named article and the target article of âChoir of St Georgeâs Chapel, Windsor Castleâ for the E5-Small and TF-IDF models. The articles listed in the table are those taken by GPT-4o. 59 Listing 14: Example of a route taken by Llama:3b when navigating to âChoir of St Georgeâs Chapel Windsor Castleâ with the reasoning given by the model at each step 0 : Magic star - Start page 1 : Schlafli symbol - The provided summary mentions âthe Quire of St George âs Chapel , by Charles Wild â which is likely related to the geometry or architecture of the chapel. Therefore , a title that relates to geometric shapes or symbols would be relevant. 2 : Dodecahedron - The target page âChoir of St George âs Chapel , Windsor Castle â contains the phrase âdodecahedron â, which is likely to be a reference to the Dodecahedron polytope. This seems like a plausible connection to make. 3 : Geometry - The target page âChoir of St George âs Chapel , Windsor Castle â is related to the concept of geometry , which deals with the study of shapes and spatial relationships. The choir âs structure and organization are likely to be discussed in a section on geometry. 4 : Euclidean geometry - The target page âChoir of St George â s Chapel , Windsor Castle â is related to the history and tradition of singing in St George âs Chapel at Windsor Castle. Euclidean geometry seems like a plausible starting point for navigation , as it deals with spatial relationships and measurement , which might be relevant to understanding the context of the choir âs performances. 5 : Euclid - The summary mentions âEuclid âs Elements â which is a strong hint towards the page about Euclid , a Greek mathematician. Additionally , there are multiple instances of âEuclid â in the list of titles , making it more likely that this is the correct page. 6 : Euclid âs Elements - The target page âChoir of St George âs Chapel , Windsor Castle â has a section titled âHistory â, which mentions that the choir has sung services continuously since 1348. The next relevant page in the list is âEuclid âs Elements â, as Euclid was an ancient Greek mathematician who lived around 300 BCE and wrote one of the most influential mathematics books in history , making it a plausible connection to the historical context of the choir. 7 : Greek mathematics - The target page âChoir of St George âs Chapel , Windsor Castle â has a section on the history of Greek mathematics , which is relevant to the topic. The title âGreek mathematics â appears in the list and seems like a plausible starting point. 60 8 : Hellenistic period - The Hellenistic period is mentioned in the provided summary as a historical context for the Choir of St George âs Chapel , Windsor Castle. It seems like a relevant and connected topic to navigate to next. 9 : Hellenistic art - The target page âChoir of St George âs Chapel , Windsor Castle â has a section on Hellenistic art in its summary. This suggests that navigating to the page with information about Hellenistic art would be relevant and useful for understanding the context of the choir. 10 : Venus de Milo - The summary mentions âVenus de Milo â as a related work , and it is likely that the target page is about this specific sculpture. 11 : Aphrodite - The target page âChoir of St George âs Chapel , Windsor Castle â mentions Aphrodite in its summary as a figure from Roman mythology and Greek sculpture. It is likely that the title âAphrodite â will be present on the page about the statue or mythological figure mentioned. 12 : Greek mythology - This title is a broad category that encompasses many of the other titles , including those related to specific gods , goddesses , and mythological figures. Itâs likely that the target page about the Choir of St George âs Chapel , Windsor Castle will be listed under this category. 13 : Classical mythology - This title is a broad category that encompasses the entire scope of Greek mythology , which includes the stories and figures mentioned in your target page âChoir of St George âs Chapel , Windsor Castle â. It seems like a good starting point to navigate to the target page. 14 : Ancient Greece - The provided summary mentions âAncient Greece â as a relevant topic , and it is likely that navigating through this title will lead to the target page âChoir of St George âs Chapel , Windsor Castle â, which is related to history and culture in ancient times. 15 : Classical antiquity - The target page âChoir of St George âs Chapel , Windsor Castle â is related to the history and culture of ancient Greece , which falls under the broader category of Classical antiquity. This title seems like a good starting point to navigate to the target page. 16 : Greek Dark Ages - The target page âChoir of St George âs Chapel , Windsor Castle â is related to ancient Greece and its history , making âGreek Dark Ages â a plausible starting point. 61 17 : Geometric art - The Geometric art category is closely related to ancient Greece and the Mediterranean region , which is also relevant to the target page about the Choir of St George âs Chapel , Windsor Castle. 18 : Dipylon Amphora - The target page âChoir of St George âs Chapel , Windsor Castle â mentions the Dipylon style in its summary. The Dipylon Amphora is a specific type of vessel that belongs to this style. 19 : National Archaeological Museum , Athens - The summary of the target page âChoir of St George âs Chapel , Windsor Castle â mentions that the choir sings regularly in the presence of the King and other members of the Royal Family . The National Archaeological Museum , Athens is a museum where royal artifacts are often kept and displayed , making it a plausible location for such events. 20 : Athens - The target page âChoir of St George âs Chapel , Windsor Castle â mentions the city of Athens in its summary . It is likely that navigating to Athens will lead you to the desired page. 21 : Acropolis of Athens - The target page is about a specific ancient monument , which is likely to be related to the Acropolis of Athens. The summary mentions â Parthenon â, which is a famous structure on the Acropolis , and the title âAcropolis of Athens â seems like a plausible match. 22 : FAILURE - Hallucination - title not in list 62 O OpenAlex Multilingual Table 9 lists the success rate of some sentence embedding models for the OpenAlex Biology, Medicine and Physics datasets in two scenarios. The first is when the target articles all have the language filtered on English and the second is when the language of the target article has been filtered on French. The success rates between the two scenarios does not differ greatly for most models. However, the filter was only applied to the target article and so the articles the models navigate are mostly going to be in English. Also, the quality of the âlanguageâ field in the OpenAlex dataset has not been fully investigated and even articles that are written in French might still have an abstract written in English. Some further work would be required to gain more insight into these modelsâ ability to navigate graphs of technical text in languages other than English. BiologyMedicinePhysics ModelEnglishFrenchEnglishFrenchEnglishFrench AllMPNetBase0.64± 0.00 0.62± 0.02 0.61± 0.00 0.60± 0.02 0.63± 0.00 0.59± 0.03 Distiluse-Base-v1 0.29± 0.00 0.30± 0.02 0.25± 0.00 0.26± 0.02 0.26± 0.00 0.25± 0.02 Distiluse-Base-v2 0.30± 0.00 0.30± 0.02 0.24± 0.00 0.22± 0.02 0.26± 0.00 0.23± 0.02 E5-Large0.61± 0.00 0.60± 0.02 0.54± 0.00 0.51± 0.02 0.52± 0.00 0.50± 0.03 E5-Small0.55± 0.00 0.54± 0.02 0.48± 0.00 0.50± 0.02 0.41± 0.00 0.35± 0.02 MiniLM-L120.37± 0.00 0.37± 0.02 0.33± 0.00 0.29± 0.02 0.29± 0.00 0.30± 0.02 Para-MPNet-Base 0.35± 0.00 0.35± 0.02 0.31± 0.00 0.29± 0.02 0.29± 0.00 0.26± 0.02 STSB-XLM-R0.15± 0.00 0.16± 0.02 0.10± 0.00 0.08± 0.01 0.13± 0.00 0.13± 0.02 Table 9: The success rate by language of target article and dataset 63 P Post Training Articles To investigate whether language models are constrained by the connections learned dur- ing their training, they were evaluated on articles published after the beginning of 2024. Since the language models only have access to information available up until their training date, they should not have explicitly learned any connections from these newer articles. However, these recent articles are not representative of the average article and typically have fewer links from other articles. This limitation can be observed in the performance reduction of the TF-IDF model, which would be unaffected by the publication date of the articles, as the model is based on a consistent dataset. The fact that the language modelsâ performance deteriorated fractionally less than the TF-IDF model gives some indication that previously seen connections do not curtail their performance. It is reassuring to note that the modelsâ performance on the academic literature dataset is comparable to Wikipedia, despite the majority of the papers being behind a paywall. This suggests that the language models may not be able to access in- formation describing the connectivity of the OpenAlex dataset, highlighting the necessity of using semantic information. ModelAll Articles Only 2024 Articles TF-IDF0.47± 0.020.10± 0.00 AllMPNetBase 0.50± 0.000.40± 0.02 E5-Large0.53± 0.000.43± 0.02 E5-Small0.53± 0.000.43± 0.02 Gemma:27b0.16± 0.020.09± 0.02 Llama:8b0.10± 0.020.03± 0.02 Table 10: The success rate by model on the English Wikipedia dataset. The models were evaluated on all of the articles or on a subset that was published in 2024, after the time horizon of the language models. 64