Paper deep dive
Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead
John T. Halloran
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/17/2026, 4:44:11 AM
Summary
This paper addresses deployment issues of the Nanbeige4.2-3B agentic model on Apple Silicon (MPS). The authors identify and fix five critical bugs in the Hugging Face transformers integration, including RoPE buffer errors and API mismatches. They further address the Looped Transformer's high memory overhead by implementing a chunked-prefill strategy, which extends context width by 2.7x. Additional fixes resolve system prompt regression and MPS-specific OOM bugs. The patched model achieves 30% success on MCPMark and near-perfect single-tool performance on BFCL, though it struggles with multi-tool calls.
Entities (8)
Relation Signals (6)
Nanbeige4.2-3B → runson → Apple Silicon
confidence 95% · Evaluated on Apple Silicon (MPS)
Nanbeige4.2-3B → usesarchitecture → Looped Transformer
confidence 95% · Nanbeige4.2-3B is a 3B-parameter agentic model built around a Looped Transformer (LT)
Nanbeige4.2-3B → evaluatedon → MCPMark
confidence 90% · On a subset of MCPMark, the debugged model completes up to 30% of real agentic tasks
Nanbeige4.2-3B → evaluatedon → BFCL
confidence 90% · on BFCL, it is near-perfect at single tool calls
Nanbeige4.2-3B → hasbug → RoPE buffer
confidence 90% · The model’s inv_freq rotary-embedding buffer is silently zeroed on load
Chunked Prefill → reducesmemoryoverheadfor → Looped Transformer
confidence 90% · chunked-prefill strategy which alleviates the incurred memory-capacity penalty
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Nanbeige4.2-3B is a 3B-parameter agentic model built around a Looped Transformer (LT) that reuses one stack of layers for a second forward pass, adding effective depth without additional parameters. Evaluated on Apple Silicon (MPS), we identify five independent bugs which prevent the released checkpoint from running via Hugging Face transformers out of the box (including a silently-zeroed RoPE buffer and calls to removed transformers cache APIs). Furthermore, we show that fixing these bugs is still not sufficient for agentic tasks, due to the LT's layer-reuse strategy (which effectively doubles peak attention memory) used to achieve parameter efficiency. We thus introduce a chunked-prefill strategy which alleviates the incurred memory-capacity penalty, extending allowable context width by $2.7 \times$ on 32~GiB shared memory. However, even with the reduced memory overhead, we show that patches are required to render Nanbeige4.2-3B usable; resolving both system prompt and MPS-native memory bugs finally allows reliable evaluation on standard MCP and tool-calling benchmarks. On a subset of MCPMark, the debugged model completes up to 30\% of real agentic tasks (up from the original's 0\%), while, on BFCL, it is near-perfect at single tool calls (yet fails the majority of multi-tool tests). We release the patched checkpoint, system prompt optimizer, and evaluation harnesses at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.13987v1
- Canonical: https://arxiv.org/abs/2608.13987v1
Trouble viewing inline? Open PDF directly →
Full Text
21,879 characters extracted from source content.
Expand or collapse full text
FandolHei-Regular.otf Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead John T. Halloran Email: halloj3@uw.edu August 14, 2026 Abstract Nanbeige4.2-3B (Nanbeige Team 2026) is a 3B-parameter agentic model built around a Looped Transformer (LT) (Bae et al. 2026) that reuses one stack of layers for a second forward pass, adding effective depth without additional parameters. Evaluated on Apple Silicon (MPS), we identify five independent bugs which prevent the released checkpoint from running via Hugging Face transformers out of the box (including a silently-zeroed RoPE buffer and calls to removed transformers cache APIs). Furthermore, we show that fixing these bugs is still not sufficient for agentic tasks, due to the LT’s layer-reuse strategy (which effectively doubles peak attention memory) used to achieve parameter efficiency. We thus introduce a chunked-prefill strategy which alleviates the incurred memory-capacity penalty, extending allowable context width by 2.7×2.7× on 32 GiB shared memory. However, even with the reduced memory overhead, we show that patches are required to render Nanbeige4.2-3B usable; resolving both system prompt and MPS-native memory bugs finally allows reliable evaluation on standard MCP and tool-calling benchmarks. On a subset of MCPMark, the debugged model completes up to 30% of real agentic tasks (up from the original’s 0%), while, on BFCL, it is near-perfect at single tool calls (yet fails the majority of multi-tool tests). We release the patched checkpoint, system prompt optimizer, and evaluation harnesses at github.com/johnhalloran/Nanbeige4.2-3B-mps-fix. 1 Introduction Nanbeige4.2-3B has recently been released as a capable small language model (SLM) specifically designed for improved capability at agentic tasks. The model utilizes a Loop Transformer (LT) architecture to trade off decoding compute without requiring scaling parameter count. The released model card reports competitive or better results than larger models (Qwen3.5-4B, Qwen3.5-9B) on agentic and office-workflow benchmarks (Nanbeige Team 2026), credited to the added effective depth of its Looped Transformer architecture. However, running the released checkpoint in a ReAct-style agentic harness via transformers on MPS (Apple Silicon) surfaces both stability and correctness problems rendering the model unusable out of the box. Herein, we identify five initial bugs—e.g., a silently-zeroed RoPE buffer, calls to removed transformers cache APIs, etc.—and the necessary fixes to render the released checkpoint usable on MPS. However, the resulting model remains unscalable for agentic tasks due to the increased memory needs of the LT architecture, despite the 3B parameters in bf16 on a 32 GiB shared memory system; inherently, the LT enables parameter efficiency at the cost of double the peak attention memory (due to the recursive looping over layers in the forward pass), which prohibits the long reasoning traces required for agentic tasks. Thus, we introduce a chunked-prefill strategy which relieves LT memory overhead, more than doubling allowable context-width evaluation. However, the resulting model reveals further deficiencies: (1) Nanbeige4.2-3B’s trained-in tool-use system prompt is silently replaced, not merged, the moment a caller supplies any system message, and (2) an MPS out-of-memory (OOM) error can permanently degrade the serving process’s usable memory budget, surfacing only while evaluating the debugged model on MCPMark. The gamut of bug and model fixes—five initial bug fixes, an alternative prefilling algorithm, system prompt correction, and MPS-specific OOM fix—allow the true reproducible evaluation of the model on standard agentic and tool-use benchmarks. 2 Five Initial Deployment Bugs Loading on Apple Silicon via ⬇ model = AutoModelForCausalLM.from_pretrained(”Nanbeige/Nanbeige4.2-3B”, trust_remote_code=True, device=”mps”, ) fails or silently misbehaves for five independent reasons. We confirm each by direct reproduction against the unmodified checkpoint. 1. RoPE buffer persistence (dominant bug). The model’s inv_freq rotary-embedding buffer is silently zeroed on load and never repopulated before the first forward pass. RoPE therefore contributes zero positional information to attention: the model runs without knowing token order. This manifests as fluent-looking but positionally-incoherent generation rather than a crash, which makes it easy to miss without directly inspecting buffer values after load. 2. RoPE-config dispatch KeyError. A bug in the custom modeling code’s RoPE-type dispatch raises KeyError for a subset of otherwise valid config values (Table 1). This fires during model construction, before device placement or a single forward pass, and is not MPS-specific (it blocks loading on any device). 3. Cache API sentinel mismatch. Calling the model’s forward() directly with the default past_key_values=None, instead of through generate(), calls an API removed in current transformers releases (DynamicCache.from_legacy_cache(...)). 4. Position-IDs re-trim. A bug during position-tracking in the custom attention code produces a hard crash on device MPS specifically (does not reproduce on CPU). 5. Tied-weights key format. An incompatible tied-weights key naming convention breaks save_pretrained(). Even a successfully-patched, running model cannot be re-serialized without an additional fix. We fix all five via sibling-file monkeypatching (never modifying cached transformers package files) in the johnhalloran/Nanbeige4.2-3B-mps-fix checkpoint. None of the five is related to the memory or system-prompt issues in Sections 3–4, which persist after all five are fixed. Table 1 gives the exact trigger, line number in the unmodified modeling_nanbeige.py, and error string or symptom for each bug, against transformers==5.8.1 (the version used throughout this paper); full diffs and reproduction scripts are in the artifact repository’s patch/ directory (Section 6). Table 1: Exact trigger, source line, and error/symptom for each bug, against transformers==5.8.1. Line numbers refer to the unmodified checkpoint. Bug Line Trigger Error / symptom 1. RoPE buffer (major) 947 from_pretrained() + first forward() no exception raised; inv_freq buffer reads all zeros (persistent=False, never restored after a meta-device load) 2. RoPE-config dispatch 1077 from_pretrained(..., trust_remote_code=True) KeyError: ’type’ 3. Cache API sentinel 2125 calling forward() directly with past_key_values=None, bypassing generate() AttributeError: type object ’DynamicCache’ has no attribute ’from_legacy_cache’; still present in the deployed checkpoint 4. Position-IDs re-trim 2630 generate() on MPS, once prepare_inputs_for_generation receives an already-populated position_ids kwarg ’mps.matmul’ op contracting dimensions differ 361 & 181 (Metal assertion, uncatchable SIGABRT); not reproduced on CPU 5. Tied-weights key format 2417 model.save_pretrained(...) AttributeError: ’list’ object has no attribute ’keys’ 3 The Looped-Transformer Memory Tradeoff Nanbeige4.2-3B’s LT (Bae et al. 2026) feeds the hidden state through the full stack of L physical transformer layers, then re-feeds the output of that pass through the same L layers a second time before producing logits: two effective passes (2L2L effective layer-executions) from L layers’ worth of parameters. This effectively improves model quality without scaling parameters—e.g., LTs outperforming larger non-looped models at a fixed parameter budget (Bae et al. 2026). However, parameter counts are kept low at the expense of additional compute and memory requirements. Naively, self-attention’s peak activation memory during prefilling is dominated by materializing an attention-score tensor proportional to (prompt_len)2(prompt\_len)^2 per layer pass. Compared to their non-looped counterparts, LTs double required prefilling memory, as the O(prompt_len2)O(prompt\_len^2) attention computation over the same prompt is repeated once per loop. Thus, despite the total-parameter count savings enabled by looped weight-sharing during pretraining, naive prefilling results in twice the full quadratic attention cost during inference. For small language models (SLMs) on large dedicated hardware (e.g., H200s), this doubling is usually not an issue. However, on Apple Silicon’s unified memory—shared between the OS, competing processes, and the model with no page-out path comparable to CUDA’s memory management—it can be catastrophic for agentic tasks. 3.1 Balancing Looped Memory Use via Chunked Prefilling As opposed to naive prefilling—wherein the full (prompt_len×prompt_len)(prompt\_len×prompt\_len) attention-score tensor is computed once per loop iteration in a single forward pass—we show that Nanbeige4.2-3B’s memory use may be significantly decreased via chunked prefilling. Chunked prefilling processes the prompt in fixed-size chunks, growing the Key-Value cache incrementally between chunks the same way ordinary autoregressive decoding already does. We replace the single-shot model(input_ids=full_prompt, ...) prefill call with a loop that processes the prompt in fixed-size chunks (256 tokens by default), incrementally growing a DynamicCache between chunks, before handing the remainder off to generate(): ⬇ total_len = input_ids.shape[1] if total_len <= chunk_size: return model.generate(input_ids=input_ids, max_new_tokens=max_new_tokens, **gen_kwargs) cache = DynamicCache() n_full_chunks = (total_len - 1) // chunk_size with torch.no_grad(): for i in range(n_full_chunks): start, end = i * chunk_size, i * chunk_size + chunk_size outputs = model( input_ids=input_ids[:, start:end], past_key_values=cache, use_cache=True, cache_position=torch.arange(start, end), ) cache = outputs.past_key_values return model.generate(input_ids=input_ids, past_key_values=cache, max_new_tokens=max_new_tokens, **gen_kwargs) This bounds the peak per-step attention-score tensor to (chunk_size×running-total)(chunk\_size×running-total), independent of how long the prompt is, at the cost of splitting one forward pass into several sequential sub-calls instead of one. Bit-identical outputs were verified against naive prefill. 3.2 LongBench-Pro results We demonstrate the memory benefits of chunked prefilling (CP) over naive prefilling (NP) using 50 long samples from LongBench-Pro (Chen et al. 2026) and single-turn queries of eight lengths varying from 1024 to 12,24412,244 tokens. Each sample is tokenized using the Nanbeige4.2-3B tokenizer and truncated to the target length. To measure maximum memory throughput, we calculate the max batch size per length and prefilling strategy by doubling the batch size until failure, repeating this process 3 times. All evaluations were performed on a Apple M2 Max with 32 GiB of shared memory. Results are in Table 2. Table 2: NP vs CP evaluated over 50 LongBench-Pro samples, averaged over 3 repeated experiments. “–” denotes the method could not complete even batch=1. prompt_len NP max batch NP tok/s CP max batch CP tok/s 1024 16 275.3275.3 32 212.5212.5 2048 4 267.5267.5 16 158.1158.1 4096 2 158.4158.4 4 124.8124.8 8192 – – 1 168.7168.7 9205 – – 1 156.5156.5 10218 – – 1 148.6148.6 11231 – – 1 142.4142.4 12244 – – – – The maximum length possible under CP (11231) is significantly larger than NP (4096). However, CP trades memory requirements for time; at prompt_len=1024prompt\_len=1024, CP allows twice the amount of batch-parallelism, while only being 22.8%22.8\% slower than NP. This tradeoff is most noticeable at prompt_len=2048prompt\_len=2048, where CP allows 4 times the batch-parallelism while being 40.9%40.9\% slower. We note that, for CP, per-chunk subcalls incur a fixed cost, paid chunk_size times regardless of batch size. Thus, this overhead becomes amortized when the batch size is large, but becomes a larger portion of total runtime when only smaller batch sizes are present, e.g., prompt_len=4096prompt\_len=4096, which achieves lower throughput than batch size =1=1 evaluations over longer prompts. We note that folding the sequential per-chunk subcalls into fewer large GPU operations via kernel fusion would reduce this per-call overhead directly. 4 System-Prompt Regression Independent of the previously discussed memory issues, Nanbeige4.2-3B’s chat template (chat_template.jinja, in the % if tools % branch) performs the following if/else on messages[0]: • If the caller supplies any system message, it is used verbatim, with a trailing " " the template appends. • Otherwise, the template injects a hardcoded default (Nanbeige’s own trained-in tool-use system prompt, beginning ‘你是一位工具函数调用专家...’11 1 Translation: “You are a tool-function-calling expert. You will be given a question and a set of possible tool functions. Based on the question, you need to make one or more function/tool calls to accomplish the goal — please do your best to explore solving the problem through tools. If no function is usable, reply directly to the user in natural language. If the given question is missing parameters required by a function, use natural language to ask the user for the necessary information. If the call results are already sufficient to answer the user’s question, summarize the results and reply to the user in natural language.”) with no trailing separator before the # Tools section that follows. Any caller-supplied system message thus silently discards the model’s own trained default instead of extending it. Any tools-plus-user-message request produces a clean, correctly-formatted single tool call, but with no system message; if a system message is added—even one such as “You are an assistant with MCP tools”—multi-tool-call outputs break into malformed text. Re-supplying the original text as an explicit system message does not fix this behavior. Byte-identical content through the explicit-system-message branch still breaks, because that branch’s own auto-appended " " differs from the zero-extra-whitespace auto-insert branch’s output by exactly two characters. The released checkpoint’s tool-calling reliability is calibrated to the exact byte sequence its own default rendering path produces, consistent with its tool-use SFT/RL data having only ever been rendered through the auto-insert branch and never with a caller-supplied system message. Fix: remove the system message from the chat template, such that the template takes its own zero-extra-whitespace auto-insert path. Then insert the caller’s system content in the rendered string after the auto-insert default (never through the default template’s aforementioned branch). This generates single-tool-call outputs while including the caller’s system content. We note that a closely related (but a mechanically distinct bug) has been independently reported against this model: llama.cpp PR #26324 documents Nanbeige4.2-3B emitting <tool_call> with a trailing space instead of <tool_call> for roughly 25% of calls, breaking tag-matching in that inference engine, and notes the same template structure is shared by (though not observed to trigger the same failure in) Qwen3-Coder and Qwen3.5-4B. Both bugs sit in the same chat-template/generation pipeline and are independent evidence that this checkpoint’s tool-calling reliability is template and whitespace sensitive. 5 Evaluation We evaluate the combined fixes on a 10 task subset of MCPMark (Wu et al. 2025)—which tests MCP-tool use capabilities on multi-turn tasks while grading tool-generated responses—and a 150 subset of the Berkeley Function-Calling Leaderboard (BFCL) (Yan et al. 2024)–which tests tool selection ability without considering tool execution outputs. MPS memory bug. Running the test suite end-to-end surfaced a bug unrelated to the model: a single caught RuntimeError: MPS backend out of memory permanently degrades the harness process’s usable MPS memory budget for the rest of its life. Neither torch.mps.empty_cache() nor gc.collect() reclaim it in a controlled tests; only a process restart does. Uncorrected, one task’s OOM—expected for MCPMark, where multi-turn tasks can grow past the max-token ceiling (12,24412,244, per Table 2)—cascades into spurious OOMs on every later, unrelated task in the same long-lived server. Restarting the harness fresh before each task (mirroring the subprocess-per-trial isolation already used for Table 2) fixes this. 5.1 MCPMark (Filesystem subset, easy tier) Table 3: MCPMark Filesystem (easy tier), patched checkpoint, per-task server isolation, using MCPMark’s default 1 hour timeout. Task Turns Outcome largest_rename 5 pass txt_merging 5 pass file_reorganize 7 pass pattern_matching 2 fail, timed out file_splitting 4 fail, tool response OOM uppercase 7 fail, tool response OOM structure_analysis 2 fail, tool response OOM papers_counting 2 fail, tool response OOM duplicate_name 2 fail, tool response OOM recommender_name 2 fail, tool response OOM Evaluations were run over MCPMark’s Filesystem suite of 10 easy tasks (which do not require API credentials), using the benchmark’s default timeout of 1 hour. The original (unpatched) checkpoint cannot be evaluated (on any device) due to bug 2 of Section 2. The patched checkpoint—with CP (Section 3.1) and all described bug fixes (Sections 2 and 4)—scores 3/10 (30%). Details for each per-task run are in Table 3. For the failed pattern_matching task, the model correctly calls the MCP tool read_multiple_files once, but repeats a long absolute path 21 times, leading to a long context-width that eventually times out. The remaining failures accumulate context over multiple turns and eventually exceed memory capacity before solving the underlying task. 5.2 Tool-calling correctness in isolation from decode throughput (BFCL) To stress test the tool calling ability of the model (and not necessarily end-to-end task effectiveness), we evaluate the debugged/scalable model on a 150 task subset of BFCL’s non-live, single-turn categories (Yan et al. 2024): simple_python (one correct call), multiple (pick 1 of N candidate functions), parallel (emit 2+ calls to the same function), parallel_multiple (2+ calls to different functions), and irrelevance (correctly emit no call at all). These are AST/exact-match graded, require no external LLM judge, and complete in one generation each (10–20s), so there is no wall-clock or multi-turn-accumulation confound (as in MCPMark). Table 4: BFCL, patched checkpoint, 30 tasks per category. Category Score Dominant failure mode simple_python 19/30 (63.3%) wrong call count (11/30) multiple 13/30 (43.3%) wrong call count (17/30) irrelevance 30/30 (100%) — parallel 1/30 (3.3%) wrong # of functions (29/30) parallel_multiple 9/30 (30.0%) wrong # of functions (21/30) The model reliably recognizes when not to call a tool (100% on irrelevance) and is moderately reliable on a single, well-specified call (63.3%). It is specifically weak at emitting multiple tool calls in one turn: on both parallel categories, the dominant failure is producing the wrong number of function calls, almost always one call where two were required. This is a distinct, format-level limitation from anything in Section 5.1—it would appear even on hardware with sufficient memory capacity to avoid OOMs and speed to avoid timeouts. 6 Conclusions and Artifacts The released Nanbeige4.2-3B checkpoint cannot run reliably via Hugging Face transformers on Apple Silicon, blocked first by five independent deployment bugs and then by the memory overhead of its Looped Transformer architecture. We fix all five bugs, introduce chunked prefilling to more than double the usable context width under Apple Silicon’s shared-memory limits, and resolve a system-prompt regression and an MPS memory bug that otherwise block reliable agentic evaluation. The resulting patched checkpoint completes up to 30% of a real MCPMark agentic subset (up from 0%) and is near-perfect at single BFCL tool calls, though it still fails the majority of multi-tool-call tests. The code, patched checkpoint, and reproduction scripts for all evaluations in are released at github.com/johnhalloran321/nanbeige-mps-fix (repository pending publication; layout and contents already finalized, see its README.md) and at johnhalloran/Nanbeige4.2-3B-mps-fix on Hugging Face. References Bae et al. (2026) Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. In Advances in Neural Information Processing Systems (NeurIPS), 2026. URL https://arxiv.org/abs/2507.10524. Chen et al. (2026) Ziyang Chen, Xing Wu, Junlong Jia, Chaochen Gao, Qi Fu, Debing Zhang, and Songlin Hu. Longbench pro: A more realistic and comprehensive bilingual long-context evaluation benchmark. arXiv preprint arXiv:2601.02872, 2026. URL https://arxiv.org/abs/2601.02872. Nanbeige Team (2026) Nanbeige Team. Nanbeige4.2-3b: Unlocking agentic capabilities in a compact model. arXiv preprint arXiv:2607.22083, 2026. URL https://arxiv.org/abs/2607.22083. Wu et al. (2025) Zijian Wu, Xiangyan Liu, Xinyuan Zhang, Lingjun Chen, Fanqing Meng, Lingxiao Du, Yiran Zhao, Fanshi Zhang, Yaoqi Ye, Jiawei Wang, Zirui Wang, Jinjie Ni, Yufan Yang, Arvin Xu, and Michael Qizhe Shieh. Mcpmark: A benchmark for stress-testing realistic and comprehensive mcp use. arXiv preprint arXiv:2509.24002, 2025. URL https://arxiv.org/abs/2509.24002. Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard (bfcl). UC Berkeley Gorilla Project, 2024. URL https://gorilla.cs.berkeley.edu/leaderboard.html.