Paper deep dive
Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders
Lucas Hideki Takeuchi Okamura, Alexandre Alcoforado, Anna Helena Reali Costa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/7/2026, 11:28:15 AM
Summary
This paper introduces MTEB-PT, a Portuguese benchmark for evaluating sentence embedding models across 14 datasets spanning STS, classification, retrieval, and reranking. Evaluating 17 models reveals that Portuguese performance is strongly task-dependent and not reliably predicted by multilingual averages. The authors fine-tune three backbone models using Portuguese contrastive supervision and Matryoshka Representation Learning (MRL), demonstrating significant gains on STS tasks and competitive performance under dimensional truncation, while releasing the benchmark, models, and code.
Entities (20)
Relation Signals (12)
MTEB-PT → spanstasks → STS, Classification, Retrieval, Reranking
confidence 97% · comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking.
MTEB-PT → evaluates → Portuguese Sentence Encoders
confidence 96% · We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol.
Portuguese Performance → isdependenton → Task Type
confidence 96% · Our results show that Portuguese performance is strongly task-dependent: multilingual rankings do not reliably predict Portuguese-specific performance across task families
e5-large-matryoshka → achieveshighestscoreon → STS
confidence 95% · In particular, e5-large-matryoshka achieves the best overall STS performance
MTEB-PT → issubsetof → MMTEB
confidence 95% · MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB
Fine-tuning Process → usestechnique → Matryoshka Representation Learning
confidence 95% · To examine this effect, we fine-tune three representative backbone models with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL).
Long-context Models → advantageousfor → Retrieval and Reranking
confidence 94% · models with stronger long-context capacity are particularly advantageous on longer-input tasks such as retrieval and reranking.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show that Portuguese performance is strongly task-dependent: multilingual rankings do not reliably predict Portuguese-specific performance across task families, no single model dominates all settings, and models with stronger long-context capacity are particularly advantageous on longer-input tasks such as retrieval and reranking. The benchmark also shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely. To examine this effect, we fine-tune three representative backbone models with Portuguese contrastive supervision and Matryoshka Representation Learning (MRL). These benchmark-informed baselines yield their strongest gains on STS, consistent with the predominantly symmetric supervision used during training, while also improving retrieval and remaining competitive under dimensional truncation. We release the MTEB-PT benchmark, the fine-tuned models, and the training and evaluation code.
Tags
Links
- Source: https://arxiv.org/abs/2607.04071v1
- Canonical: https://arxiv.org/abs/2607.04071v1
Trouble viewing inline? Open PDF directly →
Full Text
42,515 characters extracted from source content.
Expand or collapse full text
Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders Lucas Hideki Takeuchi Okamura [0000−0002−7198−6140] , Alexandre Alcoforado [0000−0003−3184−1534] , and Anna Helena Reali Costa [0000−0001−7309−4528] Escola Politécnica, Universidade de São Paulo, São Paulo, Brazil lucasokamura, alexandre.alcoforado@gmail.com anna.reali@usp.br Abstract. Portuguese remains underrepresented in text embedding eval- uation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Se- mantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show that Portuguese per- formance is strongly task-dependent: multilingual rankings do not reli- ably predict Portuguese-specific performance across task families, no sin- gle model dominates all settings, and models with stronger long-context capacity are particularly advantageous on longer-input tasks such as re- trieval and reranking. The benchmark also shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely. To examine this effect, we fine-tune three representative backbone models with Por- tuguese contrastive supervision and Matryoshka Representation Learn- ing (MRL). These benchmark-informed baselines yield their strongest gains on STS, consistent with the predominantly symmetric supervision used during training, while also improving retrieval and remaining com- petitive under dimensional truncation. We release the MTEB-PT bench- mark, the fine-tuned models, and the training and evaluation code. Keywords: Text Embeddings· Benchmark· Portuguese· Sentence En- coders· Matryoshka Representation Learning 1 Introduction Text embeddings remain a core component of modern NLP, supporting tasks such as STS, classification, retrieval, and reranking. Even as decoder-based large language models have transformed generative NLP, encoder-based models are still often preferred when the goal is to obtain compact sentence representations with strong semantic structure for downstream use. arXiv:2607.04071v1 [cs.CL] 5 Jul 2026 2Okamura et al. However, the evaluation of embedding models remains concentrated in En- glish and in broad multilingual aggregates, making it difficult to assess progress for individual languages. This is particularly limiting for Portuguese: although it is one of the most widely spoken languages in the world [17], it remains com- paratively underrepresented in standardized embedding evaluation. As a result, it is still unclear which models are genuinely strong for Portuguese, how rank- ings change across task families, and whether multilingual average performance translates into robust Portuguese performance. To address this gap, we present MTEB-PT, a Portuguese benchmark slice of MMTEB [2], composed of 14 existing datasets spanning four task families: STS, classification, retrieval, and reranking. We curate a Portuguese evaluation setting within MMTEB and evaluate 17 open- and closed-source models under a unified protocol. Our findings show that Portuguese embedding performance is strongly task- dependent. Models that perform well in multilingual settings do not transfer uniformly across Portuguese tasks, and the differences are especially visible be- tween symmetric semantic tasks and longer-input retrieval or reranking set- tings, where long-context support is often advantageous. The benchmark also shows that language-specific fine-tuning remains useful in Portuguese. To exam- ine this effect, we fine-tune three representative backbone models on Portuguese sentence-pair data and find the strongest gains on symmetric semantic tasks, especially STS, while still observing some transfer to retrieval. Beyond full-dimensional evaluation, we also study dimensional efficiency in deployment-relevant settings. We train the adapted models with MRL [6], which allows a single model to support progressive dimensional truncation and makes it possible to analyze how well Portuguese sentence representations retain quality under compact embeddings. Our main contributions are: – Portuguese benchmark: We introduce MTEB-PT, a Portuguese bench- mark constructed from a subset of MMTEB and composed of 14 existing datasets spanning STS, classification, retrieval, and reranking; – Benchmark findings: We provide a systematic evaluation of 17 open- and closed-source embedding models and show that model rankings in Por- tuguese are strongly task-dependent, with multilingual strength not trans- ferring uniformly across task families; – Strong Portuguese baselines: We release fine-tuned versions of multilingual- e5-large, bertimbau-large, and modbertbr as benchmark-informed Portuguese baselines, showing that language-specific adaptation is especially beneficial for symmetric semantic tasks; – Dimensional efficiency analysis: We show that MRL-trained models re- main competitive under dimensional truncation, enabling more storage- and latency-efficient Portuguese embeddings. MTEB-PT, a Benchmark for Portuguese Sentence Encoders3 2 Related Work Text embedding evaluation has increasingly moved from isolated semantic tasks to unified multi-task benchmarks. MTEB [11] established a common evaluation framework across diverse embedding tasks, and MMTEB [2] extended this setup to a broad multilingual setting. While these benchmarks greatly improve com- parability across models, aggregate multilingual results still do not fully reveal language-specific behavior for languages such as Portuguese. Prior work on Portuguese sentence embeddings has mostly evaluated nar- rower task subsets. Fialho et al. [3] compared multilingual and Portuguese BERT- style models on NLI and STS using ASSIN, SICK-BR, and ASSIN2, while Gomes et al. [5] introduced the Serafim family and evaluated Portuguese sen- tence encoders mainly on STS and retrieval. More recent studies broaden this perspective: Peixoto et al. [12] show that model behavior in Brazilian Portuguese is strongly task-dependent across classification, clustering, NLI, and STS, and Pinto [13] highlights the value of Matryoshka training for dimensional efficiency in Portuguese sentence embeddings. Our work builds on these efforts by providing a single Portuguese benchmark slice within the MMTEB ecosystem, spanning STS, classification, retrieval, and reranking under a unified protocol. 3 MTEB-PT: Portuguese Benchmark We introduce MTEB-PT, a Portuguese benchmark slice for evaluating sentence embedding models across multiple application settings. The benchmark follows the multi-task evaluation perspective of MTEB, which emphasizes that embed- ding quality is task-dependent and should not be inferred from semantic sim- ilarity performance alone [11]. Rather than focusing only on STS, MTEB-PT includes four task families—STS, classification, retrieval, and reranking—so that models can be compared under both symmetric and asymmetric matching set- tings. We selected datasets that satisfy three practical criteria: they are publicly available under open licenses, have sufficient quality for benchmarking, and are already accepted by the community through inclusion in the MTEB/MMTEB ecosystem. Since our goal is to provide a coherent Portuguese evaluation setting rather than introduce new datasets, we restrict the benchmark to Portuguese- labeled task subsets that can be evaluated under the official implementations. 3.1 Construction and characteristics MTEB-PT contains 14 datasets spanning four task families: 3 STS, 5 classifi- cation, 3 retrieval, and 3 reranking. The STS tasks evaluate pairwise seman- tic similarity; the classification tasks cover intent detection, sentiment analysis, toxicity, and hate speech; and the retrieval and reranking tasks extend the bench- mark to asymmetric query–document settings relevant to semantic search and 4Okamura et al. Table 1: Overview of the MTEB-PT benchmark. TaskDatasetSubsetEvaluation size STS Assin2STS–2,448 pairs SICK-BR-STS–2,048 pairs STSBenchmarkMultilingualpt1,379 pairs Classification MassiveIntentClassificationpt2,974 examples MultiHateClassificationpor1,000 examples BrazilianToxicTweetsClassification –2,048 examples HateSpeechPortugueseClassification –2,048 examples TweetSentimentClassificationportuguese 870 examples Retrieval WebFAQRetrievalpor10,000 q. / 209,353 docs WikipediaRetrievalMultilingualpt1,500 q. / 13,500 docs MultiLongDocRetrievalpt200 q. / 6,569 docs Reranking WikipediaRerankingMultilingualpt1,500 q. / 13,500 cand. XGlueWPRRerankingpt677 q. / 8,314 cand. MultiLongDocRerankingpt200 q. / 1,564 cand. retrieval-augmented generation. Including retrieval and reranking is important because strong multilingual embeddings do not necessarily transfer uniformly from short symmetric tasks to longer and structurally asymmetric search sce- narios. Table 1 summarizes the benchmark suite. In contrast to STS and classifica- tion tasks, retrieval and reranking datasets differ not only in supervision struc- ture but also in evaluation scale, since they require ranking over full corpora or candidate sets. We therefore report their sizes in terms of query and document or candidate counts, which better reflects the actual evaluation setting. Most tasks involve relatively short texts, whereas the MultiLongDoc datasets define the benchmark’s long-context regime and are particularly useful for stress-testing truncation sensitivity and long-document behavior. 4 Experimental Setup This section describes how we evaluate models on MTEB-PT under a unified and reproducible protocol. We present the evaluated models, the task-specific evalu- ation procedure, the prefix and formatting choices that ensure fair comparison, the aggregation strategy, and the dimensional truncation protocol. Throughout, our goal is to separate the definition of the benchmark from the choices made to run models on it consistently. 4.1 Evaluated Models We compare four groups of models: closed-source models, peer-reviewed open-source models, community open-source models (not peer-reviewed), and Portuguese-specialized models fine-tuned in this work. This grouping MTEB-PT, a Benchmark for Portuguese Sentence Encoders5 Table 2: Metadata of the evaluated models. SE = Sentence Encoder; PTE = Pre-Trained Encoder; MRL-SE = Matryoshka Representation Learning Sentence Encoder. ModelModel Type Model Size Embedding Dimension Context Length (Tokens) Closed-source cohere-embed-v4MRL-SE–3,072128k text-embedding-3-largeMRL-SE–1,5368,192 amazon-titan-v2MRL-SE–1,0248,192 Peer-reviewed serafim-900mSE0.9B1,536128 multilingual-e5-largeSE0.6B1,024512 multilingual-e5-large-instruct SE0.6B1,024512 multilingual-e5-baseSE0.3B768512 multilingual-e5-smallSE0.1B384512 qwen3-embedding-0.6bMRL-SE0.6B1,02432k gte-multilingual-baseMRL-SE0.3B7688,192 legal-bertimbau-large-stsSE0.3B1,024512 multilingual-mpnet-baseSE0.3B768128 bertimbau-largePTE0.3B1,024512 mmbert-basePTE0.3B7688,192 modbertbrPTE0.1B768512 Community models mmbert-embed-32kMRL-SE0.3B76832k neobertuguesPTE0.1B7688,192 PT Fine-tuned MRL e5-large-matryoshkaMRL-SE0.6B1,024512 bertimbau-large-matryoshka MRL-SE0.3B1,024512 modbertbr-matryoshkaMRL-SE0.1B768512 reflects both practical usage and scientific lineage, allowing us to compare com- mercial APIs, widely studied research encoders, strong community baselines, and Portuguese-adapted variants under the same benchmark. The fine-tuned group includes three backbone models — bertimbau-large [18], multilingual-e5-large [20], and modbertbr [22] — which we adapt for Portuguese sentence-embedding learning using MRL. These backbones were selected to cover complementary starting points: a multilingual sentence encoder already strong in general embedding benchmarks, a Portuguese BERT-family model with larger capacity, and a lighter Portuguese encoder relevant for compact deployment set- tings. The training data, contrastive objective, and hyperparameter details are presented in Section 5. Table 2 summarizes all evaluated models, including model type, parameter scale, embedding dimensionality, and maximum context length. We report these properties explicitly because they are important for interpreting performance differences across task families, especially between short symmetric tasks and longer retrieval or reranking settings. 6Okamura et al. 4.2 Evaluation Protocol We evaluate all models using the official MTEB/MMTEB task implementations and default evaluation procedures. This keeps MTEB-PT directly comparable to prior benchmark results while minimizing benchmark-specific modifications. To ensure a fair comparison across heterogeneous architectures, we encode text using each model’s recommended inference-time input format. For models explicitly trained with structured inputs—such as E5-family models that recom- mend the query: and passage: templates—we apply the corresponding prefixes during evaluation. For models without such documented conventions, we encode the raw text without adding prefixes. An exception is e5-large-matryoshka fine- tuned in this work: although the original model recommends using prefixes, our Portuguese fine-tuning was performed without them. Accordingly, we evaluate this fine-tuned variant without prefixes. For STS, we compute cosine similarity between sentence embeddings and re- port Spearman’s rank correlation with the gold similarity scores. For classifica- tion, we follow the standard MTEB protocol, training a logistic regression clas- sifier on top of frozen embeddings using the task training split and evaluating on the corresponding test split. For retrieval, we rank the full document corpus for each query using cosine similarity and report nDCG@10, following the standard MTEB/BEIR retrieval setting [19]. For reranking, we score each candidate item with respect to the query embedding, sort the candidates accordingly, and report the metric defined by the official task implementation. In our benchmark, the metric is Mean Average Precision (MAP) for WikipediaRerankingMultilingual and XGlueWPRReranking, and nDCG@10 for MultiLongDocReranking, follow- ing the definitions established in the respective MMTEB task implementations. Although MAP and nDCG@10 differ in their formulations, both are bounded in [0, 1] and capture ranking quality in comparable terms; macro-averaging across the three reranking datasets is therefore still meaningful as an aggregate indi- cator. Due to budget constraints, WebFaqRetrieval was run on only 20% of the dataset for the closed-source models; results for this task should therefore be interpreted as partial. Unless otherwise stated, we aggregate results by macro-averaging over datasets within each task family. We prefer task-family averages because they provide a more stable view of Portuguese model behavior than any individual dataset, while still preserving the distinction between symmetric semantic tasks and asymmetric search-oriented tasks. 4.3 Dimensional Truncation Protocol Besides the main evaluation protocol, we perform a truncation analysis on the sentence encoders across multiple embedding sizes to examine how performance changes under compact representations and to assess the benefit of MRL for Portuguese. This analysis is motivated by deployment constraints: full-dimensional per- formance is informative for leaderboard comparisons, but compact embeddings MTEB-PT, a Benchmark for Portuguese Sentence Encoders7 are often preferable in practical systems due to storage, memory, and latency considerations. By evaluating both regimes under the same benchmark, we can distinguish models that are strong only at maximum dimensionality from models that remain competitive after aggressive truncation. For the truncation analysis, we extract lower-dimensional representations by selecting the first d components of each model’s full embedding. For MRL-trained models, this truncation is the intended usage mode; for non-MRL models, it serves as a baseline to determine whether structured multi-resolution training is necessary to retain quality at reduced dimensions. 5 MRL Portuguese Fine-Tuning Beyond benchmarking existing encoders, we study whether Portuguese-centered supervised adaptation still improves sentence embeddings under a modern multi- task evaluation setting. Our goal is not to propose a new architecture, but to build strong Portuguese baselines from representative backbones and test whether MRL preserves these gains under dimensional truncation. We fine-tune three complementary backbone models: multilingual-e5-large, bertimbau-large, and modbertbr. The first is a strong multilingual sentence en- coder that serves as a competitive cross-lingual baseline, whereas the latter two provide Portuguese-oriented masked-language-model backbones with different capacity and architectural trade-offs, including a lighter ModernBERT-based variant [21]. For multilingual-e5-large, we use its native sentence-transformer pooling; for bertimbau-large and modbertbr, we add mean pooling over the final hidden states to obtain fixed-size sentence representations. Our training data combines three complementary supervision sources: STS- style sentence pairs with graded similarity scores, Portuguese NLI entailment pairs, and retrieval-oriented positive pairs from MLDR [1]. Concretely, the STS portion includes Portuguese data from STSBenchmark [9], ASSIN [4], ASSIN 2 [14], IRIS STS [10], and SICK-BR [15], while the positive-pair portion uses Portuguese examples derived from MultiNLI, FeverNLI, ANLI, LingNLI, and WANLI [7]. For all datasets, only the official training splits are used during fine- tuning; benchmark evaluation is conducted exclusively on the corresponding held-out test splits. This strict separation ensures that no evaluation pair is seen during training, preserving the integrity of benchmark comparisons. These mod- els should therefore be interpreted as Portuguese-adapted reference baselines trained under a common recipe in this work, enabling a controlled view of how language-specific adaptation affects performance within MTEB-PT. Training uses standard CoSENT [8] for scored STS pairs and Multiple Nega- tives Ranking Loss [16] for positive-pair supervision (NLI entailment and retrieval- oriented), both wrapped with MRL [6]. We apply the same training recipe across all three backbones for comparability and train each model so that it can be de- ployed at 64-, 128-, 256-, 512-, or full-dimensional embeddings. All models are trained for 20 epochs with an effective batch size of 512 (per-device batch size of 8Okamura et al. 100M1B 25 Number of Parameters 50 55 60 65 70 75 Mean (Task) multilingual-e5-small neobertugues mmbert-base mmbert-embed-32k modbertbr-matryoshka modbertbr multilingual-e5-base multilingual-mpnet-base gte-multilingual-base bertimbau-large legal-bertimbau-large-sts bertimbau-large-matryoshka multilingual-e5-large multilingual-e5-large-instruct e5-large-matryoshka qwen3-embedding-0.6b serafim-900m 128 500 1K 5K 10K 32K Max Tokens Fig. 1: Mean performance across tasks on MTEB-PT according to the number of parameters. The circle size denotes the full embedding size, while the color denotes the maximum sequence length of the model. Closed-source models are omitted because reliable parameter counts are not publicly available for all of them. 64 with gradient accumulation), a learning rate of 5× 10 −5 , weight decay of 0.2, and a warmup ratio of 0.1. 6 Results and Discussion This section presents the main findings on MTEB-PT using full-dimensional embeddings, analyzes how model rankings differ from MMTEB when evaluation is restricted to Portuguese, and then examines how these patterns change under dimensional truncation. 6.1 Main Benchmark Results Figure 1 summarizes how mean benchmark performance varies across open mod- els as a function of parameter count, embedding size, and maximum context length, while Table 3 shows where this aggregate view breaks down across task families. Closed-source systems are omitted from Figure 1 because consistent model-size metadata is not publicly available. Two broad patterns emerge from Figure 1. First, larger models tend to per- form better on average, but the relationship is clearly not monotonic: models with similar parameter counts can differ substantially in mean performance, indicating that architecture and training objective matter at least as much as scale alone. Second, the strongest open models in the aggregate view are sentence-oriented encoders rather than raw pretrained language models, with serafim-900m, qwen3-embedding-0.6b, and e5-large-matryoshka occupying the upper portion of the plot despite representing different training paradigms and MTEB-PT, a Benchmark for Portuguese Sentence Encoders9 Table 3: Results summary of the evaluated models across tasks, considering their full representation sizes (embeddings). *Closed-source retrieval results are approximate: WebFaqRetrieval was evaluated on only 20% of the dataset for these models, so their retrieval averages are not fully comparable to the other reported scores. ModelSTS Classification Retrieval Reranking Closed-source cohere-embed-v477.951.490.6 ∗ 86.9 text-embedding-3-large79.156.391.1 ∗ 82.0 amazon-titan-v277.050.286.9 ∗ 85.0 Peer-reviewed serafim-900m86.752.354.976.5 multilingual-e5-large79.251.374.584.2 multilingual-e5-large-instruct 81.353.070.184.5 multilingual-e5-base76.949.171.784.5 multilingual-e5-small76.949.370.284.0 qwen3-embedding-0.6b80.752.378.985.8 gte-multilingual-base80.450.276.984.2 legal-bertimbau-large-sts86.549.930.976.5 multilingual-mpnet-base78.750.658.081.1 bertimbau-large63.151.431.776.5 mmbert-base59.448.017.670.3 modbertbr51.249.427.276.7 Community Models mmbert-embed-32k76.449.567.382.7 neobertugues57.048.818.067.5 Fine-tuned PT MRL e5-large-matryoshka88.053.075.274.1 bertimbau-large-matryoshka 86.151.866.974.0 modbertbr-matryoshka85.248.855.877.6 context-length regimes. This aggregate picture is informative, but it does not fully characterize Portuguese embedding quality. Table 3 shows that performance remains strongly task-dependent, with substantial specialization across evalua- tion settings. On STS, the strongest results come from Portuguese-adapted models. In particular, e5-large-matryoshka achieves the best overall STS performance, while serafim-900m, legal-bertimbau-large-sts, and bertimbau-large-matryoshka form a closely competitive group. This indicates that Portuguese-specific supervision still matters, especially for symmetric semantic tasks where sentence-pair rank- ing objectives are closely aligned with evaluation. Meanwhile, these same mod- els do not consistently retain their advantage on search-oriented tasks, showing that strong STS performance should not be interpreted as evidence of uniformly strong Portuguese embeddings. This contrast is especially clear for serafim-900m, whose strong STS performance does not carry over to retrieval, likely reflecting the mismatch between short sentence-pair evaluation and longer retrieval docu- ments under its 128-token context limit (Table 2). 10Okamura et al. The picture is different for classification. Here, the spread between models is smaller, and the leading systems are more general-purpose embedding models rather than Portuguese specialists. This suggests that probe-based classifica- tion is less sensitive than STS to fine-grained sentence-level alignment, and that reasonably strong semantic structure is often sufficient to obtain competitive performance. This also helps explain why Portuguese fine-tuning yields more modest gains on classification than on STS. For retrieval and reranking, the benchmark reveals the strongest departure from the STS ranking. Closed-source models obtain the highest reported scores on these longer-input, asymmetric tasks, especially on retrieval, but the averages for these systems should be interpreted with caution because only 20% of Web- FaqRetrieval was used for evaluating closed-source models (due to research bud- get and dataset size). Among open models, search-oriented multilingual encoders such as qwen3-embedding-0.6b, the stronger E5 variants, and gte-multilingual- base are substantially more competitive than STS-specialized Portuguese mod- els. Taken together, these results suggest that multilingual strength does not transfer uniformly across task families, and that search-oriented training, often combined with longer context support, is advantageous on retrieval-style tasks. The comparison between raw encoder backbones and sentence-trained mod- els is also revealing. Portuguese MLM encoders such as bertimbau-large and modbertbr are weak in their off-the-shelf form, especially on STS and retrieval, despite being strong Portuguese language models in a pretraining sense. Once adapted with Portuguese contrastive supervision, however, their behavior changes substantially: bertimbau-large-matryoshka and modbertbr-matryoshka improve markedly on STS and also gain considerably on retrieval. This distinction is cen- tral to the benchmark: a strong Portuguese pretrained encoder is not necessarily a strong Portuguese sentence encoder, and sentence-level adaptation remains necessary for robust embedding quality. The Portuguese MRL baselines therefore provide a more nuanced result than a simple “fine-tuning works” conclusion. Their largest gains are concentrated on STS, which is consistent with the predominantly symmetric supervision used during adaptation. Yet the adapted models also improve on retrieval, especially relative to their non fine-tuned backbones, indicating that Portuguese sentence- level supervision transfers beyond the task structure most directly targeted dur- ing training. In contrast, reranking remains dominated by models that appear more explicitly optimized for search-style matching. Taken together, these results show that Portuguese embedding evaluation must remain task-diverse. Strong performance in the aggregate view of Figure 1, like strong performance on STS alone, is not sufficient to characterize model quality across Portuguese application settings. 6.2 MMTEB vs. MTEB-PT To further test whether multilingual benchmark performance transfers reliably to Portuguese, we compare model ranks on MMTEB and MTEB-PT across STS, classification, retrieval, and reranking. Because absolute scores are not directly MTEB-PT, a Benchmark for Portuguese Sentence Encoders11 Table 4: Models ranked by task on MTEB-PT. Each panel lists models sorted by their MTEB-PT rank. The rightmost value is the rank change relative to MMTEB: a green |▲| indicates improved rank on MTEB-PT, a red |▼| indicates worse rank, and — indicates no change. Only models with available MMTEB results for all four task families are included. STSClassification 1st multilingual-e5-large-instruct — 2nd qwen3-embedding-0.6b— 3rd gte-multilingual-base▲ 1 4th multilingual-e5-large▼ 1 5th text-embedding-3-large▲ 1 6th multilingual-mpnet-base▲ 2 7th multilingual-e5-small▼ 2 8th multilingual-e5-base▼ 1 1st text-embedding-3-large▲ 2 2nd multilingual-e5-large-instruct — 3rd qwen3-embedding-0.6b▼ 2 4th multilingual-e5-large— 5th multilingual-mpnet-base▲ 3 6th gte-multilingual-base— 7th multilingual-e5-small— 8th multilingual-e5-base▼ 3 RetrievalReranking 1st text-embedding-3-large▲ 1 2nd qwen3-embedding-0.6b▼ 1 3rd gte-multilingual-base— 4th multilingual-e5-large▲ 1 5th multilingual-e5-base▲ 1 6th multilingual-e5-small▲ 1 7th multilingual-e5-large-instruct ▼ 3 8th multilingual-mpnet-base— 1st qwen3-embedding-0.6b▲ 3 2nd multilingual-e5-base▲ 5 3rd multilingual-e5-large-instruct — 4th multilingual-e5-large▼ 2 5th gte-multilingual-base— 6th multilingual-e5-small— 7th text-embedding-3-large▼ 6 8th multilingual-mpnet-base— comparable across benchmarks, we focus on within-task rank changes. Table 4 reports model rankings on MTEB-PT and the corresponding rank differences relative to MMTEB. We include only models with available MMTEB results for all four task families. The comparison shows that MMTEB rank is informative, but not sufficient, for Portuguese model selection. Some broad structure is preserved, since several strong multilingual models remain competitive on MTEB-PT. However, the rel- ative ordering is not stable across task families, which indicates that multilingual performance does not transfer uniformly to Portuguese. The amount of reordering is, itself, task-dependent. STS is the most sta- ble setting, with only small rank changes and the top positions largely pre- served. Classification shows moderate movement, suggesting that multilin- gual strength carries over only partially in this regime. Larger shifts appear in retrieval, where models with similar multilingual standing separate more clearly on Portuguese. The strongest mismatch appears in reranking, where several models change position substantially, including a marked drop for text- embedding-3-large and strong gains for qwen3-embedding-0.6b and multilingual- e5-base. Overall, this comparison reinforces the main result of the benchmark: mul- tilingual evaluation provides useful prior information, but it cannot replace Portuguese-specific evaluation. The extent to which multilingual performance transfers depends on the task family, with the largest mismatches appearing on the more search-oriented settings. 12Okamura et al. 6.3 Dimension Truncation Analysis We next examine how performance changes under dimensional truncation for sentence encoder models. Figure 2 shows that robustness to compression is also task-dependent: STS and classification remain comparatively stable, whereas retrieval and reranking are more sensitive to aggressive truncation. 64 128256384512768 102415363072 Embedding Size 70 75 80 85 Score bertimbau-large-matryoshka e5-large-matryoshka modbertbr-matryoshka amazon-titan-v2 cohere-embed-v4 gte-multilingual-base legal-bertimbau-large-sts mmbert-embed-32k multilingual-e5-base multilingual-e5-large multilingual-e5-large-instruct multilingual-e5-small paraphrase-multilingual-mpnet-base-v2 qwen3-embedding-0.6b serafim-900m text-embedding-3-large modbertbr-matryoshka bertimbau-large-matryoshka e5-large-matryoshka (a) STS 64 128256384512768 102415363072 Embedding Size 40 42 45 48 50 52 55 Score bertimbau-large-matryoshka e5-large-matryoshka modbertbr-matryoshka amazon-titan-v2 cohere-embed-v4 gte-multilingual-base legal-bertimbau-large-sts mmbert-embed-32k multilingual-e5-base multilingual-e5-large multilingual-e5-large-instruct multilingual-e5-small paraphrase-multilingual-mpnet-base-v2 qwen3-embedding-0.6b serafim-900m text-embedding-3-large modbertbr-matryoshka bertimbau-large-matryoshka e5-large-matryoshka (b) Classification 64 128256384512768 102415363072 Embedding Size 0 20 40 60 80 Score bertimbau-large-matryoshka e5-large-matryoshka modbertbr-matryoshka amazon-titan-v2 cohere-embed-v4 gte-multilingual-base legal-bertimbau-large-sts mmbert-embed-32k multilingual-e5-base multilingual-e5-large multilingual-e5-large-instruct multilingual-e5-small paraphrase-multilingual-mpnet-base-v2 qwen3-embedding-0.6b serafim-900m text-embedding-3-large modbertbr-matryoshka bertimbau-large-matryoshka e5-large-matryoshka (c) Retrieval 64 128256384512768 102415363072 Embedding Size 70 72 75 78 80 82 85 88 Score bertimbau-large-matryoshka e5-large-matryoshka modbertbr-matryoshka amazon-titan-v2 cohere-embed-v4 gte-multilingual-base legal-bertimbau-large-sts mmbert-embed-32k multilingual-e5-base multilingual-e5-large multilingual-e5-large-instruct multilingual-e5-small paraphrase-multilingual-mpnet-base-v2 qwen3-embedding-0.6b serafim-900m text-embedding-3-large modbertbr-matryoshka bertimbau-large-matryoshka e5-large-matryoshka (d) Reranking Fig. 2: Task-wise performance under embedding truncation on MTEB-PT. The clearest MRL gains appear on STS, where the Portuguese baselines re- main strong across dimensions, especially e5-large-matryoshka. Classification is flatter overall, with smaller gaps between models; here, the adapted models re- main usable under compression, but their gains are more modest, consistent with the weaker alignment between pairwise semantic supervision and classification evaluation. The hardest settings are retrieval and reranking. In both, closed-source systems and strong multilingual search-oriented encoders remain strongest over- all, while the Portuguese MRL models improve substantially over their unfine- tuned backbones but do not displace the top search-oriented models. This is most evident for e5-large-matryoshka and bertimbau-large-matryoshka, which MTEB-PT, a Benchmark for Portuguese Sentence Encoders13 degrade more gracefully than their base encoders, indicating that Portuguese- specific adaptation transfers beyond STS even if it does not eliminate the ad- vantage of models more directly optimized for search. Overall, truncation does not remove the benchmark’s main pattern: model quality remains task-dependent, and compact embeddings are easier to obtain for symmetric semantic tasks than for search-oriented ones. MRL mainly im- proves the practicality of Portuguese-adapted models under tighter dimensional budgets, with e5-large-matryoshka remaining the most balanced adapted model across task families. 7 Limitations MTEB-PT has several important limitations. First, it is a curated Portuguese slice of existing MTEB/MMTEB tasks rather than a fully comprehensive eval- uation suite. Although it covers four task families, it does not yet capture the full diversity of Portuguese domains, genres, application settings, and linguis- tic phenomena. Notably, task families present in MTEB and MMTEB — such as clustering and bitext mining — are absent from MTEB-PT, not by design exclusion, but because no Portuguese-labeled subset meeting our quality and availability criteria was available within the MMTEB ecosystem at the time of construction. The absence of clustering is particularly relevant, as prior work on Brazilian Portuguese has shown that model behavior on grouping-based tasks can diverge substantially from STS and classification performance [12]. Expand- ing the benchmark with additional task families, and especially with clustering datasets, is therefore an important direction for future work and would allow a broader and more representative evaluation of Portuguese sentence encoders. Second, our evaluation focuses primarily on sentence encoders below one billion parameters, reflecting an emphasis on compact models that are feasible under modest memory and computational budgets. While this choice improves the practical relevance of the benchmark for low-resource deployment settings, it also limits the extent to which our results can speak to the performance of substantially larger embedding models. Evaluating larger models on MTEB-PT is therefore an important direction for future work, both to assess their actual gains on Portuguese and to determine whether those gains justify their additional computational cost relative to smaller encoders. Finally, the interpretation of closed-source systems remains limited. These models are important practical baselines, especially on retrieval and rerank- ing, but their training data and post-training procedures are not publicly docu- mented. Also, reported averages are further limited by the fact that these models were evaluated on only 20% of the WebFaqRetrieval dataset, due to dataset size and research budget. Therefore, results on the retrieval task should be viewed as preliminary, although they already indicate data contamination (the dataset might have been used in the training of these models). 14Okamura et al. 8 Conclusion We introduced MTEB-PT, a Portuguese benchmark for sentence embeddings built from 14 datasets spanning STS, classification, retrieval, and reranking. By evaluating 17 open- and closed-source models under a unified protocol, we showed that Portuguese embedding performance is strongly task-dependent, that multilingual strength does not transfer uniformly across task families, and that models with stronger long-context or search-oriented capabilities are particularly advantageous on retrieval and reranking. Our comparison with MMTEB further shows that multilingual rankings provide useful prior information, but do not reliably predict Portuguese-specific model behavior across tasks. We also used the benchmark to study Portuguese-specific adaptation through three MRL-trained baselines. These experiments show that language-specific fine-tuning remains effective, especially on STS, while also improving retrieval. Under dimensional truncation, the adapted models remain competitive, showing that compact Portuguese sentence representations can be obtained with favor- able efficiency–performance trade-offs. Overall, our findings reinforce the need for task-diverse, language-specific evaluation when selecting sentence encoders for Portuguese. We release MTEB- PT, the fine-tuned models, and the training and evaluation code to support reproducible research in Portuguese embeddings. We hope this benchmark en- courages new datasets and broader Portuguese evaluation. Acknowledgments. This study was financed in part by the Coordenação de Aper- feiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001. Anna H. Reali Costa would like to thank CNPq for the financial support, grant #312360/2023-1, and the INCT TILD-IAR, grant #408490/2024-1. References 1. Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., Liu, Z.: Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self- knowledge distillation (2024) 2. Enevoldsen, K., Chung, I., Kerboua, I., Kardos, M., Mathur, A., Stap, D., Gala, J., Siblini, W., Krzemiński, D., Winata, G.I., et al.: Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595 (2025). https://doi. org/10.48550/arXiv.2502.13595, https://arxiv.org/abs/2502.13595 3. Fialho, P., Coheur, L., Quaresma, P.: Benchmarking natural language inference and semantic textual similarity for portuguese. Information 11(10), 484 (2020) 4. Fonseca, E., Santos, L., Criscuolo, M., Aluisio, S.: Assin: Avaliacao de similaridade semantica e inferencia textual. In: Computational Processing of the Portuguese Language-12th International Conference, Tomar, Portugal. p. 13–15 (2016) 5. Gomes, L., Branco, A., Silva, J., Rodrigues, J., Santos, R.: Open sentence embed- dings for portuguese with the serafim pt* encoders family. In: EPIA Conference on Artificial Intelligence. p. 267–279. Springer (2024) 6. Kusupati, A., et al.: Matryoshka representation learning. arXiv preprint arXiv:2205.13147 (2022) MTEB-PT, a Benchmark for Portuguese Sentence Encoders15 7. Laurer, M., Atteveldt, W.v., Casas, A.S., Welbers, K.: Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint (Jun 2022), https://osf. io/74b8k, publisher: Open Science Framework 8. Li, P., Li, H.: Cosent: Consistent sentence embedding via similarity ranking. IEEE Access (2024), https://ieeexplore.ieee.org/document/10531646 9. May, P.: Machine translated multilingual sts benchmark dataset. (2021), https: //github.com/PhilipMay/stsb-multi-mt 10. Melo, R., Santos, P.A., Dias, J.: A semantic search system for the supremo tribunal de justiça. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds.) Progress in Artificial Intelligence. p. 142–154. Springer Nature Switzerland, Cham (2023) 11. Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: Mteb: Massive text embed- ding benchmark. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. p. 2014–2037. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/ v1/2023.eacl-main.148, https://aclanthology.org/2023.eacl-main.148/ 12. Peixoto, M., Silva, G., Figueredo, G., Silva, P., Luz, E.J.: Global vs. local sen- tence embeddings for brazilian portuguese: Revisiting monolingual models in the age of foundation models. In: Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026)-Vol. 1. p. 529–539 (2026) 13. Pinto, G.V.: Portuguese Sentence Embedding Models Using Matryoshka Repre- sentation Learning. Master’s thesis, Universidade Federal do Rio Grande do Sul (2025) 14. Real, L., Fonseca, E., Gonçalo Oliveira, H.: The assin 2 shared task: A quick overview. In: Proceedings of the ASSIN 2 Shared Task (2020), https://ceur-ws. org/Vol-2583/ 15. Real, L., Rodrigues, A., Vieira e Silva, A., Albiero, B., Thalenberg, B., Guide, B., Silva, C., de Oliveira Lima, G., Câmara, I.C., Stanojević, M., et al.: Sick-br: a portuguese corpus for inference. In: International Conference on Computational Processing of the Portuguese Language. p. 303–312. Springer (2018) 16. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP). p. 3982–3992 (2019) 17. SIL International: Ethnologue: Languages of the world (2023), https://w. ethnologue.com 18. Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: Pretrained BERT models for Brazilian Portuguese. In: Proceedings of BRACIS (2020) 19. Thakur, N., Reimers, N., Daxenberger, J., Srivastava, A., Gurevych, I.: Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021) 20. Wang, L., et al.: Multilingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 (2024) 21. Warner, B., et al.: Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. arXiv preprint arXiv:2412.13663 (2024) 22. Wu, W.B.T.L., Garcia, L.P.F.: Modbertbr: A modernbert-based model for brazil- ian portuguese. In: Encontro Nacional de Inteligência Artificial e Computacional (ENIAC). p. 2044–2055. SBC (2025)