Paper deep dive
DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts
Vlad Hondru, Florinel Alin Croitoru, Iuliana Georgescu, A. Sophia Koepke, Radu Tudor Ionescu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We therefore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-DD, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF-MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.23363v1
- Canonical: https://arxiv.org/abs/2608.23363v1
Trouble viewing inline? Open PDF directly →
Full Text
81,957 characters extracted from source content.
Expand or collapse full text
arXiv:2608.23363v1 [cs.CV] 24 Aug 2026 HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS1 DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts Vlad Hondru 1 Florinel Alin Croitoru 1 Iuliana Georgescu 1 A. Sophia Koepke 2,3 Radu Tudor Ionescu 1 raducu.ionescu@gmail.com 1 University of Bucharest Bucharest, Romania 2 Technical University Munich, MCML Munich, Germany 3 University of Tübingen, Tübingen AI Center Tübingen, Germany Abstract Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation meth- ods. We conjecture that overfitting can be mitigated by extracting multiple high-level cues from the available audio and visual modalities via pre-trained models. We there- fore assemble a wide variety of pre-trained models to extract features that encode mouth movements, face parsing, facial expressions, head pose, gaze tracking, heart rate, audio emotion and speech activity. We further integrate both unimodal and multimodal cues via a Mixture-of-Experts (MoE) backbone to detect deepfakes. We perform in-domain and cross-domain experiments on five benchmarks for deepfake detection (MAVOS-D, AVLips, PolyGlotFake, BioDeepAV, FakeAVCeleb) to compare our framework (DF- MoE) with state-of-the-art methods. Our results indicate that DF-MoE obtains superior deepfake detection results, surpassing all competing methods. We release our code at https://github.com/vladhondru25/DF-MoE. ✅ Real ❎ Fake Input video Legend: # frozen $ trainable Pre-trained models # MoE $ Head pose Gaze rPPG Facial expression Face segments Deep features Audio emotion Lip sync Figure 1: Overview of the proposed pipeline for deepfake detection. First, we employ mul- tiple pre-trained models to extract high-level semantic cues. Then, a lightweight Mixture- of-Experts (MoE) transformer aggregates the extracted signals and learns to classify audio- visual inputs as real or fake. Our design prevents overfitting to a specific dataset by keeping the pre-trained models frozen, while only training the lightweight MoE on the target task. 1 Introduction With the continuous advancements in generative AI [11, 30, 46, 101, 118] and the increasing democratization of generative technology, concerns towards potential misuse are rising [19]. © 2026. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. 2HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS Deepfakes, i.e. generated multimedia (audio-visual) content that aims to deceive humans into believing that the content is real, are predominantly used by malicious users in financial scams, misinformation and identity theft. Since humans, especially vulnerable groups (e.g. elders, visually impaired, etc.), may be easily deceived by deepfakes, an important area of re- search is the development of deepfake detectors [8, 38, 75, 76, 89, 104, 105, 108, 120]. One of the main challenges in this area is to obtain detectors able to generalize across deepfake generation methods, a key characteristic that is required to overcome the restless develop- ment of generative AI technology. Despite its importance, most studies in deepfake detection leave this aspect aside, focusing on improving performance across existing deepfake detec- tion benchmarks [8, 43, 76, 80, 87]. While many works report impressive performance (up to 99% accuracy) in deepfake detection on public benchmarks, they fail to determine the generalization capacity of the detectors on deepfakes produced by newer and more advanced generative methods. Against the mainstream practice in the field, several studies attempted to address the generalization concern revolving around deepfake detectors [54, 67, 70, 111]. For instance, Ma et al. [67] noticed that current deepfake models create either face incon- sistency or up-sampling artifacts, and leveraged this information to generate pseudo-fake training data. Lai et al. [54] proposed the Generalized Multi-Scenario Deepfake Detection framework to enable jointly training a model on multiple datasets, since naively training a deepfake model on multiple deepfake datasets decreases the overall performance. Unlike previous studies in this area [54, 67, 70, 111], which usually fine-tune mod- els on deepfake data, we propose to overcome the dataset overfitting issue by extracting multiple high-level cues from the available audio and video modalities via pre-trained mod- els, as shown in Figure 1. Our assumption is that models that are pre-trained on distinct tasks, e.g. facial expression recognition, audio emotion recognition, head pose estimation, or gaze tracking, provide meaningful information about the authenticity of audio-visual content, without risking overfitting to artifacts specific to a certain generative model. We therefore aggregate such pre-trained models into a unified pipeline, while deferring the training stage on real vs. fake content classification to a lightweight Mixture-of-Experts backbone [86] that combines all the high-level (semantic) features. To avoid learning the distinctive patterns of different deepfake generation methods, we further introduce a novel contractive-repulsive objective (CRO) that contracts latent vectors around corresponding class anchors (aiming to reduce distinctive patterns inside each class), while repelling class anchors beyond a given margin (aiming to enforce better discrimination between classes). By expressing our con- trastive objective through class anchors, we avoid searching for positive/negative pairs via expensive hard sample mining [27, 28, 34, 84, 90]. Our deepfake detection based on sparse Mixture-of-Experts (DF-MoE) is not only designed to improve generalization capacity, but also to maintain a reasonable compute time, achieving near real-time processing speed on a single Nvidia RTX 5090 GPU with 32GB VRAM. We perform experiments on five recent and challenging benchmarks (MAVOS-D [18], AVLips [64], PolyGlotFake [38], BioDeepAV [19], FakeAVCeleb [49]), showing that DF- MoE outperforms state-of-the-art deepfake detectors [1, 8, 21, 38, 49, 60, 72, 75, 76, 80, 89, 97, 104, 105, 108, 111, 120] across all five datasets, in both open-set and cross-domain evalu- ation scenarios. Ablation studies confirm that standalone high-level features from pre-trained models are individually well-suited for deepfake detection. Yet, combining all features via MoE is the best way to unleash their potential towards generalizable deepfake detection. Finally, we show that our framework takes a leap forward towards interpretable decisions via attributing gradient weights to the high-level features from pre-trained models, i.e. DF- MoE can determine that a video clip is fake because it exhibits unusual facial expressions, HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS3 inconsistent speech and mouth movements, unexpected gaze behavior, etc. In summary, our contribution is fourfold: • We propose DF-MoE, a novel deepfake detection framework that employs a sparse Mixture-of-Experts transformer to integrate multiple complementary cues into an end- to-end pipeline. • We introduce CRO, a novel contractive-repulsive (contrastive) loss that improves the generalization capacity across deepfake generation methods, harnessing learnable class anchors to improve latent space organization, while avoiding expensive hard sample mining techniques employed by conventional contrastive learning objectives. • We demonstrate the high generalization capacity of DF-MoE in a suite of challenging open-set and cross-domain deepfake detection experiments, where DF-MoE surpasses state-of-the-art deepfake detectors. • We show that DF-MoE provides a leap forward towards interpretable decisions, by inherently determining which high-level features contribute to the final decision. 2 Related Work Based on the input domain, deepfake detection methods can be categorized into image- level [5, 9, 10, 14, 16, 22, 23, 25, 37, 43, 45, 47, 52, 55, 56, 57, 58, 59, 62, 71, 74, 79, 87, 92, 95, 96, 98, 103, 105, 106, 109, 110, 116, 117, 121], audio-level [2, 24, 42, 48, 93, 94, 99, 100, 115] and multimodal methods [3, 4, 17, 26, 33, 44, 51, 53, 65, 73, 75, 78, 83, 89, 108, 114, 119, 120]. The initial efforts have employed convolutional neural networks (CNNs) [5, 9, 10, 12, 14, 16, 22, 23, 25, 37, 43, 45, 47, 52, 55, 56, 57, 58, 59, 62, 71, 74, 79, 87, 91, 92, 95, 96, 98, 103, 105, 106, 109, 110, 116, 117, 121] and recurrent neural networks (RNNs) [29, 40, 63, 69, 82]. More recently, most studies have adopted the transformer ar- chitecture [3, 6, 26, 33, 44, 53, 65, 73, 75, 83, 89, 108, 114, 119, 120], with the primary goal of jointly analyzing the video and audio content. Consequently, the recently proposed state-of-the-art methods [4, 75, 119, 120] for deepfake detection are multimodal. Zhou et al. [119] proposed one of the early frameworks on jointly modeling audio and video streams to perform deepfake detection. The framework was based on the synchro- nization between video and audio, and different fusion approaches, such as late-fusion and two-plus-one. Several other works [4, 75] also relied on the synchronization between audio and video. For instance, Astrid et al. [4] employed pseudo-fake generation to improve the detection of local inconsistencies between audio and video. AVFF [75] captures the corre- spondence between audio and video modalities by first pre-training the model on real train- ing data, then performing supervised training using both unimodal and cross-modal features. Switching to a more fine-grained output, Delocate [41] performs both detection and local- ization of deepfakes in videos in two stages, by first reconstructing the faces in the frames, and then classifying them. For a more efficient training, Hui et al. [68] proposed multi-task audio-visual prompt learning, by injecting prompts into each layer of an audio-visual foun- dation model, without updating the entire model. To efficiently process a video, TALL [104] transforms it into a thumbnail, and then applies the image-level Swin Transformer [66] to perform video deepfake detection. Several works [54, 67, 70, 111] focused on the generalization of the deepfake detectors to out-of-domain data. Ma et al. [67] categorized the deepfake artifacts into face incon- sistency and up-sampling artifacts, noticing that existing deepfake models exhibit either or 4HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS both kinds of artifacts. Leveraging this observation, Ma et al. [67] generated pseudo-fake data to train their deepfake model, obtaining a robust classifier. Nadimpalli et al. [70] tack- led the deepfake detection task using a hybrid combination of supervised and reinforcement learning techniques to improve cross-domain performance. Lai et al. [54] observed that naively training a model on multiple datasets does not improve the joint performance for deepfake detection. Therefore, they proposed the Generalized Multi-Scenario Deepfake De- tection framework to enable the joint training of a model on multiple datasets, including a domain-aware meta-learning strategy. Closer to our work, optimizing for both increased generalization and short training time, Yermakov et al. [111] fine-tuned only the layer nor- malization parameters of a foundational pre-trained vision encoder. They obtained good results on several cross-dataset scenarios. Different from previous works on generalizable deepfake detection [54, 67, 70, 111], we propose to extract different signals from available pre-trained models, overcoming overfitting and increasing the out-of-domain generalization capabilities by keeping these models frozen. To our knowledge, we are the first to integrate a wide range of high-level semantic cues for deepfake detection. The integration is performed by a learnable MoE block, which is carefully trained to mitigate overfitting on deepfake generative models seen at training via a novel contractive-repulsive objective. 3 Method Spotting a deepfake video requires reasoning over multiple signals simultaneously, espe- cially for videos with photorealistic facial manipulations generated by state-of-the-art mod- els, in which the artifacts are very difficult to recognize, even for humans. Nevertheless, subtle inconsistencies often persist across time: unnatural head movements, irregular gaze patterns or audio-visual mismatches. To this end, we introduce DF-MoE, a multimodal deep- fake detection framework that harnesses high-level semantic cues extracted from pre-trained models. We aggregate the semantic cues and model their temporal dynamics via a sparse MoE architecture. Instead of directly operating on raw pixels, our design decomposes the task into (i) extracting semantically meaningful visual and audio descriptors, and (i) learning cross-modal and temporal inconsistencies to capture the manipulated content. We showcase our full pipeline in Figure 2, and explain its components in detail below. 3.1 Extracting Visual and Audio Cues Our pipeline for feature extraction begins by splitting each video into frames. Then, on every fifth frame, we apply a face detector based on YOLOv11 [50] to extract the faces. The following step is to employ a face tracking model. For each pair of consecutive pro- cessed frames, we employ a face tracker based on DeepSORT [102] to match the faces and form long-term continuous tracking paths along the video. Each tracking path in a video corresponds to one person, and is processed independently through the pipeline. For each face track, we apply pre-trained models to extract a series of visual features: head pose, eye gaze, face segmentation, remote photoplethysmography (rPPG), and facial expression. At the same time, we take the associated audio signal and extract speech embeddings with Whisper-Tiny [77]. These are also used to compute emotion features from a custom pre- trained model for audio emotion recognition. Head pose. Deepfake methods may produce head movements with unnatural rigidity, subtle temporal jitter or physically implausible transitions (especially portrait animation models), HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS5 ... Head pose Gaze tracking Face parsing Face emotion rPPG Audio emotion Whisper Tiny AVFF Specialised encoders Sparse Mixture-of-Experts Classification head Linear ReLU Dropout Linear Gating Expert 1 Expert 2 Expert n ... Multi-Head Self-Attention ... Face detection & tracking AV Transformer !"# ☹%& !"# ☹%& Time (s) Signal ✅ Real ❎ Fake Input video Legend: ) frozen * * ) * ) ) ) ) ) ) ) ) ) * trainable Figure 2: Detailed overview of our DF-MoE framework for deepfake detection. Face de- tection and tracking are applied to obtain face tracks. Next, multiple pre-trained models (frozen) are employed to extract high-level semantic cues (features). The extracted features are encoded into tokens via specialized encoders (trainable adapters), ensuring consistent dimensionality across different cues. The resulting tokens are processed by a sparse MoE transformer (trainable). Best viewed in color. which can be detected by modeling their temporal consistency. Therefore, for each frame, we estimate the 3D head pose using HopeNet [81], which returns three head pose angles (roll, pitch and yaw), denoted as F 3 HP . Gaze estimation. Synthetic videos often fail to preserve natural gaze stability and coordina- tion between both eyes, leading to measurable irregularities over time. The focus of the eyes on a specific point (gaze) is estimated by regressing the three Euler angles with a ResNet- 34 [35]. We drop the roll angle, keeping only the horizontal and vertical angles, as the eyes cannot rotate along the roll direction. For each frame, its gaze features are denoted as F 2 Gaze . Face segmentation. We hypothesize that analyzing the temporal consistency of face parts can capture micro-movements and subtle spatial inconsistencies. To capture inconsistencies, we apply the Bilateral Segmentation Network [112] and obtain a facial segmentation map F 512×512 Seg with labeled face parts. rPPG. Since heart rate is estimated via a weak rPPG signal (mostly invisible to humans), we conjecture that deepfake video generators do not insert such weak signals in generated video. Therefore, extracting rPPG signals can help distinguish between real and fake videos. We extract remote photoplethysmography F 5×512 rPPG via the model proposed by Yue et al. [113]. This model captures subtle periodic color variations in facial skin regions caused by blood flow. The rPPG signal can provide an estimate of the subject’s physiological pulse (heart rate) without physical contact. We estimate the rPPG waveform for five different face points. Facial expression. Deepfakes can be characterized by irregular temporal transitions between 6HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS facial expression classes or by inconsistent facial expression and speech emotion classes. Therefore, given only the visual input, a facial expression recognition (FER) model is applied to classify the face in each frame into one of the eight basic emotions: anger, contempt, disgust, fear, happiness, neutral, sadness and surprise. We employ an EfficientNet-B0 [97] pre-trained on Video-level Group Affect [85], which produces a vector of probabilities across the eight emotion classes, denoted as F 8 FER . Speech embeddings. Since speech signal is imperative for audio-visual deepfake detec- tion, we take a pre-trained Whisper-Tiny [77] and encode speech via its latent representation F 384 Speech . The speech embeddings contain information about the phonetic content, capturing potential irregularities of synthesized or voice-cloned speech. Audio emotion. We aim to complement facial expressions with emotion classes from the audio input. To this end, we pre-train a simple MLP classifier over the extracted speech em- beddings for speech emotion recognition (SER) on the CREMA-D [7] dataset. The resulting output, denoted as F 8 SER , provides cues about the temporal consistency of speech emotion, that can be used for assessing the consistency between audio and visual emotion. Lip sync. Lip syncing is a crucial signal for detecting deepfakes. Therefore, we pre-train a custom transformer that aims to detect inconsistencies between the lip movement and speech. We use the output of the face parser (Bilateral Segmentation Network [112]) to obtain a crop around the mouth. We concatenate 50 crops and obtain their DINOv3 [88] embeddings. Sim- ilarly, we extract the Whisper-Tiny embeddings from the corresponding audio. The resulting visual and audio embeddings are fed into our cross-modal audio-visual (AV) transformer. The model consists of self-attention layers, followed by cross-attention layers, and outputs a 512-dimensional vector F 512 LS . AVFF. AVFF is a two-stage transformer framework for deepfake detection [75]. The first stage adopts unimodal encoders and decoders along with cross-modal networks that are trained with reconstruction and contrastive objectives to capture audio-visual correspon- dences. The second stage fine-tunes the model on deepfake classification, so we discard this stage to prevent overfitting. Specifically, we take the unimodal audio and video represen- tations before the classification stage. Each unimodal representation is a 1024-dimensional vector. We concatenate the two vectors to obtain a deep feature vector denoted as F 2048 AVFF . To mitigate overfitting on deepfake detection, we start from the pre-trained AVFF based on self-supervision and optimize only the unimodal transformer blocks via Effort [107]. This is a LoRA-style [39] approach that decomposes the weight matrices via Singular Value De- composition (SVD), freezes the principal components, and fine-tunes only the remaining components. Effort is specifically designed to offer strong generalization in deepfake de- tection [107]. Moreover, we keep the cross-modality fusion modules (A2V Network and V2A Network) frozen, since interactions between the audio and visual modalities can also contribute to overfitting. 3.2 Specialized Adapters To aggregate the high-level cues into a joint architecture, all features are projected into a shared embedding space of size h = 128, using specialized adapters. Movement adapter. The head pose F 3 HP and eye gaze F 2 Gaze are fed into a two-layer bidi- rectional LSTM [36]. Then, a cross-attention layer lets the gaze attend to the pose (gaze as queries, head pose as keys and values). In the end, the cross-attention output and the gaze representation (passed via a skip connection) are mean-pooled over time, concatenated, and projected. We denote the final head movement and gaze features as E HP+Gaze ∈R h . HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS7 Face parsing adapter. The input of this encoder consists of the face segmentation maps F 512×512 Seg . The architecture is represented by 3D CNN with 3D convolution and max pooling layers along time and space, followed by a global average pooling and linear layers. The resulting features are denoted as E Seg ∈R h . rPPG adapter. The rPPG encoder takes the rPPG signal F 5×512 rPPG as input. It is composed of two 1D convolutional layers, followed by average pooling and two bidirectional GRUs. The final features are denoted by E rPPG ∈R h . Emotion adapter. This encoder uses both discrete emotions (from audio F 8 AER and video F 8 FER ). Each stream is passed through a shared embedding layer, and then processed by separate bidirectional GRUs [13]. The resulting representations are combined using a cross- attention layer, concatenated with the video embeddings, and jointly projected. The resulting features are denoted by E Emo ∈R h . Lip sync adapter. This encoder takes the representation from our custom AV transformer, pools it over time with an adaptive average and projects it into the shared embedding space. We denote the final lip sync features by E LS ∈R h . Audio-visual adapter. The features obtained with AVFF, F 2048 AVFF , are simply projected through a linear layer to the shared embedding space. We denote the resulting feature vector by E AVFF ∈R h . 3.3 Sparse Mixture-of-Experts To classify a video as real or deepfake using the projected embeddings, we employ a sparse Mixture-of-Experts transformer. The projected embeddings are concatenated into a sequence S, as follows: S = E HP+Gaze ,E Seg ,E Emo ,E rPPG ,E LS ,E AVFF ∈R 6×h .(1) The sequence S is then processed by a multi-head self-attention module, enabling cross- feature modeling, before getting routed to the experts. The fact that experts can focus on different types of signals is particularly useful in our case, since tokens embed different high-level information from a wide variety of pre-trained models. Mixture-of-experts. The gating network is represented by a linear layer, denoted as g(·), that gives a ranking score for the available experts. The gating mechanism assigns routing weights to indicate which expert is most suitable for a given token. Formally, for each token s i of the sequence S =s i 6 i=1 , we rank all experts based on the returned values g(s i ) j ,∀ j∈ 1,...,n, where n is the number of experts. Subsequently, each token s i is passed through the top-k scoring experts. Let J i = j i 1 ,..., j i k ,∀i∈1,..., 6 denote the set of indices to which the token s i is routed. Based on the ranking scores, we compute the weights p i that are used to determine the output tokens of the MoE layer, given the input tokens S. Specifically, we employ the following equation, obtaining normalized weights for the experts: p i j = exp(g(s i ) j ) ∑ l∈J i exp(g(s i ) l ) , j∈ J i ,i∈1,..., 6.(2) After computingp i 6 i=1 , we can determine the output sequence ˆ S. Formally, if we denote the experts with the top-k highest scores in p i bye j (·) j∈J i ,∀i∈1,..., 6, then each output token ˆs i of the MoE layer is computed as: ˆs i = k ∑ j=1 p i j · e j (s i ).(3) Intuitively, Eq. (3) uses the values in p i as weights for the representations returned by the 8HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS experts. If an expert has a higher score in p i , then its representation is more important in Eq. (3). Dropout regularization. At training time, we include a dropout regularization on the ex- perts routing to avoid their over-specialization on a certain type of feature. The dropout is implemented by creating a mask m i 6 i=1 ∈0, 1 n that corresponds to each score vector g(s i ) 6 i=1 . We randomly set some of the positions in m i to 0, and the remaining ones to 1. The number of values assigned to 0 is controlled by a dropout rate d = 0.2. The output of the dropout layer is given by ˆg(s i ) = g(s i )∗m i , where∗ is the element-wise product. ˆg(s i ) is further used in the top-k routing logic described before. Training losses. Our final training objective comprises three components. The first one is the classic binary cross-entropy (BCE) loss over real and deepfake class labels. For this loss, we average the representations stored in the sequence ˆ S = ˆs i 6 i=1 to obtain the feature vector R = 1 6 ∑ ˆs i ∈ ˆ S ˆs i that is fed into the final classifier c(·). Given the ground-truth label y∈0, 1 of the input video, the binary cross-entropy loss is: L BCE =−[y· log(c(R))+(1−y)· log(1−c(R))].(4) If deepfakes are spread around in the latent space, e.g. all around the real class region, it is likely that deepfakes generated by unknown generative methods will be confused with real samples. We therefore propose an additional objective to assist the standard BCE in structuring the latent space of the model, so as to reduce the discrimination power among various deepfake types, while boosting discrimination between real and fake samples. Our novel contractive-repulsive objective (CRO) is designed to reduce representation diversity inside classes (intra-class contraction), while increasing the gap between real and deepfake classes in the latent space (inter-class repulsion). This is achieved via a combination of three loss terms that operate with learnable class anchors. Let A∈R h and B∈R h denote the learnable anchors for the real and fake classes, respectively. We define the components of the CRO loss as follows: L contract (A,B,R) = (1−y)·∥R−A∥ 2 2 +y·∥R−B∥ 2 2 , L repel (A,B) = 1 2 [max(0, 2·M−∥A−B∥)] 2 , L no-collapse (A,B) = 1 2 [max(0,P−∥A∥)+ max(0,P−∥B∥)] 2 , (5) where M> 0 represents the margin between the two anchors, and P is the minimum norm for each anchor. The anchors A and B are randomly initialized before training, and updated at every training iteration with the other trainable parameters. In our experiments, we set M = 1 and P = 1. L contract minimizes the distance between the feature vector R and its corresponding anchor, A or B, depending on label y. L repel enforces a minimum margin between the anchors A and B, ensuring that classes are sufficiently far apart. Since class anchors A and B are learnable, a concentric configuration of real and fake latent vectors (i.e. a cluster of real samples surrounded by a band of fake samples) might bring the anchors in the same vicinity, which might put the first two objectives into conflict. To avoid the collapse of class anchors, we introduceL no-collapse , which pushes anchors away from the origin. Finally, L CRO is defined as: L CRO =L contract +L repel +L no-collapse . (6) An advantage of our CRO loss over traditional contrastive losses is that it operates relative to a set of learnable anchors. Consequently, it achieves feature separation without requiring HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS9 computationally-expensive procedures to mine adequate sample pairs [27, 28, 34, 84, 90]. The third loss term penalizes the gating network if one expert receives too many tokens and the predicted probability for an expert is too high. This loss promotes diversity in the top-k selected tokens. To compute this loss, we determine the fraction of tokens routed to each expert f j n j=1 and the average probability score ̄p j n j=1 assigned to each expert, where ̄p j = 1 6 ∑ 6 i=1 p i j . In practice, both f j and ̄p j are estimated on an entire mini-batch. These two variables are combined into a loss function defined as follows: L aux = n n ∑ j=1 f j · ̄p j .(7) The final loss is a combination of those defined in Eq. (4), Eq. (6) and Eq. (7): L =L BCE +λ 1 ·L CRO +λ 2 ·L aux ,(8) whereλ 1 andλ 2 are hyperparameters that control the importance of the additional losses. 4 Experiments Datasets. To assess the generalization capacity of DF-MoE, we evaluate it on five datasets: AVLips [64], MAVOS-D [18], PolyGlotFake [38], BioDeepAV [19] and FakeAVCeleb [49]. In terms of evaluation setups, we conduct (i) in-domain experiments on AVLips [64] and MAVOS-D [18], (i) open-set experiments on MAVOS-D [18], as well as (i) cross- dataset experiments on PolyGlotFake [38], BioDeepAV [19] and FakeAVCeleb [49]. AVLips [64] is designed for training and evaluating lip-sync forgery detection models. The dataset contains over 3,000 real and 4,000 manipulated videos. Throughout our experi- ments, we utilize the official train and test splits. MAVOS-D is a multilingual benchmark that provides a training set (21K videos), a validation set (4K videos) and four test sets (to- gether containing over 60K test videos). The evaluation protocol is designed to support four setups: closed-set, open-set model, open-set language and open-set full. The closed-set test set evaluates detectors on samples that correspond to the set of languages and generative methods seen during training. The open-set model test set introduces new (unseen) genera- tive methods. The open-set language introduces two unseen languages, Hindi and German. Finally, the open-set full test set simultaneously evaluates performance on unseen languages and unseen methods. For the cross-dataset evaluation, we choose three recent datasets, PolyGlotFake [38], BioDeepAV [19] and FakeAVCeleb [49]. PolyGlotFake is a multi-lingual dataset that con- tains 766 real and 14,472 fake videos. Compared to MAVOS-D, PolyGlotFake has two new languages, French and Japanese. The manipulation methods of PolyGlotFake consist of text-to-speech, voice conversion and lip-synchronization methods. BioDeepAV comprises 2,010 real and 1,693 fake videos. This dataset comprises diverse generative approaches, in- cluding NeRF-based synthesis, Gaussian Splatting, and diffusion models. FakeAVCeleb is a very imbalanced dataset, comprising only 500 real and over 19,000 fake videos. We under- line that typically employed metrics, such as AUC, can be misleading for highly imbalanced datasets. For example, if a model successfully isolates high-confidence fakes at the strictest thresholds, it rapidly increases the True Positive Rate (TPR), securing a high overall AUC. To ensure a correct evaluation and to avoid any bias in our evaluation metrics, we supplement the set of real videos with additional 19,000 videos randomly sampled from VoxCeleb2 [15]. We highlight that this video addition is also suggested by the authors of FakeAVCeleb [49] in their original work. The resulting dataset is further referred to as Vox+FakeAVCeleb. 10HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS Method Closed-setOpen-set modelOpen-set lang.Open-set full mAP AUCaccmAP AUCaccmAP AUCaccmAP AUCacc TALL [104]0.84 0.85 75.610.67 0.73 62.400.77 0.78 70.140.69 0.72 63.86 MRDF [120]0.90 0.92 79.550.75 0.81 67.860.88 0.89 76.910.80 0.83 70.95 AVFF [75]0.97 0.97 90.710.93 0.94 85.100.93 0.93 86.580.91 0.93 84.85 AVH-Align [89]0.56 0.56 52.480.51 0.51 62.660.57 0.57 58.110.54 0.54 57.78 AVH-Align sup [89] 0.81 0.83 75.560.69 0.74 63.840.76 0.78 72.750.70 0.73 64.90 DF-Linear0.97 0.97 90.570.93 0.93 85.280.94 0.94 84.060.93 0.93 83.98 DF-MoE (ours)0.99 0.99 97.73 0.98 0.98 93.64 0.99 0.99 94.55 0.98 0.98 92.95 Table 1: Results on MAVOS-D obtained by TALL [104], MRDF [120], AVFF [75], AVH- Align [89], AVH-Align sup [89], DF-Linear, and our DF-MoE. The best performing method is highlighted in blue bold, and the second-best in orange. Our DF-MoE outperforms all previous state-of-the-art methods, regardless of the evaluation setup. Baselines. We evaluate our method on AVLips and MAVOS-D against several state-of-the- art methods: TALL [104], MRDF [120], AVFF [75], AVH-Align [89], AVH-Align sup [89], RealForensics [32], LipForensics, and LipFD [64]. While most of these baselines lever- age multimodal audio-visual features, TALL relies exclusively on video artifacts. For the cross-dataset evaluation on PolyGlotFake, BioDeepAV and Vox+FakeAVCeleb, we include additional baselines, e.g. UCF [105], StA [108], RECCE [8], GenD [111], among many others [1, 21, 38, 49, 60, 72, 76, 80, 89, 97, 104]. Ablated models. We carry out ablation studies to assess the impact of each high-level cue integrated in DF-MoE. We also ablate the joint MoE module, employing a custom linear classifier instead, resulting in a version called DF-Linear. DF-Linear aggregates the same semantic cues as DF-MoE, serving as a critical ablation to isolate the performance gains brought by our MoE architecture. Hyperparameters. All the models, including the baselines, are trained for 10 epochs, with the optimal checkpoint selected based on validation performance. For the baselines, the hyperparameters (learning rate, optimizer, etc.) are configured in accordance with the official recommendation from their corresponding publications. For DF-MoE and DF-Linear, we employ AdamW as the optimizer, with a learning rate of 10 −4 , and a batch size of 4. The projection dimension for the shared latent space of the specialized encoders is set to h = 128, the number of experts n is set to 6, and each token is routed to k = 2 experts, based on the scores provided by the gating network. The specialized encoders vary in architecture, as per Section 3.2. However, these encoders generally integrate a hidden bottleneck layer, with a latent representation of 64 dimensions. The weight for the CRO loss isλ 1 = 1, and the weight of the auxiliary loss is set toλ 2 = 0.01. More reproducibility details are provided in the supplementary. Evaluation measures. We report mean average precision (mAP), area under the ROC curve (AUC), and accuracy (acc). In-domain results. In Table 1, we present the results of DF-MoE on all four MAVOS-D evaluation scenarios. We observe that DF-MoE consistently yields better performance across every setup. Remarkably, DF-MoE exhibits significantly greater robustness in the open-set model scenario compared with the strongest competitor (AVFF), maintaining higher perfor- mance stability due to our integration of multiple high-level cues. Overall, feature diversity is a strong point of our method, as it helps the detection model to observe different fail- ure cases of the generative models. Our MoE-based architecture has an important role in increasing the robustness to unseen generative methods, being capable of correctly balanc- HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS11 Method AVLips mAP AUCacc RealForensics [32] 0.90-91.78 LipForensics [31]0.82-86.13 LipFD [64]0.93 0.95 95.27 AVH-Align [89]-0.89- AVFF [75]0.89 0.89 76.64 DF-MoE (ours)0.97 0.97 93.77 Table 2: In-domain results on AVLips [64] obtained by state-of-the-art models vs. DF-MoE. The best performing method is highlighted in blue bold, and the second-best in orange. Our DF-MoE outperforms all previous state-of-the-art methods in terms of mAP and AUC. rPPG Face HP+Gaze AVAVAVFF+ mAP AUCacc segmentationemotion transformerEffort ✓✗0.730.7466.18 ✗✓✗0.730.7266.58 ✗✓✗0.660.6962.51 ✗✓✗0.780.7870.48 ✗✓✗0.880.8881.40 ✗✓0.960.9688.58 ✓✗0.740.7467.33 ✓✗0.740.7568.28 ✓✗0.780.7868.18 ✓✗0.920.9285.85 ✓0.980.9892.95 Table 3: Ablation study on the impact of each feature type on the final performance of DF-MoE on MAVOS-D (open-set full). The best individual components are AV Trans- former and AVFF. However, the complementary cues brought by the other pre-trained mod- els (HP+Gaze, rPPG, audio-visual emotion, face segmentation) bring consistent performance gains, all contributing to the final performance of DF-MoE. ing complementary high-level cues to achieve substantial performance boost in the open-set model setup. In Table 2, we present the results of DF-MoE on AVLips [64]. The results demonstrate the superior performance of DF-MoE in terms of both mAP and AUC. Notably, while AVFF is an important component of our architecture, it yields significantly lower re- sults when evaluated in isolation. This performance gap further underscores the importance of integrating highly diverse features via DF-MoE. While LipFD achieves higher accuracy than our method on AVLips, it is specifically designed to detect temporal inconsistencies between lip movements and audio (making it specifically suitable for AVLips), whereas DF- MoE provides a more general framework for multimodal deepfake detection. Ablation studies. To validate the utility of the sparse MoE, we compare DF-MoE with DF- Linear in Tables 1 and Table 4. DF-Linear replaces the MoE module with a linear layer applied on the concatenation of all high-level features. While DF-Linear generally surpasses state-of-the-art detectors in the in-domain setting, its in-domain performance is consistently below our DF-MoE. Significant gaps are also observed for the cross-domain evaluation on PolyGlotFake and Vox+FakeAVCeleb. In Table A1, we present a comprehensive ablation study designed to isolate and quan- titatively evaluate the contribution of each individual feature type to the final performance. This analysis is conducted on the open-set full setup of MAVOS-D. The ablation results indicate that the most important representations are extracted by the two audio-visual mod- 12HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS Year Method PolyGlotFake [38]BioDeepAV [19]Vox+FakeAVCeleb [49] mAP AUCaccmAP AUCaccmAPAUCacc 2018 MesoNet [1]-0.57------- MesoInception [1]-0.58------- DSP-FWA [60]-0.67------- 2019 XceptionNet [80]-0.61--0.57---- EfficienNet-B4 [97]-0.58------- 2020 FFD [21]-0.60------- F3Net [76]-0.64--0.50---- 2022 CORE [72]-0.62------- RECCE [8]-0.66--0.50---- 2023 UCF [105]----0.49---- TALL ∗ [104]0.56 0.58 32.740.83 0.82 74.660.500.5755.06 2024 XRes [38]-0.68------- AVFF ∗ [75]0.88 0.91 97.930.95 0.95 70.780.690.7055.56 MRDF ∗ [120]0.51 0.395.560.53 0.53 52.120.650.6561.83 2025 StA [108]----0.62---- ForAda [20]-0.87------- Effort [107]-0.85------- 2026 GenD (CLIP) [111]-0.90------- GenD (PE) [111]-0.92------- GenD (DINO) [111]-0.92------- DF-Linear0.82 0.89 94.57 0.99 0.99 97.870.460.4550.91 DF-MoE (ours)0.93 0.94 98.49 0.99 0.99 96.77 0.86 0.88 81.59 Table 4: Cross-dataset evaluation on PolyGlotFake, BioDeepAV and Vox+FakeAVCeleb. The best performing method is highlighted in blue bold, and the second-best in orange. DF- MoE obtains the best performance on both datasets in terms of AUC and acc. Results of methods marked with an asterisk are reproduced using publicly available code. els (AV Transformer and AVFF), which already surpass some of the state-of-the-art models. Nevertheless, every high-level cue achieves non-trivial performance, when evaluated in iso- lation. This individual effectiveness indicates that the proposed features capture useful infor- mation that can be further harnessed in the full pipeline. Gradually integrating the individual components improves performance, indicating that the high-level cues exhibit a strong com- plementary effect, boosting the overall robustness and performance of DF-MoE. We report more ablations in the supplementary. Cross-dataset results. Next, we verify the robustness of DF-MoE to different manipula- tion methods and various real-world data sources. Specifically, we report results on Poly- GlotFake, BioDeepAV and Vox+FakeAVCeleb in Table 4, using MAVOS-D and AVLips as training data. The results follow the same trend observed in the open-set scenarios of MAVOS-D, namely that DF-MoE obtains state-of-the-art results on all three datasets in terms of mAP and AUC. We compare these results with several other methods. We com- pute the performance metrics of AVFF, TALL and MRDF using the code available in the corresponding public repositories. For the remaining methods, the AUC metric is taken from the official publications. While some of the evaluated methods [105, 108] are de- signed to improve generalization in deepfake detection, they still struggle to maintain opti- mal performance when encountering the significant data distribution shifts in PolyGlotFake, BioDeepAV and Vox+FakeAVCeleb. Qualitative analysis. In Figure 3, we show four videos from the MAVOS-D test set HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS13 Memo Sample/DF-MoE: Fake HifiFace Sample/DF-MoE: Fake Real Sample/DF-MoE: Real LivePortrait Sample/DF-MoE: Fake Figure 3: Examples of videos correctly classified by DF-MoE, alongside the corresponding attention scores of each high-level cue. Best viewed in color. that are correctly classified by DF-MoE. The fake videos are generated by three of the included manipulation methods (Memo, HifiFace and LivePortrait). These three methods cover all the visual manipulation types available in MAVOS-D, namely talking face syn- thesis (Memo), face swapping (HifiFace) and lip synchronization (LivePortrait). Along with the video frames, we illustrate the attention scores associated with each high-level cue. The scores are computed based on the attention weights provided by the multi-head attention layer. To obtain a single value for each feature, given the attention weights, we compute their average across the head and query dimensions. By analyzing the results shown in Figure 3, we can make several interesting observations. The AV Transformer and AVFF models have generally high contributions, essentially due to their strong individual performance coming from the direct analysis of manipulated audio- 14HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS #GPUs AverageFace detection &Feature MoETotal video durationtrackingextraction 110.973.358.471.1612.98 210.973.354.651.169.16 Table 5: Average video duration and stage-wise inference times (in seconds) for DF-MoE. The values are estimated across 20 videos on a machine with 1×AMD Ryzen Threadripper 9960X 24-core CPU and 2×Nvidia RTX 5090 GPU (32GB VRAM). visual content. The top-left example shows a slight skin tone variation between frames, which explains the high rPPG score. In the top-right example, head pose and gaze receive high importance, likely capturing gaze inconsistencies caused by face-swapping artifacts around the eyes. In the bottom-right LivePortrait sample, where lip movements are altered, the AV Transformer and AVFF effectively detect the manipulation by assessing audio-visual synchronization. Lastly, in the real sample (bottom-left), high scores from the AV Trans- former and head pose features indicate that the model correctly identifies natural audio-visual synchronization and authentic head movements. Computational complexity. Our complete pipeline, which integrates all pre-trained models, has 508.18 GFLOPs and 424.14M parameters. During training, we only update 19.9M (5%) of the 424.14M inference parameters. In Table 5, we report the average inference time in seconds for all stages of our pipeline. With all the components in place, DF-MoE reaches near real-time processing speed at inference, on both one or two GPUs. Training is conducted using two RTX 5090 GPUs. DF-MoE is trained for 10 epochs at roughly 5 hours and 20 minutes per epoch, totaling approximately 52 hours of compute time per experiment. 5 Conclusion In this work, we addressed the out-of-domain generalization issue of audio-video deepfake detectors by proposing DF-MoE. Our framework integrates audio-visual cues provided by several pre-trained models, including head pose, gaze, face segmentation maps, rPPG sig- nals, face and audio emotions, etc. Using pre-trained models prevents overfitting to a spe- cific deepfake generator or dataset. Our DF-MoE also includes trainable mixture-of-experts, improving deepfake detection with their ability to create specialized features (in our case, for different input signals), while preserving the generalization capacity of our framework. The ablation results demonstrated that each high-level cue provides useful information for deepfake detection. DF-MoE obtained state-of-the-art results on three challenging datasets, outperforming the previous methods in terms of relevant performance metrics. We also demonstrated that introducing frozen pre-trained models into an efficient architecture pro- vides state-of-the-art cross-domain performance. In future work, employing additional pre- trained models to extract complementary signals could further boost performance on deep- fake detection, and even on other complex tasks, such as video planning. 6 Acknowledgments This work was supported by a grant of the Ministry of Research, Innovation and Digitization, CCCDI - UEFISCDI, project number PN-IV-P6-6.3-SOL-2024-2-0227, within PNCDI IV. This work was in part supported by the BMFTR (FKZ: 16IS24060), and the DFG (SFB 1233, project number: 276693517). HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS15 References [1] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. MesoNet: a Compact Facial Video Forgery Detection Network. In Proceedings of WIFS, pages 1–7, 2018. [2] Agil Aghasanli, Dmitry Kangin, and Plamen Angelov.Interpretable-through- prototypes deepfake detection for diffusion models. In Proceedings of ICCVW, pages 467–474, 2023. [3] S. Asha, P. Vinod, and Varun G. Menon. A defensive attention mechanism to detect deepfake content across multiple modalities. Multimedia Systems, 30:351–356, 2024. [4] Marcella Astrid, Enjie Ghorbel, and Djamila Aouada. Audio-visual deepfake detec- tion with local temporal inconsistencies. In Proceedings of ICASSP, pages 1–5, 2025. [5] Zhongjie Ba, Qingyu Liu, Zhenguang Liu, Shuang Wu, Feng Lin, Li Lu, and Kui Ren. Exposing the deception: Uncovering more forgery clues for deepfake detection. In Proceedings of AAAI, pages 719–728, 2024. [6] Emily R. Bartusiak and Edward J. Delp. Synthesized speech detection using con- volutional transformer-based spectrogram analysis. In Proceedings of ACSSC, pages 1426–1430, 2021. [7] Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma.CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset. IEEE Transactions on Affective Computing, 5(4):377–390, 2014. [8] Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang. End-to-end reconstruction-classification learning for face forgery detection. In Pro- ceedings of CVPR, pages 4103–4112, 2022. [9] Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In Proceedings of CVPR, pages 18689–18698, 2022. [10] Liang Chen, Yong Zhang, Yibing Song, Jue Wang, and Lingqiao Liu. OST: Improving Generalization of DeepFake Detection via One-Shot Test-Time Training. In Proceed- ings of NeurIPS, pages 24597–24610, 2022. [11] Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, and Chenguang Ma. EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions. In Proceedings of AAAI, pages 2403–2410, 2024. [12] Jikang Cheng, Zhiyuan Yan, Ying Zhang, Li Hao, Jiaxin Ai, Qin Zou, Chen Li, and Zhongyuan Wang. Stacking brick by brick: Aligned feature isolation for incremental face forgery detection. In Proceedings of CVPR, pages 446–459, 2025. [13] Kyunghyun Cho, Bart Van Merriënboer, Ça ̆ glar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP, pages 1724–1734, 2014. 16HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS [14] Sungik Choi, Hankook Lee, Jaehoon Lee, Seunghyun Kim, Stanley Jungkyu Choi, and Moontae Lee. HFI: A unified framework for training-free detection and im- plicit watermarking of latent diffusion model generated images. arXiv preprint arXiv:2412.20704, 2024. [15] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep Speaker Recognition. In Proceedings of INTERSPEECH, pages 1086–1090, 2018. [16] Andrea Ciamarra, Roberto Caldelli, Federico Becattini, Lorenzo Seidenari, and Al- berto Del Bimbo. Deepfake Detection by Exploiting Surface Anomalies: The Surfake Approach. In Proceedings of WACV, pages 1024–1033, 2024. [17] Davide Cozzolino, Alessandro Pianese, Matthias Nießner, and Luisa Verdoliva. Audio-visual person-of-interest deepfake detection. In Proceedings of CVPR, pages 943–952, 2023. [18] Florinel-Alin Croitoru, Vlad Hondru, Marius Popescu, Radu Tudor Ionescu, Fa- had Shahbaz Khan, and Mubarak Shah. MAVOS-D: Multilingual Audio-Video Open-Set Deepfake Detection Benchmark. arXiv preprint arXiv:2505.11109, 2025. [19] Florinel-Alin Croitoru, Andrei-Iulian Hiji, Vlad Hondru, Nicolae Catalin Ristea, Paul Irofti, Marius Popescu, Cristian Rusu, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook. ACM Computing Surveys, 58(15):387, 2026. [20] Xinjie Cui, Yuezun Li, Ao Luo, Jiaran Zhou, and Junyu Dong. Forensics Adapter: Adapting CLIP for generalizable face forgery detection. In Proceedings of CVPR, pages 19207–19217, 2025. [21] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K. Jain. On the detec- tion of digital face manipulation. In Proceedings of CVPR, pages 5781–5790, 2020. [22] Shichao Dong, Jin Wang, Jiajun Liang, Haoqiang Fan, and Renhe Ji. Explaining deepfake detection by analysing image matching. In Proceedings of ECCV, pages 18–35, 2022. [23] Shichao Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Im- plicit identity leakage: The stumbling block to improving deepfake detection general- ization. In Proceedings of CVPR, pages 3994–4004, 2023. [24] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Ting Zhang, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. Protecting celebrities from deepfake with identity consistency transformer. In Proceedings of CVPR, pages 9458–9468, 2022. [25] Mengnan Du, Shiva Pentyala, Yuening Li, and Xia Hu. Towards generalizable deep- fake detection with locality-aware autoencoder. In Proceedings of CIKM, pages 325– 334, 2019. [26] Chao Feng, Ziyang Chen, and Andrew Owens. Self-supervised video forensics by audio-visual anomaly detection. In Proceedings of CVPR, pages 10491–10503, 2023. HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS17 [27] Mariana-Iuliana Georgescu and Radu Tudor Ionescu. Teacher-student training and triplet loss for facial expression recognition under occlusion. In Proceedings of ICPR, pages 2288–2295, 2021. [28] Mariana-Iuliana Georgescu, Georgian-Emilian Du ̧t ˇ a, and Radu Tudor Ionescu. Teacher-student training and triplet loss to reduce the effect of drastic face occlu- sion: Application to emotion recognition, gender identification and age estimation. Machine Vision and Applications, 33(1):12, 2022. [29] David Güera and Edward J. Delp. Deepfake Video Detection Using Recurrent Neural Networks. In Proceedings of AVSS, pages 1–6, 2018. [30] Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. arXiv preprint arXiv:2407.03168, 2024. [31] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don’t lie: A generalisable and robust approach to face forgery detection. In Proceedings of CVPR, pages 5039–5049, 2021. [32] Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Leveraging real talking faces via self-supervision for robust forgery detection. In Proceedings of CVPR, pages 14930–14942, 2022. [33] Yue-Hua Han, Tai-Ming Huang, Kai-Lung Hua, and Jun-Cheng Chen. Towards more general video-based deepfake detection through facial component guided adaptation for foundation model. In Proceedings of CVPR, pages 22995–23005, 2025. [34] Ben Harwood, Vijay B.G. Kumar, Gustavo Carneiro, Ian Reid, and Tom Drummond. Smart mining for deep metric learning. In Proceedings of ICCV, pages 2821–2829, 2017. [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770–778, 2016. [36] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compu- tation, 9(8):1735–1780, 1997. [37] Ashish Hooda, Neal Mangaokar, Ryan Feng, Kassem Fawaz, Somesh Jha, and Atul Prakash. D4: Detection of adversarial diffusion deepfakes using disjoint ensembles. In Proceedings of WACV, pages 3800–3810, 2024. [38] Yang Hou, Haitao Fu, Chunkai Chen, Zida Li, Haoyu Zhang, and Jianjun Zhao. Poly- GlotFake: A Novel Multilingual and Multimodal DeepFake Dataset. In Proceedings of ICPR, pages 180–193, 2024. [39] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of ICLR, 2022. [40] Juan Hu, Xin Liao, Jinwen Liang, Wenbo Zhou, and Zheng Qin. FInfer: Frame Inference-Based Deepfake Detection for High-Visual-Quality Videos. In Proceedings of AAAI, pages 951–959, 2022. 18HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS [41] Juan Hu, Xin Liao, Difei Gao, Satoshi Tsutsui, Qian Wang, Zheng Qin, and Mike Zheng Shou. Delocate: Detection and localization for deepfake videos with randomly-located tampered traces. In Proceedings of IJCAI, pages 5862–5871, 2024. [42] Guang Hua, Andrew Beng Jin Teoh, and Haijian Zhang. Towards end-to-end synthetic speech detection. IEEE Signal Processing Letters, 28:1265–1269, 2021. [43] Baojin Huang, Zhongyuan Wang, Jifan Yang, Jiaxin Ai, Qin Zou, Qian Wang, and Dengpan Ye. Implicit identity driven deepfake face swapping detection. In Proceed- ings of CVPR, pages 4490–4499, 2023. [44] Hafsa Ilyas, Ali Javed, and Khalid Mahmood Malik. AVFakeNet: A unified end-to- end Dense Swin Transformer deep learning model for audio-visual deepfakes detec- tion. Applied Soft Computing, 136:110124, 2023. [45] Yonghyun Jeong, Doyeon Kim, Seungjai Min, Seongho Joe, Youngjune Gwon, and Jongwon Choi. BiHPF: Bilateral High-Pass Filters for Robust Deepfake Detection. In Proceedings of WACV, pages 2878–2887, 2022. [46] Xiaozhong Ji, Xiaobin Hu, Zhihong Xu, Junwei Zhu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo, Yi Chen, Qin Lin, Qinglin Lu, and Chengjie Wang. Sonic: Shifting focus to global audio perception in portrait animation. In Proceedings of CVPR, pages 193–203, 2025. [47] Yan Ju, Shu Hu, Shan Jia, George H. Chen, and Siwei Lyu. Improving fairness in deepfake detection. In Proceedings of WACV, pages 4643–4653, 2024. [48] Jee-Weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong- Jin Lee, Ha-Jin Yu, and Nicholas Evans. AASIST: Audio Anti-Spoofing using In- tegrated Spectro-Temporal Graph Attention Networks. In Proceedings of ICASSP, pages 6367–6371, 2022. [49] Hasam Khalid, Shahroz Tariq, Minha Kim, and Simon S. Woo. FakeAVCeleb: A novel audio-video multimodal deepfake dataset. In Proceedings of NeurIPS, 2021. [50] Rahima Khanam and Muhammad Hussain. YOLOv11: An Overview of the Key Architectural Enhancements. arXiv preprint arXiv:2410.17725, 2024. [51] Marouane Kihal and Lamia Hamza. Robust multimedia spam filtering based on visual, textual, and audio deep features and random forest. Multimedia Tools and Applica- tions, 82(26):40819–40837, 2023. [52] Minha Kim, Shahroz Tariq, and Simon S. Woo. FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning. In Proceedings of CVPRW, pages 1001–1012, 2021. [53] Zhe Kong, Feng Gao, Yong Zhang, Zhuoliang Kang, Xiaoming Wei, Xunliang Cai, Guanying Chen, and Wenhan Luo. Let them talk: Audio-driven multi-person conver- sational video generation. In Proceedings of NeurIPS, pages 70990–71013, 2025. [54] Yingxin Lai, Hongyang Wang, Jing Yang, Xiangui Kang, Bin Li, Linlin Shen, and Zitong Yu. GM-DF: Generalized Multi-Scenario Deepfake Detection. In Proceedings of ACMMM, pages 4300–4309, 2025. HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS19 [55] Romeo Lanzino, Federico Fontana, Anxhelo Diko, Marco Raoul Marini, and Luigi Cinque. Faster than lies: Real-time deepfake detection using binary neural networks. In Proceedings of CVPR, pages 3771–3780, 2024. [56] Nicolas Larue, Ngoc-Son Vu, Vitomir Struc, Peter Peer, and Vassilis Christophides. SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deep- fakes. In Proceedings of ICCV, pages 20954–20964, 2023. [57] Binh M. Le and Simon S. Woo. ADD: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images. In Pro- ceedings of AAAI, pages 122–130, 2022. [58] Binh M Le and Simon S. Woo. Quality-agnostic deepfake detection with intra-model collaborative learning. In Proceedings of ICCV, pages 22321–22332, 2023. [59] Hanzhe Li, Jiaran Zhou, Yuezun Li, Baoyuan Wu, Bin Li, and Junyu Dong. Fre- qBlender: enhancing DeepFake detection by blending frequency knowledge. In Pro- ceedings of NeurIPS, pages 44965–44988, 2025. [60] Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping arti- facts. In Proceedings of CVPRW, pages 46–51, 2018. [61] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of CVPR, pages 3204– 3213, 2020. [62] Li Lin, Xinan He, Yan Ju, Xin Wang, Feng Ding, and Shu Hu. Preserving fairness generalization in deepfake detection. In Proceedings of CVPR, pages 16815–16825, 2024. [63] Baoping Liu, Bo Liu, Ming Ding, Tianqing Zhu, and Xin Yu. TI2Net: Temporal Identity Inconsistency Network for Deepfake Detection. In Proceedings of WACV, pages 4680–4689, 2023. [64] Weifeng Liu, Tianyi She, Jiawei Liu, Boheng Li, Dongyu Yao, Ziyou Liang, and Run Wang. Lips are lying: Spotting the temporal inconsistency between audio and visual in lip-syncing deepfakes. In Proceedings of NeurIPS, pages 91131–91155, 2024. [65] Xiaolong Liu, Yang Yu, Xiaolong Li, and Yao Zhao. Magnifying multimodal forgery clues for deepfake detection. Signal Processing: Image Communication, 118:117010, 2023. [66] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of ICCV, pages 9992–10002, 2021. [67] Long Ma, Zhiyuan Yan, Jin Xu, Yize Chen, Qinglang Guo, Zhen Bi, Yong Liao, and Hui Lin. From specificity to generality: Revisiting generalizable artifacts in detecting face deepfakes. In Proceedings of NeurIPS, pages 69306–69344, 2025. [68] Hui Miao, Yuanfang Guo, Zeming Liu, and Yunhong Wang. Multi-modal deepfake detection via multi-task audio-visual prompt learning. In Proceedings of AAAI, pages 612–621, 2025. 20HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS [69] Daniel Mas Montserrat, Hanxiang Hao, Sri K Yarlagadda, Sriram Baireddy, Ruiting Shao, János Horváth, Emily Bartusiak, Justin Yang, David Guera, Fengqing Zhu, et al. Deepfakes Detection with Automatic Face Weighting. In Proceedings of CVPR, pages 2851–2859, 2020. [70] Aakash Varma Nadimpalli and Ajita Rattani. On improving cross-dataset generaliza- tion of deepfake detectors. In Proceedings of CVPRW, pages 91–99, 2022. [71] Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjie Ghorbel, and Djamila Aouada. LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection. In Proceedings of CVPR, pages 17395–17405, 2024. [72] Yunsheng Ni, Depu Meng, Changqian Yu, Chengbin Quan, Dongchun Ren, and You- jian Zhao. Core: Consistent representation learning for face forgery detection. In Proceedings of CVPRW, pages 12–21, 2022. [73] Fan Nie, Jiangqun Ni, Jian Zhang, Bin Zhang, and Weizhe Zhang. FRADE: Forgery- aware Audio-distilled Multimodal Learning for Deepfake Detection. In Proceedings of ACMMM, pages 6297–6306, 2024. [74] Yuval Nirkin, Lior Wolf, Yosi Keller, and Tal Hassner. Deepfake detection based on discrepancies between faces and their context. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6111–6121, 2022. [75] Trevine Oorloff, Surya Koppisetti, Nicolò Bonettini, Divyaraj Solanki, Ben Colman, Yaser Yacoob, Ali Shahriyari, and Gaurav Bharaj. AVFF: Audio-Visual Feature Fu- sion for Video Deepfake Detection. In Proceedings of CVPR, pages 27102–27112, 2024. [76] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In Proceedings of ECCV, pages 86–103, 2020. [77] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Pro- ceedings of ICML, pages 28492–28518, 2023. [78] Muhammad Anas Raza and Khalid Mahmood Malik. Multimodaltrace: Deepfake detection using audiovisual representation learning. In Proceedings of CVPR, pages 993–1000, 2023. [79] Jonas Ricker, Denis Lukovnikov, and Asja Fischer. AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error. In Proceedings of CVPR, pages 9130–9140, 2024. [80] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics++: Learning to detect manipulated facial im- ages. In Proceedings of ICCV, pages 1–11, 2019. [81] Nataniel Ruiz, Eunji Chong, and James M. Rehg. Fine-grained head pose estimation without keypoints. In Proceedings of CVPR, pages 2074–2083, 2018. HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS21 [82] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. In Proceedings of CVPRW, pages 80–87, 2019. [83] Davide Salvi, Honggu Liu, Sara Mandelli, Paolo Bestagini, Wenbo Zhou, Weiming Zhang, and Stefano Tubaro. A Robust Approach to Multimodal Deepfake Detection. Journal of Imaging, 9(6), 2023. [84] Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified em- bedding for face recognition and clustering. In Proceedings of CVPR, pages 815–823, 2015. [85] Garima Sharma, Shreya Ghosh, and Abhinav Dhall. Automatic Group Level Affect and Cohesion Prediction in Videos. In Proceedings of ACIIW, pages 161–167, 2019. [86] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geof- frey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proceedings of ICLR, 2017. [87] Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended im- ages. In Proceedings of CVPR, pages 18699–18708, 2022. [88] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Rama- monjisoa, et al. DINOv3. arXiv preprint arXiv:2508.10104, 2025. [89] Stefan Smeu, Dragos-Alexandru Boldisor, Dan Oneata, and Elisabeta Oneata. Cir- cumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning. In Proceedings of CVPR, pages 18815–18825, 2025. [90] Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic Class- Based Hard Example Mining for Deep Metric Learning. In Proceedings of CVPR, pages 7244–7252, 2019. [91] Ke Sun, Shen Chen, Taiping Yao, Xiaoshuai Sun, Shouhong Ding, and Rongrong Ji. Continual face forgery detection via historical distribution preserving. International Journal of Computer Vision, pages 1067–1084, 2025. [92] Zhimin Sun, Shen Chen, Taiping Yao, Bangjie Yin, Ran Yi, Shouhong Ding, and Lizhuang Ma. Contrastive pseudo learning for open-world deepfake attribution. In Proceedings of ICCV, pages 20825–20835, 2023. [93] Hemlata Tak, Jee-weon Jung, Jose Patino, Massimiliano Todisco, and Nicholas Evans. Graph attention networks for anti-spoofing. In Proceedings of INTERSPEECH, pages 2356–2360, 2021. [94] Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-end anti-spoofing with RawNet2. In Proceedings of ICASSP, pages 6369–6373, 2021. [95] Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Frequency-aware deepfake detection: Improving generalizability through fre- quency space domain learning. In Proceedings of AAAI, pages 5052–5060, 2024. 22HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS [96] Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Rethinking the Up-Sampling Operations in CNN-Based Generative Network for Generalizable Deepfake Detection. In Proceedings of CVPR, pages 28130–28139, 2024. [97] Mingxing Tan and Quoc Le. EfficientNet: Rethinking Model Scaling for Convolu- tional Neural Networks. In Proceedings of ICML, pages 6105–6114, 2019. [98] Loc Trinh, Michael Tsang, Sirisha Rambhatla, and Yan Liu. Interpretable and trust- worthy deepfake detection via dynamic prototypes. In Proceedings of WACV, pages 1972–1982, 2021. [99] Chenglong Wang, Jiangyan Yi, Jianhua Tao, Chu Yuan Zhang, Shuai Zhang, and Xun Chen. Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features. In Proceedings of INTERSPEECH, pages 3844–3848, 2023. [100] Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Yu-Gang Jiang, and Ser-Nam Li. M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. In Proceedings of ICMR, page 615–623, 2022. [101] Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping. In Proceedings of IJCAI, pages 1136– 1142, 2021. [102] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In Proceedings of ICIP, pages 3645–3649, 2017. [103] Ying Xu, Kiran Raja, Luisa Verdoliva, and Marius Pedersen. Learning pairwise inter- action for generalizable deepfake detection. In Proceedings of WACVW, pages 1–11, 2023. [104] Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, and Ran He. TALL: Thumbnail Layout for Deepfake Video Detection. In Proceedings of ICCV, pages 22601–22611, 2023. [105] Zhiyuan Yan, Yong Zhang, Yanbo Fan, and Baoyuan Wu. UCF: Uncovering Com- mon Features for Generalizable Deepfake Detection. In Proceedings of ICCV, pages 22355–22366, 2023. [106] Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcend- ing Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection. In Proceedings of CVPR, pages 8984–8994, 2024. [107] Zhiyuan Yan, Jiangming Wang, Peng Jin, Ke-Yue Zhang, Chengchun Liu, Shen Chen, Taiping Yao, Shouhong Ding, Baoyuan Wu, and Li Yuan. Orthogonal Subspace De- composition for Generalizable AI-Generated Image Detection. In Proceedings of ICML, pages 70268–70288, 2025. [108] Zhiyuan Yan, Yandan Zhao, Shen Chen, Mingyi Guo, Xinghe Fu, Taiping Yao, Shouhong Ding, Yunsheng Wu, and Li Yuan. Generalizing deepfake video detec- tion with plug-and-play: Video-level blending and spatiotemporal adapter tuning. In Proceedings of CVPR, pages 12615–12625, 2025. HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS23 [109] Tianyun Yang, Ziyao Huang, Juan Cao, Lei Li, and Xirong Li. Deepfake network architecture attribution. In Proceedings of AAAI, pages 4662–4670, 2022. [110] Kelu Yao, Jin Wang, Boyu Diao, and Chao Li. Towards understanding the general- ization of deepfake detectors from a game-theoretical view. In Proceedings of ICCV, pages 2031–2041, 2023. [111] Andrii Yermakov, Jan Cech, Jiri Matas, and Mario Fritz. Deepfake detection that generalizes across benchmarks. In Proceedings of WACV, pages 773–783, 2026. [112] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation. In Proceedings of ECCV, pages 325–341, 2018. [113] Zijie Yue, Miaojing Shi, Hanli Wang, Shuai Ding, Qijun Chen, and Shanlin Yang. Bootstrapping vision-language models for frequency-centric self-supervised remote physiological measurement. International Journal of Computer Vision, 133(7):4112– 4133, 2025. [114] Yibo Zhang, Weiguo Lin, and Junfeng Xu. Joint audio-visual attention with con- trastive learning for more general deepfake detection. ACM Transactions on Multime- dia Computing, Communications and Applications, 20:1–23, 2024. [115] Zirui Zhang, Wei Hao, Aroon Sankoh, William Lin, Emanuel Mendiola-Ortiz, Jun- feng Yang, and Chengzhi Mao. I can hear you: Selective robust training for deepfake audio detection. In Proceedings of ICLR, 2025. [116] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deepfake detection. In Proceedings of CVPR, pages 2185–2194, 2021. [117] Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia. Learning self-consistency for deepfake detection. In Proceedings of ICCV, pages 15003–15013, 2021. [118] Longtao Zheng, Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, and Shuicheng Yan. MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation. arXiv preprint arXiv:2412.04448, 2024. [119] Yipin Zhou and Ser-Nam Lim. Joint Audio-Visual Deepfake Detection. In Proceed- ings of ICCV, pages 14800–14809, 2021. [120] Heqing Zou, Meng Shen, Yuchen Hu, Chen Chen, Eng Siong Chng, and Deepu Ra- jan. Cross-modality and within-modality regularization for audio-visual deepfake de- tection. In Proceedings of ICASSP, pages 4900–4904, 2024. [121] Dragos , -Constantin T , ânt , aru, Elisabeta Oneat , ̆ a, and Dan Oneat , ̆ a. Weakly-supervised deepfake localization in diffusion-generated images. In Proceedings of WACV, pages 6246–6256, 2024. 24HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS Dropout Ratio #Experts mAP AUCacc 0.260.980.9892.95 0.460.970.9791.70 0.2120.960.9792.54 0.4120.960.9691.24 Table A1: Ablation study on the expert dropout ratio (d) and the number of experts (n) included in DF-MoE. The study is conducted on the open-set full protocol of MAVOS-D [18]. CRO LossmAPAUCacc ✗0.919 0.936 98.42 ✓0.929 0.941 98.49 Table A2: Results of DF-MoE on PolyGlotFake [38], before and after introducing the pro- posed contractive-repulsive objective (CRO). 7 Supplementary 7.1 Additional Ablation Studies Ablation of hyperparameters. In Table A1, we present an ablation study evaluating two key hyperparameters of DF-MoE, the expert dropout ratio (d) and the number of experts (n). To prevent overfitting on the MAVOS-D dataset, we limit our exploration of these parameters to a small set of values. The results confirm that the values used in our main experiments (d = 0.2 and n = 6) yield optimal performance on the open-set full scenario of MAVOS-D. Moreover, all explored versions significantly surpass the state-of-the-art competitors [75, 89, 104, 120] (see Table 1 from the main paper), indicating that DF-MoE attains consistently high performance, even with suboptimal hyperparameter configurations. Effect of contractive-repulsive objective. To showcase the effect of the CRO loss on the latent space, we present t-SNE visualizations of real and deepfake embeddings from the latent space of DF-MoE, before and after introducing the proposed CRO loss. In Figure 1, we compare the latent space obtained after training on MAVOS-D. Upon introducing our contractive-repulsive objective, we observe that deepfakes generated by different methods are entangled in a more compact region of the latent space. Interestingly, out-of-domain (open-set) deepfake generation methods, such as Sonic [46] and HifiFace [101], share the same behavior as in-domain (closed-set) deepfake generators. In contrast, DF-MoE without our CRO loss spreads samples from different detection methods in a wider area, and different generative methods are located in distinctive regions, harming generalization capacity. Consequently, several deepfake samples produced by Hifi- Face [101] are entangled with the real samples. In general, real and deepfake entanglements in the latent space inherently lead to performance degradation in the cross-dataset scenario (see Table A2). In summary, the t-SNE visualizations depicted in Figure 1 indicate that the latent space organization induced by our CRO loss contributes to the generalization of DF-MoE. Semantic cues contributions. The advantages of incorporating multiple modalities be- yond AVFF are most evident in the cross-dataset results from the main paper (Table 4, last column). Additionally, in Table A3, we break down the contribution of each modal- ity on PolyGlotFake [38]. Most features improve upon the AVFF+Effort baseline. Although HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS25 302010010203040 t-SNE Dimension 1 20 10 0 10 20 t-SNE Dimension 2 Real HifiFace LivePortrait Memo Sonic (a) Latent space of DF-MoE before introducing the CRO loss. 40302010010203040 t-SNE Dimension 1 30 20 10 0 10 20 t-SNE Dimension 2 Real HifiFace LivePortrait Memo Sonic Anchors (b) Latent space of DF-MoE after introducing the CRO loss. Figure 1: Comparison between latent spaces of DF-MoE, before and after introducing CRO loss. In both cases, the model is trained on MAVOS-D. Real samples and deepfakes gen- erated by various methods are illustrated through different colors. Sonic and HifiFace are generative methods that do not belong to the training set. Best viewed in color. combining audio-video emotion features with AVFF features results in a slight performance drop (fourth row), further adding HP+Gaze (last row) outperforms the combination of AVFF and HP+Gaze features (third row). This result suggests that gaze provides complementary contextual information for facial expressions through self-attention, showing the benefits of jointly modeling complementary modalities for a more robust representation for deepfake detection. Fine-tuning feature extractors. Fully fine-tuning the foundational encoders is computa- tionally impractical due to the massive memory footprint and processing costs required. Be- 26HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS AVFF+Face HP+Gaze AVAV rPPG mAP AUCacc Effortsegmentationemotion transformer ✓✗0.880.9197.93 ✓✗0.900.9096.70 ✓✗✓✗0.910.9296.65 ✓✗✓✗0.870.8891.71 ✓✗✓✗0.920.9398.70 ✓✗✓0.900.9298.56 ✓✗✓✗0.930.9398.70 Table A3: Ablation study on combinations of AVFF+Effort features and other high-level features included in DF-MoE. Results are reported on PolyGlotFake [38]. MethodmAP AUCacc DF-MoE (AVFF full)0.940.9486.38 DF-MoE (AVFF+Effort)0.980.9892.95 Table A4: Full vs. parameter-efficient training of AVFF encoder on MAVOS-D (open-set full). Fine-tuning based on Effort [107] surpasses full fine-tuning. MethodAUC F3Net [76]0.789 CORE [72]0.809 RECCE [8]0.823 UCF [105]0.837 LSDA [106]0.875 DF-MoE (ours) 0.834 Table A5: Cross-dataset evaluation on Celeb-DF (v2) [61], a video-only dataset. The results demonstrate that DF-MoE obtains competitive results even when the audio modality is miss- ing. yond these computational constraints, full fine-tuning can compromise generalization, as highly parameterized models are particularly prone to overfitting to dataset-specific forgery artifacts, and thus limiting their ability to detect unseen manipulation techniques. This phe- nomenon is corroborated by Table A4, which demonstrates that applying full fine-tuning on the AVFF encoder within the DF-MoE architecture actually degrades performance. In con- trast, adapting the encoder using the Effort [107] strategy yields superior results, proving that parameter-efficient fine-tuning approach balances feature adaptation with robust gener- alization. Robustness to missing modalities. We deliberately omit positional embeddings from our token representations, enabling DF-MoE to work seamlessly when one or more modalities are missing. To demonstrate this, we evaluate DF-MoE on the Celeb-DF (v2) [61] dataset in the cross-dataset scenario in Table A5, where audio is not available. In this setting, DF- MoE safely ignores audio-specific features, while still obtaining competitive performance. This implies that DF-MoE is robust to changes in the concatenation order of features, and consequently, to variations in the set of available modalities. 7.2 Qualitative Result Analysis In Figure 2, we show the average attention scores for each dataset, computed over 100 real and 100 fake videos randomly sampled from the respective test subsets. The results indicate that feature importance remains highly consistent across different datasets. This suggests that HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS27 M A V O S - D D B i o D e e p A V P o l y G l o t F a k e RealFake Figure 2: Average attention scores across datasets and class labels. our model relies on universal, domain-agnostic features rather than overfitting to dataset- specific artifacts, which directly explains its strong generalization capabilities on unseen data. 7.3 Faliure Cases In Figure 3, we present failure cases of DF-MoE to point out some of its gaps. For the real video on the left-hand side, the AV Transformer and AVFF features are mainly responsible for the incorrect classification. The right-hand side example shows a fake video misclassified as real, where the model disproportionately attributes too much importance to emotion fea- tures. Our inspection of these predictions reveals persistent, inaccurate values of happiness and fear, that are misaligned with the actual emotions present in the audio-video streams. To- gether, these edge cases highlight two potential directions of improvement for our approach. First, we need to explicitly maintain a contribution balance when a small number of features begin to overshadow the rest of the cues. Second, we need to manage the sensitivity of the final classification to the precision of the upstream frozen feature extractors. 28HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS Real Sample/DF-MoE: Fake HifiFace Sample/DF-MoE: Real Figure 3: Examples of videos incorrectly classified by DF-MoE, alongside the corresponding attention scores of each feature type. Best viewed in color. 7.4 Implementation Details In this section, we provide details about the specialized encoders employed in our pipeline. In Figure 4, we illustrate the architectures that are briefly presented in the main paper for the rPPG, face segmentation, head movement and emotion encoders. HONDRU ET AL.: DF-MOE: DEEPFAKE DETECTION VIA MIXTURE-OF-EXPERTS29 input-tensor(1, 100, 512) Conv1D k=5 s=1 p=2 input:(1, 512, 100) output: (1, 64, 100) AvgPool1D k=2 s=2 input:(1, 64, 100) output: (1, 64, 50) Conv1D k=3 s=1 p=1 input:(1, 64, 50) output: (1, 128, 50) AvgPool1D k=2 s=2 input:(1, 128, 50) output: (1, 128, 25) GRU num_layers=2 hidden_dim=128 input:(1, 25, 128) output: (1, 25, 256), (2, 1, 128) Linear 256 → 128 input:(1, 256) output: (1, 128) output-tensor(1, 128) (a) rPPG. input-tensor(1, 100) int input:(1, 100) output: (1, 100) input-tensor(1, 100) int input:(1, 100) output: (1, 100) Embedding input:(1, 100) output: (1, 100, 64) GRU num_layers=1 hidden_dim=64 input:(1, 100, 64) output: (1, 100, 128) (2, 1, 64) Embedding input:(1, 100) output: (1, 100, 64) GRU num_layers=1 hidden_dim=64 input:(1, 100, 64) output: (1, 100, 128) (2, 1, 64) MultiheadAttention num_heads=4 embed_dim=128 input:3 x (1, 100, 128) output: (1, 100, 128), (1, 100, 100) mean input:(1, 100, 128) output: (1, 128) mean input:(1, 100, 128) output: (1, 128) cat input:2 x (1, 128) output: (1, 256) Linear 256 → 128 input:(1, 256) output: (1, 128) output-tensor(1, 128) (b) AV emotion. input-tensor(1, 100, 3) GRU num_layers=2 hidden_dim=64 input:(1, 100, 3) output: (1, 100, 128) 2 x (4, 1, 64) input-tensor(1, 100, 2) GRU num_layers=2 hidden_dim=64 input:(1, 100, 2) output: (1, 100, 128) 2 x (4, 1, 64) MultiheadAttention num_heads=4 embed_dim=128 input:3 x (1, 100, 128) output: (1, 100, 128), (1, 100, 100) mean input:(1, 100, 128) output: (1, 128) mean input:(1, 100, 128) output: (1, 128) cat input:2 x (1, 128) output: (1, 256) Linear 256 → 128 input:(1, 256) output: (1, 128) output-tensor(1, 128) (c) HP+Gaze. input-tensor(1, 1, 100, 224, 224) Conv3D k=3x3x3 s=1x1x1 p=1x1x1 input:(1, 1, 100, 224, 224) output: (1, 32, 100, 224, 224) Conv3D k=3x3x3 s=1x1x1 p=1x1x1 input:(1, 32, 100, 112, 112) output: (1, 64, 100, 112, 112) Conv3D k=3x3x3 s=1x1x1 p=1x1x1 input:(1, 64, 50, 56, 56) output: (1, 128, 50, 56, 56) AdaptiveAvgPool3d input:(1, 128, 25, 28, 28) output: (1, 128, 1, 1, 1) Flatten input:(1, 128, 1, 1, 1) output: (1, 128) Linear 128 → 128 input:(1, 128) output: (1, 128) output-tensor(1, 128) (d) Face segments. Figure 4: Architectures of the specialized encoders, namely rPPG, audio-visual emotion, head and gaze movement, and face segmentation encoders.