Paper deep dive
RandMark: On Random Watermarking of Visual Foundation Models
Anna Chistyakova, Mikhail Pautov
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 6:17:06 AM
Summary
RandMark is a novel watermarking methodology for visual foundation models (VFMs) that embeds binary signatures directly into hidden representations using a small encoder-decoder network. It is robust against functional perturbations like fine-tuning and pruning, providing a reliable mechanism for intellectual property protection and ownership verification.
Entities (5)
Relation Signals (3)
CLIP → isa → Visual Foundation Models
confidence 100% · Through experiments on state-of-the-art visual foundation models (CLIP and DINOv2)
RandMark → protects → Visual Foundation Models
confidence 95% · We propose RandMark, a novel methodology for watermarking visual foundation models.
RandMark → uses → Encoder-Decoder Network
confidence 90% · leverages a small encoder-decoder network to embed digital watermarks
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Being trained on large and diverse datasets, visual foundation models (VFMs) can be fine-tuned to achieve remarkable performance and efficiency in various downstream computer vision tasks. The high computational cost of data collection and training makes these models valuable assets, which motivates some VFM owners to distribute them alongside a license to protect their intellectual property rights. In this paper, we propose an approach to ownership verification of visual foundation models that leverages a small encoder-decoder network to embed digital watermarks into an internal representation of a hold-out set of input images. The method is based on random watermark embedding, which makes the watermark statistics detectable in functional copies of the watermarked model. Both theoretically and experimentally, we demonstrate that the proposed method yields a low probability of false detection for non-watermarked models and a low probability of false misdetection for watermarked models.
Tags
Links
- Source: https://arxiv.org/abs/2603.10695v1
- Canonical: https://arxiv.org/abs/2603.10695v1
Trouble viewing inline? Open PDF directly →
Full Text
36,870 characters extracted from source content.
Expand or collapse full text
RandMark: On Random Watermarking of Visual Foundation Models Anna Chistyakova 1 and Mikhail Pautov 1,2 1 Trusted AI Research Center, RAS 2 AXXX Abstract. Being trained on large and diverse datasets, visual founda- tion models (VFMs) can be fine-tuned to achieve remarkable perfor- mance and efficiency in various downstream computer vision tasks. The high computational cost of data collection and training makes these mod- els valuable assets, which motivates some VFM owners to distribute them alongside a license to protect their intellectual property rights. In this paper, we propose an approach to ownership verification of visual foun- dation models that leverages a small encoder-decoder network to embed digital watermarks into an internal representation of a hold-out set of input images. The method is based on random watermark embedding, which makes the watermark statistics detectable in functional copies of the watermarked model. Both theoretically and experimentally, we demonstrate that the proposed method yields a low probability of false detection for non-watermarked models and a low probability of false mis- detection for watermarked models. Keywords: Watermarking· Visual Foundation Models· Fingerprinting 1 Introduction Today, foundation models are deployed in different fields, for example, in natural language processing [3,19], computer vision [20], and biology [14]. Their impres- sive performance in a wide range of downstream tasks comes at a price of high cost of data collection, training, and maintenance. Consequently, the models be- come valuable assets of their owners: the user’s access to foundation models is mainly organized via subscription to a service where the model is deployed or via purchasing the license to use a specific instance of the model. Unfortunately, some users may violate the terms of use (for example, by integrating their in- stances of the models into other services to make a profit). Hence, it is reasonable that the models’ owners are willing to defend their intellectual property from unauthorized usage by third parties. One of the prominent approaches to protecting the intellectual property rights (IPRs) of models is watermarking [9,13,24], the set of methods that embed specific information into a model by modifying its parameters. In watermarking, ownership verification is performed by checking for the presence of this informa- tion in a model. An alternative set of methods for IPR protection is based on arXiv:2603.10695v1 [cs.CV] 11 Mar 2026 2Anna Chistyakova and Mikhail Pautov Fig. 1: Overview of the proposed RandMark watermarking pipeline. A binary message is embedded into a visual foundation model using a set of trigger images and an encoder. During verification, randomized input transformations are applied to the trigger set, and a decoder extracts the watermark message from the model outputs. The extracted messages are then compared with the original watermark to verify model ownership. fingerprinting, which typically does not alter the original model [11,16,17]. In- stead, these methods generate a unique identifier, or fingerprint, for the model; ownership verification is then conducted by comparing the fingerprint of the original model with that of the suspicious model. This work introduces a method for watermarking visual foundation models (VFMs) by embedding digital watermarks into the hidden representations of a specific set of input images. Within the framework, we experimentally verify that embedding a watermark into the representation allows us to protect the owner- ship of VFMs fine-tuned for different practical tasks, such as image classification and segmentation. We demonstrate that our approach is able to distinguish be- tween an independent model and functional copies of the watermarked model with high probability. Our contributions are summarized as follows: – We propose RandMark, a novel methodology for watermarking visual foun- dation models. Unlike prior art focused on classifiers, our approach embeds binary signatures directly into the model’s hidden representations via a set of trigger images, making it suitable for the diverse downstream use-cases of VFMs. – We theoretically derive an upper bound on the probabilities of false positive detection of a non-watermarked model and misdetection of a functional copy of the watermarked model. – Through experiments on state-of-the-art visual foundation models (CLIP and DINOv2), we demonstrate that RandMark is highly robust. It success- fully detects model ownership after various functional perturbations, includ- ing fine-tuning on downstream tasks (classification and segmentation) and unstructured pruning, where existing fingerprinting methods fail. RandMark: On Random Watermarking of Visual Foundation Models3 2 Related work 2.1 Visual Foundation Models Visual foundation models, particularly those using vision transformers (ViT, [8]), are widely used in modern computer vision due to their scalability and trans- ferability across tasks. The advancement of self-supervised learning methods [1] has facilitated the creation of general-purpose models, including SimCLR [6], DINO [5], CLIP [18], and DINOv2 [15]. These models learn representations from unlabeled images and demonstrate broad applicability across diverse tasks, often requiring minimal labeled data for fine-tuning. 2.2 Protecting Intellectual Property of Neural Networks The protection of intellectual property for visual foundation models (VFMs) has gained increasing attention within the field of trustworthy AI. Watermarking and fingerprinting techniques aim to verify model ownership and prevent unau- thorized usage or model extraction. While early works focused on large language models [22,25], recent efforts adapt these ideas to visual models, including image classifiers and foundation models [16,23]. For visual foundation models (VFMs), there are currently no watermarking approaches specifically designed for these architectures. Several existing model ownership verification methods, such as ADV-TRA [26], and IPGuard [4], have been proposed in the context of image classification. These approaches embed ownership signatures by either modifying the training objective or introducing crafted input patterns and then detect them based on the model’s responses. While these methods are effective for classification models, they are not directly tailored to the broader capabilities of VFMs, such as image feature extraction or downstream adaptation. Adapting watermarking techniques to visual founda- tion models thus remains an open challenge and motivates the work presented in this paper. Other complementary methods exploit weight-space smoothing or perturbations to embed ownership information directly into model parameters. For example, Bansal et al. [2, 21] propose model watermarking through weight smoothing in deep neural networks, making the watermark robust to fine-tuning or minor architectural changes. These approaches provide alternative mecha- nisms to mark models without relying on specific input-output triggers and are particularly relevant for large visual foundation models where modifying the backbone is costly. Overall, while watermarking for VFMs is still in early stages, these methods illustrate that both data-driven triggers and weight-space techniques can serve as practical IP protection strategies for high-capacity visual models. 3 Methodology 3.1 Problem Statement In this work, we focus on the problem of watermarking of visual foundation models. To describe the proposed method, we start by introducing the notations. 4Anna Chistyakova and Mikhail Pautov Let s be the dimension of the input image and f : R s → R k be the source VFM that maps input images to the embeddings of dimension k. Here and below, we will write f ′ ∼ f to indicate that the model f ′ is a functional copy of f that is obtained, for example, via fine-tuning, knowledge distillation or pruning of the original model. Analogously, by writing g ⊥ f we will indicate that two models, g and f, are independent of each other. In our method, we train two auxiliary models, the encoder e : R s ×0, 1 n → R k that embeds the binary message m of length n into the representation of the input object x∈ R s , and the decoder d : R k → 0, 1 n that extracts a binary message from the output embedding of the VFM. Given the input image x, the source model f and the message m embedded into f(x), the goal of the method is two-fold: on the one hand, the decoder d should extract close messages from the representations f(x) and f ′ (x) for the model f ′ ∼ f; on the other hand, given the model g ⊥ f, the messages extracted from the representations f(x) and g(x) have to be far apart. The formal problem statement goes as follows. Given x as the secret input image used for watermarking, a predefined threshold τ ≪ n and probability thresholds 0 < γ 1 ≪ γ 2 < 1, the following inequalities should hold: P (∥w− d(f ′ (x))∥ 1 ≤ τ)≥ γ 1 , P (∥w− d(g(x))∥ 1 ≤ τ)≤ γ 2 , (1) where w = e(x,m) is the embedding with the watermark, f ′ ∼ f, g ⊥ f. In Eq. 1, the probabilities are taken over the randomness induced by the encoder; this randomness will be discussed in the subsequent sections. 3.2 Threat Model In this section, we discuss the conditions under which the proposed method is expected to operate correctly and outline the potential adversary’s capabilities. The goal of an adversary is to remove an existing watermark from a model so that ownership cannot be verified. Specifically, an adversary may attempt either a watermark removal attack, aiming to eliminate the watermark while preserving the model’s functionality, or a model extraction attack, trying to obtain a copy of the watermarked model without the watermark. Possible attacks include fine-tuning the model on downstream tasks or pruning. The objective of the watermarking method is to reliably determine whether a suspect model is a functional copy of the watermarked visual foundation model. 3.3 Proposed Method We introduce RandMark, a novel watermarking approach designed for visual foundation models. RandMark embeds user-specific binary signatures into the representations of a randomly transformed set of input images. To do so, we fine-tune the source model together with the lightweight encoder and decoder RandMark: On Random Watermarking of Visual Foundation Models5 networks. This approach enables ownership verification by extracting digital fin- gerprints from the set of randomly transformed specific set of input images and computing the statistic of resulting random variables. The watermarking process goes as follows. First of all, given input image x and user-specific binary message m, we inject m into the representation of x + ε j , ε j ∼ N(0,σ 2 (x)I) by training the small encoder e and fine-tuning the source model f. Modified representation, f(x + ε j ), is then passed to the de- coder network d that extracts binary message m ′ j from it. We highlight that the extracted messages, m ′ j , are random variables due to the randomness in transfor- mation of input image. The encoder, decoder, and the source foundation model are trained jointly to minimize both the average discrepancy between m and m ′ j and the variance of m ′ j . Loss function The training objective is the combination of two terms: given the input sample x, the first one ensures that the feature representations of the watermarked and original models do not deviate much; the second term forces the extracted binary messages to be close to the embedded one. Specifically, the objective function is L(x,f, ̃ f) =∥f(x)− ̃ f(x)∥ 2 + λ K K X j=1 ∥m− m ′ j ∥ 2 , (2) where λ > 0 is a scalar parameter, ̃ f is the watermarked version of f and m ′ j = d( ̃ f(e(x + ε j ,m))) is the binary message extracted by the decoder from x + ε j and K is the total number of transformations of the input image. This formulation ensures the successful embedding and extraction of watermarks with little to no impact on the feature representation. Evaluating the efficiency of the method To evaluate the performance of the proposed method, given the user-specific watermark m and input image x, we compute both the sample average and the sample variance of the variable ∥m− m ′ ∥ 1 , where m ′ is the watermark. Here we recall that the extracted watermarks are random. Namely, if the total number of transformations of the input image is K and the length of the watermark is n, we measure the average number of matching bits between m and m ′ j in the form ˆ E∥m− m ′ ∥ 1 = 1 K K X j=1 n X i=1 1 m i ̸= (m ′ j ) i ,(3) and the sample variance is computed as ˆ V∥m− m ′ ∥ 1 = 1 K− 1 K X j=1 d j − ˆ E∥m− m ′ ∥ 1 2 ,(4) 6Anna Chistyakova and Mikhail Pautov where d j = P n i=1 1 m i ̸= (m ′ j ) i , and m ′ j = d( ̃ f(e(x + ε j ,m))). The intuition behind using this two metrics is as follows. First of all, given the extracted mes- sage m ′ , the distance from Eq. (3) is expected to be small for the watermarked model and large for an independent model. Secondly, if we introduce an auxiliary variable in the form v(f,h) = V (∥m ′ (f)− m ′ (h)∥ 1 ),(5) then v(f,f ′ ) is expected to be small for f ′ ∼ f and v(f,g) is expected to be large for g ⊥ f. We elaborate on this point in the subsequent sections. In this work, the decision rule that is used to evaluate whether the given network is watermarked is the comparison of the distance with a predefined threshold: given the suspicious model h, input image x, secret message m and the series of K watermarks m ′ 1 ,m ′ 2 ,...,m ′ K extracted from h, we treat h as watermarked if ρ(x) = ˆ E∥m− m ′ ∥ 1 = 1 K K X j=1 n X i=1 1 m i ̸= (m ′ j ) i ≤ τ,(6) where τ ≥ 0 is the threshold value. In the case of many input images used for watermarking, namely, for N images from X = x 1 ,...,x N , the performance of the method is illustrated by the watermark detection rate, R(h,X,τ), in the form below: R(h,X,τ) = 1 N X x i ∈X 1[ρ(x i )≤ τ].(7) As an auxiliary indicator of the model being watermarked, for each x ∈ X, we compute the value of statistic v(f,h). Setting the threshold value We set the threshold by formulating a hypothesis test: the null hypothesis, H 0 = “the model h is not watermarked”, is tested against an alternative hypothesis, H 1 = “the model h is watermarked”, for the given suspicious model h. In this section, we assume that the probabilities that the i ′ th bit in m ′ (f) and m ′ (h) coincide are the same for all i ∈ [1,n]. Having said so, we estimate the probability of false acceptance of hypothesis H 1 (namely, FPR 1 ) as follows: FPR 1 = P g∼Ξ [ρ(m,m ′ (g,x)) < τ] = τ X j=0 n j (1− r) j r n−j ,(8) where r = P g∼Ξ (m i = m ′ (g,x) i ). To choose a proper threshold value for τ, we set up an upper bound for FPR 1 as ε and solve for τ, namely, τ = arg max τ ′ <n τ ′ X j=0 n j (1− r) j r n−j s.t. τ ′ X j=0 n j (1− r) j r n−j < ε.(9) RandMark: On Random Watermarking of Visual Foundation Models7 3.4 Difference between watermarked and non-watermarked models Recall that a good watermarking approach should yield a high watermark detec- tion rate from (7) for the models that are functionally connected to the water- marked one and, at the same time, low detection rates for independent models. To assess the integrity of the proposed approach, we estimate the probabilities of the method to yield low detection rates for functionally dependent models and high detection rates for independent models in the form P f ′ ∼Ω f [R(f ′ ,X,τ) <R], P g∼Ξ [R(g,X,τ) > R](10) for some threshold values 0 < R <R < N. To estimate the probabilities from (10), we firstly provide one-sided interval estimations for conditional probabilities of bit collisions in the form r(Ω f |x) = P f ′ ∼Ω f [m i = m ′ (f ′ ,x) i ], r(Ξ|x) = P g∼Ξ [m i = m ′ (g,x) i ].(11) We do it by sampling M functionally dependent models, namely, f ′ 1 ,...,f ′ M ∼ Ω f , and M independent models, namely g 1 ,...,g M ∼ Ξ. Here, the space Ξ of independent models consists of visual foundation models, both of the same architecture and of different architectures as f, by either fine-tuning of non- watermarked copy of f for a downstream task, of via functionality stealing per- turbations, for example, via knowledge distillation [12] or pruning [10]. Similarly, the space Ω f consists of the models, both of the same architecture and of differ- ent architectures as f, by either fine-tuning of f for a downstream task, of via functionality stealing perturbations. Then, given the set X = x 1 ,...,x N of images used for the watermarking of f from (7), we compute the quantities 1(f ′ j ,i,x l ) = 1[m i = m ′ (f ′ j ,x l ) i ] and 1(g j ,i,x l ) = 1[m i = m ′ (g j ,x l ) i ](12) ( P(r(Ω f |x) < l(x))≤ α N , P(r(Ξ|x) > u(x))≤ α N . (13) These estimates, namely, l(x) and u(x), are used to estimate the probabilities from (10). Estimating the probability of a deviation of the detection rate In this section, we discuss how to upper-bound both the probability of false detection of a non-watermarked model as a copy of the watermarked one and the probability of misdetecting a functional copy of the watermarked model. Note that R(f ′ ,X,τ) is a sum of N independent Bernoulli variables with parameters, r(Ω f |x), so P f ′ ∼Ω f [R(f ′ ,X,τ) <R] = R−1 X l=0 X S⊂X:|S|=l Y x in ∈S r(Ω f |x in ) Y x out /∈S (1− r(Ω f |x out )). (14) 8Anna Chistyakova and Mikhail Pautov Note that replacing the parameters r(Ω f |x) with its estimations in the form l(x) from (13) yields the bound P f ′ ∼Ω f [R(f ′ ,X,τ) <R] < R−1 X l=0 X S⊂X:|S|=l Y x in ∈S l(x in ) Y x out /∈S (1− l(x out )) = p(Ω), (15) that holds with probability at least 1− α. Similarly, P g∼Ξ [R(g,X,τ) > R] < N X l=R +1 X S⊂X:|S|=l Y x in ∈S u(x in ) Y x out /∈S (1−u(x out )) = p (Ξ). (16) Remark 1. During experimentally, we used n = 32,τ = 5,M = 1000 and varied confidence level α such that probabilities α,p(Ξ),p(Ω) were close. Specifically, value α = 5× 10 −6 yields p(Ω) = 10 −6 ,p(Ξ) = 10 −4 andR = 750,R= 600. Thus, if one uses the boundary values R ,R to distinguish between the water- marked and non-watermarked model, one is guaranteed to have both error prob- abilities p (Ω),p(Ξ) low. 3.5 Alternative estimation of bit collisions According to equation 7, the quantity R(f,X,τ) = P N i=1 1[ρ(m(x i ),m ′ (f,x i ))≤ τ] is the sum of N independent Bernoulli random variables. We may rewrite R 1 = R(f ′ ,X,τ) and R 2 = R(g,X,τ) from equation 10 in the form R 1 = ξ 1 + ξ 2 +· + ξ n−1 + ξ n , R 2 = η 1 + η 2 +· + η n−1 + η n , (17) where ξ i ∼ Bernoulli(p i ),η i ∼ Bernoulli(q i ) are independent and parameters (p i ,q i ) are unknown. Letp = 1 n P n i=1 p i and q = 1 n P n i=1 q i . Then, if R < np and R > nq from equation 10, the following lemma holds. Lemma 1. Let δ > 0 and set ε = q 1 2n ln 1 δ . Let ˆp = 1 n R 1 and ˆq = 1 n R 2 be unbiased estimates of p andq, respectively. Then, with probability at least 1− δ, the following upper bounds for probabilities from equation 10 hold: P f ′ ∼Ω f [R(f ′ ,X,τ) <R]≤ h(ˆp,ε − ), P g∼Ξ [R(g,X,τ) > R]≤ h(ˆq,ε + ),(18) where h(ˆp,ε − ) = n(ˆp− ε) R R n(1− (ˆp− ε)) n−R n− R , h(ˆp,ε + ) = n(ˆp + ε) R R n(1− (ˆp + ε)) n− R n−R .(19) RandMark: On Random Watermarking of Visual Foundation Models9 Remark 2. Some words about relationR andp; proof will be moved to the appendix. Proof. We provide a proof for the upper inequality from equation 18. Specifically, we need to upper bound the probability P(R ≤ d), where R ≡ R(f ′ ,X,τ) and d≡ R. According to Chernoff bound, P(R < d)≤ inf t<0 exp(−td)E(exp(tR)).(20) Note that, according to independence of ξ i , E(exp(tR)) = n Y i=1 E(exptξ i ) = n Y i=1 (1− p i + p i e t ), and, hence, (21) P(R < d)≤ inf t<0 " exp(−td) n Y i=1 (1− p i + p i e t ) # .(22) Let φ(p) = ln(1− p + pe t ). Note that φ ′ (p) =− (e t −1) 2 (1−p+pe t ) 2 < 0 for all t < 0, and, hence, φ(p) is strictly concave on [0, 1]. From the concavity of φ(p), it follows that 1 n n X i=1 ln(1− p i + p i e t )≤ ln(1−p +pe t ), n Y i=1 1− p i + p i e t ≤ (1−p +pe t ) n ,(23) and, consequently, P(R < d)≤ inf t<0 exp(−td) 1−p +pe t n .(24) Denote ψ(t) = exp(−td)(1− p + pe t ) n . To find inf t<0 ψ(t), we analyze the derivatives of its logarithm: d dt lnψ(t) =−d + npe t 1− p + pe t (25) Note that d dt lnψ(t) = 0 iff −d + npy 1−p+py = 0, where y = e t < 1. 0 =−d + npy 1− p + py ↔ y = d− pd np− pd ↔ t = ln d− pd np− pd .(26) To satisfy y < 1, it is required that d < np. Since d 2 dt 2 lnψ(t) = npe t (1−p) (1−p+pe t ) 2 > 0 is monotonic, t = ln d−pd np−pd is a unique critical point of d dt lnψ(t). Thus, inf t<0 ψ(t) = γ(p) = np d d n(1− p) n− d n−d (27) 10Anna Chistyakova and Mikhail Pautov and the overall bound is P(R < d)≤ γ(p)(28) Note that γ(p) is impossible to compute directly (sincep) is unknown. Instead, we note that γ(p) is monotonic in p. Indeed, d dp lnγ(p) = d p − n− d 1− p → d dp lnγ(p) = 0 at p = d n ,(29) d 2 dp 2 lnγ(p) =− d p 2 − n− d (1− p) 2 < 0, (30) and hence γ(p) has a unique global maximum at p = d n , is strictly increasing on [0, d n ) and is strictly decreasing on ( d n , 1]. Recall that d < np, so γ(p) is decreasing in p. If we set ˆp = 1 n R 1 , then, according to Hoeffding inequality, P(p≤ ˆp− ε)≤ exp(−2ε 2 n)(31) and from monotonicity of γ(p) for ˆp− ε > d n , P(R < d)≤ γ(ˆp− ε)(32) with probability at least 1− δ for ε = q 1 2n ln 1 δ , what concludes the proof. The proof of the second inequality from equation 18 is similar. 4 Experiments We conducted our experiments using two large-scale VFMs, CLIP [18] and DI- NOv2 [15]. To train models on downstream tasks (namely, for classification and segmentation), we utilized three domain-specific datasets: – E-commerce Product Images: This dataset consists of 18, 175 product images categorized into 9 major classes based on Amazon’s product taxonomy. It is primarily used for image-based product categorization. – FoodSeg103: A food image segmentation dataset containing 7, 118 images annotated with fine-grained pixel-wise labels for over 100 food categories. It supports both semantic segmentation and instance-level analysis of food items. 4.1 Watermark injection Both source VFMs were initialized with publicly available pretrained weights. To embed watermarks, we use a random subset of N = 1000 images from the ImageNet dataset [7], assigning each image a randomly sampled binary vector m of size n = 32. The image and its corresponding message are jointly processed by an encoder, and the resulting output is forwarded through the VFM, allowing the RandMark: On Random Watermarking of Visual Foundation Models11 watermark information to be incorporated while preserving model functionality. The embedded watermark can later be extracted using the RandMark procedure, producing a binary message m ′ to verify model ownership. A schematic illustra- tion of the method is presented in Fig. 1, and detailed architectural descriptions are provided in the Supplementary Material. 4.2 Functional perturbations of VFM To illustrate the robustness of watermarks embedded by RandMark, we evaluate how the detection rate from (7) changes under fine-tuning of the model for downstream tasks and under pruning. Specifically, we fine-tune all the layers of the watermarked VFM for both classification and segmentation downstream tasks, using the aforementioned datasets. The fine-tuning was performed using the AdamW optimizer for 10 epochs. To investigate the impact of model sparsity on both classification accuracy and watermark robustness, we applied post-training unstructured l 1 -norm prun- ing to the entire model. We evaluated two sparsity levels: moderate pruning, where 20% of the lowest-magnitude weights were zeroed out, and aggressive pruning, where 40% of the weights were removed. This procedure enabled us to assess the effect of varying sparsity levels on watermark reconstruction. Note that the unrestricted l 1 -norm pruning is used purely as the baseline to illustrate the robustness of the proposed method to the modifications of the model. 5 Results Experimentally, we assess the efficiency of RandMark by computing the average watermark detection rate from (7) for different values of maximum number of bit errors, τ. In Fig. 2, we demonstrate that our method can be used to reliably detect models that are functionally connected to the watermarked one, namely, the ones obtained via fine-tuning for downstream tasks and pruning. At the same time, RandMark does not falsely detect the presence of the watermarks in negative suspect models. In addition to the detection rate, we analyze correlations between decoded watermark messages. Let f and g denote two models, and let m ′ (f) and m ′ (g) be the watermark bit sequences decoded from them. The covariance between the decoded messages is estimated as ∆ = V(m ′ (f)) + V(m ′ (g))− V(m ′ (f)− m ′ (g)) 2 = cov(f,g). In Fig. 3 for independent models, ∆ is close to zero, indicating negligible cor- relation. In contrast, watermark-dependent models exhibit positive covariance, reflecting correlated decoding of watermark bits. This complementary metric provides additional evidence of functional dependence between models and can help distinguish watermarked models from unrelated ones. 12Anna Chistyakova and Mikhail Pautov 0246810121416 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate 0246810121416 Max Allowed Bit Errors 0.0 0.1 0.2 0.3 0.4 True Positive Rate Positive suspect Negative suspect Positive suspect Negative suspect classification segmentation pruning 20% pruning 40% (a) The architecture of VFM is CLIP 0246810121416 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate 0246810121416 Max Allowed Bit Errors 0.0 0.1 0.2 0.3 0.4 True Positive Rate Positive suspect Negative suspect Positive suspect Negative suspect classification segmentation pruning 20% pruning 40% (b) The architecture of VFM is DINOv2 Fig. 2: Watermark detection rate R from (7), averaged over N = 1000 images used for watermarking. Classification experiments were conducted on the E-commerce Product Images dataset, segmentation experiment was conducted on the FoodSeg103 dataset. 5.1 Comparison with the baseline fingerprinting and watermarking approaches We indicate the lack of fingerprinting methods designed specifically for visual foundation models. To compare our approach with some of the general-purpose fingerprinting approaches, we add a fine-tuned classification head to the source VFM. The classification head concatenates the CLS token with the mean of patch tokens, applies normalization and dropout, and feeds the result into a linear classifier. Here, the classification backbone is fine-tuned on the ImageNet dataset. Thus, we compare fingerprinting approaches in a classification scenario. Specifically, we compare RandMark with ADV-TRA [26] and IPGuard [4] and report results in Table 2, where we present average watermark detection rates for both positive and negative suspect models. Specifically, negative suspect models here are the ones of different architecture (DINOv2 with registers and CLIP). There, experiments with the positive suspect models correspond to the ones reported in Fig. 2. It is noteworthy that the proposed method outperforms general-purpose fingerprinting techniques in terms of watermark detection rate, both for positive and negative suspect models. To evaluate the robustness of the proposed method, we compare it with the baseline approach proposed in [2,21]. In particular, we fine-tune the watermarked VFMs on an image segmentation task and monitor both the downstream task performance and the ability to recover the embedded watermark. This evalua- RandMark: On Random Watermarking of Visual Foundation Models13 0.150.100.050.000.050.100.150.200.25 = Cov(f, g) Density Independent models Watermark-dependent models (a) The architecture of VFM is DINOv2 0.100.050.000.050.100.150.200.25 = Cov(f, g) Density Independent models Watermark-dependent models (b) The architecture of VFM is CLIP Fig. 3: Distribution of covariance between decoded watermark messages from two mod- els f and g. Independent models produce covariance values near zero, while watermark- dependent models exhibit positive covariance due to correlated decoding of watermark bits. tion allows us to analyze whether the fingerprint remains detectable after task adaptation and whether the downstream performance is preserved. The results are presented in Table 2. We observe that the baseline approach introduces a substantial degradation of the downstream task performance while also failing to preserve the watermark under fine-tuning. In contrast, our method maintains high watermark extraction accuracy while achieving significantly bet- ter segmentation performance, indicating that the proposed approach preserves both task adaptability and fingerprint detectability. To assess the computational complexity of the watermarking and fingerprint- ing methods, we refer the reader to the Supplementary Material. 6 Conclusion In this work, we propose RandMark, a novel watermarking approach for visual foundation models. This method is model agnostic: it is worth mentioning that the model’s owner has to prepare a set of input images and perform the water- mark embedding procedure only once for a given instance of the model; then, 14Anna Chistyakova and Mikhail Pautov Table 1: Comparison with baseline watermarking methods under segmentation fine- tuning. Segmentation performance and watermark extraction accuracy are reported after different numbers of fine-tuning epochs; the architecture of the source VFM is CLIP. 1 epoch3 epochs5 epochs MethodSegm.↑ WM↑ Segm.↑ WM↑ Segm.↑ WM↑ Randomized Smoothing 0.14 0.27 0.36 0.00 0.46 0.00 Ours0.32 1.00 0.52 0.99 0.55 0.97 Table 2: Quantitative comparison with the general-purpose fingerprinting methods. We report the average watermark detection rate; the architecture of the source VFM is DINOv2. Model typeExperimentRandMark ADV-TRA IPGuard Positive suspect↑ classification0.8700.0210.000 segmentation0.7500.0000.000 pruning (20%)1.0001.0000.530 pruning (40%)1.0000.0860.000 Negative suspect↓ DINOv2 w/ registers 0.0000.0120.010 CLIP0.0000.0000.000 the watermarked model remains detectable by our method after fine-tuning to a particular downstream task (for example, image classification and segmenta- tion). On the other hand, we verify that RandMark does not detect benign, independent models as functional copies of the watermarked VFM, which makes the method applicable in practical scenarios. We theoretically show that our method, by design, yields low false positive and false negative detection rates. References 1. Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al.: A cookbook of self-supervised learning. In: International Conference on Learning Representations. vol. 162, p. 24388–24413. Transactions of Machine Learning Research 2. Bansal, A., Chiang, P.Y., Curry, M.J., Jain, R., Wigington, C., Manjunatha, V., Dickerson, J.P., Goldstein, T.: Certified neural network watermarks with random- ized smoothing. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, p. 1450–1465. PMLR (2022) 3. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in Neural Information Processing Systems 33, 1877–1901 (2020) RandMark: On Random Watermarking of Visual Foundation Models15 4. Cao, X., Jia, J., Gong, N.Z.: Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In: Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security. p. 14–25 (2021) 5. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 9650–9660 (2021) 6. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: International Conference on Machine Learning. p. 1597–1607. PmLR (2020) 7. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. p. 248–255. Ieee (2009) 8. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (2021) 9. Guo, J., Potkonjak, M.: Watermarking deep neural networks for embedded sys- tems. In: 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). p. 1–8. IEEE (2018) 10. Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In: ICLR (2016) 11. He, Z., Zhang, T., Lee, R.: Sensitive-sample fingerprinting of deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 4729–4737 (2019) 12. Hinton, G.: Distilling the knowledge in a neural network. In: Deep Learning and Representation Learning Workshop in Conjunction with NIPS (2014) 13. Li, Y., Zhu, L., Jia, X., Jiang, Y., Xia, S.T., Cao, X.: Defending against model stealing via verifying embedded external features. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, p. 1464–1472 (2022) 14. Ma, J., Wang, B.: Towards foundation models of biological image segmentation. Nature Methods 20(7), 953–955 (2023) 15. Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Re- search (2024) 16. Pautov, M., Bogdanov, N., Pyatkin, S., Rogov, O., Oseledets, I.: Probabilistically robust watermarking of neural networks. In: Proceedings of the Thirty-Third In- ternational Joint Conference on Artificial Intelligence. p. 4778–4787 (2024) 17. Quan, Y., Teng, H., Xu, R., Huang, J., Ji, H.: Fingerprinting deep image restoration models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. p. 13285–13295 (2023) 18. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. p. 8748–8763. PmLR (2021) 19. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019) 16Anna Chistyakova and Mikhail Pautov 20. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022) 21. Ren, J., Zhou, Y., Jin, J., Lyu, L., Yan, D.: Dimension-independent certified neural network watermarks via mollifier smoothing. In: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (eds.) Proceedings of the 40th Inter- national Conference on Machine Learning. Proceedings of Machine Learning Re- search, vol. 202, p. 28976–29008. PMLR (2023) 22. Sander, T., Fernandez, P., Durmus, A., Douze, M., Furon, T.: Watermarking makes language models radioactive. Advances in Neural Information Processing Systems 37, 21079–21113 (2024) 23. Song, H.J., Khayatkhoei, M., AbdAlmageed, W.: Manifpt: Defining and analyzing fingerprints of generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 10791–10801 (2024) 24. Uchida, Y., Nagai, Y., Sakazawa, S., Satoh, S.: Embedding watermarks into deep neural networks. In: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. p. 269–277 (2017) 25. Xu, J., Wang, F., Ma, M., Koh, P.W., Xiao, C., Chen, M.: Instructional finger- printing of large language models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). p. 3277–3306 (2024) 26. Xu, T., Wang, C., Liu, G., Yang, Y., Peng, K., Liu, W.: United we stand, divided we fall: Fingerprinting deep neural networks via adversarial trajectories. Advances in Neural Information Processing Systems 37, 69299–69328 (2024)