Paper deep dive
Conditional Factuality Controlled LLMs with Generalization Certificates via Conformal Sampling
Kai Ye, Qingtao Pan, Shuo Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/31/2026, 2:04:10 AM
Summary
The paper introduces Conditional Factuality Control (CFC), a post-hoc conformal framework for Large Language Models (LLMs) that provides set-valued outputs with conditional coverage guarantees. By utilizing augmented quantile regression on latent success scores, CFC adapts acceptance thresholds to specific prompt features, addressing the limitations of marginal conformal prediction which often under-covers hard prompts and over-covers easy ones. The authors also propose CFC-PAC, a variant with finite-sample certificates, and provide theoretical proofs demonstrating that CFC is more sample-efficient than marginal methods.
Entities (5)
Relation Signals (3)
CFC-PAC → isvariantof → CFC
confidence 99% · We further derive a PAC-style variant, CFC-PAC
CFC → provides → Conditional Coverage
confidence 98% · CFC, a post-hoc conformal framework that returns set-valued outputs with conditional coverage guarantees.
CFC → uses → Augmented Quantile Regression
confidence 97% · CFC defines a continuous, feature-conditional acceptance threshold through augmented quantile regression
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) need reliable test-time control of hallucinations. Existing conformal methods for LLMs typically provide only \emph{marginal} guarantees and rely on a single global threshold, which can under-cover hard prompts, over-cover easy ones, and produce oversized prediction sets. We propose \emph{Conditional Factuality Control} (CFC), a post-hoc conformal framework that returns \emph{set-valued} outputs with \emph{conditional} coverage guarantees. CFC defines a continuous, feature-conditional acceptance threshold through augmented quantile regression on a latent ``success'' score, and deploys it through a fixed-point threshold rule at inference time. Theoretically, we show that CFC satisfies a conditional coverage guarantee under exchangeability and analyze its \emph{efficiency}, proving that, under mild assumptions on the score distributions, the conditional rule is strictly more sample-efficient than marginal conformal prediction at the same target coverage. We further derive a PAC-style variant, CFC-PAC, which shrinks the nominal risk level based on a stability bound, yielding a finite-sample certificate that the conditional miscoverage deviates from the target by at most $O(\sqrt{\log(1/\delta)/N})$. Empirically, on synthetic data, real-world reasoning and QA benchmarks, and a Flickr8k VLM setting, CFC and CFC-PAC consistently attain near-target coverage across difficulty groups while using smaller prediction sets than CP and non-CP baselines.
Tags
Links
- Source: https://arxiv.org/abs/2603.27403v1
- Canonical: https://arxiv.org/abs/2603.27403v1
Trouble viewing inline? Open PDF directly →
Full Text
82,933 characters extracted from source content.
Expand or collapse full text
Conditional Factuality Controlled LLMs with Generalization Certificates via Conformal Sampling Kai Ye Qingtao Pan Shuo Li Case Western Reserve University Kai.Ye@pitt.edu qingtaopan33@gmail.com shuo.li11@case.edu Corresponding author. Abstract Large language models (LLMs) need reliable test-time control of hallucinations. Existing conformal methods for LLMs typically provide only marginal guarantees and rely on a single global threshold, which can under-cover hard prompts, over-cover easy ones, and produce oversized prediction sets. We propose Conditional Factuality Control (CFC), a post-hoc conformal framework that returns set-valued outputs with conditional coverage guarantees. CFC defines a continuous, feature-conditional acceptance threshold through augmented quantile regression on a latent “success” score, and deploys it through a fixed-point threshold rule at inference time. Theoretically, we show that CFC satisfies a conditional coverage guarantee under exchangeability and analyze its efficiency, proving that, under mild assumptions on the score distributions, the conditional rule is strictly more sample-efficient than marginal conformal prediction at the same target coverage. We further derive a PAC-style variant, CFC-PAC, which shrinks the nominal risk level based on a stability bound, yielding a finite-sample certificate that the conditional miscoverage deviates from the target by at most O(log(1/δ)/N)O( (1/δ)/N). Empirically, on synthetic data, real-world reasoning and QA benchmarks, and a Flickr8k VLM setting, CFC and CFC-PAC consistently attain near-target coverage across difficulty groups while using smaller prediction sets than CP and non-CP baselines. 1 Introduction Large language models (LLMs) have delivered striking progress across reasoning and generation tasks [21, 2], yet their outputs can be unreliable due to hallucinations [9]. Inference-time strategies that invest more compute on sampling often improve accuracy [5, 1], but they do not provide formal reliability guarantees. For safety‑critical or high‑stakes applications, such heuristics are insufficient: we need procedures that make uncertainty explicit and can control error rates. Figure 1: Limitation of marginal CP and advantage of proposed CFC. Left: A single global threshold learned from the marginal score mixture yields only marginal coverage and can under‑cover hard prompts while over‑covering easy ones. Right: Our CFC learns a data‑dependent threshold via conformal quantile regression, adapting the acceptance level to input features and achieving conditional coverage across subgroups. Among uncertainty-estimation methods, conformal prediction (CP) is a natural choice for adapting uncertainty control to LLMs, complementing broader efforts on trustworthy uncertainty modeling and multimodal prediction [19, 22, 23, 24, 16]. It is model‑agnostic and distribution‑free; under exchangeability between calibration and test examples, it constructs set‑valued predictions that contain the true candidate with probability at least 1−α1-α for a user‑specified risk level α [19]. Recent work adapts CP to LLMs by constructing sets of sampled responses that aim to contain at least one correct answer with high probability [14, 11, 18]. However, these methods typically rely on a single global threshold on scores, and so they only provide marginal guarantees: coverage holds on average over prompts, not for prompts with particular characteristics. This marginal coverage can hide severe heterogeneity. Hard prompts (e.g., long math questions or rare entities) may be systematically under‑covered, while easy prompts are over‑covered, potentially inflating prediction sets unnecessarily (Fig. 1, left). A global threshold is forced to compromise between easy and hard regions of the feature space, leading to miscalibration in subgroups and inefficient use of samples. Motivation. To address this failure mode, we seek conditional coverage: guarantees that coverage holds not only on average, but also when conditioning on relevant features (or groups) of the prompt. Conditional coverage is strictly stronger than marginal coverage and directly targets reliability in under‑represented or systematically hard subpopulations (Fig. 1, right). At the same time, we would like prediction sets to remain compact on average so that sampling‑based inference stays computationally practical. We tackle these challenges with Conditional Factuality Control (CFC), a post-hoc conformal layer for LLM sampling. Rather than using a single scalar threshold, CFC defines a feature-conditional acceptance rule λ^α(X) λ_α(X) by conformalizing a quantile-regression model for a latent success score S(X)S(X), namely the best score among correct candidates for prompt X. At test time, given a prompt X, we sample candidates from the base generator and accept those whose scores satisfy V(X,y)≤λ^α(X)V(X,y)≤ λ_α(X). This requires no finetuning of the base model, and lets the acceptance threshold adapt to prompt difficulty. Beyond the basic procedure, we develop a PAC-style variant, CFC-PAC, that adds a stability-based finite-sample certificate: with high probability over the draw of calibration sample, the deployed rule achieves coverage at least 1−α−εN(δ)1-α- _N(δ), where εN(δ)=O(log(1/δ)/N) _N(δ)=O\! ( (1/δ)/N ). Finally, we study the efficiency of CFC. Under natural assumptions relating prompt difficulty to the distribution of scores, we show that an oracle conditional rule can attain smaller expected prediction-set size than marginal CP rule at the same target coverage. Our learned CFC asymptotically inherits this oracle efficiency as the augmented quantile regression becomes consistent. Our main contributions are: • We introduce CFC, a conformal procedure for sampled LLM outputs that defines a continuous, feature-conditional acceptance rule via augmented quantile regression on the latent success score and yields prediction sets satisfying the conditional guarantee of [7] under exchangeability. • We develop CFC-PAC, a certified variant of CFC equipped with a PAC-style generalization bound: with probability at least 1−δ1-δ over the calibration sample, the deployed rule achieves coverage at least 1−α−εN(δ)1-α- _N(δ) for an εN(δ)=O(log(1/δ)N) _N(δ)=O\! ( (1/δ)N ). • We analyze the efficiency of CFC, proving that, under mild monotonicity and concavity assumptions on the score distribution, conditional rules can be strictly more sample-efficient than marginal CP rules at the same coverage level. • We validate CFC and CFC-PAC on synthetic data, real-world reasoning / QA benchmarks, and a Flickr8k VLM setting, showing that the same post-hoc procedure extends beyond text-only generators without finetuning the base model. Code availability. Code for reproducing our experiments is available here. 2 Related Work Inference-time sampling and reranking for LLMs. A common way to improve LLM outputs with extra test-time compute is to sample multiple candidates and rerank or filter them, as in Best-of-N decoding and pass@N evaluation [5, 3, 1]. In practice, candidate quality is often estimated with an external verifier or reward model [5, 8]. Our setting follows this line of work: we treat the base generator as a black box, sample multiple candidates, and use a verifier score to decide which candidates enter the final set. Conformal prediction and conditional guarantees. Conformal prediction provides distribution-free finite-sample coverage under exchangeability [19, 17]. Split/inductive conformal prediction (ICP) is the standard practical variant, but its guarantee is marginal: it controls coverage only on average over test inputs [13]. Exact conditional coverage is impossible without additional assumptions or relaxations [20, 6]. Recent work therefore studies weaker but useful coverages, including conformalized quantile regression [15] and the function-class conditional framework of Gibbs et al. [7], which learns feature-dependent thresholds through augmented quantile regression. Our method builds on this latter perspective. Conformal prediction for LLMs. Several recent papers adapt CP to language models by constructing sets of sampled responses that contain at least one correct answer with high probability [11, 14, 18, 4, 12]. Most existing LLM-specific methods use a global acceptance rule and therefore inherit only marginal coverage, which can under-cover hard prompts and over-cover easy ones. Relative to these marginal baselines, CFC replaces the single threshold with a feature-conditional one and is therefore designed to improve subgroup reliability. Compared with prior conditional CP works, our contributions are different in emphasis: we develop conformal factuality control for sampled LLM candidates with verifier scores, provide an efficiency analysis showing when conditional rules are more sample-efficient than marginal ones, introduce the PAC-certified variant CFC-PAC, and demonstrate transfer to a VLM setting in the main experiments. Because CFC is purely post hoc, it requires no generator finetuning and transfers across base models. 3 Preliminaries 3.1 Conformal Factuality Let X∈X be a prompt and let π:→Δ()π:X→ (Y) denote a fixed generator over candidate completions. For each prompt, we sample a candidate set C(X)=Yjj=1MC(X)=\Y_j\_j=1^M with Yj∼π(⋅∣X)Y_j π(· X), and evaluate each candidate with a verifier score V:×→[0,1]V:X×Y→[0,1], normalized so that smaller scores are better. Given a correctness indicator A(X,y)∈0,1A(X,y)∈\0,1\, the conformal factuality goal is to output a set C^α(X) C_α(X) satisfying ℙ(∃y∈C^α(Xn+1):A(Xn+1,y)=1)≥1−α.P\! (∃ y∈ C_α(X_n+1):A(X_n+1,y)=1 )≥ 1-α. (3.1) A convenient latent variable for this event is the success score S(X):=infV(X,y):y∈C(X),A(X,y)=1,S(X):= \\,V(X,y):y∈ C(X),\ A(X,y)=1\,\, (3.2) so that the event that the prediction set contains at least one correct answer is equivalent to S(X)≤λ(X)S(X)≤λ(X) for the deployed threshold rule λ(⋅)λ(·). 3.2 From Marginal to Conditional Coverage Split conformal prediction. Standard split conformal prediction calibrates a single global threshold from held-out calibration scores and therefore provides only marginal coverage. Given exchangeable data (Xi,Yi)i=1n+1\(X_i,Y_i)\_i=1^n+1 and a nonconformity score s(x,y)s(x,y), it constructs a prediction set C^α(Xn+1) C_α(X_n+1) satisfying ℙ(Yn+1∈C^α(Xn+1))≥1−α.P\! (Y_n+1∈ C_α(X_n+1) )≥ 1-α. (3.3) Using a calibration set (Xi,Yi)i=1n\(X_i,Y_i)\_i=1^n, one computes calibration scores Ri=s(Xi,Yi)R_i=s(X_i,Y_i) and forms the empirical quantile q^1−α:=R(⌈(n+1)(1−α)⌉) q_1-α:=R_( (n+1)(1-α) ) (3.4) and predicts C^α(x):=y:s(x,y)≤q^1−α. C_α(x):=\y:s(x,y)≤ q_1-α\. (3.5) In our setting, this corresponds to accepting all sampled candidates with verifier score below one constant cutoff. Such a rule can over-cover easy prompts and under-cover hard prompts, because the same acceptance level must serve the entire prompt distribution. Conditional conformal prediction. The limitation of marginal CP is that it averages over the prompt distribution. A stronger objective is pointwise conditional coverage, ℙ(Yn+1∈C^(Xn+1)∣Xn+1=x)≥1−α,P\! (Y_n+1∈ C(X_n+1) X_n+1=x )≥ 1-α, (3.6) but exact finite-sample conditional coverage is impossible without additional assumptions or relaxations [20, 6]. Following Gibbs et al. [7], we instead target function-class conditional coverage: [ \! [ f(Xn+1)(∃y∈C^(Xn+1):A(Xn+1,y)=1 f(X_n+1) ( 1\∃ y∈ C(X_n+1):A(X_n+1,y)=1\ (3.7) −(1−α))]=0 -(1-α) ) ]=0 for all f in a chosen class ℱF. When ℱ=1F=\1\, this reduces to marginal conformal prediction; when ℱ=Φ(⋅)⊤β:β∈ℝdF=\ (·) β:β ^d\, it yields a feature-conditional guarantee over the basis Φ . CFC instantiates this framework for conformal factuality by learning a feature-dependent quantile of the latent success score S(X)S(X) rather than a single global cutoff. The main paper focuses on the procedure and its guarantees. Appendix A gives the detailed background about ICP and Gibbs. [7] work. 4 Method We now present CFC, our conditional conformal rule for sampled LLM outputs, and its PAC-style variant CFC-PAC. Appendices A and B contain the extended background, derivations, and proofs. 4.1 CFC: Conditional Factuality Control Let (Xi,Si)i=1N\(X_i,S_i)\_i=1^N be calibration prompt/score pairs, where Si=S(Xi)S_i=S(X_i) is the latent success score from Sec. 3. For a test prompt XN+1X_N+1, we sample CN+1=YN+1,jj=1MC_N+1=\Y_N+1,j\_j=1^M from π(⋅∣XN+1)π(· X_N+1) and deploy a prompt-dependent threshold on verifier scores: C^α(XN+1;λ):=y∈CN+1:V(XN+1,y)≤λ. C_α(X_N+1;λ):=\y∈ C_N+1:V(X_N+1,y)≤λ\. (4.1) To learn this threshold, we follow the augmented quantile-regression construction of [7]. For a candidate test-time score s∈[0,1]s∈[0,1], define βs=argminβ∈ℝd[1N+1∑i=1Nρ1−α(Si−Φ(Xi)⊤β) _s= *arg\,min_β ^d [ 1N+1 _i=1^N _1-α\! (S_i- (X_i) β ) (4.2) +1N+1ρ1−α(s−Φ(XN+1)⊤β)], + 1N+1 _1-α\! (s- (X_N+1) β ) ], where Φ(X) (X) is the chosen feature map and ρ1−α(u)=u(1−α−u<0) _1-α(u)=u(1-α- 1\u<0\) is the pinball loss. The induced test-time map is gXN+1(s):=Φ(XN+1)⊤βs.g_X_N+1(s):= (X_N+1) _s. We then take the largest fixed point below this map as the deployed threshold: λ^α(XN+1):=sups∈[0,1]:s≤gXN+1(s), λ_α(X_N+1):= \s∈[0,1]:s≤ g_X_N+1(s)\, (4.3) and return C^α(XN+1):=y∈CN+1:V(XN+1,y)≤λ^α(XN+1). C_α(X_N+1):=\y∈ C_N+1:V(X_N+1,y)≤ λ_α(X_N+1)\. (4.4) Although Eqs. (4.2)–(4.3) define CFC conceptually through an augmented quantile-regression family, deployment only requires computing the fixed-point threshold in Eq. (4.3); it does not require repeatedly solving Eq. (4.2) on a grid of candidate scores (see Gibbs. [7]). Algorithm 1 CFC Inference 0: Calibration pairs (Xi,Si)i=1N\(X_i,S_i)\_i=1^N, test prompt x, generator π, verifier V, feature map Φ , nominal significance α, sample budget M 1: Draw C(x)=Yjj=1MC(x)=\Y_j\_j=1^M with Yj∼π(⋅∣x)Y_j π(· x) 2: Let ρ1−α(u)=u(1−α−u<0) _1-α(u)=u(1-α- 1\u<0\) 3: For s∈[0,1]s∈[0,1], define 4: βs=argminβ∈ℝd[1N+1∑i=1Nρ1−α(Si−Φ(Xi)⊤β)+1N+1ρ1−α(s−Φ(x)⊤β)] _s= *arg\,min_β ^d [ 1N+1 _i=1^N _1-α\! (S_i- (X_i) β )+ 1N+1 _1-α\! (s- (x) β ) ] 5: Define gx(s)=Φ(x)⊤βsg_x(s)= (x) _s 6: Compute λ^α(x)=sups∈[0,1]:s≤gx(s) λ_α(x)= \s∈[0,1]:s≤ g_x(s)\ 7: Set C^α(x)=y∈C(x):V(x,y)≤λ^α(x) C_α(x)=\y∈ C(x):V(x,y)≤ λ_α(x)\ 8: return λ^α(x),C^α(x) λ_α(x), C_α(x) Theorem 4.1 (Conditional coverage of CFC). Let ℱ=Φ(X)⊤β:β∈ℝdF=\ (X) β:β ^d\ be any finite-dimensional linear class, and assume exchangeability. Then for any non-negative f∈ℱf with [f(X)]>0E[f(X)]>0, the prediction set in Eq. (4.3) satisfies ℙf(∃y∈C^α(XN+1):A(XN+1,y)=1)≥1−α.P_f\! (∃ y∈ C_α(X_N+1):A(X_N+1,y)=1 )≥ 1-α. (4.5) 4.2 CFC-PAC: A High-Probability Certificate Theorem 4.1 is an expectation-level guarantee over the calibration draw. To certify the deployed rule itself, we add ridge regularization to Eq. (4.2) and shrink the nominal target by a stability slack. Assumptions. We assume: (1) bounded features ‖Φ(X)‖2≤R\| (X)\|_2≤ R almost surely; (2) ridge regularization λ2‖β‖22 λ2\|β\|_2^2 with λ>0λ>0 in the augmented quantile-regression objective; and (3) the conditional CDF FS∣X=x(t)F_S X=x(t) is L-Lipschitz in t on [0,1][0,1]. Exchangeability is as in Theorem 4.1. Theorem 4.2 (PAC conditional coverage for CFC). Assume α≥εN(δ)α≥ _N(δ) and define αeff:=α−εN(δ). _eff:=α- _N(δ). Let λ^αeff(⋅) λ_ _eff(·) be the threshold learned from algorithm 2. Then for any δ∈(0,1)δ∈(0,1), with probability at least 1−δ1-δ over the calibration sample, ℙ(S≤λ^αeff(X)∣cal)≥1−αeff−εN(δ)=1−α,P\! (S≤ λ_ _eff(X) _cal )≥ 1- _eff- _N(δ)=1-α, where εN(δ)=O(log(1/δ)N). _N(δ)=O\! ( (1/δ)N ). Equivalently, with the same probability, ℙ(∃y∈C^αeff(X):A(X,y)=1∣cal)≥1−α.P\! (∃ y∈ C_ _eff(X):A(X,y)=1 _cal )≥ 1-α. Algorithm 2 CFC-PAC Inference 0: Calibration pairs (Xi,Si)i=1N\(X_i,S_i)\_i=1^N, test prompt x, generator π, verifier V, feature map Φ , nominal significance α, confidence δ, ridge parameter λ, sample budget M 1: Compute the PAC slack εN(δ) _N(δ) from the PAC certificate 2: Set the effective target αeff=max0,α−εN(δ) _eff= \0,α- _N(δ)\ 3: Similar to Algorithm 1, run CFC with target αeff _eff, replacing the augmented objective by the ridge-regularized version below 4: βs=argminβ∈ℝd[1N+1∑i=1Nρ1−αeff(Si−Φ(Xi)⊤β)+1N+1ρ1−αeff(s−Φ(x)⊤β)+λ2‖β‖22] _s= *arg\,min_β ^d [ 1N+1 _i=1^N _1- _eff\! (S_i- (X_i) β )+ 1N+1 _1- _eff\! (s- (x) β )+ λ2\|β\|_2^2 ] 5: This produces λ^αeff(x) λ_ _eff(x) and C^αeff(x) C_ _eff(x) 6: return λ^αeff(x),C^αeff(x) λ_ _eff(x), C_ _eff(x) 4.3 Efficiency Analysis Beyond coverage, we ask whether conditioning can reduce average prediction-set size. Let GX(λ):=ℙ(V(X,Y)≤λ∣X,Y∼π(⋅∣X))G_X(λ):=P (V(X,Y)≤λ X,\ Y π(· X) ) be the score CDF under sampling at prompt X. If we draw M candidates and retain those with score at most λ, then [|C^(X)|∣X]=MGX(λ).E [| C(X)| X ]=M\,G_X(λ). Thus efficiency reduces to comparing [GX(λ(X))]E[G_X(λ(X))] across threshold rules. Marginal CP uses a constant threshold λ¯α λ_α satisfying ℙ(S≤λ¯α)≈1−αP(S≤ λ_α)≈ 1-α. In contrast, an oracle conditional rule uses the conditional quantile qα(X)q_α(X) of S∣XS X and sets λ⋆(X)=qα(X)λ (X)=q_α(X). Let T=ψ(X)T=ψ(X) be a scalar difficulty variable. For each t, define Ft(λ):=ℙ(S≤λ∣T=t),Gt(λ):=ℙ(V≤λ∣T=t),F_t(λ):=P(S≤λ T=t), G_t(λ):=P(V≤λ T=t), and Ct(u):=Gt(Ft−1(u)),u∈[0,1],C_t(u):=G_t\! (F_t^-1(u) ), u∈[0,1], where Ft−1(u):=infλ∈[0,1]:Ft(λ)≥u.F_t^-1(u):= \λ∈[0,1]:F_t(λ)≥ u\. Assumptions. (1) For each t, FtF_t is continuous and strictly increasing on [0,1][0,1]. (2) For each fixed λ∈[0,1]λ∈[0,1], the map t↦Ft(λ)t F_t(λ) is nonincreasing. (3) For each t, the map u↦Ct(u)u C_t(u) is convex and differentiable on (0,1)(0,1). (4) C has decreasing differences in (u,t)(u,t), i.e. for every u∈(0,1)u∈(0,1),t↦∂uCt(u)t _uC_t(u) is nonincreasing. Under Assumptions 1–2, the conditional (1−α)(1-α)-quantile qα(t):=Ft−1(1−α)q_α(t):=F_t^-1(1-α) is automatically nondecreasing in t. Proposition 4.3 (Oracle CFC efficiency). Let λ⋆(X):=qα(X)=FX−1(1−α).λ (X):=q_α(X)=F_X^-1(1-α). Under the assumptions above, [GX(λ⋆(X))]≤[GX(λ¯)]E [G_X(λ (X)) ] [G_X( λ) ] for any constant λ¯ λ satisfying ℙ(S≤λ¯)=1−α.P(S≤ λ)=1-α. In particular, [GX(λ⋆(X))]≤[GX(λ¯α)].E [G_X(λ (X)) ] [G_X( λ_α) ]. If, in addition, u↦Ct(u)u C_t(u) is strictly convex for almost every t and ℙ(qα(X)≠λ¯α)>0,P (q_α(X)≠ λ_α )>0, then the inequality is strict. Theorem 4.4 (CFC inherits oracle efficiency). Assume the conditions of Proposition 4.3, and let λ⋆(X):=qα(ψ(X)).λ (X):=q_α(ψ(X)). Let λ^α,N(⋅) λ_α,N(·) denote the CFC threshold learned from a calibration sample of size N, and suppose that supx∈|λ^α,N(x)−λ⋆(x)|→0as N→∞. _x | λ_α,N(x)-λ (x) | p0 N→∞. Then limN→∞[GX(λ^α,N(X))]=[GX(λ⋆(X))]≤[GX(λ¯α)]. _N→∞E [G_X( λ_α,N(X)) ]=E [G_X(λ (X)) ] [G_X( λ_α) ]. Consequently, for fixed M, limN→∞[|C^α,N(X)|]=M[GX(λ⋆(X))]≤M[GX(λ¯α)]. _N→∞E [| C_α,N(X)| ]=M\,E [G_X(λ (X)) ]≤ M\,E [G_X( λ_α) ]. If, in addition, the strictness conditions in Proposition 4.3 hold, then limN→∞[GX(λ^α,N(X))]<[GX(λ¯α)], _N→∞E [G_X( λ_α,N(X)) ]<E [G_X( λ_α) ], and hence limN→∞[|C^α,N(X)|]<M[GX(λ¯α)]. _N→∞E [| C_α,N(X)| ]<M\,E [G_X( λ_α) ]. 5 Experiments 5.1 Synthetic Data Setup. We first study a controlled synthetic setting with a scalar prompt-difficulty variable T∈[0,1]T∈[0,1]. For each prompt we draw M candidates, the probability of correctness decreases with difficulty, and verifier scores are sampled from a known difficulty-dependent distribution. Unless otherwise stated, we use target error α=0.10α=0.10, calibration size Ncal=10,000N_cal=10,000, test size Ntest=10,000N_test=10,000, and M=50M=50. We report mean ± standard deviation over random seeds. Baselines and metrics. We compare against three baselines. TopK keeps the smallest top-K prefix whose calibration coverage reaches the target level. ICP [19] calibrates a single global verifier threshold. Learnt CP fits a feature-conditional threshold from calibration data but omits the exact conformal correction. We report empirical coverage rate (ECR), average prediction set size (APSS, lower is better), and group-stratified coverage (GSC), the minimum empirical coverage over difficulty groups. On synthetic data, ECR and GSC are computed using the ground-truth correctness labels, and APSS is the average accepted set size. Figure 2: Groupwise miscoverage on synthetic data across 10 difficulty bins. The dashed line marks the target miscoverage α=0.10α=0.10. Learnt CP improves over marginal baselines, but CFC and CFC-P remain closest to the target across all bins, especially on hard prompts. Results. Figure 2 shows the main phenomenon motivating CFC: a single global threshold under-covers hard prompts and over-covers easy ones, while conditional thresholds track the target miscoverage across the entire difficulty range. Learnt CP narrows this gap, but it still fails to match the subgroup reliability of CFC/CFC-P on the hardest bins. CFC achieves the smallest average prediction sets at the target level, and CFC-P raises the worst-group coverage floor further with only a modest set-size increase. This shows that the gain does not come merely from learning a better global score; it comes from feature-conditional conformalization. Figure 3: Learned threshold λ^α(X) λ_α(X) versus prompt difficulty. Easy prompts receive stricter thresholds, while harder prompts receive looser thresholds, which is the mechanism behind the improved group-wise reliability of CFC. Threshold adaptation. Figure 3 visualizes the learned threshold itself. Relative to the single ICP cutoff, CFC assigns tighter thresholds to easy prompts and looser thresholds to hard prompts, which is exactly the behavior needed to correct the systematic under-coverage of global-threshold baselines on difficult inputs. Table 1: Synthetic-data results at α=0.10α=0.10. Learnt CP uses a learned difficulty-conditional threshold but no exact conformal correction, isolating learning from feature-conditional conformalization. Method ECR APSS↓ GSC↑ TopK 90.6 ± 0.1 16.00 ± 0.00 58.2 ± 1.3 ICP 90.2 ± 0.2 16.71 ± 0.23 57.4 ± 1.4 Learnt CP 90.2 ± 0.3 15.72 ± 0.15 84.3 ± 0.5 CFC (ours) 90.3 ± 0.5 15.53 ± 0.12 88.7 ± 0.7 CFC-P (ours) 90.8 ± 0.6 15.87 ± 0.16 89.1 ± 0.5 Table 1 shows that learning a stronger difficulty-aware threshold already helps substantially, but it still fails to match the subgroup reliability of CFC/CFC-P. This isolates the main empirical point of the method: the gains are not explained by better score fitting alone, but by the exact conditional conformal correction on top of that learned threshold. Additional Sweeps and Ablations. Appendix C reports the complete synthetic sweep across target error rates α together with additional sensitivity analyses over calibration size, sampling budget, and group granularity. Figure 4: Groupwise miscoverage on real-world datasets at representative target errors: TriviaQA α=0.25α=0.25, GSM8K α=0.10α=0.10, and Flickr8k α=0.03α=0.03. Bars show mean miscoverage over split seeds and the upper error bars show one standard deviation. The TriviaQA panel uses the chosen two-group feature map; Appendix C gives its exact construction. The GSM8K and Flickr8k panels use five equal-frequency difficulty groups ordered from easy to hard. The dashed line marks the target miscoverage α. Across all three datasets, the conditional methods flatten the miscoverage profile relative to marginal baselines, especially on the hardest groups. Table 2: TriviaQA results for α∈0.20,0.25,0.30,0.35α∈\0.20,0.25,0.30,0.35\ under the chosen calibration-defined feature map. APSS below 11 indicates abstention on some prompts. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α1-α are preferred. Appendix C reports the full experiments with additional details. Methods α=0.20α=0.20 α=0.25α=0.25 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 81.6 ± 0.3 69.7 ± 1.5 1.75 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 ICP 80.0 ± 0.5 65.1 ± 1.5 1.43 ± 0.02 74.9 ± 0.3 56.7 ± 1.9 1.08 ± 0.01 Learnt CP 79.6 ± 0.5 76.3 ± 1.5 1.69 ± 0.04 74.7 ± 0.4 74.0 ± 1.1 1.22 ± 0.03 CFC (ours) 76.2 ± 0.5 65.4 ± 1.7 1.21 ± 0.01 72.7 ± 0.4 65.2 ± 1.8 1.03 ± 0.03 CFC-P-F (ours) 80.1 ± 0.5 76.3 ± 1.6 1.72 ± 0.04 75.3 ± 0.4 74.6 ± 1.0 1.32 ± 0.10 Methods α=0.30α=0.30 α=0.35α=0.35 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 ICP 69.5 ± 0.8 49.3 ± 2.7 0.90 ± 0.02 64.5 ± 0.6 43.0 ± 2.1 0.78 ± 0.01 Learnt CP 69.5 ± 0.7 68.7 ± 1.3 0.97 ± 0.02 64.6 ± 0.7 63.0 ± 2.1 0.82 ± 0.02 CFC (ours) 68.4 ± 0.7 62.8 ± 2.0 0.88 ± 0.01 63.9 ± 0.7 59.2 ± 2.3 0.78 ± 0.01 CFC-P-F (ours) 70.0 ± 0.8 69.2 ± 1.3 0.99 ± 0.03 65.1 ± 0.8 63.7 ± 2.2 0.83 ± 0.02 Table 3: TriviaQA feature-map ablation at α=0.30α=0.30 (target coverage 70%70\%). Each cell reports ECR/GSC/APSS. TopK and ICP are unchanged across feature maps and remain fixed at 73.4/55.9/1.0073.4/55.9/1.00 and 69.5/49.3/0.9069.5/49.3/0.90, respectively. Setting CFC CFC-P-F Entropy-linear Φ 66.9 / 45.1 / 1.02 70.6 / 53.2 / 1.32 Max-loss-linear Φ 67.6 / 56.5 / 0.99 70.7 / 57.4 / 1.26 Chosen Φ 68.4 / 62.8 / 0.88 70.0 / 69.2 / 0.99 Table 4: GSM8K results at α=0.05α=0.05 using the first five sampled candidates per prompt and the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2] with T(X)T(X) equal to mean verifier loss. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α1-α are preferred. Appendix C reports the full sweep. Methods ECR GSC↑ APSS↓ TopK 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 ICP 95.09 ± 1.42 79.85 ± 6.53 4.73 ± 0.09 Learnt CP 94.91 ± 1.03 88.48 ± 3.05 4.01 ± 0.98 CFC (ours) 94.82 ± 0.97 88.48 ± 2.32 2.35 ± 0.43 CFC-P-F (ours) 95.24 ± 1.40 88.79 ± 3.01 4.59 ± 0.62 Table 5: GSM8K sample-budget ablation at α=0.10α=0.10 under the same mean-loss quadratic rule. The larger budget gives only a small subgroup-coverage gain, but it substantially inflates APSS. Method N=5N=5 N=20N=20 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ CFC (ours) 90.18 86.36 1.49 90.30 87.73 3.79 CFC-P-F (ours) 91.91 88.48 2.34 92.06 88.79 7.97 Table 6: VLM experiment on Flickr8k with Qwen2-VL-7B-Instruct at α=0.03α=0.03 using up to two cached candidates per image and the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2] with T(X)T(X) equal to mean verifier loss. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α=97%1-α=97\% are preferred. Method ECR APSS↓ GSC↑ TopK 96.37 ± 0.17 1.00 ± 0.00 93.23 ± 0.47 ICP 95.58 ± 0.54 1.84 ± 0.01 85.21 ± 3.14 Learnt CP 96.16 ± 0.17 1.22 ± 0.07 94.48 ± 0.26 CFC (ours) 95.81 ± 0.38 0.99 ± 0.00 93.23 ± 0.47 CFC-P-F (ours) 97.27 ± 0.21 1.42 ± 0.07 95.21 ± 0.77 5.2 Real-World Data Datasets. We evaluate on three settings. GSM8K [5] contains grade-school math word problems. TriviaQA [10] is an open-domain question-answering benchmark. To test transfer beyond text-only generators, we also include a Flickr8k vision-language experiment with Qwen2-VL-7B-Instruct as the base model. We use Llama-3-8B-Instruct for GSM8K and TriviaQA. Baselines and metrics. On real data we compare three baselines against CFC variants. TopK keeps the smallest top-K prefix whose calibration coverage reaches the target level. ICP calibrates a single global verifier threshold. Learnt CP learns a feature-conditional threshold from calibration data but omits the exact conformal correction. We again report ECR, APSS, and GSC. For ECR, the preferred value is the one closest to the target coverage 1−α1-α. APSS can fall below 11 because the method may abstain and return the empty set on some prompts when no sampled candidate passes the calibrated threshold. Implementation details. Unless otherwise stated, we use Llama-3-8B-Instruct as the base LLM with nucleus sampling (top-p=0.8p=0.8, temperature =0.7=0.7) and draw up to M=20M=20 samples per prompt. On reasoning tasks we use a separate verifier to compute V(X,y)V(X,y), e.g. Qwen2.5-Math-RM-72B (and Qwen2-VL-7B-Instruct for VLM tasks). On GSM8K, the strongest setting uses the first 55 sampled candidates per prompt, defines T(X)T(X) as the mean verifier loss across those samples, and uses the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. On Flickr8k, the strongest setting keeps up to 22 cached candidates per image, defines T(X)T(X) as the mean verifier loss across those candidates, and uses the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. On TriviaQA, the strongest setting uses a calibration-defined feature map built from answer-distribution entropy and verifier loss; Appendix C gives the exact construction. In the main paper, GSM8K, TriviaQA, and Flickr8k report CFC, and CFC-P-F. CFC truncates the accepted set after the best accepted candidate in sample order, while CFC-P-F applies the stability-based PAC adjustment to the full thresholded set. Appendix C includes the full TriviaQA, GSM8K, and Flickr8k sweeps with all four CFC variants. For GSM8K and Flickr8k, GSC is computed over five equal-frequency bins of the scalar difficulty proxy; for TriviaQA, GSC uses the two-group feature map, whose exact definition is given in Appendix C. All variants are purely post-hoc: they do not finetune the base generator or verifier. Groupwise behavior. Figure 4 complements Tables 2–6 by visualizing the same subgroup effect behind the real-data tables: a single global cutoff tends to under-cover the hardest inputs, while conditional thresholds flatten the miscoverage profile. This pattern is strongest on GSM8K, where the gain is concentrated on the hardest bins, and it remains visible on TriviaQA and Flickr8k as well. Results for TriviaQA. Table 2 shows that the chosen TriviaQA feature map gives the strongest overall tradeoff we found, and the left panel of Figure 4 shows why: the conditional rule mainly corrects the hard subset rather than trying to smooth every prompt equally. Learnt CP already improves subgroup reliability substantially over ICP, but it does so with larger prediction sets than CFC. At α=0.30α=0.30 (target coverage 70%70\%), CFC-P-F is closest to target at 70.0%70.0\%, while CFC attains the smallest prediction sets at 0.880.88 and still raises GSC from 49.3%49.3\% (ICP) to 62.8%62.8\%. The same division of labor is visible at α=0.20α=0.20, α=0.25α=0.25, and α=0.35α=0.35: the PAC variant is the most target-calibrated among our methods, while CFC is the most size-efficient. Feature-map ablation. Table 3 compares simple TriviaQA feature maps at α=0.30α=0.30. The chosen Φ yields the best subgroup reliability and the smallest APSS for CFC, and it also gives the strongest GSC for CFC-P-F while keeping target fit competitive. This is the main point of the ablation: the gains are not tied to a single scalar proxy, but the chosen feature map gives the best overall balance among target fit, subgroup reliability, and set size. Results for GSM8K. Table 4, Table 5, and the middle panel of Figure 4 show that on GSM8K a small candidate budget together with a smooth loss-based feature map gives the best tradeoff we found. At α=0.05α=0.05 (target coverage 95%95\%), all methods are reasonably close to target, but the conditional methods materially improve subgroup behavior over ICP: CFC reduces APSS from 4.734.73 to 2.352.35 while lifting GSC from 79.85%79.85\% to 88.48%88.48\%, and CFC-P-F raises the subgroup floor further to 88.79%88.79\%. Table 5 explains why the main paper keeps the smaller N=5N=5 budget: moving to N=20N=20 barely changes ECR or GSC, but it more than doubles APSS for CFC and more than triples it for CFC-P-F. Appendix C reports the complete GSM8K sweep. Results for Flickr8k. Table 6 and the right panel of Figure 4 show that the same post-hoc conformal layer transfers to a vision-language model and a different verifier. At the target coverage level 1−α=97%1-α=97\%, CFC-P-F is closest to target at 97.27%97.27\% while achieving the strongest subgroup reliability at 95.21%95.21\% GSC. CFC is the most size-efficient variant at 0.990.99 APSS, but on this easy benchmark it often collapses to a single caption and therefore gives up more target fit than the PAC rule. This still supports the transfer claim: the same conditional calibration layer remains effective when the base model is Qwen2-VL-7B-Instruct, and Appendix C shows the full small-α sweep together with a compact setting ablation. Additional Sweeps and Ablations. Appendix C reports the full TriviaQA, GSM8K, and Flickr8k target-risk sweeps, other CFC variants, and compact ablations for the GSM8K budget choice and the Flickr8k setting choice. 6 Conclusion Conditional Factuality Control replaces a single global factuality threshold with a feature-conditional one, and our results show that this post-hoc conformal layer improves subgroup reliability and often yields a better coverage–set-size tradeoff than marginal baselines across synthetic and real LLM/VLM settings. Acknowledgements This work was supported in part by the National Institutes of Health (NIH) under Grants R01HL173186 and R01HL177813, and by the National Science Foundation (NSF) under Grant No. 2306545. References [1] B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini (2024) Large language monkeys: scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Cited by: §1, §2. [2] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. (2023) Sparks of artificial general intelligence: early experiments with gpt-4. arXiv preprint arXiv:2303.12712. Cited by: §1. [3] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §2. [4] J. Cherian, I. Gibbs, and E. Candes (2024) Large language model validity via enhanced conformal prediction methods. Advances in Neural Information Processing Systems 37, p. 114812–114842. Cited by: §2. [5] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §1, §2, §5.2. [6] R. Foygel Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani (2021) The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA 10 (2), p. 455–482. Cited by: §A.3, §2, §3.2. [7] I. Gibbs, J. J. Cherian, and E. J. Candès (2024) Conformal prediction with conditional guarantees. External Links: 2305.12616, Link Cited by: §A.3, Theorem A.1, 1st item, §2, §3.2, §3.2, §4.1, §4.1. [8] A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal (2024) V-star: training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457. Cited by: §2. [9] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2025) A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems 43 (2), p. 1–55. Cited by: §1. [10] M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611. Cited by: §5.2. [11] B. Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam (2023) Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305.18404. Cited by: §1, §2. [12] C. Mohri and T. Hashimoto (2024) Language models with conformal factuality guarantees. In Proceedings of the 41st International Conference on Machine Learning, p. 36029–36047. Cited by: §2. [13] H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman (2002) Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19–23, 2002 proceedings 13, p. 345–356. Cited by: §2. [14] V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. S. Jaakkola, and R. Barzilay (2024) Conformal language modeling. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §1, §2. [15] Y. Romano, E. Patterson, and E. Candes (2019) Conformalized quantile regression. Advances in neural information processing systems 32. Cited by: §2. [16] M. Sensoy, L. Kaplan, and M. Kandemir (2018) Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems 31. Cited by: §1. [17] G. Shafer and V. Vovk (2008) A tutorial on conformal prediction. Journal of Machine Learning Research 9, p. 371–421. Cited by: §A.2, §2. [18] J. Su, J. Luo, H. Wang, and L. Cheng (2024-11) API is enough: conformal prediction for large language models without logit-access. In Findings of the Association for Computational Linguistics: EMNLP 2024, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 979–995. External Links: Link, Document Cited by: §1, §2. [19] V. Vovk, A. Gammerman, and G. Shafer (2005) Algorithmic learning in a random world. Vol. 29, Springer. Cited by: §A.2, §1, §2, §5.1. [20] V. Vovk (2012) Conditional validity of inductive conformal predictors. In Asian conference on machine learning, p. 475–490. Cited by: §A.3, §2, §3.2. [21] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus (2022) Emergent abilities of large language models. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, Link Cited by: §1. [22] K. Ye, T. Chen, H. Wei, and L. Zhan (2024) Uncertainty regularized evidential regression. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 16460–16468. External Links: Document Cited by: §1. [23] K. Ye, H. Tang, S. Dai, I. Fortel, P. M. Thompson, R. S. Mackin, A. D. Leow, H. Huang, and L. Zhan (2025) BPEN: brain posterior evidential network for trustworthy brain imaging analysis. Neural Networks 183, p. 106943. External Links: Document Cited by: §1. [24] K. Ye, H. Tang, S. Dai, L. Guo, J. Y. Liu, Y. Wang, A. D. Leow, P. M. Thompson, H. Huang, and L. Zhan (2023) Bidirectional mapping with contrastive learning on multimodal neuroimaging data. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, Lecture Notes in Computer Science, Vol. 14222, p. 138–148. External Links: Document Cited by: §1. Appendix A Extended Background A.1 Conformal Factuality Let X∈X be a prompt and let π:→Δ()π:X→ (Y) denote a fixed generator over completions. At inference time, we repeatedly draw candidates Y∼π(⋅∣X)Y π(· X) and seek a prediction set that contains at least one correct answer with high probability: ℙ(∃y∈C^α(XN+1):A(XN+1,y)=1)≥1−α,P\! (∃ y∈ C_α(X_N+1):A(X_N+1,y)=1 )≥ 1-α, (A.1) where α is the target error level and A(X,y)∈0,1A(X,y)∈\0,1\ indicates whether candidate y is correct for prompt X. To achieve this guarantee, each sampled candidate is evaluated with a verifier score V:×→[0,1]V:X×Y→[0,1]. In this paper, smaller verifier scores are better, so a calibrated acceptance rule amounts to choosing a threshold and retaining all candidates with score below it. A.2 Inductive Conformal Prediction Split conformal prediction transforms the outputs of a black-box model into valid prediction sets using a held-out calibration set [19, 17]. Given calibration pairs (Xi,Yi)i=1N\(X_i,Y_i)\_i=1^N and a nonconformity score function s:×→ℝs:X×Y , one computes calibration scores Vi=s(Xi,Yi)V_i=s(X_i,Y_i), sorts them, and forms the empirical quantile Q^1−α:=V(⌈(N+1)(1−α)⌉). Q_1-α:=V_( (N+1)(1-α) ). (A.2) The corresponding split-conformal prediction set is C^1−α(XN+1):=y∈:s(XN+1,y)≤Q^1−α, C_1-α(X_N+1):=\y :s(X_N+1,y)≤ Q_1-α\, (A.3) which satisfies marginal coverage ℙ(YN+1∈C^1−α(XN+1))≥1−αP\! (Y_N+1∈ C_1-α(X_N+1) )≥ 1-α (A.4) under exchangeability. In the conformal-factuality setting, this corresponds to using one global acceptance threshold for all prompts. A.3 Conditional Conformal Prediction Marginal coverage holds only on average over the prompt distribution. For LLMs, this can hide severe heterogeneity: easy prompts may be over-covered while hard prompts are under-covered. The ideal pointwise conditional guarantee ℙ(YN+1∈C^(XN+1)∣XN+1=x)≥1−αP\! (Y_N+1∈ C(X_N+1) X_N+1=x )≥ 1-α (A.5) is impossible to achieve exactly in finite samples without strong assumptions [20, 6]. Following Gibbs et al. [7], one can rewrite exact conditional coverage as an infinite family of weighted marginal constraints: ℙ(YN+1∈C^(XN+1)∣XN+1)=1−α⇔[f(XN+1)(YN+1∈C^(XN+1)−(1−α))]=0P\! (Y_N+1∈ C(X_N+1) X_N+1 )=1-α \! [f(X_N+1) ( 1\Y_N+1∈ C(X_N+1)\-(1-α) ) ]=0 (A.6) for all measurable f. Their relaxation replaces the class of all measurable functions with a chosen function class ℱF: [f(XN+1)(YN+1∈C^(XN+1)−(1−α))]=0,for all f∈ℱ.E\! [f(X_N+1) ( 1\Y_N+1∈ C(X_N+1)\-(1-α) ) ]=0, all f . (A.7) Taking ℱ=1F=\1\ recovers marginal conformal prediction, while ℱ=Φ(⋅)⊤β:β∈ℝdF=\ (·) β:β ^d\ yields a finite-dimensional feature-conditional target. For this linear class, Gibbs et al. define an augmented quantile-regression estimator using the pinball loss ρ1−α(u)=u((1−α)−u<0). _1-α(u)=u ((1-α)- 1\u<0\ ). (A.8) Given calibration scores (Xi,Si)i=1N\(X_i,S_i)\_i=1^N and a fresh candidate score S, the augmented fit is g^S:=argming∈ℱ1N+1∑i=1Nρ1−α(Si−g(Xi))+1N+1ρ1−α(S−g(XN+1)). g_S:= *arg\,min_g 1N+1 _i=1^N _1-α\! (S_i-g(X_i) )\;+\; 1N+1 _1-α\! (S-g(X_N+1) ). (A.9) The resulting prediction rule keeps labels whose score does not exceed the fitted value at the same score: C^(XN+1):=y:S(XN+1,y)≤g^S(XN+1,y)(XN+1). C(X_N+1):= \y:S(X_N+1,y)≤ g_S(X_N+1,y)(X_N+1) \. (A.10) Theorem A.1 (Gibbs et al. [7], Theorem 2). Let ℱ=Φ(⋅)⊤β:β∈ℝdF=\ (·) β:β ^d\ be a linear class over the basis Φ:→ℝd :X ^d. Then for any non-negative f∈ℱf with [f(X)]>0E[f(X)]>0, the prediction rule above satisfies ℙf(YN+1∈C^(XN+1))≥1−α.P_f\! (Y_N+1∈ C(X_N+1) )≥ 1-α. (A.11) Our method is a conformal-factuality instantiation of this framework, with the latent success score S(X)S(X) replacing the standard label-wise nonconformity score and a fixed-point construction used to obtain the deployed threshold. Appendix B Proof of Results from the Main Paper B.1 Proof of Theorem 4.1 Proof. Recall the success score S(X):=infλ∈[0,1]:ℓλ(X)=0=infV(X,y):y∈C(X),A(X,y)=1,S(X)\;:=\; \λ∈[0,1]:\ _λ(X)=0\\;=\; \V(X,y):y∈ C(X),\ A(X,y)=1\, so that for any threshold λ(X)∈[0,1]λ(X)∈[0,1], ∃y∈C^α(X):A(X,y)=1⟺S(X)≤λ(X). \∃\,y∈ C_α(X):A(X,y)=1 \ \S(X)≤λ(X) \. (B.1) Thus, if we show that ℙf(S(XN+1)≤λ^α(XN+1))≥ 1−αP_f (S(X_N+1)≤ λ_α(X_N+1) )\ ≥\ 1-α (B.2) for every non-negative f∈ℱf with [f(X)]>0E[f(X)]>0, the result follows immediately from (B.1). Throughout, for any such f we define the f-reweighted probability of an event E by ℙf(E):=[f(X) 1(X,S)∈E][f(X)].P_f(E):= E [f(X)\, 1\(X,S)∈ E\ ]E[f(X)]. Let τ:=1−ατ:=1-α and recall the pinball loss ρτ(u)=u(τ−u<0). _τ(u)\;=\;u (τ- 1\u<0\ ). Let βS _S be the augmented quantile-regression minimizer from Eq. (B.5) for the realized calibration set and the fresh pair (X,S)(X,S), and define g(S):=Φ(X)⊤βSg(S):= (X) _S. Then for any nonnegative f∈ℱf with [f(X)]>0E[f(X)]>0, [f(X) 1S≤g(S)]≥(1−α)[f(X)].E [f(X)\, 1\S≤ g(S)\ ]\ ≥\ (1-α)\,E[f(X)]. Fix γ∈ℝdγ ^d with f(X)=Φ(X)⊤γ≥0f(X)= (X) γ≥ 0 for all X and consider ψN(ε)=1N+1∑i=1Nρ1−α(Si−Φ(Xi)⊤(βS+εγ))+1N+1ρ1−α(S−Φ(X)⊤(βS+εγ)). _N( )= 1N+1 _i=1^N _1-α (S_i- (X_i) ( _S+ γ) )+ 1N+1 _1-α (S- (X) ( _S+ γ) ). By convexity and optimality of βS _S, ψN′(0+)≥0 _N (0^+)≥ 0. This gives ψN′(0+)=−1N+1[∑i=1N((1−α)−Si≤Φ(Xi)⊤βS)f(Xi)+((1−α)−S≤Φ(X)⊤βS)f(X)]≥0. _N (0^+)=- 1N+1\! [ _i=1^N\! ((1-α)- 1\S_i≤ (X_i) _S\ )f(X_i)+ ((1-α)- 1\S≤ (X) _S\ )f(X) ]\!≥ 0. Rearranging, 1N+1∑i=1NSi≤Φ(Xi)⊤βSf(Xi)+1N+1S≤Φ(X)⊤βSf(X)≥(1−α)1N+1∑i=1Nf(Xi)+(1−α)1N+1f(X). 1N+1 _i=1^N\! 1\S_i≤ (X_i) _S\f(X_i)+ 1N+1 1\S≤ (X) _S\f(X)\ ≥\ (1-α)\, 1N+1 _i=1^Nf(X_i)\;+\;(1-α)\, 1N+1f(X). Now take expectation over all N+1N+1 exchangeable draws. By exchangeability, each of the N+1N+1 summands on each side has the same distribution, so [f(X) 1S≤Φ(X)⊤βS]≥(1−α)[f(X)].E [f(X)\, 1\S≤ (X) _S\ ]\ ≥\ (1-α)\,E[f(X)]. Since g(S)=Φ(X)⊤βSg(S)= (X) _S, the claim follows. Recall the CFC threshold λ^α(X):=supt∈[0,1]:t≤g(t),g(t)=Φ(X)⊤βt. λ_α(X)\ :=\ \\,t∈[0,1]:\ t≤ g(t)\,\, g(t)= (X) _t. By definition of the supremum, for any realized S∈[0,1]S∈[0,1] we have S≤g(S)≤ 1S≤λ^α(X). 1\S≤ g(S)\\ ≤\ 1\S≤ λ_α(X)\. (B.3) Multiplying (B.3) by f(X)≥0f(X)≥ 0 and taking expectations, [f(X) 1S≤λ^α(X)]≥[f(X) 1S≤g(S)]≥(1−α)[f(X)].E [f(X)\, 1\S≤ λ_α(X)\ ]\ ≥\ E [f(X)\, 1\S≤ g(S)\ ]\ ≥\ (1-α)\,E[f(X)]. Dividing by [f(X)]>0E[f(X)]>0 gives ℙf(S≤λ^α(X))≥ 1−α.P_f (S≤ λ_α(X) )\ ≥\ 1-α. By definition of the success score and the prediction set, S(X)≤λ^α(X)⟺∃y∈C^α(X):A(X,y)=1.\\,S(X)≤ λ_α(X)\,\ \\,∃\,y∈ C_α(X):A(X,y)=1\,\. Therefore, ℙf(∃y∈C^α(X):A(X,y)=1)=ℙf(S(X)≤λ^α(X))≥ 1−α,P_f\! (∃\,y∈ C_α(X):A(X,y)=1 )\;=\;P_f\! (S(X)≤ λ_α(X) )\;≥\;1-α, as claimed. ∎ B.2 Proof of Theorem 4.2 Proof. Write the calibration set as cal=(Xi,Si)i=1N,(XN+1,SN+1)∼i.i.d. as (Xi,Si),D_cal=\(X_i,S_i)\_i=1^N, (X_N+1,S_N+1) .i.d. as (X_i,S_i), and recall the success indicator Z(x,s):= 1s≤λ^α(x),Z(x,s)\;:=\; 1 \s≤ λ_α(x) \, so that Z(X,S)=1Z(X,S)=1 iff there exists a correct candidate in C^α(X) C_α(X). Define Q(cal):=ℙ(Z(X,S)=1∣cal)=[Z(X,S)∣cal].Q(D_cal)\;:=\;P\! (Z(X,S)=1 _cal )\;=\;E\! [Z(X,S) _cal ]. By Theorem 4.1 applied with f≡1f≡ 1 and the law of total expectation, ℙ(Z(X,S)=1)=calQ(cal)≥ 1−α.P (Z(X,S)=1 )=E_D_cal\,Q(D_cal)\;≥\;1-α. Thus cal[Q(cal)]≥ 1−α.E_D_cal [Q(D_cal) ]\;≥\;1-α. (B.4) We now show that Q(cal)Q(D_cal) is Lipschitz in the calibration set, with sensitivity of order 1/N1/N to replacing one calibration pair. Let =(Xi,Si)i=1ND=\(X_i,S_i)\_i=1^N and ′=(Xi′,Si′)i=1ND =\(X_i ,S_i )\_i=1^N differ only in the k-th pair. For a fixed test prompt x and a candidate success score s∈[0,1]s∈[0,1], consider the ridge-regularized augmented quantile regression objective β↦1N+1∑i=1Nρ1−α(Si−Φ(Xi)⊤β)+1N+1ρ1−α(s−Φ(x)⊤β)+λ2‖β‖22.β\; \; 1N+1 _i=1^N _1-α\! (S_i- (X_i) β )+ 1N+1 _1-α\! (s- (x) β )+ λ2\|β\|_2^2. (B.5) Let βs(,x) _s(D,x) and βs(′,x) _s(D ,x) denote the unique minimizers of (B.5) with D and ′D respectively, and define g(x,s):=Φ(x)⊤βs(,x),g′(x,s):=Φ(x)⊤βs(′,x).g_D(x,s)\ :=\ (x) _s(D,x), g_D (x,s)\ :=\ (x) _s(D ,x). By Assumption (1), ‖Φ(Xi)‖2≤R\| (X_i)\|_2≤ R almost surely, and the pinball loss is 1-Lipschitz. Hence each sample loss is R-Lipschitz in β. By Assumption (2), adding the ridge term makes the objective (B.5) strongly convex. Standard stability results for regularized ERM imply that replacing one data point perturbs the minimizer by at most ‖βs(,x)−βs(′,x)‖2≤C1RλN \| _s(D,x)- _s(D ,x) \|_2\;≤\; C_1Rλ N for some universal constant C1>0C_1>0, uniformly over s∈[0,1]s∈[0,1] and x. Using Assumption (1) again, |g(x,s)−g′(x,s)|≤‖Φ(x)‖2‖βs(,x)−βs(′,x)‖2≤C1R2λN. |g_D(x,s)-g_D (x,s) |≤\| (x)\|_2\, \| _s(D,x)- _s(D ,x) \|_2≤ C_1R^2λ N. Thus there is a constant C2>0C_2>0 such that supx∈sups∈[0,1]|g(x,s)−g′(x,s)|≤C2N. _x _s∈[0,1] |g_D(x,s)-g_D (x,s) |\;≤\; C_2N. (B.6) The CFC threshold at x is λ^α()(x):=sups∈[0,1]:s≤g(x,s), λ_α^(D)(x)\;:=\; \\,s∈[0,1]:\ s≤ g_D(x,s)\,\, and similarly for λ^α(′)(x) λ_α^(D )(x). Under the fixed-point construction in Eq. (4.3), this threshold is a Lipschitz functional of s↦g(x,s)s g_D(x,s) in the uniform norm: there exists a constant C3>0C_3>0 such that supx∈|λ^α()(x)−λ^α(′)(x)|≤C3N. _x | λ_α^(D)(x)- λ_α^(D )(x) |\;≤\; C_3N. (B.7) For a fresh test pair (X,S)(X,S), Q()=ℙ(S≤λ^α()(X)∣)=[FS∣X(λ^α()(X))|],Q(D)=P (S≤ λ_α^(D)(X) )=E\! [F_S X ( λ_α^(D)(X) )\; |\;D ], where FS∣X=x(⋅)F_S X=x(·) is the conditional CDF of S given X=xX=x. By Assumption (3), for each x the map t↦FS∣X=x(t)t F_S X=x(t) is L-Lipschitz on [0,1][0,1]. Combining this with (B.7) yields |Q()−Q(′)|≤Lsupx∈|λ^α()(x)−λ^α(′)(x)|≤C4N, |Q(D)-Q(D ) |≤ L\, _x | λ_α^(D)(x)- λ_α^(D )(x) |≤ C_4N, where C4:=LC3C_4:=LC_3. Thus Q(cal)Q(D_cal) satisfies bounded differences with constants ci=C4/Nc_i=C_4/N for i=1,…,Ni=1,…,N. By McDiarmid’s inequality, for any ε>0 >0, ℙ(Q(cal)≤cal[Q(cal)]−ε)≤exp(−2Nε2C42).P\! (Q(D_cal) _D_cal [Q(D_cal) ]- )≤ \! (- 2N ^2C_4^2 ). Given δ∈(0,1)δ∈(0,1), set εN(δ):=C42Nlog1δ=O(log(1/δ)N). _N(δ):= C_4 2N 1δ=O\! ( (1/δ)N ). Then with probability at least 1−δ1-δ over calD_cal, Q(cal)≥cal[Q(cal)]−εN(δ).Q(D_cal)\;≥\;E_D_cal [Q(D_cal) ]- _N(δ). (B.8) Combining (B.8) with (B.4) yields Q(cal)≥ 1−α−εN(δ)Q(D_cal)\;≥\;1-α- _N(δ) with probability at least 1−δ1-δ, i.e. ℙ(Z(X,S)=1|cal)≥ 1−α−εN(δ),P\! (Z(X,S)=1\; |\;D_cal )\;≥\;1-α- _N(δ), Finally, we have αeff=max0,α−εN(δ)=α−εN(δ) _eff= \0,α- _N(δ)\=α- _N(δ) (the slack is a small term) from algorithm 2, then, ℙ(S≤λ^αeff(X)∣cal)≥1−αeff−εN(δ)=1−α,P\! (S≤ λ_ _eff(X) _cal )≥ 1- _eff- _N(δ)=1-α, ∎ B.3 Proof of Proposition 4.3 Proof. Let u0:=1−α,uλ¯(t):=Ft(λ¯).u_0:=1-α, u_ λ(t):=F_t( λ). Since ℙ(S≤λ¯)=1−αP(S≤ λ)=1-α, we have [uλ¯(T)]=[FT(λ¯)]=1−α=u0.E [u_ λ(T) ]=E [F_T( λ) ]=1-α=u_0. By Assumption 2, the map t↦uλ¯(t)t u_ λ(t) is nonincreasing. Now, [GX(λ⋆(X))]=[GT(qα(T))]=[CT(u0)],E [G_X(λ (X)) ]=E [G_T(q_α(T)) ]=E [C_T(u_0) ], because qα(t)=Ft−1(u0)q_α(t)=F_t^-1(u_0), and also [GX(λ¯)]=[GT(λ¯)]=[CT(uλ¯(T))].E [G_X( λ) ]=E [G_T( λ) ]=E [C_T(u_ λ(T)) ]. Set a(t):=uλ¯(t)−u0.a(t):=u_ λ(t)-u_0. Then a(t)a(t) is nonincreasing in t and [a(T)]=0.E[a(T)]=0. By convexity of Ct(⋅)C_t(·), for each t, Ct(uλ¯(t))≥Ct(u0)+∂uCt(u0)(uλ¯(t)−u0).C_t(u_ λ(t))≥ C_t(u_0)+ _uC_t(u_0)\, (u_ λ(t)-u_0 ). Hence Ct(uλ¯(t))−Ct(u0)≥m(t)a(t),m(t):=∂uCt(u0).C_t(u_ λ(t))-C_t(u_0)≥ m(t)\,a(t), m(t):= _uC_t(u_0). By Assumption 4, m(t)m(t) is nonincreasing in t. Since both m(t)m(t) and a(t)a(t) are nonincreasing, Chebyshev’s rearrangement inequality gives [m(T)a(T)]≥[m(T)][a(T)]=0.E[m(T)a(T)] [m(T)]\,E[a(T)]=0. Taking expectations in the previous convexity bound yields [CT(uλ¯(T))]−[CT(u0)]≥[m(T)a(T)]≥0.E [C_T(u_ λ(T)) ]-E [C_T(u_0) ] [m(T)a(T)]≥ 0. Therefore, [GX(λ¯)]=[CT(uλ¯(T))]≥[CT(u0)]=[GX(λ⋆(X))].E [G_X( λ) ]=E [C_T(u_ λ(T)) ] [C_T(u_0) ]=E [G_X(λ (X)) ]. For strictness, if ℙ(qα(X)≠λ¯α)>0P(q_α(X)≠ λ_α)>0, then by strict monotonicity of each FtF_t we also have ℙ(FT(λ¯α)≠1−α)>0.P (F_T( λ_α)≠ 1-α )>0. If Ct(⋅)C_t(·) is strictly convex for almost every t, then the supporting-line inequality is strict on a set of positive probability, which implies [GX(λ⋆(X))]<[GX(λ¯α)].E [G_X(λ (X)) ]<E [G_X( λ_α) ]. ∎ B.4 Proof of Theorem 4.4 Proof. Let T=ψ(X)T=ψ(X) and set u0:=1−α,λ⋆(X)=qα(T)=FT−1(u0).u_0:=1-α, λ (X)=q_α(T)=F_T^-1(u_0). By Proposition 4.3, [GX(λ⋆(X))]≤[GX(λ¯α)],E [G_X(λ (X)) ] [G_X( λ_α) ], (B.9) with strict inequality under the additional strictness assumptions stated there. It remains to show that [GX(λ^α,N(X))]⟶[GX(λ⋆(X))].E [G_X( λ_α,N(X)) ] [G_X(λ (X)) ]. Fix t. Since FtF_t is continuous and strictly increasing on [0,1][0,1], we have Ft−1(Ft(λ))=λfor all λ∈[0,1].F_t^-1(F_t(λ))=λ all λ∈[0,1]. By definition, Ct(u)=Gt(Ft−1(u)),C_t(u)=G_t(F_t^-1(u)), so for every λ∈[0,1]λ∈[0,1], Gt(λ)=Ct(Ft(λ)).G_t(λ)=C_t(F_t(λ)). Now u↦Ct(u)u C_t(u) is convex and differentiable on (0,1)(0,1) by Proposition 4.3, hence continuous on (0,1)(0,1). Since Ft(λ⋆(t))=Ft(Ft−1(u0))=u0=1−α∈(0,1),F_t(λ (t))=F_t (F_t^-1(u_0) )=u_0=1-α∈(0,1), it follows that λ↦Gt(λ)λ G_t(λ) is continuous at λ⋆(t)λ (t). Now fix x∈x . By the assumed uniform consistency, |λ^α,N(x)−λ⋆(x)|≤supz∈|λ^α,N(z)−λ⋆(z)|→0, | λ_α,N(x)-λ (x) |≤ _z | λ_α,N(z)-λ (z) | p0, so λ^α,N(x)→λ⋆(x). λ_α,N(x) pλ (x). Since Gx(⋅)G_x(·) is continuous at λ⋆(x)λ (x), the continuous mapping theorem yields Gx(λ^α,N(x))→Gx(λ⋆(x)).G_x ( λ_α,N(x) ) pG_x (λ (x) ). For ε>0 >0, define pN,ε(x):=ℙcal(|Gx(λ^α,N(x))−Gx(λ⋆(x))|>ε).p_N, (x):=P_D_cal\! ( |G_x ( λ_α,N(x) )-G_x (λ (x) ) |> ). Then pN,ε(x)→0p_N, (x)→ 0 for every fixed x, and 0≤pN,ε(x)≤10≤ p_N, (x)≤ 1. Since the fresh test point X is independent of the calibration sample, ℙ(|GX(λ^α,N(X))−GX(λ⋆(X))|>ε)=X[pN,ε(X)]⟶0P\! ( |G_X ( λ_α,N(X) )-G_X (λ (X) ) |> )=E_X [p_N, (X) ] 0 by dominated convergence. Therefore, GX(λ^α,N(X))→GX(λ⋆(X)).G_X ( λ_α,N(X) ) pG_X (λ (X) ). Moreover, 0≤GX(λ^α,N(X))≤1,0≤GX(λ⋆(X))≤1,0≤ G_X ( λ_α,N(X) )≤ 1, 0≤ G_X (λ (X) )≤ 1, so the sequence is uniformly integrable. Hence convergence in probability upgrades to convergence in L1L^1, and therefore limN→∞[GX(λ^α,N(X))]=[GX(λ⋆(X))]. _N→∞E [G_X( λ_α,N(X)) ]=E [G_X(λ (X)) ]. Combining this with (B.9) proves limN→∞[GX(λ^α,N(X))]≤[GX(λ¯α)], _N→∞E [G_X( λ_α,N(X)) ] [G_X( λ_α) ], with strict inequality under the additional strictness assumptions from Proposition 4.3. Finally, conditional on X and a threshold λ, each of the M sampled candidates is accepted with probability GX(λ)G_X(λ), so [|C^α,N(X)||X,λ^α,N(X)]=MGX(λ^α,N(X)).E [\,| C_α,N(X)|\, |\,X, λ_α,N(X) ]=M\,G_X ( λ_α,N(X) ). Taking expectations gives [|C^α,N(X)|]=M[GX(λ^α,N(X))].E [\,| C_α,N(X)|\, ]=M\,E [G_X( λ_α,N(X)) ]. Passing to the limit yields limN→∞[|C^α,N(X)|]=M[GX(λ⋆(X))]≤M[GX(λ¯α)], _N→∞E [| C_α,N(X)| ]=M\,E [G_X(λ (X)) ]≤ M\,E [G_X( λ_α) ], with strict inequality under the same additional assumptions. ∎ Appendix C Additional experiments C.1 Synthetic Data Setup details. All synthetic appendix results use the same clean synthetic generator as the main text. Each prompt has a scalar difficulty variable T∈[0,1]T∈[0,1], we draw M candidates, and define the prompt-level latent success score as S(X)=minVj:Aj=1S(X)= \V_j:A_j=1\, with S(X)=1S(X)=1 if no sampled candidate is correct. Unless varied explicitly in the ablations, we use Ncal=Ntest=10,000N_cal=N_test=10,000 and M=50M=50. Throughout the synthetic appendix, ECR and GSC use the ground-truth correctness labels rather than the surrogate event S(X)≤λ^(X)S(X)≤ λ(X). The main synthetic comparisons use 10 equal-frequency bins for GSC, the threshold-adaptation plot below uses 5 bins for readability, and CFC-PAC uses the same stability-mode adjustment with δ=0.90δ=0.90 as in the main synthetic experiments. Threshold adaptation versus prompt difficulty. Figure 5 visualizes the learned threshold as a function of prompt difficulty in the updated synthetic run at α=0.10α=0.10 using 5 difficulty bins. As expected, CFC assigns stricter thresholds to easy prompts and looser thresholds to hard prompts. This is exactly the adaptive behavior that a single global-threshold baseline cannot express. Figure 5: Learned threshold λ^α(X) λ_α(X) versus prompt difficulty in the updated synthetic run (α=0.10α=0.10, 5 bins). Easy prompts receive stricter thresholds, while harder prompts receive looser thresholds, explaining the improved group-wise coverage of CFC relative to global-threshold baselines. Full target-risk sweep. Table 7 reports the full synthetic sweep across target error rates. As above, ECR and GSC use ground-truth correctness labels, and CFC-PAC uses the stability-mode adjustment with δ=0.90δ=0.90 throughout. We include Learnt CP in the full sweep to separate gains from learning a better threshold from gains due to exact conditional conformalization. Table 7: Results at different target error rates α. Methods α=0.10α=0.10 α=0.15α=0.15 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 90.6 ± 0.1 58.2 ± 1.3 16.00 ± 0.00 85.0 ± 1.6 44.9 ± 2.3 10.80 ± 0.98 ICP 90.2 ± 0.2 57.4 ± 1.4 16.71 ± 0.23 85.3 ± 0.6 46.5 ± 1.4 12.11 ± 0.28 Learnt CP 90.2 ± 0.3 84.3 ± 0.5 15.72 ± 0.15 85.2 ± 0.6 77.4 ± 0.6 12.44 ± 0.17 CFC (ours) 90.3 ± 0.5 88.7 ± 0.7 15.53 ± 0.12 85.2 ± 0.6 82.7 ± 0.8 12.42 ± 0.09 CFC-PAC (ours) 90.8 ± 0.6 89.1 ± 0.5 15.87 ± 0.16 85.6 ± 0.6 83.4 ± 1.1 12.66 ± 0.07 Methods α=0.20α=0.20 α=0.25α=0.25 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 79.7 ± 0.2 36.5 ± 0.9 8.00 ± 0.00 73.6 ± 0.2 29.4 ± 1.2 6.00 ± 0.00 ICP 80.3 ± 0.7 37.9 ± 1.6 9.32 ± 0.26 75.4 ± 0.8 31.6 ± 1.4 7.43 ± 0.21 Learnt CP 80.2 ± 0.6 71.2 ± 0.9 10.31 ± 0.09 75.4 ± 0.8 65.1 ± 1.6 8.72 ± 0.13 CFC (ours) 80.2 ± 0.6 77.4 ± 0.7 10.39 ± 0.07 75.3 ± 0.8 72.1 ± 1.2 8.80 ± 0.10 CFC-PAC (ours) 80.6 ± 0.6 78.1 ± 0.6 10.53 ± 0.07 75.7 ± 0.8 72.4 ± 1.2 8.89 ± 0.10 Methods α=0.30α=0.30 α=0.35α=0.35 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 73.6 ± 0.2 29.4 ± 1.2 6.00 ± 0.00 64.2 ± 0.3 21.7 ± 1.4 4.00 ± 0.00 ICP 70.5 ± 0.8 27.1 ± 1.5 6.00 ± 0.19 65.6 ± 0.7 23.1 ± 1.0 4.91 ± 0.16 Learnt CP 70.4 ± 0.9 59.6 ± 1.7 7.37 ± 0.12 65.5 ± 1.0 54.7 ± 1.6 6.32 ± 0.14 CFC (ours) 70.4 ± 0.9 66.5 ± 1.4 7.53 ± 0.11 65.5 ± 0.8 61.3 ± 1.8 6.48 ± 0.10 CFC-PAC (ours) 70.7 ± 0.9 66.8 ± 1.3 7.60 ± 0.11 65.8 ± 0.9 61.5 ± 1.9 6.55 ± 0.11 Methods α=0.40α=0.40 α=0.45α=0.45 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 64.2 ± 0.3 21.7 ± 1.4 4.00 ± 0.00 64.2 ± 0.3 21.7 ± 1.4 4.00 ± 0.00 ICP 60.8 ± 0.8 19.4 ± 0.7 4.06 ± 0.13 55.6 ± 1.0 16.5 ± 0.9 3.32 ± 0.13 Learnt CP 60.4 ± 0.9 49.6 ± 1.8 5.40 ± 0.13 55.7 ± 1.0 45.5 ± 1.6 4.66 ± 0.10 CFC (ours) 60.5 ± 0.7 56.4 ± 1.7 5.58 ± 0.08 55.6 ± 0.8 51.9 ± 1.5 4.83 ± 0.10 CFC-PAC (ours) 60.9 ± 0.7 56.6 ± 1.6 5.62 ± 0.09 56.0 ± 0.9 52.3 ± 1.5 4.88 ± 0.11 Sensitivity to calibration size and sampling budget. Table 8 reports a representative synthetic ablation of CFC and CFC-PAC as we vary the number of calibration points NcalN_cal and the sampling budget M, using the same 10-bin setting as the main synthetic comparison. For consistency with the main synthetic experiments, CFC-PAC uses the same stability-mode adjustment with δ=0.90δ=0.90, and all reported coverages are true label-based coverages. Figure 6 shows the corresponding group-level miscoverage profile. Table 8: Ablation of CFC vs. CFC-PAC on synthetic data, with 10 bins. CFC CFC-PAC NcalN_cal M True cov. Mean set True cov. Mean set 2000 50 0.799 ± 0.006 10.60 ± 0.35 0.808 ± 0.005 10.92 ± 0.30 2000 100 0.809 ± 0.012 9.77 ± 0.31 0.819 ± 0.012 10.15 ± 0.29 2000 150 0.796 ± 0.006 9.03 ± 0.43 0.807 ± 0.006 9.37 ± 0.43 5000 50 0.801 ± 0.004 10.45 ± 0.12 0.808 ± 0.004 10.69 ± 0.16 5000 100 0.803 ± 0.008 9.52 ± 0.19 0.809 ± 0.007 9.74 ± 0.15 5000 150 0.802 ± 0.006 9.13 ± 0.26 0.808 ± 0.007 9.35 ± 0.27 10000 50 0.802 ± 0.006 10.39 ± 0.07 0.806 ± 0.006 10.53 ± 0.07 10000 100 0.803 ± 0.005 9.54 ± 0.06 0.808 ± 0.006 9.69 ± 0.09 10000 150 0.799 ± 0.008 9.15 ± 0.15 0.803 ± 0.008 9.29 ± 0.15 Figure 6: Group miscoverage for CFC and CFC-PAC, with 10 bins. C.2 Real-World Data C.2.1 TriviaQA Full target-risk sweep. For the chosen TriviaQA feature map, we compute the rank-normalized answer-distribution entropy Tent(X)T_ent(X) and the rank-normalized maximum verifier loss Tloss(X)T_loss(X) on the calibration split, then assign a prompt to the hard group when maxTent(X),Tloss(X)≥q0.925 \T_ent(X),T_loss(X)\≥ q_0.925, where q0.925q_0.925 is the calibration 92.592.5th percentile of that combined score. Table 9 reports the full TriviaQA sweep under this chosen feature map. The full table makes the main-paper tradeoff more explicit: CFC is the most size-efficient of our methods, while CFC-PAC-FULL is the strongest at hitting the target coverage with a higher subgroup floor. Table 9: Full TriviaQA sweep across target error rates α under the calibration-defined split maxTent(X),Tloss(X)≥q0.925 \T_ent(X),T_loss(X)\≥ q_0.925. APSS below 11 indicates that the method abstains and returns the empty set on some prompts. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α1-α are preferred. Methods α=0.20α=0.20 α=0.25α=0.25 α=0.30α=0.30 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 81.6 ± 0.3 69.7 ± 1.5 1.75 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 ICP 80.0 ± 0.5 65.1 ± 1.5 1.43 ± 0.02 74.9 ± 0.3 56.7 ± 1.9 1.08 ± 0.01 69.5 ± 0.8 49.3 ± 2.7 0.90 ± 0.02 Learnt CP 79.6 ± 0.5 76.3 ± 1.6 1.69 ± 0.04 74.7 ± 0.4 74.0 ± 1.1 1.22 ± 0.03 69.5 ± 0.7 68.7 ± 1.3 0.97 ± 0.02 CFC (ours) 76.2 ± 0.5 65.4 ± 1.7 1.21 ± 0.01 72.7 ± 0.4 65.2 ± 1.8 1.03 ± 0.03 68.4 ± 0.7 62.8 ± 2.0 0.88 ± 0.01 CFC-PAC (ours) 76.4 ± 0.5 65.4 ± 1.7 1.22 ± 0.01 73.1 ± 0.5 65.3 ± 1.8 1.05 ± 0.04 68.8 ± 0.7 62.9 ± 2.1 0.89 ± 0.02 CFC-FULL (ours) 79.6 ± 0.5 76.3 ± 1.6 1.69 ± 0.04 74.8 ± 0.3 74.1 ± 1.1 1.26 ± 0.09 69.6 ± 0.7 68.9 ± 1.1 0.97 ± 0.02 CFC-PAC-FULL (ours) 80.1 ± 0.5 76.3 ± 1.6 1.72 ± 0.04 75.3 ± 0.4 74.6 ± 1.0 1.32 ± 0.10 70.0 ± 0.8 69.2 ± 1.3 0.99 ± 0.03 Methods α=0.35α=0.35 α=0.40α=0.40 α=0.45α=0.45 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 73.4 ± 0.3 55.9 ± 1.6 1.00 ± 0.00 ICP 64.5 ± 0.6 43.0 ± 2.1 0.78 ± 0.01 59.9 ± 0.9 36.1 ± 3.1 0.70 ± 0.01 54.8 ± 0.9 29.6 ± 2.3 0.62 ± 0.01 Learnt CP 64.6 ± 0.7 63.0 ± 2.1 0.82 ± 0.02 59.9 ± 0.8 58.2 ± 2.4 0.72 ± 0.01 55.2 ± 1.0 53.8 ± 1.7 0.65 ± 0.01 CFC (ours) 63.9 ± 0.7 59.2 ± 2.3 0.78 ± 0.01 59.5 ± 0.9 55.8 ± 2.7 0.70 ± 0.01 54.9 ± 1.0 52.3 ± 2.4 0.63 ± 0.01 CFC-PAC (ours) 64.3 ± 0.7 59.5 ± 2.4 0.78 ± 0.01 59.9 ± 0.8 56.4 ± 2.9 0.70 ± 0.01 55.4 ± 1.0 52.8 ± 2.6 0.64 ± 0.01 CFC-FULL (ours) 64.7 ± 0.7 63.2 ± 2.0 0.82 ± 0.02 59.9 ± 0.9 58.4 ± 2.2 0.72 ± 0.01 55.2 ± 1.0 53.8 ± 1.7 0.65 ± 0.01 CFC-PAC-FULL (ours) 65.1 ± 0.8 63.7 ± 2.2 0.83 ± 0.02 60.4 ± 0.8 59.0 ± 2.2 0.73 ± 0.01 55.7 ± 1.0 54.3 ± 1.9 0.65 ± 0.01 C.2.2 GSM8K Full target-risk sweep. Table 10 reports the full GSM8K target-risk sweep for the chosen setting from the main paper: we keep the first 55 sampled candidates per prompt, define T(X)T(X) as the mean verifier loss across those candidates, and use the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. The same qualitative pattern holds across the sweep: the conditional methods remain much more efficient than ICP while sharply improving worst-group coverage. Table 10: Full GSM8K sweep across target error rates α using the first five sampled candidates per prompt, T(X)T(X) equal to mean verifier loss, and the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α1-α are preferred. Methods α=0.05α=0.05 α=0.10α=0.10 α=0.15α=0.15 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 ICP 95.09 ± 1.42 79.85 ± 6.53 4.73 ± 0.09 90.39 ± 1.44 56.36 ± 6.20 4.36 ± 0.08 83.91 ± 2.15 27.73 ± 8.14 3.97 ± 0.11 Learnt CP 94.91 ± 1.03 88.48 ± 3.05 4.01 ± 0.98 90.09 ± 1.38 86.06 ± 0.77 2.22 ± 0.07 84.70 ± 1.11 79.09 ± 1.56 1.92 ± 0.06 CFC (ours) 94.82 ± 0.97 88.48 ± 2.32 2.35 ± 0.43 90.18 ± 1.41 86.36 ± 1.07 1.49 ± 0.04 84.97 ± 1.12 79.55 ± 1.52 1.34 ± 0.04 CFC-FULL (ours) 95.03 ± 0.97 88.94 ± 2.74 4.08 ± 0.93 90.30 ± 1.40 86.36 ± 1.07 2.23 ± 0.08 85.09 ± 1.03 79.55 ± 1.52 1.96 ± 0.06 CFC-PAC (ours) 95.03 ± 1.33 88.18 ± 2.47 2.59 ± 0.29 91.79 ± 1.66 88.18 ± 1.41 1.55 ± 0.06 86.64 ± 0.95 81.67 ± 1.11 1.38 ± 0.03 CFC-PAC-FULL (ours) 95.24 ± 1.40 88.79 ± 3.01 4.59 ± 0.62 91.91 ± 1.68 88.48 ± 1.62 2.34 ± 0.11 86.76 ± 0.88 81.67 ± 1.11 2.05 ± 0.05 Methods α=0.20α=0.20 α=0.25α=0.25 α=0.30α=0.30 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 ICP 79.55 ± 2.91 18.48 ± 4.92 3.69 ± 0.16 74.64 ± 2.53 13.48 ± 1.30 3.38 ± 0.15 70.18 ± 2.51 10.15 ± 0.91 3.11 ± 0.13 Learnt CP 79.42 ± 1.87 71.36 ± 3.63 1.74 ± 0.10 74.82 ± 2.51 66.97 ± 2.90 1.57 ± 0.07 69.36 ± 2.48 60.91 ± 2.23 1.42 ± 0.08 CFC (ours) 79.94 ± 1.80 72.12 ± 3.85 1.22 ± 0.04 75.03 ± 2.52 67.12 ± 2.86 1.12 ± 0.05 69.55 ± 2.53 61.21 ± 2.11 1.01 ± 0.05 CFC-FULL (ours) 80.06 ± 1.78 72.12 ± 3.85 1.76 ± 0.08 75.15 ± 2.47 67.12 ± 2.86 1.60 ± 0.07 69.64 ± 2.48 61.21 ± 2.11 1.44 ± 0.09 CFC-PAC (ours) 81.36 ± 1.68 73.18 ± 4.27 1.25 ± 0.04 75.94 ± 2.47 68.03 ± 2.81 1.13 ± 0.05 70.82 ± 2.58 62.58 ± 2.01 1.04 ± 0.05 CFC-PAC-FULL (ours) 81.48 ± 1.65 73.18 ± 4.27 1.82 ± 0.08 76.06 ± 2.42 68.03 ± 2.81 1.63 ± 0.07 70.94 ± 2.52 62.58 ± 2.01 1.48 ± 0.08 Candidate-budget ablation. Table 11 compares the chosen N=5N=5 budget against N=20N=20 at the representative target α=0.10α=0.10, keeping the same mean-loss proxy and quadratic basis. The larger candidate budget barely changes target calibration, but it inflates APSS substantially for every threshold-based method. This is why the main paper uses the smaller budget on GSM8K: the extra samples add little new diversity but materially hurt efficiency. Table 11: GSM8K sample-budget ablation at α=0.10α=0.10 under the mean-loss quadratic rule. Each budget uses the same basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2], differing only in the number of retained sampled candidates per prompt. Methods N=5N=5 N=20N=20 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 96.42 ± 0.39 86.52 ± 1.11 1.00 ± 0.00 96.70 ± 0.36 88.48 ± 1.11 1.00 ± 0.00 ICP 90.39 ± 1.44 56.36 ± 6.20 4.36 ± 0.08 90.15 ± 1.37 55.76 ± 5.52 16.75 ± 0.35 Learnt CP 90.09 ± 1.38 86.06 ± 0.77 2.22 ± 0.07 90.15 ± 0.81 87.42 ± 1.41 7.24 ± 0.33 CFC (ours) 90.18 ± 1.41 86.36 ± 1.07 1.49 ± 0.04 90.30 ± 0.63 87.73 ± 1.47 3.79 ± 0.12 CFC-FULL (ours) 90.30 ± 1.40 86.36 ± 1.07 2.23 ± 0.08 90.45 ± 0.70 87.73 ± 1.47 7.50 ± 0.18 CFC-PAC (ours) 91.79 ± 1.66 88.18 ± 1.41 1.55 ± 0.06 91.91 ± 0.46 88.64 ± 2.30 3.99 ± 0.07 CFC-PAC-FULL (ours) 91.91 ± 1.68 88.48 ± 1.62 2.34 ± 0.11 92.06 ± 0.57 88.79 ± 2.46 7.97 ± 0.03 C.2.3 Flickr8k Full target-risk sweep. Table 12 reports the full Flickr8k sweep for the chosen clean setting from the main paper: we keep up to two cached candidates per image, define T(X)T(X) as the mean verifier loss across those candidates, and use the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. This benchmark is visibly easier than GSM8K or TriviaQA, so the most informative comparison is closeness to the target coverage together with subgroup reliability. At α=0.03α=0.03 (target coverage 97%97\%), CFC-PAC-FULL is the closest full-set variant to target while improving GSC over every baseline, whereas CFC collapses to almost one caption per image and is best viewed as the smallest-size extreme. Table 12: Full Flickr8k sweep with Qwen2-VL-7B-Instruct using up to two cached candidates per image, T(X)T(X) equal to mean verifier loss, and the quadratic basis Φ(X)=[1,T(X),T(X)2] (X)=[1,T(X),T(X)^2]. ECR and GSC are reported in percent; for ECR, values closest to the target coverage 1−α1-α are preferred. Methods α=0.01α=0.01 α=0.02α=0.02 α=0.03α=0.03 ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ ECR GSC↑ APSS↓ TopK 97.75 ± 0.19 95.62 ± 0.71 2.00 ± 0.00 97.75 ± 0.19 95.62 ± 0.71 2.00 ± 0.00 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 ICP 97.29 ± 0.29 93.33 ± 1.45 1.93 ± 0.01 96.25 ± 0.66 88.23 ± 3.66 1.87 ± 0.02 95.58 ± 0.54 85.21 ± 3.14 1.84 ± 0.01 Learnt CP 97.39 ± 0.33 95.52 ± 0.85 1.74 ± 0.06 97.06 ± 0.31 95.10 ± 0.63 1.34 ± 0.07 96.16 ± 0.17 94.48 ± 0.26 1.22 ± 0.07 CFC (ours) 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 95.81 ± 0.38 93.23 ± 0.47 0.99 ± 0.00 CFC-FULL (ours) 97.66 ± 0.16 95.62 ± 0.71 1.86 ± 0.04 97.27 ± 0.21 95.21 ± 0.77 1.40 ± 0.06 96.27 ± 0.24 94.58 ± 0.26 1.25 ± 0.08 CFC-PAC (ours) 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 96.37 ± 0.17 93.23 ± 0.47 1.00 ± 0.00 CFC-PAC-FULL (ours) 97.75 ± 0.19 95.62 ± 0.71 2.00 ± 0.00 97.64 ± 0.13 95.62 ± 0.71 1.86 ± 0.06 97.27 ± 0.21 95.21 ± 0.77 1.42 ± 0.07 Setting ablation. Table 13 compares the chosen Flickr8k setting against two nearby alternatives from the clean search at the representative target α=0.03α=0.03. The chosen N=2N=2 mean-loss quadratic rule is the best-balanced option we found: it keeps the CFC variant essentially at single-caption size, while CFC-PAC-FULL remains close to target without the larger APSS jump of the N=3N=3 alternative. Table 13: Flickr8k setting ablation at α=0.03α=0.03 (target coverage 97%97\%). Each cell reports ECR/APSS/GSC for the corresponding method. Setting CFC CFC-PAC-FULL N=2N=2, max-loss, poly2 96.02 / 1.00 / 93.02 97.62 / 1.89 / 95.93 Chosen: N=2N=2, mean-loss, poly2 95.81 / 0.99 / 93.23 97.27 / 1.42 / 95.21 N=3N=3, mean-loss, poly2 96.00 / 1.00 / 93.02 97.81 / 2.16 / 96.35