Paper deep dive

When Can Proxies Improve the Sample Complexity of Preference Learning?

Yuchen Zhu, Daniel Augusto de Souza, Zhengyan Shi, Mengyue Yang, Pasquale Minervini, Alexander D'Amour, Matt J. Kusner

Year: 2024Venue: arXiv preprintArea: Alignment TrainingType: TheoreticalEmbeddings: 361

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 5:29:14 PM

Summary

The paper addresses the problem of reward hacking in Large Language Models (LLMs) by investigating whether proxy preference data can improve the sample complexity of learning a ground truth policy. The authors establish sufficient conditions under which proxy feedback, when combined with sparse expert data, provably improves the sample complexity of learning the ground truth policy, and they propose a parameterization for LLMs to leverage this relationship.

Entities (4)

Large Language Models · technology · 99%Reward Hacking · problem · 98%Direct Preference Optimization · method · 95%Sample Complexity · metric · 95%

Relation Signals (3)

Reward Hacking → affects → Large Language Models

confidence 98% · This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective.

Direct Preference Optimization → avoids → Reward Modelling

confidence 95% · Thus, DPO avoids the need for reward modelling.

Proxy Preference Data → canimprove → Sample Complexity

confidence 90% · We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

360,879 characters extracted from source content.

Expand or collapse full text

When Can Proxies Improve the Sample Complexity of Preference Learning? Yuchen Zhu1 Daniel Augusto de Souza1 Zhengyan Shi1 Mengyue Yang2 Pasquale Minervini3 Alexander D’Amour4 Matt J. Kusner1 1University College London, 2University of Bristol, 3University of Edinburgh, 4Google Deepmind Correspondence to: yuchen.zhu.18@ucl.ac.uk Abstract We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity. 1 Introduction Large Language Models (LLMs) and other large generative models have revolutionised modern machine learning with their surprising capabilities, surpassing human-level performance in law, medicine, and other examinations (Achiam et al., 2023; Amin et al., 2023). A large part of their success is their ability to incorporate human preferences to learn complex objectives such as trustworthiness (Yu et al., 2024), sentiment preferences (Chakraborty et al., 2024), and value alignment (Ji et al., 2023). Figure 1: Medical question answering. (Illustrative purpose only. Not medical advice.) Patients 1 and 2 are put in the same group by both doctors as their key characteristics - age, lifestyle and symptom are all similar. Only the expert doctor correctly identifies that morning headache deserves a further check than headache at any other times in the day, since it could be caused by nerve in the brain pressured by a tumour. The student doctor naively attributes this to stress. Patient 3 has characteristics sufficiently different from Patient 1 and 2, so is put in a different group, and again the recommendations made by the two doctors are different. In many cases, this preference data is a proxy for the ground truth. For example, humans raters tend to prefer longer answers to a question, even if the answer is less informative Zhou et al. (2024). In this case, ‘response length’ is a proxy for the true helpfulness of an answer. If an LLM is trained on this proxy data alone it leads to a ‘length-bias’ (Shen et al., 2023; Singhal et al., 2023), as LLMs fine-tuned with this preference data generate longer and better formatted responses to appear more helpful (Chen et al., 2024). This is an example of the well-known phenomenon of reward hacking111This is also sometimes referred to as reward over-optimisation.: a model optimised to perform well with respect to a proxy reward function, performs poorly with respect to a ground truth reward function (Casper et al., 2023). Reward hacking is a fundamental problem in learning that has been observed in optimised circuits listening in on the oscillators of other computers when instead tasked to build their own (Bird & Layzell, 2002), universities rejecting the most qualified applicants to boost their ratings (Golden, 2001), and many other cases in game playing (Clark & Amodei, 2016), autonomous driving (Knox et al., 2023), and text summarisation (Paulus et al., 2018). To address reward hacking in LLMs, prior work largely designs tweaks to the model, data, and optimization procedure. This includes regularisation towards an initial policy (Schulman et al., 2017; Rafailov et al., 2023; Huang et al., 2024), changing properties of the reward model (Gao et al., 2023; Coste et al., 2024), using soft labels (Zhu et al., 2024), adjusting optimization hyperparameters (Singhal et al., 2023), reward hacking detection mechanisms (Pan et al., 2022; Miao et al., 2024), and introducing additional tools specialised to counteract length bias (Chen et al., 2024). The reasoning behind this comes from the makeup of proxy data. We can think of proxy data as having two parts: (i) a true part that brings a policy closer to the ground truth policy during learning and (i) a false part that moves it farther away. Prior work limits learning to reduce the impact that the false part has on the final model. Without any further information on proxy preferences or the ground truth, we are restricted to methods such as these, i.e., methods that are blind to the true and false parts of proxy data, to reduce the impact of reward hacking (Pan et al., 2022). Luckily, in many settings, we also have access to sparse observations of high-quality preferences (Daniels-Koch & Freedman, 2022). For instance, in demonstration-guided reinforcement learning, expert data is added to improve sample efficiency (Rajeswaran et al., 2017) and to guide exploration (Nair et al., 2018). Recent work has shown that including such expert information can help counter reward hacking in LLMs (Rita et al., 2024). Consider the following medical example depicted in Figure 1. Patient 1 and Patient 2 consult the expert doctor and the student doctor about a condition they have. They have similar characteristics and essentially the same problem: a recurring morning headache that lasted a few days, but their exact phrasings can be different. Meanwhile, Patient 3 has different characteristics and a different condition to Patient 1 and 2. We think of the experienced doctor as representing a true policy and the student as a proxy policy. The two doctors both assign Patient 1 and 2 to the same group and Patient 3 to a different group, but the two doctors’ recommendations for a given group are different, since the expert doctor can correctly recognised some easily misdiagnosed symptoms while the student doctor cannot. We assume access to sparse prescriptions from an experienced doctor (ground truth) and plentiful prescriptions from less experienced student doctors (proxy). Even with ground truth data, If we naively learn a policy on the union of the dataset, we will learn a policy close to the proxy policy, as this data is more abundant. However, given the success of preference learning methods for LLMs, there is often useful information to extract from the prolific proxy data. A natural question is: When can proxy data ever provably improve preference learning? In this paper we outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that the proxy can provably improve the sample complexity of learning the ground truth policy. As not all proxies will satisfy these conditions, they can be used to guide a data collection process for a specific task. We show that as long as the collected proxy feedback shares certain properties with the true feedback, the sample complexity of learning with true preference data is provably improved by first training on large amounts of proxy preference data. The key idea behind this is that if the proxy and true policies share a certain structure, characterised in Condition 1-4, then it is possible to express the true policy as a low-dimensional adaptation of the proxy policy (Theorem 3). This relationship implies that certain parameters of the ground truth policy can be identified solely from proxy data, reducing the number of ground truth samples needed to learn the ground truth policy (Theorem 5, 6). This result immediately implies a parametrisation for LLMs that achieves improved sample complexity. Our contributions are: • We characterise a set of sufficient conditions on proxy feedback that, if satisfied, the sample complexity of learning the true policy is reduced through learning on this proxy feedback. • We show that if proxy feedback satisfies the sufficient conditions, it implies a specific model parametrisation and learning procedure to extract information from the proxies. We detail these and describe how one can adapt existing architectures to improve sample complexity. 2 Preliminaries Consider the set of all prompts XX and completions YY, the elements of these sets are discrete sequences of tokens with arbitrary length, e.g. =[x1,…,xN]subscript1…subscriptsubscript x=[x_1,…c,x_N_ x]italic_x = [ x1 , … , xitalic_N start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT ]. By considering an enumeration of all completions YY, the space of distributions subscriptP_YPcaligraphic_Y is equivalent to the subset of positive and unit norm sequences in the sequence space ℓ1superscriptℓ1 ^1ℓ1. We consider the set of policies, in the form of language models, which are maps →subscriptX _YX → Pcaligraphic_Y. Starting from an initial policy πrefsubscriptref _refπref222In practice, πrefsubscriptref _refπref is obtained from the supervised finetuning stage of language model training (Rafailov et al., 2023)., we want to find a target policy π†:→:superscript†→subscript [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π :X _Yπ† : X → Pcaligraphic_Y which aligns with the preferences of an ideal actor in a given scenario. To learn this policy, we have preference data directly from the ideal actor, denoted †superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D D†, as well as preference data from a proxy actor, ~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Dover~ start_ARG D end_ARG. The central question we consider is: under what assumptions can ~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Dover~ start_ARG D end_ARG improve the sample complexity of learning π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†? Human preference feedback. We aim to align πrefsubscriptref _refπref using preference data of the form (,w,l)subscriptsubscript\( x, y_w, y_l)\ ( italic_x , italic_yitalic_w , italic_yitalic_l ) , where wsubscript y_witalic_yitalic_w and lsubscript y_litalic_yitalic_l are candidate completions for prompt xitalic_x, and where wsubscript y_witalic_yitalic_w is preferred to lsubscript y_litalic_yitalic_l. We assume these preferences are generated from a underlying scalar reward function r⁢(,)r( x, y)r ( italic_x , italic_y ) according to Bradley & Terry (1952): ∼p;1,2∼i.i.d.πref(⋅|); x p_X; y_1, y_2 % i.i.d. _ref ·\> |\> x ;italic_x ∼ pcaligraphic_X ; italic_y1 , italic_y2 underi.i.d. start_ARG ∼ end_ARG πref ( ⋅ | italic_x ) ; (1) b∼Bern⁡[σ⁢(r⁢(,1)−r⁢(,2))];similar-toBernsubscript1subscript2 b [σ r % x, y_1 -r x, y_2 % ];b ∼ Bern [ σ ( r ( italic_x , italic_y1 ) - r ( italic_x , italic_y2 ) ) ] ; (2) (w,l)=(1,2)if b=1,(2,1)if b=0.,subscriptsubscriptcasessubscript1subscript2if b=1,subscript2subscript1if b=0. ( y_w, y_l)= cases*( y_1, y_2)&if % $b=1$,\\ ( y_2, y_1)&if $b=0$.\\ cases*,( italic_yitalic_w , italic_yitalic_l ) = start_ROW start_CELL ( italic_y1 , italic_y2 ) end_CELL start_CELL if b = 1 , end_CELL end_ROW start_ROW start_CELL ( italic_y2 , italic_y1 ) end_CELL start_CELL if b = 0 . end_CELL end_ROW , (3) where σ⁢(⋅)⋅σ(·)σ ( ⋅ ) is the sigmoid logistic function. We assume that 1subscript1 y_1italic_y1 and 2subscript2 y_2italic_y2 are sampled from πrefsubscriptref _refπref for simplicity, whereas in practice they can be sampled from other distributions over subscriptP_YPcaligraphic_Y. In this model, higher relative rewards increase the chance of a completion being picked as the winner wsubscript y_witalic_yitalic_w. We assume that the true preference data †superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D D† and the proxy preference data ~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Dover~ start_ARG D end_ARG are generated according to distinct reward functions r†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125r r† and r~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 rover~ start_ARG r end_ARG. Bandit problem setting. Given a data-generating process G=(r,πref,p)subscriptrefsubscriptG= r, _ref,p_X = ( r , πref , pcaligraphic_X ) with reward function r, a reference policy πrefsubscriptref _refπref, and a distribution of prompts psubscriptp_Xpcaligraphic_X, the optimal KL-regularised policy πGsubscript _Gπitalic_G for the data generating process G is the one that maximises the following optimisation objective: argmaxπ∼p,∼π(⋅|)[r(,)]−βKL(π(|)∥πref(|)), _π E_ x p_X% ,\; y π ·\> |\> x [r% x, y ]- π% y\> |\> x \>\|\> _ref % y\> |\> x ,arg maxitalic_π blackboard_Eitalic_x ∼ p start_POSTSUBSCRIPT X , italic_y ∼ π ( ⋅ | italic_x ) end_POSTSUBSCRIPT [ r ( italic_x , italic_y ) ] - β KL ( π ( italic_y | italic_x ) ∥ πref ( italic_y | italic_x ) ) , (4) where the regularisation parameter β controls how close to the reference the optimum should be. Under this objective, the optimal policy is given by: πG(|)∝πref(|)exp(1βr(,)). _G y\> |\> x % _ref y\> |\> x % 1βr x, y .πitalic_G ( italic_y | italic_x ) ∝ πref ( italic_y | italic_x ) exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG r ( italic_x , italic_y ) ) . (5) The target policy we aim to learn is thus denoted π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† satisfying (5) with respect to G†=(r†,πref,)superscript†superscript†subscriptrefsubscript [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G = [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125r , _ref,P_X † = ( r† , πref , Pcaligraphic_X ). Direct preference optimisation (DPO) and implicit rewards. Following Rafailov et al. (2023), by optimising the following objective: arg⁡maxπ⁡(,w,l)∼G⁡[log⁡σ⁢(β⁢log⁡π(w|)πref(w|)−β⁢log⁡π(l|)πref(l|))], _π E_( x, y_w, y% _l) G [ σ β π y% _w\> |\> x _ref y_w\>% |\> x -β π y_l\>% |\> x _ref y_l\> % |\> x ],arg maxitalic_π blackboard_E( italic_x , italic_y start_POSTSUBSCRIPT w , italic_yitalic_l ) ∼ G end_POSTSUBSCRIPT [ log σ ( β log divide start_ARG π ( italic_yitalic_w | italic_x ) end_ARG start_ARG πref ( italic_yitalic_w | italic_x ) end_ARG - β log divide start_ARG π ( italic_yitalic_l | italic_x ) end_ARG start_ARG πref ( italic_yitalic_l | italic_x ) end_ARG ) ] , (6) we recover the same optimal policy as described in Equation 5 without directly using any reward function. Thus, DPO avoids the need for reward modelling. We note that the policy implicitly defines a reward function via r⁢(,) r x, y ( italic_x , italic_y ) =β⁢log⁡π(w|)πref(w|) =β π y_w\> |\> x % _ref y_w\> |\> x = β log divide start_ARG π ( italic_yitalic_w | italic_x ) end_ARG start_ARG πref ( italic_yitalic_w | italic_x ) end_ARG (7) We can therefore define π†,π~superscript†~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π† , over~ start_ARG π end_ARG as the policies that (implicitly) define r†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125r r† and r~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 rover~ start_ARG r end_ARG, respectively. True and proxy preference data For most interesting tasks, sampling a dataset n†:=(i,w,i,l,i)i=1n†assignsubscriptsuperscript†superscript†superscriptsubscriptsubscriptsubscriptsubscript1superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D _n :=\ % x_i, y_w,i, y_l,i \_i=1 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125n D†italic_n† := ( italic_xitalic_i , italic_yitalic_w , i , italic_yitalic_l , i ) i = 1n start_POSTSUPERSCRIPT † end_POSTSUPERSCRIPT from G†=(r†,πref,p)superscript†superscript†subscriptrefsubscript [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G =( [rgb]0,0.3828125,0.45703125% [named]pgfstrokecolorrgb0,0.3828125,0.45703125r ,% _ref,p_X)G† = ( r† , πref , pcaligraphic_X ) can be costly, thus, the size n†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125n n† of the dataset might not be large enough for adequate training. In these cases, it is common to use a much larger proxy dataset ~n~:=(~i,~w,i,~l,i)i=1n~assignsubscript~~superscriptsubscriptsubscript~subscript~subscript~1~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 D_ n:=\ % x_i, y_w,i, y_l,i % \_i=1 [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 nover~ start_ARG D end_ARGover~ start_ARG n end_ARG := ( over~ start_ARG italic_x end_ARGi , over~ start_ARG italic_y end_ARGw , i , over~ start_ARG italic_y end_ARGl , i ) i = 1over~ start_ARG n end_ARG, where each data point is sampled i.i.d. from a proxy data-generating distribution G~=(r~,πref,p)~~subscriptrefsubscript [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 G=( [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 r, _ref,p_X)over~ start_ARG G end_ARG = ( over~ start_ARG r end_ARG , πref , pcaligraphic_X ), where G~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Gover~ start_ARG G end_ARG and G†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G G† only differ in the reward function. Nonetheless, if we do not have data from G†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G G†, then even if we have access to infinitely many data samples from G~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Gover~ start_ARG G end_ARG, and even if it allows us to learn the perfect reward model r~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 rover~ start_ARG r end_ARG333The technical condition for this to be possible is to be provided subscriptP_XPcaligraphic_X and πrefsubscriptref _refπref have full support., we can at best only learn the optimal proxy policy π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, which differs from the optimal true policy in π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†, by construction due to the difference in rewards. 3 Sufficient Conditions for Proxy Feedback Theory and survey papers point out the difficulty of alignment under mismatch between the true reward function and the one reflected by the human labelers (Skalse et al., 2022; Casper et al., 2023). In other related fields such as vision, (Chi et al., 2022) shows the impossibility of leveraging pre-training data for unseen tasks unless some similarity between the two tasks is given. From these observations we can draw two conclusions. (i) We must have at least some data from G†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G G† to learn π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†; this motivates the need for both n†subscriptsuperscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D _nD†italic_n and ~n~subscript~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 D_ nover~ start_ARG D end_ARGover~ start_ARG n end_ARG. (i) In order for π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG to inform us something about π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†, they must share some similarities. Thus, we ask the following research question: Under what sufficient conditions can we use ~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 Dover~ start_ARG D end_ARG to learn π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† more efficiently than if we used †superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D D† alone? How much can we improve? With the following conditions, we show that, when we have access to large amounts of proxy data, π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† can be expressed as a low-dimension adaptation of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, and hence the sample complexity of learning π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† is drastically improved, superexponential in the data manifold dimension. While these conditions may appear strong, in practice, they can be helpful in guiding the design of the proxy data collection procedure. In particular, the conditions correspond to what expertise we might require proxy raters and true raters to share. Our first condition says that given two distinct prompts, whenever they are mapped to the same response distribution under the true policy, so are they under the proxy policy; mathematically, the two policies share level sets (Figure 2, left). Figure 2: Illustrations of conditions 1-3. Left, middle, right: Condition 1, 2, 3, respectively. Condition 1 (Shared level sets). Given 1,2∈subscript1subscript2 x_1, x_2 _x1 , italic_x2 ∈ X, we have that π†(⋅|1)=π†(⋅|2) [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π ·\> |\> x_1% = [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π ·% \> |\> x_2 π† ( ⋅ | italic_x1 ) = π† ( ⋅ | italic_x2 ) if and only if π~(⋅|1)=π~(⋅|2) [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ·\> |\>% x_1 = [rgb]0.609375,0.1328125,0.1484375 % [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % ·\> |\> x_2 ~ start_ARG π end_ARG ( ⋅ | italic_x1 ) = over~ start_ARG π end_ARG ( ⋅ | italic_x2 ). In the context of the running example (Figure 1), Condition 1 corresponds to the experienced doctor and the student doctor classifying the symptoms of patients in the same way. We could reasonably expect this because comprehending the relevant details of a patient’s query is part of the basic training for a doctor. On the other hand, if the proxy preferences were sourced from generic crowd workers with no medical background, we would not expect this assumption to hold. Our second condition says that the set of expert response distributions is contained within the set of proxy response distributions (Figure 2, middle). Condition 2 (Image inclusion). All possible responses of the true policy are included in the proxy policy. π†⁢()⊆π~⁢()superscript†~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % X π† ( X ) ⊆ over~ start_ARG π end_ARG ( X ). In the context of the running example (Figure 1), Condition 2 says that the student doctor could, in principle, express any valid medical advice distribution, even if the student doctor may not know how to assign them with appropriate symptoms with high accuracy. This is again a reasonable assumption when the proxy feedback comes from a student doctor, but is less plausible if the proxy feedback comes from a rater without general knowledge of core concepts that underlie medical advice. We also introduce two technical conditions. Our first technical condition asserts that the image of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG constitutes a smaller, easily characterized subset of the space of distributions over finite-length token sequences subscriptP_YPcaligraphic_Y (Figure 2, right). Roughly, this assumption is similar to a common condition underpinning many modern deep learning architectures: that data lie on a lower-dimensional manifold. Condition 3 (Finite-dimensional encoding of π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X )). There exists an injective function ϕ:→:italic-ϕ→subscriptφ _Yϕ : V → Pcaligraphic_Y, where its domain ⊂ℝDsuperscriptℝV ^DV ⊂ blackboard_RD is a bounded convex polytope with D+11D+1D + 1 vertices, such that: 1. Its image ϕ⁢()italic-ϕφ ϕ ( V ) contains the image of the policies: π†⁢()⊆π~⁢()⊆ϕ⁢()superscript†~italic-ϕ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % X φ π† ( X ) ⊆ over~ start_ARG π end_ARG ( X ) ⊆ ϕ ( V ); 2. It is (Lϕ,Lϕ−1)subscriptitalic-ϕsubscriptsuperscriptitalic-ϕ1(L_φ,L_φ^-1)( Litalic_ϕ , Litalic_ϕ- 1 )-bi-Lipschitz with its left inverse ϕ−1:→:superscriptitalic-ϕ1→subscriptφ^-1 _Y ϕ- 1 : Pcaligraphic_Y → V: (1/Lϕ−1)⁢∥v1−v2∥p≤d⁢(ϕ⁢(v1),ϕ⁢(v2))≤Lϕ⁢∥v1−v2∥p1subscriptsuperscriptitalic-ϕ1subscriptdelimited-∥subscript1subscript2italic-ϕsubscript1italic-ϕsubscript2subscriptitalic-ϕsubscriptdelimited-∥subscript1subscript2(1/L_φ^-1) v_1-v_2 _p≤ d φ(v_1),% φ(v_2) ≤ L_φ v_1-v_2 _p( 1 / Litalic_ϕ- 1 ) ∥ v1 - v2 ∥p ≤ d ( ϕ ( v1 ) , ϕ ( v2 ) ) ≤ Litalic_ϕ ∥ v1 - v2 ∥p, where d is a metric on subscriptP_YPcaligraphic_Y, discussed later in Section 5. Note that the condition that VV is a convex polytope is benign, since we can extend any bi-Lipschitz function which bijects a compact subset of ℝNsuperscriptℝR^Nblackboard_RN with π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X ) to a bi-Lipschitz function from a bounded convex polytope with the same Lipschitz constants to a set containing π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X ). Condition 3 says that although the topological dimension of subscriptP_YPcaligraphic_Y can be extremely large, the image of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG is identified with a Euclidean subset only of dimension D≪dim⁡[]much-less-thandimsubscriptD [P_Y]D ≪ dim [ Pcaligraphic_Y ]. To make this rigorous, we provide a proof that the topological dimension of subscriptP_YPcaligraphic_Y can be as large as ∞; since any finite n-dimensional Euclidean space has topological dimension n, this shows that subscriptP_YPcaligraphic_Y is not a finite-dimensional Euclidean space. Proposition 1 (Topological dimension of subscriptP_YPcaligraphic_Y is ∞). Let =1,⋯,k1⋯K= \1,·s,k \K = 1 , ⋯ , k denote a set of k tokens. Let YY be the set of all finite length token sequences whose tokens all come from KK. Then YY has a one-to-one identification with the natural numbers. Let subscriptP_YPcaligraphic_Y be the set of probability mass functions over YY, then the topological dimension dim⁡()=∞dimsubscriptdim _Y =∞dim ( Pcaligraphic_Y ) = ∞. If YY is instead the set of token sequences with length ≤labsent≤ l≤ l, then dim⁡()=O⁢(kl)dimsubscriptsuperscriptdim _Y =O k^l ( Pcaligraphic_Y ) = O ( kitalic_l ). Proof. Proof in Appendix A. ∎ In practical LLM training regimes, a maximum sequence length is implemented, but the dimension of subscriptP_YPcaligraphic_Y grows exponentially with l. In situations where the true and proxy policies generate responses from a small subset of all token sequences, it could be reasonable to expect Condition 3 to hold - for example, only a small subset of all token sequences form valid sentences, and an even smaller subset of those form valid medical advice, so we expect that for medical question-answering tasks the responses distributions can be encoded with fewer dimensions than for general question-answering tasks. The low-dimensional encoding can be viewed as some intermediate representation of the prompt that is sufficient for determining the response distribution. We can thus think of ϕitalic-ϕφϕ as a policy decoder Our final technical condition concerns the similarity between the policy functions π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† and π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG. Since π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG and π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† share the same level sets, as supposed in Condition 1, it can be shown that π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is a well-defined function, where it should be noted that π~−1|π~⁢()⁢()evaluated-atsuperscript~1~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % p ~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) ( italic_p ) maps a point ∈π~⁢()~ p∈ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % X _p ∈ over~ start_ARG π end_ARG ( X ) to its pre-image under π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG. The proof is provided in Appendix B. Lemma 2. Under Condition 1, π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is a well-defined function. Lemma 2 allows us to describe the ‘difference’ between the proxy and true policy as a function: given pitalic_p, π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) maps all input prompts which were assigned pitalic_p by π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG to a distinct response distribution, say ′superscript′ p italic_p′, assigned by π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†. This justifies learning an ‘adapter’ function which reassigns pitalic_p to the correct value ′superscript′ p italic_p′. Had Condition 1 not held, then attempting to learn a function which assigns pitalic_p to ′superscript′ p italic_p′ no longer makes sense since there could be some values of pitalic_p for which the corresponding ′superscript′ p italic_p′ aren’t unique. By Lemma 2, we can impose a final technical condition which helps us quantify the difference between π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG and π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†: Condition 4. π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is Lipschitz continuous. Informally, this says that if two prompts are mapped to very similar completion distributions by π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, then they cannot get mapped to very different completion distributions by π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†; for a unit difference of the former, the difference for the latter must not exceed L for some positive scaler L. This condition applies to situations where the proxy rater is within reasonable ballpark from the true rater: for example, for some medicines the correct dosage can vary by a large amount depending on the patient; if the expert doctor prescribes a certain dosage, and the student doctor prescribes a dosage different but close to that, then the condition can be considered satisfied. The condition also generalises to a broader situation: if the expert doctor’s change in prescription, for instance, after observing some improvements in a patient, is similar to the change in prescription of a student doctor, then the condition can also be considered satisfied. However, in situations of crowdsourcing human preference from places such as Amazon Mechanical Turk, we cannot consider this condition to be satisfied; likewise, if we suspect that some proxy raters are adversarial, then we also cannot expect it satisfied. 4 Adapting π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG to π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† In this section, we derive an algorithm for learning π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† that leverages the structure in π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG specified in conditions 1-4. The algorithm hinges on a decompositon of the policies π into an encoder τ, a linear layer Θ Θ, and a decoder ϕitalic-ϕφϕ. Importantly, we show that π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† can be expressed reusing these components learned from proxy data generated by π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, with the addition of a low-dimensional adapter function between known spaces. This reduction to learning an adapter function ultimately allows us to derive the sample complexity improvement (Theorem 5). We outline the main steps of the derivation now: 1. First, from Condition 2 and 3, we can show that both π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG and π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† map prompts into a common lower-dimensional space before decoding into response distributions. 2. Then, by Condition 1 and 4, it can be shown that π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG and π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† differ only by a Lipschitz continuous function mapping ΔD→ΔD→superscriptΔsuperscriptΔ ^D→ ^DΔitalic_D → Δitalic_D. 4.1 Factorising π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† and π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG through VV. Under Condition 3, the proxy policy π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG factors through VV: that is, one can view it as mapping from the space of prompts to the space of response distributions via some intermediate representation of the prompt (i.e., VV) sufficient for determining the response distribution. Specifically, there is a bi-Lipschitz injective decoder ϕitalic-ϕφϕ from VV to the image of the proxy policy π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X ). Condition 2 says that the image of π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† is included in the image of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG. Therefore, we can view π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† as a function composition of some decoder ϕ:→π~⁢():italic-ϕ→~φ:V→ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % ϕ : V → over~ start_ARG π end_ARG ( X )444Strictly, after ϕ†superscriptitalic-ϕ† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125φ ϕ† we still need to go through an inclusion to land in subscriptP_YPcaligraphic_Y, but to simplify notation we omit this technicality. and some encoder function →X → V, such that π†superscript† [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π π† :→ϕ−1∘π†→ϕπ~⁢():absentsuperscriptitalic-ϕ1superscript†→italic-ϕ→~ :X φ^-1 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π V φ% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π : X start_ARROW start_OVERACCENT ϕ- 1 ∘ π† end_OVERACCENT → end_ARROW V start_ARROW overϕ → end_ARROW over~ start_ARG π end_ARG ( X ) (8) Analogously, π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG can be viewed also as π~~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG :→ϕ−1∘π~→ϕπ~⁢():absentsuperscriptitalic-ϕ1~→italic-ϕ→~ :X φ^-1 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 πV φ% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π : X start_ARROW start_OVERACCENT ϕ- 1 ∘ over~ start_ARG π end_ARG end_OVERACCENT → end_ARROW V start_ARROW overϕ → end_ARROW over~ start_ARG π end_ARG ( X ) (9) Next, we show that π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† can be expressed by inserting a transformation into a function decomposition of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG. This transformation can be shown to be a Lipschitz map between two known D-dimensional spaces. We can thus think of this transformation as an ‘adapter’ function. 4.2 π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† and π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG differ by a function between D-simplices. It is now possible to show that π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† and π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG differ only by a transformation on the representation space VV. However, for the sample complexity arguments that follow, it is convenient to map VV to a D-simplex ΔDsuperscriptΔ ^DΔitalic_D, then to show that π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† and π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG differ only by a transformation on ΔDsuperscriptΔ ^DΔitalic_D. To this end, since VV is a D-polytope with D+11D+1D + 1 vertices, every point in VV can be expressed as a convex combination of the vertices. Then, it can be shown that π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG and π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† can be further factored through a D-simplex, ΔDsuperscriptΔ ^DΔitalic_D. With this formalism in hand, we now state our result. Theorem 3. We work under Conditions 1, 2, 3 and 4. For some D, there exists a Lipschitz invertible function ϕ~:→π~⁢():~italic-ϕ→~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ:V→ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π ~ start_ARG ϕ end_ARG : V → over~ start_ARG π end_ARG ( X ) satisfying Condition 3, Θ~∈ℝN×(D+1)~Θsuperscriptℝ1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^N×(D+1)over~ start_ARG Θ end_ARG ∈ blackboard_RN × ( D + 1 ) and τ~∘:→ΔD:superscript~→superscriptΔ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ :X→ % ^Dover~ start_ARG τ end_ARG∘ : X → Δitalic_D s.t. π~=ϕ~∘Θ~⁢τ~∘~~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π= [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG π end_ARG = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘. Moreover, for any (ϕ~,Θ~,τ~∘)~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ ) such that π~=ϕ~∘Θ~⁢τ~∘~~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π= [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG π end_ARG = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘, there exists a Lipschitz continuous function π¯†:ΔD→ΔD:superscript¯†→superscriptΔsuperscriptΔ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π : ^D→ ^Dover¯ start_ARG π end_ARG† : Δitalic_D → Δitalic_D s.t. π†=ϕ~∘Θ~⁢π¯†∘τ~∘superscript†~italic-ϕ~Θsuperscript¯†superscript~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ π† = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG† ∘ over~ start_ARG τ end_ARG∘. Proof. Proof in Appendix B. ∎ Theorem 3 has two important implications for learning. 1. It establishes that there exists a decomposition of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG with modules that can be reused to express π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†. 2. It further establishes that for any satisfactory decomposition, there exists an adapter π¯†superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π over¯ start_ARG π end_ARG†. This suggests that in practice we can first find a suitable triplet (ϕ~,Θ~,τ~∘)~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ ), then learn an adapter. 4.3 Model Parametrisation and Learning We now sketch our learning algorithm. Theorem 3 gives rise to a two-step procedure to learn the true policy π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†, firstly we recover the functional components using a large proxy dataset ~n~subscript~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 D_ nover~ start_ARG D end_ARGover~ start_ARG n end_ARG and then, secondly, we use a small true dataset n†subscriptsuperscript†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D _ [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125n D†italic_n† to learn the low-dimensional adapter. Stage 1 Based on Theorem 3, we model the proxy policy π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG with a parametric model composed of three functions: (i) τ~θ∘superscriptsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ_θ over~ start_ARG τ end_ARGθ∘, an embedding function from the prompts XX to the D-simplex ΔDsuperscriptΔ ^DΔitalic_D, (i) Θ~∈ℝD×D~Θsuperscriptℝ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^D×Dover~ start_ARG Θ end_ARG ∈ blackboard_RD × D, a linear map from the simplex to a convex polytope VV, and (i) ϕ~θsubscript~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ_θover~ start_ARG ϕ end_ARGθ, an injective function from the latent space VV to a distribution of completions subscriptP_YPcaligraphic_Y. Therefore, our model is expressed as: π~θsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π_θover~ start_ARG π end_ARGθ =ϕ~θ∘Θ~⁢τ~θ∘.absentsubscript~italic-ϕ~Θsuperscriptsubscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ_θ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ_θ .= over~ start_ARG ϕ end_ARGθ ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARGθ∘ . (10) Based on the DPO loss (Eq. 6), ϕ~θsubscript~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ_θover~ start_ARG ϕ end_ARGθ, Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG, and τ~θ∘superscriptsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ_θ over~ start_ARG τ end_ARGθ∘ are learned using the empirical preference learning objective with the proxy dataset ~n~subscript~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 D_ nover~ start_ARG D end_ARGover~ start_ARG n end_ARG: L~n~⁢(π~θ)subscript~L~subscript~ L_ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 n% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π_θ ~ start_ARG L end_ARGover~ start_ARG n end_ARG ( over~ start_ARG π end_ARGθ ) =−1n~⁢∑i=1n~log⁡σ⁢(β⁢log⁡π~θ(~w,i|~i)πref(~w,i|~i)−β⁢log⁡π~θ(~l,i|~i)πref(~l,i|~i)). =- 1 n _i=1 n σ % β [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π_θ % y_w,i\> |\> x_i _% ref y_w,i\> |\> x_i% -β [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % π_θ y_l,i\> |\> x_i% _ref y_l,i\> |\>% x_i .= - divide start_ARG 1 end_ARG start_ARG over~ start_ARG n end_ARG end_ARG ∑i = 1over~ start_ARG n end_ARG log σ ( β log divide start_ARG over~ start_ARG π end_ARGθ ( over~ start_ARG italic_y end_ARGw , i | over~ start_ARG italic_x end_ARGi ) end_ARG start_ARG πref ( over~ start_ARG italic_y end_ARGw , i | over~ start_ARG italic_x end_ARGi ) end_ARG - β log divide start_ARG over~ start_ARG π end_ARGθ ( over~ start_ARG italic_y end_ARGl , i | over~ start_ARG italic_x end_ARGi ) end_ARG start_ARG πref ( over~ start_ARG italic_y end_ARGl , i | over~ start_ARG italic_x end_ARGi ) end_ARG ) . (11) In the large sample limit, the optimal parametrised model ϕ~θ∗∘Θ~∗⁢τ~θ∘,∗superscriptsubscript~italic-ϕsuperscript~Θsuperscriptsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ_θ^* [rgb% ]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 ^* [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ_θ ,*over~ start_ARG ϕ end_ARGθ∗ ∘ over~ start_ARG Θ end_ARG∗ over~ start_ARG τ end_ARGθ∘ , ∗ minimises the population proxy preference loss, and due to Theorem 3, we know that the optimal KL-regularised proxy policy π~=ϕ~θ∗∘Θ~∗⁢τ~θ∘,∗~superscriptsubscript~italic-ϕsuperscript~Θsuperscriptsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π= [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ_θ^* [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 ^* [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ_θ ,*over~ start_ARG π end_ARG = over~ start_ARG ϕ end_ARGθ∗ ∘ over~ start_ARG Θ end_ARG∗ over~ start_ARG τ end_ARGθ∘ , ∗, thus, justifying our parametrization. Stage 2 Following this, we then model the true policy π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† using the same pre-trained components from our model of the proxy policy π~θsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π_θover~ start_ARG π end_ARGθ with the addition of a low-dimensional adapter function π¯θ†subscriptsuperscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π _θover¯ start_ARG π end_ARG†θ which maps a latent representation in the simplex ΔDsuperscriptΔ ^DΔitalic_D to another ΔDsuperscriptΔ ^DΔitalic_D as follows: πθ†subscriptsuperscript† [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π _θπ†italic_θ =ϕ~θ∘Θ~⁢π¯θ†∘τ~θ∘.absentsubscript~italic-ϕ~Θsubscriptsuperscript¯†superscriptsubscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ_θ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π _θ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ_θ .= over~ start_ARG ϕ end_ARGθ ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG†θ ∘ over~ start_ARG τ end_ARGθ∘ . (12) The adapter π¯θ†subscriptsuperscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π _θover¯ start_ARG π end_ARG†θ is learned by optimization of the empirical preference learning objective with the true dataset n†subscriptsuperscript†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D _n D†italic_n†, while keeping ϕ~θsubscript~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ_θover~ start_ARG ϕ end_ARGθ, Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG, and τ~θ∘superscriptsubscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ_θ over~ start_ARG τ end_ARGθ∘ fixed based on their previously optimised values: L†n†⁢(π¯θ†)subscriptsuperscriptL†superscript†subscriptsuperscript¯† [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125L _n^% [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125 π _θ †italic_n† ( over¯ start_ARG π end_ARG†θ ) =−1n†⁢∑i=1n†log⁡σ⁢(β⁢log⁡π¯θ†(w,i|i)πref(w,i|i)−β⁢log⁡π¯θ†(l,i|i)πref(l,i|i)). =- 1 [rgb]0,0.3828125,0.45703125 [% named]pgfstrokecolorrgb0,0.3828125,0.45703125n _i=1^% [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125n σ β % [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π _θ y_w,% i\> |\> x_i _ref y_% w,i\> |\> x_i -β [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π _θ y_l,% i\> |\> x_i _ref y_% l,i\> |\> x_i .= - divide start_ARG 1 end_ARG start_ARG n† end_ARG ∑i = 1n start_POSTSUPERSCRIPT † end_POSTSUPERSCRIPT log σ ( β log divide start_ARG over¯ start_ARG π end_ARG†θ ( italic_yitalic_w , i | italic_xitalic_i ) end_ARG start_ARG πref ( italic_yitalic_w , i | italic_xitalic_i ) end_ARG - β log divide start_ARG over¯ start_ARG π end_ARG†θ ( italic_yitalic_l , i | italic_xitalic_i ) end_ARG start_ARG πref ( italic_yitalic_l , i | italic_xitalic_i ) end_ARG ) . (13) There can be more efficient algorithm which learn the triplet (ϕ~,Θ~,τ~∘)~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ, % , τ ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ ) by using the proxy or true data simultaneously. But by splitting the learning into two stages, the first using only proxy data, and the second using only true preference data, we can make a direct sample complexity comparison between learning π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† from scratch, and learning the adapter π¯†superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π over¯ start_ARG π end_ARG† in Stage 2, in terms of the size of the true preference data n†subscriptsuperscript†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125D _n D†italic_n†. 5 Convergence Rates Analysis To illustrate the benefit of learning π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† using the outlined algorithm, we analyse its soundness by showing the sample complexity improvement given that we have identified the true ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG, Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG and τ~∘superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ over~ start_ARG τ end_ARG∘ from the proxy dataset in the first stage. This can be a reasonable approximation of the properties of the learning procedure in cases where the proxy dataset is much larger than the true dataset. To this end, we analyse the generalisation error bound for the second stage given access to true ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG, Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG and τ~∘superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ over~ start_ARG τ end_ARG∘. Following the approaches of Elesedy (2022) Mohri et al. (2012, Exercise 3.31), the generalisation error can be shown to be linear in the covering number of the hypothesis class. Our idea here is that the hypothesis class of π¯ πover¯ start_ARG π end_ARG is made smaller by having knowledge of ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG, Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG and τ~∘superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ over~ start_ARG τ end_ARG∘, hence the covering number is also smaller. In order to define covering numbers, we first need to define a notion of metric on all relevant spaces and the hypothesis classes we consider. 5.1 Metrics and Hypothesis Classes Metric on finite-dimensional vector spaces. For any finite-dimensional vector space, we use the p-norm-induced metric; for a simplex ΔDsuperscriptΔ ^DΔitalic_D we denote its metric by dΔsubscriptΔd_ droman_Δ and for any other finite dimensional space UU we use dsubscriptd_Udcaligraphic_U. Metric on the prompt space. The prompt space XX, is a discrete and unstructured space, so we define a metric, dsubscriptd_Xdcaligraphic_X on it based on some fixed embedding function f, which maps a prompt to a vector space with finite but high dimensions: d⁢(x,x′)=df⁢()⁢(f⁢(x),f⁢(x′))=∥f⁢(x)−f⁢(x′)∥psubscriptsuperscript′subscriptsuperscript′subscriptdelimited-∥superscript′d_X x,x =d_f % X f x ,f x^% = f x -f% x _pdcaligraphic_X ( x , x′ ) = ditalic_f ( X ) ( f ( x ) , f ( x′ ) ) = ∥ f ( x ) - f ( x′ ) ∥p. For intuition we can think about f as some general-purpose embedding, such as one got by retrieving some intermediate layer from a large model such as CLIP Radford et al. (2021). Importantly, this metric is only relevant when considering the complexity of the hypothesis class when we learn the target policy without proxy data, which one expects to be large; this intuition is confirmed since a general-purpose embedding may not work well across all tasks, and can result in a large Lipschitz constant for the target function. Metric on a policy space. Defining a metric on a space of policies is more involved, and we begin by relating a policy and the reward function under which it is the KL-regularised optimal policy. Given a policy π:→:→subscriptπ:X _Yπ : X → Pcaligraphic_Y, define the implicit reward as: rπ⁢(,)subscript r_π x, y _π ( italic_x , italic_y ) =β⁢log⁡π(|)πref(|) =β π y\> |\> x % _ref y\> |\> x = β log divide start_ARG π ( italic_y | italic_x ) end_ARG start_ARG πref ( italic_y | italic_x ) end_ARG (14) Using rπsubscriptr_πritalic_π, we can define a metric on the set of policies, and we express in the next lemma. Lemma 4 (rπsubscriptr_πritalic_π helps define a metric). For any π,π′:→:superscript′→subscriptπ,π :X _Yπ , π′ : X → Pcaligraphic_Y such that π and π′π π′ both satisfy ∥rπ∥∞≤∞subscriptdelimited-∥subscript r_π _∞≤∞∥ ritalic_π ∥∞ ≤ ∞, define dr⁢(π,π′)=‖rπ−rπ′‖∞subscriptsuperscript′subscriptnormsubscriptsubscriptsuperscript′d_r π,π =\|r_π-r_π \|_∞ditalic_r ( π , π′ ) = ∥ ritalic_π - ritalic_π′ ∥∞ (15) Then dr⁢(⋅,⋅)subscript⋅d_r ·,· _r ( ⋅ , ⋅ ) defines a metric over the set of policies π:→|∑π(y|x)=1 \π:X _Y\; |\; _Yπ % y\> |\>x =1 \ π : X → Pcaligraphic_Y | ∑Y π ( y | x ) = 1 . Proof. Proof in Appendix C. ∎ Metric on the completion distribution space. We also need a notion of metric on the range of any policy function, subscriptP_YPcaligraphic_Y. To this end, we define the following function d:×→ℝ∪∞:subscriptsubscript→subscriptsubscriptℝd_P_Y:P_Y×P_Y ∪ % \∞ \dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT : Pcaligraphic_Y × Pcaligraphic_Y → blackboard_R ∪ ∞ : d⁢(p,q)subscriptsubscript d_P_Y p,q _P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT ( p , q ) :=∥β⁢log⁡p⁢(⋅y)q⁢(⋅y)∥∞assignabsentsubscriptdelimited-∥subscript⋅subscript⋅ := β p ·_y % q ·_y _∞:= ∥ β log divide start_ARG p ( ⋅y ) end_ARG start_ARG q ( ⋅y ) end_ARG ∥∞ (16) This construction is motivated by a resemblance with the construction of rπsubscriptr_πritalic_π. Now define the hypothesis class of π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† as: Definition 1 (Hypothesis class Π̊Π over̊ start_ARG Π end_ARG). Let Π̊Π over̊ start_ARG Π end_ARG be a set of functions π:→:→subscriptπ:X _Yπ : X → Pcaligraphic_Y s.t. ∀x⁢∑yπ⁢(y|x)=1for-allsubscriptconditional1∀ x _yπ y|x =1∀ x ∑y π ( y | x ) = 1, and π satisfies ∥rπ∥∞≤Csubscriptdelimited-∥subscript r_π _∞≤ C∥ ritalic_π ∥∞ ≤ C. So let drsubscriptd_rditalic_r be the metric on Π̊Π over̊ start_ARG Π end_ARG and all its subsets. Later on, we will notice that the covering number of a hypothesis class depends on the smallest Lipschitz constant of the class. Let Π̊⁢(L)̊Π L ̊ start_ARG Π end_ARG ( L ) denote the subset of Π̊Π over̊ start_ARG Π end_ARG such that every policy π is L-Lipschitz wrt dsubscriptd_Xdcaligraphic_X and dYsubscriptsubscriptd_P_Ydcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT. Meanwhile, with knowledge of ϕ~,Θ~,τ~∘~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘, we define a subset of Π̊Π over̊ start_ARG Π end_ARG with respect to a Lipschitz constant. Definition 2 (Hypothesis class fixing ϕ~,Θ~,τ~∘~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ and Lipschitz-constant Lπ¯subscript¯L_ πLover¯ start_ARG π end_ARG). Fix ϕ~,Θ~,τ~∘~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ,\; [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 ,\; [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘. Π⁢(ϕ~,Θ~,τ~∘,Lπ¯)⊆Π̊Π~italic-ϕ~Θsuperscript~subscript¯̊Π [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ,L_ π % Π ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ , Lover¯ start_ARG π end_ARG ) ⊆ over̊ start_ARG Π end_ARG contains all π∈Π̊Ππ∈ π ∈ over̊ start_ARG Π end_ARG which can be written as π(⋅|x) π ·|x π ( ⋅ | x ) =ϕ~∘Θ~⁢π¯∘τ~∘⁢(x)absent~italic-ϕ~Θ¯superscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( x ) (17) for some Lπ¯subscript¯L_ πLover¯ start_ARG π end_ARG-Lipschitz π¯ πover¯ start_ARG π end_ARG, wrt dΔsubscriptΔd_ droman_Δ. Now we are ready to state our main generalisation error bounds results. Theorem 5 (Bounding sample complexity in terms of dimension). We remain in the set up of Proposition 8. The covering number of Π Π is bounded above by a function of D: Cov⁢(Π,dr,3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,3κ+3L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πδ Cov ( Π , ditalic_r , 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) ≤(2⁢Lϕ⁢∥Θ~∥p⁢Dκ)D⁢(2⁢Dδ)Dabsentsuperscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsuperscript2 ≤ 2L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _p Dκ % ^D 2 Dδ ^D≤ ( divide start_ARG 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG κ end_ARG )D ( divide start_ARG 2 square-root start_ARG D end_ARG end_ARG start_ARG δ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (18) Set κ=ϵ48italic-ϵ48κ= ε48κ = divide start_ARG ϵ end_ARG start_ARG 48 end_ARG, we need n⁢(ϵ,ω)italic-ϵ n ε,ω ( ϵ , ω ) =Ω⁢(Dϵ2⁢(96⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢Dϵ)D⁢log⁡(96⁢Lϕ⁢∥Θ~∥p⁢Dϵ)−log⁡ω)absentΩsuperscriptitalic-ϵ2superscript96subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ96subscriptitalic-ϕsubscriptdelimited-∥~Θitalic-ϵ = Dε^2 96L_% φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% π Dε ^D 96L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p% Dε - ω = Ω ( divide start_ARG D end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG )D log ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (19) samples to generalise. That is, whenever n′≥n⁢(ϵ,ω)superscript′italic-ϵn ≥ n(ε,ω)n′ ≥ n ( ϵ , ω ), we have P⁢(supπ∈Π|RG⁢(π)−RG^n′⁢(π)|≥ϵ)≤ωsubscriptsupremumΠsubscriptsubscriptsubscript^superscript′italic-ϵ P ( _π∈ |R_G π -R_% G_n π |≥ε )≤ ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n′ end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ ω (20) Proof. Proof in Appendix C.1. ∎ Theorem 6 (Bounding sample complexity of learning without proxy). Let D′ be the dimension of a given embedding function f of XX. Let Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯)̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π ̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) be the subset of Π̊Π over̊ start_ARG Π end_ARG where π is Lϕ⁢∥Θ~∥p⁢Lπ¯subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯L_φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πLitalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG-Lipschitz. Set κ=ϵ24italic-ϵ24κ= ε24κ = divide start_ARG ϵ end_ARG start_ARG 24 end_ARG, we need Ω⁢(D′ϵ2⁢(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)D′⁢log⁡(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)−log⁡ω)Ωsuperscript′italic-ϵ2superscript48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵsuperscript′48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵ D ε^2 % 48L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πE p,D D^% ε ^D 48L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πE p,D D % ε - ω Ω ( divide start_ARG D′ end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT log ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (21) samples to generalise, where D′≫Dmuch-greater-thansuperscript′D D′ ≫ D , and E′⁢(p,D′)≫1much-greater-thansuperscript′1E p,D 1E′ ( p , D′ ) ≫ 1. That is, whenever n′≥n⁢(ϵ,ω)superscript′italic-ϵn ≥ n(ε,ω)n′ ≥ n ( ϵ , ω ), we have P⁢(supπ∈Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯)|RG⁢(π)−RG^n′⁢(π)|≥ϵ)≤δsubscriptsupremum̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptsubscriptsubscript^superscript′italic-ϵ P ( _π∈ L_φ % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _pL_ π % |R_G π -R_ G_n % π |≥ε )≤ ( supitalic_π ∈ over̊ start_ARG Π end_ARG ( L start_POSTSUBSCRIPT ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) end_POSTSUBSCRIPT | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n′ end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ δ (22) Proof. Proof in Appendix C.2. ∎ Discussion. Theorem 6 says that if we learn π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† directly from expensive samples of G†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G G†, then the sample complexity scales with D′, which is the dimension of the embedding space ; this can be extremely high dimensional. However if we parametrise π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† using ϕ~,Θ~,τ~∘~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ which compose to be π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, and can be learned from cheap samples of G~~ Gover~ start_ARG G end_ARG, then the number of expensive samples we need from G†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125G G† scale with D which is assumed to be much lower-dimensional than D′. 6 Related Work Reward hacking theory. Initial work on the theory of reward hacking considered the setting where the proxy reward was a function of a subset of true reward features (Zhuang & Hadfield-Menell, 2020). This work demonstrates that optimising the proxy can lead to arbitrarily low true reward. Similarly, Tien et al. (2022) give theoretical results for reward hacking when a learned reward uses nuisance variables that correlate with true causal variables. These ideas were extended to arbitrary MDPs by Skalse et al. (2022) who define a proxy reward as hackable if it prefers policy π1subscript1 _1π1 over π2subscript2 _2π2 when the true reward has the opposite preference. Recent work has sought to develop scaling laws for reward hacking that describe how the true reward changes as the proxy reward is optimised (Gao et al., 2023). Rafailov et al. (2024) show similar over-optimisation patterns in DPO at higher KL-divergence budgets, even without an explicit reward model. In contrast to these works, our theoretical results suggest a new model parametrisation and training scheme which achieves improved sample complexity for learning the true policy; hence, our results are constructive. Addressing reward hacking in LLMs. One of the classic examples of reward hacking in LLMs is their propensity for verbose responses that are not more helpful, often called the ‘length bias’, or ‘length hacking’ of LLMs (Singhal et al., 2023). To address this, Singhal et al. (2023) modified various aspects of PPO (increasing KL regularisation, omitting outputs beyond a certain length, and reward scaling), as well as the training data, with mixed success. Chen et al. (2024) conducted a large-scale evaluation of the impact of hyperparameters and the above modifications on reward hacking. They further introduce a model that decorrelates preference predictions with length. Miao et al. (2024) formulate reward modelling as optimising a variational information bottleneck and then use this to filter out less important features in latent space. Huang et al. (2024) mitigates reward over-optimisation by replacing the KL regularisation with an alternative term which implicitly implements the principle of pessimism in the face of uncertainty. Yang et al. (2024) addresses a form of reward misalignment due to distribution shift of the prompts and responses seen in training versus test time. To the best of our knowledge, reward hacking due to a difference in the reward functions in training and test time is not discussed explicitly in existing work. More importantly, the impossibility of target policy recovery without some data from the target reward, is not yet acknowledged. RLHF with expert feedback. Human feedback often varies in quality, and one key challenge is how to incorporate these different feedbacks into learning (Daniels-Koch & Freedman, 2022). Freedman et al. (2023) formulate selecting which human to query for feedback as a bandit problem. Yamagata et al. (2024) uses the Boltzmann-rational model to account for varying levels of expertise. Our model parametrisation will leverage certain invariances between proxy and expert/true feedback that allows identification of the true policy as a low-dimensional adaptation of the proxy policy. 7 Conclusion We study the problem of reward hacking due to distribution shifts. Specifically, an abundance of preference rankings is generated by a proxy reward function, different from the true reward function, which is costly to query. We thus consider the setting where we have a large proxy dataset and a small true dataset; to the best of our knowledge, we are the first to consider this setting. We formulate conditions motivated by a real-world example, under which we prove that the optimal proxy policy can be decomposed into component functions shared with the optimal true policy, and that the true policy is only one low-dimensional adapter function away from the proxy policy given the shared component functions. We then observe that in the large sample limit of proxy data, one set of such component functions can be identified from minimising the preference loss. Leveraging this, we provide a characterisation of the sample complexity bound for learning the hypothesis class both with and without knowledge of the shared component functions; in particular, it is seen that under knowledge of the shared component functions the sample complexity bound is much lower than without such knowledge. As ongoing work, we are working on empirical evaluation of our theoretical findings, as well as relaxing some of the conditions we made. An updated version of the manuscript will be uploaded in the near future. Acknowledgments YZ acknowledges support by the Engineering and Physical Sciences Research Council with grant number EP/S021566/1. References Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Amin et al. (2023) Kanhai S Amin, Melissa A Davis, Rushabh Doshi, Andrew H Haims, Pavan Khosla, and Howard P Forman. Accuracy of chatgpt, google bard, and microsoft bing for simplifying radiology reports. Radiology, 309(2):e232561, 2023. Bird & Layzell (2002) Jon Bird and Paul Layzell. The evolved radio and its implications for modelling the evolution of novel sensors. In Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), volume 2, p. 1836–1841. IEEE, 2002. Bradley & Terry (1952) Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952. Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. Survey Certification. Chakraborty et al. (2024) Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024. Chen et al. (2024) Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024. Chi et al. (2022) Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Meta discovery: Learning to discover novel classes given very limited data. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=MEpKGLsY8f. Clark & Amodei (2016) Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https://openai.com/index/faulty-reward-functions/. Coste et al. (2024) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=dcjtMYkpXx. Daniels-Koch & Freedman (2022) Oliver Daniels-Koch and Rachel Freedman. The expertise problem: Learning from specialized feedback. arXiv preprint arXiv:2211.06519, 2022. Elesedy (2022) Bryn Elesedy. Group symmetry in pac learning. In ICLR 2022 workshop on geometrical and topological representation learning, 2022. Freedman et al. (2023) Rachel Freedman, Justin Svegliato, Kyle Wray, and Stuart Russell. Active teacher selection for reinforcement learning from human feedback. arXiv preprint arXiv:2310.15288, 2023. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, p. 10835–10866. PMLR, 2023. Golden (2001) Daniel Golden. Glass floor colleges reject top applicants, accepting only the students likely to enroll, 2001. Huang et al. (2024) Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J. Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization, 2024. URL https://arxiv.org/abs/2407.13399. Ji et al. (2023) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023. Knox et al. (2023) W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829, 2023. Miao et al. (2024) Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Mitigating reward hacking via information-theoretic reward modeling. arXiv preprint arXiv:2402.09345, 2024. Mohri et al. (2012) M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. Adaptive Computation and Machine Learning series. MIT Press, 2012. ISBN 9780262018258. URL https://books.google.co.uk/books?id=maz6AQAAQBAJ. Nair et al. (2018) Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), p. 6292–6299. IEEE, 2018. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03544, 2022. Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https://arxiv.org/abs/2103.00020. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9. Rafailov et al. (2024) Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024. URL https://arxiv.org/abs/2406.02900. Rajeswaran et al. (2017) Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017. Rita et al. (2024) Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in llm with demonstration-guided reinforcement learning. arXiv preprint arXiv:2404.19409, 2024. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shen et al. (2023) Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022. Tien et al. (2022) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601, 2022. Yamagata et al. (2024) Taku Yamagata, Tobias Oberkofler, Timo Kaufmann, Viktor Bengs, Eyke Hüllermeier, and Raul Santos-Rodriguez. Relatively rational: Learning utilities and rationalities jointly from pairwise preferences. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. Yang et al. (2024) Adam X. Yang, Maxime Robeyns, Thomas Coste, Zhengyan Shi, Jun Wang, Haitham Bou-Ammar, and Laurence Aitchison. Bayesian reward models for llm alignment, 2024. URL https://arxiv.org/abs/2402.13210. Yu et al. (2024) Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13807–13816, 2024. Zhou et al. (2024) Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, 634(8032), Sep 2024. ISSN 1476-4687. doi: 10.1038/s41586-024-07930-y. URL http://dx.doi.org/10.1038/s41586-024-07930-y. Zhu et al. (2024) Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024. Zhuang & Hadfield-Menell (2020) Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33:15763–15773, 2020. Appendix A Proof of Proposition 1 Proposition 1 Let =1,⋯,k1⋯K= \1,·s,k \K = 1 , ⋯ , k denote a set of k tokens. Let YY be the set of all finite length token sequences whose tokens all come from KK. Then YY has a one-to-one identification with the natural numbers. Let subscriptP_YPcaligraphic_Y be the set of probability mass functions over YY, then the topological dimension dim⁡()=∞dimsubscriptdim _Y =∞dim ( Pcaligraphic_Y ) = ∞. If YY is instead the set of token sequences with length ≤labsent≤ l≤ l, then dim⁡()=O⁢(kl)dimsubscriptsuperscriptdim _Y =O k^l ( Pcaligraphic_Y ) = O ( kitalic_l ). Proof of Proposition 1. We can identify YY with the natural numbers as follows. Let l denote the length of a token sequence. Since for each token in the sequence there are k options, there are klsuperscriptk^lkitalic_l distinct sequences of length l. We can define a bijective mapping from the set of klsuperscriptk^lkitalic_l sequences to the subset of natural numbers l:=(∑i=1l−1ki),⋯,(∑i=1lki)−1assignsuperscriptsuperscriptsubscript11superscript⋯superscriptsubscript1superscript1S^l:= \ ( _i=1^l-1k^i ),·s, ( _% i=1^lk^i )-1 \Sitalic_l := ( ∑i = 1l - 1 kitalic_i ) , ⋯ , ( ∑i = 1l kitalic_i ) - 1 for l≥22l≥ 2l ≥ 2 and 1:=0,⋯,k−1assignsuperscript10⋯1S^1:= \0,·s,k-1 \S1 := 0 , ⋯ , k - 1 . This is possible since the number of elements in lsuperscriptS^lSitalic_l is (∑i=1lki)−1−(∑i=1l−1ki)+1=klsuperscriptsubscript1superscript1superscriptsubscript11superscript1superscript ( _i=1^lk^i )-1- ( _i=1^l-1k^i )+1=k^l( ∑i = 1l kitalic_i ) - 1 - ( ∑i = 1l - 1 kitalic_i ) + 1 = kitalic_l. Denote one such mapping flsubscriptf_lfitalic_l. Then define f:→ℕ:→ℕf:Y : Y → blackboard_N: f⁢(y) f(y)f ( y ) =fl⁢(y)⁢ if length of y=labsentsubscript if length of y=l =f_l(y) if length of $y=l$= fitalic_l ( y ) if length of y = l (23) f is well-defined because every y has a unique length l. f is invertible because flsubscriptf_lfitalic_l is invertible for every l and l∩l′=∅superscriptsuperscriptsuperscript′S^l ^l = _l ∩ Sitalic_l start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = ∅ and ⋃l=1∞l=ℕsuperscriptsubscript1superscriptℕ _l=1^∞S^l=N⋃l = 1∞ Sitalic_l = blackboard_N. Thus, subscriptP_YPcaligraphic_Y is the set of probability mass functions whose sample space is ≅ℕabsentℕ ≅ blackboard_N. That is, an element PY∈subscriptsubscriptP_Y _YPitalic_Y ∈ Pcaligraphic_Y is an infinite positive sequence which sums to 1111. Let ΔdsuperscriptΔ ^dΔitalic_d be the d-dimensional simplex. Note that =⋃d=1∞Δdsubscriptsuperscriptsubscript1superscriptΔP_Y= _d=1^∞ ^dPcaligraphic_Y = ⋃d = 1∞ Δitalic_d, where ΔdsuperscriptΔ ^dΔitalic_d is viewed as a subset of Δd+1superscriptΔ1 ^d+1Δitalic_d + 1 via the inclusion Δd↪Δd+1:(p1,⋯,pd+1)↦(p1,⋯,pd+1,0,⋯):↪superscriptΔsuperscriptΔ1maps-tosubscript1⋯subscript1subscript1⋯subscript10⋯ ^d ^d+1:\;(p_1,·s,p_d+1) (p_1,% ·s,p_d+1,0,·s)Δitalic_d ↪ Δitalic_d + 1 : ( p1 , ⋯ , pitalic_d + 1 ) ↦ ( p1 , ⋯ , pitalic_d + 1 , 0 , ⋯ ). For each d, the topological dimension of the d-simplex ΔdsuperscriptΔ ^dΔitalic_d is d 555Theorem 5, https://personal.colby.edu/ sataylor/teaching/F14/MA331/TopologicalDimension.pdf; this is to say, d is the smallest number such that every open cover of ΔdsuperscriptΔ ^dΔitalic_d has an open refinement of order d+11d+1d + 1. Therefore, the smallest number n such that every open cover of ⋃d=1DΔdsuperscriptsubscript1superscriptΔ _d=1^D ^d⋃d = 1D Δitalic_d has an open refinement of order n+11n+1n + 1 is D. Hence, there is no finite N such that the any open cover of :=⋃d=1∞Δdassignsubscriptsuperscriptsubscript1superscriptΔP_Y:= _d=1^∞ ^dPcaligraphic_Y := ⋃d = 1∞ Δitalic_d has an open refinement with order N+11N+1N + 1. Therefore the topological dimension of subscriptP_YPcaligraphic_Y is ∞. When the maximum sequence length is l the cardinality of YY is finite and equal to ∑i=1lki=k⁢(1−kl)1−ksuperscriptsubscript1superscript1superscript1 _i=1^lk^i= k(1-k^l)1-k∑i = 1l kitalic_i = divide start_ARG k ( 1 - kitalic_l ) end_ARG start_ARG 1 - k end_ARG. subscriptP_YPcaligraphic_Y thus contains the set of positive sequences of length k⁢(1−kl)1−k1superscript1 k(1-k^l)1-kdivide start_ARG k ( 1 - kitalic_l ) end_ARG start_ARG 1 - k end_ARG which sum to 1, so subscriptP_YPcaligraphic_Y is a k⁢(1−kl)1−k−11superscript11 k(1-k^l)1-k-1divide start_ARG k ( 1 - kitalic_l ) end_ARG start_ARG 1 - k end_ARG - 1-dimensional simplex, and therefore the topological dimension of subscriptP_YPcaligraphic_Y is k⁢(1−kl)1−k−1=O⁢(kl)1superscript11superscript k(1-k^l)1-k-1=O k^l start_ARG k ( 1 - kitalic_l ) end_ARG start_ARG 1 - k end_ARG - 1 = O ( kitalic_l ). ∎ Appendix B Proof of Theorem 3 Lemma 2 Under Condition 1, π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is a well-defined function. Proof. Inverse of non-injective functions. In general, unless a function is injective 666Injective essentially means one-to-one. Formally, a function f is injective if f⁢(1)≠f⁢(2)subscript1subscript2f x_1 ≠ f x_2 ( italic_x1 ) ≠ f ( italic_x2 ) whenever 1≠2subscript1subscript2 x_1≠ x_2italic_x1 ≠ italic_x2. , its inverse is not a function, but only a set map. For instance, since π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG is many-to-one, so not injective, π~−1superscript~1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1over~ start_ARG π end_ARG- 1 would take an element from subscriptP_YPcaligraphic_Y and return a subset of XX, rather than a single element. Let π~−1|π~⁢()evaluated-atsuperscript~1~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) denote the inverse of π~−1superscript~1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1over~ start_ARG π end_ARG- 1 restricted to its image π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X ). π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is a well-defined function. It follows that π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) takes a point PYsubscriptP_YPitalic_Y in the image of π~~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 πover~ start_ARG π end_ARG, π~⁢()~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π ~ start_ARG π end_ARG ( X ), map it to its preimage π~−1⁢(PY)superscript~1subscript [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1 P_Y ~ start_ARG π end_ARG- 1 ( Pitalic_Y ), and map all points in the preimage π~−1⁢(PY)superscript~1subscript [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π^-1 P_Y ~ start_ARG π end_ARG- 1 ( Pitalic_Y ) through π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† to π†⁢(π~−1⁢(PY))superscript†superscript~1subscript [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1 P_Y % π† ( over~ start_ARG π end_ARG- 1 ( Pitalic_Y ) ). For any two points 1,2∈π~−1⁢(PY)subscript1subscript2superscript~1subscript x_1,\; x_2∈ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % π^-1 P_Y _x1 , italic_x2 ∈ over~ start_ARG π end_ARG- 1 ( Pitalic_Y ), we have π~⁢(1)=π~⁢(2)~subscript1~subscript2 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π x_1 % = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % x_2 ~ start_ARG π end_ARG ( italic_x1 ) = over~ start_ARG π end_ARG ( italic_x2 ), and then Condition 1 implies π†⁢(1)=π†⁢(2)superscript†subscript1superscript†subscript2 [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π x_1 =% [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π x_2 π† ( italic_x1 ) = π† ( italic_x2 ). Therefore, for any PY∈subscriptsubscriptP_Y _YPitalic_Y ∈ Pcaligraphic_Y, π†⁢(π~−1⁢(PY))superscript†superscript~1subscript [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1 P_Y % π† ( over~ start_ARG π end_ARG- 1 ( Pitalic_Y ) ) is a set containing exactly one element, so π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is a well-defined function. ∎ Theorem 3 We work under Assumptions 1, 2, 3 and 4. For some D, there exists a Lipschitz invertible function ϕ~:→π~⁢():~italic-ϕ→~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ:V→ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π ~ start_ARG ϕ end_ARG : V → over~ start_ARG π end_ARG ( X ), Θ~∈ℝN×(D+1)~Θsuperscriptℝ1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^N×(D+1)over~ start_ARG Θ end_ARG ∈ blackboard_RN × ( D + 1 ) and τ~∘:→ΔD:superscript~→superscriptΔ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ :X→ % ^Dover~ start_ARG τ end_ARG∘ : X → Δitalic_D s.t. π~=ϕ~∘Θ~⁢τ~∘~~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π= [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG π end_ARG = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘, and there is a Lipschitz continuous function π¯†:ΔD→ΔD:superscript¯†→superscriptΔsuperscriptΔ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π : ^D→ ^Dover¯ start_ARG π end_ARG† : Δitalic_D → Δitalic_D s.t. π†=ϕ~∘Θ~⁢π¯†∘τ~superscript†~italic-ϕ~Θsuperscript¯†~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τπ† = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG† ∘ over~ start_ARG τ end_ARG. Proof of Proposition 3. Step 1. Show that there exists Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG, τ~∘superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ over~ start_ARG τ end_ARG∘ and Lipschitz ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG such that π~=ϕ~∘Θ~⁢τ~∘~~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π= [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG π end_ARG = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘. By Condition 3, there is some invertible Lϕsubscriptitalic-ϕL_φLitalic_ϕ-Lipschitz function ϕ~:→π~⁢():~italic-ϕ→~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ:V→ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π ~ start_ARG ϕ end_ARG : V → over~ start_ARG π end_ARG ( X ), where VV is some convex polygon. Therefore, there is a finite set D+1=dd=1D+1subscript1superscriptsubscriptsubscript11V_D+1=\ v_d\_d=1^D+1Vitalic_D + 1 = italic_vitalic_d d = 1D + 1 such that every ∈ v _v ∈ V can be expressed as =∑d=1D+1pd⁢dsuperscriptsubscript11subscriptsubscript v= _d=1^D+1p_d v_ditalic_v = ∑d = 1D + 1 pitalic_d italic_vitalic_d for some ∈ΔDsuperscriptΔ p∈ ^Ditalic_p ∈ Δitalic_D. Let Θ~∈ℝN×(D+1)~Θsuperscriptℝ1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^N×(D+1)over~ start_ARG Θ end_ARG ∈ blackboard_RN × ( D + 1 ) be the matrix such that its d-th column, Θ~:,dsubscript~Θ: [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _:,dover~ start_ARG Θ end_ARG: , d, is equal to dsubscript v_ditalic_vitalic_d. Then every ∈ v _v ∈ V can be written as :=Θ~⁢assign~Θ v:= [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 pitalic_v := over~ start_ARG Θ end_ARG italic_p for some ∈ΔDsuperscriptΔ p∈ ^Ditalic_p ∈ Δitalic_D. Since ϕ~−1∘π~superscript~italic-ϕ1~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ^-1 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 πover~ start_ARG ϕ end_ARG- 1 ∘ over~ start_ARG π end_ARG is a function →X → V, then for every x, ϕ~−1∘π~⁢(x)superscript~italic-ϕ1~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ^-1 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π x ~ start_ARG ϕ end_ARG- 1 ∘ over~ start_ARG π end_ARG ( x ) is in VV. Therefore, there exists some x∈ΔDsubscriptsuperscriptΔ p_x∈ ^Ditalic_pitalic_x ∈ Δitalic_D such that ϕ~−1∘π~⁢(x)superscript~italic-ϕ1~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1 % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π x ~ start_ARG ϕ end_ARG- 1 ∘ over~ start_ARG π end_ARG ( x ) =Θ~⁢xabsent~Θsubscript = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 p_x= over~ start_ARG Θ end_ARG italic_pitalic_x (24) Let τ~∘:→ΔD:superscript~→superscriptΔ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ :X→ % ^Dover~ start_ARG τ end_ARG∘ : X → Δitalic_D be s.t. τ~∘⁢(x)superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x ~ start_ARG τ end_ARG∘ ( x ) =xabsentsubscript = p_x= italic_pitalic_x (25) then ϕ~−1∘π~⁢(x)superscript~italic-ϕ1~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1 % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π x ~ start_ARG ϕ end_ARG- 1 ∘ over~ start_ARG π end_ARG ( x ) =Θ~⁢τ~∘⁢(x)absent~Θsuperscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x = over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘ ( x ) (26) π~⁢(x)~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π x ~ start_ARG π end_ARG ( x ) =ϕ~∘Θ~⁢τ~∘⁢(x)absent~italic-ϕ~Θsuperscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘ ( x ) (27) Step 2. Let τ~⁢(x)=Θ~⁢τ~∘⁢(x)~~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ x =% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x ~ start_ARG τ end_ARG ( x ) = over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘ ( x ). We show that under the shared-level-sets assumption, π†∘τ~−1|τ~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) is well-defined. We have the following equalities: π~−1|π~⁢()evaluated-atsuperscript~1~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) =(ϕ~∘τ~)−1|π~⁢()absentevaluated-atsuperscript~italic-ϕ~1~ = [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % ^-1|_ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π % X = ( over~ start_ARG ϕ end_ARG ∘ over~ start_ARG τ end_ARG )- 1 |over~ start_ARG π end_ARG ( X ) (28) =τ~−1|∘ϕ~−1absentevaluated-atsuperscript~1superscript~italic-ϕ1 = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ^-1|_% V [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1= over~ start_ARG τ end_ARG- 1 |V ∘ over~ start_ARG ϕ end_ARG- 1 (29) π~−1|π~⁢()∘ϕ~evaluated-atsuperscript~1~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φover~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) ∘ over~ start_ARG ϕ end_ARG =τ~−1|absentevaluated-atsuperscript~1 = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ^-1|_% V= over~ start_ARG τ end_ARG- 1 |V (30) =τ~−1|ϕ~−1⁢(π~⁢())absentevaluated-atsuperscript~1superscript~italic-ϕ1~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ^-1 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % = over~ start_ARG τ end_ARG- 1 |over~ start_ARG ϕ end_ARG- 1 ( over~ start_ARG π end_ARG ( X ) ) (31) =τ~−1|τ~⁢()absentevaluated-atsuperscript~1~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ = over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) (32) Therefore, π~−1|π~⁢()∘ϕ~evaluated-atsuperscript~1~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φover~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) ∘ over~ start_ARG ϕ end_ARG =τ~−1|τ~⁢()absentevaluated-atsuperscript~1~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ^-1|_% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ = over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) (33) π†∘π~−1|π~⁢()∘ϕ~evaluated-atsuperscript†superscript~1~~italic-ϕ [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φπ† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) ∘ over~ start_ARG ϕ end_ARG =π†∘τ~−1|τ~⁢()absentevaluated-atsuperscript†superscript~1~ = [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ = π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) (34) By Condition 1 and Lemma 2, π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is well-defined. Since π†∘π~−1|π~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) is well-defined, so is π†∘τ~−1|τ~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ). Step 3. Show that under Assumptions 2, 3 and 4, π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† can be decomposed as ϕ~∘ψ∘τ~~italic-ϕ~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ ψ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τover~ start_ARG ϕ end_ARG ∘ ψ ∘ over~ start_ARG τ end_ARG for some Lipschitz function ψ:τ~⁢()→ϕ~−1⁢(π†⁢()):→~superscript~italic-ϕ1superscript†ψ: [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % X → [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % φ^-1 [rgb]0,0.3828125,0.45703125 [named% ]pgfstrokecolorrgb0,0.3828125,0.45703125π % X ψ : over~ start_ARG τ end_ARG ( X ) → over~ start_ARG ϕ end_ARG- 1 ( π† ( X ) ). Note that 1. ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG is invertible restricted to its image, 2. by Condition 2 and 3 ϕ~−1superscript~italic-ϕ1 [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ^-1over~ start_ARG ϕ end_ARG- 1 is defined on the image of π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π†, and 3. π†∘τ~−1|τ~⁢()evaluated-atsuperscript†superscript~1~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) is well-defined. Therefore, we can factor π†superscript† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π π† as π†superscript† [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π π† =ϕ~∘ϕ~−1∘π†∘τ~−1|τ~⁢()∘τ~absentevaluated-at~italic-ϕsuperscript~italic-ϕ1superscript†superscript~1~~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ^-1 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ= over~ start_ARG ϕ end_ARG ∘ over~ start_ARG ϕ end_ARG- 1 ∘ π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) ∘ over~ start_ARG τ end_ARG (36) Therefore, define: ψ:τ~⁢():~ ψ: [rgb]0.609375,0.1328125,0.1484375 [named% ]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % ψ : over~ start_ARG τ end_ARG ( X ) →ϕ~−1⁢(π†⁢())⊂τ~⁢(X)→absentsuperscript~italic-ϕ1superscript†~ → [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1 % [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolor% rgb0,0.3828125,0.45703125π % ⊂ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ X → over~ start_ARG ϕ end_ARG- 1 ( π† ( X ) ) ⊂ over~ start_ARG τ end_ARG ( X ) (37) ψ:=assignabsent ψ:=ψ := ϕ~−1∘π†∘τ~−1|τ~⁢()evaluated-atsuperscript~italic-ϕ1superscript†superscript~1~ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1 % [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ % over~ start_ARG ϕ end_ARG- 1 ∘ π† ∘ over~ start_ARG τ end_ARG- 1 |over~ start_ARG τ end_ARG ( X ) (38) = == ϕ~−1∘π†∘π~−1|π~⁢()∘ϕ~evaluated-atsuperscript~italic-ϕ1superscript†superscript~1~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ^-1 % [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π^-1|_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG- 1 ∘ π† ∘ over~ start_ARG π end_ARG- 1 |over~ start_ARG π end_ARG ( X ) ∘ over~ start_ARG ϕ end_ARG (39) is a composition of Lipschitz functions (by Assumptions 3 and 4) so is Lipschitz. Step 4. Finally show the assertion, that π†=ϕ~∘Θ~⁢π¯†∘τ~∘superscript†~italic-ϕ~Θsuperscript¯†superscript~ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125π = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ π† = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG† ∘ over~ start_ARG τ end_ARG∘ for some Lipschitz π¯†:ΔD→ΔD:superscript¯†→superscriptΔsuperscriptΔ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π : ^D→ ^Dover¯ start_ARG π end_ARG† : Δitalic_D → Δitalic_D. Substituting in τ~⁢(x)=Θ~⁢τ~∘⁢(x)~~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ x =% [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x ~ start_ARG τ end_ARG ( x ) = over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘ ( x ), we obtain: π†⁢(x)superscript† [rgb]0,0.3828125,0.45703125 [named]% pgfstrokecolorrgb0,0.3828125,0.45703125π x π† ( x ) =ϕ~∘ψ∘Θ~⁢τ~∘⁢(x)absent~italic-ϕ~Θsuperscript~ = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ ψ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x = over~ start_ARG ϕ end_ARG ∘ ψ ∘ over~ start_ARG Θ end_ARG over~ start_ARG τ end_ARG∘ ( x ) (40) Let x=τ∘⁢(x)∈ΔDsubscriptsuperscriptsuperscriptΔ p_x=τ x ∈ ^Ditalic_pitalic_x = τ∘ ( x ) ∈ Δitalic_D. We want to show that there is a Lipschitz continuous function π¯†:ΔD→ΔD:superscript¯†→superscriptΔsuperscriptΔ [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π : ^D→ ^Dover¯ start_ARG π end_ARG† : Δitalic_D → Δitalic_D such that ψ⁢(Θ~⁢x)~Θsubscript ψ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % p_x ψ ( over~ start_ARG Θ end_ARG italic_pitalic_x ) =Θ~⁢π¯†⁢(x)absent~Θsuperscript¯†subscript = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 [rgb]% 0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π p_x = over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG† ( italic_pitalic_x ) (41) For a given xsubscript p_xitalic_pitalic_x, we can try to solve the linear system in terms of π¯†⁢()superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π p ¯ start_ARG π end_ARG† ( italic_p ). We know that it must be an element of the set: Θ~+⁢ψ⁢(Θ~⁢x)+Ker⁡(Θ~),superscript~Θ~ΘsubscriptKer~Θ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 ^+ψ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 p_x% +Ker [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 ,over~ start_ARG Θ end_ARG+ ψ ( over~ start_ARG Θ end_ARG italic_pitalic_x ) + Ker ( over~ start_ARG Θ end_ARG ) , (42) where Θ~+superscript~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^+over~ start_ARG Θ end_ARG+ denotes the pseudoinverse. Take the intersection between this set and ΔDsuperscriptΔ ^DΔitalic_D; the intersection is non-empty because ψ lands in VV. Now we describe a procedure to choose a point in this intersection that is Lipschitz continuous wrt xsubscript p_xitalic_pitalic_x: we let π¯†superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π over¯ start_ARG π end_ARG† map xsubscript p_xitalic_pitalic_x to the centroid of the intersection between ΔDsuperscriptΔ ^DΔitalic_D and Θ~+⁢ψ⁢(Θ~⁢x)+Ker⁡(Θ~)superscript~Θ~ΘsubscriptKer~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^+ψ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 p_x +% Ker [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % over~ start_ARG Θ end_ARG+ ψ ( over~ start_ARG Θ end_ARG italic_pitalic_x ) + Ker ( over~ start_ARG Θ end_ARG ). The intersection is one of two convex sets, so it is convex; so the centroid lie in this set. We now proceed to show that π¯†superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π over¯ start_ARG π end_ARG† is Lipschitz continuous, in two steps. 1. First we show that the centroid of the intersection is a smooth function of the location of its vertices. 2. Then we show that the location of the vertices is a piecewise smooth function of xsubscript p_xitalic_pitalic_x. We show the centroid of the intersection is a generically smooth function of the location of its vertices. Note that the intersection of ΔDsuperscriptΔ ^DΔitalic_D and Θ~⁢ψ⁢(Θ~⁢x)+Ker⁡(Θ~)~Θ~ΘsubscriptKer~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ψ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 p_x +% Ker [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % over~ start_ARG Θ end_ARG ψ ( over~ start_ARG Θ end_ARG italic_pitalic_x ) + Ker ( over~ start_ARG Θ end_ARG ) is a convex high-dimensional polyhedron, denote it ⁢(x)subscriptS p_x ( italic_pitalic_x ). The centroid of a convex high-dimensional polyhedron can be computed as follows: every convex polyhedron admits a triangulation. Let the triangulation of ⁢(x)subscriptS p_x ( italic_pitalic_x ), denote it by T⁡(⁢(x))TsubscriptT p_x ( S ( italic_pitalic_x ) ). For the iiith simplex in T⁡(⁢(x))TsubscriptT p_x ( S ( italic_pitalic_x ) ), take its vertices i⁢0,⋯,i⁢nsubscript0⋯subscript \ v_i0,·s, x_in \ italic_vitalic_i 0 , ⋯ , italic_xitalic_i n where n is the dimension of the polyhedron. The centroid of the simplex is given by C⁡(i)=i⁢0+⋯+i⁢n+1Csubscript0⋯subscript1C i = v_i0+·s+ v_% inn+1C ( i ) = divide start_ARG italic_vitalic_i 0 + ⋯ + italic_vitalic_i n end_ARG start_ARG n + 1 end_ARG, and the volume Vol⁡(i)VolVol i ( i ) is given by 1n!⁢|(0⋯n1⋯1)|1matrixsubscript0⋯subscript1⋯1 1n! pmatrix v_0&·s& v_n\\ 1&·s&1 pmatrix start_ARG 1 end_ARG start_ARG n ! end_ARG | ( start_ARG start_ROW start_CELL italic_v0 end_CELL start_CELL ⋯ end_CELL start_CELL italic_vitalic_n end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL ⋯ end_CELL start_CELL 1 end_CELL end_ROW end_ARG ) |. The centroid of ⁢(x)subscriptS p_x ( italic_pitalic_x ), denoted C⁡(⁢(x))CsubscriptC p_x ( S ( italic_pitalic_x ) ) is given by ∑iC⁡(i)⁢Vol⁡(i)subscriptCVol _iC i % i ∑i C ( i ) Vol ( i ); since the determinant function is a polynomial in the matrix entries, this is a vector field where each entry is a polynomial. Moreover, every vertex in the triangulation but is not on ⁢(x)subscriptS p_x ( italic_pitalic_x ) is in the interior of ⁢(x)subscriptS p_x ( italic_pitalic_x ). There is some ϵitalic-ϵεϵ small enough such that we can draw an ϵitalic-ϵεϵ-ball around each such vertex such that the closure of the balls are all disjoint and still lie in the polyhedron. So, perturb move each vertex to a point on the boundary of its ball; this gives a new triangulation, but the centroid is not changed. Since we can do it for any ϵ′≤ϵsuperscriptitalic-ϵ′italic-ϵε ≤εϵ′ ≤ ϵ, C⁡(⁢(x))CsubscriptC p_x ( S ( italic_pitalic_x ) ) is constant in the interior vertices. Therefore, C⁡(⁢(x))CsubscriptC p_x ( S ( italic_pitalic_x ) ) is a polynomial of its vertices. The limiting case is when two vertices overlap; in this case, at least one element in the triangulation will collapse onto a face, which has volume zero, so its centroid will not contribute to the calculation of C⁡(⁢(x))CsubscriptC p_x ( S ( italic_pitalic_x ) ) through the formula. Therefore, the centroid of a polyhedron is a polynomial of its vertices, including when two or more vertices overlap. We show that the location of the vertices is a piecewise-smooth function of xsubscript p_xitalic_pitalic_x. Note that the set of points in ⁢(x)subscriptS p_x ( italic_pitalic_x ) is described as follows: Suppose dim⁡(⁢())=J≤Ddimdim p % =J≤ Ddim ( S ( italic_p ) ) = J ≤ D, and choose an orthonormal basis in ℝD+1superscriptℝ1R^D+1blackboard_RD + 1 whose span contains the direction vectors in ⁢()S p ( italic_p ): 1,⋯,Jsubscript1⋯subscript b_1,·s, b_Jitalic_b1 , ⋯ , italic_bitalic_J (43) Extend this to an orthonormal basis whose span contains ΔDsuperscriptΔ ^DΔitalic_D: 1,⋯,J,J+1,⋯,Dsubscript1⋯subscriptsubscript1⋯subscript b_1,·s, b_J, b_J+1,·s, b_Ditalic_b1 , ⋯ , italic_bitalic_J , italic_bitalic_J + 1 , ⋯ , italic_bitalic_D (44) And finally extend this to ℝDsuperscriptℝR^Dblackboard_RD: 1,⋯,J,J+1,⋯,D,D+1subscript1⋯subscriptsubscript1⋯subscriptsubscript1 b_1,·s, b_J, b_J+1,·s, b_D, b% _D+1italic_b1 , ⋯ , italic_bitalic_J , italic_bitalic_J + 1 , ⋯ , italic_bitalic_D , italic_bitalic_D + 1 (45) So we can express ⁢(x)=∈ℝD+1|si≥0,∑isi=1,⁢=⁢(Θ~+⁢ψ⁢(Θ~⁢)),subscriptconditional-setsuperscriptℝ1formulae-sequencesubscript0formulae-sequencesubscriptsubscript1superscript~Θ~Θ p_x = \ s∈% R^D+1\; |\;s_i≥ 0, _is_i=1, A s= A% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 ^+ψ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 p % \,S ( italic_pitalic_x ) = italic_s ∈ blackboard_RD + 1 | sitalic_i ≥ 0 , ∑i sitalic_i = 1 , italic_A italic_s = italic_A ( over~ start_ARG Θ end_ARG+ ψ ( over~ start_ARG Θ end_ARG italic_p ) ) , (46) where =(D+1−J⁢J⊤)subscript1subscriptsuperscriptsubscripttop A= I_D+1- B_J B_J italic_A = ( italic_Iitalic_D + 1 - italic_Bitalic_J italic_Bitalic_J⊤ ), Bitalic_B is the matrix whose rows are the D+11D+1D + 1 basis vectors, and Jsubscript B_Jitalic_Bitalic_J is the one taking its first J rows. Note that D+1−J⁢J⊤=−J⁢−J⊤subscript1subscriptsuperscriptsubscripttopsubscriptsuperscriptsubscripttop I_D+1- B_J B_J = B_-J B_-J italic_Iitalic_D + 1 - italic_Bitalic_J italic_Bitalic_J⊤ = italic_B- J italic_B- J⊤ where −Jsubscript B_-Jitalic_B- J is the matrix containing J+1,⋯,D+1subscript1⋯subscript1 b_J+1,·s, b_D+1italic_bitalic_J + 1 , ⋯ , italic_bitalic_D + 1 as rows. But 1 11 is orthogonal to the row space of J⊤superscriptsubscripttop B_J italic_Bitalic_J⊤, so it is contained in the row space of −J⊤superscriptsubscripttop B_-J italic_B- J⊤, and hence −J⁢−J⊤subscriptsuperscriptsubscripttop B_-J B_-J italic_B- J italic_B- J⊤. Therefore, we can remove ∑isi=1subscriptsubscript1 _is_i=1∑i sitalic_i = 1 from the set of conditions. Therefore, the set of conditions contains D+1−J1D+1-JD + 1 - J linearly independent conditions and D+11D+1D + 1 inequalities. An extrema, i.e. a vertex, is the solution of D+11D+1D + 1 linearly independent equations where all D+1−J1D+1-JD + 1 - J linearly independent equality constraints are included, together with J equations from saturating the inequality constraints. Since D+1−J⁢J⊤subscript1subscriptsuperscriptsubscripttop I_D+1- B_J B_J italic_Iitalic_D + 1 - italic_Bitalic_J italic_Bitalic_J⊤ has rank D+1−J1D+1-JD + 1 - J, there is a subset of D+1−J1D+1-JD + 1 - J rows, call the new matrix constructed from these rows ¯D+1−Jsubscript¯1 B_D+1-Jover¯ start_ARG italic_B end_ARGD + 1 - J. Select J vectors from the standard basis which are linearly independent of the rows of ¯D+1−Jsubscript¯1 B_D+1-Jover¯ start_ARG italic_B end_ARGD + 1 - J, and stack them into an invertible matrix Citalic_C. Then any vertex is a solution of one such equation ∗superscript v^*italic_v∗ =−1⁢(¯D+1−J⁢(Θ~+⁢ψ⁢(Θ~⁢x)))absentsuperscript1matrixsubscript¯1superscript~Θ~Θsubscript0 = C^-1 pmatrix B_D+1-J % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 ^+ψ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 p_x % \\ 0 pmatrix= italic_C- 1 ( start_ARG start_ROW start_CELL over¯ start_ARG italic_B end_ARGD + 1 - J ( over~ start_ARG Θ end_ARG+ ψ ( over~ start_ARG Θ end_ARG italic_pitalic_x ) ) end_CELL end_ROW start_ROW start_CELL 0 end_CELL end_ROW end_ARG ) (47) provided it still satisfies the remaining inequality constraints. Here it is clear that any ∗superscript v^*italic_v∗ varies smoothly with xsubscript p_xitalic_pitalic_x. When all vertices of ⁢(x)subscriptS p_x ( italic_pitalic_x ) are in the interior of a 1111-dimensional face of ΔDsuperscriptΔ ^DΔitalic_D, they vary locally smoothly with xsubscript p_xitalic_pitalic_x. Since the centroid of ⁢(x)subscriptS p_x ( italic_pitalic_x ) varies smoothly with its vertices, whenever its vertices vary smoothly with xsubscript p_xitalic_pitalic_x, the centroid also vary locally smoothly with xsubscript p_xitalic_pitalic_x. The only non-differentiability happens when one vertex moves out of ΔDsuperscriptΔ ^DΔitalic_D and another moves in. But around those points of xsubscript p_xitalic_pitalic_x the centroid is still continuous wrt xsubscript p_xitalic_pitalic_x , so π¯†superscript¯† [rgb]0,0.3828125,0.45703125 [named]pgfstrokecolorrgb% 0,0.3828125,0.45703125 π over¯ start_ARG π end_ARG† is piecewise differentiable function on a compact domain, and therefore is Lipschitz continuous. ∎ Appendix C Convergence rates proofs Lemma 4. (rπsubscriptr_πritalic_π helps define a metric) Define rπ⁢(,)=β⁢log⁡π(|)πref(|)r_π x, y =β π % y\> |\> x _ref y\>% |\> x ritalic_π ( italic_x , italic_y ) = β log divide start_ARG π ( italic_y | italic_x ) end_ARG start_ARG πref ( italic_y | italic_x ) end_ARG for some fixed constant β>00β>0β > 0, and let dr⁢(π,π′)=‖rπ−rπ′‖∞subscriptsuperscript′subscriptnormsubscriptsubscriptsuperscript′d_r π,π =\|r_π-r_π \|_∞ditalic_r ( π , π′ ) = ∥ ritalic_π - ritalic_π′ ∥∞ (48) Then dr⁢(⋅,⋅)subscript⋅d_r ·,· _r ( ⋅ , ⋅ ) defines a metric over any set Π Π of functions ×→[0,1]→01X×Y→[0,1]X × Y → [ 0 , 1 ] s.t. ∀∑π(|)=1∀ x _Yπ y\> |\> x % =1∀ italic_x ∑Y π ( italic_y | italic_x ) = 1, and satisfies |rπ⁢(,)|≤Csubscript r_π x, y ≤ C| ritalic_π ( italic_x , italic_y ) | ≤ C over XX and YY. Proof of Lemma 4. drsubscriptd_rditalic_r is well-defined on Π Π since for any π,π′∈Πsuperscript′Ππ,π ∈ π , π′ ∈ Π, dr⁢(π,π′)≤∥rπ∥∞+∥rπ′∥∞≤2⁢C<∞subscriptsuperscript′subscriptdelimited-∥subscriptsubscriptdelimited-∥subscriptsuperscript′2d_r π,π ≤ r_π _∞% + r_π _∞≤ 2C<∞ditalic_r ( π , π′ ) ≤ ∥ ritalic_π ∥∞ + ∥ ritalic_π′ ∥∞ ≤ 2 C < ∞. We can verify that d is a metric on Π Π. Clearly, symmetry and positivity holds, and d⁢(π,π′)=0⇔π=π′iffsuperscript′0superscript′d π,π =0 π=π d ( π , π′ ) = 0 ⇔ π = π′, so we just need to check triangle inequality. Fix π′∈Πsuperscript′Ππ ∈ π′ ′ ∈ Π, dr⁢(π,π′)+dr⁢(π′,π′)subscriptsuperscript′subscriptsuperscript′ d_r π,π +d_r % π ,π _r ( π , π′ ′ ) + ditalic_r ( π′ , π′ ′ ) =‖rπ−rπ′‖∞+‖rπ′−rπ′‖∞absentsubscriptnormsubscriptsubscriptsuperscript′subscriptnormsubscriptsuperscript′subscriptsuperscript′ =\|r_π-r_π \|_∞+\|r_π -r% _π \|_∞= ∥ ritalic_π - ritalic_π′ ′ ∥∞ + ∥ ritalic_π′ - ritalic_π′ ′ ∥∞ (49) ≥‖rπ−rπ′−rπ′+rπ′‖∞absentsubscriptnormsubscriptsubscriptsuperscript′subscriptsuperscript′subscriptsuperscript′ ≥\|r_π-r_π -r_π +r_π^% \|_∞≥ ∥ ritalic_π - ritalic_π′ ′ - ritalic_π′ + ritalic_π′ ′ ∥∞ (50) =‖rπ−rπ′‖∞absentsubscriptnormsubscriptsubscriptsuperscript′ =\|r_π-r_π \|_∞= ∥ ritalic_π - ritalic_π′ ∥∞ (51) =dr⁢(π,π′)absentsubscriptsuperscript′ =d_r π,π = ditalic_r ( π , π′ ) (52) ∎ Proposition 7 (Concentration bound). Let G be a measure on (X,Yw,Yl)subscriptsubscript(X,Y_w,Y_l)( X , Yitalic_w , Yitalic_l ) and for any π∈Π⊆Π̊Π̊Ππ∈ π ∈ Π ⊆ over̊ start_ARG Π end_ARG (Def. 1) let RG⁢(π)=G⁢[log⁡σ⁢(β⁢log⁡π⁢(Yw|X)πref⁢(Yw|X)−β⁢log⁡π⁢(Yl|X)πref⁢(Yl|X))]subscriptsubscriptdelimited-[]conditionalsubscriptsubscriptrefconditionalsubscriptconditionalsubscriptsubscriptrefconditionalsubscriptR_G π =E_G [ σ (β% π Y_w|X _ref Y% _w|X -β π Y_l|X _% ref Y_l|X ) ]Ritalic_G ( π ) = blackboard_EG [ log σ ( β log divide start_ARG π ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) ] (53) be the preference loss. Further, let (Xi,Yw,i,Yl,i)i=1nsuperscriptsubscriptsubscriptsubscriptsubscript1(X_i,Y_w,i,Y_l,i)_i=1^n( Xitalic_i , Yitalic_w , i , Yitalic_l , i )i = 1n be i.i.d. samples from G, and let G^nsubscript G_nover start_ARG G end_ARGn denote the empirical measure given by the samples, then P⁢(supπ∈Π|RG⁢(π)−RG^n⁢(π)|≥ϵ)≤2⁢infα∈(0,1)Cov⁢(Π,dr⁢(⋅,⋅),α⁢ϵ4)⁢e−2⁢(1−α)2⁢n⁢ϵ24⁢C2subscriptsupremumΠsubscriptsubscriptsubscript^italic-ϵ2subscriptinfimum01CovΠsubscript⋅italic-ϵ4superscript2superscript12superscriptitalic-ϵ24superscript2P ( _π∈ |R_G π -R_ G_n% π |≥ε )≤ 2 _α∈ % 0,1 Cov ,d_r(·,·), % αε4 e^- 2 1-α ^% 2nε^24C^2P ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ 2 infitalic_α ∈ ( 0 , 1 ) Cov ( Π , ditalic_r ( ⋅ , ⋅ ) , divide start_ARG α ϵ end_ARG start_ARG 4 end_ARG ) e- divide start_ARG 2 ( 1 - α ) start_POSTSUPERSCRIPT 2 n ϵ2 end_ARG start_ARG 4 C2 end_ARG end_POSTSUPERSCRIPT (54) where dr⁢(π,π′)=‖rπ−rπ′‖∞subscriptsuperscript′subscriptnormsubscriptsubscriptsuperscript′d_r(π,π )=\|r_π-r_π \|_∞ditalic_r ( π , π′ ) = ∥ ritalic_π - ritalic_π′ ∥∞. Proof of Proposition 7.. Adapted from Elesedy (2022). Fix π,π′∈Πsuperscript′Ππ,π ∈ π , π′ ∈ Π. |RG⁢(π)−RG⁢(π′)|=subscriptsubscriptsuperscript′absent R_G π -R_G % π =| Ritalic_G ( π ) - Ritalic_G ( π′ ) | = |G[logσ(βlogπ⁢(Yw|X)πref⁢(Yw|X)−βlogπ⁢(Yl|X)πref⁢(Yl|X)) |E_G [ σ β % π Y_w|X _ref Y_w% |X -β π Y_l|X _% ref Y_l|X | blackboard_EG [ log σ ( β log divide start_ARG π ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) −logσ(βlogπ′⁢(Yw|X)πref⁢(Yw|X)−βlogπ′⁢(Yl|X)πref⁢(Yl|X))]| \;\;\;\;\;\;\;\;- σ β π % Y_w|X _ref Y_w|X % -β π Y_l|X _% ref Y_l|X ] |- log σ ( β log divide start_ARG π′ ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π′ ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) ] | (55) ≤ ≤ G[|logσ(βlogπ⁢(Yw|X)πref⁢(Yw|X)−βlogπ⁢(Yl|X)πref⁢(Yl|X)) _G [ | σ β % π Y_w|X _ref Y_w% |X -β π Y_l|X _% ref Y_l|X _EG [ | log σ ( β log divide start_ARG π ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) −logσ(βlogπ′⁢(Yw|X)πref⁢(Yw|X)−βlogπ′⁢(Yl|X)πref⁢(Yl|X))|] \;\;\;\;\;\;\;\;- σ β π % Y_w|X _ref Y_w|X % -β π Y_l|X _% ref Y_l|X | ]- log σ ( β log divide start_ARG π′ ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π′ ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) | ] (56) When σ is the sigmoid, log⁡σ σ is concave, so the above is upper bounded: ≤ ≤ G[|(βlogπ⁢(Yw|X)πref⁢(Yw|X)−βlogπ⁢(Yl|X)πref⁢(Yl|X)) _G [ | β π % Y_w|X _ref Y_w|X % -β π Y_l|X _ref% Y_l|X _EG [ | ( β log divide start_ARG π ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) −(βlogπ′⁢(Yw|X)πref⁢(Yw|X)−βlogπ′⁢(Yl|X)πref⁢(Yl|X))|] \;\;\;\;\;\;\;\;- β π % Y_w|X _ref Y_w|X % -β π Y_l|X _% ref Y_l|X | ]- ( β log divide start_ARG π′ ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π′ ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) | ] (57) ≤ ≤ G[|(βlogπ⁢(Yw|X)πref⁢(Yw|X)−βlogπ′⁢(Yw|X)πref⁢(Yw|X)) _G [ | β π % Y_w|X _ref Y_w|X % -β π Y_w|X _% ref Y_w|X _EG [ | ( β log divide start_ARG π ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG - β log divide start_ARG π′ ( Yitalic_w | X ) end_ARG start_ARG πref ( Yitalic_w | X ) end_ARG ) −(βlogπ⁢(Yl|X)πref⁢(Yl|X)−βlogπ′⁢(Yl|X)πref⁢(Yl|X))|] \;\;\;\;\;\;\;\;- β π Y_l% |X _ref Y_l|X -β % π Y_l|X _ref % Y_l|X | ]- ( β log divide start_ARG π ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG - β log divide start_ARG π′ ( Yitalic_l | X ) end_ARG start_ARG πref ( Yitalic_l | X ) end_ARG ) | ] (58) ≤ ≤ G⁡[|(rπ⁢(X,Yw)−rπ′⁢(X,Yw))−(rπ⁢(X,Yl)−rπ′⁢(X,Yl))|]subscriptsubscriptsubscriptsubscriptsuperscript′subscriptsubscriptsubscriptsubscriptsuperscript′subscript E_G [ (r_π(X,Y_w)-r% _π (X,Y_w))-(r_π(X,Y_l)-r_π (X,Y_l)) % ]blackboard_EG [ | ( ritalic_π ( X , Yitalic_w ) - ritalic_π′ ( X , Yitalic_w ) ) - ( ritalic_π ( X , Yitalic_l ) - ritalic_π′ ( X , Yitalic_l ) ) | ] (59) ≤ ≤ G⁡[|(rπ⁢(X,Yw)−rπ′⁢(X,Yw))|]+G⁡[|(rπ⁢(X,Yl)−rπ′⁢(X,Yl))|]subscriptsubscriptsubscriptsubscriptsuperscript′subscriptsubscriptsubscriptsubscriptsubscriptsuperscript′subscript E_G [ (r_π(X,Y_w)-r% _π (X,Y_w)) ]+ E_G% [ (r_π(X,Y_l)-r_π (X,Y_l)) ]blackboard_EG [ | ( ritalic_π ( X , Yitalic_w ) - ritalic_π′ ( X , Yitalic_w ) ) | ] + blackboard_EG [ | ( ritalic_π ( X , Yitalic_l ) - ritalic_π′ ( X , Yitalic_l ) ) | ] (60) ≤ ≤ 2⁢∥rπ−rπ′∥∞2subscriptdelimited-∥subscriptsubscriptsuperscript′ 2 r_π-r_π _∞2 ∥ ritalic_π - ritalic_π′ ∥∞ (61) Now let G^nsubscript G_nover start_ARG G end_ARGn be the empirical measure of (X,Yw,Yl)subscriptsubscript(X,Y_w,Y_l)( X , Yitalic_w , Yitalic_l ) with n samples. And define LG^n⁢(π)subscriptsubscript L_ G_n π start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) =RG^n⁢(π)−RG⁢(π)absentsubscriptsubscript^subscript =R_ G_n π -R_G π = Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Ritalic_G ( π ) (62) Then |LG^n⁢(π)−LG^n⁢(π′)|subscriptsubscript^subscriptsubscript^superscript′ L_ G_n π -L_ G% _n π | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π′ ) | =|RG^n⁢(π)−RG⁢(π)−RG^n⁢(π′)+RG⁢(π′)|absentsubscriptsubscript^subscriptsubscriptsubscript^superscript′subscriptsuperscript′ = R_ G_n π -R_G% π -R_ G_n π % +R_G π = | Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π′ ) + Ritalic_G ( π′ ) | (63) ≤|RG^n⁢(π)−RG^n⁢(π′)+RG⁢(π′)−RG⁢(π)|absentsubscriptsubscript^subscriptsubscript^superscript′subscriptsuperscript′subscript ≤ R_ G_n π -R_% G_n π +R_G π % -R_G π ≤ | Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π′ ) + Ritalic_G ( π′ ) - Ritalic_G ( π ) | (64) ≤|RG^n⁢(π)−RG^n⁢(π′)|+|RG⁢(π′)−RG⁢(π)|absentsubscriptsubscript^subscriptsubscript^superscript′subscriptsuperscript′subscript ≤ R_ G_n π -R_% G_n π + R_G% π -R_G π ≤ | Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π′ ) | + | Ritalic_G ( π′ ) - Ritalic_G ( π ) | (65) ≤4⁢∥rπ−rπ′∥∞absent4subscriptdelimited-∥subscriptsubscriptsuperscript′ ≤ 4 r_π-r_π _∞≤ 4 ∥ ritalic_π - ritalic_π′ ∥∞ (66) So now let KK be a κ-cover of Π Π in dr⁢(⋅,⋅)subscript⋅d_r(·,·)ditalic_r ( ⋅ , ⋅ ). Define the sets D⁢(πk)=π∈Π:dr⁢(πk,π)≤κsubscriptconditional-setΠsubscriptsubscriptD _k =\π∈ \;:\;d_r( _k,π)≤κ\D ( πitalic_k ) = π ∈ Π : ditalic_r ( πitalic_k , π ) ≤ κ . Then P⁢(supπ∈Π|LG^n⁢(π)|≥ϵ)subscriptsupremumΠsubscriptsubscript^italic-ϵ P _π∈ L_ G_n % π ≥ε ( supitalic_π ∈ Π | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) =P⁢(⋃πk∈supπ∈D⁢(πk)|LG^n⁢(π)|≥ϵ)absentsubscriptsubscriptsubscriptsupremumsubscriptsubscriptsubscript^italic-ϵ =P _ _k \ _π% ∈ D _k L_ G_n % π ≥ε \ = P ( ⋃π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT supitalic_π ∈ D ( π start_POSTSUBSCRIPT k ) end_POSTSUBSCRIPT | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) (67) ≤∑πk∈P⁢(supπ∈D⁢(πk)|LG^n⁢(π)|≥ϵ)absentsubscriptsubscriptsubscriptsupremumsubscriptsubscriptsubscript^italic-ϵ ≤ _ _k P _π∈ D % _k L_ G_n π % ≥ε ≤ ∑π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT P ( supitalic_π ∈ D ( π start_POSTSUBSCRIPT k ) end_POSTSUBSCRIPT | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) (68) Set κ=α⁢ϵ4italic-ϵ4κ= αε4κ = divide start_ARG α ϵ end_ARG start_ARG 4 end_ARG for 0<α<1010<α<10 < α < 1. Using the above, for any π∈D⁢(πk)subscriptπ∈ D _k π ∈ D ( πitalic_k ) we have |LG^n⁢(π)−LG^n⁢(πk)|subscriptsubscript^subscriptsubscript^subscript L_ G_n π -L_ G% _n _k | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | ≤4⁢∥rπ−rπk∥∞absent4subscriptdelimited-∥subscriptsubscriptsubscript ≤ 4 r_π-r_ _k _∞≤ 4 ∥ ritalic_π - ritalic_π start_POSTSUBSCRIPT k end_POSTSUBSCRIPT ∥∞ (69) =4⁢dr⁢(πk,π)absent4subscriptsubscript =4d_r( _k,π)= 4 ditalic_r ( πitalic_k , π ) (70) ≤4⁢κabsent4 ≤ 4κ≤ 4 κ (71) By triangle inequality: |LG^n⁢(π)|−|LG^n⁢(πk)|≤|LG^n⁢(π)−LG^n⁢(πk)|subscriptsubscript^subscriptsubscript^subscriptsubscriptsubscript^subscriptsubscript^subscript L_ G_n π - % L_ G_n _k ≤ % L_ G_n π -L_ G_n % _k | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | - | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | ≤ | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) - Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | (72) So |LG^n⁢(π)|≤4⁢κ+|LG^n⁢(πk)|⁢|LG^n⁢(π)|≤α⁢ϵ+|LG^n⁢(πk)|subscriptsubscript^4subscriptsubscript^subscriptsubscriptsubscript^italic-ϵsubscriptsubscript^subscript L_ G_n π ≤ 4% κ+ L_ G_n _k % \\ L_ G_n π ≤α% ε+ L_ G_n _k | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≤ 4 κ + | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≤ α ϵ + | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | (73) Since the probability of the supremum of over a cover is greater than ϵitalic-ϵεϵ implies that the upper bound over the cover is greater than ϵitalic-ϵεϵ, we have that the probability of the latter is at least the probability of the former: P⁢(supπ∈Π|LG^n⁢(π)|≥ϵ)subscriptsupremumΠsubscriptsubscript^italic-ϵ P _π∈ L_ G_n % π ≥ε ( supitalic_π ∈ Π | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤∑πk∈P⁢(supπ∈D⁢(πk)|LG^n⁢(π)|≥ϵ)absentsubscriptsubscriptsubscriptsupremumsubscriptsubscriptsubscript^italic-ϵ ≤ _ _k P _π∈ D % _k L_ G_n π % ≥ε ≤ ∑π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT P ( supitalic_π ∈ D ( π start_POSTSUBSCRIPT k ) end_POSTSUBSCRIPT | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) (74) ≤∑πk∈P⁢(α⁢ϵ+|LG^n⁢(πk)|≥ϵ)absentsubscriptsubscriptitalic-ϵsubscriptsubscript^subscriptitalic-ϵ ≤ _ _k P αε+ % L_ G_n _k ≥ε ≤ ∑π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT P ( α ϵ + | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | ≥ ϵ ) (75) ≤∑πk∈P⁢(|LG^n⁢(πk)|≥ϵ⁢(1−α))absentsubscriptsubscriptsubscriptsubscript^subscriptitalic-ϵ1 ≤ _ _k P L_ % G_n _k ≥ε(1-α) ≤ ∑π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT P ( | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | ≥ ϵ ( 1 - α ) ) (76) Then Hoeffding’s inequality gives P⁢(supπ∈Π|LG^n⁢(π)|≥ϵ)subscriptsupremumΠsubscriptsubscript^italic-ϵ P _π∈ L_ G_n(π)% ≥ε ( supitalic_π ∈ Π | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤∑πk∈P⁢(|LG^n⁢(πk)|≥ϵ⁢(1−α))absentsubscriptsubscriptsubscriptsubscript^subscriptitalic-ϵ1 ≤ _ _k P L_ % G_n( _k) ≥ε(1-α) ≤ ∑π start_POSTSUBSCRIPT k ∈ K end_POSTSUBSCRIPT P ( | Lover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( πitalic_k ) | ≥ ϵ ( 1 - α ) ) (78) ≤2⁢||⁢exp⁡(−2⁢(1−α)2⁢n⁢ϵ2|2⁢C|2)absent22superscript12superscriptitalic-ϵ2superscript22 ≤ 2 - 2(% 1-α)^2nε^2 2C ^2 ≤ 2 | K | exp ( - divide start_ARG 2 ( 1 - α )2 n ϵ2 end_ARG start_ARG | 2 C |2 end_ARG ) (79) ∎ Proposition 8 (Covering number in terms of that of domain and range). Fix ϕ~,Θ~,τ~∘~italic-ϕ~Θsuperscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ and Lπ¯subscript¯L_ πLover¯ start_ARG π end_ARG. For ∈τ~∘⁢()superscript~ p∈ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % _p ∈ over~ start_ARG τ end_ARG∘ ( X ), let Π¯subscript¯Π _ pover¯ start_ARG Π end_ARGitalic_p :=g⁢(⋅y)=ϕ~⁢(Θ~⁢π¯⁢())⁢[⋅y]|ϕ~∘Θ~⁢π¯∘τ~∘⁢(⋅x)⁢[⋅y]∈Π⁢(ϕ~,Θ~,τ~∘,Lπ¯)⊆assignabsentconditional-setsubscript⋅~italic-ϕ~Θ¯delimited-[]subscript⋅~italic-ϕ~Θ¯superscript~subscript⋅delimited-[]subscript⋅Π~italic-ϕ~Θsuperscript~subscript¯subscript := \g ·_y = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π p % [·_y]\;|\; [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 π% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % ·_x [·_y ]∈ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ,L_ π % \ _Y:= g ( ⋅y ) = over~ start_ARG ϕ end_ARG ( over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ( italic_p ) ) [ ⋅y ] | over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( ⋅x ) [ ⋅y ] ∈ Π ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ , Lover¯ start_ARG π end_ARG ) ⊆ Pcaligraphic_Y (80) and recall the metric on subscriptP_YPcaligraphic_Y: d⁢(p,q)subscriptsubscript d_P_Y p,q _P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT ( p , q ) :=∥β⁢log⁡p⁢(⋅y)q⁢(⋅y)∥∞assignabsentsubscriptdelimited-∥subscript⋅subscript⋅ := β p ·_y % q ·_y _∞:= ∥ β log divide start_ARG p ( ⋅y ) end_ARG start_ARG q ( ⋅y ) end_ARG ∥∞ (81) Then for κ,δ>00κ,δ>0κ , δ > 0, denoting as ΔδDsuperscriptsubscriptΔ _δ^DΔitalic_δitalic_D the δ-cover of ΔDsuperscriptΔ ^DΔitalic_D under metric dΔDsubscriptsuperscriptΔd_ ^Ddroman_Δitalic_D: Cov⁢(Π,dr,3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,3κ+3L_φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ ( Π , ditalic_r , 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) (82) ≤sup′∈ΔδDCov⁢(Π¯′,d,κ)Cov⁢(ΔD,dΔ,δ)absentsubscriptsupremumsuperscript′subscriptsuperscriptΔCovsuperscriptsubscript¯Πsuperscript′subscriptsubscriptCovsuperscriptΔsubscriptΔ ≤ _ p ∈ ^D_δ Cov % _ p ,d_P_Y,κ ^% Cov ^D,d_ ,δ ≤ supbold_italic_p′ ∈ Δitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT Cov ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ )Cov ( Δ start_POSTSUPERSCRIPT D , droman_Δ , δ ) end_POSTSUPERSCRIPT (83) Proof of Proposition 8.. We wish to find the covering number of Π Π with the metric drsubscriptd_rditalic_r. Take covers of the domain and range of π¯ πover¯ start_ARG π end_ARG. Take a δ-cover of τ~∘⁢()superscript~ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ % X ~ start_ARG τ end_ARG∘ ( X ) with metric dΔsubscriptΔd_ droman_Δ, denote it ΔδDsubscriptsuperscriptΔ ^D_δΔitalic_Ditalic_δ. For every ′∈ΔδDsuperscript′subscriptsuperscriptΔ p ∈ ^D_δitalic_p′ ∈ Δitalic_Ditalic_δ, take a κ-cover of Π¯′subscript¯Πsuperscript′ _ p over¯ start_ARG Π end_ARGitalic_p′ with metric dsubscriptsubscriptd_P_Ydcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT. Denote it by Π¯′,κsubscript¯Πsuperscript′ _ p ,κover¯ start_ARG Π end_ARGitalic_p′ , κ. Construct a set of maps from the domain of π¯ πover¯ start_ARG π end_ARG to the range of π¯ πover¯ start_ARG π end_ARG, via the covers. Let h¯ΔδDsubscript¯ℎsubscriptsuperscriptΔ h_ ^D_δover¯ start_ARG h end_ARGΔitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT be a map from ΔδDsubscriptsuperscriptΔ ^D_δΔitalic_Ditalic_δ s.t. for every ′superscript′ p italic_p′, h¯ΔδD⁢(′)∈Π¯′,κsubscript¯ℎsubscriptsuperscriptΔsuperscript′subscript¯Πsuperscript′ h_ ^D_δ p ∈ % _ p ,κover¯ start_ARG h end_ARGΔitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT ( italic_p′ ) ∈ over¯ start_ARG Π end_ARGitalic_p′ , κ. For every such h¯ΔδDsubscript¯ℎsubscriptsuperscriptΔ h_ ^D_δover¯ start_ARG h end_ARGΔitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT, extend it to h¯ℎ hover¯ start_ARG h end_ARG, a function whose domain is ΔDsuperscriptΔ ^DΔitalic_D as follows: for ∈ΔDsuperscriptΔ p∈ ^Ditalic_p ∈ Δitalic_D, let: h¯⁢()¯ℎ h p ¯ start_ARG h end_ARG ( italic_p ) =h¯ΔδD⁢(),if ⁢∈ΔδDh¯ΔδD⁢(′)⁢ where ′ is selected randomly from ⁢A⁢(),if ⁢∉ΔδD,absentcasessubscript¯ℎsuperscriptsubscriptΔif superscriptsubscriptΔsubscript¯ℎsuperscriptsubscriptΔsuperscript′ where ′ is selected randomly from if superscriptsubscriptΔ = cases h_ _δ^D p% ,&if p∈ _δ^D\\ h_ _δ^D p % where $ p $ is selected randomly from A p % ,&if p ∈ _δ^D cases,= start_ROW start_CELL over¯ start_ARG h end_ARGΔ start_POSTSUBSCRIPT δitalic_D end_POSTSUBSCRIPT ( italic_p ) , end_CELL start_CELL if italic_p ∈ Δitalic_δitalic_D end_CELL end_ROW start_ROW start_CELL over¯ start_ARG h end_ARGΔ start_POSTSUBSCRIPT δitalic_D end_POSTSUBSCRIPT ( italic_p′ ) where italic_p′ is selected randomly from A ( italic_p ) , end_CELL start_CELL if italic_p ∉ Δitalic_δitalic_D end_CELL end_ROW , (84) A⁢() A p ( italic_p ) =′∈ΔδD|dΔD⁢(,′)=min′∈ΔδD⁡dΔd⁢(,′).absentconditional-setsuperscript′subscriptΔsubscriptsuperscriptΔsuperscript′subscriptsuperscript′subscriptΔsubscriptsuperscriptΔsuperscript′ = \ p ∈ _δ^D\> |\>d_% ^D p, p = _ p^% ∈ _δ^Dd_ ^d p, p^% \.= italic_p′ ∈ Δitalic_δitalic_D | droman_Δitalic_D ( italic_p , italic_p′ ) = minbold_italic_p′ ′ ∈ Δ start_POSTSUBSCRIPT δitalic_D end_POSTSUBSCRIPT droman_Δitalic_d ( italic_p , italic_p′ ′ ) . (85) Construct a set of maps ×→[0,1]→01X×Y→ [0,1 ]X × Y → [ 0 , 1 ], denoted by ℋδ,κsubscriptℋH_δ,κHitalic_δ , κ. This set will be proved to cover Π Π. For every h¯ℎ hover¯ start_ARG h end_ARG, define h:×→[0,1]:ℎ→01h:X×Y→ [0,1 ]h : X × Y → [ 0 , 1 ]: h⁢(x,y)ℎ h x,y ( x , y ) =h¯⁢(τ~∘⁢(x))⁢[y]absent¯ℎsuperscript~delimited-[] = h [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x [y ]= over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) [ y ] (86) Let ℋδ,κsubscriptℋH_δ,κHitalic_δ , κ denote all such hℎh. It can be checked that ℋδ,κ⊂ΠsubscriptℋΠH_δ,κ⊂ _δ , κ ⊂ Π: 1. h¯⁢(τ~∘⁢(x))∈Π¯′¯ℎsuperscript~subscript¯Πsuperscript′ h [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x ∈ _ p over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) ∈ over¯ start_ARG Π end_ARGitalic_p′ for some ′superscript′ p italic_p′ so is in subscriptP_YPcaligraphic_Y; therefore, ∑yh¯⁢(τ~∘⁢(x))⁢[y]=1subscript¯ℎsuperscript~delimited-[]1 _y h [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x [y]=1∑y over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) [ y ] = 1. 2. Since ′∈τ~∘⁢()superscript′~ p ∈ [rgb]0.609375,0.1328125,0.1484375 [named% ]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % _p′ ∈ over~ start_ARG τ end_ARG∘ ( X ), take x′ so that ′=τ~∘⁢(x′)superscript′~superscript′ p = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x _p′ = over~ start_ARG τ end_ARG∘ ( x′ ). Then wlog for every x, there is some ′superscript′ p italic_p′ and x′ such that, h¯⁢(τ~∘⁢(x))⁢[y]¯ℎsuperscript~delimited-[] h [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x [y]over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) [ y ] =h¯⁢(′)⁢[y]absent¯ℎsuperscript′delimited-[] = h p [y]= over¯ start_ARG h end_ARG ( italic_p′ ) [ y ] (87) =h¯ΔδD⁢(′)⁢[y]absentsubscript¯ℎsubscriptsuperscriptΔsuperscript′delimited-[] = h_ ^D_δ p % [y]= over¯ start_ARG h end_ARGΔitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT ( italic_p′ ) [ y ] (88) =ϕ~∘Θ~⁢π¯⁢(′)⁢[y]absent~italic-ϕ~Θ¯superscript′delimited-[] = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π p^% [y]= over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ( italic_p′ ) [ y ] (89) =ϕ~∘Θ~⁢π¯⁢(τ~∘⁢(x′))⁢[y],absent~italic-ϕ~Θ¯superscript~superscript′delimited-[] = [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % [y],= over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ( over~ start_ARG τ end_ARG∘ ( x′ ) ) [ y ] , (90) where the third equality holds since h¯ΔδD⁢(′)⁢[y]∈Π¯′subscript¯ℎsubscriptsuperscriptΔsuperscript′delimited-[]subscript¯Πsuperscript′ h_ ^D_δ p [y]∈% _ p over¯ start_ARG h end_ARGΔitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT ( italic_p′ ) [ y ] ∈ over¯ start_ARG Π end_ARGitalic_p′. Therefore for every x′, |β⁢log⁡h¯⁢(τ~∘⁢(x))⁢[y]πr⁢e⁢f⁢(y|x)|≤C¯ℎsuperscript~delimited-[]subscriptconditional β h [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % [y] _ref y|x ≤ C| β log divide start_ARG over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) [ y ] end_ARG start_ARG πitalic_r e f ( y | x ) end_ARG | ≤ C. 3. Now let us show that for some π¯hsubscript¯ℎ π_hover¯ start_ARG π end_ARGh, h=ϕ~∘Θ~⁢π¯h∘τ~∘∈Π̊ℎ~italic-ϕ~Θsubscript¯ℎsuperscript~̊Πh= [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ π% _h [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ ∈% h = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARGh ∘ over~ start_ARG τ end_ARG∘ ∈ over̊ start_ARG Π end_ARG. For every ∈τ~∘⁢()superscript~ p∈ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % _p ∈ over~ start_ARG τ end_ARG∘ ( X ), h¯⁢()∈Π¯′,κ¯ℎsubscript¯Πsuperscript′ h p ∈ _ p ,κover¯ start_ARG h end_ARG ( italic_p ) ∈ over¯ start_ARG Π end_ARGitalic_p′ , κ for some ′superscript′ p italic_p′, i.e. there is some π¯subscript¯ π_ pover¯ start_ARG π end_ARGitalic_p s.t. h¯⁢()=ϕ~⁢(Θ~⁢π¯⁢(′))¯ℎ~italic-ϕ~Θ¯superscript′ h p = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π p^% ¯ start_ARG h end_ARG ( italic_p ) = over~ start_ARG ϕ end_ARG ( over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ( italic_p′ ) ). Construct π¯h⁢():=π¯⁢(′)assignsubscript¯ℎsubscript¯superscript′ π_h p := π_ p % p ¯ start_ARG π end_ARGh ( italic_p ) := over¯ start_ARG π end_ARGitalic_p ( italic_p′ ). We need to show that ℋδ,κsubscriptℋH_δ,κHitalic_δ , κ covers Π Π with metric drsubscriptd_rditalic_r. To this end we need to show that for every π∈Ππ∈ π ∈ Π, there is a hπ∈ℋδ,κsubscriptℎsubscriptℋh_π _δ,κhitalic_π ∈ Hitalic_δ , κ, such that dr⁢(π,hπ)<ϵ⁢(δ,κ)subscriptsubscriptℎitalic-ϵd_r π,h_π <ε δ,κ _r ( π , hitalic_π ) < ϵ ( δ , κ ) for some small value ϵitalic-ϵεϵ which depends monotonically on δ and κ, and ϵ⁢(0,0)=0italic-ϵ000ε 0,0 =0ϵ ( 0 , 0 ) = 0. So consider dr⁢(π,h)subscriptℎd_r π,h _r ( π , h ) for some π∈Ππ∈ π ∈ Π and h∈ℋδ,κℎsubscriptℋh _δ,κh ∈ Hitalic_δ , κ: dr⁢(π,h)subscriptℎ d_r π,h _r ( π , h ) =∥rπ−rh∥∞absentsubscriptdelimited-∥subscriptsubscriptℎ = r_π-r_h _∞= ∥ ritalic_π - ritalic_h ∥∞ (91) =supx∈,y∈|rπ⁢(x,y)−rh⁢(x,y)|absentsubscriptsupremumformulae-sequencesubscriptsubscriptℎ = _x ,y r_π % x,y -r_h x,y = supitalic_x ∈ X , y ∈ Y | ritalic_π ( x , y ) - ritalic_h ( x , y ) | (92) =supx∈,y∈|β⁢log⁡πref⁢(x,y)−β⁢log⁡hπref⁢(x,y)|absentsubscriptsupremumformulae-sequencesubscriptrefℎsubscriptref = _x ,y β % π _ref x,y -β h _% ref x,y = supitalic_x ∈ X , y ∈ Y | β log divide start_ARG π end_ARG start_ARG πref end_ARG ( x , y ) - β log divide start_ARG h end_ARG start_ARG πref end_ARG ( x , y ) | (93) =supx∈supy∈|β⁢log⁡πref⁢(x,y)−β⁢log⁡hπref⁢(x,y)|absentsubscriptsupremumsubscriptsupremumsubscriptrefℎsubscriptref = _x \ _y \ % β π _ref x,y -β% h _ref x,y % \ \= supitalic_x ∈ X supitalic_y ∈ Y | β log divide start_ARG π end_ARG start_ARG πref end_ARG ( x , y ) - β log divide start_ARG h end_ARG start_ARG πref end_ARG ( x , y ) | (94) =supx∈supy∈|β⁢log⁡π⁢(x,y)−β⁢log⁡h⁢(x,y)|absentsubscriptsupremumsubscriptsupremumℎ = _x \ _y \ % β π x,y -β h x,y% \ \= supitalic_x ∈ X supitalic_y ∈ Y | β log π ( x , y ) - β log h ( x , y ) | (95) So it is sufficient to show that for every π there is some hπsubscriptℎh_πhitalic_π such that for every x, supy∈|β⁢log⁡π⁢(x,y)−β⁢log⁡hπ⁢(x,y)|subscriptsupremumsubscriptℎ _y \ β π % x,y -β h_π x,y % \supitalic_y ∈ Y | β log π ( x , y ) - β log hitalic_π ( x , y ) | (96) =∥β⁢log⁡π⁢(x,⋅y)−β⁢log⁡hπ⁢(x,⋅y)∥∞absentsubscriptdelimited-∥subscript⋅subscriptℎsubscript⋅ = β π x,·_y -% β h_π x,·_y _∞= ∥ β log π ( x , ⋅y ) - β log hitalic_π ( x , ⋅y ) ∥∞ (97) ≤ϵ⁢(δ,κ)absentitalic-ϵ ≤ε δ,κ ≤ ϵ ( δ , κ ) (98) Decompose the distance between a general π∈Ππ∈ π ∈ Π and h∈ℋδ,κℎsubscriptℋh _δ,κh ∈ Hitalic_δ , κ for a fixed x. This helps us later choose the hπsubscriptℎh_πhitalic_π which makes the bound small enough. Let π∈Ππ∈ π ∈ Π, h∈ℋδ,κ,x∈formulae-sequenceℎsubscriptℋh _δ,κ,x ∈ Hitalic_δ , κ , x ∈ X and x′∈superscript′x ′ ∈ X s.t. ′:=τ~∘⁢(x′)∈A⁢(τ~∘⁢(x))assignsuperscript′~superscript′~ p := [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x ∈ A [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % _p′ := over~ start_ARG τ end_ARG∘ ( x′ ) ∈ A ( over~ start_ARG τ end_ARG∘ ( x ) ). ∥β⁢log⁡π⁢(x,⋅y)−β⁢log⁡h⁢(x,⋅y)∥∞subscriptdelimited-∥subscript⋅ℎsubscript⋅ β π x,·_y -% β h x,·_y _∞∥ β log π ( x , ⋅y ) - β log h ( x , ⋅y ) ∥∞ (99) ≤∥β⁢log⁡π⁢(x,⋅y)−β⁢log⁡π⁢(x′,⋅y)∥∞absentsubscriptdelimited-∥subscript⋅superscript′subscript⋅ ≤ β π x,·_y -β% π x ,·_y _∞≤ ∥ β log π ( x , ⋅y ) - β log π ( x′ , ⋅y ) ∥∞ (100) +∥β⁢log⁡π⁢(x′,⋅y)−β⁢log⁡h⁢(x′,⋅y)∥∞subscriptdelimited-∥superscript′subscript⋅ℎsuperscript′subscript⋅ + β π x ,·_y % -β h x ,·_y _∞+ ∥ β log π ( x′ , ⋅y ) - β log h ( x′ , ⋅y ) ∥∞ (101) +∥β⁢log⁡h⁢(x′,⋅y)−β⁢log⁡h⁢(x,⋅y)∥∞subscriptdelimited-∥ℎsuperscript′subscript⋅ℎsubscript⋅ + β h x ,·_y -% β h x,·_y _∞+ ∥ β log h ( x′ , ⋅y ) - β log h ( x , ⋅y ) ∥∞ (102) Consider ∥β⁢log⁡π⁢(x,⋅y)−β⁢log⁡π⁢(x′,⋅y)∥∞subscriptdelimited-∥subscript⋅superscript′subscript⋅ β π x,·_y -β π % x ,·_y _∞∥ β log π ( x , ⋅y ) - β log π ( x′ , ⋅y ) ∥∞. We can show this term is bounded by L⁢δL δ. Let x′∈τ~∘,−1⁢(′)superscript′~1superscript′x ∈ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ ,-1% p ′ ∈ over~ start_ARG τ end_ARG∘ , - 1 ( italic_p′ ) be s.t. ∥β⁢log⁡π⁢(x,⋅y)−β⁢log⁡π⁢(x′,⋅y)∥∞subscriptdelimited-∥subscript⋅superscript′subscript⋅ β π x,·_y -% β π x ,·_y _∞∥ β log π ( x , ⋅y ) - β log π ( x′ , ⋅y ) ∥∞ (103) ≤∥β⁢log⁡π⁢(x,⋅y)π⁢(x′,⋅y)∥∞absentsubscriptdelimited-∥subscript⋅superscript′subscript⋅ ≤ β π x,·_y % π x ,·_y _∞≤ ∥ β log divide start_ARG π ( x , ⋅y ) end_ARG start_ARG π ( x′ , ⋅y ) end_ARG ∥∞ (104) ≤d⁢(π⁢(x,⋅y),π⁢(x′,⋅y))absentsubscriptsubscriptsubscript⋅superscript′subscript⋅ ≤ d_P_Y π x,·_y% ,π x ,·_y ≤ dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT ( π ( x , ⋅y ) , π ( x′ , ⋅y ) ) (105) ≤Lϕ⁢∥Θ~∥p⁢Lπ¯⁢dΔ⁢(τ~∘⁢(x),τ~∘⁢(x′))absentsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptΔsuperscript~superscript~superscript′ ≤ L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πd_ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % , [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x ≤ Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG droman_Δ ( over~ start_ARG τ end_ARG∘ ( x ) , over~ start_ARG τ end_ARG∘ ( x′ ) ) (106) ≤Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δabsentsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ ≤ L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδ≤ Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (107) Consider ∥β⁢log⁡π⁢(x′,⋅y)−β⁢log⁡h⁢(x′,⋅y)∥∞subscriptdelimited-∥superscript′subscript⋅ℎsuperscript′subscript⋅ β π x ,·_y -β% h x ,·_y _∞∥ β log π ( x′ , ⋅y ) - β log h ( x′ , ⋅y ) ∥∞; we show that we can choose hπsubscriptℎh_πhitalic_π to make this be upper bounded by κ. Since π∈Ππ∈ π ∈ Π, it can be written as π⁢(⋅x,⋅y)=ϕ~∘Θ~⁢π¯∘τ~∘⁢(⋅x)⁢[⋅y]subscript⋅subscript⋅~italic-ϕ~Θ¯superscript~subscript⋅delimited-[]subscript⋅π ·_x,·_y = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ·_x% [·_y ]π ( ⋅x , ⋅y ) = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( ⋅x ) [ ⋅y ] for some π¯:ΔD→ΔD:¯→superscriptΔsuperscriptΔ π: ^D→ ^Dover¯ start_ARG π end_ARG : Δitalic_D → Δitalic_D. Note that π⁢(x′,⋅y)∈Π¯τ~⁢(x′)superscript′subscript⋅subscript¯Π~superscript′π x ,·_y ∈ _ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x π ( x′ , ⋅y ) ∈ over¯ start_ARG Π end_ARGover~ start_ARG τ end_ARG ( x′ ). And since τ~∘⁢(x′)∈ΔδDsuperscript~superscript′subscriptΔ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ x^% ∈ _δ^Dover~ start_ARG τ end_ARG∘ ( x′ ) ∈ Δitalic_δitalic_D, there is some h¯π¯∈Π¯τ~∘⁢(x′),κsubscript¯ℎ¯subscript¯Πsuperscript~superscript′ h_ π∈ _ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x ,κover¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ∈ over¯ start_ARG Π end_ARGover~ start_ARG τ end_ARG∘ ( x′ ) , κ s.t. d⁢(π⁢(x′,⋅y),h¯π¯⁢(τ~∘⁢(x′)))≤κsubscriptsubscriptsuperscript′subscript⋅subscript¯ℎ¯superscript~superscript′d_P_Y π x ,·_y % , h_ π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % ≤ _P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT ( π ( x′ , ⋅y ) , over¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ( over~ start_ARG τ end_ARG∘ ( x′ ) ) ) ≤ κ. But expanding this, κ κ ≥d⁢(π⁢(x′,⋅y),h¯π¯⁢(τ~∘⁢(x′)))absentsubscriptsubscriptsuperscript′subscript⋅subscript¯ℎ¯superscript~superscript′ ≥ d_P_Y π x ,% ·_y , h_ π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % ≥ dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT ( π ( x′ , ⋅y ) , over¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ( over~ start_ARG τ end_ARG∘ ( x′ ) ) ) (108) =∥β⁢log⁡π⁢(x′,⋅y)h¯π¯⁢(τ~∘⁢(x′))⁢[⋅y]∥∞absentsubscriptdelimited-∥superscript′subscript⋅subscript¯ℎ¯superscript~superscript′delimited-[]subscript⋅ = β π x ,·_y% h_ π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % [·_y ] _∞= ∥ β log divide start_ARG π ( x′ , ⋅y ) end_ARG start_ARG over¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ( over~ start_ARG τ end_ARG∘ ( x′ ) ) [ ⋅y ] end_ARG ∥∞ (109) =∥β⁢log⁡π⁢(x′,⋅y)−h¯π¯∘τ~∘⁢(x′,⋅y)∥∞absentsubscriptdelimited-∥superscript′subscript⋅subscript¯ℎ¯superscript~superscript′subscript⋅ = β π x ,·_y % - h_ π [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x ,·_y _∞= ∥ β log π ( x′ , ⋅y ) - over¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( x′ , ⋅y ) ∥∞ (110) So choose hπ:=h¯π¯∘τ~∘assignsubscriptℎsubscript¯ℎ¯superscript~h_π:= h_ π τ hitalic_π := over¯ start_ARG h end_ARGover¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘. From now on replace hℎh by hπsubscriptℎh_πhitalic_π. Consider ∥β⁢log⁡hπ⁢(x′,⋅y)−β⁢log⁡hπ⁢(x,⋅y)∥∞subscriptdelimited-∥subscriptℎsuperscript′subscript⋅subscriptℎsubscript⋅ β h_π x ,·_y -% β h_π x,·_y _∞∥ β log hitalic_π ( x′ , ⋅y ) - β log hitalic_π ( x , ⋅y ) ∥∞. We show that this is bounded above by 2⁢κ+2⁢L⁢δ222κ+2L 2 κ + 2 L δ. Let 1′,2′∈A⁢(τ~∘⁢(x))subscriptsuperscript′1subscriptsuperscript′2superscript~ p _1, p _2∈ A [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % _p′1 , italic_p′2 ∈ A ( over~ start_ARG τ end_ARG∘ ( x ) ), then let xi′∈τ~∘,−1⁢(i′)subscriptsuperscript′~1subscriptsuperscript′x _i∈ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ ,-1% p _i ′italic_i ∈ over~ start_ARG τ end_ARG∘ , - 1 ( italic_p′italic_i ). ∥β⁢log⁡hπ⁢(x1′,⋅y)−β⁢log⁡hπ⁢(x2′,⋅y)∥∞subscriptdelimited-∥subscriptℎsubscriptsuperscript′1subscript⋅subscriptℎsubscriptsuperscript′2subscript⋅ β h_π x _1% ,·_y -β h_π x _2,·_% y _∞ 28.45274ptstart_ARG ∥ β log hitalic_π ( x′1 , ⋅y ) - β log hitalic_π ( x′2 , ⋅y ) ∥∞ end_ARG ≤∥β⁢log⁡hπ⁢(x1′,⋅y)−β⁢log⁡π⁢(x1′,⋅y)∥∞+∥β⁢log⁡π⁢(x1′,⋅y)−β⁢log⁡π⁢(x2′,⋅y)∥∞+∥β⁢log⁡π⁢(x2′,⋅y)−β⁢log⁡hπ⁢(x2′,⋅y)∥∞absentsubscriptdelimited-∥subscriptℎsubscriptsuperscript′1subscript⋅subscriptsuperscript′1subscript⋅subscriptdelimited-∥subscriptsuperscript′1subscript⋅subscriptsuperscript′2subscript⋅subscriptdelimited-∥subscriptsuperscript′2subscript⋅subscriptℎsubscriptsuperscript′2subscript⋅ multlined≤ β h_π x^% _1,·_y -β π x _1,% ·_y _∞+\\ β π x _1,·_y -% β π x _2,·_y _% ∞+\\ β π x _2,·_y -% β h_π x _2,·_y % _∞ multlined≤ β h_π x^% _1,·_y -β π x _1,% ·_y _∞+\\ β π x _1,·_y -% β π x _2,·_y _% ∞+\\ β π x _2,·_y -% β h_π x _2,·_y % _∞start_ROW start_CELL ≤ ∥ β log hitalic_π ( x′1 , ⋅y ) - β log π ( x′1 , ⋅y ) ∥∞ + end_CELL end_ROW start_ROW start_CELL ∥ β log π ( x′1 , ⋅y ) - β log π ( x′2 , ⋅y ) ∥∞ + end_CELL end_ROW start_ROW start_CELL ∥ β log π ( x′2 , ⋅y ) - β log hitalic_π ( x′2 , ⋅y ) ∥∞ end_CELL end_ROW (114) ≤κ+∥β⁢log⁡π⁢(x1′,⋅y)−β⁢log⁡π⁢(x2′,⋅y)∥∞+κabsentsubscriptdelimited-∥subscriptsuperscript′1subscript⋅subscriptsuperscript′2subscript⋅ ≤κ+ β π x _1,% ·_y -β π x _2,·_y % _∞+κ≤ κ + ∥ β log π ( x′1 , ⋅y ) - β log π ( x′2 , ⋅y ) ∥∞ + κ (115) =2⁢κ+β⁢∥log⁡π⁢(x1′,⋅y)π⁢(x2′,⋅y)∥∞absent2subscriptdelimited-∥subscriptsuperscript′1subscript⋅subscriptsuperscript′2subscript⋅ =2κ+β π x _1% ,·_y π x _2,·_y % _∞= 2 κ + β ∥ log divide start_ARG π ( x′1 , ⋅y ) end_ARG start_ARG π ( x′2 , ⋅y ) end_ARG ∥∞ (116) ≤2⁢κ+Lϕ⁢∥Θ~∥p⁢Lπ¯⁢dΔ⁢(τ~∘⁢(x1),τ~∘⁢(x2))absent2subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptΔsuperscript~subscript1superscript~subscript2 ≤ 2κ+L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πd_ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x_1 , [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x_2 ≤ 2 κ + Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG droman_Δ ( over~ start_ARG τ end_ARG∘ ( x1 ) , over~ start_ARG τ end_ARG∘ ( x2 ) ) (117) ≤2⁢κ+Lϕ⁢∥Θ~∥p⁢Lπ¯⁢dΔ⁢(τ~∘⁢(x1),τ~∘⁢(x))+Lϕ⁢∥Θ~∥p⁢Lπ¯⁢dΔ⁢(τ~∘⁢(x),τ~∘⁢(x2))absent2subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptΔsuperscript~subscript1superscript~subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptΔsuperscript~superscript~subscript2 ≤ 2κ+L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πd_ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x_1 , [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % τ x +L_φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πd_ % [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x , [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 τ % x_2 ≤ 2 κ + Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG droman_Δ ( over~ start_ARG τ end_ARG∘ ( x1 ) , over~ start_ARG τ end_ARG∘ ( x ) ) + Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG droman_Δ ( over~ start_ARG τ end_ARG∘ ( x ) , over~ start_ARG τ end_ARG∘ ( x2 ) ) (118) ≤2⁢κ+2⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δabsent22subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ ≤ 2κ+2L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ≤ 2 κ + 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (119) If τ~⁢(x)∉ΔδD~superscriptsubscriptΔ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ x % ∈ _δ^Dover~ start_ARG τ end_ARG ( x ) ∉ Δitalic_δitalic_D, h⁢(x)=h¯⁢(τ~∘⁢(x))=h¯Δδ⁢(′)=h⁢(x′)ℎ¯ℎsuperscript~subscript¯ℎsubscriptΔsuperscript′ℎsuperscript′h x = h [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % = h_ _δ p % =h x ( x ) = over¯ start_ARG h end_ARG ( over~ start_ARG τ end_ARG∘ ( x ) ) = over¯ start_ARG h end_ARGΔ start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT ( italic_p′ ′ ) = h ( x′ ′ ) with some ′∈A⁢(τ~∘⁢(x))superscript′~ p ∈ A [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ x % _p′ ′ ∈ A ( over~ start_ARG τ end_ARG∘ ( x ) ) and τ~∘⁢(x′)=′~superscript′ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 τ x^% = p over~ start_ARG τ end_ARG∘ ( x′ ′ ) = italic_p′ ′, therefore ∥β⁢log⁡hπ⁢(x′,⋅y)−β⁢log⁡hπ⁢(x,⋅y)∥∞subscriptdelimited-∥subscriptℎsuperscript′subscript⋅subscriptℎsubscript⋅ β h_π x ,·_y% -β h_π x,·_y % _∞∥ β log hitalic_π ( x′ , ⋅y ) - β log hitalic_π ( x , ⋅y ) ∥∞ =∥β⁢log⁡hπ⁢(x′,⋅y)−rhπ⁢(x′,⋅y)∥∞absentsubscriptdelimited-∥subscriptℎsuperscript′subscript⋅subscriptsubscriptℎsuperscript′subscript⋅ = β h_π x ,·_y% -r_h_π x ,·_y % _∞= ∥ β log hitalic_π ( x′ , ⋅y ) - ritalic_h start_POSTSUBSCRIPT π end_POSTSUBSCRIPT ( x′ ′ , ⋅y ) ∥∞ (120) ≤2⁢κ+2⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δabsent22subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ ≤ 2κ+2L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ≤ 2 κ + 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (121) Therefore, ∥rπ⁢(x,⋅y)−β⁢log⁡hπ⁢(x,⋅y)∥∞subscriptdelimited-∥subscriptsubscript⋅subscriptℎsubscript⋅ r_π x,·_y -β% h_π x,·_y _∞∥ ritalic_π ( x , ⋅y ) - β log hitalic_π ( x , ⋅y ) ∥∞ ≤3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δabsent33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ ≤ 3κ+3L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ≤ 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (122) Since the upper bound is constant in x, we can conclude that dr⁢(π,hπ)≤3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δsubscriptsubscriptℎ33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯d_r π,h_π ≤ 3κ+3L_φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π _r ( π , hitalic_π ) ≤ 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ. So ℋδ,κsubscriptℋH_δ,κHitalic_δ , κ covers Π Π in drsubscriptd_rditalic_r with radius 3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯3κ+3L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ π 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ. Therefore, we have that Cov⁢(Π,dr,3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,3κ+3L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πδ Cov ( Π , ditalic_r , 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) ≤|ℋδ,κ|absentsubscriptℋ ≤ _δ,κ ≤ | Hitalic_δ , κ | (123) ≤∏′∈ΔδD|Π¯′,κ|absentsubscriptproductsuperscript′subscriptsuperscriptΔsubscript¯Πsuperscript′ ≤ _ p ∈ ^D_δ % _ p ,κ ≤ ∏italic_p′ ∈ Δitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT | over¯ start_ARG Π end_ARGitalic_p′ , κ | (124) ≤sup′∈ΔδDCov⁢(Π¯′,d,κ)Cov⁢(ΔδD,dΔD,δ)absentsubscriptsupremumsuperscript′subscriptΔCovsuperscriptsubscript¯Πsuperscript′subscriptsubscriptCovsubscriptsuperscriptΔsubscriptsuperscriptΔ ≤ _ p ∈ _δ^D Cov % _ p ,d_P_Y,κ ^% Cov ^D_δ,d_ ^D,δ ≤ supbold_italic_p′ ∈ Δ start_POSTSUBSCRIPT δitalic_D end_POSTSUBSCRIPT Cov ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ )Cov ( Δ start_POSTSUPERSCRIPT Ditalic_δ , droman_Δitalic_D , δ ) end_POSTSUPERSCRIPT (125) ∎ C.1 Proof of Theorem 5 Theorem 5(Bounding sample complexity in terms of dimension) We remain in the set up of Proposition 8. The covering number of Π Π is bounded above by a function of D: Cov⁢(Π,dr,3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,3κ+3L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πδ Cov ( Π , ditalic_r , 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) ≤(2⁢Lϕ⁢∥Θ~∥p⁢Dκ)D⁢(2⁢Dδ)Dabsentsuperscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsuperscript2 ≤ 2L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _p Dκ % ^D 2 Dδ ^D≤ ( divide start_ARG 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG κ end_ARG )D ( divide start_ARG 2 square-root start_ARG D end_ARG end_ARG start_ARG δ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (126) Set κ=ϵ48italic-ϵ48κ= ε48κ = divide start_ARG ϵ end_ARG start_ARG 48 end_ARG, we need n⁢(ϵ,ω)italic-ϵ n ε,ω ( ϵ , ω ) =Ω⁢(Dϵ2⁢(96⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢Dϵ)D⁢log⁡(96⁢Lϕ⁢∥Θ~∥p⁢Dϵ)−log⁡ω)absentΩsuperscriptitalic-ϵ2superscript96subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ96subscriptitalic-ϕsubscriptdelimited-∥~Θitalic-ϵ = Dε^2 96L_% φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% π Dε ^D 96L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p% Dε - ω = Ω ( divide start_ARG D end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG )D log ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (127) samples to generalise. That is, whenever n′≥n⁢(ϵ,ω)superscript′italic-ϵn ≥ n(ε,ω)n′ ≥ n ( ϵ , ω ), we have P⁢(supπ∈Π|RG⁢(π)−RG^n′⁢(π)|≥ϵ)≤ωsubscriptsupremumΠsubscriptsubscriptsubscript^superscript′italic-ϵ P ( _π∈ |R_G π -R_% G_n π |≥ε )≤ ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n′ end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ ω (128) Proof of Theorem 5.. We will bound both Cov⁢(Π¯′,d,κ)Covsubscript¯Πsuperscript′subscriptsubscript Cov _ p ,d_P_Y,κ ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ ) and Cov⁢(ΔD,dΔ,δ)CovsuperscriptΔsubscriptΔ Cov ^D,d_ ,δ ( Δitalic_D , droman_Δ , δ ) in terms of D. First consider Cov⁢(Π¯′,d,κ)Covsubscript¯Πsuperscript′subscriptsubscript Cov _ p ,d_P_Y,κ ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ ). Recall Π¯′subscript¯Πsuperscript′ _ p over¯ start_ARG Π end_ARGitalic_p′: Π¯′subscript¯Πsuperscript′ _ p over¯ start_ARG Π end_ARGitalic_p′ =g⁢(⋅y)=ϕ~∘Θ~⁢π¯⁢(′)⁢[⋅y]|ϕ~∘Θ~⁢π¯∘τ~∘⁢(⋅x)⁢[⋅y]∈Πabsentconditional-setsubscript⋅~italic-ϕ~Θ¯superscript′delimited-[]subscript⋅~italic-ϕ~Θ¯superscript~subscript⋅delimited-[]subscript⋅Π = \g ·_y = [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π p^% [·_y ]\> |\> [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ·_x% [·_y ]∈ \= g ( ⋅y ) = over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ( italic_p′ ) [ ⋅y ] | over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( ⋅x ) [ ⋅y ] ∈ Π (129) Now we create a Lipschitz function such that the image is Π¯′subscript¯Πsuperscript′ _ p over¯ start_ARG Π end_ARGitalic_p′: note that ϕ~∘Θ~~italic-ϕ~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG is Lϕ⁢∥Θ~∥psubscriptitalic-ϕsubscriptdelimited-∥~ΘL_φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pLitalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p-Lipschitz, where we recall that Lϕsubscriptitalic-ϕL_φLitalic_ϕ is the Lipschitz constant for ϕ~~italic-ϕ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φover~ start_ARG ϕ end_ARG and ∥Θ~∥psubscriptdelimited-∥~Θ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p∥ over~ start_ARG Θ end_ARG ∥p is the operator-p-norm of Θ~~Θ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 over~ start_ARG Θ end_ARG on ΔDsuperscriptΔ ^DΔitalic_D. For a given ′superscript′ p italic_p′, let K⁢(′)=π¯⁢(′)|ϕ~∘Θ~⁢π¯∘τ~∘⁢(⋅x)⁢[⋅y]∈Π⊆ΔDsuperscript′conditional-set¯superscript′~italic-ϕ~Θ¯superscript~subscript⋅delimited-[]subscript⋅ΠsuperscriptΔK( p )= \ π p \>% |\> [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 π [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ·_x% [·_y ]∈ \ ^DK ( italic_p′ ) = over¯ start_ARG π end_ARG ( italic_p′ ) | over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG over¯ start_ARG π end_ARG ∘ over~ start_ARG τ end_ARG∘ ( ⋅x ) [ ⋅y ] ∈ Π ⊆ Δitalic_D. Then ϕ~∘Θ~:K⁢(′)→Π¯′:~italic-ϕ~Θ→superscript′subscript¯Πsuperscript′ [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 :K p % → _ p over~ start_ARG ϕ end_ARG ∘ over~ start_ARG Θ end_ARG : K ( italic_p′ ) → over¯ start_ARG Π end_ARGitalic_p′. Now use the covering number of K⁢(′)superscript′K p ( italic_p′ ) to bound that of Π¯′subscript¯Πsuperscript′ _ p over¯ start_ARG Π end_ARGitalic_p′. Cov⁢(Π¯′,d,κ)Covsubscript¯Πsuperscript′subscriptsubscript Cov _ p ,d_P% _Y,κ ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ ) ≤Cov⁢(K⁢(′),dΔ,κLϕ⁢∥Θ~∥p)≤Cov⁢(ΔD,dΔ,κLϕ⁢∥Θ~∥p)absentCovsuperscript′subscriptΔsubscriptitalic-ϕsubscriptdelimited-∥~ΘCovsuperscriptΔsubscriptΔsubscriptitalic-ϕsubscriptdelimited-∥~Θ ≤ Cov K p % ,d_ , κL_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _p ≤% Cov ^D,d_ , κL_φ % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _p ≤ Cov ( K ( italic_p′ ) , droman_Δ , divide start_ARG κ end_ARG start_ARG Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p end_ARG ) ≤ Cov ( Δitalic_D , droman_Δ , divide start_ARG κ end_ARG start_ARG Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p end_ARG ) (130) Finally, since all vectors on ΔDsuperscriptΔ ^DΔitalic_D have bounded p-norm, we can bound, for some constant E⁢(p,D)E p,D ( p , D ) depending on the norm: Cov⁢(ΔD,dΔ,κ)CovsuperscriptΔsubscriptΔ Cov ^D,d_ ,κ ( Δitalic_D , droman_Δ , κ ) ≤(2⁢E⁢(p,D)⁢Lϕ⁢∥Θ~∥p⁢Dκ)Dabsentsuperscript2subscriptitalic-ϕsubscriptdelimited-∥~Θ ≤ 2E p,D L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p% Dκ ^D≤ ( divide start_ARG 2 E ( p , D ) Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG κ end_ARG )D (131) This gives us: Cov⁢(Π,dr,3⁢κ+3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript33subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,3κ+3L_φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ ( Π , ditalic_r , 3 κ + 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) (132) ≤sup′∈ΔδDCov⁢(Π¯′,d,κ)Cov⁢(ΔD,dΔ,δ)absentsubscriptsupremumsuperscript′subscriptsuperscriptΔCovsuperscriptsubscript¯Πsuperscript′subscriptsubscriptCovsuperscriptΔsubscriptΔ ≤ _ p ∈ ^D_δ Cov % _ p ,d_P_Y,κ ^% Cov ^D,d_ ,δ ≤ supbold_italic_p′ ∈ Δitalic_D start_POSTSUBSCRIPT δ end_POSTSUBSCRIPT Cov ( over¯ start_ARG Π end_ARGitalic_p′ , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ )Cov ( Δ start_POSTSUPERSCRIPT D , droman_Δ , δ ) end_POSTSUPERSCRIPT (133) ≤Cov⁢(ΔD,dΔ,κLϕ⁢∥Θ~∥p)Cov⁢(ΔD,dΔ,δ)absentCovsuperscriptsuperscriptΔsubscriptΔsubscriptitalic-ϕsubscriptdelimited-∥~ΘCovsuperscriptΔsubscriptΔ ≤ Cov ^D,d_ , κL% _φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p% Cov ^D,d_ ,δ ≤ Cov ( Δitalic_D , droman_Δ , divide start_ARG κ end_ARG start_ARG Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p end_ARG )Cov ( Δ start_POSTSUPERSCRIPT D , droman_Δ , δ ) end_POSTSUPERSCRIPT (134) ≤(2⁢E⁢(p,D)⁢Lϕ⁢∥Θ~∥p⁢Dκ)D⁢(2⁢E⁢(p,D)⁢Dδ)Dabsentsuperscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsuperscript2 ≤ 2E p,D L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _p% Dκ ^D 2E p,D % Dδ ^D≤ ( divide start_ARG 2 E ( p , D ) Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG κ end_ARG )D ( divide start_ARG 2 E ( p , D ) square-root start_ARG D end_ARG end_ARG start_ARG δ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (135) And let κ=Lϕ⁢∥Θ~∥p⁢(Lπ¯)⁢δsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯κ=L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _p L_ π δκ = Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p ( Lover¯ start_ARG π end_ARG ) δ. Then the covering number bound becomes Cov⁢(Π,dr,6⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ)CovΠsubscript6subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov ,d_r,6L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ ( Π , ditalic_r , 6 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ ) ≤Cov⁢(ΔD,dΔ,Lπ¯⁢δ)Cov⁢(ΔD,dΔ,δ)absentCovsuperscriptsuperscriptΔsubscriptΔsubscript¯CovsuperscriptΔsubscriptΔ ≤ Cov ^D,d_ ,L_ π% δ Cov ^D,d_ ,δ% ≤ Cov ( Δitalic_D , droman_Δ , Lover¯ start_ARG π end_ARG δ )Cov ( Δ start_POSTSUPERSCRIPT D , droman_Δ , δ ) end_POSTSUPERSCRIPT (136) ≤(2⁢E⁢(p,D)⁢DLπ¯⁢δ)D⁢(2⁢E⁢(p,D)⁢Dδ)Dabsentsuperscript2subscript¯superscript2 ≤ 2E p,D D% L_ πδ ^D 2E p,D% Dδ ^D≤ ( divide start_ARG 2 E ( p , D ) square-root start_ARG D end_ARG end_ARG start_ARG Lover¯ start_ARG π end_ARG δ end_ARG )D ( divide start_ARG 2 E ( p , D ) square-root start_ARG D end_ARG end_ARG start_ARG δ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (137) Recall that the generalisation error bound is P⁢(supπ∈Π|RG⁢(π)−RG^n⁢(π)|≥ϵ)≤2⁢infα∈(0,1)Cov⁢(Π,dr,α⁢ϵ4)⁢e−2⁢(1−α)2⁢n⁢ϵ24⁢C2subscriptsupremumΠsubscriptsubscriptsubscript^italic-ϵ2subscriptinfimum01CovΠsubscriptitalic-ϵ4superscript2superscript12superscriptitalic-ϵ24superscript2P ( _π∈ |R_G π -R_ G_n% π |≥ε )≤ 2 _α∈ % 0,1 Cov ( ,d_r, αε4% )e^- 2 1-α ^2nε^24C^2P ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ 2 infitalic_α ∈ ( 0 , 1 ) Cov ( Π , ditalic_r , divide start_ARG α ϵ end_ARG start_ARG 4 end_ARG ) e- divide start_ARG 2 ( 1 - α ) start_POSTSUPERSCRIPT 2 n ϵ2 end_ARG start_ARG 4 C2 end_ARG end_POSTSUPERSCRIPT (138) For simplicity let α=1212α= 12α = divide start_ARG 1 end_ARG start_ARG 2 end_ARG. So, set ϵ8italic-ϵ8 ε8divide start_ARG ϵ end_ARG start_ARG 8 end_ARG =6⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δabsent6subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ =6L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδ= 6 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (139) So δ δ =ϵ48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯.absentitalic-ϵ48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ = ε48L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π.= divide start_ARG ϵ end_ARG start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG end_ARG . (140) and κ κ =Lϕ⁢∥Θ~∥p⁢Lπ¯⁢ϵ48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯absentsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ = L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πε48L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π= divide start_ARG Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ϵ end_ARG start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG end_ARG (141) =ϵ48absentitalic-ϵ48 = ε48= divide start_ARG ϵ end_ARG start_ARG 48 end_ARG (142) We have P⁢(supπ∈Π|RG⁢(π)−RG^n⁢(π)|≥ϵ)subscriptsupremumΠsubscriptsubscriptsubscript^italic-ϵ P ( _π∈ |R_G π -R_% G_n π |≥ε )P ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) (143) ≤Cov⁢(Π,dr,ϵ/8)⁢e−n⁢ϵ28⁢C2absentCovΠsubscriptitalic-ϵ8superscriptsuperscriptitalic-ϵ28superscript2 ≤ Cov ,d_r,ε/8 e^% - nε^28C^2≤ Cov ( Π , ditalic_r , ϵ / 8 ) e- divide start_ARG n ϵ start_POSTSUPERSCRIPT 2 end_ARG start_ARG 8 C2 end_ARG end_POSTSUPERSCRIPT (144) ≤e−n⁢ϵ28⁢C2⁢(96⁢Lϕ⁢∥Θ~∥p⁢E⁢(p,D)⁢Dϵ)D⁢(96⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E⁢(p,D)⁢Dϵ)Dabsentsuperscriptsuperscriptitalic-ϵ28superscript2superscript96subscriptitalic-ϕsubscriptdelimited-∥~Θitalic-ϵsuperscript96subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ ≤ e^- nε^28C^2 96L_φ% _pE p,D % Dε ^D 96L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πE % p,D Dε ^D≤ e- divide start_ARG n ϵ start_POSTSUPERSCRIPT 2 end_ARG start_ARG 8 C2 end_ARG end_POSTSUPERSCRIPT ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p E ( p , D ) square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG )D ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E ( p , D ) square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (145) For simplices, E⁢[p,D]≤11E[p,D]≤ 1E [ p , D ] ≤ 1. So let’s say we want the probility upper bound to be ω, then the number of samples n we need to generalise is ω ω =e−n⁢ϵ28⁢C2⁢(96⁢Lϕ⁢∥Θ~∥p⁢Dϵ)D⁢(96⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢Dϵ)Dabsentsuperscriptsuperscriptitalic-ϵ28superscript2superscript96subscriptitalic-ϕsubscriptdelimited-∥~Θitalic-ϵsuperscript96subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ =e^- nε^28C^2 96L_φ% _p Dε ^D% 96L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ π Dε ^D= e- divide start_ARG n ϵ start_POSTSUPERSCRIPT 2 end_ARG start_ARG 8 C2 end_ARG end_POSTSUPERSCRIPT ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG )D ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG ) start_POSTSUPERSCRIPT D end_POSTSUPERSCRIPT (146) n n =Ω⁢(Dϵ2⁢(96⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢Dϵ)D⁢log⁡(96⁢Lϕ⁢∥Θ~∥p⁢Dϵ)−log⁡ω)absentΩsuperscriptitalic-ϵ2superscript96subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯italic-ϵ96subscriptitalic-ϕsubscriptdelimited-∥~Θitalic-ϵ = Dε^2 96L_% φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% π Dε ^D 96L_φ% _p Dε - % ω = Ω ( divide start_ARG D end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG )D log ( divide start_ARG 96 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p square-root start_ARG D end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (147) ∎ C.2 Proof of Theorem 6 Theorem 6 (Bounding sample complexity of learning without proxy) Let Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯)̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π ̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) be the subset of Π̊Π over̊ start_ARG Π end_ARG where π is Lϕ⁢∥Θ~∥p⁢Lπ¯subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯L_φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πLitalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG-Lipschitz. Set κ=ϵ24italic-ϵ24κ= ε24κ = divide start_ARG ϵ end_ARG start_ARG 24 end_ARG, we need Ω⁢(D′ϵ2⁢(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)D′⁢log⁡(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)−log⁡ω)Ωsuperscript′italic-ϵ2superscript48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵsuperscript′48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵ D ε^2 % 48L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πE p,D D^% ε ^D 48L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πE p,D D % ε - ω Ω ( divide start_ARG D′ end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT log ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (148) samples to generalise, where D′≫Dmuch-greater-thansuperscript′D D′ ≫ D , and E′⁢(p,D′)≫1much-greater-thansuperscript′1E p,D 1E′ ( p , D′ ) ≫ 1. That is, whenever n′≥n⁢(ϵ,ω)superscript′italic-ϵn ≥ n(ε,ω)n′ ≥ n ( ϵ , ω ), we have P⁢(supπ∈Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯)|RG⁢(π)−RG^n′⁢(π)|≥ϵ)≤δsubscriptsupremum̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscriptsubscriptsubscript^superscript′italic-ϵ P ( _π∈ L_φ % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _pL_ π % |R_G π -R_ G_n % π |≥ε )≤ ( supitalic_π ∈ over̊ start_ARG Π end_ARG ( L start_POSTSUBSCRIPT ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) end_POSTSUBSCRIPT | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n′ end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) ≤ δ (149) Proof of Theorem 6. For learning Π⁢(ϕ~,Θ~,τ~∘,Lπ)Π~italic-ϕ~Θsuperscript~subscript [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 φ, [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 , [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 τ ,L_π Π ( over~ start_ARG ϕ end_ARG , over~ start_ARG Θ end_ARG , over~ start_ARG τ end_ARG∘ , Litalic_π ), the covering number bound is as in Eq 145. For learning Π̊Π over̊ start_ARG Π end_ARG with the same Lipschitz constant (i.e. Lϕ⁢∥Θ~∥p⁢Lπ¯subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯L_φ [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πLitalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG) as above but without proxy data, the covering number bound can be read off from Elesedy (2022). Since XX is a discrete space, we use the p-norm-induced metric in the embedding space of XX; denote the embedding function f. Additionally denote the feasible subset in subscriptP_YPcaligraphic_Y by PP. Denote the hypothesis class as Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯)̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ π ̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) to mean the subset with smallest Lipschitz-constant in the argument, Cov⁢(Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯),dr,2⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ+κ)Cov̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov L_φ % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _pL_ π % ,d_r,2L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδ+κ ( over̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) , ditalic_r , 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ + κ ) (150) =Cov⁢(,d,κ)Cov⁢(f⁢(),dp,δ)absentCovsuperscriptsubscriptsubscriptCovsubscript = Cov ,d_P_Y,κ% Cov f ,% d_p,δ = Cov ( Y , dcaligraphic_P start_POSTSUBSCRIPT Y end_POSTSUBSCRIPT , κ )Cov ( f ( X ) , ditalic_p , δ ) (151) ≤(2⁢E′⁢(p,D′)⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢D′κ)D′⁢(2⁢E′⁢(p,D′)⁢D′δ)D′absentsuperscript2superscript′subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript2superscript′superscript′ ≤ 2E p,D % L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ π D κ ^D % 2E p,D D % δ ^D ≤ ( divide start_ARG 2 E′ ( p , D′ ) Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG square-root start_ARG D′ end_ARG end_ARG start_ARG κ end_ARG )D start_POSTSUPERSCRIPT ′ ( divide start_ARG 2 E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG δ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT (152) Setting κ=Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯κ=L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδκ = Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ, Cov⁢(Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯),dr,2⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ+κ)Cov̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ Cov L_φ % [rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolor% rgb0.609375,0.1328125,0.1484375 _pL_ π % ,d_r,2L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδ+κ ( over̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) , ditalic_r , 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ + κ ) (153) ≤(2⁢E′⁢(p,D′)⁢D′δ)D′⁢(2⁢E′⁢(p,D′)⁢D′δ)D′absentsuperscript2superscript′superscript′superscript2superscript′superscript′ ≤ 2E p,D % D δ ^D 2E% p,D D δ% ^D ≤ ( divide start_ARG 2 E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG δ end_ARG )D start_POSTSUPERSCRIPT ′ ( divide start_ARG 2 E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG δ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT (154) then setting ϵ/8=3⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δitalic-ϵ83subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯ε/8=3L_φ [rgb]0.609375,0.1328125,0.1484375% [named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πδϵ / 8 = 3 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ (i.e. setting κ=ϵ24italic-ϵ24κ= ε24κ = divide start_ARG ϵ end_ARG start_ARG 24 end_ARG) P⁢(supπ∈Π|RG⁢(π)−RG^n⁢(π)|≥ϵ)subscriptsupremumΠsubscriptsubscriptsubscript^italic-ϵ P ( _π∈ |R_G π -R_% G_n π |≥ε )P ( supitalic_π ∈ Π | Ritalic_G ( π ) - Rover start_ARG G end_ARG start_POSTSUBSCRIPT n end_POSTSUBSCRIPT ( π ) | ≥ ϵ ) (155) ≤Cov⁢(Π̊⁢(Lϕ⁢∥Θ~∥p⁢Lπ¯),dr,2⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢δ+κ)⁢e−n⁢ϵ28⁢C2absentCov̊Πsubscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯subscript2subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscriptsuperscriptitalic-ϵ28superscript2 ≤ Cov L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% π ,d_r,2L_φ [rgb]% 0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πδ+% κ e^- nε^28C^2≤ Cov ( over̊ start_ARG Π end_ARG ( Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG ) , ditalic_r , 2 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG δ + κ ) e- divide start_ARG n ϵ start_POSTSUPERSCRIPT 2 end_ARG start_ARG 8 C2 end_ARG end_POSTSUPERSCRIPT (156) ≤e−n⁢ϵ28⁢C2⁢(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)D′⁢(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)D′absentsuperscriptsuperscriptitalic-ϵ28superscript2superscript48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵsuperscript′48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵsuperscript′ ≤ e^- nε^28C^2 48L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πE p,D D % ε ^D 48L_φ [% rgb]0.609375,0.1328125,0.1484375 [named]pgfstrokecolorrgb% 0.609375,0.1328125,0.1484375 _pL_ πE % p,D D ε % ^D ≤ e- divide start_ARG n ϵ start_POSTSUPERSCRIPT 2 end_ARG start_ARG 8 C2 end_ARG end_POSTSUPERSCRIPT ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG )D start_POSTSUPERSCRIPT ′ ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT (157) A similar analysis show that we need n n =Ω⁢(D′ϵ2⁢(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)D′⁢log⁡(48⁢Lϕ⁢∥Θ~∥p⁢Lπ¯⁢E′⁢(p,D′)⁢D′ϵ)−log⁡ω)absentΩsuperscript′italic-ϵ2superscript48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵsuperscript′48subscriptitalic-ϕsubscriptdelimited-∥~Θsubscript¯superscript′superscript′italic-ϵ = D ε^2 % 48L_φ [rgb]0.609375,0.1328125,0.1484375 [% named]pgfstrokecolorrgb0.609375,0.1328125,0.1484375 % _pL_ πE p,D D^% ε ^D 48L_φ% [rgb]0.609375,0.1328125,0.1484375 [named]% pgfstrokecolorrgb0.609375,0.1328125,0.1484375 _pL_% πE p,D D % ε - ω = Ω ( divide start_ARG D′ end_ARG start_ARG ϵ2 end_ARG ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG )D start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT log ( divide start_ARG 48 Litalic_ϕ ∥ over~ start_ARG Θ end_ARG ∥p Lover¯ start_ARG π end_ARG E′ ( p , D′ ) square-root start_ARG D′ end_ARG end_ARG start_ARG ϵ end_ARG ) - log ω ) (158) ∎

Instant research discovery