Paper deep dive
The Computational Complexity of Circuit Discovery for Inner Interpretability
Federico Adolfi, Martina G. Vilas, Todd Wareham
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/12/2026, 6:54:52 PM
Summary
This paper provides a formal computational complexity analysis of 'circuit discovery' in neural networks, specifically multi-layer perceptrons. It establishes a conceptual framework for interpretability queries (description, explanation, prediction, control), proves that many such queries are intractable (NP-hard, W[1]-hard, or inapproximable), and identifies specific tractable subsets and transformations to guide future interpretability research.
Entities (6)
Relation Signals (3)
Federico Adolfi ā authored ā The Computational Complexity of Circuit Discovery for Inner Interpretability
confidence 100% Ā· The Computational Complexity of Circuit Discovery for Inner Interpretability Federico Adolfi
Circuit Discovery ā isanalyzedusing ā Computational Complexity Theory
confidence 100% Ā· we study circuit discovery with classical and parameterized computational complexity theory
Multi-Layer Perceptron ā issubjectof ā Circuit Discovery
confidence 95% Ā· we use it to settle the complexity of many query variants and relaxations of practical interest on multi-layer perceptrons.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Many proposed applications of neural networks in machine learning, cognitive/brain science, and society hinge on the feasibility of inner interpretability via circuit discovery. This calls for empirical and theoretical explorations of viable algorithmic options. Despite advances in the design and testing of heuristics, there are concerns about their scalability and faithfulness at a time when we lack understanding of the complexity properties of the problems they are deployed to solve. To address this, we study circuit discovery with classical and parameterized computational complexity theory: (1) we describe a conceptual scaffolding to reason about circuit finding queries in terms of affordances for description, explanation, prediction and control; (2) we formalize a comprehensive set of queries for mechanistic explanation, and propose a formal framework for their analysis; (3) we use it to settle the complexity of many query variants and relaxations of practical interest on multi-layer perceptrons. Our findings reveal a challenging complexity landscape. Many queries are intractable, remain fixed-parameter intractable relative to model/circuit features, and inapproximable under additive, multiplicative, and probabilistic approximation schemes. To navigate this landscape, we prove there exist transformations to tackle some of these hard problems with better-understood heuristics, and prove the tractability or fixed-parameter tractability of more modest queries which retain useful affordances. This framework allows us to understand the scope and limits of interpretability queries, explore viable options, and compare their resource demands on existing and future architectures.
Tags
Links
- Source: https://arxiv.org/abs/2410.08025
- Canonical: https://arxiv.org/abs/2410.08025
Trouble viewing inline? Open PDF directly ā
Full Text
492,494 characters extracted from source content.
Expand or collapse full text
The Computational Complexity of Circuit Discovery for Inner Interpretability Federico Adolfi ESI Neuroscience, Max-Planck Society & University of Bristol fede.adolfi@bristol.ac.uk &Martina G. Vilas Department of Computer Science Goethe University Frankfurt martinagvilas@em.uni-frankfurt.de &Todd Wareham Department of Computer Science Memorial University of Newfoundland harold@mun.ca Abstract Many proposed applications of neural networks in machine learning, cognitive/brain science, and society hinge on the feasibility of inner interpretability via circuit discovery. This calls for empirical and theoretical explorations of viable algorithmic options. Despite advances in the design and testing of heuristics, there are concerns about their scalability and faithfulness at a time when we lack understanding of the complexity properties of the problems they are deployed to solve. To address this, we study circuit discovery with classical and parameterized computational complexity theory: (1) we describe a conceptual scaffolding to reason about circuit finding queries in terms of affordances for description, explanation, prediction and control; (2) we formalize a comprehensive set of queries for mechanistic explanation, and propose a formal framework for their analysis; (3) we use it to settle the complexity of many query variants and relaxations of practical interest on multi-layer perceptrons. Our findings reveal a challenging complexity landscape. Many queries are intractable, remain fixed-parameter intractable relative to model/circuit features, and inapproximable under additive, multiplicative, and probabilistic approximation schemes. To navigate this landscape, we prove there exist transformations to tackle some of these hard problems with better-understood heuristics, and prove the tractability or fixed-parameter tractability of more modest queries which retain useful affordances. This framework allows us to understand the scope and limits of interpretability queries, explore viable options, and compare their resource demands on existing and future architectures. 1 Introduction As artificial neural networks (ANNs) grow in size and capabilities, Inner Interpretability ā an emerging field tasked with explaining their inner workings (RƤuker et al., 2023; Vilas et al., 2024a) ā attempts to devise scalable, automated procedures to understand systems mechanistically. Many proposed applications of neural networks in machine learning, cognitive and brain sciences, and society, hinge on the feasibility of inner interpretability. For instance, we might have to rely on interpretability methods to improve system safety (Bereska & Gavves, 2024), detect and control vulnerabilities (GarcĆa-Carrasco et al., 2024), prune for efficiency (Hooker et al., 2021), find and use task subnetworks (Zhang et al., 2024), explain internal concepts underlying decisions (Lee et al., 2023), experiment with neuro-cognitive models of language, vision, etc. (Lindsay, 2024; Lindsay & Bau, 2023; Pavlick, 2023), describe determinants of ANN-brain alignment (Feghhi et al., 2024; Oota et al., 2023), improve architectures, and extract domain insights (RƤuker et al., 2023). We will have to solve different instances of these interpretability problems, ideally automatically, for increasingly large models. We therefore need efficient interpretability procedures, and this requires empirical and theoretical explorations of viable algorithmic options. Circuit discovery and its challenges. Since top-down approaches to inner interpretability (see Vilas et al., 2024a) work their way down from high-level concepts or algorithmic hypotheses (Lieberum et al., 2023), there is interest in a complementary bottom-up methodology: circuit discovery (see Shi et al., 2024; Tigges et al., 2024). It starts from neuron- and circuit-level isolation or description (e.g., Hoang-Xuan et al., 2024; Lepori et al., 2023) and attempts to build up higher-level abstractions. The motivation is the circuit hypothesis: models might implement their capabilities via small subnetworks (Shi et al., 2024). Advances in the design and testing of interpretability heuristics (see Shi et al., 2024; Tigges et al., 2024) come alongside interest in the automation of circuit discovery (e.g., Conmy et al., 2023; Ferrando & Voita, 2024; Syed et al., 2023) and concerns about its feasibility (Voss et al., 2021; RƤuker et al., 2023). One challenge is scaling up methods to larger networks, more naturalistic datasets, and more complex tasks (e.g., Lieberum et al., 2023; Marks et al., 2024), given their manual-intensive search over large spaces (Voss et al., 2021). A related issue is that current heuristics, though sometimes promising (e.g., Merullo et al., 2024), often yield discrepant results (see e.g., Shi et al., 2024; Niu et al., 2023; Zhang & Nanda, 2023). They often find circuits that are not functionally faithful (Yu et al., 2024a) or lack the expected affordances (e.g., effects on behavior; Shi et al., 2024). This questions whether certain localization methods yield results that inform editing (Hase et al., 2023), and vice versa (Wang & Veitch, 2024). More broadly, we run into āinterpretability illusionsā (Friedman et al., 2024) when our simplifications (e.g., circuits) mimic the local input-output behavior of the system but lack global faithfulness (Jacovi & Goldberg, 2020). Exploring viable algorithmic options. These challenges come at a time when, despite emerging theoretical frameworks (e.g., Vilas et al., 2024a; Geiger et al., 2024), there are notable gaps in the formalization and analysis of the computational problems that interpretability heuristics attempt to solve (see Wang & Veitch, 2024, Ā§8). Issues around scalability of circuit discovery and faithfulness have a natural formulation in the language of Computational Complexity Theory (Arora & Barak, 2009; Downey & Fellows, 2013). A fundamental source of breakdown of scalability ā which lack of faithfulness is one manifestation of ā is the intrinsic resource demands of interpretability problems. In order to design efficient and effective solutions, we need to understand the complexity properties of circuit discovery queries and the constraints that might be leveraged to yield the desired results. Although experimental efforts have made promising inroads, the complexity-theoretic properties that naturally impact scalability and faithfulness remain open questions (see e.g., Subercaseaux, 2020, Ā§6C). We settle them here by complementing these efforts with a systematic study of the computational complexity of circuit discovery for inner interpretability. We present a framework that allows us to (a) understand the scope and limits of interpretability queries for description/explanation and prediction/control, (b) explore viable options, and (c) compare their resource demands among existing and future architectures. 1.1 Contributions ā¢ We present a conceptual scaffolding to reason about circuit finding queries in terms of affordances for description, explanation, prediction and control. ā¢ We formalize a comprehensive set of queries that capture mechanistic explanation, and propose a formal framework for their analysis. ā¢ We use this framework to settle the complexity of many query variants, parameterizations, approximation schemes and relaxations of practical interest on multi-layer perceptrons, relevant to various architectures such as transformers. ā¢ We demonstrate how our proof techniques can also be useful to draw links between interpretability and explainability by using them to improve existing results on the latter. 1.2 Overview of results ā¢ We uncover a challenging complexity landscape (see Table 4) where many queries are intractable (NP-hard, Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-hard), remain fixed-parameter intractable (W[1]-hard) when constraining model/circuit features (e.g., depth), and are inapproximable under additive, multiplicative, and probabilistic approximation schemes. ā¢ We prove there exist transformations to potentially tackle some hard problems (NP- vs. Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-complete) with better-understood heuristics, and prove the tractability (PTIME) or fixed-parameter tractability (FPT) of other queries of interest, and we identify open problems. ā¢ We describe a quasi-minimality property of ANN circuits and exploit it to generate tractable queries which retain useful affordances as well as efficient algorithms to compute them. ā¢ We establish a separation between local and global query complexity. Together with quasi-minimality, this explains interpretability illusions of faithfulness observed in experiments. 1.3 Related work This paper gives the first systematic exploration of the computational complexity of inner interpretability problems.111 This work expands on FAās PhD dissertation at University of Bristol (Adolfi, 2023; Adolfi et al., 2024). An adjacent area is the complexity analysis of explainability problems (Bassan & Katz, 2023; Ordyniak et al., 2023). It differs from our work in its focus on input queries ā aspects of the input that explain model decisions ā as we look at the inner workings of neural networks via circuit queries. BarcelĆ³ et al. (2020) study the explainability of multi-layer perceptrons compared to simpler models through a set of input queries. Bassan et al. (2024) extend this idea with a comparison between local and global explainability. None of these works formalize or analyze circuit queries (although Subercaseaux, 2020, identifies it as an open problem); we adapt the local versus global distinction in our framework and show how our proof techniques can tighten some results on explainability queries. Ramaswamy (2019) and Adolfi & van Rooij (2023) explore a small set of circuit queries and only on abstract biological networks modeled as general graphs, which cannot inform circuit discovery in ANNs. Efforts in characterizing the complexity of learning neural networks (e.g., Song et al., 2017; Chen et al., 2020; Livni et al., 2014) might eventually connect to our work, although a number of differences between the formalizations makes results in one area difficult to predict from those in the other. Likewise, efforts to settle the complexity of finding small circuits consistent with a truth table (Hitchcock & Pavan, 2015) are currently too general to be applicable to interpretability problems. More generally, we join efforts to build a solid theoretical foundation for interpretability (Bassan & Katz, 2023; Geiger et al., 2024; Vilas et al., 2024a). 2 Mechanistic understanding of neural networks Mechanistic understanding is a contentious topic (Ross & Bassett, 2024), but for our purposes it will suffice to adopt a pragmatic perspective. In many cases of practical interest, we want our interpretability methods to output objects that allow us to, in some limited sense, (1) describe or explain succinctly, and (2) control or predict precisely. Such objects (e.g., circuits) should be āefficiently queriableā; they are often referred to as āa way of making an explanation tractableā (Cao & Yamins, 2023). Roughly, this means that we would like short descriptions (e.g., small circuits) with useful affordances (e.g., to readily answer questions and perform interventions of interest). Circuits have the potential to fulfill these criteria (Olah et al., 2020). Here we preview some special circuits with useful properties which we formalize and analyze later on. Table 1 maps the main circuits we study to their corresponding affordances for description, explanation, prediction and control. Formal definitions of circuit queries are given alongside results in Section 4 (see also Appendix: Definitions, Theorems and Proofs). Table 1: Circuit affordances for description, explanation, prediction, and control. Circuit Affordance Description / Explanation Prediction / Control Sufficient Circuit Which neurons suffice in isolation to cause a behavior? Minimum: shortest description. Inference in isolation. Minimal: ablating any neuron breaks behavior of the circuit. Quasi-minimal Sufficient Circuit Which neurons suffice in isolation to cause a behavior and which is a breaking point? Ablating the breaking point breaks behavior of the circuit. Necessary Circuit Which neurons are part of all circuits for a behavior? Key subcomputations? Ablating the neurons breaks behavior of any sufficient circuit in the network. Circuit Ablation & Clamping Which neurons are necessary in the current configuration of the network? Ablating/Clamping the neurons breaks behavior of the network. Circuit Robustness How much redundancy supports a behavior? Resilience to perturbations. Ablating any set of neurons of size below threshold does not break behavior. Patched Circuit Which neurons drive a behavior in a given input context, i.e., are control nodes? Patching neurons changes network behavior for inputs of interest. Steering; Editing. Quasi-minimal Patched Circuit Which neurons can drive a behavior in a given input context and which neuron is a breaking point? Patching neurons causes target behavior for inputs of interest; Unpatching breaking point breaks target behavior. Gnostic Neurons Which neurons respond preferentially to a certain concept? Concept editing; guided synthesis. 3 Inner interpretability queries as computational problems We model post-hoc interpretability queries on neural networks as computational problems in order to analyze their intrinsic complexity properties. These circuit queries also formalize criteria for desired circuits, including those appearing in the literature, such as āfaithfulnessā, ācompletenessā, and āminimalityā (Wang et al., 2022; Yu et al., 2024a). Query variants: coverage, size and minimality. The coverage of a circuit is the domain over which it behaves in a certain way (e.g., faithful to the modelās prediction). Local circuits do so over a finite set of known inputs and global circuits do so over all possible inputs. The size of a circuit is the number of neurons. Some circuit queries require circuits of bounded size whereas others leave the size unbounded. A circuit with a certain property (e.g., local sufficiency) is minimal if there is no subset of its neurons that also has that property (cf. minimum size among all such circuits present in the network; see Figure 1). Figure 1: Relationships between circuit types. Sufficient Circuits (SCs) are faithful to the model. The entire network is a trivial SC. Necessary Circuits (NCs) are units shared by all minimal SCs. Quasi-minimal SCs contain a known breaking point (here, NC) and unknown superfluous units. To fit our comprehensive suite of problems, we explain how to generate problem variants and later on only present one representative definition of each. Problem 0 (ProblemName (PN)) Input: A multi-layer perceptron ā³MM, CoverageIN, SizeIN. Output: A Property circuit CC of ā³MM, SizeOUT, s.t. CoverageOUT ā¢()=ā³ā¢()ā³C(x)\!=\!M(x)C ( x ) = M ( x ), Suffix. Input: and Table 2 illustrate how to generate problem variants using a template, and ProblemName = Sufficient Circuit as an example (e.g., the Coverage[IN/OUT] variables specify parts of the input/output description that vary according to whether the requested circuit must have global or local faithfulness). Problem definitions will be given for search (return specified circuits) or decision (answer yes/no circuit queries) versions. Others, including optimization (return maximum/minimum-size circuits), can be generated by assigning variables. Problems presented later on are obtained similarly. We also explore various parameterizations, approximation schemes, and relaxations that we explain in the following sections as needed. Table 2: Generating query variants from problem templates. Description variables Query variants Local Global Bounded Unbounded Optimal Bounded Unbounded Optimal CoverageIN an input xx an input xx an input xx ā__ā ā__ā ā__ā CoverageOUT ā__ā ā__ā ā__ā āsubscriptfor-all _xāx āsubscriptfor-all _xāx āsubscriptfor-all _xāx SizeIN int. uā¤|ā³|ā³uā¤|M|u ā¤ | M | ā__ā ā__ā int. uā¤|ā³|ā³uā¤|M|u ā¤ | M | ā__ā ā__ā SizeOUT size ||ā¤u|C|ā¤ u| C | ā¤ u ā__ā min. size size ||ā¤u|C|ā¤ u| C | ā¤ u ā__ā min. size Property minimal / ā__ā minimal / ā__ā ā__ā minimal / ā__ā minimal / ā__ā ā__ā Suffix if it exists, otherwise ā„bottom ā„ ā__ā ā__ā if it exists, otherwise ā„bottom ā„ ā__ā ā__ā 3.1 Complexity analyses Classical and parameterized complexity. We prove theorems about interpretability queries building on techniques from classical (Garey & Johnson, 1979) and parameterized complexity (Downey & Fellows, 2013). Given our limited knowledge of the problem space of interpretability, worst-case analysis is appropriate to explore which problems might be solvable without requiring any additional assumptions (e.g., Bassan et al., 2024; BarcelĆ³ et al., 2020), and experimental results suggest it captures a lower bound on real-world complexity (e.g., Friedman et al., 2024; Shi et al., 2024; Yu et al., 2024a). Here we give a brief, informal overview of the main concepts underlying our analyses (see Appendix: Definitions, Theorems and Proofs for extensive formal definitions). We will explore beyond classical polynomial-time tractability (PTIME) by studying fixed-parameter tractability (FPT), a more novel and finer-grained look at the sources of complexity of problems to test aspects that possibly make interpretability feasible in practice. NP-hard queries are considered intractable because they cannot be computed by polynomial-time algorithms. A relaxation is to allow unreasonable (e.g., exponential) resource demands to be confined to problem parameters that can be kept small in practice. Parameterizing a given ANN and requested circuit leads to parameterized problems (see Table 3 for problem parameters we study later). Parameterized queries in the class FPT admit fixed-parameter tractable algorithms. W-hard queries (by analogy: to FPT as NP-hard is to PTIME), however, do not. We study counting problems via analogous classes #P and #W[1]. We also investigate completeness for NP and classes higher up the polynomial hierarchy such as Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 and Ī 2psubscriptsuperscriptĪ 2 ^p_2Ī italic_p2 to identify aspects of hard problems that make them even harder, and to explore the possibility to tackle hard interpretability problems with better-understood methods for well-known NP-complete problems (de Haan & Szeider, 2017). Most proofs involve reductions between computational problems which establish the complexity status of interpretability queries based on the known complexity of canonical problems in other areas. Table 3: Model and circuit parameterizations. Parameter Model (given) Circuit (requested) Number of layers (depth) L^ Lover start_ARG L end_ARG l^ lover start_ARG l end_ARG Maximum layer width L^wsubscript L_wover start_ARG L end_ARGw l^wsubscript l_wover start_ARG l end_ARGw Total number of units22footnotemark: 2 U^=|ā³|ā¤L^ā L^w^ā³ā ^subscript U=|M|ā¤ LĀ· L_wover start_ARG U end_ARG = | M | ā¤ over start_ARG L end_ARG ā over start_ARG L end_ARGw ||=u^^|C|= u| C | = over start_ARG u end_ARG Number of input units U^Isubscript U_Iover start_ARG U end_ARGI u^Isubscript u_Iover start_ARG u end_ARGI Number of output units U^Osubscript U_Oover start_ARG U end_ARGO u^Osubscript u_Oover start_ARG u end_ARGO Maximum weight W^ Wover start_ARG W end_ARG w^ wover start_ARG w end_ARG Maximum bias B^ Bover start_ARG B end_ARG b^ bover start_ARG b end_ARG Approximation. Although sometimes computing optimal solutions is intractable, it is conceivable we could devise tractable interpretability procedures to obtain approximate solutions that are useful in practice. We consider 5 notions of approximation: additive, multiplicative, and three probabilistic schemes (=c,PTAS,3PAPTAS3PAA=\c,PTAS,3PA\A = c , PTAS , 3PA ; see Appendix: Definitions, Theorems and Proofs for formal definitions). Additive approximation algorithms return solutions at most a fixed distance c away from optimal (e.g., from the minimum-sized circuit), ensuring that errors cannot get impractically large (c-approximability). Multiplicative approximation returns solutions at most a factor of optimal away. Some hard problems allow for polynomial-time multiplicative approximation schemes (PTAS) where we can get arbitrarily close to optimal solutions as long as we expend increasing compute time (Ausiello et al., 1999). Finally, we consider three types of probabilistic polynomial-time approximability (henceforth 3PA) that may be acceptable in situations where always getting the correct output for an input is not required: algorithms that (1) always run in polynomial time and produce the correct output for a given input in all but a small number of cases (Hemaspaandra & Williams, 2012); (2) always run in polynomial time and produce the correct output for a given input with high probability (Motwani & Raghavan, 1995); and (3) run in polynomial time with high probability but are always correct (Gill, 1977). Model architecture. The Multi-Layer Perceptron (MLP) is a natural first step in our exploration because (a) it is proving useful as a stepping stone in current experimental (e.g., Lampinen et al., 2024) and theoretical work (e.g., Rossem & Saxe, 2024; McInerney & Burke, 2023); (b) it exists as a leading standalone architecture (Yu et al., 2024b), as the central element of all-MLP architectures (Tolstikhin et al., 2021), and as a key component of state-of-the-art models such as transformers (Vaswani et al., 2017); (c) it is of active interest to the interpretability community (e.g., Geva et al., 2022; 2021; Dai et al., 2022; Meng et al., 2024; 2022; Niu et al., 2023; Vilas et al., 2024b; Hanna et al., 2023); and (d) we can relate our findings in inner interpretability to those in explainability, which also begins with MLPs (e.g., BarcelĆ³ et al., 2020; Bassan et al., 2024). Although MLP blocks can be taken as units to simplify search, it is recommended to investigate MLPs by treating each neuron as a unit (e.g., Gurnee et al., 2023; Cammarata et al., 2020; Olah et al., 2017), as it better reflects the semantics of computations in ANNs (Lieberum et al., 2023, sec. 2.3.1). We adopt this perspective in our analyses. We write ā³MM for an MLP model and ā³ā¢()ā³M(x)M ( x ) for its output on input vector xx. Its size |ā³|ā³|M|| M | is the number of neurons. A circuit CC is a subset of |||C|| C | neurons which induce a (possibly end-to-end) subgraph of ā³MM (see Appendix: Definitions, Theorems and Proofs for formal definitions). 4 Results & Discussion: the complexity of circuit queries In this section we present each circuit query with its computational problem and a discussion of the complexity profile we obtain across variants, relaxations, and parameterizations. For an overview of the results for all queries, see Table 4. Proofs of the theorems can be found in the Appendix: Definitions, Theorems and Proofs. 4.1 Sufficient Circuit Sufficient circuits (SCs) are sets of neurons connected end-to-end that suffice, in isolation, to reproduce some model behavior over an input domain (see faithfulness; Wang et al., 2022; Yu et al., 2024a). They are conceptually related to the desired outcome of zero-ablating components that do not contribute to the behavior of interest (small, parameter-efficient subnetworks). Zero-ablation as a method (e.g., to find sufficient circuits) has been criticized on the grounds that the patched value (zero) is somewhat arbitrary and therefore can mischaracterize the functioning of the neuron/circuit when operating in the context of the rest of the network during inference. This gives rise to alternative methods such as activation patching with activation means or specific input activations, which we study later. SCs remain relevant as, despite valid criticisms of zero-ablation (e.g., Conmy et al., 2023), circuit discovery through pruning might be justified at least in some cases (Yu et al., 2024a). Problem 1 (Bounded Local Sufficient Circuit (BLSC)) Input: A multi-layer perceptron ā³MM, an input vector xx, and an integer uā¤|ā³|ā³uā¤|M|u ā¤ | M |. Output: A circuit CC in ā³MM of size ||ā¤u|C|ā¤ u| C | ā¤ u, such that ā¢()=ā³ā¢()ā³C(x)=M(x)C ( x ) = M ( x ), if it exists, otherwise ā„bottom ā„. We find that many variants of SC are NP-hard (see Table 4). Counterintuitively, this intractability does not depend straightforwardly on parameters such as network depth (W[1]-hard relative to PP). Therefore, hardness is not mitigated by keeping models shallow. Given this barrier, we explore the possibility of obtaining approximate solutions but find that hard SC variants are inapproximable relative to all schemes in Table 3. An alternative is to consider the membership of these problems in a well-studied class whose solvers are better understood than interpretability heuristics (de Haan & Szeider, 2017). We prove that local versions of SC are NP-complete. This implies there exist efficient transformations from instances of SC to those of the satisfiability problem (SAT; Biere et al., 2021), opening up the possibility to borrow techniques that work reasonably well in practice for SAT that might be suitable for some versions of neural network problems. Interestingly, this is not possible for the global version, which we prove is complete for a class higher up the complexity hierarchy (Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-complete). This result establishes a formal separation between local and global query complexity that partly explains āinterpretability illusionsā (Friedman et al., 2024; Yu et al., 2024a), which we conjecture holds for other queries we investigate later. These illusions come about when an interpretability abstraction (e.g., a circuit) seems empirically faithful to its target (e.g., model behavior) by some criterion (e.g., ālocalā tests on a dataset), but actually lacks faithfulness in the way it generalizes to other criteria (e.g., tests of its āglobalā behavior outside the original distribution). Next we explore whether we could diagnose if SCs with some desired property (e.g., minimality) are abundant, which would be informative of the ability of heuristic search to stumble upon one of them. We analyze various queries where the output is a count of SCs (i.e., counting problems). We find that both local and global, bounded and unbounded variants are #P-complete and remain intractable (#W[1]-hard) when parameterized by many network features including depth (Table 3). The hardness profile of SC over all these variants calls for exploring more substantial relaxations. We introduce the notion of quasi-minimality for this purpose (similar to Ramaswamy, 2019) and later demonstrate its usefulness beyond this particular problem. Any neuron in a minimal/minimum SC is a breaking point in the sense that removing it will break the target behavior. In quasi-minimal SCs we are merely guaranteed to know at least one neuron that causes this breakdown. By introducing this relaxation, which gives up some affordances but retains others of interest, we get a feasible interpretability query. Problem 2 (Unbounded Quasi-minimal Local Sufficient Circuit (UQLSC)) Input: A multi-layer perceptron ā³MM, and an input vector xx. Output: A circuit CC in ā³MM and a neuron vāv ā C s.t. ā¢()=ā³ā¢()ā³C(x)=M(x)C ( x ) = M ( x ) and [āv]ā¢()ā ā³ā¢()delimited-[]ā³[C \v\](x) (x)[ C ā v ] ( x ) ā M ( x ). Input: is in PTIME. We describe an efficient algorithm to compute it which can be heuristically biased towards finding smaller circuits and combined with techniques that exploit weights and gradients (see Appendix: Definitions, Theorems and Proofs). 4.2 Gnostic Neuron Gnostic neurons, sometimes called āgrandmother neuronsā in neuroscience (Gale et al., 2020) and āconcept neuronsā or āobject detectorsā in AI (e.g., Bau et al., 2020), are one of the oldest and still current interpretability queries of interest (see also āknowledge neuronsā; Niu et al., 2023). Problem 3 (Bounded Gnostic Neurons (BGN)) Input: A multi-layer perceptron ā³MM and two sets of input vectors XX and YY, an integer k, and an activation threshold t. Output: A set of neurons V in ā³MM of size |V|ā„k|V|ā„ k| V | ā„ k such that āvāVsubscriptfor-all _vā Vāv ā V it is the case that āāsubscriptfor-all _x āx ā X, ā³ā¢()ā³M(x)M ( x ) produces activations Avā„tsubscriptsuperscriptA^v_xā„ tAitalic_vbold_x ā„ t and āā:Av<t:subscriptfor-allsubscriptsuperscript _y :A^v_y<tāy ā Y : Aitalic_vbold_y < t, if it exists, else ā„bottom ā„. Input: is in PTIME. Alternatives might require GNs to have some behavioral effect when intervened; such variants would remain tractable. 4.3 Circuit Ablation and Clamping The idea that some neurons perform key subcomputations for certain tasks naturally leads to the hypothesis that ablating them should have downstream effects on the corresponding model behaviors. Searching for neuron sets with this property has been one strategy (i.e., zero-ablation) to get at important circuits (Wang & Veitch, 2024). The circuit ablation (CA) problem formalizes this idea. Problem 4 (Bounded Local Circuit Ablation (BLCA)) Input: A multi-layer perceptron ā³MM, an input vector xx, and an integer uā¤|ā³|ā³uā¤|M|u ā¤ | M |. Output: A neuron set CC in ā³MM of size ||ā¤u|C|ā¤ u| C | ā¤ u, s.t. [ā³ā]ā¢()ā ā³ā¢()delimited-[]ā³[M ](x) (x)[ M ā C ] ( x ) ā M ( x ), if it exists, else ā„bottom ā„. A difference between CAs and minimal SCs is that the former can be interpreted as a possibly non-minimal breaking set in the context of the whole network whereas the latter is by default a minimal breaking set when the SC is taken in isolation. In this sense, CA can be seen as a less stringent criterion for circuit affordances. A related idea is circuit clamping (C): fixing the activations of certain neurons to a level that produces a change in the behavior of interest. Problem 5 (Bounded Local Circuit Clamping (BLCC)) Input: A multi-layer perceptron ā³MM, vector xx, value r, and an integer u s.t. 1<uā¤|ā³|1ā³1<uā¤|M|1 < u ā¤ | M |. Output: A subset of neurons CC in ā³MM of size ||ā¤u|C|ā¤ u| C | ā¤ u, such that for the ā³āsuperscriptā³M^*Mā induced by clamping all cāc ā C to value r, ā³āā¢()ā ā³ā¢()superscriptā³M^*(x) (x)Mā ( x ) ā M ( x ), if it exists, otherwise ā„bottom ā„. Despite these more modest criteria, we find that both the local and global variants of CA and C are NP-hard, fixed-parameter intractable W[1]-hard relative to various parameters, and inapproximable in all 5 senses studied. However, we prove these problems are NP-complete, which opens up practical options not available for other problems we study (see remarks in Section 4.1). 4.4 Circuit Patching A critique of zero-ablation is the arbitrariness of the value, leading to alternatives such as mean-ablation (e.g., Wang et al., 2022). This contrasts studying circuits in isolation versus embedded in surrounding subnetworks. Activation patching (Ghandeharioun et al., 2024; Zhang & Nanda, 2023; Hanna et al., 2024) and path patching (Goldowsky-Dill et al., 2023) try to pinpoint which activations play an in-context role in model behavior, which inspires the circuit patching (CP) problem. Problem 6 (Bounded Local Circuit Patching (BLCP)) Input: A multi-layer perceptron ā³MM, an integer k, an input vector yy, and a vector set XX. Output: A subset CC in ā³MM of size ||ā¤k|C|ā¤ k| C | ā¤ k, such that for the ā³āsuperscriptā³M^*Mā induced by patching CC with activations from ā³ā¢()ā³M(y)M ( y ) and ā³āā³M ā C with activations from ā³ā¢()ā³M(x)M ( x ), ā³āā¢()=ā³ā¢()superscriptā³M^*(x)=M(y)Mā ( x ) = M ( y ) for all āx ā X, if it exists, otherwise ā„bottom ā„. We find that local/global variants are intractable (NP-hard) in a way that does not depend on parameters such as network depth or size of the patched circuit (W[1]-hard), and are inapproximable (c,PTAS,3PAPTAS3PA\c,PTAS,3PA\ c , PTAS , 3PA -inapprox.). Although we also prove the local variant of CP is NP-complete and therefore approachable in practice with solvers for hard problems not available for the global variants (see remarks in Section 4.1), these complexity barriers motivate exploring further relaxations. With some modifications the idea of quasi-minimality can be repurposed to do useful work here. Problem 7 (Unbounded Quasi-minimal Local Circuit Patching (UQLCP)) Input: A multi-layer perceptron ā³MM, an input vector yy, and a set XX of input vectors. Output: A subset CC in ā³MM and a neuron vāv ā C, such that for the ā³āsuperscriptā³M^*Mā induced by patching CC with activations from ā³ā¢()ā³M(y)M ( y ) and ā³āā³M ā C with activations from ā³ā¢()ā³M(x)M ( x ), āā:ā³āā¢()=ā³ā¢():subscriptfor-allsuperscriptā³ _x :M^*(x)=M(% y)āx ā X : Mā ( x ) = M ( y ), and for ā³ā²M Mā² induced by patching identically except for vāv ā C, āā:ā³ā²ā¢()ā ā³ā¢():subscriptsuperscriptā³ā²ā³ _x :M (x)ā % M(y)āx ā X : Mā² ( x ) ā M ( y ). In this way we obtain a tractable query (PTIME) for quasi-minimal patching, sidestepping barriers while retaining some useful affordances (see Table 1). We present an algorithm to compute Input: efficiently that can be combined with strategies exploiting weights and gradients (see Appendix: Definitions, Theorems and Proofs). Table 4: Classical and parameterized complexity results by problem variant. Classical & parameterized queries333Circuits are bounded-size unless otherwise stated. Each cell contains the complexity of the problem variant in terms of classical and FP (in)tractability, membership in complexity classes, and (in)approximability (=c,PTAS,3PAPTAS3PAA=\c,PTAS,3PA\A = c , PTAS , 3PA ). ā?ā marks potentially fruitful open problems. āN/Aā stands for not applicable. =ā³āŖsubscriptā³subscriptP=P_M _CP = Pcaligraphic_M āŖ Pcaligraphic_C ā³=L^,U^I,U^O,W^,B^subscriptā³^subscript^subscript^^^P_M=\ L, U_I, U_O, W, B\Pcaligraphic_M = over start_ARG L end_ARG , over start_ARG U end_ARGI , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG =l^,l^w,u^,u^I,u^O,w^,b^subscript^subscript^^subscript^subscript^^^P_C=\ l, l_w, u, u_I, u_O% , w, b\Pcaligraphic_C = over start_ARG l end_ARG , over start_ARG l end_ARGw , over start_ARG u end_ARG , over start_ARG u end_ARGI , over start_ARG u end_ARGO , over start_ARG w end_ARG , over start_ARG b end_ARG Problem variants Local Global Decision/Search Optimization Decision/Search Optimization Sufficient Circuit (SC) NP-complete AA-inapprox. Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-complete AA-inapprox. PP-SC W[1]-hard AA-inapprox. W[1]-hard AA-inapprox. Minimal SC NP-complete ? āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard ? PP-Minimal SC W[1]-hard ? W[1]-hard ? Unbounded Minimal SC ? N A ? N A PP-Unbounded Minimal SC ? ? Unbounded Quasi-Minimal SC PTIME ? Count SC #P-complete N A #P-hard N A PP-Count SC #W[1]-hard #W[1]-hard Count Minimal SC #P-complete #P-hard PP-Count Minimal SC #W[1]-hard #W[1]-hard Count Unbounded Minimal SC #P-complete #P-hard Gnostic Neuron (GN) PTIME N/A ? N/A Circuit Ablation (CA) NP-complete AA-inapprox. āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard AA-inapprox. L^,U^I,U^O,W^,B^,u^^subscript^subscript^^^^\ L, U_I, U_O, W, B, u\ over start_ARG L end_ARG , over start_ARG U end_ARGI , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG , over start_ARG u end_ARG -CA W[1]-hard AA-inapprox. W[1]-hard AA-inapprox. Circuit Clamping (C) NP-complete AA-inapprox. āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard AA-inapprox. L^,U^O,W^,B^,u^^subscript^^^^\ L, U_O, W, B, u\ over start_ARG L end_ARG , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG , over start_ARG u end_ARG -C W[1]-hard AA-inapprox. W[1]-hard AA-inapprox. Circuit Patching (CP) NP-complete AA-inapprox. āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard AA-inapprox. L^,U^O,W^,B^,u^^subscript^^^^\ L, U_O, W, B, u\ over start_ARG L end_ARG , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG , over start_ARG u end_ARG -CP W[2]-hard AA-inapprox. W[2]-hard AA-inapprox. Unbounded Quasi-Minimal CP PTIME N/A ? N/A Necessary Circuit (NC) āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard AA-inapprox. āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard AA-inapprox. L^,U^I,U^O,W^,u^^subscript^subscript^^^\ L, U_I, U_O, W, u\ over start_ARG L end_ARG , over start_ARG U end_ARGI , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG u end_ARG -NC W[1]-hard AA-inapprox. W[1]-hard AA-inapprox. Circuit Robustness (CR) coNP-complete ? āĪ 2pabsentsubscriptsuperscriptĪ 2ā ^p_2ā Ī italic_p2 coNP-hard ? L^,U^I,U^O,W^,B^,u^^subscript^subscript^^^^\ L, U_I, U_O, W, B, u\ over start_ARG L end_ARG , over start_ARG U end_ARGI , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG , over start_ARG u end_ARG -CR coW[1]-hard ? coW[1]-hard ? |H||H|| H |-CR FPT FPT ? ? |H||H|| H |, U^Isubscript U_Iover start_ARG U end_ARGI-CR FPT FPT FPT FPT Sufficient Reasons (SR) āĪ£2pabsentsubscriptsuperscriptĪ£2ā ^p_2ā Ī£italic_p2 NP-hard 3PA-inapprox. N A L^,U^O,W^,B^,u^^subscript^^^^\ L, U_O, W, B, u\ over start_ARG L end_ARG , over start_ARG U end_ARGO , over start_ARG W end_ARG , over start_ARG B end_ARG , over start_ARG u end_ARG -SR W[1]-hard 3PA-inapprox. 4.5 Necessary Circuit The criterion of necessity is a stringent one, and consequently necessary circuits (NCs) carry powerful affordances (see Table 1). Since neurons in NCs collectively interact with all possible sufficient circuits for a target behavior, they are candidates to describe key task subcomputations and intervening on them is guaranteed to have effects even in the presence of high redundance. This relates to the notion of circuit overlap and therefore to efforts in identifying circuits shared by various tasks (e.g., Merullo et al., 2024), and the link between overlap and faithfulness (e.g., Hanna et al., 2024). Problem 8 (Bounded Global Necessary Circuit (BGNC)) Input: A multi-layer perceptron ā³MM, and an integer k. Output: A subset SS of neurons in ā³MM of size ||ā¤k|S|ā¤ k| S | ā¤ k, such that ā©ā ā S ā ā© C ā ā for every circuit CC in ā³MM that is sufficient relative to all possible input vectors, if it exists, otherwise ā„bottom ā„. Unfortunately both local and global versions of NC are NP-hard (in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2; Table 4), remain intractable even when keeping parameters such as network depth, number of input and output neurons, and others small (Table 3), and does not admit any of the available approximation schemes (Table 3). Tractable versions of NC are unlikely unless substantial restrictions or relaxations are introduced. 4.6 Circuit Robustness A behavior of interest might be over-determined or resilient in the sense that many circuits in the model implement it and one can take over when the other breaks down. This is related to the notion of redundancy used in neuroscience (e.g., Nanda et al., 2023). Intuitively, when a model implements a task in this way, the behavior should be more robust to a number of perturbations. The possibility of verifying this property experimentally motivates the circuit robustness (CR) problem, and a related interpretability effort is diagnosing nodes that are excluded by circuit discovery procedures but still have an impact on behavior (false negatives; KramĆ”r et al., 2024). Problem 9 (Bounded Local Circuit Robustness (BLCR)) Input: A multi-layer perceptron ā³MM, a subset H of ā³MM, an input vector xx, and an integer k with 1ā¤kā¤|H|11ā¤ kā¤|H|1 ā¤ k ā¤ | H |. Output: <YES> if for each Hā²āHsuperscriptā²H Hā² ā H, with |Hā²|ā¤ksuperscriptā²|H |ā¤ k| Hā² | ā¤ k, ā³ā¢()=[ā³āHā²]ā¢()ā³delimited-[]ā³superscriptā²M(x)=[M H ](x)M ( x ) = [ M ā Hā² ] ( x ), else <NO>. We find that Local CR is coNP-complete while Global CR is in Ī 2psubscriptsuperscriptĪ 2 ^p_2Ī italic_p2 and coNP-hard. It remains fixed-parameter intractable (coW[1]-hard) relative to model parameters (Table 3). Pushing further, we explore parameterizing CR by |H|\|H|\ | H | and prove fixed-parameter tractability of |H|\|H|\ | H | -CR which holds both for the local and global versions. There exist algorithms for CR that scale well as long as |H||H|| H | is reasonable; a scenario that might be useful to probe robustness in practice. This wraps up our results for circuit queries. We briefly digress into explainability before discussing some implications. 4.7 Sufficient Reasons Understanding the sufficient reasons (SR) for a model decision in terms of input features consists of knowledge of values of the input components that are enough to determine the output. Given a model decision on an input, the most interesting reasons are those with the least components. Problem 10 (Bounded Local Sufficient Reasons (BLSR)) Input: A multi-layer perceptron ā³MM, an input vector xx of length ||=u^Isubscript^|x|= u_I| x | = over start_ARG u end_ARGI, and an integer k with 1ā¤kā¤u^I1subscript^1ā¤ kā¤ u_I1 ā¤ k ā¤ over start_ARG u end_ARGI. Output: A subset ssuperscriptx^sxitalic_s of xx of size |s|=ksuperscript|x^s|=k| xitalic_s | = k, such that for every possible completion csuperscriptx^cxitalic_c of ssuperscriptx^sxitalic_s ā³ā¢()=ā³ā¢()ā³superscriptā³M(x^c)=M(x)M ( xbold_c ) = M ( x ), if it exists, otherwise ā„bottom ā„. To demonstrate the usefulness of our framework beyond inner interpretability, we show how it links to explainability. Using our techniques for circuit queries, we significantly tighten existing results for SR (BarcelĆ³ et al., 2020; WƤldchen et al., 2021) by proving that hardness (NP-hard, W[1]-hard, 3PA-inapprox.) holds even when the model has only one hidden layer. 5 Implications, limitations, and future directions We presented a framework based on parameterized complexity to accompany experiments on inner interpretability with theoretical explorations of viable algorithms. With this grasp of circuit query complexity, we can understand the challenges of scalability and the mixed outcomes of experiments with heuristics for circuit discovery. There is ample complexity-theoretic evidence that there is a limit (often underestimated) to how good the performance of heuristics on intractable problems can be (Hemaspaandra & Williams, 2012). We can explain āinterpretability illusionsā (Friedman et al., 2024) due to lack of faithfulness, minimality (e.g., Shi et al., 2024; Yu et al., 2024a) and other affordances (Wang & Veitch, 2024; Hase et al., 2023), in terms of the kinds of circuits that our current heuristics are well-equipped to discover. For instance, consider the algorithm for automated circuit discovery proposed by Conmy et al. (2023), which eliminates one network component at a time if the consequence on behavior is reasonably small. Since this algorithm runs in polynomial time, it is not likely to solve the problems proven hard here, such as Minimal Sufficient Circuit. However, one reason we observe interesting results in some cases is because it is well-equipped to solve Quasi-Minimal Circuit problems. As our conceptual and formal analyses show, quasi-minimal circuits can mimic various desirable aspects of sufficient circuits (Table 1), and the former can be found tractably (results for Input: and Input:). At the same time, understanding these properties of circuit discovery heuristics helps us explain observed discrepancies: why we often see (1) lack of faithfulness (i.e., global coverage is out of reach for QMC algorithms), (2) non-minimality (i.e., QM circuits can have many non-breaking points), and (3) large variability in performance across tasks and analysis parameters (e.g., Shi et al., 2024; Conmy et al., 2023). Although we find that many queries of interest are intractable in the general case (and empirical results are in line with this characterization), this should not paralyze efforts to interpret neural network models. As our exploration of the current complexity landscape shows, reasonable relaxations, restrictions and problem variants can yield tractable queries for circuits with useful properties. Consider a few out of many possible avenues to continue these explorations. (i) Study query parameters. Faced with an intractable query, we can investigate which parameters of the problem (e.g., network or circuit aspects) might be responsible for its core hardness. If these problematic parameters can be kept small in real-world applications, this yields a fixed-parameter tractable query. We have explored some, but more are possible as any aspect of the problem can be parameterized. A close dialogue between theorists and experimentalists is important for this, as empirical regularities suggest which parameters might be fruitful to explore theoretically, and experiments test whether theoretically conjectured parameters can be kept small in practice. (i) Generate novel queries. Our formalization of quasi-minimal circuit problems illustrates the search for viable algorithmic options with examples of tractable problems for inner interpretability. When the use case is well defined, efficient queries that return circuits with useful affordances for applications can be designed. Alternative circuits might also mimic the affordances for prediction/control of ideal circuits while avoiding intractability. (i) Explore network output as axis of approximation. Some of our constructions use binary input/output (following previous work; e.g., Bassan et al., 2024; BarcelĆ³ et al., 2020). Although continuous output does not necessarily matter complexity-wise (see Appendix: Definitions, Theorems and Proofs for [counter]examples), this is an interesting direction for future work, as it opens the door to studying the network output as an axis of approximation, which in turn might be a useful relaxation. (iv) Design more abstract queries. A different path is to design queries that partially rely on mid-level abstractions (Vilas et al., 2024a) to bridge the gap between circuits and human-intelligible algorithms (e.g., key-value mechanisms; Geva et al., 2022; Vilas et al., 2024b). (v) Characterize actual network structure. It is in principle possible that some real-world, trained neural networks possess internal structure that is benevolent to general (ideal) circuit queries (e.g., redundancy; see Appendix: Definitions, Theorems and Proofs). In such optimistic scenarios, general-purpose heuristics might work well. The empirical evidence available to date, however, speaks against this. In any case, it will always be important to characterize any such structure to use it explicitly to design algorithms with useful guarantees. (vi) Compare resource demands of interpretability/explainability across architectures. Our results for inner interpretability complement those of explainability (e.g., BarcelĆ³ et al., 2020; Bassan et al., 2024; WƤldchen et al., 2021). These aspects can be studied together for different architectures to assess their intrinsic interpretability. To some extent our results already transfer to some cases of interest. Since the transformer architecture contains MLPs, the complexity status of our circuit queries bears on neuron-level circuit discovery efforts in transformers (e.g., Large Language/Vision/Audio models). Acknowledgments We thank Ronald de Haan for comments on proving membership using alternating quantifier formulas. References Adolfi (2023) Federico Adolfi. Computational Meta-Theory in Cognitive Science: A Theoretical Computer Science Framework. PhD thesis, University of Bristol, Bristol, UK, 2023. Adolfi & van Rooij (2023) Federico Adolfi and Iris van Rooij. Resource demands of an implementationist approach to cognition. In Proceedings of the 21st International Conference on Cognitive Modeling, 2023. Adolfi et al. (2024) Federico Adolfi, Martina G. Vilas, and Todd Wareham. Complexity-Theoretic Limits on the Promises of Artificial Neural Network Reverse-Engineering. Proceedings of the Annual Meeting of the Cognitive Science Society, 46(0), 2024. Arora & Barak (2009) Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach. Cambridge University Press, Cambridge ; New York, 2009. ISBN 978-0-521-42426-4. Arora et al. (1998) Sanjeev Arora, Carsten Lund, Rajeev Motwani, Madhu Sudan, and Mario Szegedy. Proof verification and the hardness of approximation problems. Journal of the ACM (JACM), 45(3):501ā555, 1998. Ausiello et al. (1999) Giorgio Ausiello, Alberto Marchetti-Spaccamela, Pierluigi Crescenzi, Giorgio Gambosi, Marco Protasi, and Viggo Kann. Complexity and Approximation. Springer Berlin Heidelberg, Berlin, Heidelberg, 1999. ISBN 978-3-642-63581-6 978-3-642-58412-1. BarcelĆ³ et al. (2020) Pablo BarcelĆ³, MikaĆ«l Monet, Jorge PĆ©rez, and Bernardo Subercaseaux. Model Interpretability through the lens of Computational Complexity. In Advances in Neural Information Processing Systems, volume 33, p. 15487ā15498. Curran Associates, Inc., 2020. Bassan & Katz (2023) Shahaf Bassan and Guy Katz. Towards Formal XAI: Formally Approximate Minimal Explanations of Neural Networks. In Tools and Algorithms for the Construction and Analysis of Systems: 29th International Conference, TACAS 2023, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2023, Paris, France, April 22ā27, 2023, Proceedings, Part I, p. 187ā207, Berlin, Heidelberg, 2023. Springer-Verlag. ISBN 978-3-031-30822-2. Bassan et al. (2024) Shahaf Bassan, Guy Amir, and Guy Katz. Local vs. Global Interpretability: A Computational Complexity Perspective. In Proceedings of the 41st International Conference on Machine Learning, p. 3133ā3167. PMLR, 2024. Bau et al. (2020) David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071ā30078, 2020. Bereska & Gavves (2024) Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. Survey Certification, Expert Certification. Biere et al. (2021) Armin Biere, Marijn J. H. Heule, Hans van Maaren, and Toby Walsh (eds.). Handbook of Satisfiability. Number Volume 336,1 in Frontiers in Artificial Intelligence and Applications. IOS Press, Amsterdam Berlin Washington, DC, second edition edition, 2021. ISBN 978-1-64368-160-3. Cammarata et al. (2020) Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 5(3):e24, 2020. Cao & Yamins (2023) Rosa Cao and Daniel Yamins. Explanatory models in neuroscience, Part 2: Functional intelligibility and the contravariance principle. Cognitive Systems Research, p. 101200, 2023. Chen et al. (2020) Sitan Chen, Adam R. Klivans, and Raghu Meka. Learning Deep ReLU Networks Is Fixed-Parameter Tractable. (arXiv:2009.13512), 2020. Chen & Lin (2019) Yijia Chen and Bingkai Lin. The Constant Inapproximability of the Parameterized Dominating Set Problem. SIAM Journal on Computing, 48(2):513ā533, 2019. Conmy et al. (2023) Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and AdriĆ Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability. Advances in Neural Information Processing Systems, 36:16318ā16352, 2023. Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge Neurons in Pretrained Transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8493ā8502, Dublin, Ireland, 2022. Association for Computational Linguistics. de Haan & Szeider (2017) Ronald de Haan and Stefan Szeider. Parameterized complexity classes beyond para-NP. Journal of Computer and System Sciences, 87:16ā57, 2017. Downey & Fellows (1999) R. G. Downey and M. R. Fellows. Parameterized Complexity. Monographs in Computer Science. Springer New York, New York, NY, 1999. ISBN 978-1-4612-6798-0 978-1-4612-0515-9. Downey & Fellows (2013) Rod G. Downey and Michael R. Fellows. Fundamentals of Parameterized Complexity. Texts in Computer Science. Springer, London [u.a.], 2013. ISBN 978-1-4471-5559-1. Feghhi et al. (2024) Ebrahim Feghhi, Nima Hadidi, Bryan Song, Idan A. Blank, and Jonathan C. Kao. What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores. (arXiv:2406.01538), 2024. Ferrando & Voita (2024) Javier Ferrando and Elena Voita. Information Flow Routes: Automatically Interpreting Language Models at Scale. (arXiv:2403.00824), 2024. Flum & Grohe (2006) Jƶrg Flum and Martin Grohe. Parameterized Complexity Theory. Texts in Theoretical Computer Science. Springer, Berlin Heidelberg New York, 2006. ISBN 978-3-540-29953-0. Fortnow (2009) Lance Fortnow. The status of the P versus NP problem. Communications of the ACM, 52(9):78ā86, 2009. Friedman et al. (2024) Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability Illusions in the Generalization of Simplified Models. (arXiv:2312.03656), 2024. Gale et al. (2020) Ella M. Gale, Nicholas Martin, Ryan Blything, Anh Nguyen, and Jeffrey S. Bowers. Are there any āobject detectorsā in the hidden layers of CNNs trained to identify objects or scenes? Vision Research, 176:60ā71, 2020. GarcĆa-Carrasco et al. (2024) Jorge GarcĆa-Carrasco, Alejandro MatĆ©, and Juan Trujillo. Detecting and Understanding Vulnerabilities in Language Models via Mechanistic Interpretability. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, p. 385ā393, 2024. Garey & Johnson (1979) Michael R Garey and David S Johnson. Computers and intractability. W.H. Freeman, 1979. Gasarch et al. (1995) William I. Gasarch, Mark W. Krentel, and Kevin J. Rappoport. OptP as the normal behavior of NP-complete problems. Mathematical Systems Theory, 28(6):487ā514, 1995. Geiger et al. (2024) Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. (arXiv:2301.04709), 2024. Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Value Memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 5484ā5495, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 30ā45, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. Ghandeharioun et al. (2024) Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. In Proceedings of the 41st International Conference on Machine Learning, p. 15466ā15490. PMLR, 2024. Gill (1977) John Gill. Computational Complexity of Probabilistic Turing Machines. SIAM Journal on Computing, 6(4):675ā695, 1977. Goldowsky-Dill et al. (2023) Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing Model Behavior with Path Patching. (arXiv:2304.05969), 2023. Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding Neurons in a Haystack: Case Studies with Sparse Probing. Transactions on Machine Learning Research, 2023. Hanna et al. (2023) Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-Seventh Conference on Neural Information Processing Systems, 2023. Hanna et al. (2024) Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. (arXiv:2403.17806), 2024. Hase et al. (2023) Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. Advances in Neural Information Processing Systems, 36:17643ā17668, 2023. Hemaspaandra & Williams (2012) Lane A. Hemaspaandra and Ryan Williams. SIGACT News Complexity Theory Column 76: An atypical survey of typical-case heuristic algorithms. ACM SIGACT News, 43(4):70ā89, 2012. Hitchcock & Pavan (2015) John M. Hitchcock and A. Pavan. On the NP-Completeness of the Minimum Circuit Size Problem. LIPIcs, Volume 45, FSTTCS 2015, 45:236ā245, 2015. Hoang-Xuan et al. (2024) Nhat Hoang-Xuan, Minh Vu, and My T. Thai. LLM-assisted Concept Discovery: Automatically Identifying and Explaining Neuron Functions. (arXiv:2406.08572), 2024. Hooker et al. (2021) Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What Do Compressed Deep Neural Networks Forget? (arXiv:1911.05248), 2021. Jacovi & Goldberg (2020) Alon Jacovi and Yoav Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4198ā4205, Online, 2020. Association for Computational Linguistics. Karmarkar (1984) N. Karmarkar. A new polynomial-time algorithm for linear programming. Combinatorica, 4(4):373ā395, 1984. KramĆ”r et al. (2024) JĆ”nos KramĆ”r, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP*: An efficient and scalable method for localizing LLM behaviour to components. (arXiv:2403.00745), 2024. Krentel (1988) MW Krentel. The complexity of optimization functions. Journal of Computer and System Sciences, 36(3):490ā509, 1988. Lampinen et al. (2024) Andrew Kyle Lampinen, Stephanie C. Y. Chan, and Katherine Hermann. Learned feature representations are biased by complexity, learning order, position, and more. Transactions on Machine Learning Research, 2024. Lee et al. (2023) Jae Hee Lee, Sergio Lanza, and Stefan Wermter. From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networks. (arXiv:2310.11884), 2023. Lepori et al. (2023) Michael A. Lepori, Thomas Serre, and Ellie Pavlick. Uncovering Causal Variables in Transformers using Circuit Probing. (arXiv:2311.04354), 2023. Lieberum et al. (2023) Tom Lieberum, Matthew Rahtz, JĆ”nos KramĆ”r, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla. (arXiv:2307.09458), 2023. Lindsay (2024) Grace W Lindsay. Grounding neuroscience in behavioral changes using artificial neural networks. Current Opinion in Neurobiology, 2024. Lindsay & Bau (2023) Grace W. Lindsay and David Bau. Testing methods of neural systems understanding. Cognitive Systems Research, 82:101156, 2023. Livni et al. (2014) Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. On the Computational Efficiency of Training Neural Networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. Marks et al. (2024) Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. (arXiv:2403.19647), 2024. McInerney & Burke (2023) Andrew McInerney and Kevin Burke. Feedforward neural networks as statistical models: Improving interpretability through uncertainty quantification. (arXiv:2311.08139), 2023. Meng et al. (2022) Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-Editing Memory in a Transformer. In The Eleventh International Conference on Learning Representations, 2022. Meng et al. (2024) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ā22, p. 17359ā17372, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 978-1-71387-108-8. Merullo et al. (2024) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit Component Reuse Across Tasks in Transformer Language Models. In The Twelfth International Conference on Learning Representations, 2024. Motwani & Raghavan (1995) Rajeev Motwani and Prabhakar Raghavan. Randomized Algorithms. Cambridge University Press, Cambridge ; New York, 1995. ISBN 978-0-521-47465-8. Nanda et al. (2023) Vedant Nanda, Till Speicher, John Dickerson, Krishna Gummadi, Soheil Feizi, and Adrian Weller. Diffused redundancy in pre-trained representations. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, p. 4055ā4079. Curran Associates, Inc., 2023. Niu et al. (2023) Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the Knowledge Neuron Thesis Have to do with Knowledge? In The Twelfth International Conference on Learning Representations, 2023. Olah et al. (2017) Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(11):e7, 2017. Olah et al. (2020) Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024.001, 2020. Oota et al. (2023) Subba Reddy Oota, Emin Ćelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. (arXiv:2311.04664), 2023. Ordyniak et al. (2023) Sebastian Ordyniak, Giacomo Paesani, and Stefan Szeider. The Parameterized Complexity of Finding Concise Local Explanations. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, p. 3312ā3320, Macau, SAR China, 2023. International Joint Conferences on Artificial Intelligence Organization. ISBN 978-1-956792-03-4. Papadimitriou & Yannakakis (1991) C.H. Papadimitriou and M. Yannakakis. Optimization, approximation, and complexity classes. Journal of Computer and System Sciences, 43:425ā440, 1991. Pavlick (2023) Ellie Pavlick. Symbols and grounding in large language models. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2251):20220041, 2023. Provan & Ball (1983) J. Scott Provan and Michael O. Ball. The Complexity of Counting Cuts and of Computing the Probability that a Graph is Connected. SIAM Journal on Computing, 12(4):777ā788, 1983. Ramaswamy (2019) Venkatakrishnan Ramaswamy. An Algorithmic Barrier to Neural Circuit Understanding. bioRxiv, 2019. doi: 10.1101/639724. RƤuker et al. (2023) Tilman RƤuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), p. 464ā483. IEEE Computer Society, 2023. ISBN 978-1-66546-299-0. Ross & Bassett (2024) Lauren N. Ross and Dani S. Bassett. Causation in neuroscience: Keeping mechanism meaningful. Nature Reviews Neuroscience, 25(2):81ā90, 2024. Rossem & Saxe (2024) Loek Van Rossem and Andrew M. Saxe. When Representations Align: Universality in Representation Learning Dynamics. In Proceedings of the 41st International Conference on Machine Learning, p. 49098ā49121. PMLR, 2024. Schaefer & Umans (2002) Marcus Schaefer and Christopher Umans. Completeness in the polynomial-time hierarchy: A compendium. SIGACT News, 33(3):32ā49, 2002. Shi et al. (2024) Claudia Shi, Nicolas Beltran-Velez, Achille Nazaret, Carolina Zheng, AdriĆ Garriga-Alonso, Andrew Jesson, Maggie Makar, and David Blei. Hypothesis Testing the Circuit Hypothesis in LLMs. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. Song et al. (2017) Le Song, Santosh Vempala, John Wilmes, and Bo Xie. On the Complexity of Learning Neural Networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. Subercaseaux (2020) Bernardo AnĆbal Subercaseaux. Model Interpretability through the Lens of Computational Complexity. PhD thesis, Universidad de Chile, 2020. Syed et al. (2023) Aaquib Syed, Can Rager, and Arthur Conmy. Attribution Patching Outperforms Automated Circuit Discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023. Tigges et al. (2024) Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. LLM Circuit Analyses Are Consistent Across Training and Scale. (arXiv:2407.10827), 2024. Toda (1991) Seinosuke Toda. P is as Hard as the Polynomial-Time Hierarchy. SIAM Journal on Computing, 20(5):865ā877, 1991. Tolstikhin et al. (2021) Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An all-MLP Architecture for Vision. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021), volume 34, p. 24261ā24272. Curran Associates, Inc., 2021. Valiant (1979) Leslie G. Valiant. The Complexity of Enumeration and Reliability Problems. SIAM Journal on Computing, 8(3):410ā421, 1979. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Å ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. Vilas et al. (2024a) Martina G. Vilas, Federico Adolfi, David Poeppel, and Gemma Roig. Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience. In Proceedings of the 41st International Conference on Machine Learning, p. 49506ā49522. PMLR, 2024a. Vilas et al. (2024b) Martina G. Vilas, Timothy Schaumlƶffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ā23, p. 40030ā40041, Red Hook, NY, USA, 2024b. Curran Associates Inc. Voss et al. (2021) Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing Weights. Distill, 6(2):e00024.007, 2021. WƤldchen et al. (2021) Stephan WƤldchen, Jan Macdonald, Sascha Hauch, and Gitta Kutyniok. The computational complexity of understanding binary classifier decisions. Journal of Artificial Intelligence Research, 70:351ā387, 2021. Wang et al. (2022) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, 2022. Wang & Veitch (2024) Zihao Wang and Victor Veitch. Does Editing Provide Evidence for Localization? In ICML 2024 Workshop on Mechanistic Interpretability, 2024. Wareham (1999) Harold T. Wareham. Systematic Parameterized Complexity Analysis in Computational Phonology. PhD thesis, University of Victoria, Canada, 1999. Wareham (2022) Todd Wareham. Creating teams of simple agents for specified tasks: A computational complexity perspective. (arXiv:2205.02061), 2022. Yu et al. (2024a) Lei Yu, Jingcheng Niu, Zining Zhu, and Gerald Penn. Functional Faithfulness in the Wild: Circuit Discovery with Differentiable Computation Graph Pruning. (arXiv:2407.03779), 2024a. Yu et al. (2024b) Runpeng Yu, Weihao Yu, and Xinchao Wang. KAN or MLP: A Fairer Comparison. (arXiv:2407.16674), 2024b. Zhang et al. (2024) Enyan Zhang, Michael A. Lepori, and Ellie Pavlick. Instilling Inductive Biases with Subnetworks. (arXiv:2310.10899), 2024. Zhang & Nanda (2023) Fred Zhang and Neel Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. In The Twelfth International Conference on Learning Representations, 2023. Appendix: Definitions, Theorems and Proofs Appendix A Preliminaries Each section of this appendix is self-contained except for the following definitions. We re-state interpretability query definitions in a more detailed form in each section for convenience. To err here on the side of rigor, here we will use more cumbersome notation that we avoided in the main manuscript for succinctness. A.1 Computational problems of known complexity Some of our proofs construct reductions from the following computational problems. Clique (Garey & Johnson, 1979, Problem GT19) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer k. Question: Does G have a clique of size at least k, i.e., a subset Vā²āVsuperscriptā²V Vā² ā V, |Vā²|ā„ksuperscriptā²|V |ā„ k| Vā² | ā„ k, such that for all pairs v,vā²āVā²superscriptā²v,v ā V v , vā² ā Vā², (v,vā²)āEsuperscriptā²(v,v )ā E( v , vā² ) ā E? Vertex cover (VC) (Garey & Johnson, 1979, Problem GT1) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer k. Question: Does G contain a vertex cover of size at most k, i.e., a subset Vā²āVsuperscriptā²V Vā² ā V, |Vā²|ā¤ksuperscriptā²|V |ā¤ k| Vā² | ā¤ k, such that for all (u,v)āE(u,v)ā E( u , v ) ā E, at least one of u or v is in Vā²? Dominating set (DS) (Garey & Johnson, 1979, Problem GT2) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer k. Question: Does G contain a dominating set of size at most k, i.e., a subset Vā²āVsuperscriptā²V Vā² ā V, |Vā²|ā¤ksuperscriptā²|V |ā¤ k| Vā² | ā¤ k, such that for all vāVvā Vv ā V, either vāVā²vā V v ā Vā² or there is at least one vā²āVā²superscriptā²v ā V vā² ā Vā² such that (v,vā²)āEsuperscriptā²(v,v )ā E( v , vā² ) ā E? Hitting set (HS) (Garey & Johnson, 1979, Problem SP8) Input: A collection of subsets C of a finite set S and a positive integer k. Question: Is there a subset Sā² of S, |Sā²|ā¤ksuperscriptā²|S |ā¤ k| Sā² | ā¤ k, such that Sā² has a non-empty intersection with each set in C? Minimum DNF Tautology (3DT) (Schaefer & Umans, 2002, Problem L7) Input: A 3-DNF tautology Ļitalic-ĻĻĻ with T terms over a set of variables V and a positive integer k. Question: Is there a 3-DNF formula Ļā²italic-Ļā²Ļ Ļā² made up of ā¤kabsentā¤ kā¤ k of the terms in Ļitalic-ĻĻĻ that is a also a tautology? A.2 Classical and parameterized complexity Definition 1 (Polynomial-time tractability). An algorithm is said to run in polynomial-time if the number of steps it performs is Oā¢(nc)superscriptO(n^c)O ( nitalic_c ), where n is a measure of the input size and c is some constant. A problem Ī Ī is said to be tractable if it has a polynomial-time algorithm. P denotes the class of such problems. Consider a more fine-grained look at the sources of complexity of problems. The following is a relaxation of the notion of tractability, where unreasonable resource demands are allowed as long as they are constrained to a set of problem parameters. Definition 2 (Fixed-parameter tractability). Let PP be a set of problem parameters. A problem PP-Ī Ī is fixed-parameter tractable relative to PP if there exists an algorithm that computes solutions to instances of PP-Ī Ī of any size n in time fā¢()ā ncā superscriptf(P)Ā· n^cf ( P ) ā nitalic_c, where c is a constant and fā¢(ā )ā f(Ā·)f ( ā ) some computable function. FPT denotes the class of such problems and includes all problems in P. A.3 Hardness and reductions Most proof techniques in this work involve reductions between computational problems. Definition 3 (Reducibility). A problem Ī 1subscriptĪ 1 _1Ī 1 is polynomial-time reducible to Ī 2subscriptĪ 2 _2Ī 2 if there exists a polynomial-time algorithm (reduction) that transforms instances of Ī 1subscriptĪ 1 _1Ī 1 into instances of Ī 2subscriptĪ 2 _2Ī 2 such that solutions for Ī 2subscriptĪ 2 _2Ī 2 can be transformed in polynomial-time into solutions for Ī 1subscriptĪ 1 _1Ī 1. This implies that if a tractable algorithm for Ī 2subscriptĪ 2 _2Ī 2 exists, it can be used to solve Ī 1subscriptĪ 1 _1Ī 1 tractably. Fpt-reductions transform an instance (x,k)(x,k)( x , k ) of some problem parameterized by k into an instance (xā²,kā²)superscriptā²(x ,k )( xā² , kā² ) of another problem, with kā²ā¤gā¢(k)superscriptā²k ā¤ g(k)kā² ā¤ g ( k ), in time fā¢(k)ā pā¢(|x|)ā f(k)Ā· p(|x|)f ( k ) ā p ( | x | ) where p is a polynomial and gā¢(ā )ā g(Ā·)g ( ā ) is an arbitrary function. These reductions analogously transfer fixed-parameter tractability results between problems. Hardness results are generally conditional on two conjectures with extensive theoretical and empirical support. Intractability statements build on these as follows. Conjecture 1. P ā NP. Definition 4 (Polynomial-time intractability). The class NP contains all problems in P and more. Assuming 1, NP-hard problems lie outside P. These problems are considered intractable because they cannot be solved in polynomial-time (unless 1 is false; see Fortnow, 2009). Conjecture 2. FPT ā W[1]. Definition 5 (Fixed-parameter intractability). The class W[1] contains all problems in the class FPT and more. Assuming 2, W[1]-hard parameterized problems lie outside FPT. These problems are considered fixed-parameter intractable, relative to a given parameter set, because no fixed-parameter tractable algorithm can exist to solve them (unless 2 is false; see Downey & Fellows, 2013). The following two easily-proven lemmas will be useful in our parameterized complexity proofs. Lemma 1. (Wareham, 1999, Lemma 2.1.30) If problem Ī Ī is fp-tractable relative to aspect-set K then Ī Ī is fp-tractable for any aspect-set Kā² such that KāKā²ā K K ā Kā². Lemma 2. (Wareham, 1999, Lemma 2.1.31) If problem Ī Ī is fp-intractable relative to aspect-set K then Ī Ī is fp-intractable for any aspect-set Kā² such that Kā²āKsuperscriptā²K ā Kā² ā K. A.4 Approximation Although sometimes computing optimal solutions might be intractable, it is still conceivable that we could devise tractable procedures to obtain approximate solutions that are useful in practice. We consider two natural notions of additive and multiplicative approximation and three probabilistic schemes. A.4.1 Multiplicative approximation For a minimization problem Ī Ī , let Oā¢Pā¢TĪ ā¢(I)subscriptĪ OPT_ (I)O P Troman_Ī ( I ) be an optimal solution for Ī Ī on instance I, AĪ ā¢(I)subscriptĪ A_ (I)Aroman_Ī ( I ) be a solution for Ī Ī returned by an algorithm A, and mā¢(Oā¢Pā¢TĪ ā¢(I))subscriptĪ m(OPT_ (I))m ( O P Troman_Ī ( I ) ) and mā¢(AĪ ā¢(I))subscriptĪ m(A_ (I))m ( Aroman_Ī ( I ) ) be the values of these solutions. Definition 6 (Multiplicative approximation algorithm). [Ausiello et al. 1999, Def. 3.5]. Given a minimization problem Ī Ī , an algorithm A is a multiplicative Ļµitalic-ĻµĪµĻµ-approximation algorithm for Ī Ī if for each instance I of Ī Ī , mā¢(AĪ ā¢(I))āmā¢(Oā¢Pā¢TĪ ā¢(I))ā¤ĻµĆmā¢(Oā¢Pā¢TĪ ā¢(I))subscriptĪ subscriptĪ italic-ĻµsubscriptĪ m(A_ (I))-m(OPT_ (I))ā¤ĪµĆ m(OPT_ (I))m ( Aroman_Ī ( I ) ) - m ( O P Troman_Ī ( I ) ) ā¤ Ļµ Ć m ( O P Troman_Ī ( I ) ). It would be ideal if one could obtain approximate solutions for a problem Ī Ī that are arbitrarily close to optimal if one is willing to allow extra algorithm runtime. Definition 7 (Multiplicative approximation scheme). [Adapted from Ausiello et al., 1999, Def. 3.10]. Given a minimization problem Ī Ī , a polynomial-time approximation scheme (PTAS) for Ī Ī is a set AA of algorithms such that for each integer k>00k>0k > 0, there is a 1k1 1kdivide start_ARG 1 end_ARG start_ARG k end_ARG-approximation algorithm AĪ kāsubscriptsuperscriptĪ A^k_ ā AAitalic_kroman_Ī ā A that runs in time polynomial in |I||I|| I |. A.4.2 Additive approximation It would be useful to have guarantees that an approximation algorithm for our problems returns solutions at most a fixed distance away from optimal. This would ensure errors cannot get impractically large. Definition 8 (Additive approximation algorithm). [Adapted from Ausiello et al., 1999, Def. 3.3]. An algorithm AĪ subscriptĪ A_ Aroman_Ī for a problem Ī Ī is a d-additive approximation algorithm (d-A) if there exists a constant d such that for all instances x of Ī Ī the error between the value mā¢(ā )ā m(Ā·)m ( ā ) of an optimal solution oā¢pā¢tā¢sā¢oā¢lā¢(x)optsol(x)o p t s o l ( x ) and the output AĪ ā¢(x)subscriptĪ A_ (x)Aroman_Ī ( x ) is such that |mā¢(oā¢pā¢tā¢sā¢oā¢lā¢(x))āmā¢(AĪ ā¢(x))|ā¤dsubscriptĪ |\ m(optsol(x))-m(A_ (x))\ |ā¤ d| m ( o p t s o l ( x ) ) - m ( Aroman_Ī ( x ) ) | ā¤ d. A.4.3 Probabilistic approximation Finally, consider three other types of probabilistic polynomial-time approximability (henceforth 3PA) that may be acceptable in situations where always getting the correct output for an input is not required: (1) algorithms that always run in polynomial time and produce the correct output for a given input in all but a small number of cases (Hemaspaandra & Williams, 2012); (2) algorithms that always run in polynomial time and produce the correct output for a given input with high probability (Motwani & Raghavan, 1995); and (3) algorithms that run in polynomial time with high probability but are always correct (Gill, 1977). A.5 Model architecture Definition 9 (Multi-Layer Perceptron). [Adapted from BarcelĆ³ et al. 2020]. A multi-layer perceptron (MLP) is a neural network model ā³MM, with L^ Lover start_ARG L end_ARG layers, defined by sequences of weight matrices (1,2,ā¦,L^),iāādiā1Ćdisubscript1subscript2ā¦subscript^subscriptsuperscriptāsubscript1subscript(W_1,W_2,ā¦,W_ L), _i% ^d_i-1Ć d_i( W1 , W2 , ā¦ , Wover start_ARG L end_ARG ) , Witalic_i ā blackboard_Qditalic_i - 1 Ć ditalic_i, bias vectors (1,2,ā¦,L^),iāādisubscript1subscript2ā¦subscript^subscriptsuperscriptāsubscript(b_1,b_2,ā¦,b_ L), _i% ^d_i( b1 , b2 , ā¦ , bover start_ARG L end_ARG ) , bitalic_i ā blackboard_Qditalic_i, and (element-wise) ReLU functions (f1,f2,ā¦,fL^ā1),fiā¢(x):=maxā”(0,x).assignsubscript1subscript2ā¦subscript^1subscript0(f_1,f_2,ā¦,f_ L-1), f_i(x):= (0,x).( f1 , f2 , ā¦ , fover start_ARG L end_ARG - 1 ) , fitalic_i ( x ) := max ( 0 , x ) . The final function is, without loss of generality, the binary step function fL^ā¢(x):=1ā¢ifā¢xā„0,otherwiseā¢ 0.formulae-sequenceassignsubscript^1if0otherwise 0f_ L(x):=1\ if\ xā„ 0,otherwise\ 0.fover start_ARG L end_ARG ( x ) := 1 if x ā„ 0 , otherwise 0 . The computation rules for ā³MM are given by i:=fiā¢(iā1ā¢+i),0:=,formulae-sequenceassignsubscriptsubscriptsubscript1subscriptassignsubscript0h_i:=f_i(h_i-1W+b_i), % h_0:=x,hitalic_i := fitalic_i ( hitalic_i - 1 W + bitalic_i ) , h0 := x , where xx is the input. The output of ā³MM on xx is defined as ā³ā¢():=L^assignā³subscript^M(x):=h_ LM ( x ) := hover start_ARG L end_ARG. The graph Gā³=(V,E)subscriptā³G_M=(V,E)Gcaligraphic_M = ( V , E ) of ā³MM has a vertex for each component of each isubscripth_ihitalic_i. All vertices in layer i are connected by edges to all vertices of layer i+11i+1i + 1, with no intra-layer connections. Edges carry weights according to isubscriptW_iWitalic_i, and vertices carry the components of isubscriptb_ibitalic_i as biases. The size of ā³MM is defined as |ā³|:=|V|assignā³|M|:=|V|| M | := | V |. A.6 Preliminary remarks As is the case for other work in this area ((which might be called āApplied Complexity Theoryā Bassan et al., 2024; BarcelĆ³ et al., 2020), we are not aiming at developing new mathematical techniques but rather deploying existing mathematical tools to answer important questions that connect to applications. Part of the technical challenge we take up is to formalize problems of practical interest in simple (and if possible, elegant) ways that are readily understandable, and to prove their complexity properties efficiently (i.e., obtaining a one to many relation between proof constructions and meaningful results). This allows us to gain insights into the sources of complexity of problems, an investigation where the difficulty/complexity/intricacy of proofs are a liability. We use āinput queriesā to refer to computational problems in explainability and ācircuit queriesā for circuit discovery in inner interpretability. We make no claims as to whether one or the other query relates more to intuitive ideas of explanation or interpretation and merely use the latter as familiar pointers to the literature. All of our proofs for local problem variants assume a particular input vector I, be it the all-0 or all-1 vector. Note that we can simulate these vectors by having zero weights on the input lines and putting appropriate 0 and 1 biases on the input neurons (a technique developed and used in our later-derived proofs but readily applicable to earlier ones). This causes the input to be āignoredā, which renders our proofs correct under both integer and continuous inputs. Note this construction is in line with previous work (e.g., Bassan et al., 2024; BarcelĆ³ et al., 2020). Importantly, this highlights that the characteristics of the input are not important but rather there is a combinatorial āheartā beating at the center of our circuit problems; namely, the selection of a subcircuit from exponential number of subcircuits. This combinatorial core can, if not tamed by appropriate restrictions on network and input structure, give rise to non-polynomial worst-case algorithmic complexity. Proofs for global problem variants often employ constructions similar to those for local variants, with minor to medium (though crucial) differences. For completeness, the full construction is stated again to minimize errors and the need to check proofs other than those being examined. On the issue of real-world structure and formal complexity. One example scenario where real-world statistics might act as mitigating forces with respect to computational hardness (of the general problems) is the case of high redundancy (related to our Circuit Robustness problem). Redundancy can in some sense make circuit finding easier (as solutions are more abundant), but the benefit comes at a cost for interpretability through introducing identifiably issues. As circuits supporting a particular behavior are more numerous (i.e., there is more redundancy), it might get easier to find them with heuristics, but since they are more numerous, they potentially represent competing explanations, which leads to the issue of identifiability. This redundancy would be important to diagnose and characterize, an issue that our Circuit Robustness problem touches on. Appendix B Local and Global Sufficient Circuit Minimum locally sufficient circuit (MLSC) Input: A multi-layer perceptron M of depth cā¢dgsubscriptcd_gc ditalic_g with #ā¢ntā¢oā¢t,g#subscript\#n_tot,g# nitalic_t o t , g neurons and maximum layer width cā¢wgsubscriptcw_gc witalic_g, connection-value matrices W1,W2,ā¦,Wcā¢dgsubscript1subscript2ā¦subscriptsubscriptW_1,W_2,ā¦,W_cd_gW1 , W2 , ā¦ , Witalic_c d start_POSTSUBSCRIPT g end_POSTSUBSCRIPT, neuron bias vector B, a Boolean input vector I of length #ā¢ng,iā¢n#subscript\#n_g,in# nitalic_g , i n, and integers d, w, and #ā¢n#\#n# n such that 1ā¤dā¤cā¢dg1subscript1ā¤ dā¤ cd_g1 ā¤ d ā¤ c ditalic_g, 1ā¤wā¤cā¢wg1subscript1ā¤ wā¤ cw_g1 ā¤ w ā¤ c witalic_g, and 1ā¤#ā¢nā¤#ā¢ntā¢oā¢t,g1##subscript1ā¤\#nā¤\#n_tot,g1 ā¤ # n ā¤ # nitalic_t o t , g. Question: Is there a subcircuit C of M of depth cā¢drā¤dsubscriptcd_rā¤ dc ditalic_r ā¤ d with #ā¢ntā¢oā¢t,rā¤#ā¢n#subscript#\#n_tot,rā¤\#n# nitalic_t o t , r ā¤ # n neurons and maximum layer width cā¢wrā¤wsubscriptcw_rā¤ wc witalic_r ā¤ w that produces the same output on input I as M? Minimum globally sufficient circuit (MGSC) Input: A multi-layer perceptron M of depth cā¢dgsubscriptcd_gc ditalic_g with #ā¢ntā¢oā¢t,g#subscript\#n_tot,g# nitalic_t o t , g neurons and maximum layer width cā¢wgsubscriptcw_gc witalic_g, connection-value matrices W1,W2,ā¦,Wcā¢dgsubscript1subscript2ā¦subscriptsubscriptW_1,W_2,ā¦,W_cd_gW1 , W2 , ā¦ , Witalic_c d start_POSTSUBSCRIPT g end_POSTSUBSCRIPT, neuron bias vector B, and integers d, w, and #ā¢n#\#n# n such that 1ā¤dā¤cā¢dg1subscript1ā¤ dā¤ cd_g1 ā¤ d ā¤ c ditalic_g, 1ā¤wā¤cā¢wg1subscript1ā¤ wā¤ cw_g1 ā¤ w ā¤ c witalic_g, and 1ā¤#ā¢nā¤#ā¢ntā¢oā¢t,g1##subscript1ā¤\#nā¤\#n_tot,g1 ā¤ # n ā¤ # nitalic_t o t , g. Question: Is there a subcircuit C of M of depth cā¢drā¤dsubscriptcd_rā¤ dc ditalic_r ā¤ d with #ā¢ntā¢oā¢t,rā¤#ā¢n#subscript#\#n_tot,rā¤\#n# nitalic_t o t , r ā¤ # n neurons and maximum layer width cā¢wrā¤wsubscriptcw_rā¤ wc witalic_r ā¤ w that produces the same output as M on every possible Boolean input vector of length #iā¢n,gsubscript#\#_in,g#i n , g? Given a subset x of the neurons in M, the subcircuit C of M based on x has the neurons in x and all connections in M among these neurons. Note that in order for the output of C to be equal to the output of M on input I, the numbers #ā¢niā¢n,g#subscript\#n_in,g# nitalic_i n , g and #ā¢noā¢uā¢t,g#subscript\#n_out,g# nitalic_o u t , g of input and output neurons in M must exactly equal the numbers #ā¢niā¢n,r#subscript\#n_in,r# nitalic_i n , r and #ā¢noā¢uā¢t,r#subscript\#n_out,r# nitalic_o u t , r of input and output neurons in C; hence, no input or output neurons can be deleted from M in creating C. Following BarcelĆ³ et al. 2020, page 4, all neurons in M use the ReLU activation function and the output x of each output neuron is stepped as necessary to be Boolean, i.e, sā¢tā¢eā¢pā¢(x)=00step(x)=0s t e p ( x ) = 0 if xā¤00xā¤ 0x ā¤ 0 and is 1111 otherwise. For a graph G=(V,E)G=(V,E)G = ( V , E ), we shall assume an ordering on the vertices and edges in V and E, respectively. For each vertex vāVvā Vv ā V, let the complete neighbourhood NCā¢(v)subscriptN_C(v)Nitalic_C ( v ) of v be the set composed of v and the set of all vertices in G that are adjacent to v by a single edge, i.e., vāŖu|uāVā¢andā¢(u,v)āEconditional-setanduvEvāŖ\u~|~u~ā V~ and~(u,v)ā E\v āŖ u | u ā V and ( u , v ) ā E . Finally, let VCB be the version of VC in which each vertex in G has degree at most B. We will prove various classical and parameterized results for MLSC and MGSC using reductions from Clique (Theorem 1 and 7). These reductions are summarized in Figure 2 and the parameterized results are proved relative to the parameters in Table 5. Additional reductions from VC and DS (Theorems 5 and 103) use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Figure 2: This figure summarizes the reduction from Clique. Table 5: Parameters for the minimum sufficient subcircuit and reason problems. Parameter Description Problem cā¢dgsubscriptcd_gc ditalic_g # layers in given MLP All cā¢wgsubscriptcw_gc witalic_g max # neurons in layer in given MLP All #ā¢ntā¢oā¢t,g#subscript\#n_tot,g# nitalic_t o t , g total # neurons in given MLP All #ā¢niā¢n,g#subscript\#n_in,g# nitalic_i n , g # input neurons in given MLP All #ā¢noā¢uā¢t,g#subscript\#n_out,g# nitalic_o u t , g # output neurons in given MLP All Bmax,gsubscriptB_ ,gBroman_max , g max neuron bias in given MLP All Wmax,gsubscriptW_ ,gWroman_max , g max connection weight in given MLP All cā¢drsubscriptcd_rc ditalic_r # layers in requested subcircuit ML,GSC cā¢wrsubscriptcw_rc witalic_r max # neurons in layer in requested subcircuit ML,GSC #ā¢ntā¢oā¢t,r#subscript\#n_tot,r# nitalic_t o t , r total # neurons in requested subcircuit ML,GSC #ā¢niā¢n,r#subscript\#n_in,r# nitalic_i n , r # input neurons in requested subcircuit ML,GSC #ā¢noā¢uā¢t,r#subscript\#n_out,r# nitalic_o u t , r # output neurons in requested subcircuit ML,GSC Bmax,rsubscriptB_ ,rBroman_max , r max neuron bias in requested subcircuit ML,GSC Wmax,rsubscriptW_ ,rWroman_max , r max connection weight in requested subcircuit ML,GSC k Size of requested subset of input vector MSR B.1 Results for MLSC Towards proving NP-completeness, we first prove membership and then follow up with hardness. Membership of MLSC in NP can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]:ā¢()=ā³ā¢():delimited-[]ā³ā[C ]:C(x)=M(% x)ā [ C ā M ] : C ( x ) = M ( x ) Theorem 1. If MLSC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from Clique to MLSC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of Clique, construct the following instance āØM,I,d,w,#ā¢nā©# M,I,d,w,\#n āØ M , I , d , w , # n ā© of MLSC: Let M be an MLP based on #ā¢ntā¢oā¢t,g=|V|+|E|+2#subscript2\#n_tot,g=|V|+|E|+2# nitalic_t o t , g = | V | + | E | + 2 neurons spread across four layers: 1. Input layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias 0). 2. Hidden vertex layer: The vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V | (all with bias 0). 3. Hidden edge layer: The edge neurons nā¢e1,nā¢e2,ā¦ā¢nā¢e|E|subscript1subscript2ā¦subscriptne_1,ne_2,ā¦ ne_|E|n e1 , n e2 , ā¦ n e| E | (all with bias ā11-1- 1). 4. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t (bias ā(kā¢(kā1)/2ā1)121-(k(k-1)/2-1)- ( k ( k - 1 ) / 2 - 1 )). The non-zero weight connections between adjacent layers are as follows: ā¢ The input neuron niā¢nsubscriptn_innitalic_i n is connected to each vertex neuron with weight 1. ā¢ Each vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge neuron nā¢eisubscriptne_in eitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(1)1I=(1)I = ( 1 ), d=44d=4d = 4, w=kā¢(kā1)/212w=k(k-1)/2w = k ( k - 1 ) / 2, and #ā¢n=kā¢(kā1)/2+k+2#122\#n=k(k-1)/2+k+2# n = k ( k - 1 ) / 2 + k + 2. Observe that this instance of MLSC can be created in time polynomial in the size of the given instance of Clique. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢v1ā¢(1),nā¢v2ā¢(1),ā¦ā¢nā¢v|V|ā¢(1)subscript11subscript21ā¦subscript1nv_1(1),nv_2(1),ā¦ nv_|V|(1)n v1 ( 1 ) , n v2 ( 1 ) , ā¦ n v| V | ( 1 ) 3 nā¢e1ā¢(1),nā¢e2ā¢(1),ā¦ā¢nā¢e|E|ā¢(1)subscript11subscript21ā¦subscript1ne_1(1),ne_2(1),ā¦ ne_|E|(1)n e1 ( 1 ) , n e2 ( 1 ) , ā¦ n e| E | ( 1 ) 4 noā¢uā¢tā¢(|E|ā(kā¢(kā1)/2ā1))subscript121n_out(|E|-(k(k-1)/2-1))nitalic_o u t ( | E | - ( k ( k - 1 ) / 2 - 1 ) ) Note that it is the stepped output of noā¢uā¢tsubscriptn_outnitalic_o u t in timestep 4 that yields output 1. We now need to show the correctness of this reduction by proving that the answer for the given instance of Clique is āYesā if and only if the answer for the constructed instance of MLSC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a clique in G of size k. Consider the subcircuit C based on neurons niā¢nsubscriptn_innitalic_i n, noā¢uā¢tsubscriptn_outnitalic_o u t, nā¢vā²|vā²āVā²conditional-setsuperscriptā²superscriptā²\nv ~|~v ā V \ n vā² | vā² ā Vā² , and nā¢eā²|eā²=(x,y)ā¢andā¢vā¢x,vā¢yāVā²conditional-setsuperscriptā²formulae-sequencesuperscriptā²andsuperscriptā²\ne ~|~e =(x,y)~ and~vx,vyā V \ n eā² | eā² = ( x , y ) and v x , v y ā Vā² . Observe that in this subcircuit, cā¢dr=d=4subscript4cd_r=d=4c ditalic_r = d = 4, cā¢wr=r=kā¢(kā1)/2subscript12cw_r=r=k(k-1)/2c witalic_r = r = k ( k - 1 ) / 2, and #ā¢ntā¢oā¢t,r=#ā¢n=kā¢(kā1)/2+k+2#subscript#122\#n_tot,r=\#n=k(k-1)/2+k+2# nitalic_t o t , r = # n = k ( k - 1 ) / 2 + k + 2. The output behaviour of the neurons in C from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢v1ā²ā¢(1),nā¢v2ā²ā¢(1),ā¦ā¢nā¢v|Vā²|ā²ā¢(1)subscriptsuperscriptā²11subscriptsuperscriptā²21ā¦subscriptsuperscriptā²1nv _1(1),nv _2(1),ā¦ nv _|V |(1)n vā²1 ( 1 ) , n vā²2 ( 1 ) , ā¦ n vā²| Vā² | ( 1 ) 3 nā¢e1ā²ā¢(1),nā¢e2ā²ā¢(1),ā¦ā¢nā¢ekā¢(kā1)/2ā²ā¢(1)subscriptsuperscriptā²11subscriptsuperscriptā²21ā¦subscriptsuperscriptā²121ne _1(1),ne _2(1),ā¦ ne _k(k-1)/2(1)n eā²1 ( 1 ) , n eā²2 ( 1 ) , ā¦ n eā²italic_k ( k - 1 ) / 2 ( 1 ) 4 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) The output of noā¢uā¢tsubscriptn_outnitalic_o u t in timestep 4 is stepped to 1, which means that C is behaviorally equivalent to M on I. ā ā : Let C be a subcircuit of M that is behaviorally equivalent to M on input I and has #ā¢ntā¢oā¢t,rā¤#ā¢n=kā¢(kā1)/2+k+2#subscript#122\#n_tot,rā¤\#n=k(k-1)/2+k+2# nitalic_t o t , r ā¤ # n = k ( k - 1 ) / 2 + k + 2 neurons. As neurons in all four layers in M must be present in C to produce the required output, cā¢dr=d=cā¢dgsubscriptsubscriptcd_r=d=cd_gc ditalic_r = d = c ditalic_g and both niā¢nsubscriptn_innitalic_i n and noā¢uā¢tsubscriptn_outnitalic_o u t are in C. In order for noā¢uā¢tsubscriptn_outnitalic_o u t to produce a non-zero output, there must be at least kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons in C, each of which must be activated by the inclusion of the vertex neurons corresponding to both of their endpoint vertices. This requires the inclusion of at least k vertex neurons in C, as a set Vā²V Vā² ā² of vertices in graph can have at most |Vā²|ā¢(|Vā²|ā1)/2superscriptā²12|V |(|V |-1)/2| Vā² ā² | ( | Vā² ā² | - 1 ) / 2 distinct edges between them (with this maximum occurring if all pairs of vertices in Vā²V Vā² ā² have an edge between them). As #ā¢ntā¢oā¢t,rā¤kā¢(kā1)/2+k+2#subscript122\#n_tot,rā¤ k(k-1)/2+k+2# nitalic_t o t , r ā¤ k ( k - 1 ) / 2 + k + 2, all of the above implies that there must be exactly kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons and exactly k vertex neurons in C and the vertices in G corresponding to these vertex neurons must form a clique of size k in G. As Clique is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLSC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 2. If āØcdg,#niā¢n,g,#noā¢uā¢t,g,Bmax,g,Wmax,g,cdr,cwr,#niā¢n,r,#noā¢uā¢t,r,#ntā¢oā¢t,r, cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_% in,r,\#n_out,r,\#n_tot,r,āØ c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Bmax,r,Wmax,rā©B_ ,r,W_ ,r _max , r , Wroman_max , r ā©-MLSC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MLSC constructed in the reduction in the proof of Theorem 1, cā¢dg=cā¢dr=4subscriptsubscript4cd_g=cd_r=4c ditalic_g = c ditalic_r = 4, #ā¢niā¢n,g=#ā¢niā¢n,r=#ā¢noā¢uā¢t,r=#ā¢noā¢uā¢t,r=Wmax,g=Wmax,r=1#subscript#subscript#subscript#subscriptsubscriptsubscript1\#n_in,g=\#n_in,r=\#n_out,r=\#n_out,r=W_ ,g=W_ ,r=1# nitalic_i n , g = # nitalic_i n , r = # nitalic_o u t , r = # nitalic_o u t , r = Wroman_max , g = Wroman_max , r = 1, and Bmax,g,Bmax,r,#ā¢ntā¢oā¢t,r,subscriptsubscript#subscriptB_ ,g,B_ ,r,\#n_tot,r,Broman_max , g , Broman_max , r , # nitalic_t o t , r , and cā¢wrsubscriptcw_rc witalic_r are all functions of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 3. āØ#ā¢ntā¢oā¢t,gā©delimited-āØā©#subscript \#n_tot,g āØ # nitalic_t o t , g ā©-MLSC is fixed-parameter tractable. Proof. Consider the algorithm that generates each possible subcircuit of M and checks if that subcircuit is behaviorally equivalent to M on input I. If such a subcircuit is found, return āYesā; otherwise, return āNoā. As each such subcircuit can be run on I in time polynomial in the size of the given instance of MLSC and the total number of subcircuits that need to be checked is at most 2#ā¢ntā¢oā¢t,gsuperscript2#subscript2^\#n_tot,g2# nitalic_t o t , g, the above is a fixed-parameter tractable algorithm for MLSC relative to parameter-set #ā¢ntā¢oā¢t,g#subscript\\#n_tot,g\ # nitalic_t o t , g . ā Theorem 4. āØcā¢dg,cā¢wgā©subscriptsubscript cd_g,cw_g āØ c ditalic_g , c witalic_g ā©-MLSC is fixed-parameter tractable. Proof. Follows from the observation that #ā¢ntā¢oā¢t,gā¤cā¢dgĆcā¢wg#subscriptsubscriptsubscript\#n_tot,gā¤ cd_gĆ cw_g# nitalic_t o t , g ā¤ c ditalic_g Ć c witalic_g and the algorithm in the proof of Theorem 3. ā Though we have already proved the polynomial-time intractability of MLSC in Theorem 1, the reduction in the proof of the following theorem will be useful in proving a certain type of polynomial-time inapproximability for MLSC (see Figure 3). Theorem 5. If MLSC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from VC to MLSC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of VC, construct the following instance āØM,I,d,w,#ā¢nā©# M,I,d,w,\#n āØ M , I , d , w , # n ā© of MLSC: Let M be an MLP based on #ā¢ntā¢oā¢t,g=|V|+2ā¢|E|+2#subscript22\#n_tot,g=|V|+2|E|+2# nitalic_t o t , g = | V | + 2 | E | + 2 neurons spread across five layers: 1. Input layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias 0). 2. Hidden vertex layer: The vertex neurons nā¢vā¢N1,nā¢vā¢N2,ā¦ā¢nā¢vā¢N|V|subscript1subscript2ā¦subscriptnvN_1,nvN_2,ā¦ nvN_|V|n v N1 , n v N2 , ā¦ n v N| V |, all of which are NOT ReLU gates. 3. Hidden edge layer I: The edge AND neurons nā¢eā¢A1,nā¢eā¢A2,ā¦ā¢nā¢eā¢A|E|subscript1subscript2ā¦subscriptneA_1,neA_2,ā¦ neA_|E|n e A1 , n e A2 , ā¦ n e A| E |, all of which are 2-way AND ReLU gates. 4. Hidden edge layer I: The edge NOT neurons nā¢eā¢N1,nā¢eā¢N2,ā¦ā¢nā¢eā¢N|E|subscript1subscript2ā¦subscriptneN_1,neN_2,ā¦ neN_|E|n e N1 , n e N2 , ā¦ n e N| E |, all of which are NOT ReLU gates. 5. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is an |E||E|| E |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ The input neuron niā¢nsubscriptn_innitalic_i n is connected to each vertex NOT neuron with weight 1. ā¢ Each vertex NOT neuron nā¢vā¢NisubscriptnvN_in v Nitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge AND neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge AND neuron nā¢eā¢AisubscriptneA_in e Aitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to its corresponding edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i with weight 1. ā¢ Each edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(1)1I=(1)I = ( 1 ), d=55d=5d = 5, w=|E|w=|E|w = | E |, and #ā¢n=2ā¢|E|+k+2#22\#n=2|E|+k+2# n = 2 | E | + k + 2. Observe that this instance of MLSC can be created in time polynomial in the size of the given instance of VC, Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢vā¢N1ā¢(0),nā¢vā¢N2ā¢(0),ā¦ā¢nā¢vā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0nvN_1(0),nvN_2(0),ā¦ nvN_|V|(0)n v N1 ( 0 ) , n v N2 ( 0 ) , ā¦ n v N| V | ( 0 ) 3 nā¢eā¢A1ā¢(0),nā¢eā¢A2ā¢(0),ā¦ā¢nā¢eā¢A|E|ā¢(0)subscript10subscript20ā¦subscript0neA_1(0),neA_2(0),ā¦ neA_|E|(0)n e A1 ( 0 ) , n e A2 ( 0 ) , ā¦ n e A| E | ( 0 ) 4 nā¢eā¢N1ā¢(1),nā¢eā¢N2ā¢(1),ā¦ā¢nā¢eā¢N|E|ā¢(1)subscript11subscript21ā¦subscript1neN_1(1),neN_2(1),ā¦ neN_|E|(1)n e N1 ( 1 ) , n e N2 ( 1 ) , ā¦ n e N| E | ( 1 ) 5 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of VC is āYesā if and only if the answer for the constructed instance of MLSC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a vertex cover in G of size k. Consider the subcircuit C based on neurons niā¢nsubscriptn_innitalic_i n, noā¢uā¢tsubscriptn_outnitalic_o u t, nā¢vā²ā¢N|vā²āVā²conditional-setsuperscriptā²superscriptā²\nv N~|~v ā V \ n vā² N | vā² ā Vā² , nā¢eā¢A1,nā¢eā¢A2,ā¦,nā¢eā¢A|E|subscript1subscript2ā¦subscript\neA_1,neA_2,ā¦,neA_|E|\ n e A1 , n e A2 , ā¦ , n e A| E | , and nā¢eā¢N1,nā¢eā¢N2,ā¦,nā¢eā¢N|E|subscript1subscript2ā¦subscript\neN_1,neN_2,ā¦,neN_|E|\ n e N1 , n e N2 , ā¦ , n e N| E | . Observe that in this subcircuit, cā¢dr=d=5subscript5cd_r=d=5c ditalic_r = d = 5, cā¢wr=w=|E|subscriptcw_r=w=|E|c witalic_r = w = | E |, and #ā¢ntā¢oā¢t,r=#ā¢n=2ā¢|E|+k+2#subscript#22\#n_tot,r=\#n=2|E|+k+2# nitalic_t o t , r = # n = 2 | E | + k + 2. The output behaviour of the neurons in C from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢vā¢N1ā²ā¢(0),nā¢vā¢N2ā²ā¢(0),ā¦ā¢nā¢vā¢N|Vā²|ā²ā¢(0)subscriptsuperscriptā²10subscriptsuperscriptā²20ā¦subscriptsuperscriptā²0nvN _1(0),nvN _2(0),ā¦ nvN _|V |(0)n v Nā²1 ( 0 ) , n v Nā²2 ( 0 ) , ā¦ n v Nā²| Vā² | ( 0 ) 3 nā¢eā¢A1ā¢(0),nā¢eā¢A2ā¢(0),ā¦ā¢nā¢eā¢A|E|ā¢(0)subscript10subscript20ā¦subscript0neA_1(0),neA_2(0),ā¦ neA_|E|(0)n e A1 ( 0 ) , n e A2 ( 0 ) , ā¦ n e A| E | ( 0 ) 4 nā¢eā¢N1ā¢(1),nā¢eā¢N2ā¢(1),ā¦ā¢nā¢eā¢N|E|ā¢(1)subscript11subscript21ā¦subscript1neN_1(1),neN_2(1),ā¦ neN_|E|(1)n e N1 ( 1 ) , n e N2 ( 1 ) , ā¦ n e N| E | ( 1 ) 5 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) This means that C is behaviorally equivalent to M on I. ā ā : Let C be a subcircuit of M that is behaviorally equivalent to M on input I and has #ā¢ntā¢oā¢t,rā¤#ā¢n=2ā¢|E|+k+2#subscript#22\#n_tot,rā¤\#n=2|E|+k+2# nitalic_t o t , r ā¤ # n = 2 | E | + k + 2 neurons. As neurons in all five layers in M must be present in C to produce the required output, cā¢dr=cā¢dgsubscriptsubscriptcd_r=cd_gc ditalic_r = c ditalic_g and both niā¢nsubscriptn_innitalic_i n and noā¢uā¢tsubscriptn_outnitalic_o u t are in C. In order for noā¢uā¢tsubscriptn_outnitalic_o u t to produce a non-zero output, there must be at least |E||E|| E | edge NOT neurons and |E||E|| E | AND neurons in C, and each of the latter must be connected to at least one of the vertex NOT neurons corresponding to their endpoint vertices. As #ā¢ntā¢oā¢t,rā¤2ā¢|E|+k+2#subscript22\#n_tot,rā¤ 2|E|+k+2# nitalic_t o t , r ā¤ 2 | E | + k + 2, there must be exactly |E||E|| E | edge NOT neurons, |E||E|| E | edge AND neurons, and k vertex NOT neurons in C and the vertices in G corresponding to these vertex NOT neurons must form a vertex cover of size k in G. As VC is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLSC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Figure 3: This figure summarizes the reduction from Vertex Cover We now define our two notions of polynomial-time approximation. For a minimization problem Ī Ī , let Oā¢Pā¢TĪ ā¢(I)subscriptĪ OPT_ (I)O P Troman_Ī ( I ) be an optimal solution for Ī Ī , AĪ ā¢(I)subscriptĪ A_ (I)Aroman_Ī ( I ) be a solution forĪ Ī returned by an algorithm A, and mā¢(Oā¢Pā¢TĪ ā¢(I))subscriptĪ m(OPT_ (I))m ( O P Troman_Ī ( I ) ) and mā¢(AĪ ā¢(I))subscriptĪ m(A_ (I))m ( Aroman_Ī ( I ) ) be the values of these solutions. Consider the following alternative to approximation algorithms that give solutions that are within an additive factor of optimal. Definition 10. (Ausiello et al., 1999, Definition 3.5) Given a minimization problem Ī Ī , an algorithm A is a (multiplicative) Ļµitalic-ĻµĪµĻµ-approximation algorithm for Ī Ī if for each instance I of Ī Ī , mā¢(AĪ ā¢(I))āmā¢(Oā¢Pā¢TĪ ā¢(I))ā¤ĻµĆmā¢(Oā¢Pā¢TĪ ā¢(I))subscriptĪ subscriptĪ italic-ĻµsubscriptĪ m(A_ (I))-m(OPT_ (I))ā¤ĪµĆ m(OPT_ (I))m ( Aroman_Ī ( I ) ) - m ( O P Troman_Ī ( I ) ) ā¤ Ļµ Ć m ( O P Troman_Ī ( I ) ). It would be ideal if one could obtain approximate solutions for a problem Ī Ī that are arbitrarily close to optimal if one is willing to allow extra algorithm runtime. This is encoded in the following entity. Definition 11. (Adapted from Definition 3.10 in Ausiello et al. 1999) Given a minimization problem Ī Ī , a polynomial-time approximation scheme (PTAS) for Ī Ī is a set AA of algorithms such that for each integer k>00k>0k > 0, there is a 1k1 1kdivide start_ARG 1 end_ARG start_ARG k end_ARG-approximation algorithm AĪ kāsubscriptsuperscriptĪ A^k_ ā AAitalic_kroman_Ī ā A that runs in time polynomial in |I||I|| I |. The question of whether or not a problem has a PTAS can be answered using the following type of approximation-preserving reducibility. Definition 12. (Papadimitriou & Yannakakis, 1991, page 427) Given two minimization problems Ī Ī and Ī ā², Ī Ī L-reduces to Ī ā², i.e., Ī ā¤LĪ ā²subscriptĪ superscriptĪ ā² _L Ī ā¤L Ī ā² if there are polynomial-time algorithms f and g and constants Ī±,Ī²>00Ī±,Ī²>0Ī± , Ī² > 0 such that for each instance I of Ī Ī (L1) Algorithm f produces an instance Iā² of Ī ā² such that mā¢(Oā¢Pā¢TĪ ā²ā¢(Iā²))ā¤Ī±Ćmā¢(Oā¢Pā¢TĪ ā¢(I))subscriptsuperscriptĪ ā²subscriptĪ m(OPT_ (I ))ā¤Ī±Ć m(OPT_ (I))m ( O P Troman_Ī ā² ( Iā² ) ) ā¤ Ī± Ć m ( O P Troman_Ī ( I ) ); and (L2) For any solution for Iā² with value vā², algorithm g produces a solution for I of value v such that vāmā¢(Oā¢Pā¢TĪ ā¢(I))ā¤Ī²Ć(vā²āmā¢(Oā¢Pā¢TĪ ā²ā¢(Iā²)))subscriptĪ superscriptā²subscriptsuperscriptĪ ā²v-m(OPT_ (I))ā¤Ī²Ć(v -m(OPT_ (I )))v - m ( O P Troman_Ī ( I ) ) ā¤ Ī² Ć ( vā² - m ( O P Troman_Ī ā² ( Iā² ) ) ). Lemma 3. (Arora et al., 1998, Theorem 1.2.2) If an optimization problem that is Mā¢Aā¢Xā¢Sā¢Nā¢PMAX~SNPM A X S N P-hard under L-reductions has a PTAS then P=Nā¢P=NPP = N P. Theorem 6. If MLSC has a PTAS then P=Nā¢P=NPP = N P. Proof. We prove that the reduction from VC to MLSC in the proof of Theorem 5 is also an L-reduction from VCB to MLSC as follows: ā¢ Observe that mā¢(Oā¢Pā¢TVā¢CBā¢(I))ā„|E|/Bsubscriptsubscriptm(OPT_VC_B(I))ā„|E|/Bm ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) ā„ | E | / B (the best case in which G is a collection of B-star subgraphs such that each edge is uniquely covered by the central vertex of its associated star) and mā¢(Oā¢Pā¢TMā¢Lā¢Sā¢Cā¢(Iā²))ā¤2ā¢|E|+2ā¢|E|+2=4ā¢|E|+2subscriptsuperscriptā²22242m(OPT_MLSC(I ))ā¤ 2|E|+2|E|+2=4|E|+2m ( O P Titalic_M L S C ( Iā² ) ) ā¤ 2 | E | + 2 | E | + 2 = 4 | E | + 2 (the worst case in which the vertex neurons corresponding to the two endpoints of every edge in G are selected). This gives us mā¢(Oā¢Pā¢TMā¢Lā¢Sā¢Cā¢(Iā²))subscriptsuperscriptā² m(OPT_MLSC(I ))m ( O P Titalic_M L S C ( Iā² ) ) ā¤ ā¤ 4ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))+24subscriptsubscript2 4Bm(OPT_VC_B(I))+24 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) + 2 ā¤ ā¤ 4ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))+2ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))4subscriptsubscript2subscriptsubscript 4Bm(OPT_VC_B(I))+2Bm(OPT_VC_B(I))4 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) + 2 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) ā¤ ā¤ 6ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))6subscriptsubscript 6Bm(OPT_VC_B(I))6 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) which satisfies condition L1 with Ī±=6ā¢B6Ī±=6BĪ± = 6 B. ā¢ Observe that any solution Sā² for for the constructed instance Iā² of MLSC of value k+2ā¢|E|+222k+2|E|+2k + 2 | E | + 2 implies a solution S for the given instance I of VCB of size k, i.e., the vertices in V corresponding to the selected vertex neurons in S. Hence, it is the case that mā¢(S)āmā¢(Oā¢Pā¢TVā¢CBā¢(I))=mā¢(Sā²)āmā¢(Oā¢Pā¢TMā¢Lā¢Sā¢Cā¢(Iā²))subscriptsubscriptsuperscriptā²subscriptsuperscriptā²m(S)-m(OPT_VC_B(I))=m(S )-m(OPT_MLSC(I ))m ( S ) - m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) = m ( Sā² ) - m ( O P Titalic_M L S C ( Iā² ) ), which satisfies condition L2 with Ī²=11Ī²=1Ī² = 1. As VCB is Mā¢Aā¢Xā¢Sā¢Nā¢PMAX~SNPM A X S N P-hard under L-reductions (Papadimitriou & Yannakakis, 1991, Theorem 2(d)), the L-reduction above proves that MLSC is also Mā¢Aā¢Xā¢Sā¢Nā¢PMAX~SNPM A X S N P-hard under L-reductions. The result follows from Lemma 3. ā B.2 Results for MGSC Theorem 7. If MGSC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from Clique to MGSC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of Clique, construct an instance āØM,d,w,#ā¢nā©# M,d,w,\#n āØ M , d , w , # n ā© of MGSC as in the reduction in the proof of Theorem 1, omitting input vector I. Observe that this instance of MGSC can be created in time polynomial in the size of the given instance of Clique. As #ā¢niā¢n,g=1#subscript1\#n_in,g=1# nitalic_i n , g = 1, there are only two possible Boolean input vectors, (0)0(0)( 0 ) and (1)1(1)( 1 ). Given input vector (1)1(1)( 1 ), as MLP M in this reduction is the same as M in the proof of Theorem 1, the output in timestep 4 is once again 1; moreover, given input vector (0)0(0)( 0 ), no vertex or edge neurons can have output 1 and hence the output in timestep 4 is 0. We now need to show the correctness of this reduction by proving that the answer for the given instance of Clique is āYesā if and only if the answer for the constructed instance of MGSC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a clique in G of size k. Consider the subcircuit C based on neurons niā¢nsubscriptn_innitalic_i n, noā¢uā¢tsubscriptn_outnitalic_o u t, nā¢vā²|vā²āVā²conditional-setsuperscriptā²superscriptā²\nv ~|~v ā V \ n vā² | vā² ā Vā² , and nā¢eā²|eā²=(x,y)ā¢andā¢vā¢x,vā¢yāVā²conditional-setsuperscriptā²formulae-sequencesuperscriptā²andsuperscriptā²\ne ~|~e =(x,y)~ and~vx,vyā V \ n eā² | eā² = ( x , y ) and v x , v y ā Vā² . Observe that in this subcircuit, cā¢dr=4subscript4cd_r=4c ditalic_r = 4, cā¢wr=kā¢(kā1)/2subscript12cw_r=k(k-1)/2c witalic_r = k ( k - 1 ) / 2, and #ā¢ntā¢oā¢t,r=kā¢(kā1)/2+k+2#subscript122\#n_tot,r=k(k-1)/2+k+2# nitalic_t o t , r = k ( k - 1 ) / 2 + k + 2. Given input (1)1(1)( 1 ), the output behaviour of the neurons in C from the presentation of input until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢v1ā²ā¢(1),nā¢v2ā²ā¢(1),ā¦ā¢nā¢v|Vā²|ā²ā¢(1)subscriptsuperscriptā²11subscriptsuperscriptā²21ā¦subscriptsuperscriptā²1nv _1(1),nv _2(1),ā¦ nv _|V |(1)n vā²1 ( 1 ) , n vā²2 ( 1 ) , ā¦ n vā²| Vā² | ( 1 ) 3 nā¢e1ā²ā¢(1),nā¢e2ā²ā¢(1),ā¦ā¢nā¢ekā¢(kā1)/2ā²ā¢(1)subscriptsuperscriptā²11subscriptsuperscriptā²21ā¦subscriptsuperscriptā²121ne _1(1),ne _2(1),ā¦ ne _k(k-1)/2(1)n eā²1 ( 1 ) , n eā²2 ( 1 ) , ā¦ n eā²italic_k ( k - 1 ) / 2 ( 1 ) 4 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) Moreover, given input (0)0(0)( 0 ), no vertex or edge neurons in C can have output 1 and the output of C at timestep 4 is 0. This means that C is behaviorally equivalent to M on all possible Boolean input vectors ā ā : Let C be a subcircuit of M that is behaviorally equivalent to M on all possible Boolean input vectors and has #ā¢ntā¢oā¢t,rā¤#ā¢n=kā¢(kā1)/2+k+2#subscript#122\#n_tot,rā¤\#n=k(k-1)/2+k+2# nitalic_t o t , r ā¤ # n = k ( k - 1 ) / 2 + k + 2 neurons. Consider the case of input vector (1)1(1)( 1 ). This vector must cause C to generate output 1 at timestep 4 as C is behaviorally equivalent to M on all Boolean input vectors. As neurons in all four layers in M must be present in C to produce the required output, cā¢dr=cā¢dgsubscriptsubscriptcd_r=cd_gc ditalic_r = c ditalic_g and both niā¢nsubscriptn_innitalic_i n and noā¢uā¢tsubscriptn_outnitalic_o u t are in C. In order for noā¢uā¢tsubscriptn_outnitalic_o u t to produce a non-zero output, there must be at least kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons in C, each of which must be activated by the inclusion of the vertex neurons corresponding to both of their endpoint vertices. This requires the inclusion of at least k vertex neurons in C, as a set Vā²V Vā² ā² of vertices in graph can have at most |Vā²|ā¢(|Vā²|ā1)/2superscriptā²12|V |(|V |-1)/2| Vā² ā² | ( | Vā² ā² | - 1 ) / 2 distinct edges between them (with this maximum occurring if all pairs of vertices in Vā²V Vā² ā² have an edge between them). As #ā¢ntā¢oā¢t,rā¤kā¢(kā1)/2+k+2#subscript122\#n_tot,rā¤ k(k-1)/2+k+2# nitalic_t o t , r ā¤ k ( k - 1 ) / 2 + k + 2, all of the above implies that there must be exactly kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons and exactly k vertex neurons in C and the vertices in G corresponding to these vertex neurons must form a clique of size k in G. As Clique is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MGSC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 8. If āØcdg,#niā¢n,g,#noā¢uā¢t,g,Bmax,g,Wmax,g,cdr,cwr,#niā¢n,r,#noā¢uā¢t,r,#ntā¢oā¢t,r, cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_% in,r,\#n_out,r,\#n_tot,r,āØ c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Bmax,r,Wmax,rā©B_ ,r,W_ ,r _max , r , Wroman_max , r ā©-MGSC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MGSC constructed in the reduction in the proof of Theorem 7, cā¢dg=cā¢dr=4subscriptsubscript4cd_g=cd_r=4c ditalic_g = c ditalic_r = 4, #ā¢niā¢n,g=#ā¢niā¢n,r=#ā¢noā¢uā¢t,r=#ā¢noā¢uā¢t,r=Wmax,g=Wmax,r=1#subscript#subscript#subscript#subscriptsubscriptsubscript1\#n_in,g=\#n_in,r=\#n_out,r=\#n_out,r=W_ ,g=W_ ,r=1# nitalic_i n , g = # nitalic_i n , r = # nitalic_o u t , r = # nitalic_o u t , r = Wroman_max , g = Wroman_max , r = 1, and Bmax,g,Bmax,r,#ā¢ntā¢oā¢t,r,subscriptsubscript#subscriptB_ ,g,B_ ,r,\#n_tot,r,Broman_max , g , Broman_max , r , # nitalic_t o t , r , and cā¢wrsubscriptcw_rc witalic_r are all functions of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 9. āØ#ā¢ntā¢oā¢t,gā©delimited-āØā©#subscript \#n_tot,g āØ # nitalic_t o t , g ā©-MGSC is fixed-parameter tractable. Proof. Consider the algorithm that generates each possible subcircuit of M and checks if that subcircuit is behaviorally equivalent to M on all possible Boolean input vectors of length #ā¢niā¢n,g#subscript\#n_in,g# nitalic_i n , g. If such a subcircuit is found, return āYesā; otherwise, return āNoā. There are 2#ā¢niā¢n,gsuperscript2#subscript2^\#n_in,g2# nitalic_i n , g possible Boolean input vectors and the total number of subcircuits that need to be checked is at most 2#ā¢ntā¢oā¢t,gsuperscript2#subscript2^\#n_tot,g2# nitalic_t o t , g. As #ā¢niā¢n,gā¤#ā¢ntā¢oā¢t,g#subscript#subscript\#n_in,gā¤\#n_tot,g# nitalic_i n , g ā¤ # nitalic_t o t , g and each such subcircuit can be run on an input vector in time polynomial in the size of the given instance of MGSC, the above is a fixed-parameter tractable algorithm for MGSC relative to parameter-set #ā¢ntā¢oā¢t,g#subscript\\#n_tot,g\ # nitalic_t o t , g . ā Theorem 10. āØcā¢dg,cā¢wgā©subscriptsubscript cd_g,cw_g āØ c ditalic_g , c witalic_g ā©-MGSC is fixed-parameter tractable. Proof. Follows from the observation that #ā¢ntā¢oā¢t,gā¤cā¢dgĆcā¢wg#subscriptsubscriptsubscript\#n_tot,gā¤ cd_gĆ cw_g# nitalic_t o t , g ā¤ c ditalic_g Ć c witalic_g and the algorithm in the proof of Theorem 9. ā Let us now consider the PTAS-approximability of MGSC. A first thought would be to -reuse the reduction in the proof of Theorem 5 if the given MLP M and VC subcircuit are behaviorally equivalent under both possible input vectors, (1)1(1)( 1 ) and (0)0(0)( 0 ). We already know the former is true. With respect to the latter, observe that the output behaviour of the neurons in M from the presentation of input (0)0(0)( 0 ) until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(0)subscript0n_in(0)nitalic_i n ( 0 ) 2 nā¢vā¢N1ā¢(1),nā¢vā¢N2ā¢(1),ā¦ā¢nā¢vā¢N|V|ā¢(1)subscript11subscript21ā¦subscript1nvN_1(1),nvN_2(1),ā¦ nvN_|V|(1)n v N1 ( 1 ) , n v N2 ( 1 ) , ā¦ n v N| V | ( 1 ) 3 nā¢eā¢A1ā¢(1),nā¢eā¢A2ā¢(1),ā¦ā¢nā¢eā¢A|E|ā¢(1)subscript11subscript21ā¦subscript1neA_1(1),neA_2(1),ā¦ neA_|E|(1)n e A1 ( 1 ) , n e A2 ( 1 ) , ā¦ n e A| E | ( 1 ) 4 nā¢eā¢N1ā¢(0),nā¢eā¢N2ā¢(0),ā¦ā¢nā¢eā¢N|E|ā¢(0)subscript10subscript20ā¦subscript0neN_1(0),neN_2(0),ā¦ neN_|E|(0)n e N1 ( 0 ) , n e N2 ( 0 ) , ā¦ n e N| E | ( 0 ) 5 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) However, in the VC subcircuit, we are no longer guaranteed that both endpoint vertex NOT neurons for any edge AND neuron (let alone the endpoint vertex NOT neurons for all edge AND neurons) will be the vertex cover encoded in the subcircuit. This means that all edge AND neurons could potentially output 0, which would cause M to output 1 at timestep 4. This problem can be fixed if we can modify the given VC graph G to create a graph Gā² such that 1. we can guarantee that both endpoint vertex NOT neurons for at least one edge AND neuron are present in a VC subcircuit C constructed for Gā² (which would make at least one edge AND neuron output 1 and cause C to output 0 at timestep 4); and 2. we can easily extract a graph vertex cover of size at most k for G from any vertex cover of a particular size for Gā². Figure 4: The c-way Bowtie Graph BcsubscriptB_cBitalic_c. To do this, we shall use the c-way bowtie graph BcsubscriptB_cBitalic_c. For c>00c>0c > 0, BcsubscriptB_cBitalic_c consists of a central edge eBsubscripte_Beitalic_B between vertices vBā¢1subscript1v_B1vitalic_B 1 and vBā¢2subscript2v_B2vitalic_B 2 such that c edges radiate outwards from vBā¢1subscript1v_B1vitalic_B 1 and vBā¢2subscript2v_B2vitalic_B 2 to the c-sized vertex-sets VBā¢1=vBā¢1,1,vBā¢1,2,ā¦,vBā¢1,csubscript1subscript11subscript12ā¦subscript1V_B1=\v_B1,1,v_B1,2,ā¦,v_B1,c\Vitalic_B 1 = vitalic_B 1 , 1 , vitalic_B 1 , 2 , ā¦ , vitalic_B 1 , c and VBā¢2=vBā¢2,1,vBā¢2,2,ā¦,vBā¢2,csubscript2subscript21subscript22ā¦subscript2V_B2=\v_B2,1,v_B2,2,ā¦,v_B2,c\Vitalic_B 2 = vitalic_B 2 , 1 , vitalic_B 2 , 2 , ā¦ , vitalic_B 2 , c , respectively (see Figure 4). Note that such a graph has 2ā¢c+2222c+22 c + 2 vertices and 2ā¢c+1212c+12 c + 1 edges. Given a graph G=(V,E)G=(V,E)G = ( V , E ) with no isolated vertices such that |V|ā¤2ā¢|E|2|V|ā¤ 2|E|| V | ā¤ 2 | E | (with the minimum occurring in a graph consisting of |E||E|| E | endpoint-disjoint edges), let Bā¢oā¢wā¢(G)=B4ā¢|E|āŖGsubscript4Bow(G)=B_4|E|āŖ GB o w ( G ) = B4 | E | āŖ G. This graph has the following useful property. Lemma 4. Given a graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer kā¤|V|kā¤|V|k ā¤ | V |, if Bā¢oā¢wā¢(G)Bow(G)B o w ( G ) has a vertex cover Vā² of size at most k+22k+2k + 2 then (1) vBā¢1,vBā¢2āVā²subscript1subscript2superscriptā²\v_B1,v_B2\ā V vitalic_B 1 , vitalic_B 2 ā Vā² and (2) G has a vertex cover of size at most k. Proof. Let us prove the two consequent clauses as follows: 1. Any vertex cover Vā² of Bā¢oā¢wā¢(G)Bow(G)B o w ( G ) must cover all the edges in both B4ā¢|E|subscript4B_4|E|B4 | E | and G. Suppose vBā¢1,vBā¢2āVā²subscript1subscript2superscriptā²\v_B1,v_B2\ ā V vitalic_B 1 , vitalic_B 2 ā Vā². In order to cover the edges in B4ā¢|E|subscript4B_4|E|B4 | E |, all 8ā¢|E|88|E|8 | E | vertices in VBā¢1āŖVBā¢2subscript1subscript2V_B1āŖ V_B2Vitalic_B 1 āŖ Vitalic_B 2 must be in Vā². This is however impossible as |Vā²|ā¤k+2ā¤|V|+2ā¤2ā¢|E|+2ā¤4ā¢|Eā¢<8|ā¢E|superscriptā²22224bra8|V |ā¤ k+2ā¤|V|+2ā¤ 2|E|+2ā¤ 4|E<8|E|| Vā² | ā¤ k + 2 ā¤ | V | + 2 ā¤ 2 | E | + 2 ā¤ 4 | E < 8 | E |. Similarly, suppose only one of vBā¢1subscript1v_B1vitalic_B 1 and vBā¢2subscript2v_B2vitalic_B 2 is in Vā²; let us assume it is vBā¢1subscript1v_B1vitalic_B 1. In that case, all vertices in VBā¢2subscript2V_B2Vitalic_B 2 must be in Vā². However, this too is impossible as |Vā²āvBā¢1|ā¤k+1ā¤|V|+1ā¤2ā¢|E|+1ā¤3ā¢|E|<4ā¢|E|=|VBā¢2|superscriptā²subscript1112134subscript2|V -\v_B1\|ā¤ k+1ā¤|V|+1ā¤ 2|E|+1ā¤ 3|E|<4|E|=|V_B2|| Vā² - vitalic_B 1 | ā¤ k + 1 ā¤ | V | + 1 ā¤ 2 | E | + 1 ā¤ 3 | E | < 4 | E | = | Vitalic_B 2 |. Hence, both vBā¢1subscript1v_B1vitalic_B 1 and vBā¢2subscript2v_B2vitalic_B 2 must be in Vā². 2. Given (1), kā²ā¤ksuperscriptā²k ā¤ kā² ā¤ k vertices remain in Vā² to cover G. All kā² of these vertices need not be in G, e.g., some may be scattered over VBā¢1subscript1V_B1Vitalic_B 1 and VBā¢2subscript2V_B2Vitalic_B 2. That being said, it is still the case that G must have a vertex cover of size at most k. This concludes the proof. ā Theorem 11. If MGSC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from VC to MGSC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of VC, construct the following instance āØM,d,w,#ā¢nā©# M,d,w,\#n āØ M , d , w , # n ā© of MGSC based on Gā²=(Vā²,Eā²)=Bā¢oā¢wā¢(G)superscriptā²superscriptā²G =(V ,E )=Bow(G)Gā² = ( Vā² , Eā² ) = B o w ( G ): Let M be an MLP based on #ā¢ntā¢oā¢t,g=|Vā²|+2ā¢|Eā²|+2#subscriptsuperscriptā²2superscriptā²2\#n_tot,g=|V |+2|E |+2# nitalic_t o t , g = | Vā² | + 2 | Eā² | + 2 neurons spread across five layers: 1. Input layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias 0). 2. Hidden vertex layer: The vertex neurons nā¢vā¢N1,nā¢vā¢N2,ā¦ā¢nā¢vā¢N|Vā²|subscript1subscript2ā¦subscriptsuperscriptā²nvN_1,nvN_2,ā¦ nvN_|V |n v N1 , n v N2 , ā¦ n v N| Vā² |, all of which are NOT ReLU gates. 3. Hidden edge layer I: The edge AND neurons nā¢eā¢A1,nā¢eā¢A2,ā¦ā¢nā¢eā¢A|Eā²|subscript1subscript2ā¦subscriptsuperscriptā²neA_1,neA_2,ā¦ neA_|E |n e A1 , n e A2 , ā¦ n e A| Eā² |, all of which are 2-way AND ReLU gates. 4. Hidden edge layer I: The edge NOT neurons nā¢eā¢N1,nā¢eā¢N2,ā¦ā¢nā¢eā¢N|Eā²|subscript1subscript2ā¦subscriptsuperscriptā²neN_1,neN_2,ā¦ neN_|E |n e N1 , n e N2 , ā¦ n e N| Eā² |, all of which are NOT ReLU gates. 5. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is an |Eā²|superscriptā²|E || Eā² |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ The input neuron niā¢nsubscriptn_innitalic_i n is connected to each vertex NOT neuron with weight 1. ā¢ Each vertex NOT neuron nā¢vā¢NisubscriptnvN_in v Nitalic_i, 1ā¤iā¤|Vā²|1superscriptā²1ā¤ iā¤|V |1 ā¤ i ā¤ | Vā² |, is connected to each edge AND neuron whose corresponding edge has an endpoint viā²subscriptsuperscriptā²v _ivā²italic_i with weight 1. ā¢ Each edge AND neuron nā¢eā¢AisubscriptneA_in e Aitalic_i, 1ā¤iā¤|Eā²|1superscriptā²1ā¤ iā¤|E |1 ā¤ i ā¤ | Eā² |, is connected to its corresponding edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i with weight 1. ā¢ Each edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i, 1ā¤iā¤|Eā²|1superscriptā²1ā¤ iā¤|E |1 ā¤ i ā¤ | Eā² |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let d=55d=5d = 5, w=|Eā²|superscriptā²w=|E |w = | Eā² |, and #ā¢n=2ā¢|Eā²|+(k+2)+2=2ā¢|Eā²|=k+4#2superscriptā²222superscriptā²4\#n=2|E |+(k+2)+2=2|E |=k+4# n = 2 | Eā² | + ( k + 2 ) + 2 = 2 | Eā² | = k + 4. Observe that this instance of MGSC can be created in time polynomial in the size of the given instance of VC, the output behaviour of the neurons in M from the presentation of input (1)1(1)( 1 ) until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢vā¢N1ā¢(0),nā¢vā¢N2ā¢(0),ā¦ā¢nā¢vā¢N|Vā²|ā¢(0)subscript10subscript20ā¦subscriptsuperscriptā²0nvN_1(0),nvN_2(0),ā¦ nvN_|V |(0)n v N1 ( 0 ) , n v N2 ( 0 ) , ā¦ n v N| Vā² | ( 0 ) 3 nā¢eā¢A1ā¢(0),nā¢eā¢A2ā¢(0),ā¦ā¢nā¢eā¢A|Eā²|ā¢(0)subscript10subscript20ā¦subscriptsuperscriptā²0neA_1(0),neA_2(0),ā¦ neA_|E |(0)n e A1 ( 0 ) , n e A2 ( 0 ) , ā¦ n e A| Eā² | ( 0 ) 4 nā¢eā¢N1ā¢(1),nā¢eā¢N2ā¢(1),ā¦ā¢nā¢eā¢N|Eā²|ā¢(1)subscript11subscript21ā¦subscriptsuperscriptā²1neN_1(1),neN_2(1),ā¦ neN_|E |(1)n e N1 ( 1 ) , n e N2 ( 1 ) , ā¦ n e N| Eā² | ( 1 ) 5 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) and the output behaviour of the neurons in M from the presentation of input (0)0(0)( 0 ) until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(0)subscript0n_in(0)nitalic_i n ( 0 ) 2 nā¢vā¢N1ā¢(1),nā¢vā¢N2ā¢(1),ā¦ā¢nā¢vā¢N|Vā²|ā¢(1)subscript11subscript21ā¦subscriptsuperscriptā²1nvN_1(1),nvN_2(1),ā¦ nvN_|V |(1)n v N1 ( 1 ) , n v N2 ( 1 ) , ā¦ n v N| Vā² | ( 1 ) 3 nā¢eā¢A1ā¢(1),nā¢eā¢A2ā¢(1),ā¦ā¢nā¢eā¢A|Eā²|ā¢(1)subscript11subscript21ā¦subscriptsuperscriptā²1neA_1(1),neA_2(1),ā¦ neA_|E |(1)n e A1 ( 1 ) , n e A2 ( 1 ) , ā¦ n e A| Eā² | ( 1 ) 4 nā¢eā¢N1ā¢(0),nā¢eā¢N2ā¢(0),ā¦ā¢nā¢eā¢N|Eā²|ā¢(0)subscript10subscript20ā¦subscriptsuperscriptā²0neN_1(0),neN_2(0),ā¦ neN_|E |(0)n e N1 ( 0 ) , n e N2 ( 0 ) , ā¦ n e N| Eā² | ( 0 ) 5 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of VC is āYesā if and only if the answer for the constructed instance of MGSC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v % _k\ Vā² ā² = vā² ā²1 , vā² ā²2 , ā¦ , vā² ā²k ā V be a vertex cover in G of size k. Consider the subcircuit C based on neurons niā¢nsubscriptn_innitalic_i n, noā¢uā¢tsubscriptn_outnitalic_o u t, nā¢vā²ā¢N|vā²āVā²āŖnā¢vBā¢1ā¢N,nā¢vBā¢2ā¢Bconditional-setsuperscriptā²superscriptā²subscript1subscript2\nv N~|~v ā V \āŖ\nv_B1% N,nv_B2B\ n vā² ā² N | vā² ā² ā Vā² ā² āŖ n vitalic_B 1 N , n vitalic_B 2 B , nā¢eā¢A1,nā¢eā¢A2,ā¦,nā¢eā¢A|Eā²|subscript1subscript2ā¦subscriptsuperscriptā²\neA_1,neA_2,ā¦,neA_|E |\ n e A1 , n e A2 , ā¦ , n e A| Eā² | , and nā¢eā¢N1,nā¢eā¢N2,ā¦,nā¢eā¢N|Eā²|subscript1subscript2ā¦subscriptsuperscriptā²\neN_1,neN_2,ā¦,neN_|E |\ n e N1 , n e N2 , ā¦ , n e N| Eā² | . Observe that in this subcircuit, cā¢dr=d=5subscript5cd_r=d=5c ditalic_r = d = 5, cā¢wr=w=|Eā²|subscriptsuperscriptā²cw_r=w=|E |c witalic_r = w = | Eā² |, and #ā¢ntā¢oā¢t,r=#ā¢n=2ā¢|Eā²|+k+4#subscript#2superscriptā²4\#n_tot,r=\#n=2|E |+k+4# nitalic_t o t , r = # n = 2 | Eā² | + k + 4. The output behaviour of the neurons in C from the presentation of input (1)1(1)( 1 ) until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 All vertex NOT neurons (0) 3 nā¢eā¢A1ā¢(0),nā¢eā¢A2ā¢(0),ā¦ā¢nā¢eā¢A|Eā²|ā¢(0)subscript10subscript20ā¦subscriptsuperscriptā²0neA_1(0),neA_2(0),ā¦ neA_|E |(0)n e A1 ( 0 ) , n e A2 ( 0 ) , ā¦ n e A| Eā² | ( 0 ) 4 nā¢eā¢N1ā¢(1),nā¢eā¢N2ā¢(1),ā¦ā¢nā¢eā¢N|Eā²|ā¢(1)subscript11subscript21ā¦subscriptsuperscriptā²1neN_1(1),neN_2(1),ā¦ neN_|E |(1)n e N1 ( 1 ) , n e N2 ( 1 ) , ā¦ n e N| Eā² | ( 1 ) 5 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) Moreover, the output behaviour of the neurons in C from the presentation of input (0)0(0)( 0 ) until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(0)subscript0n_in(0)nitalic_i n ( 0 ) 2 All vertex NOT neurons (1) 3 At least one edge AND neuron has output 1, e.g., nā¢eBā¢Asubscriptne_BAn eitalic_B A 4 At least one edge NOT neuron has output 0, e.g., nā¢eBā¢Nsubscriptne_BNn eitalic_B N 5 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) This means that C is behaviorally equivalent to M on all possible Boolean input vectors. ā ā : Let C be a subcircuit of M that is behaviorally equivalent to M on all possible Boolean input vectors and has #ā¢ntā¢oā¢t,rā¤#ā¢n=2ā¢|Eā²|+k+4#subscript#2superscriptā²4\#n_tot,rā¤\#n=2|E |+k+4# nitalic_t o t , r ā¤ # n = 2 | Eā² | + k + 4 neurons. As neurons in all five layers in M must be present in C to produce the required output, cā¢dr=cā¢dgsubscriptsubscriptcd_r=cd_gc ditalic_r = c ditalic_g and both niā¢nsubscriptn_innitalic_i n and noā¢uā¢tsubscriptn_outnitalic_o u t are in C. In order for noā¢uā¢tsubscriptn_outnitalic_o u t to produce a non-zero output, there must be at least |Eā²|superscriptā²|E || Eā² | edge NOT neurons and |Eā²|superscriptā²|E || Eā² | AND neurons in C, and each of the latter must be connected to at least one of the vertex NOT neurons corresponding to their endpoint vertices. As #ā¢ntā¢oā¢t,rā¤2ā¢|Eā²|+k+4#subscript2superscriptā²4\#n_tot,rā¤ 2|E |+k+4# nitalic_t o t , r ā¤ 2 | Eā² | + k + 4, there must be exactly |E||E|| E | edge NOT neurons, |Eā²|superscriptā²|E || Eā² | edge AND neurons, and k+22k+2k + 2 vertex NOT neurons in C and the vertices in Gā² corresponding to these vertex NOT neurons must form a vertex cover Vā²V Vā² ā² of size k+22k+2k + 2 in Gā². However, as Gā²=Bā¢oā¢wā¢(G)superscriptā²G =Bow(G)Gā² = B o w ( G ), Lemma 4 implies not only that nā¢vBā¢1ā¢N,nā¢vBā¢2ā¢NāVā²subscript1subscript2superscriptā²\nv_B1N,nv_B2N\ā V n vitalic_B 1 N , n vitalic_B 2 N ā Vā² ā² but that G has a vertex cover of size at most k. As VC is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MGSC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 12. If MGSC has a PTAS then P=Nā¢P=NPP = N P. Proof. We prove that the reduction from VC to MLSC in the proof of Theorem 11 is also an L-reduction from VCB to MGSC as follows: ā¢ Observe that mā¢(Oā¢Pā¢TVā¢CBā¢(I))ā„|E|/Bsubscriptsubscriptm(OPT_VC_B(I))ā„|E|/Bm ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) ā„ | E | / B (the best case in which G is a collection of B-star subgraphs such that each edge is uniquely covered by the central vertex of its associated star) and mā¢(Oā¢Pā¢TMā¢Gā¢Sā¢Cā¢(Iā²))ā¤2ā¢|Eā²|+2ā¢|Eā²|+2=4ā¢|Eā²|+2subscriptsuperscriptā²2superscriptā²2superscriptā²24superscriptā²2m(OPT_MGSC(I ))ā¤ 2|E |+2|E |+2=4|E |+2m ( O P Titalic_M G S C ( Iā² ) ) ā¤ 2 | Eā² | + 2 | Eā² | + 2 = 4 | Eā² | + 2 (the worst case in which the vertex neurons corresponding to the two endpoints of every edge in G are selected). As |Eā²|=9ā¢|E|+1superscriptā²91|E |=9|E|+1| Eā² | = 9 | E | + 1, this gives us mā¢(Oā¢Pā¢TMā¢Lā¢Sā¢Cā¢(Iā²))subscriptsuperscriptā² m(OPT_MLSC(I ))m ( O P Titalic_M L S C ( Iā² ) ) ā¤ ā¤ 36ā¢|E|+4+23642 36|E|+4+236 | E | + 4 + 2 ā¤ ā¤ 36ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))+636subscriptsubscript6 36Bm(OPT_VC_B(I))+636 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) + 6 ā¤ ā¤ 36ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))+6ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))36subscriptsubscript6subscriptsubscript 36Bm(OPT_VC_B(I))+6Bm(OPT_VC_B(I))36 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) + 6 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) ā¤ ā¤ 42ā¢Bā¢mā¢(Oā¢Pā¢TVā¢CBā¢(I))42subscriptsubscript 42Bm(OPT_VC_B(I))42 B m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) which satisfies condition L1 with Ī±=42ā¢B42Ī±=42BĪ± = 42 B. ā¢ Observe that any solution Sā² for for the constructed instance Iā² of MGSC of value k+2ā¢|Eā²|+42superscriptā²4k+2|E |+4k + 2 | Eā² | + 4 implies a solution S for the given instance I of VCB of size k in S. Hence, it is the case that mā¢(S)āmā¢(Oā¢Pā¢TVā¢CBā¢(I))=mā¢(Sā²)āmā¢(Oā¢Pā¢TMā¢Gā¢Sā¢Cā¢(Iā²))subscriptsubscriptsuperscriptā²subscriptsuperscriptā²m(S)-m(OPT_VC_B(I))=m(S )-m(OPT_MGSC(I ))m ( S ) - m ( O P Titalic_V C start_POSTSUBSCRIPT B end_POSTSUBSCRIPT ( I ) ) = m ( Sā² ) - m ( O P Titalic_M G S C ( Iā² ) ), which satisfies condition L2 with Ī²=11Ī²=1Ī² = 1. As VCB is Mā¢Aā¢Xā¢Sā¢Nā¢PMAX~SNPM A X S N P-hard under L-reductions (Papadimitriou & Yannakakis, 1991, Theorem 2(d)), the L-reduction above proves that MGSC is also Mā¢Aā¢Xā¢Sā¢Nā¢PMAX~SNPM A X S N P-hard under L-reductions. The result follows from Lemma 3. ā Appendix C Sufficient Circuit Search and Counting Problems Definition 13. An entity x with property P is minimal if there is no non-empty subset of elements in x that can be deleted to create an entity xā² with property P. We shall assume here that all subcircuits are non-trivial, i.e., the subcircuit is of size <|M|absent<|M|< | M |. Consider the following search problem templates: Name local sufficient circuit (AccLSC) Input: A multi-layer perceptron M of depth cā¢dgsubscriptcd_gc ditalic_g with #ā¢ntā¢oā¢t,g#subscript\#n_tot,g# nitalic_t o t , g neurons and maximum layer width cā¢wgsubscriptcw_gc witalic_g, connection-value matrices W1,W2,ā¦,Wcā¢dgsubscript1subscript2ā¦subscriptsubscriptW_1,W_2,ā¦,W_cd_gW1 , W2 , ā¦ , Witalic_c d start_POSTSUBSCRIPT g end_POSTSUBSCRIPT, neuron bias vector B, a Boolean input vector I of length #ā¢ng,iā¢n#subscript\#n_g,in# nitalic_g , i n, integers d and w such that 1ā¤dā¤cā¢dg1subscript1ā¤ dā¤ cd_g1 ā¤ d ā¤ c ditalic_g and 1ā¤wā¤cā¢wg1subscript1ā¤ wā¤ cw_g1 ā¤ w ā¤ c witalic_g PrmAdd. Output: A CType subcircuit C of M of depth cā¢drā¤dsubscriptcd_rā¤ dc ditalic_r ā¤ d with maximum layer width cā¢wrā¤wsubscriptcw_rā¤ wc witalic_r ā¤ w that Cā¢(I)=Mā¢(I)C(I)=M(I)C ( I ) = M ( I ), if such a subcircuit exists, and special symbol ā„bottom ā„ otherwise. Name global sufficient circuit (AccGSC) Input: A multi-layer perceptron M of depth cā¢dgsubscriptcd_gc ditalic_g with #ā¢ntā¢oā¢t,g#subscript\#n_tot,g# nitalic_t o t , g neurons and maximum layer width cā¢wgsubscriptcw_gc witalic_g, connection-value matrices W1,W2,ā¦,Wcā¢dgsubscript1subscript2ā¦subscriptsubscriptW_1,W_2,ā¦,W_cd_gW1 , W2 , ā¦ , Witalic_c d start_POSTSUBSCRIPT g end_POSTSUBSCRIPT, neuron bias vector B, integers d and w such that 1ā¤dā¤cā¢dg1subscript1ā¤ dā¤ cd_g1 ā¤ d ā¤ c ditalic_g and 1ā¤wā¤cā¢wg1subscript1ā¤ wā¤ cw_g1 ā¤ w ā¤ c witalic_g PrmAdd. Output: A CType subcircuit C of M of depth cā¢drā¤dsubscriptcd_rā¤ dc ditalic_r ā¤ d with maximum layer width cā¢wrā¤wsubscriptcw_rā¤ wc witalic_r ā¤ w such that Cā¢(I)=Mā¢(I)C(I)=M(I)C ( I ) = M ( I ) for every possible Boolean input vector I of length #iā¢n,gsubscript#\#_in,g#i n , g, if such a subcircuit exists, and special symbol ā„bottom ā„ otherwise.. Name local necessary circuit (AccLNC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n PrmAdd. Output: A CType subcircuit C of M such that Cā©Cā²ā ā superscriptā²Cā© C ā ā© Cā² ā ā for every sufficient circuit Cā² of M relative to I, if such a s subcircuit exists, and special symbol ā„bottom ā„ otherwise. Name global necessary circuit (AccGNC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B PrmAdd. Output: A CType subcircuit C of M such that for for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, Cā©Cā²ā ā superscriptā²Cā© C ā ā© Cā² ā ā for every sufficient circuit Cā² of M relative to I, if such a subcircuit exists, and special symbol ā„bottom ā„ otherwise. Name local circuit ablation (AccLCA) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n PrmAdd. Output: A CType subcircuit C of M such that (M/C)ā¢(I)ā Mā¢(I)(M/C)(I)ā M(I)( M / C ) ( I ) ā M ( I ), if such a subcircuit exists, and special symbol ā„bottom ā„ otherwise. Name global circuit ablation (AccGCA) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B PrmAdd. Output: A CType subcircuit C of M such that (M/C)ā¢(I)ā Mā¢(I)(M/C)(I)ā M(I)( M / C ) ( I ) ā M ( I ) every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, if such a subcircuit exists, and special symbol ā„bottom ā„ otherwise. Each these templates can be filled out to create six search problem variants as follows: 1. Exact-size Problem: ā¢ Name === āExactā ā¢ Acc === āExā ā¢ PrmAdd === ā, and a positive integer k<|M|k<|M|k < | M |ā ā¢ CType === āsize-kā 2. Bounded-size Problem: ā¢ Name === āBoundedā ā¢ Acc === āBā ā¢ PrmAdd === ā, and a positive integer k<|M|k<|M|k < | M |ā ā¢ CType === āsize-ā¤kabsentā¤ kā¤ kā 3. Minimal Problem: ā¢ Name === āMinimalā ā¢ Acc === āMnlā ā¢ PrmAdd === āā ā¢ CType === āminimalā 4. Minimal Exact-size Problem: ā¢ Name === āMinimal exactā ā¢ Acc === āMnlExā ā¢ PrmAdd === ā, and a positive integer k<|M|k<|M|k < | M |ā ā¢ CType === āminimal size-kā 5. Minimal Bounded-size Problem: ā¢ Name === āMinimal boundedā ā¢ Acc === āMnlBā ā¢ PrmAdd === ā, and a positive integer k<|M|k<|M|k < | M |ā ā¢ CType === āminimal size-ā¤kabsentā¤ kā¤ kā 6. Minimum Problem: ā¢ Name === āMinimumā ā¢ Acc === āMinā ā¢ PrmAdd === āā ā¢ CType === āminimumā We will use previous results for the following problems to prove our results for the problems above. Exact clique (ExClique) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer kā¤|V|kā¤|V|k ā¤ | V |. Output: A k-size vertex subset Vā² of G that is a clique of size k in G, if such a Vā² exists, and special symbol ā„bottom ā„ otherwise. Exact vertex cover (ExVC) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ) and a positive integer kā¤|V|kā¤|V|k ā¤ | V |. Output: A k-size vertex subset Vā² of G that is a vertex cover of size k in G, if such a Vā² exists, and special symbol ā„bottom ā„ otherwise. Minimal vertex cover (MnlVC) (Valiant, 1979, Problem 4) Input: An undirected graph G=(V,E)G=(V,E)G = ( V , E ). Output: A minimal subset Vā² of the vertices in G that is a vertex cover of G. Given any search problem X above, let #X be the problem that returns the number of solution outputs. To assess the complexity of these counting problems, we will use the following definitions in Garey & Johnson 1979, Section 7.3, adapted from those originally given in Valiant 1979. Definition 14. (Garey & Johnson, 1979, p. 168) A counting problem #Ī Ī is in #P if there is a nondeterministic algorithm such that for each input I of Ī Ī , (1) the number of ādistinct āguessesā that lead to acceptance of I exactly equals the number solutions of Ī Ī for input I and (2) the length of the longest accepting computation is bounded by a polynomial in |I||I|| I |. #P contains very hard problems, as it is known every class in the Polynomial Hierarchy (which include P and Nā¢PNPN P as its lowest members) Turing reduces to #P, i.e., Pā¢HāP#ā¢Psuperscript#PH P^\#PP H ā P# P (Toda, 1991). We use the following type of reduction to isolate problems that are the hardest (#P-complete) and at least as hard as the hardest (#P-hard) in #P. Definition 15. (Garey & Johnson, 1979, p. 168-169) Given two search problems Ī Ī and Ī ā², a (polynomial time) parsimonious reduction from Ī Ī to Ī ā² is a function f:IĪ āIĪ ā²:āsubscriptĪ subscriptsuperscriptĪ ā²f:I_ ā I_ f : Iroman_Ī ā Iroman_Ī ā² that can be computed in polynomial time such that for every IāĪ Iā I ā Ī , the number of solutions of Ī Ī for input I is exactly equal to the number of Ī ā² for input fā¢(I)f(I)f ( I ). We will also derive parameterized counting results using the framework given in Flum & Grohe 2006, Chapter 14. The definition of class #Wā¢[1]delimited-[]1W[1]W [ 1 ] (Flum & Grohe, 2006, Definition 14.11) is rather intricate and need not concern us here. We will use the following type of reduction to isolate problems that are at least as hard as the hardest (#Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard) in #Wā¢[1]delimited-[]1W[1]W [ 1 ]. Definition 16. (Adapted from Flum & Grohe 2006, Definition 14.10.a) Given two parameterized search problems āØkā©delimited-āØā© k āØ k ā©-Ī Ī and āØKā©delimited-āØā© K āØ K ā©-Ī ā², a (fpt) parsimonious reduction from āØkā©delimited-āØā© k āØ k ā©-Ī Ī to āØKā©delimited-āØā© K āØ K ā©-Ī ā² is a function f:IĪ āIĪ ā²:āsubscriptĪ subscriptsuperscriptĪ ā²f:I_ ā I_ f : Iroman_Ī ā Iroman_Ī ā² computable in fixed-parameter time relative to parameter k such that for every IāĪ Iā I ā Ī (1) the number of solutions of Ī Ī for input I is exactly equal to the number of Ī ā² for input fā¢(I)f(I)f ( I ) and (2) for every parameter kā²āKsuperscriptā²k ā Kkā² ā K, kā²ā¤gkā²ā¢(k)superscriptā²subscriptsuperscriptā²k ā¤ g_k (k)kā² ā¤ gitalic_kā² ( k ) for some function gkā²ā¢()subscriptsuperscriptā²g_k ()gitalic_kā² ( ). Reductions are often established to be parsimonious by proving bijections between solution-sets for I and fā¢(I)f(I)f ( I ), i.e., each solution to I corresponds to exactly one solution for fā¢(I)f(I)f ( I ) and vice versa. We will prove various parameterized results for our problems using reductions from ExClique. The parameterized results are proved relative to the parameters in Table 6. Lemmas 1 and 2 will be useful in deriving additional parameterized results from proved ones. Table 6: Parameters for the sufficient circuit, necessary circuit, circuit ablation problems. Note that for sufficient circuit problems, there are two versions of each parameter k ā namely, those describing the given MLP M and the derived subcircuit C (distinguished by g- ands r-subscripts, respectively). Parameter Description cā¢dcdc d # layers in given MLP cā¢wcwc w max # neurons in layer in given MLP #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP BmaxsubscriptB_ Broman_max max neuron bias in given MLP WmaxsubscriptW_ Wroman_max max connection weight in given MLP k Size of requested neuron subset C.1 Results for Sufficient Circuit Problems Theorem 13. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B , ExClique polynomial-time parsimoniously reduces to Ī Ī . Proof. Consider first the local sufficient circuit problem variants. Observe that for the reduction from Clique to MLSC in the proof of Theorem 1 (1) the reduction is also from ExClique, (2) each clique of size k in the given instance of ExClique has exactly one corresponding sufficient circuit of size kā²=k+kā¢(kā1)/2+2superscriptā²122k =k+k(k-1)/2+2kā² = k + k ( k - 1 ) / 2 + 2 in the constructed instance of MLSC and vice versa, and (3) courtesy of the bias in neuron noā¢uā¢tsubscriptn_outnitalic_o u t and the structure of the MLP M in the constructed instance of MLSC, no sufficient circuit can have size <kā²absentsuperscriptā²<k < kā² and hence problem variants ExLSC, BLSC, MnlExLSC, and MnlBLSC (when kā²=k+kā¢(kā1)/2+2superscriptā²122k =k+k(k-1)/2+2kā² = k + k ( k - 1 ) / 2 + 2) have the same set of sufficient circuit solutions Hence, this reduction is also a polynomial-time parsimonious reduction from ExClique to each local sufficient circuit problem variant in L. As for the global sufficient circuit problem variants, it was pointed out that in the proof of Theorem 7 that the reduction above is also a reduction from Clique to MGSC; moreover, all three properties above also hold modulo MGSC, ExGSC, BGSC, MnlExGSC, and MnlBGSC. Hence, this reduction is also a polynomial-time parsimonious reduction from ExClique to each global sufficient circuit problem variant in L. ā Theorem 14. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B , āØkā©delimited-āØā© k āØ k ā©-ExClique fpt parsimoniously reduces to āØcdg,#niā¢n,g,#noā¢uā¢t,g,Bmax,g,Wmax,g,cdr,cwr, cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,āØ c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , #niā¢n,r,#noā¢uā¢t,r,#ntā¢oā¢t,r,Bmax,r,Wmax,rā©\#n_in,r,\#n_out,r,\#n_tot,r,B_ ,r,W_ ,r # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Broman_max , r , Wroman_max , r ā©-Ī Ī . Proof. Observe that in the instance of MLSC constructed in the reduction in the proof of Theorem 1, cā¢dg=cā¢dr=4subscriptsubscript4cd_g=cd_r=4c ditalic_g = c ditalic_r = 4, #ā¢niā¢n,g=#ā¢niā¢n,r=#ā¢noā¢uā¢t,r=#ā¢noā¢uā¢t,r=Wmax,g=Wmax,r=1#subscript#subscript#subscript#subscriptsubscriptsubscript1\#n_in,g=\#n_in,r=\#n_out,r=\#n_out,r=W_ ,g=W_ ,r=1# nitalic_i n , g = # nitalic_i n , r = # nitalic_o u t , r = # nitalic_o u t , r = Wroman_max , g = Wroman_max , r = 1, and Bmax,g,Bmax,r,#ā¢ntā¢oā¢t,r,subscriptsubscript#subscriptB_ ,g,B_ ,r,\#n_tot,r,Broman_max , g , Broman_max , r , # nitalic_t o t , r , and cā¢wrsubscriptcw_rc witalic_r are all functions of k in the given instance of Clique. The result then follows by the reasoning in the proof of Theorem 13. ā Theorem 15. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B , if Ī Ī is polynomial-time solvable then P=Nā¢P=NPP = N P. Proof. Suppose there is a polynomial-time algorithm A for some Ī āLĪ ā LĪ ā L. Let R be the polynomial-time algorithm underlying the polynomial-time parsimonious reduction from ExClique to Ī Ī specified in the proof of Theorem 13. Construct an algorithm Aā² for the decision version of ExClique as follows: Given an input I for ExCliqueD, create input Iā² for Ī Ī using R, and apply A to Iā² to create solution S. If S=ā„bottomS= = ā„, return āNoā; otherwise, return āYesā. Algorithm Aā² is a polynomial-time algorithm for ExCliqueD; however, as ExCliqueD is Nā¢PNPN P-complete (Garey & Johnson, 1979, Problem GT19), this implies that P=Nā¢P=NPP = N P, giving the result. ā Theorem 16. if MinLSC or MinGSC is polynomial-time solvable then P=Nā¢P=NPP = N P. Proof. Suppose there is a polynomial-time algorithm A for MinLSC (MinGSC). Let R be the polynomial-time algorithm underlying the polynomial-time parsimonious reduction from ExClique to BLSC (BGSC) specified in the proof of Theorem 13. Construct an algorithm Aā² for the decision version of ExClique as follows: Given an input I for ExCliqueD, create input Iā² for BSC )LSC) using R, and apply A to Iā² to create solution S. If |S|ā¤k|S|ā¤ k| S | ā¤ k, return āYesā; otherwise, return āNoā. Algorithm Aā² is a polynomial-time algorithm for ExCliqueD; however, as ExCliqueD is Nā¢PNPN P-complete (Garey & Johnson, 1979, Problem GT19), this implies that P=Nā¢P=NPP = N P, giving the result. ā Theorem 17. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B and K=cdg,#niā¢n,g,#noā¢uā¢t,g,Bmax,g,Wmax,g,cdr,cwr,#niā¢n,r,#noā¢uā¢t,r,#ntā¢oā¢t,r,Bmax,r,K=\cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_in,% r,\#n_out,r,\#n_tot,r,B_ ,r,K = c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Broman_max , r , Wmax,rW_ ,r\Wroman_max , r , if āØKā©delimited-āØā© K āØ K ā©-Ī Ī is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Suppose there is a fixed-parameter tractable algorithm A for āØKā©delimited-āØā© K āØ K ā©-Ī Ī for some Ī āLĪ ā LĪ ā L. Let R be the fixed-parameter algorithm underlying the fpt parsimonious reduction from āØkā©delimited-āØā© k āØ k ā©-ExClique to āØKā©delimited-āØā© K āØ K ā©-Ī Ī specified in the proof of Theorem 14. Construct an algorithm Aā² for the decision version of āØkā©delimited-āØā© k āØ k ā©-ExClique as follows: Given an input I for āØkā©delimited-āØā© k āØ k ā©-ExCliqueD, create input Iā² for Ī Ī using R, and apply A to Iā² to create solution S. If S=ā„bottomS= = ā„, return āNoā; otherwise, return āYesā. Algorithm Aā² is a fixed-parameter tractable algorithm for āØkā©delimited-āØā© k āØ k ā©-ExCliqueD; however, as āØkā©delimited-āØā© k āØ k ā©-ExCliqueD is Wā¢[1]delimited-[]1W[1]W [ 1 ]-complete (Downey & Fellows, 1999), this implies that Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ], giving the result. ā Theorem 18. For K=cdg,#niā¢n,g,#noā¢uā¢t,g,Bmax,g,Wmax,g,cdr,cwr,#niā¢n,r,#noā¢uā¢t,r,K=\cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_in,% r,\#n_out,r,K = c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , #ntā¢oā¢t,r,Bmax,r,Wmax,r\#n_tot,r,B_ ,r,W_ ,r\# nitalic_t o t , r , Broman_max , r , Wroman_max , r , if āØKā©delimited-āØā© K āØ K ā©-MinLSC or āØK]ā© K] āØ K ] ā©-MinGSC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Suppose there is a fixed-parameter tractable algorithm A for āØKā©delimited-āØā© K āØ K ā©-MinLSC (āØKā©delimited-āØā© K āØ K ā©-MinGSC). Let R be the fixed-parameter algorithm underlying the fpt parsimonious reduction from āØkā©delimited-āØā© k āØ k ā©-ExClique to āØKā©delimited-āØā© K āØ K ā©-BLSC (āØKā©delimited-āØā© K āØ K ā©-BGSC) specified in the proof of Theorem 14. Construct an algorithm Aā² for the decision version of āØkā©delimited-āØā© k āØ k ā©-ExClique as follows: Given an input I for āØkā©delimited-āØā© k āØ k ā©-ExCliqueD, create input Iā² for Ī Ī using R, and apply A to Iā² to create solution S. If |S|ā¤k|S|ā¤ k| S | ā¤ k, return āYesā; otherwise, return āNoā. Algorithm Aā² is a fixed-parameter tractable algorithm for āØkā©delimited-āØā© k āØ k ā©-ExCliqueD; however, as āØkā©delimited-āØā© k āØ k ā©-ExCliqueD is Wā¢[1]delimited-[]1W[1]W [ 1 ]-complete (Downey & Fellows, 1999), this implies that Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ], giving the result. ā Theorem 19. For Ī āL=ā¢Lā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C | V ā E x , B , M n l E x , M n l B , #Ī Ī is #P-complete. Proof. As #ExVC is #P-complete (Provan & Ball 1983, Page 781; see also Garey & Johnson 1979, Page 169), #ExClique is #P-hard by the polynomial-time parsimonious reduction from ExVC to ExClique implicit in Garey & Johnson 1979, Lemma 3.1. The #P-hardness of #Ī Ī then follows from the appropriate polynomial-time parsimonious reduction from ExClique to Ī Ī specified in the proof of Theorem 13. Membership of #Ī Ī in #P and the result follows from the nondeterministic algorithm for #Ī Ī that, on each computation path, guesses a subcircuit C of M and then verified that C satisfies the properties required by Ī Ī relative to MLP M input I. ā Theorem 20. For Ī āL=ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V G S C | V ā E x , B , M n l E x , M n l B , #Ī Ī is #P-hard. Proof. The result follows from the #P-hardness of #ExClique noted in the proof of Theorem 20 and the appropriate polynomial-time parsimonious reduction from ExClique to Ī Ī specified in the proof of Theorem 13. ā Theorem 21. #MnlLSC is #P-complete. Proof. Consider the reduction from MnlVC to MnlLSC created by modifying the reduction from VC to MLSC given in the proof of Theorem 5 such that the bias and input-line weight of input neuron niā¢nsubscriptn_innitalic_i n are changed to 0 and 1 to ensure that all possible input vectors (namely, 00\0\ 0 and 11\1\ 1 ) cause niā¢nsubscriptn_innitalic_i n to output 1. Observe that this modified reduction runs in time polynomial in the size of the given instance of MnlVC. We now need to show that this reduction is parsimonious, i.e., this reduction creates a bijection between the solution-sets of the given instance of MnlVC and the constructed instance of MnlLSC. We prove the two directions of this bijection separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a minimal vertex cover in G. Consider the subcircuit C based on neurons niā¢nsubscriptn_innitalic_i n, noā¢uā¢tsubscriptn_outnitalic_o u t, nā¢vā²ā¢N|vā²āVā²conditional-setsuperscriptā²superscriptā²\nv N~|~v ā V \ n vā² N | vā² ā Vā² , nā¢eā¢A1,nā¢eā¢A2,ā¦,nā¢eā¢A|E|subscript1subscript2ā¦subscript\neA_1,neA_2,ā¦,neA_|E|\ n e A1 , n e A2 , ā¦ , n e A| E | , and nā¢eā¢N1,nā¢eā¢N2,ā¦,nā¢eā¢N|E|subscript1subscript2ā¦subscript\neN_1,neN_2,ā¦,neN_|E|\ n e N1 , n e N2 , ā¦ , n e N| E | . As shown in the ā ā-portion of the proof of correctness of the reduction in the proof of Theorem 5, C is behaviorally equivalent to M on I. As Vā² is minimal and only vertex NOT neurons can be deleted from M to create C, C must itself be minimal. Moreover, note that any such set Vā² in G is associated with exactly one set of vertex NOT neurons in (and thus exactly one sufficient circuit of) M. ā ā : Let C be a minimal subcircuit of M that is behaviorally equivalent to M on input I. As neurons in all five layers in M must be present in C to produce the required output, both niā¢nsubscriptn_innitalic_i n and noā¢uā¢tsubscriptn_outnitalic_o u t are in C. In order for noā¢uā¢tsubscriptn_outnitalic_o u t to produce a non-zero output, all |E||E|| E | edge NOT neurons and all |E||E|| E | AND neurons must also be in C, and each of the latter must be connected to at least one of the vertex NOT neurons corresponding to their endpoint vertices. Hence, the vertices in G corresponding to the vertex NOT neurons in C must form a vertex cover in G. As C is minimal and only vertex NOT neurons can be deleted from M to create C, this vertex cover must itself be minimal. Moreover, note that any such set of vertex NOT neurons in M is associated with exactly one set of vertices in (and hence exactly one vertex cover of) G. As #MnlVC is #P-complete (Valiant, 1979, Theorem 1(4)), the reduction above establishes that #MnlLSC is #P-hard. Membership of #MnlLSC in #P and hence the result follows from the nondeterministic algorithm for #MnlLSC that, on each computation path, guesses a subcircuit C of M and then verifies that C is minimal and Cā¢(I)=Mā¢(I)C(I)=M(I)C ( I ) = M ( I ). ā Theorem 22. #MnlGSC is #P-hard. Proof. Recall that the parsimonious reduction from MnlVC to MnlLSC in the proof of Theorem 21 creates an instance of MnlLSC whose MLP M has the same output for every possible input vector; hence, this reduction is also a parsimonious reduction from MnlVC to MnlGSC. As #MnlVC is #P-complete (Valiant, 1979, Theorem 1(4)), this reduction establishes that #MnlGSC is #P-hard, giving the result. ā Theorem 23. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B , if #Ī Ī is polynomial-time solvable then P=Nā¢P=NPP = N P. Proof. Suppose there is a polynomial-time algorithm A for #Ī Ī for some Ī āLĪ ā LĪ ā L. Let R be the polynomial-time algorithm underlying the polynomial-time parsimonious reduction from ExClique to Ī Ī specified in the proof of Theorem 13. Construct an algorithm Aā² for the decision version of ExClique as follows: Given an input I for ExCliqueD, create input Iā² for Ī Ī using R, and apply A to Iā² to create solution S. If S=00S=0S = 0, return āNoā; otherwise, return āYesā. Algorithm Aā² is a polynomial-time algorithm for ExCliqueD; however, as ExCliqueD is Nā¢PNPN P-complete (Garey & Johnson, 1979, Problem GT19), this implies that P=Nā¢P=NPP = N P, giving the result. ā Theorem 24. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B and K=cā¢dg,#ā¢niā¢n,g,#ā¢noā¢uā¢t,g,Bmax,g,Wmax,g,cā¢dr,cā¢wr,#ā¢niā¢n,r,#ā¢noā¢uā¢t,r,#ā¢ntā¢oā¢t,r,Bmax,r,Wmax,rsubscript#subscript#subscriptsubscriptsubscriptsubscriptsubscript#subscript#subscript#subscriptsubscriptsubscriptK=\cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_in,% r,\#n_out,r,\#n_tot,r,B_ ,r,W_ ,r\K = c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Broman_max , r , Wroman_max , r , āØKā©delimited-āØā© K āØ K ā©-#Ī Ī is #Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard. Proof. The result follows from the #Wā¢[1]delimited-[]1W[1]W [ 1 ]-hardness of āØkā©delimited-āØā© k āØ k ā©-#ExClique (Flum & Grohe, 2006, Theorem 14.18) and the appropriate fpt parsimonious reduction from āØkā©delimited-āØā© k āØ k ā©-ExClique to āØKā©delimited-āØā© K āØ K ā©-Ī Ī specified in the proof of Theorem 14. ā Theorem 25. For Ī āL=ā¢Lā¢Sā¢C,ā¢Gā¢Sā¢C|āEā¢x,B,Mā¢nā¢lā¢Eā¢x,Mā¢nā¢lā¢BĪ conditional-set ā L=\ VLSC, VGSC~|~ Vā\Ex,B,MnlEx,MnlB\\Ī ā L = V L S C , V G S C | V ā E x , B , M n l E x , M n l B and K=cā¢dg,#ā¢niā¢n,g,#ā¢noā¢uā¢t,g,Bmax,g,Wmax,g,cā¢dr,cā¢wr,#ā¢niā¢n,r,#ā¢noā¢uā¢t,r,#ā¢ntā¢oā¢t,r,Bmax,r,Wmax,rsubscript#subscript#subscriptsubscriptsubscriptsubscriptsubscript#subscript#subscript#subscriptsubscriptsubscriptK=\cd_g,\#n_in,g,\#n_out,g,B_ ,g,W_ ,g,cd_r,cw_r,\#n_in,% r,\#n_out,r,\#n_tot,r,B_ ,r,W_ ,r\K = c ditalic_g , # nitalic_i n , g , # nitalic_o u t , g , Broman_max , g , Wroman_max , g , c ditalic_r , c witalic_r , # nitalic_i n , r , # nitalic_o u t , r , # nitalic_t o t , r , Broman_max , r , Wroman_max , r , if āØKā©delimited-āØā© K āØ K ā©-#Ī Ī is fixed-parameter tractable then Fā¢Pā¢T=#ā¢Wā¢[1]#delimited-[]1FPT=\#W[1]F P T = # W [ 1 ]. Proof. Suppose there is a fixed-parameter tractable algorithm A for āØKā©delimited-āØā© K āØ K ā©-#Ī Ī for some Ī āLĪ ā LĪ ā L. Let R be the fixed-parameter algorithm underlying the fpt parsimonious reduction from āØkā©delimited-āØā© k āØ k ā©-ExClique to āØKā©delimited-āØā© K āØ K ā©-Ī Ī specified in the proof of Theorem 14. Construct an algorithm Aā² for āØkā©delimited-āØā© k āØ k ā©-#ExClique as follows: Given an input I for āØk]ra k]raāØ k ] r a-#ExClique, create input Iā² for #Ī Ī using R, and apply A to Iā² to create solution S. If S=00S=0S = 0, return āNoā; otherwise, return āYesā. Algorithm Aā² is a fixed-parameter tractable algorithm for āØkā©delimited-āØā© k āØ k ā©-#ExCliqueD; however, as āØkā©delimited-āØā© k āØ k ā©-ExCliqueD is #ā¢Wā¢[1]#delimited-[]1\#W[1]# W [ 1 ]-complete (Flum & Grohe, 2006, Theorem 14.18), this implies that Fā¢Pā¢T=#ā¢Wā¢[1]#delimited-[]1FPT=\#W[1]F P T = # W [ 1 ], giving the result. ā Appendix D Global Sufficient Circuit Problem (sigma completeness) Minimum global sufficient circuit (MGSC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dgsubscript1subscript2ā¦subscriptsubscriptW_1,W_2,ā¦,W_cd_gW1 , W2 , ā¦ , Witalic_c d start_POSTSUBSCRIPT g end_POSTSUBSCRIPT, neuron bias vector B, and a positive integer k. Question: Is there a subcircuit C of M based on ā¤kabsentā¤ kā¤ k neurons from M such that for every possible input I of M, Cā¢(I)=Mā¢(I)C(I)=M(I)C ( I ) = M ( I )? Given a subset N of the neurons in M, the subcircuit C of M based on x has the neurons in x and all connections in M among these neurons. Note that in order for the output of C to be equal to the output of M on input I, the numbers #ā¢niā¢n#subscript\#n_in# nitalic_i n and #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t of input and output neurons in M must exactly equal the numbers of input and output neurons in C; hence, no input or output neurons can be deleted from M in creating C. Following BarcelĆ³ et al. 2020, page 4, all neurons in M use the ReLU activation function and the output x of each output neuron is stepped as necessary to be Boolean, i.e, sā¢tā¢eā¢pā¢(x)=00step(x)=0s t e p ( x ) = 0 if xā¤00xā¤ 0x ā¤ 0 and is 1111 otherwise. We will prove our result for MGSC using a polynomial-time reduction from the problem Minimum DNF Tautology. Given a DNF formula Ļitalic-ĻĻĻ over a set V of variables, Ļitalic-ĻĻĻ is a tautology if Ļitalic-ĻĻĻ evaluates to Tā¢rā¢uā¢eTrueT r u e for every possible truth-assignment to the variables in V. Our reduction will use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Theorem 26. MGSC is Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-complete. Proof. Let us first show the membership of MGSC in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2. Using the alternating-quantifier definition of classes in the polynomial hierarchy, membership of a decision problem Ī Ī in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proved by showing that solving Ī Ī for input I is equivalent to solving a quantified formula of the form ā(x)ā¢ā(y):pā¢(x,y):for-allā(x)ā(y):p(x,y)ā ( x ) ā ( y ) : p ( x , y ) where both the sizes of x and y and the evaluation time of predicate formula pā¢()p()p ( ) are upper-bounded by polynomials in |I||I|| I |. Such a formula for MGSC is ā(āā³)ā¢āā0,1#ā¢niā¢n:ā¢()=ā³ā¢():ā³for-allsuperscript01#subscriptā³ā(C ) ā\0,1\^\#n_in% :C(x)=M(x)ā ( C ā M ) ā x ā 0 , 1 # nitalic_i n : C ( x ) = M ( x ) We now show the Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-hardness of MGSC. Consider the following reduction from 3DT to MGSC. Given an instance āØĻ,T,V,kā©italic-Ļ Ļ,T,V,k āØ Ļ , T , V , k ā© of 3DT, construct the following instance āØM,kā²ā©superscriptā² M,k āØ M , kā² ā© of MGSC: Let M be an MLP based on 3ā¢|V|+2ā¢T+23223|V|+2T+23 | V | + 2 T + 2 neurons spread across five layers: 1. Input neuron layer: The input neurons nā¢i1,nā¢i2,ā¦,nā¢i|V|subscript1subscript2ā¦subscriptni_1,ni_2,ā¦,ni_|V|n i1 , n i2 , ā¦ , n i| V | (all with bias 0). 2. Hidden layer I: The unnegated variable identity neurons nā¢vā¢U1,nā¢vā¢U2,ā¦,nā¢vā¢U|V|subscript1subscript2ā¦subscriptnvU_1,nvU_2,ā¦,nvU_|V|n v U1 , n v U2 , ā¦ , n v U| V | (all with bias 0) and negated variable NOT neurons nā¢vā¢N1,nā¢vā¢N2,ā¦,nā¢vā¢N|V|subscript1subscript2ā¦subscriptnvN_1,nvN_2,ā¦,nvN_|V|n v N1 , n v N2 , ā¦ , n v N| V | (all with bias 1). 3. Hidden layer I: (a) The term 3-way AND neurons nā¢T1,xā¢T2,ā¦,xā¢T|T|subscript1subscript2ā¦subscriptnT_1,xT_2,ā¦,xT_|T|n T1 , x T2 , ā¦ , x T| T | (all with bias -2). (b) The gadget neuron ngsubscriptn_gnitalic_g (bias |V|ā11|V|-1| V | - 1). 4. Hidden layer I: The modified term 2-way AND neurons nā¢Tā¢m1,nā¢Tā¢M2,ā¦,nā¢Tā¢M|T|subscript1subscript2ā¦subscriptnTm_1,nTM_2,ā¦,nTM_|T|n T m1 , n T M2 , ā¦ , n T M| T | (all with bias ā11-1- 1). 5. Output layer: The stepped output neuron noā¢uā¢tsubscriptn_outnitalic_o u t. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input neuron nā¢iisubscriptni_in iitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is input-connected with weight 1 to its corresponding input line and output-connected with weights 1 and -1 to unnegated and negated variable neurons nā¢vā¢UisubscriptnvU_in v Uitalic_i and nā¢vā¢NisubscriptnvN_in v Nitalic_i, respectively. ā¢ Each term neuron nā¢TisubscriptnT_in Titalic_i, 1ā¤iā¤|T|11ā¤ iā¤|T|1 ā¤ i ā¤ | T |, is input-connected with weight 1 to each of the 3 variable neurons corresponding to that termās literals. ā¢ The gadget neuron ngsubscriptn_gnitalic_g is input-connected with weight 1 to all of the unnegated and negated variable neurons. ā¢ Each modified term neuron nā¢Tā¢misubscriptnTm_in T mitalic_i, 1ā¤iā¤|T|11ā¤ iā¤|T|1 ā¤ i ā¤ | T |, is input-connected with weight 1 to the term neuron nā¢TisubscriptnT_in Titalic_i and the gadget neuron ngsubscriptn_gnitalic_g and output-connected with weight 1 to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t. All other connections between neurons in adjacent layers have weight 0. Finally, let kā²=3ā¢|V|+2ā¢k+2superscriptā²322k =3|V|+2k+2kā² = 3 | V | + 2 k + 2. Observe that this instance of MGSC can be constructed time polynomial in the size of the given instance of 3DT. The following observations about the MLP M constructed above will be of use: ā¢ The input to M is exactly that of the 3-DNF formula Ļitalic-ĻĻĻ. ā¢ The output neuron of M outputs 1 if and only if one or more of the modified term neurons output 1. ā¢ Modified term neuron nā¢Tā¢misubscriptnTm_in T mitalic_i outputs 1 if and only both the term neuron nā¢TisubscriptnT_in Titalic_i and the gadget neuron ngsubscriptn_gnitalic_g output 1. ā¢ The gadget neuron ngsubscriptn_gnitalic_g outputs 1 for input I to a subcircuit C of M if and only if the negated and unnegated variable neurons corresponding to I each output 1; hence, ngsubscriptn_gnitalic_g outputs 1 for all possible inputs to C if and only if all negated and unnegated variable neurons in hidden layer I are part of C. As Ļitalic-ĻĻĻ is a tautology, the above implies that (1) M outputs 1 for every possible input and (2) every global sufficient circuit of M must include all input and variable neurons, the gadget and output neurons, and at least one term / modified term neuron-pair. We now need to show the correctness of this reduction by proving that the answer for the given instance of 3DT is āYesā if and only if the answer for the constructed instance of MGSC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Tā², |Tā²|=ksuperscriptā²|T |=k| Tā² | = k be a subset of the terms in Ļitalic-ĻĻĻ that is a tautology. As noted above, any global sufficient circuit C for the constructed MLP M must include all input and variable neurons, the gadget and output neurons, and at least one term / modified term neuron-pair. Let C contain the term / modified term neuron-pairs corresponding to the terms in Tā². Such a C is therefore a global sufficient circuit for M of size kā²=3ā¢|V|=2ā¢k+2superscriptā²322k =3|V|=2k+2kā² = 3 | V | = 2 k + 2. ā ā : Let C be a global sufficient circuit for M of size kā²=3ā¢|V|+2ā¢k+2superscriptā²322k =3|V|+2k+2kā² = 3 | V | + 2 k + 2. As noted above, C must include all input and variable neurons, the gadget and output neurons, and k term / modified term neuron-pairs. Let Tā² be the subset of k terms in Ļitalic-ĻĻĻ corresponding to these neuron-pairs. Given the input-output equivalence Ļitalic-ĻĻĻ and M (and hence any sufficient circuit for M), the disjunction of the k terms in Tā² must be a tautology. As 3DT is Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-hard (Schaefer & Umans, 2002, Problem L7), the reduction above establishes that MGSC is also Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2-hard. The result then follows from the membership of MGSC in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 shown at the beginning of this proof. ā Appendix E Quasi-Minimal Sufficient Circuit Problem Quasi-Minimal Sufficient Circuit (QMSC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a set XX of input vectors of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. Output: a circuit CC in ā³MM and a neuron vāv ā C such that ā¢()=ā³ā¢()ā³C(x)=M(x)C ( x ) = M ( x ) and [āv]ā¢()ā ā³ā¢()delimited-[]ā³[C \v\](x) (x)[ C ā v ] ( x ) ā M ( x ) Theorem 27. QMSC is in PTIME (i.e., polynomial-time tractable). Proof. Consider the following algorithm for QMSC. Build a sequence of MLPs by taking ā³MM with all neurons labeled 1, and generating subsequent ā³isubscriptā³M_iMitalic_i in the sequence by labeling an additional neuron with 0 each time (this choice can be based on any heuristic strategy, for instance, one based on gradients). The first MLP, ā³1subscriptā³1M_1M1, obtained by removing all neurons labeled 0 (i.e., none) is such that ā³1ā¢(x)=ā³ā¢(x)subscriptā³1ā³M_1(x)=M(x)M1 ( x ) = M ( x ), and the last ā³nsubscriptā³M_nMitalic_n is guaranteed to give ā³nā¢(x)ā ā³ā¢(x)subscriptā³M_n(x) (x)Mitalic_n ( x ) ā M ( x ) because all neurons are removed. Label the first MLP YES, and the last NO. Perform a variant of binary search on the sequence as follows. Evaluate the ā³isubscriptā³M_iMitalic_i halfway between YES and NO while removing all its neurons labeled 0. If it satisfies the condition, label it YES, and repeat the same strategy with the sequence starting from the YES just labeled until the last ā³nsubscriptā³M_nMitalic_n. If it does not satisfy the condition, label it NO and repeat the same strategy with the sequence starting from the YES at the beginning of the original sequence until the NO just labeled. This iterative procedure halves the sequence each time. Halt when you find two adjacent āØYES,NOā©YESNO YES, NO āØ YES , NO ā© circuits (guaranteed to exist), and return the circuit set of the YES network V and the single neuron difference between YES and NO (the breaking point), vāVvā Vv ā V. The complexity of this algorithm is roughly Oā¢(nā¢logā”n)O(n n)O ( n log n ). ā Appendix F Gnostic Neurons Problem Gnostic Neurons (GN) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, and two sets XX and YY of input vectors of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Output: a subset of neurons V in M of size |V|ā„k|V|ā„ k| V | ā„ k such that āvāVsubscriptfor-all _vā Vāv ā V it is the case that āāsubscriptfor-all _x āx ā X computing Mā¢()M(x)M ( x ) produces activations Avā„tsubscriptsuperscriptA^v_xā„ tAitalic_vbold_x ā„ t and āā:Av<t:subscriptfor-allsubscriptsuperscript _y :A^v_y<tāy ā Y : Aitalic_vbold_y < t. Theorem 28. GN is in PTIME (i.e., polynomial-time tractable). Proof. Consider the complexity of the following subroutines of an algorithm for GN. Computing the activations of all neurons of M for all āx ā X and all āy ā Y takes polynomial time in |M|,|||M|,|X|| M | , | X | and |||Y|| Y |. Labeling neurons that pass or not the activation threshold takes time polynomial in |M|,|||M|,|X|| M | , | X | and |||Y|| Y |. Finally, checking whether the set of neurons that fulfils the condition is of size at least k can be done in polynomial time in |M||M|| M |. These subroutines can be put together to yield a polynomial-time algorithm for GN. ā Remark 1 One could also add to the output of the computational problem the requirement that if we silence (or activate) the neuron, we should elicit (or abolish) a behavior. Note that checking these effects can be done in polynomial time in all of the input parts given above and also in the size of the behavior set (which should be added to the input in these variants). Appendix G Necessary Circuit Problem Minimum Local Necessary Circuit (MLNC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that Nā²ā©Cā ā superscriptā²N ā© Cā ā² ā© C ā ā for every sufficient circuit C of M relative to I? Minimum Global Necessary Circuit (MGNC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that for for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, Nā²ā©Cā ā superscriptā²N ā© Cā ā² ā© C ā ā for every sufficient circuit C of M relative to I? We will use reductions from the Hitting Set problem to prove our results for the problems above. Regarding the Hitting Set problem, we shall assume an ordering on the sets and elements in C and S, respectively. Our parameterized results are proved relative to the parameters in Table 7. Lemmas 1 and 2 will be useful in deriving additional parameterized results from proved ones. Our reductions will use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Table 7: Parameters for the minimum necessary circuit problem. Parameter Description cā¢dcdc d # layers in given MLP cā¢wcwc w max # neurons in layer in given MLP #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP BmaxsubscriptB_ Broman_max max neuron bias in given MLP WmaxsubscriptW_ Wroman_max max connection weight in given MLP k Size of requested neuron subset G.1 Results for MLNC Membership of MLNC in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[Nāā³]ā¢ā[āā³]:[ā¢()=ā³ā¢()]ā¹Nā©ā ā :delimited-[]ā³for-alldelimited-[]ā³delimited-[]ā³ā[N ]\ ā[C ]:[% C(x)=M(x)] N ā ā [ N ā M ] ā [ C ā M ] : [ C ( x ) = M ( x ) ] ā¹ N ā© C ā ā Theorem 29. If MLNC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from HS to MLNC. Given an instance āØC,S,kā© C,S,k āØ C , S , k ā© of HS, construct the following instance āØM,I,kā²ā©superscriptā² M,I,k āØ M , I , kā² ā© of MLNC: Let M be an MLP based on #ā¢ntā¢oā¢t=|S|+|C|+2#subscript2\#n_tot=|S|+|C|+2# nitalic_t o t = | S | + | C | + 2 neurons spread across four layers: 1. Input neuron layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias +11+1+ 1). 2. Hidden element layer: The element neurons nā¢s1,nā¢s2,ā¦ā¢nā¢s|S|subscript1subscript2ā¦subscriptns_1,ns_2,ā¦ ns_|S|n s1 , n s2 , ā¦ n s| S | (all with bias 0). 3. Hidden set layer: The set AND neurons nā¢c1,nā¢c2,ā¦,nā¢c|C|subscript1subscript2ā¦subscriptnc_1,nc_2,ā¦,nc_|C|n c1 , n c2 , ā¦ , n c| C | (such that neuron nā¢cisubscriptnc_in citalic_i has bias ā|ci|subscript-|c_i|- | citalic_i |). 4. Output layer: The single stepped output neuron noā¢uā¢tsubscriptn_outnitalic_o u t (bias 0). The non-zero weight connections between adjacent layers are as follows: ā¢ The input neuron has an edge of weight 0 coming from its input and is in turn connected to each of the element neurons with weight 1. ā¢ Each element neuron nā¢sisubscriptns_in sitalic_i, 1ā¤iā¤|S|11ā¤ iā¤|S|1 ā¤ i ā¤ | S |, is connected to each set neuron nā¢cjsubscriptnc_jn citalic_j, 1ā¤jā¤|C|11ā¤ jā¤|C|1 ā¤ j ā¤ | C |, such that siācjsubscriptsubscripts_iā c_jsitalic_i ā citalic_j with weight 1. ā¢ Each set neuron nā¢cisubscriptnc_in citalic_i, 1ā¤iā¤|C|11ā¤ iā¤|C|1 ā¤ i ā¤ | C |, is connected to the output neuron with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(0)0I=(0)I = ( 0 ) and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLNC can be created in time polynomial in the size of the given instance of HS. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢s1,nā¢s2,ā¦,nā¢s|S|subscript1subscript2ā¦subscriptns_1,ns_2,ā¦,ns_|S|n s1 , n s2 , ā¦ , n s| S | (1) 3 nā¢c1,nā¢c2,ā¦,nā¢c|C|subscript1subscript2ā¦subscriptnc_1,nc_2,ā¦,nc_|C|n c1 , n c2 , ā¦ , n c| C | (1) 4 noā¢uā¢t(1))n_out(1))nitalic_o u t ( 1 ) ) Note the following about the behavior of M: Observation 1. For any set neuron nā¢cisubscriptnc_in citalic_i to output 1, it must receive input 1 from all of its incoming element neurons connected with weight 1. Observation 2. For the output neuron to output 1, it is sufficient to get input 1 from any of its incoming set neurons with weight 1. Observations 1 and 2 imply that any sufficient circuit for M must contain at least one set neuron and all of its associated element neurons. We now need to show the correctness of this reduction by proving that the answer for the given instance of HS is āYesā if and only if the answer for the constructed instance of MLNC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Sā²=s1ā²,s2ā²,ā¦,skā²āSsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²S =\s _1,s _2,ā¦,s _k\ Sā² = sā²1 , sā²2 , ā¦ , sā²italic_k ā S be a hitting set of size k for C. By the construction above, the k element neurons corresponding to the elements in Sā² collectively connect with weight 1 to all set neurons in M. By Observations 1 and 2, this means that the set N of these element neurons has a non-empty intersection with every sufficient circuit for M, and hence that N is a necessary circuit for M of size k=kā²=k k = kā². ā ā : Let N be a necessary circuit for M of size kā². Let NSsubscriptN_SNitalic_S and NCsubscriptN_CNitalic_C be the subsets of N that are element and set neurons. We can create a set Nā² consisting only of element neurons by replacing each set neuron ncsubscriptn_cnitalic_c in NCsubscriptN_CNitalic_C with an arbitrary element neuron that is not already in NSsubscriptN_SNitalic_S and is connected to ncsubscriptn_cnitalic_c with weight 1. Observe that Nā² (whose size may be less than kā² if any ncsubscriptn_cnitalic_c already had an associated element neuron in NSsubscriptN_SNitalic_S) remains a necessary circuit for M. Moreover, as the element neurons in Nā² by definition have a non-empty intersection with each sufficient circuit for M, by Observations 1 and 2 above, the set Sā² of elements in S corresponding to the element neurons in Nā² has a non-empty intersection with each set in C and hence is a hitting set of size Nā²ā¤kā²=ksuperscriptā²N ā¤ k =kNā² ā¤ kā² = k As HS is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLNC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 30. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,kā©#subscript#subscriptsubscript cd,\#n_in,\#n_out,W_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , k ā©-MLNC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MLNC constructed in the reduction in the proof of Theorem 29, #ā¢niā¢n=#ā¢noā¢uā¢t=Wmax=1#subscript#subscriptsubscript1\#n_in=\#n_out=W_ =1# nitalic_i n = # nitalic_o u t = Wroman_max = 1, cā¢d=44cd=4c d = 4, and kā² is a function of k in the given instance of HS. The result then follows from the facts that āØkā©delimited-āØā© k āØ k ā©-HS is Wā¢[2]delimited-[]2W[2]W [ 2 ]-hard (by a reduction from āØkā©delimited-āØā© k āØ k ā©-Dominating set; Downey & Fellows 1999) and Wā¢[1]āWā¢[2]delimited-[]1delimited-[]2W[1] W[2]W [ 1 ] ā W [ 2 ]. ā Theorem 31. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLNC is fixed-parameter tractable. Proof. Consider the algorithm that generates every possible subset Nā² of size at most k of the neurons in MLP M and for each such subset, generates every possible subset Nā²N Nā² ā² of M, checks if Nā²N Nā² ā² is a sufficient circuit for M relative to I and, if so, checks if Nā² has a non-empty intersection with Nā²N Nā² ā². If an Nā² is found that has a non-empty intersection with each sufficient circuit for M relative to I, return āYesā; otherwise, return āNoā. The number of possible subsets Nā² and Nā²N Nā² ā² are both at most 2#ā¢ntā¢oā¢tsuperscript2#subscript2^\#n_tot2# nitalic_t o t. As all subsequent checking operations can be done in time polynomial in the size of the given instance of MLNC, the above is a fixed-parameter tractable algorithm for MLNC relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 32. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLNC is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 31 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 30ā32 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MLNC relative to many subsets of the parameters listed in Table 7. Let us now consider the polynomial-time cost approximability of MLNC. Theorem 33. If MLNC has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then P=Nā¢P=NPP = N P. Proof. Recall from the proof of correctness of the reduction in the proof of Theorem 29 that a given instance of HS has a hitting set of size k if and only if the constructed instance of MLNC has a necessary circuit of size kā²=ksuperscriptā²k =kā² = k. This implies that, given a polynomial-time c-approximation algorithm A for MLNC for some constant c>00c>0c > 0, we can create a polynomial-time c-approximation algorithm for HS by applying the reduction to the given instance x of HS to construct an instance xā² of MLNC, applying A to xā² to create an approximate solution yā², and then using yā² to create an approximate solution y for x that has the same cost as yā². The result then follows from Ausiello et al. 1999, Problem SP7, which states that if HS has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 a and is hence in approximation problem class APX then P=Nā¢P=NPP = N P. ā Note that this theorem also renders MLNC PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. G.2 Results for MGNC Membership of MLNC in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[Nāā³]ā[āā³s.t.ā(ā0,1#ā¢niā¢n)]:Nā©ā ā ā[N ]\ ā[C \ s.t.% \ ā(xā\0,1\^\#n_in)]:N ā ā [ N ā M ] ā [ C ā M s . t . ā ( x ā 0 , 1 # nitalic_i n ) ] : N ā© C ā ā Theorem 34. If MGNC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLNC constructed by the reduction in the proof of Theorem 29, the input-connection weight 0 and bias 1 of the input neuron force this neuron to output 1 for both of the possible input vectors (1)1(1)( 1 ) and (0)0(0)( 0 ). Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of MGNC. ā Theorem 35. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,kā©#subscript#subscriptsubscript cd,\#n_in,\#n_out,W_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , k ā©-MGNC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MGNC constructed in the reduction in the proof of Theorem 34, #ā¢niā¢n=#ā¢noā¢uā¢t=Wmax=1#subscript#subscriptsubscript1\#n_in=\#n_out=W_ =1# nitalic_i n = # nitalic_o u t = Wroman_max = 1, cā¢d=44cd=4c d = 4, and kā² is a function of k in the given instance of HS The result then follows from the facts that āØkā©delimited-āØā© k āØ k ā©-HS is Wā¢[2]delimited-[]2W[2]W [ 2 ]-hard (by a reduction from āØkā©delimited-āØā© k āØ k ā©-Dominating set; Downey & Fellows 1999) and Wā¢[1]āWā¢[2]delimited-[]1delimited-[]2W[1] W[2]W [ 1 ] ā W [ 2 ]. ā Theorem 36. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MGNC is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 31 such that each potential sufficient circuit Nā²N Nā² ā² is checked to ensure that Mā¢(I)=Nā²ā¢(I)superscriptā²M(I)=N (I)M ( I ) = Nā² ā² ( I ) for every possible Boolean input vector of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢n<2#ā¢ntā¢oā¢tsuperscript2#subscriptsuperscript2#subscript2^\#n_in<2^\#n_tot2# nitalic_i n < 2# nitalic_t o t, the above is a fixed-parameter tractable algorithm for MGNC relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 37. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MGNC is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 36 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 35ā37 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MGNC relative to many subsets of the parameters listed in Table 7. Let us now consider the polynomial-time cost approximability of MGNC. Theorem 38. If MGNC has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then P=Nā¢P=NPP = N P. Proof. As the reduction in the proof of Theorem 34 is essentially the same as the reduction in the proof of Theorem 29, the result follows by the same reasoning as given in the proof of Theorem 33. ā Note that this theorem also renders MGNC PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Appendix H Circuit Ablation and Clamping Problems Given an MLP M and a subset N of the neurons in M, the MLP Mā² induced by N is said to be active if there is at least one path between the the input and output neurons in Mā²; otherwise, Mā² is inactive. As we are interested in inductions that preserve or violate output behaviour, all output neurons of M must be preserved in Mā²; however, we only require that at least one input neuron be so preserved. Unless otherwise stated, all inductions discussed wrt MLCA and MGCA below will be assumed to result in active MLP. Minimum Local Circuit Ablation (MLCA) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for the MLP Mā² induced by NāNā² N N ā Nā²? Minimum Global Circuit Ablation (MGCA) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that for the MLP Mā² induced by NāNā² N N ā Nā², Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n? Given an MLP M and a neuron v in M, v is clamped to value vā¢aā¢lvalv a l if the output of v is always vā¢aā¢lvalv a l regardless of the inputs to v. As one can trivially change the output of an MLP by clamping one or more of its output neurons, we shall not allow the clamping of output neurons in the problems below. Minimum Local Circuit Clamping (MLCC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, a Boolean value vā¢aā¢lvalv a l, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for the MLP Mā² in which all neurons in Nā² are clamped to value vā¢aā¢lvalv a l? Minimum Global Circuit Clamping (MGCC) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean value vā¢aā¢lvalv a l, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that for the MLP Mā² in which all neurons in Nā² are clamped to value vā¢aā¢lvalv a l, Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n? Following BarcelĆ³ et al. 2020, page 4, all neurons in M use the ReLU activation function and the output x of each output neuron is stepped as necessary to be Boolean, i.e, sā¢tā¢eā¢pā¢(x)=00step(x)=0s t e p ( x ) = 0 if xā¤00xā¤ 0x ā¤ 0 and is 1111 otherwise. For a graph G=(V,E)G=(V,E)G = ( V , E ), we shall assume an ordering on the vertices and edges in V and E, respectively. For each vertex vāVvā Vv ā V, let the complete neighbourhood NCā¢(v)subscriptN_C(v)Nitalic_C ( v ) of v be the set composed of v and the set of all vertices in G that are adjacent to v by a single edge, i.e., vāŖu|uāVā¢andā¢(u,v)āEconditional-setanduvEvāŖ\u~|~u~ā V~ and~(u,v)ā E\v āŖ u | u ā V and ( u , v ) ā E . We will prove various classical and parameterized results for MLCA, MGCA,MLCC, and MGCC using reductions from Clique. The parameterized results are proved relative to the parameters in Table 8. Lemmas 1 and 2 will be useful in deriving additional parameterized results from proved ones. Additional reductions from DS used to prove polynomial-time cost inapproximability use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Table 8: Parameters for the minimum circuit ablation and clamping problems. Parameter Description cā¢dcdc d # layers in given MLP cā¢wcwc w max # neurons in layer in given MLP #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP BmaxsubscriptB_ Broman_max max neuron bias in given MLP WmaxsubscriptW_ Wroman_max max connection weight in given MLP k Size of requested neuron subset H.1 Results for Minimal Circuit Ablation The following hardness and inapproximability results are notable for holding when the given MLP M has three hidden layers. H.1.1 Results for MLCA Towards proving NP-completeness, we first prove membership and then follow up with hardness. Membership in NP can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]:[ā³ā]ā¢()ā ā³ā¢():delimited-[]ā³delimited-[]ā³ā[S ]:[M ](% x) (x)ā [ S ā M ] : [ M ā S ] ( x ) ā M ( x ) Theorem 39. If MLCA is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from Clique to MLCA. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of Clique, construct the following instance āØM,I,kā²ā©superscriptā² M,I,k āØ M , I , kā² ā© of MLCA: Let M be an MLP based on #ā¢ntā¢oā¢t=3ā¢|V|+|E|+2#subscript32\#n_tot=3|V|+|E|+2# nitalic_t o t = 3 | V | + | E | + 2 neurons spread across five layers: 1. Input neuron layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias +11+1+ 1). 2. Hidden vertex pair layer: The vertex neurons nā¢vā¢Pā¢11,nā¢vā¢Pā¢12,ā¦ā¢nā¢vā¢Pā¢1|V|subscript11subscript12ā¦subscript1nvP1_1,nvP1_2,ā¦ nvP1_|V|n v P 11 , n v P 12 , ā¦ n v P 1| V | and nā¢vā¢Pā¢21,nā¢vā¢Pā¢22,ā¦ā¢nā¢vā¢Pā¢2|V|subscript21subscript22ā¦subscript2nvP2_1,nvP2_2,ā¦ nvP2_|V|n v P 21 , n v P 22 , ā¦ n v P 2| V | (all with bias 0). 3. Hidden vertex regulator layer: The vertex neurons nā¢vā¢R1,nā¢vā¢R2,ā¦ā¢nā¢R|V|subscript1subscript2ā¦subscriptnvR_1,nvR_2,ā¦ nR_|V|n v R1 , n v R2 , ā¦ n R| V | (all with bias 0). 4. Hidden edge layer: The edge neurons nā¢e1,nā¢e2,ā¦ā¢nā¢e|E|subscript1subscript2ā¦subscriptne_1,ne_2,ā¦ ne_|E|n e1 , n e2 , ā¦ n e| E | (all with bias ā11-1- 1). 5. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t (bias ā(kā¢(kā1)/2ā1)121-(k(k-1)/2-1)- ( k ( k - 1 ) / 2 - 1 )). The non-zero weight connections between adjacent layers are as follows: ā¢ Each input neuron has an edge of weight 0 coming from its corresponding input and is in turn connected to each of the vertex pair neurons with weight 1. ā¢ Each P1 (P2) vertex pair neuron nā¢vā¢Pā¢1isubscript1nvP1_in v P 1i (nā¢vā¢Pā¢2isubscript2nvP2_in v P 2i), 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to vertex regulator neuron nā¢vā¢RisubscriptnvR_in v Ritalic_i with weight ā22-2- 2 (1). ā¢ Each vertex regulator neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge neuron nā¢eisubscriptne_in eitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(1)1I=(1)I = ( 1 ) and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLCA can be created in time polynomial in the size of the given instance of Clique. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢vā¢Pā¢11ā¢(1),nā¢vā¢Pā¢12ā¢(1),ā¦ā¢nā¢vā¢Pā¢1|V|ā¢(1),nā¢vā¢Pā¢21ā¢(1),nā¢vā¢Pā¢22ā¢(1),ā¦ā¢nā¢vā¢Pā¢2|V|ā¢(1)subscript111subscript121ā¦subscript11subscript211subscript221ā¦subscript21nvP1_1(1),nvP1_2(1),ā¦ nvP1_|V|(1),nvP2_1(1),nvP2_2(1),ā¦ nvP% 2_|V|(1)n v P 11 ( 1 ) , n v P 12 ( 1 ) , ā¦ n v P 1| V | ( 1 ) , n v P 21 ( 1 ) , n v P 22 ( 1 ) , ā¦ n v P 2| V | ( 1 ) 3 nā¢vā¢R1ā¢(0),nā¢vā¢R2ā¢(0),ā¦ā¢nā¢vā¢R|V|ā¢(0)subscript10subscript20ā¦subscript0nvR_1(0),nvR_2(0),ā¦ nvR_|V|(0)n v R1 ( 0 ) , n v R2 ( 0 ) , ā¦ n v R| V | ( 0 ) 4 nā¢e1ā¢(0),nā¢e2ā¢(0),ā¦ā¢nā¢e|E|ā¢(0)subscript10subscript20ā¦subscript0ne_1(0),ne_2(0),ā¦ ne_|E|(0)n e1 ( 0 ) , n e2 ( 0 ) , ā¦ n e| E | ( 0 ) 5 noā¢uā¢t(0))n_out(0))nitalic_o u t ( 0 ) ) We now need to show the correctness of this reduction by proving that the answer for the given instance of Clique is āYesā if and only if the answer for the constructed instance of MLCA is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a clique in G of size kā²ā„ksuperscriptā²k ā„ kā² ā² ā„ k and Nā² be the kā²ā„kā²=ksuperscriptā²k ā„ k =kā² ā² ā„ kā² = k-sized subset of the P1 vertex pair neurons corresponding to the vertices in Vā². Let Mā² be the version of M in which all neurons in Nā² are ablated. As each of these vertex pair neurons previously forced their associated vertex regulator neurons to output 0 courtesy of their connection-weight of ā22-2- 2, their ablation now allows these kā²k kā² ā² vertex regulator neurons to output 1. As Vā² is a clique of size kā²k kā² ā², exactly kā²ā¢(kā²ā1)/2ā„kā¢(kā1)/2superscriptā²1212k (k -1)/2ā„ k(k-1)/2kā² ā² ( kā² ā² - 1 ) / 2 ā„ k ( k - 1 ) / 2 edge neurons in Mā² receive the requisite inputs of 1 on both of their endpoints from the vertex regulator neurons associated with the P1 vertex pair neurons in Nā². This in turn ensures the output neuron produces output 1. Hence, Mā¢(I)=0ā 1=Mā²ā¢(I)01superscriptā²M(I)=0ā 1=M (I)M ( I ) = 0 ā 1 = Mā² ( I ). ā ā : Let Nā² be a subset of N of size at most kā²=ksuperscriptā²k =kā² = k such that for the MLP Mā² induced by ablating all neurons in Nā², Mā¢(I)ā Mā¢(Iā²)superscriptā²M(I)ā M(I )M ( I ) ā M ( Iā² ). As Mā¢(I)=00M(I)=0M ( I ) = 0 and circuit outputs are stepped to be Boolean, Mā²ā¢(I)=1superscriptā²1M (I)=1Mā² ( I ) = 1. Given the bias of the output neuron, this can only occur if at least kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons in Mā² have output 1 on input I, which requires that each of these neurons receives 1 from both of its endpoint vertex regulator neurons. These vertex regulator neurons can only output 1 if all of their associated P1 vertex neurons have been ablated; moreover, there must be exactly k such neurons. This means that the vertices in G corresponding to the P1 vertex pair neurons in Nā² must form a clique of size k in G. As Clique is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLCA is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 40. If āØcā¢d,#ā¢niā¢n.#ā¢noā¢uā¢t,Wmax,Bmax,kā©delimited-āØā©formulae-sequence#subscript#subscriptsubscriptsubscript cd,\#n_in.\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n . # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCA is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MLCA constructed in the reduction in the proof of Theorem 39, #ā¢niā¢n=#ā¢noā¢uā¢t=Wmax=1#subscript#subscriptsubscript1\#n_in=\#n_out=W_ =1# nitalic_i n = # nitalic_o u t = Wroman_max = 1, cā¢d=55cd=5c d = 5, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 41. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCA is fixed-parameter tractable. Proof. Consider the algorithm that generates every possible subset Nā² of size at most k of the neurons N in MLP M and for each such subset, creates the MLP Mā² induced from M by ablating the neurons in Nā² and (assuming Mā² is active) checks if Mā²ā¢(I)ā Mā¢(I)superscriptā²M (I)ā M(I)Mā² ( I ) ā M ( I ). If such a subset is found, return āYesā; otherwise, return āNoā. The number of possible subsets Nā² is at most kĆ#ā¢ntā¢oā¢tkā¤#ā¢ntā¢oā¢tĆ#ā¢ntā¢oā¢t#ā¢ntā¢oā¢t#superscriptsubscript#subscript#superscriptsubscript#subscriptkĆ\#n_tot^kā¤\#n_totĆ\#n_tot^\#n_totk Ć # nitalic_t o titalic_k ā¤ # nitalic_t o t Ć # nitalic_t o t# nitalic_t o t. As any such Mā² can be generated from M, checked or activity, and run on I in time polynomial in the size of the given instance of MLCA, the above is a fixed-parameter tractable algorithm for MLCA relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 42. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLCA is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 41 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 40ā42 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MLCA relative to many subsets of the parameters listed in Table 8. Let us now consider the polynomial-time cost approximability of MLCA. As MLCA is a minimization problem, we cannot do this using reductions from a maximization problem like Clique. Hence we will instead use a reduction from another minimization problem, namely DS. Theorem 43. If MLCA is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from DS to MLCA. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of DS, construct the following instance āØM,I,kā²ā©superscriptā² M,I,k āØ M , I , kā² ā© of MLCA: Let M be an MLP based on #ā¢ntā¢oā¢t,g=3ā¢|V|+1#subscript31\#n_tot,g=3|V|+1# nitalic_t o t , g = 3 | V | + 1 neurons spread across four layers: 1. Input layer: The input vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V |, all of which have bias 1. 2. Hidden vertex neighbourhood layer I: The vertex neighbourhood AND neurons nā¢vā¢nā¢A1,nā¢vā¢nā¢A2,ā¦ā¢nā¢vā¢nā¢A|V|subscript1subscript2ā¦subscriptnvnA_1,nvnA_2,ā¦ nvnA_|V|n v n A1 , n v n A2 , ā¦ n v n A| V |, where nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i is an x-way AND ReLU gates such that x=|NCā¢(vi)|subscriptsubscriptx=|N_C(v_i)|x = | Nitalic_C ( vitalic_i ) |. 3. Hidden vertex neighbourhood layer I: The vertex neighbourhood NOT neurons nā¢vā¢nā¢N1,nā¢vā¢nā¢N2,ā¦ā¢nā¢vā¢nā¢N|V|subscript1subscript2ā¦subscriptnvnN_1,nvnN_2,ā¦ nvnN_|V|n v n N1 , n v n N2 , ā¦ n v n N| V |, all of which are NOT ReLU gates. 4. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is a |V||V|| V |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its input line with weight 0 and to each vertex neighbourhood AND neuron nā¢vā¢nā¢AjsubscriptnvnA_jn v n Aitalic_j such that viāNCā¢(vj)subscriptsubscriptsubscriptv_iā N_C(v_j)vitalic_i ā Nitalic_C ( vitalic_j ) with weight 1. ā¢ Each vertex neighbourhood AND neuron nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its corresponding vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i with weight 1. ā¢ Each vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I be the |V||V|| V |-length one-vector and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLCA can be created in time polynomial in the size of the given instance of DS, Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 nā¢v1ā¢(1),nā¢v2ā¢(1),ā¦ā¢nā¢v|V|ā¢(1)subscript11subscript21ā¦subscript1nv_1(1),nv_2(1),ā¦ nv_|V|(1)n v1 ( 1 ) , n v2 ( 1 ) , ā¦ n v| V | ( 1 ) 2 nā¢vā¢nā¢A1ā¢(1),nā¢vā¢nā¢A2ā¢(1),ā¦ā¢nā¢vā¢nā¢A|V|ā¢(1)subscript11subscript21ā¦subscript1nvnA_1(1),nvnA_2(1),ā¦ nvnA_|V|(1)n v n A1 ( 1 ) , n v n A2 ( 1 ) , ā¦ n v n A| V | ( 1 ) 3 nā¢vā¢nā¢N1ā¢(0),nā¢vā¢nā¢N2ā¢(0),ā¦ā¢nā¢vā¢nā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0nvnN_1(0),nvnN_2(0),ā¦ nvnN_|V|(0)n v n N1 ( 0 ) , n v n N2 ( 0 ) , ā¦ n v n N| V | ( 0 ) 4 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of DS is āYesā if and only if the answer for the constructed instance of MLCA is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a dominating set in G of size k and Nā² be the kā²=ksuperscriptā²k =kā² = k-sized subset of the input vertex neurons in M corresponding to the vertices in Vā². Create MLP Mā² by ablated in M the neurons in Nā². As Vā² is a dominating set, each vertex neighbourhood AND neuron in Mā² is missing a i-input from at least one input vertex neuron in Nā², which in turn ensures that each vertex neighbourhood AND neuron in Mā² has output 0. This in turn ensures that M produces output 1 on input I such that Mā¢(I)=0ā 1=Mā²ā¢(I)01superscriptā²M(I)=0ā 1=M (I)M ( I ) = 0 ā 1 = Mā² ( I ). ā ā : Let Nā² be a kā²ā¤kā²=ksuperscriptā²k ā¤ k =kā² ā² ā¤ kā² = k-sized subset of the set N of neurons in M whose ablation in M creates an MLP Mā² such that Mā¢(I)=0ā Mā²ā¢(I)0superscriptā²M(I)=0ā M (I)M ( I ) = 0 ā Mā² ( I ). As all MLP outputs are stepped to be Boolean, this implies that Mā²ā¢(I)=1superscriptā²1M (I)=1Mā² ( I ) = 1. This can only happen if all vertex neighbourhood NOT neurons output 1, which in turn can happen only if all vertex neighbourhood AND gates output 0. As I=1|V|superscript1I=1^|V|I = 1| V |, this can only happen if for each vertex neighbourhood AND neuron, at least one input vertex neuron previously producing a 1-input to that vertex neighbourhood AND neuron has been ablated in creating Mā². This in turn implies that the kā²k kā² ā² vertices in G corresponding to the elements of Nā² form a dominating set of size kā²ā¤ksuperscriptā²k ā¤ kā² ā² ā¤ k for G. As DS is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLCA is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 44. If MLCA has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Recall from the proof of correctness of the reduction in the proof of Theorem 43 that a given instance of DS has a dominating set of size k if and only if the constructed instance of MLCA has a subset Nā² of size kā²=ksuperscriptā²k =kā² = k of the neurons in given MLP M such that the ablation in M of the neurons in Nā² creates an MLP Mā² such that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ). This implies that, given a polynomial-time c-approximation algorithm A for MLCA for some constant c>00c>0c > 0, we can create a polynomial-time c-approximation algorithm for DS by applying the reduction to the given instance x of DS to construct an instance xā² of MLCA, applying A to xā² to create an approximate solution yā², and then using yā² to create an approximate solution y for x that has the same cost as yā². The result then follows from Chen & Lin 2019, Corollary 2, which implies that if DS has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. ā Note that this theorem also renders MLCA PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. H.1.2 Results for MGCA Membership in in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]ā¢ā[ā0,1#ā¢niā¢n]:[ā³ā]ā¢()ā ā³ā¢():delimited-[]ā³for-alldelimited-[]superscript01#subscriptdelimited-[]ā³ā[S ]\ ā[xā\0,1\^\#n_% in]:[M ](x) (x)ā [ S ā M ] ā [ x ā 0 , 1 # nitalic_i n ] : [ M ā S ] ( x ) ā M ( x ) Theorem 45. If MGCA is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLCA constructed by the reduction in the proof of Theorem 39, the input-connection weight 0 and bias 1 of the input neuron force this neuron to output 1 for both of the possible input vectors (1)1(1)( 1 ) and (0)0(0)( 0 ). Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of MGCA. ā Theorem 46. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCA is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MGCA constructed in the reduction in the proof of Theorem 45, #ā¢niā¢n=#ā¢noā¢uā¢t=Wmax=1#subscript#subscriptsubscript1\#n_in=\#n_out=W_ =1# nitalic_i n = # nitalic_o u t = Wroman_max = 1, cā¢d=55cd=5c d = 5, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 47. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MGCA is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 41 such that each created MLP M is checked to ensure that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for every possible Boolean input vector of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢nā¤2#ā¢ntā¢oā¢tsuperscript2#subscriptsuperscript2#subscript2^\#n_inā¤ 2^\#n_tot2# nitalic_i n ā¤ 2# nitalic_t o t, the above is a fixed-parameter tractable algorithm for MGCA relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 48. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MGCA is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 47 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 46ā48 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MGCA relative to many subsets of the parameters listed in Table 8. Let us now consider the polynomial-time cost approximability of MGCA. As MGCA is a minimization problem, we cannot do this using reductions from a maximization problem like Clique. Hence we will instead use a reduction from another minimization problem, namely DS. Theorem 49. If MGCA is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLCA constructed by the reduction in the proof of Theorem 43, the input-connection weight 0 and bias 1 of the vertex input neurons force each such neuron to output 1 for input value 0 or 1. Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of MGCA. ā Theorem 50. If MGCA has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. As the reduction in the proof of Theorem 49 is essentially the same as the reduction in the proof of Theorem 43, the result follows by the same reasoning as given in the proof of Theorem 44. ā Note that this theorem also renders MGCA PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. H.2 Results for Minimal Circuit Clamping Towards proving NP-completeness, we first prove membership and then follow up with hardness. Membership in NP can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]:ā³ā¢()ā ā³ā¢():delimited-[]ā³subscriptā³ā[S ]:M_S(x)% (x)ā [ S ā M ] : Mcaligraphic_S ( x ) ā M ( x ) The following hardness results are notable for holding when the given MLP M has only one hidden layer. H.2.1 Results for MLCC Theorem 51. If MLCC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from Clique to MLCC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of Clique, construct the following instance āØM,I,vā¢aā¢l,kā²ā©superscriptā² M,I,val,k āØ M , I , v a l , kā² ā© of MLCC: Let M be an MLP based on #ā¢ntā¢oā¢t=|V|+|E|+1#subscript1\#n_tot=|V|+|E|+1# nitalic_t o t = | V | + | E | + 1 neurons spread across three layers: 1. Input vertex layer: The vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V | (all with bias ā22-2- 2). 2. Hidden edge layer: The edge neurons nā¢e1,nā¢e2,ā¦ā¢nā¢e|E|subscript1subscript2ā¦subscriptne_1,ne_2,ā¦ ne_|E|n e1 , n e2 , ā¦ n e| E | (all with bias ā11-1- 1). 3. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t (bias ā(kā¢(kā1)/2ā1)121-(k(k-1)/2-1)- ( k ( k - 1 ) / 2 - 1 )). Note that this MLP has only one hidden layer. The non-zero weight connections between adjacent layers are as follows: ā¢ Each vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge neuron nā¢eisubscriptne_in eitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=0#ā¢niā¢n)I=0^\#n_in)I = 0# nitalic_i n ), vā¢aā¢l=11val=1v a l = 1, and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLCC can be created in time polynomial in the size of the given instance of Clique. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 nā¢v1ā¢(0),nā¢v2ā¢(0),ā¦ā¢nā¢v|V|ā¢(0)subscript10subscript20ā¦subscript0nv_1(0),nv_2(0),ā¦ nv_|V|(0)n v1 ( 0 ) , n v2 ( 0 ) , ā¦ n v| V | ( 0 ) 2 nā¢e1ā¢(0),nā¢e2ā¢(0),ā¦ā¢nā¢e|E|ā¢(0)subscript10subscript20ā¦subscript0ne_1(0),ne_2(0),ā¦ ne_|E|(0)n e1 ( 0 ) , n e2 ( 0 ) , ā¦ n e| E | ( 0 ) 3 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of Clique is āYesā if and only if the answer for the constructed instance of MLCC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a clique in G of size kā²ā„ksuperscriptā²k ā„ kā² ā² ā„ k and Nā² be the kā²ā„kā²=ksuperscriptā²k ā„ k =kā² ā² ā„ kā² = k-sized subset of the input vertex neurons corresponding to the vertices in Vā². Let Mā² be the version of M in which all neurons in Nā² are clamped to value vā¢aā¢l=11val=1v a l = 1. As Vā² is a clique of size kā²k kā² ā², exactly kā²ā¢(kā²ā1)/2ā„kā¢(kā1)/2superscriptā²1212k (k -1)/2ā„ k(k-1)/2kā² ā² ( kā² ā² - 1 ) / 2 ā„ k ( k - 1 ) / 2 edge neurons in Mā² receive the requisite inputs of 1 on both of their endpoints from the vertex neurons in Nā². This in turn ensures the output neuron produces output 1. Hence, Mā¢(I)=0ā 1=Mā²ā¢(I)01superscriptā²M(I)=0ā 1=M (I)M ( I ) = 0 ā 1 = Mā² ( I ). ā ā : Let Nā² be a subset of N of size at most kā²=ksuperscriptā²k =kā² = k such that for the MLP Mā² induced by clamping all neurons in Nā² to value vā¢aā¢l=11val=1v a l = 1, Mā¢(I)ā Mā¢(Iā²)superscriptā²M(I)ā M(I )M ( I ) ā M ( Iā² ). As Mā¢(I)=00M(I)=0M ( I ) = 0 and circuit outputs are stepped to be Boolean, Mā²ā¢(I)=1superscriptā²1M (I)=1Mā² ( I ) = 1. Given the bias of the output neuron, this can only occur if at least kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons in Mā² have output 1 on input I, which requires that each of these neurons receives 1 from both of its endpoint vertex neurons. As I=0#ā¢niā¢nsuperscript0#subscriptI=0^\#n_inI = 0# nitalic_i n, these 1-inputs could only have come from the clamped vertex neurons in Nā²; moreover, there must be exactly k such neurons. This means that the vertices in G corresponding to the vertex neurons in Nā² must form a clique of size k in G. As Clique is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLCC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 52. If āØcā¢d,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscriptsubscriptsubscript cd,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MLCC constructed in the reduction in the proof of Theorem 51, #ā¢noā¢uā¢t=Wmax=1#subscriptsubscript1\#n_out=W_ =1# nitalic_o u t = Wroman_max = 1, cā¢d=33cd=3c d = 3, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 53. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCC is fixed-parameter tractable. Proof. Consider the algorithm that generates every possible subset Nā² of size at most k of the neurons N in MLP M and for each such subset, creates the MLP Mā² induced from M by clamping the neurons in Nā² to vā¢aā¢lvalv a l and checks if Mā²ā¢(I)ā Mā¢(I)superscriptā²M (I)ā M(I)Mā² ( I ) ā M ( I ). If such a subset is found, return āYesā; otherwise, return āNoā. The number of possible subsets Nā² is at most kĆ#ā¢ntā¢oā¢tkā¤#ā¢ntā¢oā¢tĆ#ā¢ntā¢oā¢t#ā¢ntā¢oā¢t#superscriptsubscript#subscript#superscriptsubscript#subscriptkĆ\#n_tot^kā¤\#n_totĆ\#n_tot^\#n_totk Ć # nitalic_t o titalic_k ā¤ # nitalic_t o t Ć # nitalic_t o t# nitalic_t o t. As any such Mā² can be generated from M and Mā² can be run on I in time polynomial in the size of the given instance of MLCC, the above is a fixed-parameter tractable algorithm for MLCC relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 54. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLCC is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 53 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 52ā54 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MLCC relative to many subsets of the parameters listed in Table 8. Let us now consider the polynomial-time cost approximability of MLCC. As MLCC is a minimization problem, we cannot do this using reductions from a maximization problem like Clique. Hence we will instead use a reduction from another minimization problem, namely DS. Theorem 55. If MLCC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from DS to MLCC. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of DS, construct the following instance āØM,I,vā¢aā¢l,kā²ā©superscriptā² M,I,val,k āØ M , I , v a l , kā² ā© of MLCC: Let M be an MLP based on #ā¢ntā¢oā¢t,g=3ā¢|V|+1#subscript31\#n_tot,g=3|V|+1# nitalic_t o t , g = 3 | V | + 1 neurons spread across four layers: 1. Input layer: The input vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V |, all of which have bias 1. 2. Hidden vertex neighbourhood layer I: The vertex neighbourhood AND neurons nā¢vā¢nā¢A1,nā¢vā¢nā¢A2,ā¦ā¢nā¢vā¢nā¢A|V|subscript1subscript2ā¦subscriptnvnA_1,nvnA_2,ā¦ nvnA_|V|n v n A1 , n v n A2 , ā¦ n v n A| V |, where nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i is an x-way AND ReLU gates such that x=|NCā¢(vi)|subscriptsubscriptx=|N_C(v_i)|x = | Nitalic_C ( vitalic_i ) |. 3. Hidden vertex neighbourhood layer I: The vertex neighbourhood NOT neurons nā¢vā¢nā¢N1,nā¢vā¢nā¢N2,ā¦ā¢nā¢vā¢nā¢N|V|subscript1subscript2ā¦subscriptnvnN_1,nvnN_2,ā¦ nvnN_|V|n v n N1 , n v n N2 , ā¦ n v n N| V |, all of which are NOT ReLU gates. 4. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is a |V||V|| V |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its input line with weight 0 and to each vertex neighbourhood AND neuron nā¢vā¢nā¢AjsubscriptnvnA_jn v n Aitalic_j such that viāNCā¢(vj)subscriptsubscriptsubscriptv_iā N_C(v_j)vitalic_i ā Nitalic_C ( vitalic_j ) with weight 1. ā¢ Each vertex neighbourhood AND neuron nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its corresponding vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i with weight 1. ā¢ Each vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I be the |V||V|| V |-length one-vector, vā¢aā¢l=00val=0v a l = 0, and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLCC can be created in time polynomial in the size of the given instance of DS, Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 nā¢vā¢N1ā¢(1),nā¢vā¢N2ā¢(1),ā¦ā¢nā¢vā¢N|V|ā¢(1)subscript11subscript21ā¦subscript1nvN_1(1),nvN_2(1),ā¦ nvN_|V|(1)n v N1 ( 1 ) , n v N2 ( 1 ) , ā¦ n v N| V | ( 1 ) 2 nā¢vā¢nā¢A1ā¢(1),nā¢vā¢nā¢A2ā¢(1),ā¦ā¢nā¢vā¢nā¢A|V|ā¢(1)subscript11subscript21ā¦subscript1nvnA_1(1),nvnA_2(1),ā¦ nvnA_|V|(1)n v n A1 ( 1 ) , n v n A2 ( 1 ) , ā¦ n v n A| V | ( 1 ) 3 nā¢vā¢nā¢N1ā¢(0),nā¢vā¢nā¢N2ā¢(0),ā¦ā¢nā¢vā¢nā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0nvnN_1(0),nvnN_2(0),ā¦ nvnN_|V|(0)n v n N1 ( 0 ) , n v n N2 ( 0 ) , ā¦ n v n N| V | ( 0 ) 4 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of DS is āYesā if and only if the answer for the constructed instance of MLCC is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a dominating set in G of size k and Nā² be the kā²=ksuperscriptā²k =kā² = k-sized subset of the input vertex neurons in M corresponding to the vertices in Vā². Create MLP Mā² by clamping in M the neurons in Nā² to vā¢aā¢l=00val=0v a l = 0. As Vā² is a dominating set, each vertex neighbourhood AND neuron in Mā² is now missing a i-input from at least one input vertex neuron in Nā², which in turn ensures that each vertex neighbourhood AND neuron in Mā² has output 0. This in turn ensures that M produces output 1 on input I such that Mā¢(I)=0ā 1=Mā²ā¢(I)01superscriptā²M(I)=0ā 1=M (I)M ( I ) = 0 ā 1 = Mā² ( I ).. ā ā : Let Nā² be a kā²ā¤kā²=ksuperscriptā²k ā¤ k =kā² ā² ā¤ kā² = k-sized subset of the set N of neurons in M whose clamping to vā¢aā¢l=00val=0v a l = 0 in M creates an MLP Mā² such that Mā¢(I)=0ā Mā²ā¢(I)0superscriptā²M(I)=0ā M (I)M ( I ) = 0 ā Mā² ( I ). As all MLP outputs are stepped to be Boolean, this implies that Mā²ā¢(I)=1superscriptā²1M (I)=1Mā² ( I ) = 1. This can only happen if all vertex neighbourhood NOT neurons output 1, which in turn can happen only if all vertex neighbourhood AND gates output 0. As I=1|V|superscript1I=1^|V|I = 1| V |, this can only happen if for each vertex neighbourhood AND neuron, at least one input vertex neuron previously producing a 1-input to that vertex neighbourhood AND neuron has been clamped to 0 in creating Mā². This in turn implies that the kā²k kā² ā² vertices in G corresponding to the elements of Nā² form a dominating set of size kā²ā¤ksuperscriptā²k ā¤ kā² ā² ā¤ k for G. As DS is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLCC is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 56. If MLCC has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Recall from the proof of correctness of the reduction in the proof of Theorem 55 that a given instance of DS has a dominating set of size k if and only if the constructed instance of MLCC has a subset Nā² of size kā²=ksuperscriptā²k =kā² = k of the neurons in given MLP M such that the clamping to vā¢aā¢l=00val=0v a l = 0 in M of the neurons in Nā² creates an MLP Mā² such that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ). This implies that, given a polynomial-time c-approximation algorithm A for MLCC for some constant c>00c>0c > 0, we can create a polynomial-time c-approximation algorithm for DS by applying the reduction to the given instance x of DS to construct an instance xā² of MLCC, applying A to xā² to create an approximate solution yā², and then using yā² to create an approximate solution y for x that has the same cost as yā². The result then follows from Chen & Lin 2019, Corollary 2, which implies that if DS has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. ā Note that this theorem also renders MLCC PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. H.2.2 Results for MGCC Membership in in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]ā¢ā[ā0,1#ā¢niā¢n]:ā³ā¢()ā ā³ā¢():delimited-[]ā³for-alldelimited-[]superscript01#subscriptsubscriptā³ā[S ]\ ā[xā\0,1\^\#n_% in]:M_S(x) (x)ā [ S ā M ] ā [ x ā 0 , 1 # nitalic_i n ] : Mcaligraphic_S ( x ) ā M ( x ) Theorem 57. If MGCC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLCC constructed by the reduction in the proof of Theorem 51, the biases of ā22-2- 2 in the input vertex neurons force these neurons to map any given Boolean input vector onto 0#ā¢niā¢nsuperscript0#subscript0^\#n_in0# nitalic_i n. Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of MGCC. ā Theorem 58. If āØcā¢d,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscriptsubscriptsubscript cd,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCC is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MGCC constructed in the reduction in the proof of Theorem 57, #ā¢noā¢uā¢t=Wmax=1#subscriptsubscript1\#n_out=W_ =1# nitalic_o u t = Wroman_max = 1, cā¢d=33cd=3c d = 3, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 59. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCC is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 53 such that each created MLP M is checked to ensure that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for every possible Boolean input vector of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢nā¤2#ā¢ntā¢oā¢tsuperscript2#subscriptsuperscript2#subscript2^\#n_inā¤ 2^\#n_tot2# nitalic_i n ā¤ 2# nitalic_t o t, the above is a fixed-parameter tractable algorithm for MGCC relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 60. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MGCC is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 59 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Observe that the results in Theorems 58ā60 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MGCC relative to many subsets of the parameters listed in Table 8. Let us now consider the polynomial-time cost approximability of MGCC. As MGCC is a minimization problem, we cannot do this using reductions from a maximization problem like Clique. Hence we will instead use a reduction from another minimization problem, namely DS. Theorem 61. If MGCC is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLCC constructed by the reduction in the proof of Theorem 55, the input-connection weight 0 and bias 1 of the vertex input neurons force each such neuron to output 1 for input value 0 or 1. Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of MGCC. ā Theorem 62. If MGCC has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. As the reduction in the proof of Theorem 61 is essentially the same as the reduction in the proof of Theorem 55, the result follows by the same reasoning as given in the proof of Theorem 56. ā Note that this theorem also renders MGCC PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Appendix I Circuit Patching Problem Minimum local circuit patching (MLCP) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, Boolean input vectors x and y of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤(#ā¢ntā¢oā¢tā(#ā¢niā¢n+#ā¢noā¢uā¢t))1#subscript#subscript#subscript1ā¤ kā¤(\#n_tot-(\#n_in+\#n_out))1 ā¤ k ā¤ ( # nitalic_t o t - ( # nitalic_i n + # nitalic_o u t ) ). Question: Is there a subset C, |C|ā¤k|C|ā¤ k| C | ā¤ k, of the internal neurons in M such that for the MLP Mā² created when M is y-patched wrt C, i.e., Mā² is created when M/CM/CM / C is patched with activations from Mā¢(x)M(x)M ( x ) and C is patched with activations from Mā¢(y)M(y)M ( y ), Mā²ā¢(x)=Mā¢(y)superscriptā²M (x)=M(y)Mā² ( x ) = M ( y )? Minimum Global Circuit Patching (MGCP) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, Boolean input vector y of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤(#ā¢ntā¢oā¢tā(#ā¢niā¢n+#ā¢noā¢uā¢t))1#subscript#subscript#subscript1ā¤ kā¤(\#n_tot-(\#n_in+\#n_out))1 ā¤ k ā¤ ( # nitalic_t o t - ( # nitalic_i n + # nitalic_o u t ) ). Question: Is there a subset C, |C|ā¤k|C|ā¤ k| C | ā¤ k, of the internal neurons in M such that, for all possible input vectors x, for the MLP Mā² created when M is y-patched wrt C, i.e., Mā² is created when M/CM/CM / C is patched with activations from Mā¢(x)M(x)M ( x ) and C is patched with activations from Mā¢(y)M(y)M ( y ), Mā²ā¢(x)=Mā¢(y)superscriptā²M (x)=M(y)Mā² ( x ) = M ( y )? Following BarcelĆ³ et al. 2020, page 4, all neurons in M use the ReLU activation function and the output x of each output neuron is stepped as necessary to be Boolean, i.e, sā¢tā¢eā¢pā¢(x)=00step(x)=0s t e p ( x ) = 0 if xā¤00xā¤ 0x ā¤ 0 and is 1111 otherwise. For a graph G=(V,E)G=(V,E)G = ( V , E ), we shall assume an ordering on the vertices and edges in V and E, respectively. For each vertex vāVvā Vv ā V, let the complete neighbourhood NCā¢(v)subscriptN_C(v)Nitalic_C ( v ) of v be the set composed of v and the set of all vertices in G that are adjacent to v by a single edge, i.e., vāŖu|uāVā¢andā¢(u,v)āEconditional-setanduvEvāŖ\u~|~u~ā V~ and~(u,v)ā E\v āŖ u | u ā V and ( u , v ) ā E . We will prove various classical and parameterized results for MLCP and MGCP using reductions from Dominating Set. The parameterized results are proved relative to the parameters in Table 9. Our reductions (Theorems 63 and 67) uses specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Table 9: Parameters for the minimum circuit patching problem. Parameter Description cā¢dcdc d # layers in given MLP cā¢wcwc w max # neurons in layer in given MLP #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP BmaxsubscriptB_ Broman_max max neuron bias in given MLP WmaxsubscriptW_ Wroman_max max connection weight in given MLP k Size of requested patching-subset of M I.1 Results for MLCP Towards proving NP-completeness, we first prove membership and then follow up with hardness. Membership in NP can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]:ā³ā¢()=ā³ā¢():delimited-[]ā³subscriptā³ā[S ]:M_S(x)% =M(y)ā [ S ā M ] : Mcaligraphic_S ( x ) = M ( y ) Theorem 63. If MLCP is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from DS to MLCP adapted from the reduction from DS to MSR in Theorem 103. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of DS, construct the following instance āØM,x,y,kā²ā©superscriptā² M,x,y,k āØ M , x , y , kā² ā© of MLCP: Let M be an MLP based on #ā¢ntā¢oā¢t=4ā¢|V|+1#subscript41\#n_tot=4|V|+1# nitalic_t o t = 4 | V | + 1 neurons spread across five layers: 1. Input layer: The input vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V |, all of which have bias 0. 2. Hidden vertex layer: The hidden vertex neurons nā¢hā¢v1,nā¢hā¢v2,ā¦ā¢nā¢hā¢v|V|āsubscript1āsubscript2ā¦āsubscriptnhv_1,nhv_2,ā¦ nhv_|V|n h v1 , n h v2 , ā¦ n h v| V |, all of which are identity ReLU gates with bias 0. 3. Hidden vertex neighbourhood layer I: The vertex neighbourhood AND neurons nā¢vā¢nā¢A1,nā¢vā¢nā¢A2,ā¦ā¢nā¢vā¢nā¢A|V|subscript1subscript2ā¦subscriptnvnA_1,nvnA_2,ā¦ nvnA_|V|n v n A1 , n v n A2 , ā¦ n v n A| V |, where nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i is an x-way AND ReLU gates such that x=|NCā¢(vi)|subscriptsubscriptx=|N_C(v_i)|x = | Nitalic_C ( vitalic_i ) |. 4. Hidden vertex neighbourhood layer I: The vertex neighbourhood NOT neurons nā¢vā¢nā¢N1,nā¢vā¢nā¢N2,ā¦ā¢nā¢vā¢nā¢N|V|subscript1subscript2ā¦subscriptnvnN_1,nvnN_2,ā¦ nvnN_|V|n v n N1 , n v n N2 , ā¦ n v n N| V |, all of which are NOT ReLU gates. 5. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is a |V||V|| V |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its associated hidden vertex neuron nā¢hā¢viāsubscriptnhv_in h vitalic_i with weight 1. ā¢ Each hidden vertex neuron nā¢hā¢viāsubscriptnhv_in h vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each vertex neighbourhood AND neuron nā¢vā¢nā¢AjsubscriptnvnA_jn v n Aitalic_j such that viāNCā¢(vj)subscriptsubscriptsubscriptv_iā N_C(v_j)vitalic_i ā Nitalic_C ( vitalic_j ) with weight 1. ā¢ Each vertex neighbourhood AND neuron nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its corresponding vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i with weight 1. ā¢ Each vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let x and y be the |V||V|| V |-length one- and zero-vectors and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MLCP can be created in time polynomial in the size of the given instance of DS, Moreover, the output behaviour of the neurons in M from the presentation of input x until the output is generated is timestep neurons (outputs) 0 ā 1 nā¢v1ā¢(1),nā¢v2ā¢(1),ā¦ā¢nā¢v|V|ā¢(1)subscript11subscript21ā¦subscript1nv_1(1),nv_2(1),ā¦ nv_|V|(1)n v1 ( 1 ) , n v2 ( 1 ) , ā¦ n v| V | ( 1 ) 2 nā¢hā¢v1ā¢(1),nā¢hā¢v2ā¢(1),ā¦ā¢nā¢hā¢v|V|ā¢(1)āsubscript11āsubscript21ā¦āsubscript1nhv_1(1),nhv_2(1),ā¦ nhv_|V|(1)n h v1 ( 1 ) , n h v2 ( 1 ) , ā¦ n h v| V | ( 1 ) 3 nā¢vā¢nā¢A1ā¢(1),nā¢vā¢nā¢A2ā¢(1),ā¦ā¢nā¢vā¢nā¢A|V|ā¢(1)subscript11subscript21ā¦subscript1nvnA_1(1),nvnA_2(1),ā¦ nvnA_|V|(1)n v n A1 ( 1 ) , n v n A2 ( 1 ) , ā¦ n v n A| V | ( 1 ) 4 nā¢vā¢nā¢N1ā¢(0),nā¢vā¢nā¢N2ā¢(0),ā¦ā¢nā¢vā¢nā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0nvnN_1(0),nvnN_2(0),ā¦ nvnN_|V|(0)n v n N1 ( 0 ) , n v n N2 ( 0 ) , ā¦ n v n N| V | ( 0 ) 5 noā¢uā¢tā¢(0)subscript0n_out(0)nitalic_o u t ( 0 ) and the output behaviour of the neurons in M from the presentation of input y until the output is generated is timestep neurons (outputs) 0 ā 1 nā¢v1ā¢(0),nā¢v2ā¢(0),ā¦ā¢nā¢v|V|ā¢(0)subscript10subscript20ā¦subscript0nv_1(0),nv_2(0),ā¦ nv_|V|(0)n v1 ( 0 ) , n v2 ( 0 ) , ā¦ n v| V | ( 0 ) 2 nā¢hā¢v1ā¢(0),nā¢hā¢v2ā¢(0),ā¦ā¢nā¢hā¢v|V|ā¢(0)āsubscript10āsubscript20ā¦āsubscript0nhv_1(0),nhv_2(0),ā¦ nhv_|V|(0)n h v1 ( 0 ) , n h v2 ( 0 ) , ā¦ n h v| V | ( 0 ) 3 nā¢vā¢nā¢A1ā¢(0),nā¢vā¢nā¢A2ā¢(0),ā¦ā¢nā¢vā¢nā¢A|V|ā¢(0)subscript10subscript20ā¦subscript0nvnA_1(0),nvnA_2(0),ā¦ nvnA_|V|(0)n v n A1 ( 0 ) , n v n A2 ( 0 ) , ā¦ n v n A| V | ( 0 ) 4 nā¢vā¢nā¢N1ā¢(1),nā¢vā¢nā¢N2ā¢(1),ā¦ā¢nā¢vā¢nā¢N|V|ā¢(1)subscript11subscript21ā¦subscript1nvnN_1(1),nvnN_2(1),ā¦ nvnN_|V|(1)n v n N1 ( 1 ) , n v n N2 ( 1 ) , ā¦ n v n N| V | ( 1 ) 5 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of DS is āYesā if and only if the answer for the constructed instance of MLCP is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a dominating set in G of size k and C be the kā²=ksuperscriptā²k =kā² = k-sized subset of the hidden vertex neurons in M corresponding to the vertices in Vā². As Vā² is a dominating set, each vertex neighbourhood AND neuron receives input 0 from at least one hidden vertex neuron in C when M is y-patched wrt C. This ensures that each vertex neighbourhood AND neuron has output 0, which in turn ensures that each vertex neighbourhood NOT neuron has output 1 and for Mā² created by y-patching M wrt C, Mā²ā¢(x)=Mā¢(y)=1superscriptā²1M (x)=M(y)=1Mā² ( x ) = M ( y ) = 1. ā ā : Let C be a kā²=ksuperscriptā²k =kā² = k-sized subset of the internal neurons of M such that when Mā² is created by y-patching M wrt C, Mā²ā¢(x)=Mā¢(y)=1superscriptā²1M (x)=M(y)=1Mā² ( x ) = M ( y ) = 1. The output of Mā² on X can be 1 (and hence equal to the output of M on y) only if all vertex neighbourhood NOT neurons output 1, which in turn can happen only if all vertex neighbourhood AND gates output 0. However, as all elements of x have value 1, this means that each vertex neighbourhood AND neuron must be connected to at least one patched hidden vertex neuron (all of which have output 0 courtesy of y), which in turn implies that the kā²=ksuperscriptā²k =kā² = k vertices in G corresponding to the patched hidden vertex neurons in C form a dominating set of size k for G. As DS is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MLCP is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 64. If āØcā¢d,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscriptsubscriptsubscript cd,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCP is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MLCP constructed in the reduction in the proof of Theorem 63, #ā¢noā¢uā¢t=Wmax=1#subscriptsubscript1\#n_out=W_ =1# nitalic_o u t = Wroman_max = 1, cā¢d=44cd=4c d = 4, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of DS. The result then follows from the facts that āØkā©delimited-āØā© k āØ k ā©-DS is Wā¢[2]delimited-[]2W[2]W [ 2 ]-hard (Downey & Fellows, 1999) and Wā¢[1]āWā¢[2]delimited-[]1delimited-[]2W[1] W[2]W [ 1 ] ā W [ 2 ]. ā Theorem 65. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCP is fixed-parameter tractable. Proof. Consider the algorithm that generates all possible subset C of the internal neurons in M and for each such subset, checks if Mā² created by y-patching M wrt C is such that Mā²ā¢(x)=Mā¢(y)superscriptā²M (x)=M(y)Mā² ( x ) = M ( y ). If such a C is found, return āYesā; otherwise, return āNoā. The number of possible subsets C is at most 2(#ntā¢oā¢t2^(\#n_tot2( # nitalic_t o t. Given this, as M can be patched relative to C and run on x and y in time polynomial in the size of the given instance of MLCP, the above is a fixed-parameter tractable algorithm for MLCP relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Observe that the results in Theorems 64 and 65 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MLCP relative to many subset of the parameters listed in Table 9. Let us now consider the polynomial-time cost approximability of MLCP. Theorem 66. If MLCP has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Recall from the proof of correctness of the reduction in the proof of Theorem 63 that a given instance of DS has a dominating set of size k if and only if the constructed instance of Mā¢Lā¢Cā¢PMLCPM L C P has a subset C of the internal neurons in M of size kā²=ksuperscriptā²k =kā² = k such that for the MLP Mā² created from M by y-patching M wrt C, Mā²ā¢(x)=Mā¢(y)superscriptā²M (x)=M(y)Mā² ( x ) = M ( y ) This implies that, given a polynomial-time c-approximation algorithm A for MLCP for some constant c>00c>0c > 0, we can create a polynomial-time c-approximation algorithm for DS by applying the reduction to the given instance I of DS to construct an instance Iā² of MLCP, applying A to Iā² to create an approximate solution Sā², and then using Sā² to create an approximate solution S for I that has the same cost as Sā². The result then follows from Chen & Lin 2019, Corollary 2, which implies that if DS has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. ā Note that this theorem also renders MLCP PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. I.2 Results for MGCP Membership in in Ī£2psubscriptsuperscriptĪ£2 ^p_2Ī£italic_p2 can be proven via the definition of the polynomial hierarchy and the following alternating quantifier formula: ā[āā³]ā¢ā[ā0,1#ā¢niā¢n]:ā³ā¢()=ā³ā¢():delimited-[]ā³for-alldelimited-[]superscript01#subscriptsubscriptā³ā[S ]\ ā[xā\0,1\^\#n_% in]:M_S(x)=M(y)ā [ S ā M ] ā [ x ā 0 , 1 # nitalic_i n ] : Mcaligraphic_S ( x ) = M ( y ) Theorem 67. If MGCP is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Modify the reduction in the proof of Theorem 63 such that each hidden vertex neuron has input weight 0 and bias 1; this will force all hidden vertex neurons to output 1 for all input vectors x instead of just when x is the all-one vector. Hence, with slight modifications to the proof of reduction correctness for the reduction in the proof of Theorem 63, this modified reduction establishes the Nā¢PNPN P-hardness of MGCP. ā Theorem 68. If āØcā¢d,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscriptsubscriptsubscript cd,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCP is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MGCP constructed in the reduction in the proof of Theorem 67, #ā¢noā¢uā¢t=Wmax=1#subscriptsubscript1\#n_out=W_ =1# nitalic_o u t = Wroman_max = 1, cā¢d=44cd=4c d = 4, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of DS. The result then follows from the facts that āØkā©delimited-āØā© k āØ k ā©-DS is Wā¢[2]delimited-[]2W[2]W [ 2 ]-hard (Downey & Fellows, 1999) and Wā¢[1]āWā¢[2]delimited-[]1delimited-[]2W[1] W[2]W [ 1 ] ā W [ 2 ]. ā Theorem 69. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MGCP is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 65 such that each circuit Mā² created by y-patching M is checked to ensure that Mā²ā¢(x)=Mā¢(y)superscriptā²M (x)=M(y)Mā² ( x ) = M ( y ) for every possible Boolean input vector x of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢n<2#ā¢ntā¢oā¢tsuperscript2#subscriptsuperscript2#subscript2^\#n_in<2^\#n_tot2# nitalic_i n < 2# nitalic_t o t, the above is a fixed-parameter tractable algorithm for MGCP relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Observe that the results in Theorems 68 and 69 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MGCP relative to many subset of the parameters listed in Table 9. Let us now consider the polynomial-time cost approximability of MGCP. Theorem 70. If MGCP has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. As the reduction in the proof of Theorem 67 is essentially the same as the reduction in the proof of Theorem 63, the result follows by the same reasoning as given in the proof of Theorem 66. ā Note that this theorem also renders MGCP PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Appendix J Quasi-Minimal Circuit Patching Problem Quasi-Minimal Circuit Patching (QMCP) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, an input vector yy and a set XX of input vectors of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. Output: a subset CC in ā³MM and a neuron vāv ā C, such that for the ā³āsuperscriptā³M^*Mā induced by patching CC with activations from ā³ā¢()ā³M(y)M ( y ) and ā³āā³M ā C with activations from ā³ā¢()ā³M(x)M ( x ), āā:ā³āā¢()=ā³ā¢():subscriptfor-allsuperscriptā³ _x :M^*(x)=M(% y)āx ā X : Mā ( x ) = M ( y ), and for ā³ā²M Mā² induced by patching identically except for vāv ā C, āā:ā³ā²ā¢()ā ā³ā¢():subscriptsuperscriptā³ā²ā³ _x :M (x)ā % M(y)āx ā X : Mā² ( x ) ā M ( y ). Theorem 71. QMCP is in PTIME (i.e., polynomial-time tractable). Proof. Consider the following algorithm for QMCP. Build a sequence of MLPs by taking ā³MM with all neurons labeled 0, and generating subsequent ā³isubscriptā³M_iMitalic_i in the sequence by labeling an additional neuron with 1 each time (this choice can be based on any heuristic strategy, for instance, one based on gradients). The first MLP, ā³1subscriptā³1M_1M1, obtained by patching all neurons labeled 1 (i.e., none) is such that ā³1ā¢(x)ā ā³ā¢(y)subscriptā³1ā³M_1(x) (y)M1 ( x ) ā M ( y ), and the last ā³nsubscriptā³M_nMitalic_n is guaranteed to give ā³nā¢(x)=ā³ā¢(y)subscriptā³M_n(x)=M(y)Mitalic_n ( x ) = M ( y ) because all neurons are patched. Label the first MLP NO, and the last YES. Perform a variant of binary search on the sequence as follows. Evaluate the ā³subscriptā³M_iMcaligraphic_i halfway between NO and YES while patching all its neurons labeled 1. If it satisfies the condition, label it YES, and repeat the same strategy with the sequence starting from the first ā³isubscriptā³M_iMitalic_i until the YES just labeled. If it does not satisfy the condition, label it NO and repeat the same strategy with the sequence starting from the NO just labeled until the YES at the end of the original sequence. This iterative procedure halves the sequence each time. Halt when you find two adjacent āØNO,YESā©NOYES NO, YES āØ NO , YES ā© patched networks (guaranteed to exist), and return the patched neuron set of the YES network V and the single neuron difference between YES and NO (the breaking point), vāVvā Vv ā V. The complexity of this algorithm is roughly Oā¢(nā¢logā”n)O(n n)O ( n log n ). ā Appendix K Circuit Robustness Problem Definition 17. Given an MLP M, a subset H of the elements in M, an integer kā¤|H|kā¤|H|k ā¤ | H |, and an input I to M, M is k-robust relative to H for I if for each subset Hā²āHsuperscriptā²H Hā² ā H, |Hā²|ā¤ksuperscriptā²|H |ā¤ k| Hā² | ā¤ k, Mā¢(I)=(M/Hā²)ā¢(I)superscriptā²M(I)=(M/H )(I)M ( I ) = ( M / Hā² ) ( I ). Maximum Local Circuit Robustness (MLCR) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a subset H of the neurons in M, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤|H|11ā¤ kā¤|H|1 ā¤ k ā¤ | H |. Question: Is M k-robust relative to H for I? Restricted Maximum Local Circuit Robustness (MLCRā) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤|M|11ā¤ kā¤|M|1 ā¤ k ā¤ | M |. Question: Is M k-robust relative to H=MH=MH = M for I? Maximum Global Circuit Robustness (MGCR) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a subset H of the neurons in M, and a positive integer k such that 1ā¤kā¤|H|11ā¤ kā¤|H|1 ā¤ k ā¤ | H |. Question: Is M k-robust relative to H for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n? Restricted Maximum Global Circuit Robustness (MGCRā) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, and a positive integer k such that 1ā¤kā¤|M|11ā¤ kā¤|M|1 ā¤ k ā¤ | M |. Question: Is M k-robust relative to H=MH=MH = M for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n? Following BarcelĆ³ et al. 2020, page 4, all neurons in M use the ReLU activation function and the output x of each output neuron is stepped as necessary to be Boolean, i.e, sā¢tā¢eā¢pā¢(x)=00step(x)=0s t e p ( x ) = 0 if xā¤00xā¤ 0x ā¤ 0 and is 1111 otherwise. We will use previous results for Minimum Local/Global Circuit Ablation, Clique and Vertex Cover to prove our results for the problems above. For a graph G=(V,E)G=(V,E)G = ( V , E ), we shall assume an ordering on the vertices and edges in V and E, respectively. We will prove various classical and parameterized results for MLCR, MLCRā, MGCR, and MGCRā. The parameterized results are proved relative to the parameters in Table 10. Lemmas 1 and 2 will be useful in deriving additional parameterized results from proved ones. Table 10: Parameters for the minimum circuit robustness problem. Parameter Description Appl. cā¢dcdc d # layers in given MLP All cā¢wcwc w max # neurons in layer in given MLP All #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP All #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP All #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP All BmaxsubscriptB_ Broman_max max neuron bias in given MLP All WmaxsubscriptW_ Wroman_max max connection weight in given MLP All k Requested level of robustness All |H||H|| H | Size of investigated region M\L,G\CR Several of our proofs involve problems that are hard for cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ] and cā¢oā¢Nā¢PcoNPc o N P, the complement classes of Wā¢[1]delimited-[]1W[1]W [ 1 ] and Nā¢PNPN P (Garey & Johnson, 1979, Section 7.1). Given a decision problem X let co-X be the complement problem in which all instances answers are switched. Observation 3. MLCRā is the complement of MLCA. Proof. Let X=āØM,I,kā©X= M,I,k = āØ M , I , k ā© be a shared input of MLCRā and MLCA. If the the answer to MLCA on input X is āYesā, this means that there is a subset H of size ā¤kabsentā¤ kā¤ k of the neurons of M that can be ablated such that Mā¢(I)ā (M/H)ā¢(I)M(I)ā (M/H)(I)M ( I ) ā ( M / H ) ( I ). This implies that the answer to MLCRā on input X is āNoā. Conversely, if the answer to MLCA on input X is āNoā, this means that there is no subset H of size ā¤kabsentā¤ kā¤ k of the neurons of M that can be ablated such that Mā¢(I)ā (M/H)ā¢(I)M(I)ā (M/H)(I)M ( I ) ā ( M / H ) ( I ). This implies that the answer to MLCRā on input X is āYesā. The observation follows from the definition of complement problem. ā The following lemmas will be of use in the derivation and interpretation of results involving complement problems and classes. Lemma 5. (Garey & Johnson, 1979, Section 7.1) Given a decision problem X, if X is Nā¢PNPN P-hard then co-X is cā¢oā¢Nā¢PcoNPc o N P-hard. Lemma 6. Given a decision problem X, if X is cā¢oā¢Nā¢PcoNPc o N P-hard and X is polynomial-time solvable then P=Nā¢P=NPP = N P. Proof. Suppose decision problem X is cā¢oā¢Nā¢PcoNPc o N P-hard and solvable in polynomial-time by algorithm A. By the definition of problem, class hardness, for every problem Y in cā¢oā¢Nā¢PcoNPc o N P there is a polynomial-time many-one reduction Ī Ī from Y to X; let AĪ subscriptĪ A_ Aroman_Ī be the polynomial-time algorithm encoded in Ī Ī . We can create a polynomial-time algorithm Aā² for co-Y by running AĪ subscriptĪ A_ Aroman_Ī on a given input, running A, and then complementing the produced output, i.e., āYesā ā ā āNoā and āNoā ā ā āYesā. However, as coX is Nā¢PNPN P-hard by Lemma 5, this implies that P=Nā¢P=NPP = N P. ā Lemma 7. (Flum & Grohe, 2006, Lemma 8.23) Let CC be a parameterized complexity class. Given a parameterized decision problem X, if X is CC-hard then co-X is cā¢oā¢co Cc o C-hard. Lemma 8. Given a parameterized decision problem X, if X is fixed-parameter tractable then co-X is fixed-parameter tractable. Proof. Given a fixed-parameter tractable algorithm A for X, we can create a fixed-parameter tractable algorithm Aā² for co-X by running A on a given input and then complementing the produced output, i.e., āYesā ā ā āNoā and āNoā ā ā āYesā. ā Lemma 9. Given a parameterized decision problem X, if X is cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hard and fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Suppose decision problem X is cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hard and solvable in polynomial-time by algorithm A. By the definition of problem class hardness, for every problem Y in cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ] there is a parameterized reduction Ī Ī from Y to X; let AĪ subscriptĪ A_ Aroman_Ī be the fixed-parameter tractable algorithm encoded in Ī Ī . We can create a polynomial-time algorithm Aā² for co-Y by running AĪ subscriptĪ A_ Aroman_Ī on a given input, running A, and then complementing the produced output, i.e., āYesā ā ā āNoā and āNoā ā ā āYesā. However, as coX is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard by Lemma 7, this implies that P=Nā¢P=NPP = N P. ā We will also be deriving polynomial-time inapproximability results for optimization versions of MLCR, MLCRā, MGCR, and MGCRā, i.e., ā¢ Max-MLCR, which asks for the maximum value k such that M is k-robust relative to H for I. ā¢ Max-MLCRā, which asks for the maximum value k such that M is k-robust relative to H=MH=MH = M for I. ā¢ Max-MGCR, which asks for the maximum value k such that M is k-robust relative to H for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n ā¢ Max-MGCRā, which asks for the maximum value k such that M is k-robust relative to H=MH=MH = M for every possible Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n The derivation of such results is complicated by both the cā¢oā¢Nā¢PcoNPc o N P-hardness of the decision versions of these problems (and the scarcity of inapproximability results for cā¢oā¢Nā¢PcoNPc o N P-hard problems which one could transfer to our problems by L-reductions) and the fact that the optimization versions of our problems are evaluation problems that return numbers rather than graph structures (which makes the use of instance-copy and gap inapproximability proof techniques [Garey & Johnson 1979, Chapter 6] extremely difficult). We shall sidestep many of these issues by deriving our results within the Oā¢pā¢tā¢POptPO p t P framework for analyzing evaluation problems developed in Krentel 1988; Gasarch et al. 1995. In particular, we shall find the following of use. Definition 18. (Adapted from (Krentel, 1988, page 493)) Let f,g:Ī£āā:āsuperscriptĪ£f,g: ^* , g : Ī£ā ā Z. A metric reduction from f to g is a pair (T1,T2)subscript1subscript2(T_1,T_2)( T1 , T2 ) of polynomial-time computable functions where T1:Ī£āāĪ£ā:subscript1āsuperscriptĪ£superscriptĪ£T_1: ^*ā ^*T1 : Ī£ā ā Ī£ā and T2:Ī£āĆā:subscript2āsuperscriptĪ£T_2: ^*ĆZ 2 : Ī£ā Ć Z ā Z such that fā¢(x)=T2ā¢(x,gā¢(T1ā¢(x)))subscript2subscript1f(x)=T_2(x,g(T_1(x)))f ( x ) = T2 ( x , g ( T1 ( x ) ) ) for all xāĪ£āsuperscriptĪ£xā ^*x ā Ī£ā. Lemma 10. (Corollary of (Krentel, 1988, Theorem 4.3)) Given an evaluation problem Ī Ī that is Oā¢pā¢tā¢Pā¢[Oā¢(logā”n)]delimited-[]OptP[O( n)]O p t P [ O ( log n ) ]-hard under metric reductions, if Ī Ī has a c-additive approximation algorithm for some cāoā¢(pā¢oā¢lā¢y)cā o(poly)c ā o ( p o l y )444 Note that oā¢(pā¢oā¢lā¢y)o(poly)o ( p o l y ) is the set of all functions f that are strictly upper bounded by all polynomials of n, i.e., fā¢(n)ā¤cĆgā¢(n)f(n)ā¤ cĆ g(n)f ( n ) ā¤ c Ć g ( n ) for nā„n0subscript0nā„ n_0n ā„ n0 for all c>00c>0c > 0 and gā¢(n)āāŖknk=nOā¢(1)subscriptsuperscriptsuperscript1g(n)ā _kn^k=n^O(1)g ( n ) ā āŖk nitalic_k = nitalic_O ( 1 ). then P=Nā¢P=NPP = N P. Some of our metric reductions use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. The hardness (inapproximability) results in this section hold when the given MLP M has three (six) hidden layers. K.1 Results for MLCR-special and MLCR Let us first consider problem MLCRā. Theorem 72. If MLCRā is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. As MLCRā is the complement of MLCA by Observation 3 and MLCA is Nā¢PNPN P-hard by the proof of Theorem 39, Lemma 5 implies that MLCRā is cā¢oā¢Nā¢PcoNPc o N P-hard. The result then follows from Lemma 6. ā Theorem 73. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCRā is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. The cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCRā follows from the Wā¢[1]delimited-[]1W[1]W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCA (Theorem 40), Observation 3, and Lemma 7. The result then follows from Lemma 9. ā Theorem 74. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCRā is fixed-parameter tractable. Proof. The result follows from the fixed-parameter tractability of āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCA Theorem 41, Observation 3, and Lemma 8. ā Theorem 75. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLCRā is fixed-parameter tractable. Proof. The result follows from the fixed-parameter tractability of āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLCA Theorem 42, Observation 3, and Lemma 8. ā Observe that the results in Theorems 73ā75 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MLCRā relative to many subsets of the parameters listed in Table 10. We can derive a polynomial-time additive inapproximability result for Max-MLCRā using the following chain of metric reductions based on two evaluation problems: ā¢ Min-VC, which asks for the minimum value k such that G has a vertex cover of size k. ā¢ Min-MLCA, which asks for the minimum value k such that there is a k-size subset Nā² of the |N||N|| N | neurons in M such that Mā¢(I)ā (M/Nā²)ā¢(I)superscriptā²M(I)ā (M/N )(I)M ( I ) ā ( M / Nā² ) ( I ). Lemma 11. Min-VC metric reduces to Min-MLCA. Proof. Consider the following reduction from Min-VC to Min-MLCA. Given an instance X=āØG=(V,E)ā©delimited-āØā©X= G=(V,E) = āØ G = ( V , E ) ā© of Min-VS, construct the following instance Xā²=āØM,Iā©superscriptā²X = M,I ā² = āØ M , I ā© of Min-MLCA: Let M be an MLP based on #ā¢ntā¢oā¢t=3ā¢|V|+2ā¢|E|+2#subscript322\#n_tot=3|V|+2|E|+2# nitalic_t o t = 3 | V | + 2 | E | + 2 neurons spread across six layers: 1. Input neuron layer: The single input neuron niā¢nsubscriptn_innitalic_i n (bias +11+1+ 1). 2. Hidden vertex pair layer: The vertex neurons nā¢vā¢Pā¢11,nā¢vā¢Pā¢12,ā¦ā¢nā¢vā¢Pā¢1|V|subscript11subscript12ā¦subscript1nvP1_1,nvP1_2,ā¦ nvP1_|V|n v P 11 , n v P 12 , ā¦ n v P 1| V | and nā¢vā¢Pā¢21,nā¢vā¢Pā¢22,ā¦ā¢nā¢vā¢Pā¢2|V|subscript21subscript22ā¦subscript2nvP2_1,nvP2_2,ā¦ nvP2_|V|n v P 21 , n v P 22 , ā¦ n v P 2| V | (all with bias 0). 3. Hidden vertex AND layer: The vertex neurons nā¢vā¢A1,nā¢vā¢A2,ā¦ā¢nā¢A|V|subscript1subscript2ā¦subscriptnvA_1,nvA_2,ā¦ nA_|V|n v A1 , n v A2 , ā¦ n A| V |, all of which are 2-way AND ReLU gates. 4. Hidden edge AND layer: The edge neurons nā¢eā¢A1,nā¢eā¢A2,ā¦ā¢nā¢eā¢A|E|subscript1subscript2ā¦subscriptneA_1,neA_2,ā¦ neA_|E|n e A1 , n e A2 , ā¦ n e A| E |, all of which are 2-way AND ReLU gates. 5. Hidden edge NOT layer: The edge neurons nā¢eā¢N1,nā¢eā¢N2,ā¦ā¢nā¢eā¢N|E|subscript1subscript2ā¦subscriptneN_1,neN_2,ā¦ neN_|E|n e N1 , n e N2 , ā¦ n e N| E |, all of which are NOT ReLU gates. 6. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is an |E||E|| E |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input neuron has an edge of weight 0 coming from its corresponding input and is in turn connected to each of the vertex pair neurons with weight 1. ā¢ Each vertex pair neuron nā¢vā¢Pā¢1isubscript1nvP1_in v P 1i (nā¢vā¢Pā¢2isubscript2nvP2_in v P 2i), 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to vertex AND neuron nā¢vā¢RisubscriptnvR_in v Ritalic_i with weight 2 (0). ā¢ Each vertex AND neuron nā¢vā¢AisubscriptnvA_in v Aitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge AND neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge AND neuron nā¢eā¢AisubscriptneA_in e Aitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i with weight 1. ā¢ Each edge NOT neuron nā¢eā¢NisubscriptneN_in e Nitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(1)1I=(1)I = ( 1 ). Observe that this instance of Min-MLCA can be created in time polynomial in the size of the given instance of Min-VC. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 niā¢nā¢(1)subscript1n_in(1)nitalic_i n ( 1 ) 2 nā¢vā¢Pā¢11ā¢(1),nā¢vā¢Pā¢12ā¢(1),ā¦ā¢nā¢vā¢Pā¢1|V|ā¢(1),nā¢vā¢Pā¢21ā¢(1),nā¢vā¢Pā¢22ā¢(1),ā¦ā¢nā¢vā¢Pā¢2|V|ā¢(1)subscript111subscript121ā¦subscript11subscript211subscript221ā¦subscript21nvP1_1(1),nvP1_2(1),ā¦ nvP1_|V|(1),nvP2_1(1),nvP2_2(1),ā¦ nvP% 2_|V|(1)n v P 11 ( 1 ) , n v P 12 ( 1 ) , ā¦ n v P 1| V | ( 1 ) , n v P 21 ( 1 ) , n v P 22 ( 1 ) , ā¦ n v P 2| V | ( 1 ) 3 nā¢vā¢A1ā¢(1),nā¢vā¢A2ā¢(1),ā¦ā¢nā¢vā¢A|V|ā¢(1)subscript11subscript21ā¦subscript1nvA_1(1),nvA_2(1),ā¦ nvA_|V|(1)n v A1 ( 1 ) , n v A2 ( 1 ) , ā¦ n v A| V | ( 1 ) 4 nā¢eā¢A1ā¢(1),nā¢eā¢A2ā¢(1),ā¦ā¢nā¢eā¢A|V|ā¢(1)subscript11subscript21ā¦subscript1neA_1(1),neA_2(1),ā¦ neA_|V|(1)n e A1 ( 1 ) , n e A2 ( 1 ) , ā¦ n e A| V | ( 1 ) 5 nā¢eā¢N1ā¢(0),nā¢eā¢N2ā¢(0),ā¦ā¢nā¢eā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0neN_1(0),neN_2(0),ā¦ neN_|V|(0)n e N1 ( 0 ) , n e N2 ( 0 ) , ā¦ n e N| V | ( 0 ) 6 noā¢uā¢t(0))n_out(0))nitalic_o u t ( 0 ) ) Note that, given the 0 (2) connection-weights of P2 (P1) vertex pair neurons to vertex AND neurons, it is the outputs of P1 vertex pair neurons in timestep 2 that enables vertex AND neurons to output 1 in timestep 3. We now need to show the correctness of this reduction by proving that the answer for the given instance of Min-VC is k if and only if the answer for the constructed instance of Min-MLCA is k. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a minimum-size vertex cover in G of size k and Nā² be the k-sized sized subset of the P1 vertex pair neurons corresponding to the vertices in Vā². Let Mā² be the version of M in which all neurons in Nā² are ablated. As each of these vertex pair neurons previously allowed their associated vertex AND neurons to output 1, their ablation now allows these k vertex AND neurons to output 0. As Vā² is a vertex cover of size k, all edge AND neurons in Mā² receive inputs of 0 on at least one of their endpoints from the vertex AND neurons associated with the P1 vertex pair neurons in Nā². This in turn ensures that all of the edge NOT neurons and the output neuron produces output 1. Hence, Mā¢(I)=0ā 1=Mā²ā¢(I)01superscriptā²M(I)=0ā 1=M (I)M ( I ) = 0 ā 1 = Mā² ( I ). ā ā : Let Nā² be a minimum-sized subset of N of size k such that for the MLP Mā² induced by ablating all neurons in Nā², Mā¢(I)ā Mā¢(Iā²)superscriptā²M(I)ā M(I )M ( I ) ā M ( Iā² ). As Mā¢(I)=00M(I)=0M ( I ) = 0 and circuit outputs are stepped to be Boolean, Mā²ā¢(I)=1superscriptā²1M (I)=1Mā² ( I ) = 1. Given the bias of the output neuron, this can only occur if all |E||E|| E | edge NOT (AND) neurons in Mā² have output 1 (0) on input I, the latter of which requiring that each edge AND neuron receives 0 from at least one of its endpoint vertex AND neurons. These vertex AND neurons can only output 0 if all of their associated P1 vertex neurons have been ablated. This means that the vertices in G corresponding to the P1 vertex pair neurons in Nā² must form a vertex cover of size k in G. As this proves that Min-VC(X) === Min-MLCA(Xā²), the reduction above is a metric reduction from Min-VC to Min-MLCA. ā Lemma 12. Min-MLCA metric reduces to Max-MLCRā. Proof. As MLCA is the complement problem of MLCRā (Observation 3), we already have a trivial reduction from MLCA to MLCRā on their common input X=āØM,Iā©X= M,I = āØ M , I ā©. We can then show that k is the minimum value such that M has a k circuit ablation relative to I if and only if kā11k-1k - 1 is the maximum value such that M is kā11k-1k - 1-robust relative to I: ā ā If k is is the minimum value such that M has a k-sized circuit ablation relative to I then no subset of M of size kā11k-1k - 1 can be a circuit ablation of M relative to I, and M is (kā1)1(k-1)( k - 1 )-robust relative to I. Moreover, M cannot be k-robust relative to I as that would contradict the existence of a k-sized circuit ablation for M relative to I. Hence, kā11k-1k - 1 is the maximum robustness value for M relative to I. ā ā If kā11k-1k - 1 is the maximum value such that M is (kā1)1(k-1)( k - 1 )-robust relative to I then there must be a subset H of M of size k that ensures M is not kā11k-1k - 1-robust, i.e., (M/H)ā¢(I)ā Mā¢(I)(M/H)(I)ā M(I)( M / H ) ( I ) ā M ( I ). Such an H is a k-sized circuit ablation of M relative to I. Moreover, there cannot be a (kā1)1(k-1)( k - 1 )-sized circuit ablation of M relative to I as that would contradict the (kā1)1(k-1)( k - 1 )-robustness of M relative to I. Hence, k is the minimum size of circuit ablations for M relative to I. As this proves that Min-MLCA(X) === Max-MLCRā(X) +11+1+ 1, the reduction above is a metric reduction from Min-MLCA to Max-MLCRā. ā Theorem 76. If Max-MLCRāhas a c-additive approximation algorithm for some cāoā¢(pā¢oā¢lā¢y)cā o(poly)c ā o ( p o l y ) then P=Nā¢P=NPP = N P. Proof. The Oā¢pā¢tā¢Pā¢[Oā¢(logā”n)]delimited-[]OptP[O( n)]O p t P [ O ( log n ) ]-hardness of Max-MLCRā follows from the Oā¢pā¢tā¢Pā¢[Oā¢(logā”n)]delimited-[]OptP[O( n)]O p t P [ O ( log n ) ]-hardness of Min-VC (Gasarch et al., 1995, Theorem Theorem 3.3) and the metric reductions in Lemmas 11 and 12. The result then follows from Lemma 4 ā Let us now consider problem MLCR. Lemma 13. MLCRā many-one polynomial-time reduces to MLCR. Proof. Follows from the trivial reduction in which an instance āØM,I,kā© M,I,k āØ M , I , k ā© of MLCRā is transformed into an instance āØM,H=M,I,kā©delimited-āØā©formulae-sequence M,H=M,I,k āØ M , H = M , I , k ā© of MLCR. ā Theorem 77. If MLCR is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Follows from the cā¢oā¢Nā¢PcoNPc o N P-hardness of MLCRā (Theorem 72), the reduction in Lemma 13, and Lemma 6. ā Theorem 78. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCR is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Follows from the cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MLCRā (Theorem 73), the reduction in Lemma 13, and Lemma 9. ā Theorem 79. āØ|H|ā©delimited-āØā© |H| āØ | H | ā©-MLCR is fixed-parameter tractable. Proof. Consider the algorithm that generates every possible subset Hā² of size at most k of the neurons N in H and for each such subset (assuming M/Hā²M/H M / Hā² is active) checks if (M/Hā²)ā¢(I)ā Mā¢(I)superscriptā²(M/H )(I)ā M(I)( M / Hā² ) ( I ) ā M ( I ). If such a subset is found, return āNoā; otherwise, return āYesā. The number of possible subsets Hā² is at most kĆ|H|kā¤|H|Ć|H||H|superscriptsuperscriptkĆ|H|^kā¤|H|Ć|H|^|H|k Ć | H |k ā¤ | H | Ć | H || H |. As any such Mā² can be generated from M, checked or activity, and run on I in time polynomial in the size of the given instance of MLCR, the above is a fixed-parameter tractable algorithm for MLCR relative to parameter-set |H|\|H|\ | H | . ā Theorem 80. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MLCR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 79 and the observation that |H|ā¤cā¢wĆcā¢d|H|ā¤ cwĆ cd| H | ā¤ c w Ć c d. ā Theorem 81. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MLCR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 79 and the observation that |H|ā¤#ā¢ntā¢oā¢t#subscript|H|ā¤\#n_tot| H | ā¤ # nitalic_t o t. ā Theorem 82. If Max-MLCR has a c-additive approximation algorithm for some cāoā¢(pā¢oā¢lā¢y)cā o(poly)c ā o ( p o l y ) then P=Nā¢P=NPP = N P. Proof. As Max-MLCRā is a special case of Max-MLCR, if Max-MLCR has a c-additive approximation algorithm for some c then so does Max-MLCRā. The result then follows from Theorem 76 ā K.2 Results for MGCR-special and MGCR Consider the following variant of MGCA: Special Circuit Ablation (SCA) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, and a positive integer k such that 1ā¤kā¤#ā¢ntā¢oā¢t1#subscript1ā¤ kā¤\#n_tot1 ā¤ k ā¤ # nitalic_t o t. Question: Is there a subset Nā², |Nā²|ā¤ksuperscriptā²|N |ā¤ k| Nā² | ā¤ k, of the |N||N|| N | neurons in M such that for the MLP Mā² induced by NāNā² N N ā Nā², Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for some Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n? Theorem 83. If SCA is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Observe that in the instance of MLCA constructed by the reduction from Clique in the proof of Theorem 40, the input-connection weight 0 and bias 1 of the input neuron force this neuron to output 1 for both of the possible input vectors (1)1(1)( 1 ) and (0)0(0)( 0 ). This means that the answer to the given instance of Clique is āYesā if and only if the answer to the constructed instance of MLCA relative to any of its possible input vectors is āYesā. Hence, with slight modifications to the proof of reduction correctness, this reduction also establishes the Nā¢PNPN P-hardness of SCA. ā Theorem 84. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-SCA is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of SCA constructed in the reduction in the proof of Theorem 83, #ā¢niā¢n=#ā¢noā¢uā¢t=Wmax=1#subscript#subscriptsubscript1\#n_in=\#n_out=W_ =1# nitalic_i n = # nitalic_o u t = Wroman_max = 1, cā¢d=55cd=5c d = 5, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 85. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-SCA is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 41 such that each created MLP M is checked to ensure that Mā¢(I)ā Mā²ā¢(I)superscriptā²M(I)ā M (I)M ( I ) ā Mā² ( I ) for every possible Boolean input vector of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢nā¤2#ā¢ntā¢oā¢tsuperscript2#subscriptsuperscript2#subscript2^\#n_inā¤ 2^\#n_tot2# nitalic_i n ā¤ 2# nitalic_t o t, the above is a fixed-parameter tractable algorithm for SCA relative to parameter-set #ā¢ntā¢oā¢t#subscript\\#n_tot\ # nitalic_t o t . ā Theorem 86. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-SCA is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 85 and the observation that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d. ā Our results above for SCA above gain importance for us here courtesy of the following observation. Observation 4. MGCRā is the complement of SCA. Proof. Let X=āØM,kā©X= M,k = āØ M , k ā© be a shared input of MGCRā and SCA. If the the answer to SCA on input X is āYesā, this means that there is a subset H of size ā¤kabsentā¤ kā¤ k of the neurons of M that can be ablated such that Mā¢(I)ā (M/H)ā¢(I)M(I)ā (M/H)(I)M ( I ) ā ( M / H ) ( I ) for some input vector I. This implies that the answer to MGCRā on input X is āNoā. Conversely, if the answer to SCA on input X is āNoā, this means that there is no subset H of size ā¤kabsentā¤ kā¤ k of the neurons of M that can be ablated such that Mā¢(I)ā (M/H)ā¢(I)M(I)ā (M/H)(I)M ( I ) ā ( M / H ) ( I ) for any input vector I. This implies that the answer to MGCRā on input X is āYesā. The observation follows from the definition of complement problem. ā Theorem 87. If MGCRā is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. As MGCRā is the complement of SCA by Observation 4 and SCA is Nā¢PNPN P-hard (Theorem 83), Lemma 5 implies that MGCRā is cā¢oā¢Nā¢PcoNPc o N P-hard. The result then follows from Lemma 6. ā Theorem 88. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCRā is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. The cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCRā follows from the Wā¢[1]delimited-[]1W[1]W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-SCA (Theorem 84), Observation 4, and Lemma 7. The result then follows from Lemma 9. ā Theorem 89. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MGCRā is fixed-parameter tractable. Proof. The result follows from the fixed-parameter tractability of āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-SCA (Theorem 85), Observation 4, and Lemma 8. ā Theorem 90. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MGCRā is fixed-parameter tractable. Proof. The result follows from the fixed-parameter tractability of āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-SCA (Theorem 86), Observation 4, and Lemma 8. ā Observe that the results in Theorems 88ā90 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MGCRā relative to many subsets of the parameters listed in Table 10. We can derive a polynomial-time additive inapproximability result for Max-MGCRā using the following chain of metric reductions based on two evaluation problems: ā¢ Min-VC, which asks for the minimum value k such that G has a vertex cover of size k. ā¢ Min-SCA, which asks for the minimum value k such that there is a k-size subset Nā² of the |N||N|| N | neurons in M such that Mā¢(I)ā (M/Nā²)ā¢(I)superscriptā²M(I)ā (M/N )(I)M ( I ) ā ( M / Nā² ) ( I ) for some Boolean input vector I of length#ā¢niā¢n#subscript\#n_in# nitalic_i n. Lemma 14. Min-VC metric reduces to Min-SCA. Proof. Observe that in the instance of Min-MLCA constructed by the reduction from Min-VC in the proof of Theorem 11, the input-connection weight 0 and bias 1 of the input neuron force this neuron to output 1 for both of the possible input vectors (1)1(1)( 1 ) and (0)0(0)( 0 ). This means that the answer to the constructed instance of Min-MLCA is the same relative to any of its possible input vectors. Hence, with slight modifications to the proof of reduction correctness, this reduction is also a metric reduction from Min-VC to Min-SCA such that for given instance X of Min-VC and constructed instance Xā² of Min-SCA, Min-VC(X) === Min-SCA(Xā²). ā Lemma 15. Min-SCA metric reduces to Max-MGCRā. Proof. As SCA is the complement problem of MGCRā (Observation 4), we already have a trivial reduction from SCA to MGCRā on their common input X=āØMā©delimited-āØā©X= M = āØ M ā©. We can then show that k is the minimum value such that M has a k circuit ablation relative to some possible I if and only if kā11k-1k - 1 is the maximum value such that M is kā11k-1k - 1-robust relative to all possible I: ā ā If k is is the minimum value such that M has a k-sized circuit ablation relative to some possible I then no subset of M of size kā11k-1k - 1 can be a circuit ablation of M relative to any possible I, and M is (kā1)1(k-1)( k - 1 )-robust relative to all possible I. Moreover, M cannot be k-robust relative to all possible I as that would contradict the existence of a k-sized circuit ablation for M relative to some possible I. Hence, kā11k-1k - 1 is the maximum robustness value for M relative to all possible I. ā ā If kā11k-1k - 1 is the maximum value such that M is (kā1)1(k-1)( k - 1 )-robust relative to all possible I then there must be a subset H of M of size k that ensures M is not kā11k-1k - 1-robust relative to all possible I, i.e., (M/H)ā¢(I)ā Mā¢(I)(M/H)(I)ā M(I)( M / H ) ( I ) ā M ( I ) for some possible I. Such an H is a k-sized circuit ablation of M relative to that I. Moreover, there cannot be a (kā1)1(k-1)( k - 1 )-sized circuit ablation of M relative to some possible I as that would contradict the (kā1)1(k-1)( k - 1 )-robustness of M relative to all possible I. Hence, k is the minimum size of circuit ablations for M relative to some possible I. As this proves that Min-SCA(X) === Max-MGCRā(X) +11+1+ 1, the reduction above is a metric reduction from Min-SCA to Max-MGCRā. ā Theorem 91. If Max-MGCRāhas a c-additive approximation algorithm for some cāoā¢(pā¢oā¢lā¢y)cā o(poly)c ā o ( p o l y ) then P=Nā¢P=NPP = N P. Proof. The Oā¢pā¢tā¢Pā¢[Oā¢(logā”n)]delimited-[]OptP[O( n)]O p t P [ O ( log n ) ]-hardness of Max-MGCRā follows from the Oā¢pā¢tā¢Pā¢[Oā¢(logā”n)]delimited-[]OptP[O( n)]O p t P [ O ( log n ) ]-hardness of Min-VC (Gasarch et al., 1995, Theorem Theorem 3.3) and the metric reductions in Lemmas 14 and 15. The result then follows from Lemma 4 ā Let us now consider problem MGCR. Lemma 16. MGCRā many-one polynomial-time reduces to MGCR. Proof. Follows from the trivial reduction in which an instance āØM,kā© M,k āØ M , k ā© of MGCRā is transformed into an instance āØM,H=M,kā©delimited-āØā©formulae-sequence M,H=M,k āØ M , H = M , k ā© of MGCR. ā Theorem 92. If MGCR is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Follows from the cā¢oā¢Nā¢PcoNPc o N P-hardness of MGCRā (Theorem 87), the reduction in Lemma 16, and Lemma 6. ā Theorem 93. If āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCR is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Follows from the cā¢oā¢Wā¢[1]delimited-[]1coW[1]c o W [ 1 ]-hardness of āØcā¢d,#ā¢niā¢n,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscript#subscriptsubscriptsubscript cd,\#n_in,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_i n , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MGCRā (Theorem 88), the reduction in Lemma 16, and Lemma 9. ā Theorem 94. āØ#ā¢niā¢n,|H|ā©#subscript \#n_in,|H| āØ # nitalic_i n , | H | ā©-MGCR is fixed-parameter tractable. Proof. Modify the algorithm in the proof of Theorem 79 such that each created MLP M is checked to ensure that Mā¢(I)ā (H/Hā²)ā¢(I)superscriptā²M(I)ā (H/H )(I)M ( I ) ā ( H / Hā² ) ( I ) for every possible Boolean input vector of length #ā¢niā¢n#subscript\#n_in# nitalic_i n. As the number of such vectors is 2#ā¢niā¢nsuperscript2#subscript2^\#n_in2# nitalic_i n, the above is a fixed-parameter tractable algorithm for MGCR relative to parameter-set #ā¢niā¢n,|H|#subscript\\#n_in,|H|\ # nitalic_i n , | H | . ā Theorem 95. āØcā¢w,cā¢dā© cw,cd āØ c w , c d ā©-MGCR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 94 and the observations that #ā¢ntā¢oā¢tā¤cā¢wĆcā¢d#subscript\#n_totā¤ cwĆ cd# nitalic_t o t ā¤ c w Ć c d and |H|ā¤cā¢wĆcā¢d|H|ā¤ cwĆ cd| H | ā¤ c w Ć c d. ā Theorem 96. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MGCR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 94 and the observations that #ā¢ntā¢oā¢tā¤#ā¢ntā¢oā¢t#subscript#subscript\#n_totā¤\#n_tot# nitalic_t o t ā¤ # nitalic_t o t and |H|ā¤#ā¢ntā¢oā¢t#subscript|H|ā¤\#n_tot| H | ā¤ # nitalic_t o t. ā Theorem 97. If Max-MGCR has a c-additive approximation algorithm for some cāoā¢(pā¢oā¢lā¢y)cā o(poly)c ā o ( p o l y ) then P=Nā¢P=NPP = N P. Proof. As Max-MGCRā is a special case of Max-MGCR, if Max-MGCR has a c-additive approximation algorithm for some c then so does Max-MGCRā. The result then follows from Theorem 91 ā Appendix L Sufficient Reasons Problem Minimum sufficient reason (MSR) Input: A multi-layer perceptron M of depth cā¢dcdc d with #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t neurons and maximum layer width cā¢wcwc w, connection-value matrices W1,W2,ā¦,Wcā¢dsubscript1subscript2ā¦subscriptW_1,W_2,ā¦,W_cdW1 , W2 , ā¦ , Witalic_c d, neuron bias vector B, a Boolean input vector I of length #ā¢niā¢n#subscript\#n_in# nitalic_i n, and a positive integer k such that 1ā¤kā¤#ā¢niā¢n1#subscript1ā¤ kā¤\#n_in1 ā¤ k ā¤ # nitalic_i n. Question: Is there a k-sized subset Iā² of I such that for each possible completion Iā²I Iā² ā² of Iā², I and Iā²I Iā² ā² are behaviorally equivalent with respect to M? For a graph G=(V,E)G=(V,E)G = ( V , E ), we shall assume an ordering on the vertices and edges in V and E, respectively. For each vertex vāVvā Vv ā V, let the complete neighbourhood NCā¢(v)subscriptN_C(v)Nitalic_C ( v ) of v be the set composed of v and the set of all vertices in G that are adjacent to v by a single edge, i.e., vāŖu|uāVā¢andā¢(u,v)āEconditional-setanduvEvāŖ\u~|~u~ā V~ and~(u,v)ā E\v āŖ u | u ā V and ( u , v ) ā E . We will prove various classical and parameterized results for MSR using reductions from Clique. The parameterized results are proved relative to the parameters in Table 11. An additional reduction from DS (Theorem 103) use specialized ReLU logic gates described in BarcelĆ³ et al. 2020, Lemma 13. These gates assume Boolean neuron input and output values of 0 and 1 and are structured as follows: 1. NOT ReLU gate: A ReLU gate with one input connection weight of value ā11-1- 1 and a bias of 1. This gate has output 1 if the input is 0 and 0 otherwise. 2. n-way AND ReLU gate: A ReLU gate with n input connection weights of value 1 and a bias of ā(nā1)1-(n-1)- ( n - 1 ). This gate has output 1 if all inputs have value 1 and 0 otherwise. 3. n-way OR ReLU gate: A combination of an n-way AND ReLU gate with NOT ReLU gates on all of its inputs and a NOT ReLU gate on its output that uses DeMorganās Second Law to implement (x1āØx2āØā¦ā¢xn)subscript1subscript2ā¦subscript(x_1 x_2 ā¦ x_n)( x1 āØ x2 āØ ā¦ xitalic_n ) as Ā¬(Ā¬x1ā§Ā¬x2ā§ā¦ā¢Ā¬xn)subscript1subscript2ā¦subscript ( x_1 x_2 ā¦ x_n)Ā¬ ( Ā¬ x1 ā§ Ā¬ x2 ā§ ā¦ Ā¬ xitalic_n ). This gate has output 1 if any input has value 1 and 0 otherwise. Table 11: Parameters for the minimum sufficient reason problem. Parameter Description cā¢dcdc d # layers in given MLP cā¢wcwc w max # neurons in layer in given MLP #ā¢ntā¢oā¢t#subscript\#n_tot# nitalic_t o t total # neurons in given MLP #ā¢niā¢n#subscript\#n_in# nitalic_i n # input neurons in given MLP #ā¢noā¢uā¢t#subscript\#n_out# nitalic_o u t # output neurons in given MLP BmaxsubscriptB_ Broman_max max neuron bias in given MLP WmaxsubscriptW_ Wroman_max max connection weight in given MLP k Size of requested subset of input vector L.1 Results for MSR The following hardness results are notable for holding when the given MLP M has only one hidden layer. As such, they complement and significantly tighten results given in (BarcelĆ³ et al., 2020; WƤldchen et al., 2021), respectively. Theorem 98. If MSR is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from Clique to MSR. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of Clique, construct the following instance āØM,I,kā²ā©superscriptā² M,I,k āØ M , I , kā² ā© of MSR: Let M be an MLP based on #ā¢ntā¢oā¢t=|V|+|E|+1#subscript1\#n_tot=|V|+|E|+1# nitalic_t o t = | V | + | E | + 1 neurons spread across three layers: 1. Input vertex layer: The vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V | (all with bias 0). 2. Hidden edge layer: The edge neurons nā¢e1,nā¢e2,ā¦ā¢nā¢e|E|subscript1subscript2ā¦subscriptne_1,ne_2,ā¦ ne_|E|n e1 , n e2 , ā¦ n e| E | (all with bias ā11-1- 1). 3. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t (bias ā(kā¢(kā1)/2ā1)121-(k(k-1)/2-1)- ( k ( k - 1 ) / 2 - 1 )). Note that this MLP has only one hidden layer. The non-zero weight connections between adjacent layers are as follows: ā¢ Each vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each edge neuron whose corresponding edge has an endpoint visubscriptv_ivitalic_i with weight 1. ā¢ Each edge neuron nā¢eisubscriptne_in eitalic_i, 1ā¤iā¤|E|11ā¤ iā¤|E|1 ā¤ i ā¤ | E |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I=(1)1I=(1)I = ( 1 ) and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MSR can be created in time polynomial in the size of the given instance of Clique. Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 nā¢v1ā¢(1),nā¢v2ā¢(1),ā¦ā¢nā¢v|V|ā¢(1)subscript11subscript21ā¦subscript1nv_1(1),nv_2(1),ā¦ nv_|V|(1)n v1 ( 1 ) , n v2 ( 1 ) , ā¦ n v| V | ( 1 ) 2 nā¢e1ā¢(1),nā¢e2ā¢(1),ā¦ā¢nā¢e|E|ā¢(1)subscript11subscript21ā¦subscript1ne_1(1),ne_2(1),ā¦ ne_|E|(1)n e1 ( 1 ) , n e2 ( 1 ) , ā¦ n e| E | ( 1 ) 3 noā¢uā¢tā¢(|E|ā(kā¢(kā1)/2ā1))subscript121n_out(|E|-(k(k-1)/2-1))nitalic_o u t ( | E | - ( k ( k - 1 ) / 2 - 1 ) ) Note that it is the stepped output of noā¢uā¢tsubscriptn_outnitalic_o u t in timestep 3 that yields output 1. We now need to show the correctness of this reduction by proving that the answer for the given instance of Clique is āYesā if and only if the answer for the constructed instance of MSR is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a clique in G of size k and Iā² be the kā²=ksuperscriptā²k =kā² = k-sized subset of I corresponding to the vertices in Vā². As Vā² is a clique of size k, exactly kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons in the constructed MLP M receive the requisite inputs of 1 on both of their endpoints from the vertex neurons associated with Iā². This in turn ensures the output neuron produces output 1. No other possible inputs to the vertex neurons not corresponding to elements of Iā² can change the outputs of these activated edge neurons (and hence the output neuron as well) from 1 to 0. Hence, all completions of Iā² cause M to output 1 and are behaviorally equivalent to I with respect to M. ā ā : Let Iā² be a kā²=ksuperscriptā²k =kā² = k-sized subset of I such that all possible completions of Iā² are behaviorally equivalent to I with respect to M, i.e., all such completions cause M to output 1. Consider the completion Iā²I Iā² ā² of Iā² in which all non-Iā² elements have value 0. The output of M on Iā²I Iā² ā² can be 1 (and hence equal to the output of M on I) only if at least kā¢(kā1)/212k(k-1)/2k ( k - 1 ) / 2 edge neurons have output 1. As all non-Iā² elements of Iā²I Iā² ā² have value 0, this means that both endpoints of each of these edge neuron must be connected to elements of Iā² with output 1, which in turn implies that the kā²=ksuperscriptā²k =kā² = k vertices in G corresponding to the elements of Iā² form a clique of size k for G. As Clique is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MSR is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 99. If āØcā¢d,#ā¢noā¢uā¢t,Wmax,Bmax,kā©#subscriptsubscriptsubscript cd,\#n_out,W_ ,B_ ,k āØ c d , # nitalic_o u t , Wroman_max , Broman_max , k ā©-MSR is fixed-parameter tractable then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Observe that in the instance of MSR constructed in the reduction in the proof of Theorem 98, #ā¢noā¢uā¢t=Wmax=1#subscriptsubscript1\#n_out=W_ =1# nitalic_o u t = Wroman_max = 1, cā¢d=33cd=3c d = 3, and BmaxsubscriptB_ Broman_max and k are function of k in the given instance of Clique. The result then follows from the fact that āØkā©delimited-āØā© k āØ k ā©-Clique is Wā¢[1]delimited-[]1W[1]W [ 1 ]-hard (Downey & Fellows, 1999). ā Theorem 100. āØ#ā¢niā¢nā©delimited-āØā©#subscript \#n_in āØ # nitalic_i n ā©-MSR is fixed-parameter tractable. Proof. Consider the algorithm that generates each possible subset Iā² of of I of size k and for each such Iā², checks if all possible completions of Iā² are behaviorally equivalent to I with respect to M. If such an Iā² is found, return āYesā; otherwise, return āNoā. The number of possible Iā² is at most (#ā¢niā¢n)kā¤(#ā¢niā¢n)#ā¢niā¢nsuperscript#subscriptsuperscript#subscript#subscript(\#n_in)^kā¤(\#n_in)^\#n_in( # nitalic_i n )k ā¤ ( # nitalic_i n )# nitalic_i n and the number of possible completions of any such Iā² is less than 2#ā¢niā¢nsuperscript2#subscript2^\#n_in2# nitalic_i n. Given this, as M can be run on each completion of Iā² is time polynomial in the size of the given instance of MSR, the above is a fixed-parameter tractable algorithm for MSR relative to parameter-set #ā¢niā¢n#subscript\\#n_in\ # nitalic_i n . ā Theorem 101. āØ#ā¢ntā¢oā¢tā©delimited-āØā©#subscript \#n_tot āØ # nitalic_t o t ā©-MSR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 100 and the observation that #ā¢niā¢nā¤#ā¢ntā¢oā¢t#subscript#subscript\#n_inā¤\#n_tot# nitalic_i n ā¤ # nitalic_t o t. ā Theorem 102. āØcā¢wā©delimited-āØā© cw āØ c w ā©-MSR is fixed-parameter tractable. Proof. Follows from the algorithm in the proof of Theorem 100 and the observation that #ā¢niā¢nā¤cā¢w#subscript\#n_inā¤ cw# nitalic_i n ā¤ c w. ā Observe that the results in Theorems 99ā102 in combination with Lemmas 1 and 2 suffice to establish the parameterized complexity status of MSR relative to every subset of the parameters listed in Table 11. Let us now consider the polynomial-time cost approximability of MSR. As MSR is a minimization problem, we cannot do this using reductions from a maximization problem like Clique. Hence we will instead use a reduction from another minimization problem, namely DS. Theorem 103. If MSR is polynomial-time tractable then P=Nā¢P=NPP = N P. Proof. Consider the following reduction from DS to MSR. Given an instance āØG=(V,E),kā©delimited-āØā© G=(V,E),k āØ G = ( V , E ) , k ā© of DS, construct the following instance āØM,I,kā²ā©superscriptā² M,I,k āØ M , I , kā² ā© of MSR: Let M be an MLP based on #ā¢ntā¢oā¢t,g=3ā¢|V|+1#subscript31\#n_tot,g=3|V|+1# nitalic_t o t , g = 3 | V | + 1 neurons spread across four layers: 1. Input layer: The input vertex neurons nā¢v1,nā¢v2,ā¦ā¢nā¢v|V|subscript1subscript2ā¦subscriptnv_1,nv_2,ā¦ nv_|V|n v1 , n v2 , ā¦ n v| V |, all of which have bias 0. 2. Hidden vertex neighbourhood layer I: The vertex neighbourhood AND neurons nā¢vā¢nā¢A1,nā¢vā¢nā¢A2,ā¦ā¢nā¢vā¢nā¢A|V|subscript1subscript2ā¦subscriptnvnA_1,nvnA_2,ā¦ nvnA_|V|n v n A1 , n v n A2 , ā¦ n v n A| V |, where nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i is an x-way AND ReLU gates such that x=|NCā¢(vi)|subscriptsubscriptx=|N_C(v_i)|x = | Nitalic_C ( vitalic_i ) |. 3. Hidden vertex neighbourhood layer I: The vertex neighbourhood NOT neurons nā¢vā¢nā¢N1,nā¢vā¢nā¢N2,ā¦ā¢nā¢vā¢nā¢N|V|subscript1subscript2ā¦subscriptnvnN_1,nvnN_2,ā¦ nvnN_|V|n v n N1 , n v n N2 , ā¦ n v n N| V |, all of which are NOT ReLU gates. 4. Output layer: The single output neuron noā¢uā¢tsubscriptn_outnitalic_o u t, which is a |V||V|| V |-way AND ReLU gate. The non-zero weight connections between adjacent layers are as follows: ā¢ Each input vertex neuron nā¢visubscriptnv_in vitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to each vertex neighbourhood AND neuron nā¢vā¢nā¢AjsubscriptnvnA_jn v n Aitalic_j such that viāNCā¢(vj)subscriptsubscriptsubscriptv_iā N_C(v_j)vitalic_i ā Nitalic_C ( vitalic_j ) with weight 1. ā¢ Each vertex neighbourhood AND neuron nā¢vā¢nā¢AisubscriptnvnA_in v n Aitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to its corresponding vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i with weight 1. ā¢ Each vertex neighbourhood NOT neuron nā¢vā¢nā¢NisubscriptnvnN_in v n Nitalic_i, 1ā¤iā¤|V|11ā¤ iā¤|V|1 ā¤ i ā¤ | V |, is connected to the output neuron noā¢uā¢tsubscriptn_outnitalic_o u t with weight 1. All other connections between neurons in adjacent layers have weight 0. Finally, let I be the |V||V|| V |-length zero-vector and kā²=ksuperscriptā²k =kā² = k. Observe that this instance of MSR can be created in time polynomial in the size of the given instance of DS, Moreover, the output behaviour of the neurons in M from the presentation of input I until the output is generated is as follows: timestep neurons (outputs) 0 ā 1 nā¢vā¢N1ā¢(0),nā¢vā¢N2ā¢(0),ā¦ā¢nā¢vā¢N|V|ā¢(0)subscript10subscript20ā¦subscript0nvN_1(0),nvN_2(0),ā¦ nvN_|V|(0)n v N1 ( 0 ) , n v N2 ( 0 ) , ā¦ n v N| V | ( 0 ) 2 nā¢vā¢nā¢A1ā¢(0),nā¢vā¢nā¢A2ā¢(0),ā¦ā¢nā¢vā¢nā¢A|V|ā¢(0)subscript10subscript20ā¦subscript0nvnA_1(0),nvnA_2(0),ā¦ nvnA_|V|(0)n v n A1 ( 0 ) , n v n A2 ( 0 ) , ā¦ n v n A| V | ( 0 ) 3 nā¢vā¢nā¢N1ā¢(1),nā¢vā¢nā¢N2ā¢(1),ā¦ā¢nā¢vā¢nā¢N|V|ā¢(1)subscript11subscript21ā¦subscript1nvnN_1(1),nvnN_2(1),ā¦ nvnN_|V|(1)n v n N1 ( 1 ) , n v n N2 ( 1 ) , ā¦ n v n N| V | ( 1 ) 4 noā¢uā¢tā¢(1)subscript1n_out(1)nitalic_o u t ( 1 ) We now need to show the correctness of this reduction by proving that the answer for the given instance of DS is āYesā if and only if the answer for the constructed instance of MSR is āYesā. We prove the two directions of this if and only if separately as follows: ā ā : Let Vā²=v1ā²,v2ā²,ā¦,vkā²āVsuperscriptā²subscriptsuperscriptā²1subscriptsuperscriptā²2ā¦subscriptsuperscriptā²V =\v _1,v _2,ā¦,v _k\ Vā² = vā²1 , vā²2 , ā¦ , vā²italic_k ā V be a dominating set in G of size k and Iā² be the kā²=ksuperscriptā²k =kā² = k-sized subset of I corresponding to the vertices in Vā². As Vā² is a dominating set, each vertex neighbourhood AND neuron receives input 0 from at least one input vertex neuron in the set of input vertex neurons associated with Iā², which in turn ensures that each vertex neighbourhood AND neuron has output 0. This in turn ensures that M produces output 1. No other possible inputs to the vertex neighbourhood AND neurons can change the output of these neurons from 0 to 1. Hence, all completions of Iā² cause M to output 1 and are behaviorally equivalent to I with respect to M. ā ā : Let Iā² be a kā²=ksuperscriptā²k =kā² = k-sized subset of I such that all possible completions of Iā² are behaviorally equivalent to I with respect to M, i.e., all such completions cause M to output 1. Consider the completion Iā²I Iā² ā² of Iā² in which all non-Iā² elements have value 1. The output of M on Iā²I Iā² ā² can be 1 (and hence equal to the output of M on I) only if all vertex neighbourhood NOT neurons output 1, which in turn can happen only if all vertex neighbourhood AND gates output 0. However, as all non-Iā² elements of Iā²I Iā² ā² have value 1, this means that each vertex neighbourhood AND neuron must be connected to at least one element of Iā², which in turn implies that the kā²=ksuperscriptā²k =kā² = k vertices in G corresponding to the elements of Iā² form a dominating set of size k for G. As DS is Nā¢PNPN P-hard (Garey & Johnson, 1979), the reduction above establishes that MSR is also Nā¢PNPN P-hard. The result follows from the definition of Nā¢PNPN P-hardness. ā Theorem 104. If MSR has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Proof. Recall from the proof of correctness of the reduction in the proof of Theorem 103 that a given instance of DS has a dominating set of size k if and only if the constructed instance of Mā¢Sā¢RMSRM S R has a subset Iā² of I of size kā²=ksuperscriptā²k =kā² = k such that every possible completion of Iā² is behaviorally equivalent to I with respect to M. This implies that, given a polynomial-time c-approximation algorithm A for MSR for some constant c>00c>0c > 0, we can create a polynomial-time c-approximation algorithm for DS by applying the reduction to the given instance x of DS to construct an instance xā² of MSR, applying A to xā² to create an approximate solution yā², and then using yā² to create an approximate solution y for x that has the same cost as yā². The result then follows from Chen & Lin 2019, Corollary 2, which implies that if DS has a polynomial-time c-approximation algorithm for any constant c>00c>0c > 0 then Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. ā Note that this theorem also renders MSR PTAS-inapproximable unless Fā¢Pā¢T=Wā¢[1]delimited-[]1FPT=W[1]F P T = W [ 1 ]. Appendix M Probabilistic approximation schemes Let us now consider three other types of polynomial-time approximability that may be acceptable in situations where always getting the correct output for an input is not required: 1. algorithms that always run in polynomial time but are frequently correct in that they produce the correct output for a given input in all but a small number of cases (i.e., the number of errors for input size n is bounded by function eā¢rā¢rā¢(n)err(n)e r r ( n )) Hemaspaandra & Williams (2012); 2. algorithms that always run in polynomial time but are frequently correct in that they produce the correct output for a given input with high probability Motwani & Raghavan (1995); and 3. algorithms that run in polynomial time with high probability but are always correct (Gill, 1977). Unfortunately, none of these options are in general open to us, courtesy of the following result. Theorem 105. None of the hard problems in Table 4 are polynomial-time approximable in senses (1ā3). Proof. (Sketch) Holds relative to several strongly-believed or established complexity-class relation conjectures courtesy of the Nā¢PNPN P-hardness of the problems and the reasoning in the proof of (Wareham, 2022, Result E). ā Appendix N Supplementary discussion Search space size versus intrinsic complexity. Some of our hardness results can be surprising (e.g., fixed-parameter intractability indicating that taming intuitive network and circuit parameters is not enough to make queries feasible). Other findings might be unsurprising/surprising for the wrong reasons. Often intractability is assumed based on observing that a problem of interest has an exponential search space. But this is not a sufficient condition for intractability. For instance, although the Minimum Spanning Tree problem has an exponential search space, there is enough structure in it that can be exploited to get optimal solutions tractably. A more directly relevant example is our Quasi-Minimal Circuit problems, which also have exponential search spaces. This is a scenario where the typical reasoning in the literature would lead us astray. Given our tractability results, jumping to intractability conclusions would miss valuable opportunities to design tractable algorithms with guarantees. Worst-case analysis. Given our limited knowledge of the problem space of interpretability, worst-case analysis is appropriate to explore what problems might be solvable without requiring any additional assumptions (e.g., Bassan et al., 2024; BarcelĆ³ et al., 2020) and experimental results suggest it captures a lower bound on real-world complexity (e.g., Friedman et al., 2024; Shi et al., 2024; Yu et al., 2024a). One possibly fruitful avenue would be to conduct an empirical and formal characterization of learned weights in search of structure that could potentially distinguish conditions of (in)tractability. This could inform future average-case analyses on plausible distributional assumptions. Strategies for exploring the viability of interpretability queries. Although we find that many queries of interest are intractable in the general case (and empirical results are in line with this characterization), this should not paralyze real-world efforts to interpret models. As our exploration of the current complexity landscape shows, reasonable relaxations, restrictions and problem variants can yield tractable queries for circuits with useful properties. Consider a few out of many possible avenues to continue these explorations. (i) Faced with an intractable query, we can investigate which parameters of the problem (e.g., network, circuit aspects) might be responsible for the core hardness of the general problem. If these problematic parameters can be kept small in real-world applications, this can yield a fixed-parameter tractable query which can be answered efficiently in practice. We have explored some of these parameters, but many more could be, as any aspect of the problem can be parameterized. For this, a close dialogue between theorists and experimentalists will be crucial, as often empirical regularities suggest which parameters might be fruitful to explore theoretically, and experiments can test whether theoretically conjectured parameters are or can be kept small in practice. (i) Generating altogether different circuit query variants is another way of making interpretability feasible. Our formalization of quasi-minimal circuit problems illustrates the search for viable algorithmic options with examples of tractable problems for inner interpretability. When the use case is well defined, efficient queries that return circuits with useful affordances for applications can be designed. Some circuits (e.g., quasi-minimal circuits, but likely others) might mimic the affordances for prediction/control that ideal circuits have, while shedding the intractability that plagues the latter. (i) It could be fruitful to investigate properties of the network output. Although for some problems, our constructions use step functions in the output layer (following the literature; Bassan et al., 2024; BarcelĆ³ et al., 2020), for many problems we do not or provide alternative proofs without them. This suggests this is not likely a significant source of complexity. Another aspect could be the binary input/output, although continuous input/output does not necessarily matter complexity-wise. Sometimes it does, as in the case of Linear Programming (PTIME; Karmarkar, 1984) versus 0-1 Integer Programming (NP-complete Garey & Johnson, 1979), and sometimes it does not, as in Euclidean Steiner Tree (NP-hard, not in NP for technical reasons; Garey & Johnson, 1979) versus Rectilinear Steiner Tree (NP-complete; Garey & Johnson, 1979). Still, this is an interesting direction for future work, as it suggests studying the output as an axis of approximation. (iv) A different path is to design queries that partially rely on mid-level abstractions (Vilas et al., 2024a) to bridge the gap between circuits and human-intelligible algorithms (e.g., key-value mechanisms; Geva et al., 2022; Vilas et al., 2024b). (v) It is in principle possible that real-world trained neural networks possess an internal structure that is somehow benevolent to general (ideal) circuit queries (e.g., redundancy). In such optimistic scenarios, general-purpose heuristics might work well. The empirical evidence available, however, speaks against this possibility. In any case, it will always be important to characterize any ābenevolent structureā in the problems such that we can leverage it explicitly to design algorithms with useful guarantees.