Indistinguishability by adaptive procedures with advice, and lower bounds on hardness amplification proofs

We study how well can q-query decision trees distinguish between the following two distributions: (i) R = (R ₁ ,⋯, R _N ) that are i.i.d. indicator random variables, (ii) X = (R|R ∈ A) where A is an event s.t. Pr[R∈A] ≥ 2 ^-a . We prove two lemmas: • Forbidden-set lemma: There exists B ⊆ [N] of size poly(a, q, 1/η) such that q-query trees that do not query variables in B cannot distinguish X from R with advantage η. • Fixed-set lemma: There exists B ⊆ [N] of size poly(a, q, 1/η) and υ ∈ {0, 1} ^B such that q-query trees do not distinguish (X|X _B = v) from (R|R _B = v) with advantage η. The first can be seen as an extension of past work by Edmonds, Impagliazzo, Rudich and Sgall (Computational Complexity 2001), Raz (SICOMP 1998), and Shaltiel and Viola (SICOMP 2010) to adaptive decision trees. It is independent of recent work by Meir and Wigderson (ECCC 2017) bounding the number of i ∈ [N] for which there exists a q-query tree that predicts Xi from the other bits. We use the second, fixed-set lemma to prove lower bounds on black-box proofs for hardness amplification that amplify hardness from δ to 1/2 - ϵ. Specifically: • Reductions must make q = Ω(log(1/δ)/ϵ ² ) queries, implying a "size loss factor" of q. We also prove the lower bound q = Ω(log(1/δ)/ϵ) for "error-less" hardness amplification proofs, and for direct-product lemmas. These bounds are tight. • Reductions can be used to compute Majority on Ω(1/ϵ) bits, implying that black box proofs cannot amplify hardness of functions that are hard against constant depth circuits (unless they are allowed to use Majority gates). Both items extend to pseudorandom-generator constructions. These results prove 15-year-old conjectures by Viola, and improve on three incomparable previous works (Shaltiel and Viola, SICOMP 2010; Gutfreund and Rothblum, RANDOM 2008; Artemenko and Shaltiel, Computational Complexity 2014).