Shuffling by semi-random transpositions

1. Shuffling by semi-random transpositions Elchanan Mossel, U.C. Berkeley Joint work with Yuval Peres and Alistair Sinclair

2. Shuffling by random transpositions At each step choose two independent uniformly chosen cards and exchange them. A A 2 6 3 3 4 4 5 5 6 2 7 7 8 8

3. Shuffling by random transpositions • Thm[Diaconis-Shahshahani-81]: Themixing time of the random transpositions shuffle is (½ + o(1)) n log n. • One can prove an O(n log n)upper bound can using “marking” (more later). • Proof of an (n log n)lower bound: • At each step “touch” 2 random cards. • Until time (n log n)/4 there are (n1/2) untouched cards • ) permutation is not random.

4. The cyclic to random shuffle At step i exchange card at location (i mod n) with a uniformly chosen card. A 2 3 4 5 6 7 8

5. History of the cyclic to random shuffle • Shuffle introduced by Thorp (65). • Aldous and Diaconis (86) asked what is the mixing time? • Mironov posed again and proved O(n log n)upper bound using marking.

6. Why do we care? • General question: Is systematic scan faster than random update? (other examples: Diaconis-Ram ; Benjamini-Berger-Hoffman-Mfor asymmetric exclusion; Gaussian fields etc.). • Would be nice to find a “natural problem” where the mixing time is strictly between (n) and (n log n) • Mironov: Cyclic to random may tell us a lot about a widely used crypto algorithm RC4.

7. The RC4 algorithm More than 106 hits in google • Mironov: Let’s study algorithm assuming j is random. • Slow mixingcorresponds to weak crypto.

8. Upper Bounds - Broder’s Marking • Broder’s Marking argument: • Call the two pointers Lt and Rt. • Start by marking the first card that is pointed by L1. • At time t, mark card pointed by Lt if either: • The card at Rt is marked or • Rt = Lt.

9. Broder’s Marking A A A 2 6 6 3 3 3 4 4 4 5 5 5 6 2 2 7 7 7 8 8 8 R L R=L

10. Broder’s marking • By induction: Given the time and • set of marked cards and • their positions, • the permutation on the marked cards is uniform. • )The time when all cards are marked is a strong uniform time (permutation is random given the time). • In order to prove upper bound, need to bound the “marking time”. • For random transpositions easy: By coupon collector estimate this time is O(n log n). • Mironov: delicate analysis for cyclic to random.

11. A general n log nupper bound • Thm: [M-Peres-Sinclair] An O(n log n) upper bound on the mixing time holds for any shuffle where: • At step t we exchange cards Lt and Rt where • Rtare i.i.d. uniform in {0,…,n-1}. • The sequence Ltis independent of Rt. • Ltcan be random, deterministic etc. • Cyclic to random is given by Lt = t mod n. • Top to random is given by Lt = 0 for all n. • Random transpositionsis given by Lt i.i.d uniform. • Pf:Careful analysis of the marking process.

12. A general n log nupper bound • Proof In more detail: • May assume that Lt is deterministic. • Partition time into intervals of length 2n. • In such an interval look at pairs of times s < t such that Ls = Lt (there are at least n such pairs). • We can mark card x if: • at time s, x is chosen by Rs. • Rr Lt for s < r < t. • Rt is one of the marked cards. • Letting mi (ui) be the (un)-marked card at interval i, gives • E[ui+1 | Fi] · ui (1 – c mi) for c > 0. • Will skip the rest of the proof. Rs Ls x Lt x Rt

13. Cyclic to random shuffle – lower bound? • Mironov proved c n lower bound for some c > 1 using parity as a test function: • Each shuffle changes the parity with probability • (1 – 1/n). • After t steps, resulting parity=original parity with probability: • Q: Is next to random faster than random transpositions? • Note: All cards are touched by time n.

14. n log nlower bound for cyclic to random shuffle • Thm[M-Peres-Sinclair]: • The cyclic to random shuffle has a mixing time (n log n). • More precisely: • And here is how the proof goes:

15. Step 1: Homogenizing the chain • Problem: The chain is nottime homogenous. • Can be easily fixed: Consider a chain where at time t: • (0) swaped with (U), where U is uniform. • Rotate all cards to the left: ’(k) = (k+1 mod n). • Clearly chain is equivalent + • It is homogenous. • From now on study homogenized chain.

16. One card chain Markov chain for a single card: • Eigenvalues satisfy  = (1 – 1/n) where • (n-1)n – n n-1 + 1 = 0. • Want to show slow mixing) want  close to 1.

17. Asymptotics of eigen values and functions •  = (1 – 1/n)  where (n-1)n – n n-1 + 1 = 0. • Let -1 = 1 + z/n and get • (1+z/n)n – n (1+z/n) + (n-1) = 0!ez – z – 1 = 0. • Lemma 1:ez – z – 1 has non-zero complex roots. • Lemma 2: If  is a root, then M has an eigenvalue  such that 1-|| = (1+<)/n + O(1/n2). • Lemma 3: The eigenvectorf corresponding to  is “smooth”: |f|1· C |f|2. Will write |f| for either. • Pfs: Complex analysis … • Remark: Numerically, the smallest non-zero root is • = 2.088… + 7.416… i

18. The test function • Take f to be an eigenfunction of M corresponding to the eigenvalue closest to 1. • Define the test functionF • Easy: E[f] = 0 ) E[F] = 0. • Easy: E[F(idt)] = t |f|2. • A Longer calculation gives: E(F2) = |f|4/n E(F) = 0 E(F(idt)) = t |f|2

19. The main Lemma E(F2) = |f|4/n • Remains to bound E[|F(idt)|2]. • Main Lemma: E(F) = 0 E(F(idt)) = t |f|2 • )as long as ||2t ~¸ (4t + n)/n2 the idt and  (where  is uniform) have large total variation distance (2nd moment method). • Since 1 - || = O(1/n): • )1¸ (n log n)

20. Proof of main Lemma • The main lemma can be proved using Wilson’s • method and the properties of  and f. • Or it can be done more directly using coupling: • Lemma:

21. Proof of main Lemma • Pf idea: “Couple” the following two processes: • Process 1: cards i and j move independently. • Process 2: The location of cards i and j in the real process. • In process 1: • Remains to bound the difference between the processes • using coupling. • Will skip the details …

22. Conclusion and Open problems • We’ve seen that the mixing time of the pseudo-random next to random shuffle has the same mixing time as the random transposition shuffle. • Proof is not that hard. • Problem: How general is the phenomenon? • In particular: • Open problem: Are there any sequences (deterministic/random) It, such that the It to random shuffle mixes in less than n log n time?