Lower Bounds for 2-Dimensional Range Counting

1. Lower Bounds for 2-Dimensional Range Counting Mihai Pătraşcu summer @

2. Range Counting SELECT count(*) FROM employees WHERE salary <= 70000 AND startdate <= 1998 71000 70000 69000 68000 ’96 ’97 ’98 ’99

3. Some Theory • d dimensions • range trees (roughly): space S=nlgd-1n, query t=lgd-1n • space S=n2d, query t=O(1) • geometric extensions: range = disks, half-spaces, polygons… PROBLEM: lower bounds

4. The Semigroup Industry Let (U,+) be a semigroup Points have weights in U Given w1,…,wn: precompute S sums query: add t precomputed sums w7 w3 w1 w4 w6 w2 w5 w1+w2+w4+w5

5. Why are we so good at this? [Caravaggio] Why “Industry”? • tight semigroup lower bounds [Fredman JACM’81] [Chazelle FOCS’86] • many, many excellent bounds for many range problems

6. Semigroup = Low Dimension • say Mem[17]=w2+w6 • can Mem[17] help with this query? NO: cannot subtract w6 • Mem[17] described by bounding box can only be used when query dominates bounding box “How well do rectangles decompose into rectangles?” X(disks, half-planes…) Y w7 w3 w1 w4 w2 w6 w5 All these objects are low-dimensional!

7. Computation = High Dimension New concept… weights come from a group (U,+,-) • can cancel any weight => no bounding box • relevant information about Mem[17]:O(1)-D rectangle → linear combination of n variables • decomposability in O(1)-D → decomposability in n-D • n-D means: geometry → information theory

8. The Ultimate Frontier The cell-probe model: • plain old counting (weights {0,1}) • each cell stores any function of inputs • query probes t cells (adaptively), computes any function “Theory of the computer in your lap”

9. To boldly go where no one has gone before General lack of group lower bounds (never mind cell-probe!) [Chazelle STOC’95] Ω(lglgn) [Fredman JACM’82] [Chazelle FOCS’86] [Agarwal ’9x, ’0x] etc etc yeah, we have nice semigroup lower bounds but prove something in the group model

10. Our Small Step If S=nlgO(1)n, then t=Ω(lgn/lglgn)  group model!  …and cell-probe model!  tight in 2D Maybe not so giant leap for mankind…  does not grow with d => really only relevant for 2D

11. Basic Idea • space S=nlgd-1n, query t=lgd-1n • space S=n2d, query t=O(1) Remember: Separating linear from polynomial space: [P,Thorup STOC’06] Ω(lglgn) [P,Thorup SODA’06] --”-- randomized [P,Thorup FOCS’06] Ω(lgn/lglgn) (“unnatural” problems; not tight) NOW Ω(lgn/lglgn) (natural, fundamental problem; tight) The trick: “Consider n/lgnqueries, see what happens” This paper: what happens for range counting “technique” “framework” “trick”

12. Hard Instance The bit-reversal permutation(see FFT, etc): e.g. π(6) = π(“110”) = “011” = 3 Well-known hard instance: • points at (x, π(x)) • random query [0,a]X[0,b][in fact, many independent random queries]

13. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7

14. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7

15. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7

16. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

17. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

18. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

19. The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

20. 8 7 6 5 4 3 2 1 The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

21. 8 7 6 5 4 3 2 1 The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

22. 8 7 6 5 4 3 2 1 The Shuffle View 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 0 0 2 2 4 4 6 6 1 1 3 3 5 5 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7

23. 8 7 6 5 4 3 2 1 Shuffling the Queries 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 query[0,4]X[0,5]

24. 8 7 4 4 0 0 2 2 6 6 6 1 1 3 3 5 5 7 7 5 3 2 1 Shuffling the Queries 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 query[0,5]X[0,4] 4

25. 8 7 6 5 3 4 4 2 0 0 6 6 1 2 2 1 1 5 5 3 3 7 7 Shuffling the Queries 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 query[0,5]X[0,4] 4 4 0 0 2 2 6 6 1 1 3 3 5 5 7 7 4

26. 8 7 6 5 3 2 1 0 0 4 4 2 2 6 6 1 1 5 5 3 3 7 7 Shuffling the Queries 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 query[0,5]X[0,4] 4 4 0 0 2 2 6 6 1 1 Specifying a query = Saying how it shuffles at every level 3 3 5 5 7 7 4 4 4 0 0 6 6 2 2 1 1 5 5 3 3 7 7

27. S k Proof Sketch (I) • k queries, space S => one probe revelslg = Θ(klg(S/k)) bits of information about the queries • t probes => Θ(tklg(S/k)) bits • k=n/lgn S=nlgO(1)n t=o(lgn/lglgn) =>o(klgn) bits revealed by the probes about the queries

28. Proof Sketch (II) • I(queries)≤ I(shuffling at level 1)+I(shuffling at level 2) + …+ I(shuffling at level lgn)≤ o(klgn) • so for one level the data structure learns o(k) bits of information about the queries[i.e. doesn’t know where most queries fit] • so it cannot precompute useful sums [see Proceedings]

29. T T T H H H E E E E E E N N N D D D

30. 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15 8 8 10 10 12 12 14 14 9 9 11 11 13 13 15 15 8 8 12 12 10 10 14 14 9 9 13 13 11 11 15 15 8 8 12 12 10 10 14 14 9 9 13 13 11 11 15 15