Logic and Lattices for Distributed Programming

1. Logic and Lattices for Distributed Programming Neil Conway, William R. Marczak, Peter Alvaro, Joseph M. Hellerstein UC Berkeley David Maier Portland State University

2. Distributed Programming: Key Challenges Partial Failure Asynchrony

4. Dealing with Disorder Enforce global order • Paxos, Two-Phase Commit, GCS, … • “Strong Consistency” Tolerate disorder • Programmer must ensure correct behavior for many possible network orders • “Eventual Consistency” • Typical goal: replicas converge to same final state

5. Dealing with Disorder Enforce global order • Paxos, Two-Phase Commit, GCS, … • “Strong Consistency” Tolerate disorder • Programmer must ensure correct behavior for many possible network orders • “Eventual Consistency” • Typical goal: replicas converge to same final state

6. Goal:Make it easier to writeprogramson top ofeventual consistency

7. This Talk • Prior Work • Convergent Modules (CRDTs) • Monotonic Logic (CALM) • BloomL • Case Study

8. Write: {Alice, Bob, Carol} Read: {Alice, Bob} Students{Alice, Bob, Carol} Students{Alice, Bob} Client0 How to resolve? Write: {Alice, Bob, Dave} Read: {Alice, Bob} Students{Alice, Bob, Dave} Students{Alice, Bob} Client1

10. Students{Alice, Bob, Carol, Dave} Client0 Merge = Set Union Students{Alice, Bob, Carol, Dave} Client1

11. Commutative Operations • Common design pattern • Formalized as CRDTs: Convergent and Commutative Replicated Data Types • Shapiro et al., INRIA (2009-2012) • Based on join semilattices

12. Lattices hS,t,?iis a bounded join semilatticeiff: • S is a set • t is a binary operator (“least upper bound”) • Associative, commutative, and idempotent • Induces a partial order on S: x·Sy if xty = y • Informally, “merge function” for elements of S • ? is the “least” element in S • 8x2S: ?t x= x

13. Time IncreasingInteger(LUB = Max) Set (LUB = Union) Boolean(LUB = Or)

14. Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Read: {Alice, Bob, Carol, Dave} Client0 Teams{<Alice, Bob>} Teams{<Alice, Bob>,<Carol, Dave>} Teams{<Alice, Bob>,<Carol, Dave>} Read: {<Alice,Bob>} Write: {<Alice,Bob>, <Carol,Dave>} Replica Synchronization Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Remove: {Dave} Client1 Teams{<Alice, Bob>,<Carol, Dave>} Teams{<Alice, Bob>} Teams{<Alice, Bob>,<Carol, Dave>}

15. Students{Alice, Bob, Carol, Dave} Students{Alice, Bob, Carol} Read: {Alice, Bob, Carol} Client0 Teams{<Alice, Bob>} Teams{<Alice, Bob>} Read: {<Alice,Bob>} Replica Synchronization Nondeterministic Outcome! Students{Alice, Bob, Carol} Students{Alice, Bob, Carol, Dave} Remove: {Dave} Client1 Teams{<Alice, Bob>} Teams{<Alice, Bob>}

16. Problem:Composition of CRDTs canresult in non-determinism

17. Possible Solution:Encapsulate all distributedstate in a single CRDT Hard to design,verify, and test Doesn’t scale with application size

18. Goal:Design a language that allowssafe composition of CRDTs

19. Solution: … Datalog? • Concurrent work: distributed programming using Datalog • P2 (2006-2010) • Bloom (2010-2012) • Monotonic logic: building block for convergent distributed programs

20. Monotonic Logic • As input set grows, output set does not shrink • “Retraction-free” • Order independent • e.g., map, filter, join, union, intersection Non-Monotonic Logic • New inputs might retract previous outputs • Order sensitive • e.g., aggregation, negation

21. Monotonicity and Determinism Agents learn strictly more knowledge over time Different learning order, same final outcome Result:Program is deterministic!

22. CALM Analysis All monotone programs are deterministic Simple syntactic test for monotonicity Result: Whole-program static analysis foreventual consistency ConsistencyAs LogicalMonotonicity

23. Problem:CALM only applies to programs over growing sets Version Numbers Timestamps Threshold Tests

24. Quorum Vote • A coordinator accepts votes from agents • Count # of votes • When Count(Votes) > k, send “success” message

25. Quorum Vote • A coordinator accepts votes from agents • Count # of votes • When Count(Votes) > k, send “success” message Aggregation isnon-monotonic!

27. BloomL Constructs

28. Monotone Functions f : ST is a monotone function iff 8a,b2S : a·Sb) f(a)·Tf(b)

29. Time Monotone function fromset  increase-int Monotone functionfromincrease-intboolean size() >= 5 IncreasingInteger(LUB = Max) Set(LUB = Union) Boolean(LUB = Or)

30. Quorum Vote in BloomL QUORUM_SIZE=5 RESULT_ADDR="example.org" classQuorumVote includeBud state do channel :vote_chn, [:@addr, :voter_id] channel :result_chn, [:@addr] lset:votes lmax:vote_cnt lbool:got_quorum end bloom do votes <=vote_chn {|v|v.voter_id} vote_cnt<=votes.size got_quorum<=vote_cnt.gt_eq(QUORUM_SIZE) result_chn<~got_quorum.when_true { [RESULT_ADDR] } end end Annotated Ruby class Communication interfaces Program state Lattice state declarations Monotonic  CALM Accumulate votesinto set Monotone function: set ! max Monotone function: max !bool Program logic Merge function for set lattice Threshold test on bool (monotone)

31. BloomL Features • Generalizes logic programming to lattices • Integration of relational-style queries and functions over lattices • Efficient incremental evaluation scheme • Library of built-in lattices • Booleans, increasing/decreasing integers, sets, multisets, maps, … • API for defining custom lattices

32. Case Studies Key-Value Store • Object versioning via vector clocks • Quorum replication Replicated Shopping Cart • Using custom lattice types to encode domain-specific knowledge

33. Case Studies Key-Value Store • Object versioning via vector clocks • Quorum replication Replicated Shopping Cart • Using custom lattice types to encode domain-specific knowledge

34. Case Study: Shopping Carts

35. Case Study: Shopping Carts

36. Case Study: Shopping Carts

37. Case Study: Shopping Carts

38. Perspectives on Shopping • CRDTs • Individual server replicas converge • Bloom • Checkout is non-monotonic  requires distributed coordination • Built-in BloomL lattice types • Checkout is not a monotone function of any of the built-in lattices

39. Observation:Once a checkoutoccurs, no more shopping actionscan be performed

40. Observation:Each client knowswhena checkout can beprocessed“safely”

41. Monotone Checkout OPS = [1,2,3]Complete OPS = [2,3]Incomplete OPS = [1,2]Incomplete OPS = [2]Incomplete OPS = [3]Incomplete OPS = [1]Incomplete

42. Monotone Checkout

43. Monotone Checkout

44. Monotone Checkout

45. Monotone Checkout

46. Shopping Takeaways • Checkout summary is a monotone function of client’s activities • Custom lattice type captures application-specific notion of “forward progress” • “Unsafe” state hidden behind ADT interface

47. Recap • How to build eventually consistent systems • Write disorderlyprograms • Disorderly state • Lattices • Disorderly computation • Monotone functions over lattices • BloomL • Type system for deterministic behavior • Support for custom lattice types

48. Thank You!http://www.bloom-lang.net

49. Backup Slides

50. Strong Consistency in Industry “… there was a single overarching theme within the keynote talks… strong synchronization of the sort provided by a locking service must be avoided like the plague… [the key] challenge is to find ways of transforming services that might seem to need locking into versions that … can operate correctly without locking.” -- Birman et al.,“Toward a Cloud Computing Research Agenda” (LADIS, 2009)