Better Software Variability Using A Functional, Traversal-Based Implicit Invocation Architecture

1. Better Software Variability Using A Functional, Traversal-Based Implicit Invocation Architecture by Karl Lieberherr Northeastern University, CCIS/PRL Joint work with Bryan Chadwick, Ahmed Abdelmeged and Therapon Skotiniotis ABB

2. Vision • If ABB wants to take advantage of multi-core computers, it is beneficial to write programs in a style that is amenable to parallel execution. • Today I will present such a style: AP-F and its implementation in Java called DemeterF. • AP-F has other benefits: easier software variability ABB

3. interface ShapeI extends Remote { double get_x() throws RemoteException ; void set_x(int x) throws RemoteException ; double get_y() throws RemoteException ; void set_y(int y) throws RemoteException ; double get_width() throws RemoteException ; void set_width(int w) throws RemoteException ; double get_height() throws RemoteException ; void set_height(int h) throws RemoteException ; void adjustLocation() throws RemoteException ; void adjustDimensions() throws RemoteException ; } publicclass Shape implements ShapeI { protected AdjustableLocation loc; protected AdjustableDimension dim; public Shape() { loc = new AdjustableLocation(0, 0); dim = new AdjustableDimension(0, 0); } double get_x() throws RemoteException { returnloc.x(); } void set_x(int x) throws RemoteException { loc.set_x(); } double get_y() throws RemoteException { returnloc.y(); } void set_y(int y) throws RemoteException { loc.set_y(); } double get_width() throws RemoteException { returndim.width(); } void set_width(int w) throws RemoteException { dim.set_w(); } double get_height() throws RemoteException { returndim.height(); } void set_height(int h) throws RemoteException { dim.set_h(); } void adjustLocation() throws RemoteException { loc.adjust(); } void adjustDimensions() throws RemoteException { dim.adjust(); } } publicclass Shape { protecteddouble x_= 0.0, y_= 0.0; protecteddouble width_=0.0, height_=0.0; double get_x() { return x_(); } void set_x(int x) { x_ = x; } double get_y() { return y_(); } void set_y(int y) { y_ = y; } double get_width(){ return width_(); } void set_width(int w) { width_ = w; } double get_height(){ return height_(); } void set_height(int h) { height_ = h; } void adjustLocation() { x_ = longCalculation1(); y_ = longCalculation2(); } void adjustDimensions() { width_ = longCalculation3(); height_ = longCalculation4(); } } Write this coordinator Shape { selfex adjustLocation, adjustDimensions; mutex {adjustLocation, get_x, set_x, get_y, set_y}; mutex {adjustDimensions, get_width, get_height, set_width, set_height}; } class AdjustableLocation { protecteddouble x_, y_; public AdjustableLocation(double x, double y) { x_ = x; y_ = y; } synchronizeddouble get_x() { return x_; } synchronizedvoid set_x(int x) {x_ = x;} synchronizeddouble get_y() { return y_; } synchronizedvoid set_y(int y) {y_ = y;} synchronizedvoid adjust() { x_ = longCalculation1(); y_ = longCalculation2(); } } class AdjustableDimension { protecteddouble width_=0.0, height_=0.0; public AdjustableDimension(double h, double w) { height_ = h; width_ = w; } synchronizeddouble get_width() { return width_; } synchronizedvoid set_w(int w) {width_ = w;} synchronizeddouble get_height() { return height_; } synchronizedvoid set_h(int h) {height_ = h;} synchronizedvoid adjust() { width_ = longCalculation3(); height_ = longCalculation4(); } } portal Shape { double get_x() {} ; void set_x(int x) {}; double get_y() {}; void set_y(int y) {}; double get_width() {}; void set_width(int w) {}; double get_height() {}; void set_height(int h) {}; void adjustLocation() {}; void adjustDimensions() {}; } Modularization of crosscutting concerns Instead of writing this ABB

4. Outline • Application Domain: Algorithmic Trading Robots • need for structure-shy software (works in many structural contexts) • DemeterF: a traversal abstraction tool that supports structure-shy software • Generalized data abstraction • Intro to AP-F • Intro to DemeterF • important classes • important methods • Examples • Composition in DemeterF • Conclusions ABB

5. SDG Example • derivative(CNF{(2,0),(1,1)}, 0.70) • 4 variables maximum • would you like to buy it? • you will get two rights • you will receive a CNF R of the given type. • if you can satisfy fraction q of clauses in R, I will pay back q to you. CNF = SEQUENCE of clauses ABB

6. CNF{(2,0),(1,1)} raw materials 2: a 2: b 2: c 1: !a !b 1: !b !c 2: a 2: b 3: c 1: d 3: !a !b 9: !a !c 7: !b !c 1: !a !d 6: !b !d 6: !c !d 2: a 3: b 1: c 8: !a !b 6: !b !c Find best satisfaction ratio 1: a 1: b 8: !a !b ABB

7. CNF{(2,0),(1,1)} raw materials,finished products 2: a 2: b 2: c 1: !a !b 1: !b !c 2: a 2: b 3: c 1: d 3: !a !b 9: !a !c 7: !b !c 1: !a !d 6: !b !d 6: !c !d 6/8=0.75 2: a 3: b 1: c 8: !a !b 6: !b !c 17/20=0.85 price of 0.7 seems fair!? 1: a 1: b 8: !a !b 9/10=0.9 35/40 = 0.875 ABB

8. Oops • Our analysis was not thorough enough! • 2 kinds of uncertainty: • worst formula? • best assignment? ABB

9. Playing with the weights x: a x: b x: c x: d y: !a !b y: !a !c y: !b !c y: !a !d y: !b !d y: !c !d x =1, y=1 best assignment a=1, b=0, c=0, d=0: (1+6)/10=7/10=0.7 x = 2, y=1 best assignment a=1, b=1, c=0, d=0: (4+5)/14=9/14=0.64 derivative(CNF{(2,0),(1,1)}, 0.70) LOSS: 0.06 ABB

10. The SDG Game: Two high level views • Financial: Implement trading robots that survive in an artificial derivative market through offering derivatives, and buying and processing derivatives produced by other trading robots. • Biological: Implement organisms that survive in an artificial world through offering outsourced services, and consuming and processing outsourced services produced by other organisms. ABB

11. Virtual World • Trading Robots that need to survive in a virtual world of financial derivatives. • Trading Robots that don’t follow the world rules don’t survive. • Trading Robots are ranked based on their bank account. • Basic question: How to give a robot enough intelligence so that it can survive. ABB

12. Robot Structure • This past year all my students in my software development classes developed trading robots for the game SDG(Max). Each robot contains a: • Derivative buying agent • Derivative offering agent • Raw material production agent • Finished product agent (solves Max) • Winning in robot competitions strongly influences the final grade. • Game is interesting even if players are far from perfect. ABB

13. SDG(Max) • Choose a maximization problem. • Partition instances using predicates. • Derivative = (Predicate, Price in [0,1], Player). • Players offer and buy derivatives. • Buying a derivative gives you the rights: • to receive raw material R satisfying the predicate. • upon finishing the raw material R at quality q (trying to find the maximum solution), you receive q in [0,1]. ABB

14. cutting across behavior • Each robot contains a: • Derivative buying agent • needs break-even price • Derivative offering agent • needs break-even price • Raw material production agent • RM algorithm • Finished product agent (solves Max) • FP algorithm ABB

15. Analysis • Seller (before offering) • does not know the algorithmic strategy FP the buyer will use. With algorithmic strategy RM wants to hide best assignment using instance satisfying predicate pred, making it costly to find by algorithm FP of buyer. • Buyer (before buying) • does not know the algorithmic strategy RM the seller will use to hide the best assignment using instance satisfying predicate pred. Needs to find an algorithmic strategy FP that works well independent of RM. • Buyer (after buying) • wants to find the best assignment using FP. ABB

16. Analysis • How well can seller hide the best assignment and keep the satisfaction ratio low? ABB

17. Analysis of SDG(Max) The better the Buyer approximates the maximum, the harder the Seller has to hide it. tpred = lim n -> ∞ min all raw materials rm of size n satisfying predicate pred max all finished products fp produced for rm q(fp) The better the Seller hides the maximum, the harder the Seller has to work to find it. Seller approximates minimum efficiently Buyer approximates maximum efficiently ABB

18. AP-F/DemeterF: a useful tool to implement trading robots • OOP and Functional Adaptive Programming • Traversal abstraction library • similar to a library supporting the visitor design pattern • new: better traversal abstraction through multi-methods and traversal control ABB

19. Important Software Topic To Which We Contribute with AP-F • Generalizing the Data Abstraction Problem • Through Better Abstractions for Traversals ABB

20. Generalizing Data Abstraction well-accepted, since 70s • Data Abstraction Principle: the implementation of objects can be changed without affecting clients provided the interface holds. • Adaptive Programming (AP) Principle: the interface of objects can be changed without affecting clients provided the generalized interface holds. since 90s ABB

21. Traversal Abstraction • Traversal of an object is fundamental to any computation on the object. • Express WhereToGo using a domain-specific language for navigation control (a-la XPath). • But traversals perform diverse computations: Interpretation and Translation. ABB

22. Motto of AP-F • Law of Demeter: Talk only to your friends. • Law of AP-F: Listen only to your friends and selectively receive information from your superiors. • You might not need all the information you get from your friends. You only take what you need and what the communication protocol requires. ABB

23. What happens if you don’t use traversal abstraction? • You write a lot of boiler plate code! Related work: Scrap Your Boilerplate (SYB) started by Ralf Laemmel and Simon Peyton-Jones. • You tightly couple structure (changes frequently) and computation (more stable). That is against common sense. • You always deal with the worst-case scenario of AP-F. ABB

24. A Quick Intro to AP-F • An AP-F program is defined by two sets of functions which adapt behavior of a predefined traversal with a multiple dispatch function. The two sets of functions: • down • up ABB

25. interface ShapeI extends Remote { double get_x() throws RemoteException ; void set_x(int x) throws RemoteException ; double get_y() throws RemoteException ; void set_y(int y) throws RemoteException ; double get_width() throws RemoteException ; void set_width(int w) throws RemoteException ; double get_height() throws RemoteException ; void set_height(int h) throws RemoteException ; void adjustLocation() throws RemoteException ; void adjustDimensions() throws RemoteException ; } publicclass Shape implements ShapeI { protected AdjustableLocation loc; protected AdjustableDimension dim; public Shape() { loc = new AdjustableLocation(0, 0); dim = new AdjustableDimension(0, 0); } double get_x() throws RemoteException { returnloc.x(); } void set_x(int x) throws RemoteException { loc.set_x(); } double get_y() throws RemoteException { returnloc.y(); } void set_y(int y) throws RemoteException { loc.set_y(); } double get_width() throws RemoteException { returndim.width(); } void set_width(int w) throws RemoteException { dim.set_w(); } double get_height() throws RemoteException { returndim.height(); } void set_height(int h) throws RemoteException { dim.set_h(); } void adjustLocation() throws RemoteException { loc.adjust(); } void adjustDimensions() throws RemoteException { dim.adjust(); } } publicclass Shape { protecteddouble x_= 0.0, y_= 0.0; protecteddouble width_=0.0, height_=0.0; double get_x() { return x_(); } void set_x(int x) { x_ = x; } double get_y() { return y_(); } void set_y(int y) { y_ = y; } double get_width(){ return width_(); } void set_width(int w) { width_ = w; } double get_height(){ return height_(); } void set_height(int h) { height_ = h; } void adjustLocation() { x_ = longCalculation1(); y_ = longCalculation2(); } void adjustDimensions() { width_ = longCalculation3(); height_ = longCalculation4(); } } Write this coordinator Shape { selfex adjustLocation, adjustDimensions; mutex {adjustLocation, get_x, set_x, get_y, set_y}; mutex {adjustDimensions, get_width, get_height, set_width, set_height}; } class AdjustableLocation { protecteddouble x_, y_; public AdjustableLocation(double x, double y) { x_ = x; y_ = y; } synchronizeddouble get_x() { return x_; } synchronizedvoid set_x(int x) {x_ = x;} synchronizeddouble get_y() { return y_; } synchronizedvoid set_y(int y) {y_ = y;} synchronizedvoid adjust() { x_ = longCalculation1(); y_ = longCalculation2(); } } class AdjustableDimension { protecteddouble width_=0.0, height_=0.0; public AdjustableDimension(double h, double w) { height_ = h; width_ = w; } synchronizeddouble get_width() { return width_; } synchronizedvoid set_w(int w) {width_ = w;} synchronizeddouble get_height() { return height_; } synchronizedvoid set_h(int h) {height_ = h;} synchronizedvoid adjust() { width_ = longCalculation3(); height_ = longCalculation4(); } } portal Shape { double get_x() {} ; void set_x(int x) {}; double get_y() {}; void set_y(int y) {}; double get_width() {}; void set_width(int w) {}; double get_height() {}; void set_height(int h) {}; void adjustLocation() {}; void adjustDimensions() {}; } Modularization of crosscutting concerns In AP-F: concern 1: class graph concern 2: WhereToGo concern 3: WhatToDo Instead of writing this ABB

26. Intro to AP-F • Think of object computation as moving information down and pushing up. • Send information down as needed. • Default behavior: copy object. ABB

27. Motto of AP-F down A object graph Listen to C and D Receive from A B up Does NOT Receive from A C Receive from A D E Receive from A ABB

28. Illustration of up 19a 19 :C (w1+w2+w3+w4,2) :Cons (w1+w2+w3+w4,1) :E (w4,0) 1 20 2 18a 3 18 3a 13a (w1+w2+w3,1) :Cons after blue arrow up is active (like after) 17a :C (w2+w3,1) 4 12a 17 :Cons (w1,0) 5 13 14 12 16a (w2+w3,0) :Cons 15a 6 (0,0) :E (w1,0) :Empty 11a 7a 15 16 11 :E (w3,0) :Cons (w2,0) 8 7 10a 9a (w2,0) :E :Empty (0,0) 9 ABB 10

29. DemeterF • A significant class of applications can be coded modularly without tangled code and without having to write boiler plate code. • DemeterF has an optional code generation tool to generate Java classes from grammars (data binding) and traversal strategies (navigation binding). ABB

30. Family of Programs • Writing programs for a family of grammars. • Grammar also defines data types. ABB

31. Example of a Grammar // Parameters of Maximization Problem Type = "Typ"  /*...*/. Outcome = "O" /*...*/. Quality = "Q" /*...*/. RawMaterial = "R" <type> Type /*...*/ . Derivative = "D" <type> Type <price> double. FinishedProd = "F" <r> RawMaterial <o> Outcome. // Helper classes Pair(A,B) = <a> A <b> B. PlayerID = <id> int. // Main Game structures... SDG = "SDG"       <players> List(Player)       <account> List(Pair(PlayerID,Double))       <store> List(Pair(PlayerID,PlayerStore))       <config> Config. Player = "player" <id> PlayerID <name> String. PlayerStore = "pstore" <forSale> List(Derivative)                        <bought> List(BoughtDeriv). BoughtDeriv = "bought" <d> Derivative                        <seller> PlayerID                        <r> Option(RawMaterial)                        <f> Option(FinishedProd). ABB

32. Architectural View of DemeterF: Traversal-based Implicit Invocation (II) • POS • Data format abstracted away • Variability: Computation separate from data representation • Either can be changed independently of other • NEG • Efficiency ABB

33. DemeterF Function Objects • up methods • several parameters (from your friends) • default: reconstruct using (new) parameters • down methods • three parameters (where, which field/path, what came from above) • default: identity (return third parameter) ABB

34. What is new with DemeterF • easy generalization of functionality thanks to multimethods: exploit similarities between up methods at different nodes. Type-checking captures missing methods. • easy generalization of traversals • easy parallelization ABB

35. What Hinders Creation of Flexible Software • Manually following rules like: Follow the structure, follow the grammar. • Actively call traversal methods (explicit traversal problem). • Also leads to manual passing of arguments (explicit argument problem). ABB

36. Still follow the grammar? • Might have to write customized methods for every node. Extreme case. • Encode local information about structure in customized methods. ABB

37. Related Work • Modularity First: A Case for Mixing AOP and Attribute Grammars, Pavel Avgustinov, Torbjorn Ekman, Julian Tibble, Programming Tools Group, University of Oxford, AOSD 2008 • Other Oxford group (Jeremy Gibbons): OOPSLA 2008 • SYB home page • AP home page ABB

38. The 4 most important classes in DemeterF ID == id-for-trav-arguments and id-for-prims (int,String, etc.) ID inherits Bc Bc == ID + rebuild Use ID for type-unifying processing. Use Bc for type-preserving processing. Traversal(ID, EdgeControl) EdgeControl ABB

39. The 3 most important methods in DemeterF • for initiating a traversal: traverse • for modifying a traversal (WhatToDo): • down: • method is activated when the down method of the parent is complete. • up: • method is activated when all subtraversals have returned. ABB

40. Type-unifying: ID, Traversalup, traverse // Produces a string representation of the tree static class Str extends ID { String up(leaf l) { return ""; } String up(node n, int d, String l, String r) { return "("+d+" "+l+" "+r+")"; } public static String doStr(bst t){ return new Traversal(new Str()).<String>traverse(t); } } how generic? ABB

41. Example: Class Graph bst bst right left is-a is-a leaf node data int ABB

42. Example: Class bst public static abstract class bst { public abstract bst insert(int d); public static bst create(int ... ns) { bst t = new leaf(); for(int i:ns) t = t.insert(i); return t; } } ABB

43. Example: Class leaf // Functional Leafs public static class leaf extends bst { public bst insert(int d) { return new node(d, this, this); } } ABB

44. Example: class node // Functional Nodes public static class node extends bst { int data; bst left, right; public node(int d, bst l, bst r) { data = d; left = l; right = r; } public bst insert(int d) { if(d <= data) return new node(data, left.insert(d), right); return new node(data, left, right.insert(d)); } // Field classes... Must be public+static. public static class data { } public static class left { } public static class right { } } } ABB

45. Type-unifying: ID, Traversalup, traverse // Produces a string representation of the tree static class Str extends ID { String up(leaf l) { return ""; } String up(node n, int d, String l, String r) { return "("+d+" "+l+" "+r+")"; } public static String doStr(bst t){ return new Traversal(new Str()).<String>traverse(t); } } how generic? ABB

46. Type-preserving: Bc, Traversalup, traverse // Increments each int element and rebuilds the // resulting tree static class Incr extends Bc { int up(int i) { return i+1; } public static String doIncr(bst t){ return new Traversal(new Incr()). <bst>traverse(t); } } how generic? ABB

47. ID, Traversal, EdgeControlup, traverse // Find the minimum data element in the BST... keep going left static class Min extends ID { int up(node n, int d, leaf l) { return d; } int up(node n, int d, int l) { return l; } public static int min(bst t){ EdgeControl ec = EdgeControl.create( new Edge(node.class,"right")); ec.addBuiltIn(leaf.class); return new Traversal(new Min(), ec).<Integer>traverse(t); } } ABB

48. ID, Traversaldown, up, traverse // Produces a list of strings representing the paths to each // leaf element in the tree static class Paths extends ID { String down(node n, node.left f, String path) { return path+".left"; } String down(node n, node.right f, String path) { return path+".right"; } List<String> up(leaf l) { return new Empty<String>(); } List<String> up(node n, int d, List<String> l, List<String> r, String p) { return l.append(r).push(" "+d+" : "+p); } public static String doPaths(bst t){ return new Traversal(new Paths()). < List<String>>traverse(t, “root”); } } ABB

49. C# version • see separate file ABB

50. Type-unifying • Tree Height, bottom up: • Unifying type: (int height) • fold: max + 1 class Height extends ID{ int up(node t, int d, int l, int r){ return Math.max(l,r)+1; } int up(leaf l){return 0;} } ABB