1 / 67

A Popperian Platform for Programming and Teaching the Global Brain

A Popperian Platform for Programming and Teaching the Global Brain. Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston. A claim is …. information about one’s performance when interacting with another clever being in a specific domain.

Download Presentation

A Popperian Platform for Programming and Teaching the Global Brain

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A Popperian Platform for Programming and Teaching the Global Brain Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston

  2. A claim is … • information about one’s performance when interacting with another clever being in a specific domain. • information about the performance of one’s program. Crowdsourcing

  3. Outline • Introduction Introduction • Theoretical Background Theory • Methods of Exploration Methods • Results Results • Conclusions and Future Work Conclusion

  4. Conclusion Introduction Theory Methods Results Introduction SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … • Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. • SCG usage for teaching • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but for competitive innovation) • Theoretical Properties of SCG

  5. Conclusion Introduction Theory Methods Results Specker • Claims: • Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c • Example 1 • X = Conjunctive Normal Forms with various restrictions • Y(X) = Assignments to CNFs • f(x,y) = fraction of satisfied clauses in x under y • c in [0,1], e.g., c= 0.61 • Example 2 • X = [0,1] • Y(X) = [0,1] • f(x,y)=x*y+(1-x)(1-y^2)) • c in [0,1], e.g., c=0.61

  6. Conclusion Introduction Theory Methods Results Popper • One of the philosophers of science who has had the biggest impact. • Popper’s solution: Falsificationism: A claim is falsifiable if you can imagine an observation that would cause you to reject the theory. • That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it.

  7. Conclusion Introduction Theory Methods Results What SCG helps with • How to identify experts? • How to decide if an answer is worthwhile? • Use scholars to choose the winners • How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. • The scholars try to reverse engineer the solutions of winning scholars.

  8. Claim involving Experiment Claim ExperimentalTechnique(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. Crowdsourcing

  9. Who are the scholars? • Scientists • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees (Facebookon kaggle.com) • Anyone with web access; Intelligent crowd.

  10. Kaggle.com Competitions2012 • Facebook recruiting competitions • Task: Data scientist • Reward: Job • Teams: 197 • Heritage Health Prize • Task: Hospital admissions • Reward: $ 3 million • Teams: 1118 • Chess ratings – Elo versus the Rest of the World • Task: Predict outcome of chess games • Reward: $ 617 • Teams: 257

  11. Kaggle.com Competitions2012 • Eye Movements Verification and Identification • Task: Identify people • Reward: Kudos • Teams: 51 • EMC Data Science Global Hackathon • Task: Air Quality Prediction • Reward $ 7030 • Teams: 114

  12. Conclusion Introduction Theory Methods Results What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? • Scholars are free to invent; game rules don’t limit creativity!

  13. Conclusion Introduction Theory Methods Results Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge

  14. happy = can contribute, not ignored, can be creative, can thrive, have opportunity to learn Organizational Problem Solved • How to design a happy scientific community that encourages its members to really contribute. • Control of scientific community • tunable SCG rules • Specific domain, claim definition to narrow scope.

  15. Conclusion Introduction Theory Methods Results What is a loose collaboration? • Scholars can work independently on an aspect of the same problem. • Problem = decide which claims in playground to oppose or agree with. • How is know-how combined? Using a protocol. • Alice claimed that for the input that Alice provides, Bob cannot find an output of quality q. But Bob finds such an output. Alice corrects. • Bug reports that need to be addressed and corrections. Playground = Instantiation of Platform

  16. Conclusion Introduction Theory Methods Results Theory • Extensive Form Representation of Game • Community Property: All faulty actions can be exposed. • SCG Equilibrium • Convergence to optimum claim

  17. Conclusion Introduction Theory Methods Results Extensive-form representation • the players of a game: 1 and 2 • for every player every opportunity they have to move • what each player can do at each of their moves • what each player knows for every move • the payoffs received by every player for every possible combination of moves

  18. Large Action Spaces • Thick arrows mean: select from a usually large number of choices 1 2

  19. Conclusion Introduction Theory Methods Results Refutation Protocol • Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation)

  20. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)

  21. Reinterpret Refutation • Refutation leads to successful strengthening or successful agreement.

  22. Conclusion Introduction Theory Methods Results Game Rules for Playground • All objects exchanged during protocol must be legal and valid. • Each move must be within time-limit. • Scholar who first violates a playground rule, loses.

  23. Example: Independent Set • Alice = proposer, Bob = other. • Protocol / claim: AtLeastAsGood. Alice claims to be at least as good as Bob at IS. • Bob provides undirected graph G. • Bob computes independent set sB for G (secret). • Alice computes independent set sA for G. • Alice wins, if size(sA) >= size(sB) (= p(sA,sB)).

  24. Conclusion Introduction Theory Methods Results Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Crowdsourcing

  25. Conclusion Introduction Theory Methods Results Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars there exists two-party certificate to expose misclassification Crowdsourcing

  26. Comparison Logic and SCG Logic Scientific Community Game sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences • sentences • true • false • proof for being true • proof system, checkable • guaranteed defense • proof for being false • proof system, checkable • guaranteed refutation • Universal sentences Crowdsourcing

  27. Conclusion Introduction Theory Methods Results Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort.

  28. Community PropertyAlternative formulation • If all decisions by Alice are not faulty, there is no chance of Alice losing against Bob. • if Alice is perfect, there is no chance of losing. • If there exists a faulty decision by Alice, there is a chance of Alice losing against Bob. • egalitarian game

  29. Summary: faulty decisions • propose(Alice,C),C=false • propose(Alice,C),C=not optimum, C=true • refute(Alice,Bob,C),C=true • strengthen(Alice,Bob,c,cs),c=optimum • strengthen(Alice,Bob,c,cs),c=false • agree(Alice,Bob,c),c=false • agree(Alice,Bob,c),c=not optimum, c=true

  30. Conclusion Introduction Theory Methods Results SCG Equilibrium • Points (reputations) of scholars are stable. • The science does not progress; bugs are not fixed, no new ideas are introduced. • Extreme, desirable situation: All scholars are perfect: they propose optimal claims that can neither be strengthened nor refuted.

  31. Claims: convergence to optimum over strengthening 1 false claims (refutable) correct valuation quality strengthening true claims (defendable) 31 0

  32. Conclusion Introduction Theory Methods Results Convergence • Given a family of claims totally ordered with respect to strengthening and containing an optimum claim. If every faulty action is exposed, convergence to the optimum claim is guaranteed by using a search strategy such as binary search.

  33. Conclusion Introduction Theory Methods Results Methods of Exploration • Developed Platform SCG Court = Generator of teaching/innovation playgrounds • http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ • Developed numerous playgrounds for avatars. • Developed Algorithms Course using Piazza based on SCG Court experience • role of scholar played by humans • piazza.com: encourages students to answer each other’s questions.

  34. Conclusion Introduction Theory Methods Results Avatar Interface • AvatarI • public List<Claim> propose(List<Claim> forbiddenClaims); • public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); • public InstanceI provide(Claim claimToBeProvided); • public SolutionI solve(SolveRequest solveRequest); • from http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/src/scg/scg.beh

  35. Conclusion Introduction Theory Methods Results Instance Interface • InstanceI • boolean valid(SolutionI solution, Config config); • double quality(SolutionI solution);

  36. InstanceSet Interface • InstanceSetI • Option<String> belongsTo(InstanceI instance); • Option<String> valid(Config config); }}

  37. Protocol Interface • ProtocolI • double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); • ProtocolSpec getProtocolSpec(); • boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

  38. Claim Class, for all playgrounds • Claim • public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

  39. Conclusion Introduction Theory Methods Results Protocol Library • ExistsForAll.java • ForAllExists.java • Renaissance.java • AsGoodAsYou.java • Survivor.java

  40. Conclusion Introduction Theory Methods Results Claim Kinds in SCG Court • Claim C(instance, q) • Claim C(InstanceSet, q) • Claim MaxResource(Algorithm,InstanceSet,n,ResExp) • Claim MinResource(Algorithm,InstanceSet,n,ResExp) • Claim IAmTheBest(), AtLeastAsGoodAsYou() • Claims from predicate logic and some of its generalizations (IF Logic)

  41. Piazza: Gale-Shapley • Gale-Shapley (GS) algorithm is fixed. • Alice provides secret instance of size n, solves it with GS (counts number of iterations) and publishes it: Ac(n). • Bob provides secret instance of size n, solves it with GS and publishes it Bc(n). • Alice wins if Ac(n) > Bc(n).

  42. Reverse EngineeringPlayground Design Issues • input, output, algorithm unknown • for some problems input-output pairs help to invent the algorithm • input unknown, output, algorithm • e.g., output for worst-case running-time, find input which is worst-case • for some problems finding

  43. Conclusion Introduction Theory Methods Results Playground Design • Make sure that the scholars don’t have to reveal their secrets. If they do. • Remember who was first? • Provenance of winning idea.

  44. Conclusion Introduction Theory Methods Results SCG with piazza.com • use piazza.com for posting • playground definition by playground designer • for scholars • claims • executions of refutation protocols • Community policing • Used JSON to exchange claims, instances and solutions. • Used links when they did not fit in piazza window.

  45. Conclusion Introduction Theory Methods Results JSON example • We define the JSON notation for defining a preference p as follows: • {"n":3, • "manPref" : [[2,1,0],[1,0,2],[0,1,2]], • "womanPref : [[2,1,0],[1,3,2],[3,1,2]] } • Easy to parse in a variety of programming languages and also readable by humans.

  46. Piazza ExperienceGale-Shapley • We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1.Note: Thus far, the algorithm used for all other claims arrives at only (n(n+1))/2.

  47. Piazza Experience • Leaf Covering: Improved running time from quadratic to constant time.

  48. Conclusion Introduction Theory Methods Results Results SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … • SCG usage for teaching using forum • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but good for competitive innovation) • Theoretical Properties of SCG

  49. Competition tuning: minimum • For each scholar • count claims that were successfully opposed (refuted or strengthened) • encourages strong claims • gather information from competitors for free • count claims that were not successfully agreed • Good for teaching • students want minimum competition • good students want to build social capital and help weaker students

  50. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 High competition refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)

More Related