1 / 84

Virtual Scientific-Community-Based Foundations for Popperian e-Science

Virtual Scientific-Community-Based Foundations for Popperian e-Science. Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston. inspired by ScienceWISE. Ontology. Organization. Mathematics. Computer Science. Mechanism Design. Mathematical Logic. Programming.

stefan
Download Presentation

Virtual Scientific-Community-Based Foundations for Popperian e-Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Virtual Scientific-Community-Based Foundations for Popperian e-Science Karl Lieberherr Ahmed Abdelmeged Northeastern University, CCIS, PRL, Boston

  2. inspired by ScienceWISE Ontology Organization Mathematics Computer Science Mechanism Design Mathematical Logic Programming Game Theory MetaGaming ExtensiveForm Socio-Technical System The Global Brain Dialog Games IF Logic

  3. A claim is … • information about one’s performance when interacting with another clever being in a specific domain. • information about the performance of one’s program. Crowdsourcing

  4. Outline • Introduction Introduction • Theoretical Background Theory • Methods of Exploration Methods • Results Results • Conclusions and Future Work Conclusion

  5. Conclusion Introduction Theory Methods Results Introduction SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, tunedit.org, Renaissance, … • Make SCG a part of cyber-infrastructure (e-science) to support teaching and innovation in constructive domains. • SCG usage for teaching • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but for competitive innovation) • Theoretical Properties of SCG • Take Home: EASY USE WITH STANDARD TOOLS • group research, teaching, (intelligent) crowd sourcing

  6. Conclusion Introduction Theory Methods Results Popper • One of the philosophers of science who has had a big impact. • Popper’s solution: Falsification: A claim is falsifiable if you can imagine an observation that would cause you to reject the claim. • That a claim is "falsifiable" does not mean it is false; rather, that if it is false, then some observation or experiment will produce a reproducible result that is in conflict with it.

  7. Conclusion Introduction Theory Methods Results What SCG helps with • Build and maintain knowledge bases (sets of claims believed to be true). • How to identify experts? • How to decide if an answer is worthwhile? • Use scholars to choose the winners • How to organize egoistic scholars to produce social welfare: knowledge base and know-how how to defend it. • The scholars try to reverse engineer the solutions of winning scholars.

  8. Abstraction from 4 Examples • From a CS journal paper • Insilico experiment • From kaggle.com: Facebook competition • From a calculus problem

  9. Example 1: From an Abstract of a 2005 Journal Paper • An instance of a constraint satisfaction problem (CSP) is variable k-consistent if any subinstance with at most k variables has a solution. • For a fixed constraint language L, r(k,L) is the largest ratio such that any variable k-consistent instance has a solution that satisfies at least a fraction of r(k,L) of the constraints.

  10. Example 1 • From a 2005 TCS paper: Locally Consistent Constraint Satisfaction Problems by Manuel Bodirsky and Daniel Kral. • Example • L = CNF • k = 1 • What is r(1,CNF)? • Claims: r(1,CNF) = 0.6, r(1,CNF) = 0.7

  11. Example 1: Making a game to determine r(1,CNF) • Observation: claims are falsifiable playing a two person game.

  12. Example 2: Claim involving Insilico Experiment Claim InsilicoExperimental(X,Y,q,r) I claim, given raw materials x in X, I can produce product y in Y of quality q and using resources at most r. Crowdsourcing

  13. Example 2: Making a game to determine InsilicoExperimental(X,Y,q,r) • Observation: claims are falsifiable playing a two person game.

  14. Conclusion Introduction Theory Methods Results Example 3: Data mining • Facebook competition from Kaggle.com: • Given a social network graph x with deleted edges and the original social network graph gs (secret, from a family X of social networks) • guess the complete social network graph y • quality(x, gs, y) = mean average precision (adapted from IR) • I claim I can achieve a mean average precision of q • for social graphs in family X: DM1(X,q) • for a specific reduced social graph: DM2(x,q)

  15. Example 3: Making a game to determine the optimal claims • Observation: claims DM1(X,q) and DM2(x,q) are falsifiable playing a two person game.

  16. Conclusion Introduction Theory Methods Results Example 4: Specker • Claims: • Specker(set X, set Y(X), function f(X,Y)->[0,1], constant c): ForAll x in X Exists y in Y(X): f(x,y)≥c • Example 1 • X = Conjunctive Normal Forms with various restrictions • Y(X) = Assignments to CNFs • f(x,y) = fraction of satisfied clauses in x under y • c in [0,1], e.g., c= 0.61 • Example 2 (a reduction of example 1) • X = [0,1] • Y(X) = [0,1] • f(x,y)=x*y+(1-x)(1-y^2)) • c in [0,1], e.g., c=0.61

  17. Example 4: Specker • Observation: claims Specker(X,Y,f,c) are falsifiable playing a two person game.

  18. What is the abstraction? • Sets of claims • Claims are falsifiable • …

  19. Playgrounds • Each playground defines: • domain • claims language • specific protocol • data exchanged • configuration data PG1 PG2 SC2 SC4 claims C21 C22 C23 … claims C11 C12 C13 … SC1 SC3 D1 SC5 SC1 RP1 RP2 D2 SC1 SC1 • SCG defines: • refutation protocol interface • generic rules for all playgrounds

  20. Example 1: Making a game to determine r(1,CNF) • Observation: claims are falsifiable playing a two person game. defendable = !refutable • propose r(1,CNF) = 0.7 • refutable • propose r(1,CNF) = 0.6 can be strengthened to r(1,CNF) = 0.61 which is defendable (refutation attempts will be unsuccessful) • propose r(1,CNF) = (sqrt(5)-1)/2 ~ 0.618 … optimum: defendable and cannot be strengthened

  21. Who are the scholars? • Scientists • Students in a class room • High school • University • Members of the Gig Economy • Between 1995 and 2005, the number of self-employed independent workers grew by 27 percent. • Potential employees (Facebookon kaggle.com) • Anyone with web access; Intelligent crowd.

  22. Kaggle.com Competitions2012 • Facebook recruiting competitions • Task: Data scientist • Reward: Job • Teams: 197 • Heritage Health Prize • Task: Hospital admissions • Reward: $ 3 million • Teams: 1118 • Chess ratings – Elo versus the Rest of the World • Task: Predict outcome of chess games • Reward: $ 617 • Teams: 257

  23. Kaggle.com Competitions2012 • Eye Movements Verification and Identification • Task: Identify people • Reward: Kudos • Teams: 51 • EMC Data Science Global Hackathon • Task: Air Quality Prediction • Reward $ 7030 • Teams: 114

  24. Conclusion Introduction Theory Methods Results What Scholars think about! • If I propose claim C, what is the probability that • C is successfully refuted • C is successfully strengthened • If I try to refute claim C, what is the probability that I will fail. • If I try to strengthen claim C, what is the probability that I will fail? • Scholars are free to invent; game rules don’t limit creativity!

  25. Conclusion Introduction Theory Methods Results Degree of automation with SCG(X) avatar Bob scholar Alice degree of automation used by scholar 1 0 no automation human plays some automation human plays full automation avatar plays transfer to reliable, efficient software more applications: test constructive knowledge

  26. happy = can be creative, can thrive, have opportunity to learn, not ignored Organizational Problem Solved • How to design a happy scientific community that encourages its members to really contribute. • Control of scientific community • tunable SCG rules • Specific domain, claim definition to narrow scope.

  27. Playground • defines • what is wanted, e.g., an algorithm S in a particular domain (inputs/outputs) • evaluation, e.g., how S is evaluated (quality) • claims, e.g., what kind of claims can be made about S (expression with quantifiers) • A playground defines WHAT is desired and the scholars/avatars define the HOW.

  28. Conclusion Introduction Theory Methods Results Theory • Extensive Form Representation of Game • Properties • Community Property: All faulty actions can be exposed. • SCG Equilibrium • Convergence to optimum claim

  29. Conclusion Introduction Theory Methods Results Extensive-form representation • the players of a game: 1 and 2 • for every player every opportunity they have to move • what each player can do at each of their moves • what each player knows for every move • the payoffs received by every player for every possible combination of moves

  30. Large Action Spaces • Thick arrows mean: select from a usually large number of choices 1 2

  31. Conclusion Introduction Theory Methods Results Refutation Protocol • Collects data given to predicate p. Alternates. refute(C,proposer,other) other tries to make p false while proposer tries to make p true. p false means successful refutation. p true means successful defense. p(C, …)?(1,-1):(-1,1) claim payoff for proposer if p true (defense) payoff for other if p true (defense) payoff for proposer if p false (refutation) payoff for other if p false (refutation)

  32. Conclusion Introduction Theory Methods Results 1 scholar 2 scholar 1 SCG Core refute(C, proposer,other) p(…)?(proposer,other): (proposer,other) propose claim C from Claims agree attempt C 2 refute attempt C strengthen attempt C’ => C refute(C,1,2) p(C, …)?(1,-1):(-1,1) refute(C’,2,1) refute(C,2,1) p(C, …)?(1,-1):(-1,1) p(C’, …)?(1,-1):(-1,1) p(C, …)?(0,0):(1,-1) p(C’, …)?(-1,1):(1,-1)

  33. Conclusion Introduction Theory Methods Results Game Rules for Playground • All objects exchanged during protocol must be legal and valid. • Each move must be within time-limit. • Scholar who first violates a playground rule, loses.

  34. Conclusion Introduction Theory Methods Results Logic with Soundness claims sentences good bad not just true/false claims, but optimum/non-optimum claims: good: true/optimum bad: false/non-optimum Crowdsourcing

  35. Conclusion Introduction Theory Methods Results Scientific Community Game Logic with Community Principle claims sentences good bad disagreed by two scholars agreed by two scholars there exists a two-party certificate to expose misclassification Crowdsourcing

  36. Comparison Logic and SCG Logic Scientific Community Game sentences = claims good bad evidence for goodness defense, checkable uncertainty of defense evidence for badness refutation, checkable uncertainty of refutation Personified sentences • sentences • true • false • proof for being true • proof system, checkable • guaranteed defense • proof for being false • proof system, checkable • guaranteed refutation • Universal sentences Crowdsourcing

  37. Conclusion Introduction Theory Methods Results Community Property • For every faulty decision action there exists an exposing reaction that blames the bad decision. • Reasons: • We want the system to be egalitarian. • It is important that clever crowd members can shine and expose others who don’t promote the social welfare of the community. • Faulty decisions must be exposable. It may take effort.

  38. Conclusion Introduction Theory Methods Results Methods of Exploration • Developed Platform SCG Court = Generator of teaching/innovation playgrounds • http://sourceforge.net/p/generic-scg/code-0/11 0/tree/GenericSCG/ • Developed numerous playgrounds for avatars. • Developed Algorithms Course using Piazza based on SCG Court experience • role of scholar played by humans • piazza.com: encourages students to answer each other’s questions.

  39. Conclusion Introduction Theory Methods Results Avatar Interface • AvatarI • public List<Claim> propose(List<Claim> forbiddenClaims); • public List<OpposeAction> oppose(List<Claim> claimsToBeOpposed); • public InstanceI provide(Claim claimToBeProvided); • public SolutionI solve(SolveRequest solveRequest); • from http://sourceforge.net/p/generic-scg/code-0/110/tree/GenericSCG/src/scg/scg.beh

  40. Conclusion Introduction Theory Methods Results Instance Interface (Domain) • InstanceI • boolean valid(SolutionI solution, Config config); • double quality(SolutionI solution);

  41. InstanceSet Interface (Domain) • InstanceSetI • Option<String> belongsTo(InstanceI instance); • Option<String> valid(Config config); }}

  42. Protocol Interface • ProtocolI • double getResult(Claim claim, SolutionI[] solutions, InstanceI[] instances); • ProtocolSpec getProtocolSpec(); • boolean strengthenP(Claim oldClaim, Claim strengthenedClaim);

  43. Claim Class, for all playgrounds • Claim • public Claim(InstanceSetI instanceSet, ProtocolI protocol, double quality, double confidence)

  44. Conclusion Introduction Theory Methods Results Protocol Library • ExistsForAll.java • ForAllExists.java • Renaissance.java • AsGoodAsYou.java • Survivor.java

  45. Second Method: Piazza ExperienceGale-Shapley • We propose that, for all integers n > 0, the maximum iterations the Gale-Shapely algorithm with n men and n women can produce is n(n-1)+1.Note: Thus far, the inputs used for all other claims arrives at only (n(n+1))/2.

  46. Piazza Experience • Leaf Covering: Improved running time from quadratic to constant time.

  47. Conclusion Introduction Theory Methods Results Results SCG = Scientific Community Game = Specker Challenge Game • Explanation: SCG as a general pattern behind many different competitions: topcoder.com, kaggle.com, Operations Research Competitions, tunedit.org, http://eterna.cmu.edu/ … • SCG usage for teaching using forum • Innovation Success with Undergraduates using SCG on piazza.com: Qualitative Data Sources & Analysis • Avatar competitions are not for teaching (but good for competitive innovation) • Theoretical Properties of SCG

  48. Conclusion Introduction Theory Methods Results Competition tuning: minimum • For each scholar • count claims that were successfully opposed (refuted or strengthened) • encourages strong claims • gather information from competitors for free • count claims that were not successfully agreed • Good for teaching • students want minimum competition • good students want to build social capital and help weaker students

  49. Conclusion Introduction Theory Methods Results Piazza Results • Do not give hints at solutions. This significantly decreased the amount of discourse taking place.

  50. Conclusion Introduction Theory Methods Results Conclusions and Future Work • We propose a systematic gamification of teaching STEM domains: • Design an SCG playground where the winning students demonstrate superior domain knowledge. STEM = Science, Technology, Engineering, and Mathematics

More Related