Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003

1. Concrete Approaches to Quality Assessment:Moving Beyond Peer ReviewAugust 6, 2003 Howard BurrowsAutonomous Systems Institute Lee, NH

2. Outline Hardcore philosophical underpinnings Evaluating peer review Alternatives to peer review Justifying decisions about quality

3. Hardcore Underpinnings There is a dialectic between ontology and epistemology. John Burke Grammar of Motives

4. Ontology, Epistemology, Dialectic Ontology – what is taken to exist (what needs a name) Epistemology – what do you know about it Dialectic – thesis, antithesis, synthesis

5. Dialectic betweenOntology and Epistemology What counts as data when you form your beliefs The relation between evidence and justice. Are your beliefs “sensible” and “reasonable” The relation between metrics and evaluation

6. Outline Hardcore philosophical underpinnings Evaluating peer review Alternatives to peer review Justifying decisions about quality

7. Evaluating Peer Review Selecting reviewers; providing incentives Informed objectivity; conflict of interest Review criteria; comparing across panels Ronald N. KostoffUS Office of Naval Research

8. Selecting Reviewers: Peer vs Non-Peer Balancing interests: Economic- government, academia, industry, society Intellectual- engineers, scientists, policy agencies Practical- curiosity, problem based, market driven

9. Informed objectivity “Good old boy” consensus Natural bias often subtle, needs diversity With paradigm shifts, experience becomes a liability

10. US National Science FoundationReview Criteria Intellectual merit creative and original concepts Broader impacts benefits to society

11. Outline Hardcore philosophical underpinnings Evaluating peer review Alternatives to peer review Justifying decisions about quality

12. The Age of Unreason “Changes are not what they used to be.” from book “The Age of Unreason” Charles Handy, 1989 • Change by discontinuous leaps • Learning from the past dangerous • Evolution yes, but allow for revolution

13. NASA’s Earth Science Information Partners Courtesy Don Collins of the DAAC Alliance

14. Federalism Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly) Heterogeneous, diverse communities Data centers, academics, government, and industry Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

15. Outline Hardcore philosophical underpinnings Evaluating peer review Alternatives to peer review Justifying decisions about quality

16. Justifying Decisions about Quality Concept spaces and mapping Fact-value continuum? Fate of Knowledge (social construction)

17. Concept spaces and mapping

18. Fact-value continuum? Can we distinguish fact claims from value judgments? Are there really objective as opposed to subjective distinctions? Amartya Sen introduces value judgments in economics

19. Fate of knowledge Knowledge is social Cognitive processes are social (reasonable) Actions based on knowledge are justified through social processes.

20. Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

21. Promising developments Semantic web not words; rather “meaningful” data, concepts, and ideas. Science draws meaning from data; and has changed the way it justifies this. The semantic web offers to improve or supplant “peer” review (and education). The semantic web provides a “marketplace” for learning.

22. Profiles for discussion People Content Technology Values and priorities Economic Environment

23. Evolution to a Semantic Web HTML – Generic display of hypertext XML – Generic display of data RDF – Triples (subject verb object) OWL – Taxonomy, inference rules, and proofs

24. Changes in Science 1930 –Reason and Sense (logical positivism) 1960 – Beyond Reason (linguistic turn) 1990 – Social Construction (peer review) 2020 – Back to Reason and Sense

25. “Peers” and “Educators” Ontology – Locally coherent naming structures Realism – “Reifying” the structures Under standing – Logical foundations Knowledge – Justifying decisions (personal beliefs)

26. Business Plan and “Marketplace” Sales agents – Know-bots negotiate deals Value chains – Multiple entities in assembly line Public choice – Market forces and “fair” voting rules Learning Economy – From food chain to ecosystem

27. Back to the Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

28. Problems in Public Funding National Archives - GPRA - Curriculum - 2% GNP - What data is valuable? Is science making progress? What is “educated”? How much to spend?

29. The Age of Unreason “Changes are not what they used to be.” from book “The Age of Unreason” Charles Handy, 1989 • Change by discontinuous leaps • Learning from the past dangerous • Evolution yes, but allow for revolution

30. Category I Earth Science Information Partners Courtesy Don Collins of the DAAC Alliance

31. structure Data Structure Features Models Semantic Images Video Audio Multimedia Formats Layout Regions Segments Mosaics Relationship(Spatio-temporal) Color Texture Shape Motion Camera motion Clusters Classes Collections Probabilities Confidences Objects Events Actions People Labels Relationship MPEG-7: Metadata for Content Description Signal Data Features Model Semantics Courtesy John Smith, IBM

32. Data, Community, Public Use Application solution provider Science data provider Backbone data center Users Farmers USDA Johns Hopkins NCDC Yes, treat your crops today Real-time data Real-time riskmap $$$ on pest control GHRC Real-time data $$ on fire ants studies $ on real time data USGS Historical data Technology provider IBM Software Courtesy Yuan-Chi Chang, IBM

33. Other Cross Discipline ESIP’s Science Oceanography (2) Terrestrial Studies (4) Climate (3) Technology (3) Public Use • Education (3) • Regional Policy (4) • Public Health • Media • Legal

35. Community I • “A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.” E. O. Wilson Consilience: The Unity of Knowledge

36. Community II • “It is the disorder of the scientific community—the laminated, finite, partially independent strata supporting one another; it is the disunification of science—the intercalation of different patterns of argument—that is responsible for its strength and coherence..” Peter Galison Image and Logic, 1997

37. Federalism Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly) Heterogeneous, diverse communities Data centers, academics, government, and industry Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

38. Federations Preserve Heterogeneity Economic- government, academia, industry Intellectual- academic disciplines, policy agencies Practical- problem based, market driven

39. Economic Model Public funding caps out at less than 3% of the GNP Dynamic “pricing” is needed to demonstrate value to end-users “Market forces” or other methods of public choice provide robust mechanism to rank options

40. Trading Zones Economic- government, academia, industry Intellectual- engineers, scientists, policy agencies Practical- curiosity, problem based, market driven

41. Peer Review Nurturing cooperation vs competition Economic- government, academia, industry Intellectual- engineers, scientists, policy agencies Practical- curiosity, problem based, market driven

42. Back to the Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

43. Business Plan and “Marketplace” Sales agents – Know-bots negotiate deals Value chains – Multiple entities in assembly line Public choice – Market forces and “fair” voting rules Learning Economy – From food chain to ecosystem