This page intentionally left blank

1. This page intentionally left blank Please focus here Mats Björkman, MdH

2. Measurement-based Research Methods in Computer Engineering Mats Björkman Mälardalens Högskola Mats Björkman, MdH

3. Overview • Introduction • Experimental-based research methodology • Statistics • Measurements • Methodology • Examples • Pitfalls • Conclusions Mats Björkman, MdH

4. Introduction • Measurement-based research is founded in: • Experimental research methodology • Statistics Mats Björkman, MdH

5. Experimental-based research methodology • Overview (repetition) • Comments Mats Björkman, MdH

6. Experimental-based research methodology - Overview • Already the old Greeks… • Two main standpoints: • Rational methods, it all comes from the brain, everything can be thought out • Idealistic methods, everything we observe give us knowledge about the ideal world (e.g. Plato) Mats Björkman, MdH

7. Different Methodologies • Rational research meant thinking, thus deductive (logical) methodologies • Idealistic research meant observing and drawing conclusions, thus inductive methodologies Mats Björkman, MdH

8. Practice often a mixture • E.g. Astronomy, a combination of induction from observations and deduction through e.g. Mathematics Mats Björkman, MdH

9. Medieval times • Much debate around whether or not God stands above the laws of logic • The question “Why?” important • Research always seen through the glasses of religion Mats Björkman, MdH

10. The Scientific Revolution • Bacon, Copernicus, Kepler, Newton etc. • Focus on “How?” • Nature and religion may be treated separately as long as focus is on How • Led to the development of the “traditional” sciences Mats Björkman, MdH

11. Modern Science • Karl Popper - important philosopher • Science is a process of testing and refining hypotheses • Induction problem: Can experience be generalized? • Popper says ‘no’, experiments cannot prove general hypotheses Mats Björkman, MdH

12. Modern Science • Falsification is the most important feature of science according to Popper • Hypotheses cannot be proven, but they can be falsified by counter-examples • Theories are compared by their expressiveness and by their abilities to withstand falsification Mats Björkman, MdH

13. Modern Science • Since hypotheses cannot be generally proven, corroboration and statistics play important roles • Hypothetical-deductive research methods build on these views of science • However, corroboration versus verification can be and is discussed Mats Björkman, MdH

14. Hypothetical-deductive methods • Problem formulation • Hypothesis • Deduction to find evaluation criteria • Experiment(s)/observation(s) • Conclusion (corroboration/verification or falsification) Mats Björkman, MdH

15. Problem formulation • The research problem is formulated • Typically, a good problem is addressable through empirical studies Mats Björkman, MdH

16. Hypothesis • A hypothesis regarding the answer to the research question is formulated • This is the really creative part of research, scientific intuition and a good “educated guess” are important to success • (Popper: The more “risky” the hypothesis, the “better” the result.) Mats Björkman, MdH

17. Deduction • From the hypothesis, criteria are deduced, the criteria to be used to test the hypothesis Mats Björkman, MdH

18. Experiments/observations • The “hard work” part of research • Experiments are set up and/or observations are performed in order to corroborate/verify (or falsify) the hypothesis Mats Björkman, MdH

19. Corroboration/verification or falsification • Using the deduced criteria on the results of the experiments/observations leads to either corroboration/verification or falsification of the hypothesis Mats Björkman, MdH

20. Then iterate… • Modern scientific research is typically a series of hypothetical-deductive situations; each corroboration/ verification or falsification gives input to a new or modified research question etc. etc. • Through this process, our scientific theories are expanded and refined Mats Björkman, MdH

21. What is “the Truth”? • Experimental research is often more quantitative than qualitative • For quantitative results, confidence levels or margins of errors are used in attempts to “encircle” the Truth (should it exist) • Experiments are repeated and/or modified until confidence levels or error margins are satisfactory Mats Björkman, MdH

22. What is “the Truth”? • For qualitative results, we must also use statistics. • Even if we believe in induction and that the Truth is possible to find, there are always experimental errors and the like that makes 100% impossible to reach • Here too, repeated experiments are needed Mats Björkman, MdH

23. Conclusions • Experimental research is an iterative process • Potential falsification is important: experiments without risks are not interesting Mats Björkman, MdH

24. Conclusions • Examples: If we know the outcome beforehand, the experiment is of no scientific value. • If there is no way to falsify the hypothesis (e.g. pseudoscience), the experiment is of no scientific value. Mats Björkman, MdH

25. Experimental-based research methodology - Comments • 100 % does not exist in reality! Mats Björkman, MdH

26. Experimental-based research methodology - Comments • In reality, there is always a residual chance/risk that something really weird will happen Mats Björkman, MdH

27. Experimental-based research methodology - Comments • Therefore, Popper and his followers are maybe not wrong, but it is kind of irrelevant whether a hypothesis can be “generally proven” or not • (I’m trying to be provocative here…) Mats Björkman, MdH

28. Experimental-based research methodology - Comments • If I show that in X percent of all cases, some hypothesis Y holds… • …then according to Popper, this cannot prove the general case… • …but if 1 – X (the risk of Y not holding) is smaller than e.g. the risk of the world being ended by a comet… • …then who cares? Mats Björkman, MdH

29. Experimental-based research methodology - Comments • In reality, we always take calculated risks • If a hypothesis is true for all practical purposes, then it is an academic question (a philosophical question) whether or not the hypothesis is TRUE Mats Björkman, MdH

30. Experimental-based research methodology - Conclusions • Conclusion: The hypothetical-deductive method is the modern methodology in experimental research • However, not everyone agrees that hypotheses cannot be generally proven (and others don’t care…) Mats Björkman, MdH

31. An Experimental Example • The work I did for my PhD thesis around the performance of parallel implementations of communication protocols Mats Björkman, MdH

32. Parallel TCP and UDP stacks • On a shared-memory multiprocessor we implemented parallel TCP/IP/Ethernet and UDP/IP/Ethernet stacks • The performance behavior of these stacks gave rise to the research question “What factors limit the performance of these parallel implementations?” Mats Björkman, MdH

33. Performance limiting factors • In parallel processing, critical resources must be protected from simultaneous access, in our case by using locks • Hence, these critical sections were main suspects as performance limiting factor Mats Björkman, MdH

34. Our research hypothesis • Our hypothesis then was “locking is the main performance limiting factor” • We built a performance model using only locking and processing • If our hypothesis was right, then the model should behave like the real system Mats Björkman, MdH

35. Experiment • Our experiment was to run the model with the same input as the real implementation and compare results Mats Björkman, MdH

36. Results • For the TCP stack, results were fairly accurate for low numbers of processors (but far from perfect). • Conclusion: locking “is probably” one major factor (but not the only) Mats Björkman, MdH

37. Results • For the UDP stack, results differed widely. • Conclusion: locking is not a major factor here Mats Björkman, MdH

38. Results from conclusions • We need to rethink and refine (iterate) • Locking obviously is one factor, but not the only • Need to think again and formulate a new hypothesis Mats Björkman, MdH

39. New hypothesis • Next to contention for shared software resources, contention for shared hardware resources (e.g. buses, memory) is a likely candidate • New hypothesis: Contention for locks and contention for the bus/memory system are the two main factors Mats Björkman, MdH

40. New model • We then built a new model that captured the effects of both locking and bus/memory contention • The same evaluation criteria as before, model and reality should agree Mats Björkman, MdH

41. New results • For the new model, the TCP results were very good Mats Björkman, MdH

42. New results • While not perfect, UDP results also showed that our new model captured the main behavior of the UDP stack Mats Björkman, MdH

43. New results • Conclusion: Lock and bus/memory contention “are” the two main performance limiting factors for the observed implementations Mats Björkman, MdH

44. Statistics Statistics are used for many purposes: • Quantify results • Measure confidence • Statistics needed in corroboration process Mats Björkman, MdH

45. Statistics – Result quantification • Assume we are measuring some property P that has a certain (but to us unknown) value V • When measuring, we get a measured value V* Mats Björkman, MdH

46. Statistics – Result quantification • How is V and V* related? • It depends on our measurement methods • Ideally, our measurement method gives an exact result, i.e. V* = V Mats Björkman, MdH

47. Statistics – Result quantification • However, most measurement methods are: • Statistical by their nature • Inexact • Deliberately simplified (model) Mats Björkman, MdH

48. Statistics – Measurement methods • Statistical by nature: • Sampling a typical example: • Instead of observing a long and possibly continuous process, we take a number of snapshots • These snapshots are statistically representative of the process Mats Björkman, MdH

49. Statistics – Measurement methods • Example: Counting cars (vehicles) • We want to know the number of cars/vehicles passing outside Rosenhill on one day • Instead of counting for 24 hours, we can count 10 randomly chosen minutes and multiply by 144. Mats Björkman, MdH

50. Statistics – Measurement methods • Inexactness: • If our measurement tools have lower resolution than the property we are measuring, we introduce measurement errors Mats Björkman, MdH