1 / 32

"Methodology" of Research in Computer Science

"Methodology" of Research in Computer Science. Lionel Brunie National Institute of Applied Sciences (INSA) LIRIS Laboratory/DRIM Team – UMR CNRS 5205 Lyon, France http://liris.cnrs.fr/lionel.brunie. Agenda. ( Disclaimer) Discovery vs. invention

emier
Download Presentation

"Methodology" of Research in Computer Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. "Methodology"ofResearch in Computer Science Lionel Brunie National Institute of Applied Sciences (INSA) LIRIS Laboratory/DRIM Team – UMR CNRS 5205 Lyon, France http://liris.cnrs.fr/lionel.brunie

  2. Agenda • (Disclaimer) • Discovery vs. invention • What is a research result in computer science? • What is a research paper in computer science? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes • How is research evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  3. Agenda  Disclaimer Discovery vs. invention What is a research result in computer science? What is a research paper in computer science? General structure(s) of a research paper in CS Searching for black boxes, traps and black holes How is research evaluated? And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS 3

  4. Disclaimer • This is NOT a lecture ! • I apologize not to be an epistemologist • I am just a practitioner of computer science that have co-authored some research papers and read/evaluated several hundreds ones • This open discussion is absolutely NOT objective • Goal : open the windows and discuss how we ACTUALLY work (and not how we should work…) • All questions, remarks, contradictions are welcome!!! Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  5. Agenda • Disclaimer •  Discovery vs. invention • What is a research result in computer science? • What is a research paper in computer science ? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes • How is research evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  6. Discovery vs. Invention • My feeling is that there are two main ways of practicing science : discovery vs. invention • Biologists, physicists, chemists, researchers in psychology… are discoverers • (at least, software) Computer scientists, researchers in nanotech, researchers in process engineering or in industrial engineering… are inventors • However inventing often needs a priori discoveries (e.g., of your target/domain of interest); discovering often needs inventing (e.g., measuring instruments) • Mathematicians… are mathematicians :-) Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  7. Discovering (just two words) • Understanding the world : what are atoms constituted of, why a disease in inherited, why do people have dreams, etc. • Understanding means first asking questions, then observing, inquiring, modeling, evaluating • At the end of the research process, one has an answer to the initial question (question of research) • An answer is the most often not definitive. It is an explanation of a small piece of the natural world under some hypotheses • Famous recent example: the “proof of the existence” of Higgs’ boson at CERN/LHC Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  8. Inventing (1/2) • Software computer science produces inventions • Computers do not exist by themselves. They have been created by human beings => there is nothing to discover in a computer or in a software • The objective of research in CS is “just” to make computers/computer networks and computer applications more efficient, more easy to use, more reliable, more powerful… i.e. more useable/useful Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  9. Inventing (2/2) • As a consequence, a research result in (software) CS has no (real) intrinsic value. It has only the value that the research community and/or the society gives to it. A useless invention has no value! • This has to be moderated when one considers research in fundamental and theoretical CS where people study formal problems => context actually close to research in maths • This has also to be moderated when one considers social- or bio- or something-inspired CS where researchers attempt to understand and model an “external” (social or biological or…) phenomenon Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  10. Agenda • Disclaimer • Discovery vs. invention •  What is a research result in computer science? • What is a research paper in computer science? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes • How is research evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  11. A difficult answer… • The endless debate about fundamental research, basic research, applied research, finalized research… • A first and definitive answer: a result has some scientific value as soon as it has been accepted by a scientific publication (a scientific committee), i.e., research is a social activity (hence, fashions, trends…) • A second answer: a result has some scientific value if it interests people and if it is novel (original) (and if it has some genericity) i.e., refer to the scientific method • The most difficult (and so the most interesting) problems have often been raised by applications, e.g. medical applications. Investigating these problems have allowed going through theoretical scientific developments Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  12. Agenda • Disclaimer • Discovery vs. invention • What is a research result in computer science? •  What is a research paper in computer science? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes • How is research evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  13. Structure of a research paper in CS • Introduction-Motivations • State of the art • Proposal • Experiments • Discussion • Conclusion and future work => Computing the D (or d or e)! Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  14. 3 (+1) key sections (1/4) • Having 1 very good idea in a scientist’s life is… very good; having two is exceptional • One paper => one problem and at most one (two) contribution(s) to defend • A first key section: the statement of the problem • Why is the problem important? Remark: notion of fashion (e.g., Big Data) • Why is the problem difficult (→ link to the state of the art)? • What are the hypotheses? What is/are the context/conditions? • What are the objectives i.e., what are the performance/quality criterions with respect to which the contribution should be evaluated? • Then, add the outline of the contribution Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  15. 3 (+1) key sections (2/4) A second key section: the state of the art • A state of the art must not be “flat” or general purpose • It must be focused on your scientific and applicative targets and “critical and compared/taxonomic” • Critical: the contribution/limitations of each cited paper should be analyzed wrt your problem • Compared: papers should be organized in taxonomies • The state of the art establishes/describes the scientific basis wrt which your work should be compared, i.e., your // will be evaluated • Ideally, all pertinent papers should be analyzed. Practically, it is often untracktable • “Making the bibliography” is a key component of the scientific method Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  16. 3 (+1) key sections (3/4) • A third key section: the discussion • Goal of the discussion: analyzing your contribution/proposal • Wrt to your initial objective/motivation • Wrt to the state of the art • Wrt the experiments you have run… and have not run • Wrt the implementation choices you have made • Wrt genericity • A discussion is not a conclusion • The discussion is often distributed over the paper ( :-( ?) • Do not believe authors: make your personal discussion! Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  17. 3 (+1) key sections (4/4) • A fourth obvious section : your contribution • Must be “sound” • Must be comprehensive and complete • Should allow people to reproduce the experiments you use • Must be self-contained • Kill ambiguities Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  18. A short focus on the“Introduction – Motivation” section • Inventing some “thing” that interests nobody has no real sense • Inventing some “thing” that addresses a problem that has an existing satisfactory solution has a limited interest • To be interesting, a work/paper should address a problem : • With a high (scientific or economic or social or…) value • And no (perfect) solution • (already noted) A paper should list the criteria wrt which authors consider that their proposal is to be evaluated Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  19. Experiments (1/3) • The often dark side of research in CS • A first reason : cultural/educational : computer scientists have an unsatisfactory education in statistics/experiment planning/performance evaluation • However, things become better :-) Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  20. Experiments (2/3) • A second reason : real life experiments are often intractable because comparing two inventions is often intractable • Concurrent software are often not available • Concurrent complex systems/softwares cannot be re-develop from scratch at a reasonable price • Evaluation criteria are different • Users of your software are reluctant to spend/waste their time on alpha versions • It often exists no benchmarks • Research works often address very small range/specific issues that are part of a complex systems but integration is often not realized • Research teams have neither the manpower nor the competence to develop “products” and limit their development to demonstrator or prototypes that cannot be tested by real end users Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  21. Experiments (3/3) • However experiments are mandatory to evaluate the pertinence of your work • Experiments should allow • To evaluate the performance/pertinence/efficiency of your proposal wrt your initial objective and wrt all the possible contexts (not only in the best case) • To evaluate the influence of the various parameters on the performance • To compare your proposal wrt the state of the art • Experimental conditions must be made completely explicit Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  22. Agenda • Disclaimer • Discovery vs. invention • What is a research result in computer science? • What is a research paper in computer science? General structure(s) of a research paper in CS •  Searching for black boxes, traps and black holes • How is research evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  23. Reading a paper as an investigation • A key behavior: criticism (cf. methodical doubt by Descartes) • A first obvious step : analyze what is written • is the proposal scientifically sound? • are the performances good? • is the problem really important? • is the state of the art exhaustive? • etc. • A second step : analyze what is not written Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  24. Black boxes and assumptions • Computer systems/applications/software are too complex/too large so that a single team/paper can address the entire problem => Focus on a small part or specific issue => Assumptions on the other parts + inputs: it exists/will exist (soon) some “system” can can provide your prototype with the data/information it needs + outputs: the data produced by your prototype could be/will (soon) be used by the end-user side of the global system • Assumptions are not always explicit • Always ask yourself if these assumptions are valid/realistic, what do they imply Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  25. Analyzing experiments (1/3) • The border line • “Science” definitely condemns treachery and falsification: modifying the result of an experiment is beyond the “red line” • Experimental measurements that are presented in a paper are (supposed to be) true (I personally have always supposed it) • The question is about omitted/”forgotten” results • The question is about uncertainty and statistical errors on the measurements • The question is about some generalization of results/experiments • From a scientific point of view, a negative result is often as interesting as a positive result • However, as CS has to deal with invention, a negative result is usually considered as a depreciated fact Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  26. Analyzing experiments (2/3) • So the temptation to “avoid” including negative measurements often exists: • “This measurement is not really important but as I do not have enough space to explain it, it is more important, from a scientific point of view (of course), to discuss in deep other more important experiments” • “The situation that leads to this bad result never happens in the real life” • “With a slight modification of the prototype, it is obvious that one would never get such a result, so let us focus on the results concerning the core of the prototype” • etc. are reasons that a scientist may use to self-persuade him that presenting this experiment is not pertinent Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  27. Analyzing experiments (3/3) • Where is the yellow line? Where is the red line? • Always • analyze if experiments are exhaustive • If not, always analyze if the missing experiments are important to evaluate the performance of the prototype • and try to infer what could be the behavior of the prototype wrt these experiments Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  28. Agenda • Disclaimer • Discovery vs. invention • What is a research result in computer science? • What is a research paper in computer science? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes •  How is a research paper evaluated? • And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  29. Evaluation of a research paper • Is the topic of the paper in the domain of interest of the conference/journal? • (Is the bibliography complete, critical and taxonomic?) • Is the contribution original? • Is the paper technically sound/correct? • Is the paper easy to read? • Is the contribution significant? • What is the level of expertise of the evaluator / what is the confidence of the evaluator in his judgement • Discussion within the program or scientific committee Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  30. Menu • Disclaimer • Discovery vs. invention • What is a research result in computer science? • What is a research paper in computer science? General structure(s) of a research paper in CS • Searching for black boxes, traps and black holes • How is research evaluated? •  And now? Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  31. Conclusion? • Read, read, read, read! • Be positively critical • Never admit supposed evidences. Always doubt (Descartes) • All questions are good • Be modest Master Course, Lyon, 2015/2016 - Methodology of Research in CS

  32. Bring your creativity! Master Course, Lyon, 2015/2016 - Methodology of Research in CS

More Related