ISERN Open Issues, Grand Challenges or Have we made any progress and where are going? Vic Basili

1. ISERNOpen Issues, Grand ChallengesorHave we made any progress and where are going?Vic Basili 2001

2. ISERN Areas of Contribution and Influence • What have been our contributions to Software Engineering: • Experimentation • Design and Implementation • Replication • Research Knowledge • Levels of Understanding • Depth of Hypotheses and Questions • Software Engineering Practice • Domains of Knowledge • Practicality of Information • And what has been our influence on: • the ISERN community • SE research • SE Practice

3. An example: Reading Experiments • Before ISERN (1986): Code Testing vs. Reading Experiment (UMD) • … • First ISERN 1992: First discussion of the requirements reading (DBR)(UMD) • ICSE 1996: • Basili presented a talk on the state of experimentation and discussed reading experiments to that point in time. • Set some goals for the maturing of an experimental discipline • The next 6 slides come from that talk

4. The Maturing of the Experimental DisciplineHow is experimentation maturing? • Sign of maturity in a field: • level of sophistication of the goals of an experiment • understanding interesting things about the discipline • For software engineering that might mean: • Can we build models that allow use to measure and differentiate processes and products? • Can we measure the effect of a change in a particular process variable on the product variable? • Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context? • Can we control for product effects, based upon goals, given a particular set of context variables?

5. The Maturing of the Experimental DisciplineHow is experimentation maturing? • Sign of maturity in a field: • a pattern of knowledge built from a series of experiments • Does the discipline build on prior (knowledge, models, experiments). • Was the study an isolated event? • Did it lead to other studies that made use of this information • Have studies been replicated under similar or differing conditions? • Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work? • For example, inspections, in general, are well studied experimentally • However, there has been very little combining of results, replication, analysis of the differentiating variables

6. The Maturing of the Experimental DisciplineHow is experimentation maturing? • There is some evidence that researchers appear to be • - asking more sophisticated questions • - studying relationships between processes/product characteristics • - using more studies in the field than in the controlled laboratory • - combining various experimental classes to build knowledge • On such example of evolving knowledge over time, • - based upon experimentation and learning is • - the evolution of the SEL knowledge • - of the effectiveness of reading techniques and methods • Software Engineering Laboratory is a consortium (established in 1975) • - NASA/Goddard Space Flight Center • - University of Maryland • - Computer Sciences Corporation

7. Evolution of Knowledge over Time Reading Technology Experiments • This example • - shows the combination of multiple experimental designs • - provides insight into the effects of different variables on reading • - demonstrates replication by other researchers • The experiments start with • the early reading vs. testing experiments • to various Cleanroom experiments • to the development of new reading techniques currently under study to replications at other groups • The experiments are based upon the ideas that • Reading is a key technical activity • for improving the analysis of all kinds of software documents • and we need to better understand its effect • Early experiments (Hetzel, Meyers) showed very little difference between reading and testing • But reading was simply reading, without a technological base

8. The Maturing of the Experimental DisciplineHow is experimentation maturing? • Several of these experiments have been replicated • - under the same and differing contexts • The original analysis technique comparison has been replicated • University of Kaiserslautern • Scenario-based reading study variations • University of Bari, Italy • University of New South Wales, Australia • Bell Laboratories, USA • University of Trondheim, Norway • Bosch, Germany • to better understand the reading variable • ISERN • organized explicitly to share knowledge and experiments • has membership in the U.S., Europe, Asia, and Australia • represents both industry and academia • supports the publication of artifacts and laboratory manuals • Its goal is to evolve software engineering knowledge over time, based upon experimentation and learning

9. What will our future look like? • Experimentation can provide us with • - better scientific and engineering basis for the software engineering • - better models of • - software processes and products • - various environmental factors, e.g. the people, the organization • - better understanding of the interactions of these models • Practitioners will be provided with • - the ability to control and manipulate project solutions • - based upon the environment and goals set for the project • - knowledge based upon empirical and experimental evidence • - of what works and does not work and under what conditions • Researchers will be provided laboratories for experimentation • This will require a research plan that will take place over many years • - coordinating experiments • - evolving with new knowledge

10. Continuing: Reading Experiments • Since then: new variations and replications • Code Reading vs. Testing • UMD (USA), UKL (Germany), U Strathclyde (Scotland), • U Sao Paolo (Brazil), • Defect Based Reading • UMD, Lucent, U Bari (Italy), • Perspective Based Reading • UMD, IESE, FC-MD, U Bari, UMBC (USA), U Sao Paulo, • NTNU (Norway), USC (USA), COPPE/UFRJ (Brazil), • Lund U (Sweden) • Object Oriented Reading • UMD, NTNU, U Rio deJaniero, USC, IESE • Other Reading • UMD, UNSW (Australia), U Strathclyde, TU Vienna (Austria), IESE

11. Reading Experiments: Contribution and Influence • Experimentation • Design and Implementation • Several designs form full fractional factorial to constant comparison approaches • Are learning the limits of controlled experiments • Replication • Several at universities, often with support fro original experimenters • Combining Results • Identifying common factors (Goal Template) • Need to • build a stronger body of experimental design • be able to combine the results of experiments of all kinds • make “controlled” type experiments cheaper • And what has been our influence on: • the ISERN community • work on the combined experiment for inspections • SE research • two books on experimentation from ISERN participants • SE Practice • not much so far but some nation wide programs have begun, • e.g., CeBASE, ViSEK, …

12. Reading Experiments: Contribution and Influence • Research Knowledge • Levels of Understanding • evolution of process based upon empirical study • use of experiments as learning rather than validation • Depth of Hypotheses and Questions • asking questions about meta issues, such as context variables • Questions from ICSE: • Can we build models that allow use to measure and differentiate processes and products? Yes • Can we measure the effect of a change in a particular process variable on the product variable? Yes • Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context? Some, but not really

13. Reading Experiments: Contribution and Influence • Research Knowledge • Levels of Understanding • Depth of Hypotheses and Questions • Questions from ICSE: • Does the discipline build on prior (knowledge, models, experiments)? • More than before • Was the study an isolated event? Few isolated events by ISERN personnel • Did it lead to other studies that made use of this information? Yes • Have studies been replicated under similar or differing conditions? Yes • Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work? More building together

14. Reading Experiments: Contribution and Influence • Research Knowledge • Levels of Understanding • Depth of Hypotheses and Questions • And what has been our influence on: • the ISERN community • More working together, sharing experiments • SE research • More experimentalists joining ISERN • SE Practice • Some interest in national projects like CeBASE, ViSEK, ESERNET,.. • What results do we have so far? For example …

15. Combining Results from multiple Reading Experiments • We have generated useful empirical results for technique definition guidance • For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one. • Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. • A set of readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus.

16. Reading Experiments: Contribution and Influence • Software Engineering Practice • Domains of Knowledge • have been primarily limited to inspection area • have opened up to areas like COTS • have attracted empirical modelers which include areas such as cost, quality, and schedule estimation • Practicality of Information • national experience base activities are trying to reach out to the development community • not too many members other than researchers • And what has been our influence on: • the ISERN community • SE research • SE Practice • What can be packaged for practice, research, education?

17. Examples of Using Empirical Results for development, research, education Technique Tailoring • Is tailoring the reading process associated with an inspection worth the effort? • Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. (UMD) • Implications for empirically based software development process: • The better you can articulate your goals, the more effectively you can choose and tailor process. • Implications for software engineering research: • It is important to empirically study the effects of processes on product • Implications for software engineering education: • Don’t teach that there is a one size fits all process; teach how to tailor processes

18. Examples of Using Empirical Results for development, research, educationTechnique Selection Guidance • When should you use a procedural approach to code reviewing? • For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one (UMD, USC) • Implications for empirically based software development process: • Experts might be more effective working on their own but most people should apply a procedural approach. Novices need training. • Implications for software engineering research: • How can we improve document reading procedures based upon how experts analyze documents? • Implications for software engineering education: • Effective procedures that can be taught for reviewing documents

19. Examples of Using Empirical Results for development, research, education Technique Selection Guidance • When are peer reviews more effective than functional testing? • Peer reviews are more effective than functional testing for faults of omissionand incorrect specification(UMD, USC) • Implications for empirically based software development process: • If, for a given project set, there is an expectation of a larger number of faults of omission or incorrect facts than use peer reviews. • Implications for software engineering research: • How can peer reviews be improved with better reading techniques for faults of omission and incorrect fact? • Implications for software engineering education: • Teach how to experiment with and choose the appropriate analytic techniques

20. ISERN Areas Challenges • Experimentation • How do we define more effective and efficient experimental techniques? • How do we combine results from experiments? • How do we deal with (define, study) meta data? • How doe we package lab manuals, experience bases? • Research Knowledge • How do we define the next levels of research questions? • What other processes should we study? Other analytic (testing), creative (design)? What product properties should we be studying? • Mostly have looked at the relationship between process and product characteristics. What else should we be studying? • In our grand inspection experiment, should we be studying ourselves? • How do we get more researchers interested in empirically studying their techniques? • Software Engineering Practice • How do we ,package our results for practice? • How do we get more industrial members involved?

21. Example of Specific Experimental Issues • In looking at the relationship between process and product, there are many questions that need to be covered: • Process variable (Independent variable) issues: • How do we define/specify the process? • How do we account for process conformance? • Effectiveness of Product (Dependent variable) issues: • How do we select good criteria for effectiveness? • Context Variables (meta data) Issues: • What subjects are performing the process? • What characterizes their experience, environment,…? • Questions associated with the variables need to be further specified and documented for replication • Varying the values of these variables allow us to • vary the detailed hypotheses • support validity of study results