Validation of the Method Adoption Model for Functional Size Measurement of Web Applications

1. Validation of the Method Adoption Model for Functional Size Measurement of Web Applications 2nd Workshop on Software Metrics Validation • Geert Poels • Ghent University, Belgium • geert.poels@UGent.be • Silvia Abrahão • Valencia University of Technology, Spain • sabrahao@dsic.upv.es • Ghent, July 06, 2004

2. Contents • Functional Size Measurement (FSM) • How to evaluate FSM methods? • A Process Model for FSM • A Theoretical Model for Evaluating FSM Methods • A Laboratory Experiment to test the Theoretical Model • Summary of Findings • Next Steps

3. Functional Size Measurement • Functional size measurement (FSM) methods • Define functional size measures • Describe how to apply functional size measurement • New FSM methods for new technologies • E.g. OO-Method Function Points (OOmFP) and its extension to the Web (OOmFPWeb) • No methodology for systematic evaluation of FSM methods • Little evidence of validity of functional size measures

4. How to evaluate FSM methods? • OOmFPWeb: A FSM method for Web Applications • OOmFPWeb Measurement Procedure

5. Possibly existing solutions • Theoretical validation • Verification of empirical and numerical properties using Measurement Theory • Conformity evaluation to ISO/IEC 14143-1:1998 International Standard for FSM concepts • Empiricalvalidation • Demonstrate the relationship with relevant software process and management variables • Evaluate performance properties of FSM method (e.g. using ISO/IEC TR 14143-3:2003) • Pragmatic approach: method’s success in practice

6. Our position • Comprehensive evaluation framework needed • Multi-perspective evaluation • Systematic evaluation • Based on software measurement process model • Design => Application => Analysis => Use • Based on multi-disciplinary approach • Measurement Theory • IT / IS method evaluation models • Empirical software engineering

7. Explotation of the measurement method result Application of the measurement method rules Design of the Measurement method Measurement result A Process Model for FSM Step 2 Step 3 Step 1 Step 4 (Jacquet and Abran, 1997) Validation of the design of the FSM method Validation of the application of the FSM method Validation of the models thatuse the FSM method results Includes measurevalidation Includes evaluationof performance-basedand perception-basedproperties of the method Includes evaluationof model properties(e.g. predictionaccuracy, causality) Empirical softwareengineering approach MeasurementTheoretic approach Evaluation model forFSM methods

8. The Method Evaluation ModelAn Overview • Method should enable task to be performed faster, more cheaply or with less effort (efficiency) and/or improve the quality of the result (effectiveness) • Efficiency is defined by the effort required to apply the method. • Effectiveness is defined by how well the method achieves its objectives. • Efficacy is the combination of efficiency and effectiveness. Inputs Outputs Task reduce inputs improve outputs (increase efficiency) (increase effectiveness) (Moody, 2001) Method

9. Validation of the application of OOmFPWeb • Method Evaluation Model (MEM): a theoretical model for evaluating IS design methods, which incorporates both aspects of method “success”: efficacy and adoption in practice. (Moody, 2001)

10. Theoretical Model for Evaluating FSM Methods • Model’s variables: • Measurement Time: the time taken by a subject to size a OOWS conceptual schema. • Reproducibility: the agreement between the measurement results of different subjects using OOmFPWeb. • Perceived Ease of Use: the degree to which a subject believes that using OOmFPWeb would be free of effort. • Perceived Usefulness: the degree to which a subject believes that OOmFPWeb will be effective in achieving its intended objectives. • Intention to Use: the degree to which an individual intends to use OOmFPWeb as a result of her perception of the method’s efficacy.

11. Validation of the application of OOmFPWeb • Laboratory experiment to test the Theoretical Model • Analyze functional size measurements • For the purpose of evaluating OOmFPWeb • With respect to itsefficacy and likely adoption in practice • From the point of view of theresearcher. • The context of the experiment was an OOWS conceptual schema of a Web application that is measured by PhD students in the Department of Computer Science at the Valencia University of Technology.

12. Research Questions • The research questions addressed were: • RQ1: Is OOmFPWeb efficacious? • RQ2: Is OOmFPWeb likely to be adopted in practice? • RQ3: Is Method Adoption Model a valid theoretical model for evaluating OOmFPWeb?

13. Hypothesis Formulation • We therefore test the following hypotheses: • Hypothesis 1:OOmFPWeb is efficient when compared to current industry practices. • Hypothesis 2: OOmFPWeb is effective when compared to similar studies reported in the literature. • Hypothesis 3: OOmFPWeb is perceived as easy to use. • Hypothesis 4: OOmFPWeb is perceived as useful. • Hypothesis 5: There is an intention to use OOmFPWeb.

14. Results Obtained • H1:Efficiency:Productivity of subjects X Reported industry averages • The mean measurement productivity that was observed (108.79 FP/hour) is about three times the size of the industry benchmark. • H2:Effectiveness: Reproducibility X Previous studies • Compared to the results reported by Kemerer the consistency of measurements was high (mean REPi was 6%). The variation around the mean subject assessment (i.e. range of values divided by mean value) was 24.5%, comparing well with Rudolph’s study.

15. Results Obtained • H3, H4 and H5: PEOU, PU and ITU:survey instrument included 14 closed questions • Example of a question to measure PEOU: • These hypotheses was tested by verifying whether the scores that students assign to the constructs of the MAM are significantly better than the middle score (i.e. the score 3). • All hypothesis were confirmed

16. Analyzing the Validity and Reliability of the MAM Constructs • Validity Analysis: inter-item correlation analysis was carried out. • Reliability Analysis: inter-rater reliabilities  .7 considered to be acceptable.

17. Regression Models

18. Summary of Findings

19. Work done so far.. • Developing and validating measurement instruments for perception-based variables • Three replications of the study presented here: • Valencia University of Technology, Spain • 46 students in the last year of Computer Science degree • Valencia University of Technology, Spain • 18 PhD students • University of Klagenfurt, Austria. • 28 Master’s students

20. Next Steps • Analyzing the collected data to verify whether the proposed theoretical model is a valid model and measurement instrument for evaluating FSM methods. • A new experiment using practitioners of OOWS in the context of the Spanish Association of Software Metrics.