Paired Comparison: A User Perspective

1. Paired Comparison: A User Perspective Shigeru Sasao Master of Software Engineering Carnegie Mellon University

2. Agenda • Introduction to Paired Comparison • Experiment comparing Ad-hoc, Planning Poker, and Paired Comparison • Observation of Usage in a Project • Points of Improvement • New Version Using Incomplete Cyclic Design • Conclusion

3. Introduction to Paired Comparison • We want to estimate the size of the following objects. • We know that size of C is 10 units big. B A C D

4. B A C D • Estimate the relative sizes for all pairs: • I think A is 3 times as large as D. • I think A is half the size of B. • I think B is 6 times as large as D. • ...

5. I think A is 3 times bigger than D… • This is the “Judgment Matrix”. • Principal diagonal is always 1. • The other shaded regions are reciprocals of the un-shaded region. • Does not have to be perfectly consistent, since it is an estimate.

6. From the Judgment Matrix, we can calculate the estimates of the object size: • Calculate the geometric mean of each row. • Obtain the “ratio scale”. • Calculate the estimated size.

7. For our example, • Calculate the geometric mean of each row. • Obtain the “ratio scale”. • Sum the geometric means • 1.32 + 1.57 + 1.19 + 0.41 = 4.49 • Divide geometric mean of the row by the sum of the geometric mean.

8. Calculate the estimated size. • We know from the beginning that C has size of 10 units. • So, the reference value is C with ratio scale entry of 0.26.

9. Introduction to Paired Comparison • What did we do? • We knew the size of 1 object (reference size). • We compared pairs of relative sizes, and entered it into a judgment matrix. • Calculated the estimates for the objects. • The method reduces individual judgment errors by requiring multiple pair-wise comparisons of relative values.

10. Experiment Comparing Ad-hoc, Planning Poker, and Paired Comparison • Comparison was made between ad-hoc methods, planning poker and paired comparison. • Study conducted among students from the Master of Software Engineering program at Carnegie Mellon. • Students have strong technical background with two to three years of industry experience.

11. Experiment Setup • Students divided into three groups, with each group using either ad-hoc, planning poker, or paired comparison. • Estimations conducted in pairs. • Five pairs per estimation technique. • Students were asked to estimate the size (Lines of Code) of different data structures (stack, queue, binary tree, etc). • Students were told that the size of “linked list (a)” was 40 LOC.

12. Experiment Result

13. Experiment Result • Planning poker and paired comparison show a vast improvement in precision over ad-hoc methods. • This result is consistent with previous reports by Miranda. Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.

14. Experiment Result • Comparison of standard deviation between planning poker and paired comparison:

15. Experiment Result • Paired comparison produced more consistent estimations among estimators as compared to ad-hoc and planning poker. • Earlier studies by both Miranda and Shepperd show that paired comparison produces more reliable estimations than ad-hoc estimation methods.

16. Observation of Usage in a Project • In the Master of Software Engineering program, a team of four to five members work for an external customer. • 16 months with a total resource of 4608 to 5760 person hours. • Observed the usage of paired comparison by a MSE team. • The project was to produce an integrated development environment to support global distributed teams. • Estimation of effort for architectural components prior to the detailed design/implementation stage.

17. Observation of Usage in a Project • Five team members and observers gathered in a conference for the estimation session. • A list of 12 components was prepared beforehand. • Pair-wise comparison between all 12 components, for a total of 66 comparisons. • For each comparison, team discussed size, complexity, number of state transitions to agree on a relative size. • Values entered into a spreadsheet tool.

18. Observation of Usage in a Project • Snapshot of the resulting judgment matrix: • Total of 66 comparisons in 54 minutes.

19. Points of Improvement • Estimators became noticeably tired during the 54 minute estimation session. • As estimators became tired, they started extrapolating the new relative size values from old ones that have already been determined. • “Since A was 3 times the size of C and C was 2 times the size of D, A must be 6 times the size of D…”

20. New Version Using Incomplete Cyclic Design • To consider the estimators’ stamina, we need to reduce the number of comparison. • Use incomplete cyclic designs to reduce comparisons. • Maintains balance and connectedness.

21. New Version Using Incomplete Cyclic Design • Control the number of comparisons by the replication factor. • Example for a 10 component comparison:

22. New Version UsingIncomplete Cyclic Design • Full comparison of 8 components (28 comparisons) • Replication factor of 4 (16 comparisons)

23. New Version UsingIncomplete Cyclic Design • The reduction of comparison is made possible through: • Incomplete cyclic design (ICD) • Imputing missing values as the geometric mean of comparisons made • Studies have shown that very low replication factors still produce reliable estimates, although precision may diminish as number of comparisons decrease.

24. New Version Using Cyclic Design • Developed a tool which uses the new version with cyclic design: Replication factor Randomized comparison order

25. Integration with Cocomo • Use paired comparison to estimate LOC size as input into COCOMO.

26. Conclusion • Paired comparison produces more consistent estimations than ad-hoc methods and planning poker. • Can be used to directly estimate effort, or to estimate LOC size for input into parametric models such as COCOMO. • New version using incomplete cyclic designs reduce the number of comparison. • If you are interested in the tool, please contact me.

27. References • [Aguaron et al. 2003] Aguaron, Juan, Moreno-Jimenez, Jose Maria, The Geometric Consistency Index: Approximated Thresholds, European Journal of Operational Research, 147 (2003), Pages 137-145. • [Crawford, Williams 1985] Crawford, Gordon, Williams, Cindy, The Analysis of Subjective Judgment Matrices, Rand Corporation, 1985. • [Miranda 2000] Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000. • [Miranda 2001] Miranda, Eduardo, Improving Subjective Estimates Using Paired Comparison, IEEE Software, 2001. • [Miranda et al. 2009] Miranda, Eduardo, Bourque, Pierre, Abran, Alain, Sizing User Stories Using Paired Comparisons, Information and Software Technology Volume 51, Issue 9, September 2009, Pages 1327-1337, Butterworth-Heinemann, 2009. • [PlanningPoker] Cohn, Mike, Planning Poker, www.planningpoker.com, Mountain Goat Software. • [Shepperd et al. 2001] Shepperd, Martin, Cartwright, Michelle, Predicting with Sparse Data, IEEE Transactions on Software Engineering, VOL. 27, NO. 11, 2001. • [Spencer 1982] Spencer, Ian, Incomplete Experimental Designs for Multidimensional Scaling, Chapter 3. In R.G. Golledge and J.N. Rayner (Eds.), Proximity and Preference: Problems in the Multidimensional Analysis of Large Data Sets, University of Minnesota Press, 1982. • [Spencer 1983] Spencer, Ian, Monte Carlo Simulation Studies, Applied Psychological Measurement Vol. 7, No. 4, 1983.