Open ECBCheck

1. Open ECBCheck Methods for Quality Development Rafael García Rodríguez University of Augsburg, 2010

2. Introduction Open ECBCheck label makes use of five central methods for quality evaluation and validation that all have distinct characteristics and potential advantages and disadvantages. Benchmarking Self-assessment Qualitative Weighting and Summation Community of Practice Quality Assessment Benchlearning Peer-review

3. Benchmarking Definition Benchmarking is a systematic and continuous process to evaluate products, services or processes of an organization with the recognized market leaders (“best in class"). The target, in addition to improve himself, would be to get in long term market leadership or excellence.

4. Benchmarking Benchmarking is divided into four main stages with a concluding fifth phase maturity: Self-assessment Identification of object, organisations and methods Planning Analysis Identification of gaps and performance levels Communication of results and setting of targets Integration Implementation Development, execution and check Leading position and integration of benchmarking Maturity

5. Benchmarking There are, however, a number of restrictions associated with benchmarking: First, learning from benchmarking is mostly concerned with the past as the competitor has already achieved this level before. There is a high uncertainty in identifying the “best in class” organisation or best practice. It's necessary a step by step approach. Benchmarking only focuses on current best practices and is not a source for innovation and possible future best practice. The best practice cases are not specific, they need to be generalisable to be transferable to one organisation.

6. Benchlearning Definition Benchlearning is the process of learning from the "best in class" with the purpose of integrating these best practices in all organizational levels of the company.

7. Benchlearning Benchlearning is connected to benchmarking. Benchmarking = (Foundation) Benchlearning = (Learning process) These two processes search the way of how the best practices can be transferred to the own organisation and how skills and processes can be improved, and they distinguish benchaction as the actual implementation of all changes that have been set as targets.

8. Self-assessment Definition • Self-assessment is a regular and systematic analysis of strengths and weaknesses of a company or organisation to determine one position, to identify areas for improvement and to transfer these insights into implementation. • The initiative is supposed to come from the organisation itself and the organisation that conducts the assessment is also responsible for the process. Usually, the self-assessment is conducted against a set of criteria.

9. Self-assessment The European Foundation For Quality Management (EFQM) offers the EFQM Excellence Model as a reference point for a self-assessment. EFQM Excellence Model

10. Self-assessment The EFQM manual discusses data collection methods that could be used for self-assessments that differ in results, time or resources required. Easy to use Don’t consume many resources Used as foundation for other methods Questionnaires Data collection Presentation and assessment Reach a consensus Assessment workshop Full documentation for “European Quality Award” Results from organisations itself or outside “Award Simulation”

11. Peer-review Definition • Peer-review is a external evaluation of an organisation or individual by experts that belong to a different organisation or may also be colleagues within the same organisation for a Quality Assurance. • Peer-review doesn’t focus on data collection but on a competent review of existing data to derive core issues, however, peer-reviewers may collect additional data by a peer-review visit or interviews.

12. Peer-review Design review as the subject of interest: Review of the self-assessment report/results. Review of the conclusions in the self-assessment report/ site visits and interviews with stakeholders. Review of the subject of interest itself Basic design of the Peer-review process Self-evaluation External evaluation (+ on-site visit) Final review report Even so, there are numerous possible designs for peer-reviews

13. Peer-reviews Advantages Disadvantages • In a formative way: • - Mutual learning possibility • Relatively good cost-benefit ratio • More acceptable perception of colleagues or experts in the same field of study than external evaluators. • SRCIVEN (1991, p. 255) • - Halo-effects • - Secret-contract bias • - Fear of retaliatory action • Gutknecht Gmeiner (2008, p. 23) • - Lack of qualification • - Lack of objectivity • (refers to the reviewers)

14. Qualitative Weighting and Summation Definition • Qualitative Weighting and Summation is a method for evaluation based on criteria catalogues. Is described in the context of the evaluation of learning management systems. This method is proposed as an alternative to the Numerical Weighting and Summation method to avoid some disadvantages of this latter. • In Numerical Weighting and Summation method: 1. Each criterion is weighted according to importance. 2. The evaluand is rated in all criteria 3. The products of all ratings with the corresponding weighing are summed up to one final score that indicates the relative rank of product, organisation or service)

15. Qualitative Weighting and Summation Qualitative Weighting and Summation Process • All criteria are weighted based on non numeric characters (Essential / Very Valuable / Valuable / Marginally Valuable / Zero). All criteria that are rated with “Zero” are excluded. • Only criteria that have been weighted as essential are assessed. • All remaining products are rated only with help of the remaining criteria (“Very Valuable” to “Marginally Valuable”) • The ratings in the three different importance levels are then summarised resulting in three aggregated values for each product.

16. Qualitative Weighting and Summation Disadvantages Qualitative Weighting and Summation Numerical Weighting and Summation • Existence of minimum criteria. • Existence of interconnections between criteria. • The assumption of linear scales. • How to weigh and consider the relevance of the criteria. • The procedure of weighting is unreproducible. • Method relatively complex. • It doesn’t offer a definite decision algorithm. • Sometimes needs to be applied several times in an iterative process to final conclusion.

17. References • Ehlers, Ulf-Daniel (2009).Selected Methods for Quality Development in Open • http://www.qualityfoundation.org/openecbcheck/ • http://www.efqm.org/en/ • More information about Open ECB-Check in: http://www.qualityfoundation.org/openecbcheck/ ECBCheck. University Duisburg-Essen, Germany.