Enhancing Comparability of Standards through Validation and Moderation A study funded by the National Quality Council

1. Enhancing Comparability of Standards through Validation and ModerationA study funded by the National Quality Council Shelley Gillis Andrea Bateman Berwyn Clayton

2. Rationale Some key stakeholders have raised concerns with the quality and consistency of assessments being undertaken by RTOs. That is, concerns have been raised about comparability of standards. In recent times, some key stakeholders have raised concerns with the quality and consistency of assessments being undertaken by Registered Training Organisations (RTOs). That is, there are some concerns that assessment standards in the VET sector are often not comparable. Ensuring the comparability of standards[1] has become particularly pertinent in the VET sector, as assessments can now be made across a range of contexts (e.g., vocational education, educational and industrial contexts) by a diverse range of assessors using highly contextualised performance based tasks that require professional judgement by assessors. Comparability of Standards � Achieved when the performance levels expected (eg C/NYC) for a unit(s) of competency is similar between assessors assessing the same unit(s) within and across RTOs. In recent times, some key stakeholders have raised concerns with the quality and consistency of assessments being undertaken by Registered Training Organisations (RTOs). That is, there are some concerns that assessment standards in the VET sector are often not comparable. Ensuring the comparability of standards[1] has become particularly pertinent in the VET sector, as assessments can now be made across a range of contexts (e.g., vocational education, educational and industrial contexts) by a diverse range of assessors using highly contextualised performance based tasks that require professional judgement by assessors. Comparability of Standards � Achieved when the performance levels expected (eg C/NYC) for a unit(s) of competency is similar between assessors assessing the same unit(s) within and across RTOs.

3. Aim To develop a series of products that would: Improve the consistency in assessment decisions within VET; Increase the level of confidence in industry in assessment in VET; Increase awareness of, and consistency in, the application of reasonable adjustments in making assessment decisions; Increase capability in RTOs to demonstrate compliance with AQTF 2007 Essential Standards for Registration, Standard 1.

4. Products Volume 1 describes the methodology and findings of the study. It includes the following sections: Section 1: Background Section 2: Structure of the Outputs Section 3: Methodology Section 4: Findings Section 5: Equity Considerations Section 6: Recommendations for Future Research and Development Volume II provides the Code of Professional Practice. This volume contains a set of high level principles designed to provide guidance on how to conduct assessment validation and moderation within a vocational education and training (VET) setting. The Code is intended to complement Elements 1.1 and 1.5 of the Australian Quality Training Framework (AQTF) Essential Standards for Registration and be consistent with the TAA04 Training and Assessment Training Package. The Code is not intended to be mandatory, exhaustive or definitive, and may not be applicable to every situation. Instead, the Code is intended to be aspirational and educative in nature. Volume III contains the Implementation Guide to Validation and Moderation. This was designed to be a practical resource for training organisations intending to implement and/or review validation and/or moderation involving consensus meetings[1]. It provides guidance on how to implement the Code of Professional Practice within one�s own organisation. The Guide provides practical suggestions for: Adhering to the Principles within the Code of Professional Practice; Designing assessment tools; Planning and conducting consensus meetings; Recording and Reporting outcomes; and Handling complaints and appeals. Volume 1 describes the methodology and findings of the study. It includes the following sections: Section 1: Background Section 2: Structure of the Outputs Section 3: Methodology Section 4: Findings Section 5: Equity Considerations Section 6: Recommendations for Future Research and Development Volume II provides the Code of Professional Practice. This volume contains a set of high level principles designed to provide guidance on how to conduct assessment validation and moderation within a vocational education and training (VET) setting. The Code is intended to complement Elements 1.1 and 1.5 of the Australian Quality Training Framework (AQTF) Essential Standards for Registration and be consistent with the TAA04 Training and Assessment Training Package. The Code is not intended to be mandatory, exhaustive or definitive, and may not be applicable to every situation. Instead, the Code is intended to be aspirational and educative in nature. Volume III contains the Implementation Guide to Validation and Moderation. This was designed to be a practical resource for training organisations intending to implement and/or review validation and/or moderation involving consensus meetings[1]. It provides guidance on how to implement the Code of Professional Practice within one�s own organisation. The Guide provides practical suggestions for: Adhering to the Principles within the Code of Professional Practice; Designing assessment tools; Planning and conducting consensus meetings; Recording and Reporting outcomes; and Handling complaints and appeals.

5. Changes to the AQTF User Guide Validity Reliability Assessment tool Validation Moderation As a result of this study, the following definitions have been changed in national documents to ensure greater clarity and accuracy of technical terms used in assessment. The Code and Guides introduce a number of technical assessment concepts that may not be very familiar to a number of assessors within the VET Sector, particularly the different types of validity and reliability that need to be considered when designing and reviewing assessment tools. The technical definitions provided throughout the Code and Guides are consistent with the international Standards for Educational and Psychological Testing (American Education Research Association, American Psychological Association and the National Council on Measurement in Education, 1999). These standards were written for the professional and for the educated layperson and address professional and technical issues of test development and use in education, psychology and employment. The research team have attempted to redefine the terms used in the International Standards to maximise understanding by VET assessors, but at the same time, care has been taken to ensure the original intent and meaning has been preserved. The Code and Guides include an extensive Glossary of Terms. As a result of this study, the following definitions have been changed in national documents to ensure greater clarity and accuracy of technical terms used in assessment. The Code and Guides introduce a number of technical assessment concepts that may not be very familiar to a number of assessors within the VET Sector, particularly the different types of validity and reliability that need to be considered when designing and reviewing assessment tools. The technical definitions provided throughout the Code and Guides are consistent with the international Standards for Educational and Psychological Testing (American Education Research Association, American Psychological Association and the National Council on Measurement in Education, 1999). These standards were written for the professional and for the educated layperson and address professional and technical issues of test development and use in education, psychology and employment. The research team have attempted to redefine the terms used in the International Standards to maximise understanding by VET assessors, but at the same time, care has been taken to ensure the original intent and meaning has been preserved. The Code and Guides include an extensive Glossary of Terms.

6. The Guide for Developing Assessment Tools This Guide is a practical resource material for assessors and assessor trainers seeking technical guidance on how to develop and/or review assessment tools. The Guide is not intended to be mandatory, exhaustive or definitive but instead it is intended to be aspirational and educative in nature. This Guide is a practical resource material for assessors and assessor trainers seeking technical guidance on how to develop and/or review assessment tools. The Guide is not intended to be mandatory, exhaustive or definitive but instead it is intended to be aspirational and educative in nature.

7. Essential Characteristics of an Assessment Tool An assessment tool includes the following components: The learning or competency unit(s) to be assessed The target group, context and conditions for the assessment The tasks to be administered to the candidate An outline of the evidence to be gathered from the candidate The evidence criteria used to judge the quality of performance (i.e., the assessment decision making rules); as well as the The administration, recording and reporting requirements.

8. Ideal Characteristics The context Competency mapping The information to be provided to the candidate The evidence to be collected from the candidate Decision making rules Range and conditions Materials/resources required Assessor intervention Reasonable adjustments Validity evidence Reliability evidence Recording requirements Reporting Requirements The Guide also includes a number of ideal characteristics of an assessment tool and provides four examples of how each of these characteristics can be built into the design for four methods of assessment: observation, interview, portfolio and product-based assessments. These four examples encapsulate methods that require candidates to either do (observation), say (interview), write (portfolio) or create (product) something. In fact, any assessment activity can be classified according to these four broad methods. To assist with validation and/or moderation, the tool should also provide evidence of how validity and reliability have been tested and built into the design and use of the tool. In some instances, all the components within the assessment tool may not necessarily be present within the same document. That is, it is not necessary that the hard copy tool holds all components. It may be that the tool makes reference to information in another document/material/tool held elsewhere. This would help avoid repetition across a number of tools (e.g., the context, as well as the recording and reporting requirements of the tool may be the same for a number of tools and therefore, may be just cited within one document but referred to within all tools). The quality test of any assessment tool is the capacity for another assessor to use and replicate the assessment procedures without any need for further clarification by the tool developer. That is, it should be a stand-alone assessment tool. The Guide also includes a number of ideal characteristics of an assessment tool and provides four examples of how each of these characteristics can be built into the design for four methods of assessment: observation, interview, portfolio and product-based assessments. These four examples encapsulate methods that require candidates to either do (observation), say (interview), write (portfolio) or create (product) something. In fact, any assessment activity can be classified according to these four broad methods. To assist with validation and/or moderation, the tool should also provide evidence of how validity and reliability have been tested and built into the design and use of the tool. In some instances, all the components within the assessment tool may not necessarily be present within the same document. That is, it is not necessary that the hard copy tool holds all components. It may be that the tool makes reference to information in another document/material/tool held elsewhere. This would help avoid repetition across a number of tools (e.g., the context, as well as the recording and reporting requirements of the tool may be the same for a number of tools and therefore, may be just cited within one document but referred to within all tools). The quality test of any assessment tool is the capacity for another assessor to use and replicate the assessment procedures without any need for further clarification by the tool developer. That is, it should be a stand-alone assessment tool.

9. Competency Mapping The components of the Unit(s) of Competency that the tool should cover should be described. This could be as simple as a mapping exercise between the components within a task (eg each structured interview question) and components within a Unit or cluster of Units of Competency. The mapping will help determine the suffiency of the evidence to be collected as well as the content validity.

10. Decision Making Rules The rules to be used to: Check evidence quality (i.e., the rules of evidence). Judge how well the candidate performed according to the standard expected. Synthesise evidence from multiple sources to make an overall judgement.

11. Reasonable Adjustments This section should describe the guidelines for making reasonable adjustments to the way in which evidence of performance is gathered without altering the expected performance standards (as outlined in the decision making rules).

12. Validity Evidence Validity is concerned with the extent to which an assessment decision about a candidate, based on the performance by the candidate, is justified. Requires determining conditions that weaken the truthfulness of the decision, exploring alternative explanations for good or poor performance, and feeding them back into the assessment process to reduce errors when making inferences about competence. Evidence of validity (such as face, construct, predictive, concurrent, consequential and content) should be provided to support the use of the assessment evidence for the defined purpose and target group of the tool. .

13. Reliability Evidence Reliability is concerned with how much error is included in the evidence. If using a performance based task that requires professional judgement of the assessor, evidence of reliability could include providing evidence of: The level of agreement between two different assessors who have assessed the same evidence of performance for a particular candidate (i.e., inter-rater reliability). The level of agreement of the same assessor who has assessed the same evidence of performance of the candidate, but at a different time (i.e., intra-rater reliability). If using objective test items (e.g., multiple choice tests) than other forms of reliability should be considered such as the internal consistency of a test (i.e., internal reliability) as well as the equivalence of two alternative assessment tests (i.e., parallel forms).

14. Examples Write Say Do Create

15. Quality Checks Panel Pilot Trial There are several checks that could be undertaken (as part of the quality assurance procedures of the organisation) prior to implementing a new assessment tool. For example, the tool could be: Panelled with subject matter experts (e.g., industry representatives and/or other colleagues with subject matter expertise) to examine the tool to ensure that the content of the tool is correct and relevant. The panelist should critique the tool for its: Clarity; Content accuracy; Relevance; Content validity (i.e., match to unit of competency and/or learning outcomes); Avoidance of bias; and/or Appropriateness of language for the target population. Panelled with colleagues who are not subject matter experts but have expertise in assessment tool development. Such individuals could review the tool to check that it has: Clear instructions for completion by candidates; Clear instructions for administration by assessors; Avoidance of bias. Piloted with a small number of individuals who have similar characteristics to the target population. Those piloting the tool should be encouraged to think out aloud when responding to the tool. The amount of time required to complete the tool should be recorded and feedback from the participants should be gathered about the clarity of the administration instructions, the appropriateness of its demands (i.e., whether it is too difficult or easy to perform), it perceived relevance to the workplace etc. Trialled with a group of individuals who also have similar characteristics to the target population. The trial should be treated as though it is a dress rehearsal for the �real assessment�. It is important during the trial period that an appropriate sample size is employed and that the sample is representative of the expected levels of ability of the target population. The findings from the trial will help predict whether the tool would: Be cost effective to implement; Be engaging to potential candidates; Produce valid and reliable evidence; Be too difficult and/or too easy for the target population; Possibly disadvantage some individuals; Able to produce sufficient and adequate evidence to address the purpose of the assessment; as well as Satisfy the reporting needs of the key stakeholder groups. This process may need to be repeated if the original conditions under which the assessment tool were developed have been altered such as the: Target group; Unit(s) of Competency and/or learning outcomes; Context (e.g., location, technology); Purpose of the assessment; Reporting requirements of the key stakeholder groups; and/or Legislative/regulatory changes. A risk assessment will help determine whether it is necessary to undertake all three processes (i.e., panelling, piloting and trialling) for ensuring the quality of the assessment tool prior to use. If there is a high likelihood of unexpected and/or unfortunate consequences of making incorrect assessment judgements (in terms of safety, costs, equity etc), then it may be necessary to undertake all three processes. When the risks have been assessed as minimal, then it may only be necessary to undertake a panelling exercise with one�s colleagues who are either subject matter experts and/or assessment experts. There are several checks that could be undertaken (as part of the quality assurance procedures of the organisation) prior to implementing a new assessment tool. For example, the tool could be: Panelled with subject matter experts (e.g., industry representatives and/or other colleagues with subject matter expertise) to examine the tool to ensure that the content of the tool is correct and relevant. The panelist should critique the tool for its: Clarity; Content accuracy; Relevance; Content validity (i.e., match to unit of competency and/or learning outcomes); Avoidance of bias; and/or Appropriateness of language for the target population. Panelled with colleagues who are not subject matter experts but have expertise in assessment tool development. Such individuals could review the tool to check that it has: Clear instructions for completion by candidates; Clear instructions for administration by assessors; Avoidance of bias. Piloted with a small number of individuals who have similar characteristics to the target population. Those piloting the tool should be encouraged to think out aloud when responding to the tool. The amount of time required to complete the tool should be recorded and feedback from the participants should be gathered about the clarity of the administration instructions, the appropriateness of its demands (i.e., whether it is too difficult or easy to perform), it perceived relevance to the workplace etc. Trialled with a group of individuals who also have similar characteristics to the target population. The trial should be treated as though it is a dress rehearsal for the �real assessment�. It is important during the trial period that an appropriate sample size is employed and that the sample is representative of the expected levels of ability of the target population. The findings from the trial will help predict whether the tool would: Be cost effective to implement; Be engaging to potential candidates; Produce valid and reliable evidence; Be too difficult and/or too easy for the target population; Possibly disadvantage some individuals; Able to produce sufficient and adequate evidence to address the purpose of the assessment; as well as Satisfy the reporting needs of the key stakeholder groups. This process may need to be repeated if the original conditions under which the assessment tool were developed have been altered such as the: Target group; Unit(s) of Competency and/or learning outcomes; Context (e.g., location, technology); Purpose of the assessment; Reporting requirements of the key stakeholder groups; and/or Legislative/regulatory changes. A risk assessment will help determine whether it is necessary to undertake all three processes (i.e., panelling, piloting and trialling) for ensuring the quality of the assessment tool prior to use. If there is a high likelihood of unexpected and/or unfortunate consequences of making incorrect assessment judgements (in terms of safety, costs, equity etc), then it may be necessary to undertake all three processes. When the risks have been assessed as minimal, then it may only be necessary to undertake a panelling exercise with one�s colleagues who are either subject matter experts and/or assessment experts.

16. A Code of Professional Practice for Validation and Moderation

17. Assessment Quality Management Quality Assurance Quality Control Quality Review There are a number of different quality management processes that could be used to help achieve national comparability of standards whilst still maintaining sufficient flexibility at the RTO level to conduct assessments. Typically, there are three major components to quality management of educational assessments: quality assurance; quality control; and quality review (Maxwell, 2006). A quality assurance approach attempts to assure quality of assessment in VET through focusing on the procedures used in the assessment process. Such an approach is based upon the assumption that the introduction of products and processes such as policies, competency standards, professional development support materials and training within the sector can improve the quality of assessments. Hence, it is referred to as an �input approach� to quality management. The second approach to quality management, referred to as �quality control� focuses on monitoring, and where necessary making adjustments to judgements made by assessors prior to the finalisation of assessment results/outcomes. This approach therefore involves the direct management of assessment judgements to ensure consistency in the interpretation and application of the competency standards. As it occurs prior to the finalisation of the result, in which alterations can be made to assessor judgements, it is referred to as an �active process�. The third approach to quality management is referred to as �quality review� as it involves the review of the assessment procedures and outcomes for the sole purpose of improving assessment processes and procedures for future use. It is referred to as a retrospective approach as the outcomes of the review are aimed at making recommendations for future improvements. The outcomes of the review have no direct impact on any current or past assessments. A number of mechanisms or potential mechanisms for enhancing quality assurance, quality control and quality review within the Australian VET sector can be identified. These have been displayed in Table 1. There are a number of different quality management processes that could be used to help achieve national comparability of standards whilst still maintaining sufficient flexibility at the RTO level to conduct assessments. Typically, there are three major components to quality management of educational assessments: quality assurance; quality control; and quality review (Maxwell, 2006). A quality assurance approach attempts to assure quality of assessment in VET through focusing on the procedures used in the assessment process. Such an approach is based upon the assumption that the introduction of products and processes such as policies, competency standards, professional development support materials and training within the sector can improve the quality of assessments. Hence, it is referred to as an �input approach� to quality management. The second approach to quality management, referred to as �quality control� focuses on monitoring, and where necessary making adjustments to judgements made by assessors prior to the finalisation of assessment results/outcomes. This approach therefore involves the direct management of assessment judgements to ensure consistency in the interpretation and application of the competency standards. As it occurs prior to the finalisation of the result, in which alterations can be made to assessor judgements, it is referred to as an �active process�. The third approach to quality management is referred to as �quality review� as it involves the review of the assessment procedures and outcomes for the sole purpose of improving assessment processes and procedures for future use. It is referred to as a retrospective approach as the outcomes of the review are aimed at making recommendations for future improvements. The outcomes of the review have no direct impact on any current or past assessments. A number of mechanisms or potential mechanisms for enhancing quality assurance, quality control and quality review within the Australian VET sector can be identified. These have been displayed in Table 1.

18. The Professional Code of Practice is a QA approach to assessment Quality Management. There is an assumption that What is validation? Validation is a quality review process. It involves checking that the assessment tool[1] produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes. What is moderation? Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.The Professional Code of Practice is a QA approach to assessment Quality Management. There is an assumption that What is validation? Validation is a quality review process. It involves checking that the assessment tool[1] produced valid, reliable, sufficient, current and authentic evidence to enable reasonable judgements to be made as to whether the requirements of the relevant aspects of the Training Package or accredited course had been met. It includes reviewing and making recommendations for future improvements to the assessment tool, process and/or outcomes. What is moderation? Moderation is the process of bringing assessment judgements and standards into alignment. It is a process that ensures the same standards are applied to all assessment results within the same Unit(s) of Competency. It is an active process in the sense that adjustments to assessor judgements are made to overcome differences in the difficulty of the tool and/or the severity of judgements.

19. This table summarises the similarities and differences between validation and moderation. This table summarises the similarities and differences between validation and moderation.

20. Focus - Tool Has clear, documented evidence of the procedures for collecting, synthesising, judging and recording outcomes (i.e., to help improve the consistency of assessments across assessors [inter-rater reliability]). Has evidence of content validity (i.e., whether the assessment task(s) as a whole, represents the full range of knowledge and skills specified within the Unit(s) of competency. Reflect work-based contexts, specific enterprise language and job-tasks and meets industry requirements (i.e., face validity). Adheres to the literacy and numeracy requirements of the Unit(s) of Competency (construct validity). Has been designed to assess a variety of evidence over time and contexts (predictive validity). Has been designed to minimise the influence of extraneous factors (i.e., factors that are not related to the unit of competency) on candidate performance (construct validity).

21. Focus - Tool Has clear decision making rules to ensure consistency of judgements across assessors (inter-rater reliability) as well as consistency of judgements within an assessor (intra-rater reliability). Has a clear instruction on how to synthesise multiple sources of evidence to make an overall judgement of performance (inter-rater reliability). Has outlined appropriate reasonable adjustments that could be made to the gathering of assessment evidence for specific individuals and/or groups. Has evidence that the principles of fairness and flexibility have been adhered to. Has been designed to produce sufficient, current and authentic evidence. Is appropriate in terms of the level of difficulty of the task(s) to be performed in relation to the skills and knowledge specified within the relevant Unit(s) of Competency. Has adhered to the relevant organisation assessment policy. When validating assessment tools, you may want to focus on whether the tool ���.When validating assessment tools, you may want to focus on whether the tool ���.

22. Focus - Judgement Check whether the judgement was too harsh or too lenient by reviewing samples of judged candidate evidence against the: Requirements set out in the Unit(s) of Competency; Benchmark samples of candidate evidence at varying levels of achievement (including borderline cases); and the Assessment decision making rules specified within the assessment tools. Desirable for validation, mandatory for moderation When checking the appropriateness of assessor judgements, it may be necessary to check�..When checking the appropriateness of assessor judgements, it may be necessary to check�..

23. Types of Approaches � Assessor Partnerships Validation only Informal, self-managed, collegial Small group of assessors May involve: Sharing, discussing and/or reviewing one another�s tools and/or judgements Benefit Low costs, personally empowering, non-threatening Weakness Potential to reinforce misconceptions and mistakes Types of Approaches Types of Approaches

24. Types of Approaches - Consensus Typically involves reviewing their own & colleagues assessment tools and judgements as a group Can occur within and/or across organisations Strength Professional development, networking, promotes collegiality and sharing Weakness Less quality control than external and statistical approaches as they can also be influenced by local values and expectations Requires a culture of sharing

25. Types of Approaches - External Types Site Visit Versus Central Agency Strengths Offer authoritative interpretations of standards Improve consistency of standards across locations by identifying local bias and/or misconceptions (if any) Educative Weakness Expensive Less control than statistical

26. Types of Approaches - Statistical Limited to moderation Yet to be pursued at the national level in VET Requires some form of common assessment task at the national level Adjusts level and spread of RTO based assessments to match the level and spread of the same candidates scores on a common assessment task Maintains RTO-based rank ordering but brings the distribution of scores across groups of candidates into alignment Strength Strongest form of quality control Weakness Lacks face validity, may have limited content validity

27. Summary of major distinguishing features Validation is concerned with quality review whilst moderation is concerned with quality control; The primary purpose of moderation is to help achieve comparability of standards across organisations whilst validation is primarily concerned with continuous improvement of assessment practices and outcomes; Whilst validation and moderation can both focus on assessment tools, moderation requires access to judged (or scored) candidate evidence. The latter is only desirable for validation; Both consensus and external approaches to validation and moderation are possible. Moderation can also be based upon statistical procedures whilst validation can include less formal arrangements such as assessor partnerships; and The outcomes of validation are in terms of recommendations for future improvement to the assessment tools and/or processes; whereas moderation may also include making adjustments to assessor judgements to bring standards into alignment, where determined necessary. 

28. Principles Transparent Representative Confidential Educative Equitable Tolerable Principles were selected based upon universal acceptance across all educational sectors, nationally and internationally, and non susceptible to political/system level changes. The first five principles most people would be familiar with, the final principle, tolerability was included to emphasis the fact that there will always be some level of variability in assessment, the challenge is to determine how much is acceptable. See next slide�Principles were selected based upon universal acceptance across all educational sectors, nationally and internationally, and non susceptible to political/system level changes. The first five principles most people would be familiar with, the final principle, tolerability was included to emphasis the fact that there will always be some level of variability in assessment, the challenge is to determine how much is acceptable. See next slide�

29. Tolerable

30. Associate Professor Shelley Gillis Deputy Director, Work-based Education Research Centre Ph: 61 3 9689 3280 Mobile: 0432 756 638 email: shelley.gillis@vu.edu.au web: www.werc.vu.edu.au� � CONTACT DETAILS