The Conceptualization and Validation of the Vanderbilt Assessment of Leadership in Education (VAL-ED...
This presentation is the property of its rightful owner.
Sponsored Links
1 / 33

The Conceptualization and Validation of the Vanderbilt Assessment of Leadership in Education (VAL-ED) Ellen Goldring, Vanderbilt University PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on
  • Presentation posted in: General

The Conceptualization and Validation of the Vanderbilt Assessment of Leadership in Education (VAL-ED) Ellen Goldring, Vanderbilt University Henry May, University of Pennsylvania June 2010 . The Goal – Produce a Psychometrically Sound Assessment of School Leadership.

Download Presentation

The Conceptualization and Validation of the Vanderbilt Assessment of Leadership in Education (VAL-ED) Ellen Goldring, Vanderbilt University

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The conceptualization and validation of the vanderbilt assessment of leadership in education val ed ellen goldring

The Conceptualization and Validation of the Vanderbilt Assessment of Leadership in Education (VAL-ED)

Ellen Goldring, Vanderbilt University

Henry May, University of Pennsylvania

June 2010


The goal produce a psychometrically sound assessment of school leadership

The Goal – Produce a Psychometrically Sound Assessment of School Leadership

The team – Andy Porter, Joe Murphy, Ellen Goldring, Steve Elliott, Xiu Cravens, Morgan Polikoff, Henry May,

Funding: The Wallace Foundation and IES


Acknowledgments

Acknowledgments

  • The VAL-ED instrument is published and distributed by Discovery Education Assessment.

  • We are authors of the VAL-Ed, and while we have made every effort to be objective and data-based in my statements about this instrument in this presentation, readers should judge the facts and related information materials for themselves and make independent decisions regarding the use of the instrument.


The case for leadership assessment

The Case for Leadership Assessment

  • Most school leadership improvement focuses on professional development, mentoring, licensing policies, and standards.

  • Minimal attention has been paid to assessment, feedback, and subsequent action.

  • Leadership assessment and feedback is an important missing link to improving and strengthening school leadership.

  • Assess behaviors associated with student achievement and learning


Learning centered leadership

Learning –Centered Leadership

  • Staying consistently focused on learning, teaching, curriculum, and assessment

  • Making all the other dimensions of schooling (e.g., administration, organization, finance) work in the service of improved student learning


Leadership behavior framework

Leadership Behavior Framework


Definitions of core components

Definitions of Core Components

  • High Standards for Student Learning—There are individual, team,

    and school goals for rigorous student academic and social learning.

  • Rigorous Curriculum (content)—There is ambitious academic content provided to all students in core academic subjects.

  • Quality Instruction (pedagogy)—There are effective instructional practices that maximize student academic and social learning.

  • Culture of Learning & Professional Behavior—There are integrated communities of professional practice in the service of student academic and social learning. There is a healthy school environment in which student learning is the central focus.

  • Connections to External Communities—There are linkages to family and/or other people and institutions in the community that advance academic and social learning.

  • Performance Accountability— Leadership holds itself and others responsible for realizing high standards of performance for student academic and social learning. There is individual and collective responsibility among the professional staff and students.


Definitions of key processes

Definitions of Key Processes

  • Planning—Articulate shared direction and coherent policies, practices, and procedures for realizing high standards of student performance.

  • Implementing—Engage people, ideas, and resources to put into practice the activities necessary to realize high standards for student performance.

  • Supporting—Create enabling conditions; secure and use the financial, political, technological, and human resources necessary to promote academic and social learning.

  • Advocating—Promotes the diverse needs of students within and beyond the school.

  • Communicating—Develop, utilize, and maintain systems of exchange among members of the school and with its external communities.

  • Monitoring—Systematically collect and analyze data to make judgments that guide decisions and actions for continuous improvement.


The need for a valid and reliable principal leadership assessment tool

The Need for a Valid and Reliable Principal Leadership Assessment Tool

  • Virtually every school district in the United States (N = 14,000 school districts and over 90,000 schools) requires some form of evaluation of its principals.

  • Many states and districts have developed their own leadership assessment tools.

  • Our analysis of the assessments with large urban school districts indicates that few have a conceptual framework based on how leaders improve student learning, nor have they been validated for their intended uses.


Comparing sampled evaluations with the learning centered core components

Comparing Sampled Evaluations with the Learning-Centered Core Components


Comparing sampled evaluations with the learning centered core components1

Comparing Sampled Evaluations with the Learning-Centered Core Components


From the sampled principal leadership assessment instruments we find

From the sampled principal leadership assessment instruments, we find:

  • Various levels of specificity

  • Wide spread of assessed areas

  • Limited depth

  • Locally designed procedures for growth and development

  • More on knowledge and skills, less on behavior

  • Lack of consistent focus on school performance as measured by student achievement

  • No psychometric development or reporting

  • No 360 degree feedback


Assessing learning centered leadership the val ed vision

Assessing Learning-Centered Leadership:The VAL-ED vision…

A leadership assessment system that has the following properties:

Works well in a variety of settings and circumstances,

Is construct valid,

Is reliable,

If feasible for widespread use,

Provides accurate and useful reporting of results.

Is unbiased,

Yields a diagnostic profile for summative and formative purposes.

Can be used to measure progress over time in the development of leadership, and

Predicts important outcomes.

Part of a comprehensive assessment of the effectiveness of a leader's behaviors


The val ed instrument

The VAL-ED Instrument


The val ed instrument1

The VAL-ED Instrument

  • The instrument consists of 72 items defining six core component subscales and six key process subscales.

  • Principal, Teachers, & Supervisor provide a 360-degree, evidenced-based assessment of leadership behaviors.

  • Respondents rate effectiveness of 72 behaviors on scale 1=Ineffective to 5=Outstandingly effective.

  • Each respondent rates the principal’s effectiveness after indicating the sources of evidence on which the effectiveness is rated.

  • Two parallel forms of the assessment facilitate measuring growth over time.

  • The instrument is available in both paper and online versions.


The conceptualization and validation of the vanderbilt assessment of leadership in education val ed ellen goldring

Directions for Completing Rating Scale


An example set of responses

An Example Set of Responses


Interpretation of rating scale results

Interpretation of Rating Scale Results

Descriptive Analysis

Total Score

Core Components Subscale Scores

Key Process Subscale Scores

Norm-Referenced Profiles

Principal

Teacher

Supervisor

Total respondent composite

Criterion-Referenced Profiles

Distinguished

Proficient

Basic

Below basic


Aggregated effectiveness ratings

Aggregated Effectiveness Ratings


Leadership behaviors for possible improvement example of a potential area of growth

Leadership Behaviors for Possible ImprovementExample of a potential area of growth:


Instrument development and early studies of psychometric properties

Instrument Development and Early Studies of Psychometric Properties


Psychometric piloting

Psychometric Piloting

Item sorting study

Established content validity by asking education leaders to sort the items into 36 cells.

Cognitive interviews of paper/pencil version

Two rounds of cognitive interviews in three districts each.

Three respondents evaluated the format and items.

Nine-school pilot test (320/440 teachers = 73%)

Estimated reliability of each of 12 scales.

Established construct validity through factor analysis.

Established face validity through questions to respondents.

Bias review

Submitted to urban districts to evaluate language;


Val ed field test

VAL-ED Field Test

  • Target of 300 Schools

    • 100 elementary, 100 middle, 100 high schools

    • 150 urban schools, 100 suburban schools, 50 rural schools

    • Of the 150 urban schools, 50 from Wallace grantee districts and 50 from Wallace grantee states

    • 75 schools from each of four geographic areas

  • The Obtained Sample

    • 60 districts (9 Wallace districts)

    • 235 schools for principal data, 253 schools for supervisor data, 245 schools for teachers data (w/ fairly even distribution across grade levels)

    • 8,863 teachers

    • 218 schools for which there was data from all three respondent groups

    • 50% of the schools had a 68 percent teacher response or better


Reliability

Reliability

  • Internal Consistency – Internal consistency reliability was high for both total score and each of the core components and each of the key processes with all reliabilities .87 or higher.

  • Reliability of differences among core components and among key processes

    • Contrasts among Culture of Learning, External Communities, Performance Accountability, and Rigorous Curriculum were all good, ranging from .35 to .92.

    • For key processes, Supporting and Advocating could each be reliably contrasted with the other key processes, with reliabilities ranging from .25 to .92.


Factor analyses to investigate the conceptual framework

Factor Analyses to Investigate the Conceptual Framework

Confirmatory factor analysis provided support for both core components and key processes, despite high co-linearity among the core components and among the key processes.

Exploratory factor analysis with oblique rotation yielded an 8-factor solution with fairly consistent results between Forms A and C.

There was considerable support for the core components of Performance Accountability and Connections to External Communities and some support for the core component of Culture of Learning and Professional Behavior. There was also support for the key process of Supporting and the key process of Advocating.


Correlations among response groups

Correlations among Response Groups

For total score, the correlation between principal and supervisor was .13, principal and teacher .27, supervisor and teacher .18.

For core components, the average correlation between teacher and principal was .26, between teacher and supervisor .19, and supervisor and principal .13.

For key processes, the average correlation was .27 between principal and teacher, .17 between supervisor and teacher, and .13 between principal and supervisor.


Performance level descriptors

Performance Level Descriptors

  • Distinguished

    • A distinguished leader exhibits learning centered leadership behaviors at levels of effectiveness that are virtually certain to influence teachers positively and result in strong value-added to student achievement and social learning for all students.

  • Proficient

    • A proficient leader exhibits learning centered leadership behaviors at levels of effectiveness that are likely to influence teachers positively and result in acceptable value-added to student achievement and social learning for all students.

  • Basic

    • A leader at the basic level of proficiency exhibits learning centered leadership behaviors at levels of effectiveness that are likely to influence teachers positively and result in acceptable value-added to student achievement and social learning for some sub-groups of students, but not all.

  • Below basic

    • A leader at the below basic level of proficiency exhibits learning centered leadership behaviors at levels of effectiveness that are unlikely to influence teachers positively nor result in acceptable value-added to student achievement and social learning for students.


The bookmark procedure

The Bookmark Procedure

  • Twenty-two panelists: Ten principals, four teachers, four supervisors of principals, two researchers of school leadership, and two education policymakers

  • Item-ordered booklet based on the National Field Trial (items from Form A)

  • The cut between proficient and basic was set at 3.60.

  • The cut between distinguished and proficient was set at 3.77.

  • The cut between basic and below basic was set at 3.42.

  • The impact data are

    • 30% of the principals fall below basic

    • 50% fall below proficient

    • 70% fall below distinguished


Ongoing validity studies

Ongoing Validity Studies


  • Login