Personal attribute assessments for higher education admissions
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

“Personal Attribute” Assessments for Higher Education Admissions PowerPoint PPT Presentation


  • 139 Views
  • Uploaded on
  • Presentation posted in: General

“Personal Attribute” Assessments for Higher Education Admissions. Patrick Kyllonen, Director Center For New Constructs ETS [email protected] Kyllonen, P.C. (May, 2009). “Personal Attribute” Assessments for Higher Education Admissions. Colloquium. Catholic University, Santiago, Chile.

Download Presentation

“Personal Attribute” Assessments for Higher Education Admissions

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Personal attribute assessments for higher education admissions

“Personal Attribute” Assessments for Higher Education Admissions

Patrick Kyllonen, DirectorCenter For New ConstructsETS

[email protected]

Kyllonen, P.C. (May, 2009). “Personal Attribute” Assessments for Higher Education Admissions. Colloquium. Catholic University, Santiago, Chile.


Outline

Outline

  • Importance of personal attributes in higher education

  • Projects designed to assess personal attributes

    • European Self Assessment Approaches

    • Robert Sternberg’s efforts

    • Michigan State/College Board project

    • ETS® Personal Potential Index (PPI)

  • Future steps


Validity of personal attributes

Validity of Personal Attributes

  • Noftle & Robins meta analysis (2007)

    • 20 studies (N = 5000), r (Conscientiousness, college grades) = .29

    • 4 new studies (N = 8500)

      • r (conscientiousness, college GPA) = .19 (median over 3 samples), controlling for Gender, high school GPA, SAT-V, SAT-M

      • Partially mediated by

        • Academic effort

        • Perceived academic ability

      • Key facets of Conscientiousness

        • Diligence

        • Perfectionism

        • Prudence

        • Competence

        • Achievement-striving

        • Self Discipline

  • Not

    • Organization

    • Order

    • Dutifulness

    • Deliberation


Employers rating skill very important

% employers rating skill “very important”

4-yrgrads

2-yrgrads

HS

grads

Non-cog

cog


European self assessment approaches

European Self Assessment Approaches

  • Several German (Aachen, Frankfurt) and Austrian (Vienna) Universities are using self assessments to help students select majors (see EAPA symposium, 2007)

  • Example: Prof. Psychology, Lutz Hornke, Aachen University

    • Engineering Department

    • Problem: High drop out rate

    • Solution: Self Assessments for high school students (internet delivered)

  • Content

    • Personality assessments, interest inventories, cognitive tests

  • Approach

    • Compare student applicants with department averages to help student decide “fit”

  • Evaluation

    • Compare outcomes after implementation


Robert sternberg s efforts

Robert Sternberg’s Efforts

  • Based on decades of research on “triarchic theory”

    • 3 kinds of intelligence—practical, creative, analytic

  • Study 1: University of Michigan Business School

    • Practical intelligence measures

      • Situational Judgment & Case Scenario problems

    • Findings: Incremental validity over GMAT, particularly for course projects

    • Results: Difficulty in implementation

Hedlund, J., Wilt, J.M., Nebel, K., Ashford, S.E., & Sternberg, R.J. (2006). Assessing Practical Intelligence in Business School Admissions: A Supplement to the Graduate Management Admissions Test. Learning and Individual Differences, 16, 101-127.


Successful intelligence assessment personnel shortage 5 others

“Successful Intelligence Assessment” (“Personnel Shortage” (5 others))

The setting is a manufacturing plant facing a personnel shortage. Employees are working excessive amounts of overtime and morale is low. (5-15 pages of further description.) You are the Human Resources Manager. What do you do?

Situational Judgment Problems (3 subproblems each)

Immediately hire 200 new workers (rate 1-7)

Meet with unit heads to discuss strategies (rate 1-7)

Case scenario problems

What is the main problem?

What would you do to address the main problem?

What information did you focus on?

What outcome do you expect? What obstacles?


Robert sternberg s efforts1

Robert Sternberg’s Efforts

  • Study 2: Rainbow Project (undergraduate admissions)

    • Consortium of several universities; sponsored by the College Board

    • Measures

      • Creative—interpreting cartoons; writing essays

      • Analytic—traditional ability measures

      • Practical—situational judgment tests (next page)

    • Results

      • Incremental validity of new measures

      • Reduced impact with minority groups

    • Outcomes

      • Sponsor decided to experiment with other measures


Example tacit knowledge intentory item

Example “Tacit Knowledge Intentory” Item

You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do?

___Tell the worker that you think he is worthless.

___Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor.

___Avoid all conversation and eye contact with the other worker.

___Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day.

___Tell your supervisor that you refuse to work with this man.

___The project is going to be impossible to accomplish with this worker, so you may as well not even try--you can always blame your bad work partner.

___See if you can convince one of your friends to take your place and work with this employee.

___Demand a raise from your supervisor; you should not have to tolerate these conditions.


Robert sternberg s efforts2

Robert Sternberg’s Efforts

  • Study 3: Kaleidoscope Project (Tufts University undergraduate admissions)

    • Optional essays, evaluated for creative (“what if”), practical, and wisdom skills using a rubric

    • Results

      • New measures did not overlap much with standardized tests

      • New measures did not predict grades

      • New measures were correlated with extracurricular activities

    • Consequence

      • Increase in minority group enrollments

      • Sends message that other skills are important for college


College board michigan state university effort

12 dimension taxonomy

College Board/Michigan State University Effort

  • Taxonomy of Valued Academic Skills (mission statements from 35 universities)

VII. Social responsibility, citizenship and involvement

VIII. Physical and psychological health

IX. Career orientation

X. Adaptability and life skills

XI. Perseverance

XII. Ethics and integrity

I. Knowledge, learning, mastery of general principles

II. Continuous learning, intellectual interest and curiosity

III. Artistic cultural appreciation and curiosity

IV. Multicultural tolerance and appreciation

V. Leadership

VI. Interpersonal skills

Schmitt, N., Oswald, F. L., Kim, B. H., Imus, A., Drzakowski, S., Friede, A., & Shivpuri, S. (2007). The use of background and ability profiles to predict college student outcomes. Journal of Applied Psychology, 92, 165-180.

Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-207.


College board michigan state university effort1

12 dimension taxonomy

College Board/Michigan State University Effort

  • Sample Situational Judgment Item (“Leadership”)

An important class project you have been working on with a group of other students is not developing as it should because of petty differences and the need of some members to satisfy their own agenda. How would you proceed? (mark “most” and “least” likely to do)

__Try to solve the group problems before starting on the work.

__Work hard by yourself to make sure the project is finished, taking on others’ share of the work if necessary.

__Talk to the professor and get suggestions about solving the problem. If that doesn’t work, try to switch groups or have an independent project.

__Schedule a number of meetings, forcing the group to interact.

__Take charge and delegate tasks to each person. Make them responsible for their part of the project.

__Talk to the group and demand that they start working together.

  • Sample Biodata Item (“Leadership”)

How many times in the past year have you tried to get someone to join an activity in which you were involved or leading?

a.  Never; b. Once; c. Twice; d. 3 or 4 times; e. 5 times or more.


College board michigan state university effort2

12 dimension taxonomy

College Board/Michigan State University Effort

  • Results (N = 1000--2000; k = 10 institutions)

    • Small incremental validity (over grades and tests)

      • With grade point average as criterion

    • Large incremental validity

      • Time to graduate; Absenteeism, Self-ratings, Peer-ratings

    • Smaller disparate impact (vs. minority groups)

  • Status

    • Larger study outcome data currently being analyzed (2009)

      • N = 8000; k = 15

    • Concerns about faking (especially biodata)

    • Concerns about cost and politics of implementation


The case for letters of recommendation

The Case for Letters of Recommendation

  • Top three criteria for evaluating graduate students (interview studies):

    • Grades

    • Standardized-test scores (GRE)

    • Letters of recommendation

  • Letters are useful

    • Give qualitative information about applicants

    • Provide information about students’ motivation, persistence, creativity, and personality that are otherwise difficult to obtain


But letters have their problems

But Letters have their Problems

  • May reflect the writing skills of the referee more than the qualities of the applicant

  • Ambiguous language

  • Problem of the omission

  • Hard to determine reliability, inter-rater agreement, and validity

  • Time intensive for letter writers & readers


What is the ets ppi

What is the ETS® PPI?

  • Rating system for advisors to rate prospective graduate students

    • Supplement to GRE

  • Based on extensive research on the critical attributes for success

    • Pilot tested with ETS summer interns; Project 1000

  • Reveals candidate’s critical personal attributes beyond grades, test scores, and recommendation letters

  • Gives a more complete picture of a candidate’s potential

  • Enriches the evaluation process—offers greater opportunity for students

    • Gives students the opportunity to provide evidence of a broad range of capabilities

  • Helps create a more diverse and successful graduate program


Personal attribute assessments for higher education admissions

PPI scales and items

Resilience

Accepts feedback without getting defensive

Works well under stress

Can overcome challenges and setbacks

Works extremely hard

Planning and Organization

Sets realistic goals

Organizes work and time effectively

Meets deadlines

Makes plans and sticks to them

Ethics and Integrity

Is among the most honest persons I know

Maintains high ethical standards

Is worthy of trust from others

Demonstrates sincerity

Knowledge and Creativity

Has a broad perspective on the field

Is among the brightest persons I know

Produces novel ideas

Is intensely curious about the field

Communication Skills

Speaks in a clear, organized, and logical manner

Writes with precision and style

Speaks in a way that is interesting

Organizes writing well

Teamwork

Supports the efforts of others

Behaves in an open and friendly manner

Works well in group settings

Gives criticism/feedback to others in a helpful way


Findings and next steps

Findings and Next Steps

  • PPI has good psychometric properties

    • Reliable

    • Measures several factors reliably (cognitive and noncognitive)

    • Predicts GPA in high school over standardized tests

    • Faculty ratings match mentor ratings

    • Project 1000 analysis underway

    • Is well received

  • Will be part of GRE starting summer 2009

    • Fall 2009 – initial PPI data collection from several universities

    • Link PPI to variety of graduate school outcomes

  • Long term consequential validity

    • Will signal the importance of personal attributes

    • Will help create more diverse and successful graduate programs


Summary

Summary

  • Several uses for personal attributes

    • Self assessments for career decision making

    • High stakes assessments

  • For high stakes, several promising approaches for “personal attribute” assessment

    • Biodata

    • Situational Judgment Tests

    • Ratings by Others (e.g., teachers), as in PPI

  • We will know a lot more in one or two years!

  • Long term consequential validity

    • Will signal the importance of personal attributes

    • Will help create more diverse and successful classes/graduate programs


  • Login