employability and thinking skills a validity study
Download
Skip this Video
Download Presentation
Employability and Thinking Skills: A Validity Study

Loading in 2 Seconds...

play fullscreen
1 / 26

Employability and Thinking Skills: A Validity Study - PowerPoint PPT Presentation


  • 123 Views
  • Uploaded on

Employability and Thinking Skills: A Validity Study. Megan Rodgers AERA 2013. What’s the Value of College?. Many colleges and universities uphold some form of a liberal arts education (e.g., general education).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Employability and Thinking Skills: A Validity Study' - javier


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
what s the value of college
What’s the Value of College?
  • Many colleges and universities uphold some form of a liberal arts education (e.g., general education).
  • Most higher education institutions include a set of common core outcomes (AAC&U, 2005)
  • Unfortunately, there has been

recent public debate about the

value of college

    • Many graduates have record levels

of debt and are unemployed

there is value in college
There is Value in College
  • Specifically, the skills obtained college should increase graduate employability, a student’s potential to be employed (Rampell, 2011; Yorke, 2004).
  • What is needed within higher education is a scale that measures a student’s potential to be employed
measurement of employability
Measurement of Employability
  • Typically, employability is measured by alumni surveys (Volkein, 2010)
    • Not good indicators of potential to be employed
      • Job market, personal circumstance, etc.
  • “Employability” instruments exist
    • Focus on knowledge of the employment process (e.g., Rothwell, Herbert, & Rothwell, 2008).
    • Basic skills (e.g., Brigance, 1995)
  • The Employability Skills Inventory (ESI; Liptak, 2010)
    • Purports to measure skills needed to be employed based on the U.S. Department of Labor’s SCANS Report (U.S. Department of Labor, 1991)
scans report
SCANS Report

Research Question: What skills are needed for young people to succeed in the increasing globalized world of work?

employability skills inventory
Employability Skills Inventory
  • 8 Subscales that directly relate to the SCANS Report
  • 80 items
    • 3 Point Response Option: Not True, Somewhat True, Very True
    • Commercial Scale- $4 per administration
    • No detail about item development; No validity evidence
focus of study
Focus of Study
  • Focus on gathering structural validity evidence for one subscale (Cronbach & Meehl, 1955; Bollen, 2010)
  • Cross-section of Liberal Arts Education and Employability
thinking skills
Thinking Skills
  • As defined in ESI (Liptak, 2010):
  • These abilities also align well with the critical thinking literature (Fisher, 2001).
  • In ESI measured with 10 items- 6 positively worded (e.g., I am able to…) and 4 negatively worded (e.g., I cannot…)

Thinking Skills: People who score high on this scale tend to be able to think creatively, make effective decisions, solve problems logically and efficiently, visualize how things work, apply sound reasoning skills, and use effective learning techniques to acquire new knowledge and skills.

construct validity
Construct Validity
  • Benson’s (1998) strong program of construct validity
    • Substantive Stage
      • Specifying and operationally defining construct
        • Backwards Translation
    • Structural Stage
      • Relating the structure of the items to the construct
        • CFA on Thinking Skills
    • External Stage
      • Relating the construct to related constructs
        • Not conducted due to earlier findings
substantive validity
Substantive Validity

Do the items on the ESI align with the defined domain for each of the eight subscales?

  • Backwards Translation
  • 12 graduate students, 1 professor
backwards translation results
Backwards Translation Results
  • Overall, alignment is not bad
  • Dual-loading common
  • Results useful for ESI revisions
testing structural validity

Thinking Skills

7

9

10

47

49

6

8

46

48

50

Negative Wording Method Effect

6*

48*

50*

7*

47*

8*

49*

9*

46*

10*

Testing Structural Validity

Unidimensional Model

Bifactor Model

Negatively worded items may create a method effect(Tomas & Oliver, 1999)

method
Method

Participants

  • ESI was electronically administered to 283 undergraduates through the participant pool in a proctored setting

Procedure

  • Finney (2001) suggests that a structured data collection effort will evoke greater attention by the student and reduce negative wording effects

40 items

40 items

After finishing the scales, students worked on their puzzle until others completed

data analysis
Data Analysis
  • Confirmatory Factor Analysis (CFA)
  • Data is categorical in nature (Finney & DiStefano, in press)
    • 3 Point Response Option: Not True, Somewhat True, Very True
    • Observed data are not continuous; however, underlying latent variable “Thinking Skills” is thought to be continuous
      • Estimate polychoric correlations
  • ML Estimation assumes data are continuous
    • Use Robust Diagonally Weighted Least Squares estimation (Finney & DiStefano, in press)
data analysis continued
Data Analysis (Continued)
  • Polychoric Correlations are LOW, limited variability
  • Can the proposed model reproduce the data?
data analysis continued1
Data Analysis (continued)
    • Fit Indices
      • χ2 (p>.05), RMSEA (~.06), CFI (~.95 or greater)
        • Tend to be fairly accurate with categorical data
      • Polychoric correlation residuals (~.10 or greater)
  • Nine cases were omitted using listwise deletion; N=274
  • All analyses were conducted using LISREL 8.80
results
Results
  • Unidimensional model did not fit the data
    • rDWLS χ2 (35) = 81.95, p <.001, RMSEA=0.07, CFI=0.86
    • 16 residuals greater than .10
  • Local misfit permeated the model
results continued

Thinking Skills

Thinking Skills

7

7

9

9

10

10

47

47

49

49

6

6

8

8

46

46

48

48

50

50

Negative Wording Method Effect

6*

6*

48*

48*

50*

50*

7*

7*

47*

47*

8*

8*

49*

49*

9*

9*

46*

46*

10*

10*

Results (Continued)
  • Bifactor model
    • Did not converge to an admissible solution
  • Ancillary analysis

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

e

results continued1
Results (Continued)
  • Ancillary Analysis did converge, as expected
    • Also did not fit the data rDWLS χ2 (29) = 67.12, p <.001, RMSEA=0.07, CFI=0.89.
  • Bifactor didn’t converge because I was “Overfactoring” the data (Rindskopf, 1984)
discussion
Discussion
  • Limited substantive validity support
  • The proposed unidimensional model did not fit the Thinking Skills data
    • Local misfit permeated the model
  • No structural validity evidence for this subscale
  • Negative wording did not appear to affect data
    • Could be due to proctoring and data collection methodology (Finney, 2001)
  • Need for a measure of employability still exists
    • Unlikely that this need will be fulfilled with the ESI
references
References

Arum, R., & Roksa, J. (2011). Academically adrift: Limited learning on college campuses. Chicago, IL: University of Chicago Press.

Association of American Colleges and Universities. (2005).Liberal education outcomes. Washington, DC: Association of American Colleges and Universities. Retrieved from http://www.aacu.org/advocacy/pdfs/leap_report_final.pdf.

Brigance, A. H. (1995). Brigance(r)diagnostic employability skills inventory. North Billerica, MA: Curriculum Associates.

Finney, S.J. (1991). A comparison of the psychometric properties of negatively and positively worded questionnaire items. (Doctoral dissertation). University of Nebraska, Lincoln, Nebraska.

Finney, S. J., & DiStefano, C. (in press). Nonnormal and categorical data in structural equation modeling. In G. Hancock & R. Mueller (Eds.), Structural Equation Modeling: A Second Course, 2nd Edition. Information Age Publishing Inc.

Fisher, A. (2001). Critical thinking: an introduction. Cambridge, United Kingdom: Press Syndicate of the University of Cambridge. Retrieved from http://pactiss.org/wp-content/uploads/2011/09/Critical-thinking-introduction.pdf

Hacker, A., & Dreifus, C. (2010). Higher education?:how colleges are wasting our money and failing our kids--and what we can do about it. New York, NY: Times Books.

Liptak, J. (2010) Employability Skills Inventory. Indianapolis, IN: JIST Publishing.

references continued
References (continued)

Nye, C.D., & Drasgow, F. (2011). Assessing goodness of fit: simple rules of thumb simply do not work. Organizational Research Methods, 14 (3), 548-570.

Rampell, C. (2011, May 20). Once again: is college worth it?. The New York Times. Retrieved from http://economix.blogs.nytimes.com/2011/05/20/once-again-is-college-worth-it/

Rindskopf , D. (1984). Structural equation models empirical identification, heywood cases, and related problems.Sociological Methods and Research, 42, 104-120.

Rothwell, A., Herbert, I., & Rothwell, F. (2008). Self-perceived employability: construction and initial validation of a scale for university students. Journal of Vocational Behavior, 73, 1-12.

Tomas, J. M., & Oliver, A. (1999). Rosenberg\'s self-esteem scale: two factors or method effects. Structural Equation Modeling, 6(1), 84-98.

U.S. Department of Labor, Secretary\'s Commission on Achieving Necessary Skills. (1991). What work requires of schools. Washington, DC.

Volkwein, J. F. (2010), Assessing alumni outcomes. New Directions for Institutional Research, 2010: 125–139. doi: 10.1002/ir.335

Yorke, M. (2004). Employability in the undergraduate curriculum: some student perspectives. European Journal of Education, 39(4), 409-427.

ad