A national certification exam for child and youth care workers prelimary results
This presentation is the property of its rightful owner.
Sponsored Links
1 / 30

A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on
  • Presentation posted in: General

A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS. Dale Curry, Kent State University Basil Qaqish, University of North Carolina-Greensboro. The NACP Competencies. Organization of Competencies Domains Sub-Domains Categories Competencies. Domains.

Download Presentation

A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


A national certification exam for child and youth care workers prelimary results

A NATIONAL CERTIFICATION EXAM FOR CHILD AND YOUTH CARE WORKERS: PRELIMARY RESULTS

Dale Curry, Kent State University

Basil Qaqish, University of North Carolina-Greensboro


A national certification exam for child and youth care workers prelimary results

The NACP Competencies

Organization of Competencies

  • Domains

  • Sub-Domains

  • Categories

  • Competencies


Domains

Domains

I. Professionalism

II. Cultural & Human Diversity

III. Applied Human Development

IV.Relationship & Communication

V. Developmental Practice Methods


Example from nacp competencies document

ExamplefromNACP Competencies Document


A national certification exam for child and youth care workers prelimary results

I. PROFESSIONALISM 6. Advocacy – MEDIUM WEIGHT a. demonstrate knowledge and skills in use of advocacy -SUPERVISOR ASSESSMENTb. access information on the rights of children, youth and families - PORTFOLIO c. describe the rights of children youth and families in relevant setting/s and systems – EXAMINATIONd. advocate for the rights of children, youth, and families in relevant setting/s and systems - PORTFOLIO e. describe safeguards for protection from abuse including institutional abuse - EXAMINATION f. advocate for safeguards for protection from abuse including institutional abuse - SUPERVISOR ASSESSMENT g. ensure that children, youth and family views are heard and considered during the decision making processes which directly affect them -EXAMINATION


Situational judgment items

Situational Judgment Items

  • Can be used to place competencies in a context of relationships, events, and settings

  • Can help establish face validity of test if they are based on real case studies collected from the field

  • Moves beyond the simple recitation of learned knowledge


Situational judgment items1

Situational Judgment Items

  • Case studies of real incidents were collected from child and youth care practitioners in a variety of practice settings

  • Used as the basis for the exam questions.


Example of situational judgment item

Example ofSituational Judgment Item


Example of situational judgment item1

Example of Situational Judgment Item

Competencies assessed by examination:

I.6.c. Describe the rights of children youth and families in relevant setting/s and systems

I.6.e. Describe safeguards for protection from abuse including institutional abuse


A national certification exam for child and youth care workers prelimary results

Item 8: Read this case study of an incident and answer the questions at the end:

Lisa (a girl in care) became angry during individual counseling session with Social Service Manager (MH) and cursed at the staff member. Lisa was escorted to “off dorm” time out due to disrupting milieu. While in time out, Lisa continued to use profanity toward staff members, urinated on the floor, and said “maybe I should pimp again and make some money…


A national certification exam for child and youth care workers prelimary results

After giving her several expectations and after Lisa refused to calm down, staff member (JT) began to physically escort Lisa to seclusion when Lisa became physically assaultive, hitting staff member. Staff members JT and SR placed Lisa in a therapeutic hold and eventually removed her to seclusion room. By 8:15 P.M., Lisa regained physical control of herself and agreed to a plan to maintain the safety of Lisa and the staff members. Lisa cleaned seclusion room and time out room.


Which of the following statements are true circle true or false for each statement

Which of the following statements are true? [Circle True or False for each statement]

T/F Lisa’s rights were properly protected in

this situation.

T/F The physical restraint, as explained in

this case study was justified and proper.

T/F An alternative intervention such as

having the staff member (JT) switch off

with another staff member could have

been more effective.


Pilot testing

Pilot Testing


Study goals

STUDY GOALS

  • Administer exam and get feedback regarding face validity & ways to improve

  • Conduct item analysis

  • Examine relationship between test scores and supervisory assessment of worker performance (concurrent validity)

  • Explore differential performance results

  • Determine a cut score (pass/fail)


Method

METHOD

  • Assessment Work Group developed 100 item situational judgment exam.

  • Exam was administered to 775 child and youth care workers in 29 sites.

  • Sites located in six states & two provinces.


Method1

METHOD

  • Scores were statistically compared with supervisor’s rating of competence.

  • Supervisor ratings of worker competence (six items) administered.

  • Examinee feedback on exam and face validity solicited through questionnaire


Procedures

PROCEDURES

  • Item analysis

  • Correlation of exam total score with supervisor composite ratings

  • Modified Angoff procedure to determine cut point (pass/fail)


Item analysis

ITEM ANALYSIS

  • Reliability analysis

  • Difficulty analysis

  • Discrimination analysis

  • Distracter analysis

  • Differential Item Analysis (DIF)


Reliability analysis

RELIABILITY ANALYSIS

  • Excellent internal reliability (Chronbach’s alpha=.90)

  • 18 items identified for possible deletion or revision


Difficulty analysis

DIFFICULTY ANALYSIS

  • Difficulty scores obtained for each item (acceptable difficulty score range from .10-.80)

  • Also examined top and bottom 20%

  • Most items were of moderate difficulty


Discrimination analysis

DISCRIMINATION ANALYSIS

  • Point Bi-serial correlation between the item and total test score

  • Also compared top 20% with bottom 20%

  • .15 minimum discrimination score


Distracter analysis

DISTRACTER ANALYSIS

Figure 1


Differential item functioning dif

DIFFERENTIAL ITEM FUNCTIONING (DIF)

  • Exploring differences for race and gender for those with similar abilities on the test

  • Use of computer program SIBTEST and Mantel Haenszel procedure

  • 7 items identified by either SIBTEST or Mantel Haenszel for race and 1 item for gender (must give these items more scrutiny)


Face validity and other feedback from examinees

FACE VALIDITY AND OTHER FEEDBACK FROM EXAMINEES

  • 90% indicate that the items in the exam accurately assess important aspects of child and youth care work

  • 90% indicate that the case studies provide realistic samples of child and youth care work

  • 80% indicate that the exam content is similar to their job duties

  • 59% indicate that their performance on the exam is an excellent indicator of their job performance (34% neither agreed or disagreed)


Other feedback

OTHER FEEDBACK

  • Test is consistent with real situations-overall a good and fair test

  • Motivating and thought provoking

  • Test too long – perhaps give a break or do it in two parts

  • Not familiar with some of the terminology

  • Some of the case studies were hard to follow


Canadian results

CANADIAN RESULTS

  • Mean total exam score - 75.7 (U.S. 59.3)

  • Mean competency rating (composite of 6 items, 5 point scale) – 28.4 (U.S. 26.2)

  • Mean face validity rating (composite of 4 items, 5 point scale) – 15.4 (no significant difference with U.S. examinees)


Comments canadian examinees

COMMENTS CANADIAN EXAMINEES

  • Exam was too long, a break was needed

  • Well-thought questions with very realistic, engaging scenarios but a mix in style of question type (e.g. true/false) may have been a welcome

  • The exam did not reflect the cultural aspects of British Columbia

  • Knew about N.A. Code but only familiar with Ontario Code of Ethics


Post pilot activities

POST PILOT ACTIVITIES

  • Explored items for possible deletion to improve reliability (18 items)

  • Examined content of items identified by DIF for gender/racial bias

  • Examined examinee feedback pertaining to each item and regarding the test in general

  • Completed process of setting cut-off (passing scores) for exam


What remains to be done

What remains to be done?

  • Continue to collect and analyze data to improve the exam and accumulate evidence of the exam’s validity

  • Continue to develop and research new exam items

  • Continue to explore reliability and validity of exam in future as it is implemented


More information

More Information

North American Certification Project

www.acycp.org


  • Login