The need for international guidelines on computer based testing and the internet
This presentation is the property of its rightful owner.
Sponsored Links
1 / 22

The need for international guidelines on computer-based testing and the Internet. PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on
  • Presentation posted in: General

The need for international guidelines on computer-based testing and the Internet. Dave Bartram Research Director SHL Group plc. Why do we need International Guidelines?. Individual countries are no longer ‘closed’ systems.

Download Presentation

The need for international guidelines on computer-based testing and the Internet.

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The need for international guidelines on computer based testing and the internet

The need for international guidelines on computer-based testing and the Internet.

Dave Bartram

Research Director

SHL Group plc


Why do we need international guidelines

Why do we need International Guidelines?

  • Individual countries are no longer ‘closed’ systems.

    • Use of WWW for distance assessment – especially in Recruitment and 360 Feedback.

    • Mobility of test users and cross-border access to materials, especially in EU.

    • Impact of globalisation on assessment procedures

  • Variations between countries in

    • standards adopted and training provision

    • regulation of access to test materials

    • statutory and non-statutory controls over testing and test use


A thought

A Thought

  • It took:

    • 38 years for radio

    • 13 years for TV

    • 4 years for WWW

      ……….to reach 50m Users.

      Now it is over 500m


Issues

Issues

  • Internet is an international medium. It no longer makes sense to try to control testing with restrictions in one country if there are no restrictions in the next country.

  • Rapid proliferation of testing on the Internet.

    • Standard inventories being administered remotely

    • Timed ability tests being put on the Internet in un-timed and unproctored modes

    • Lots of DIY tests appearing – some for serious use, many for ‘amusement’ – with little or no information about provenance for the user.


Risk management

Risk Management

  • Testing is an exercise in risk management.

  • For example, we balance:

    • Risks of item exposure against benefits of volume assessment

    • Risk of collusion on self-report measures against logistic benefits and cost savings of unsupervised administration

  • Internet changes the balance and creates need to review some of these risks and consider new risk mitigation strategies.

  • Good practice is about managing risk appropriately


Risk is managed by exercising control

Risk is managed by exercising control

  • Control over prior practice – informing test takers, starting from a level playing field, avoiding uncontrolled exposure of sensitive items.

  • Control over materials – psychometric quality, security, updating, everyone using the same versions.

  • Control over test takers – authentication: knowing who is taking the test

  • Control over test conditions - ensuring conformity to critical administration constraints: e.g. timing, clarity of display etc.


What are the functions of assessment supervision

What are the functions of assessment supervision?

  • To authenticate identity of test takers

  • To establish rapport with the test taker

  • To ensure the standardised conditions are followed

  • To ‘validate’ results (prevent cheating and collusion)

  • To deal with problems

  • To ensure security of materials


What are the functions of assessment supervision1

What are the functions of assessment supervision?

  • To authenticate identity of test takers

  • To establish rapport with the test taker

  • To ensure the standardised conditions are followed

  • To ‘validate’ results (prevent cheating and collusion)

  • To deal with problems

  • To ensure security of materials

    These are difficult to manage remotely


Internet test administration modes

Internet Test Administration modes

Insecure mode

Moderately secure mode

Secure

modes


High stakes testing zones in the recruitment funnel

Controlled =

Moderate control +

Major cost benefits

Supervised/

Managed =

traditional

& comfortable

Open =

High volume,

but low control

HIGH STAKES TESTING & ZONES IN THE RECRUITMENT FUNNEL


Cognitive ability testing supervised

Cognitive ability testing: supervised

  • Traditional tests need high security to protect items.

  • Most tests are timed or need good time control

    • Standard HTML web pages not good for this

    • Need downloadable aplets which can run even if server link is down.

    • Difficult to manage adaptive testing unless bandwidth is high: Managed Mode allows for download of item sets.


Issues of cheating fakeability

ISSUES OF CHEATING & FAKEABILITY

  • What is it sensible to put in the Open end of the selection funnel?

    • Practice tests

    • Work samples and tools aimed at optimising self-selection in/out

  • Cognitive tests can be administered in ‘open’ modes but:

    • There is no control over cheating

    • There is no candidate authentication

    • There is no means of safeguarding item security

  • OK for low stakes assessment with item sets where item exposure is of no concern.

    • E.g. Self-assessment for career guidance


Moving timed testing up the funnel

MOVING TIMED TESTING UP THE FUNNEL

  • Constrained random generation of parallel tests from large item banks

    • Multiple versions of items

    • Unique test delivered to each candidate

    • Standardised scoring through IRT item calibration

  • Used unsupervised as a short-form pre-sift, with a supervised retest at Assessment Centre for final short-list.

  • Supported by ‘honesty contract’.


Cheating verifiability australian study with n 716 graduate applicants

ZONE A

CHEATING & VERIFIABILITY(Australian study with N=716 graduate applicants)

Supervised

Re-test

CUT-OFF

Online

Test


The need for international guidelines on computer based testing and the internet

Frequency

0

25

NMG4 at Assessment Centre (cut-off =25)

Pre ONT

Post ONT

Equivalent to a saving of c.$2,700

per successful candidate

Pass rate 30% to 50% (67%)


Conclusions

CONCLUSIONS

  • Main challenges for testing on the internet are:

    • Differentiating genuine tests from all the pseudo-tests

    • Technical, delivery issues

    • Security and authentication issues

  • Need to consider best practice in relation to different modes of administration

    • Open – low stakes and low security

    • Controlled – can be high stakes (if followed up), moderate security

    • Supervised or Managed– high stakes, high security

  • Need to consider how testing fits into the overall assessment process in order to manage content security, cheating and authentication.

  • Need international guidelines setting out best practice.


Thank you

Thank you


Honesty policy and honesty management

Honesty policy and honesty management

The management of test-taker honesty within a high stakes assessment process is not just a matter of supervision. It is also a matter of the design of the whole process and the extent to which cheating or dishonest behaviour is perceived as being likely to be detected.

Assessment processes for selection can be backed by an explicit honesty policy to which candidates are asked to sign-up. This is supported by the process of re-assessing, under controlled supervised conditions in the final stages of the selection process, any key competencies which formed the basis for the sift. Such agreements are used to 'click-wrap' the application. In order to enter the application process, the applicant has formally to agree to a statement such as the following:

“I understand that the results of this assessment may be used in the process of deciding whether my application will be progressed further.

I understand that if it is progressed, I may be required to complete a similar assessment again under supervised conditions.

I confirm that I will endeavour to complete the assessment in accordance with the instructions given.

I will do so on my own, without seeking or accepting assistance or support from any other person or persons.

I undertake neither to make copies of any of the questions that will appear on screen during this assessment nor to pass on to any other person any information about the content of the assessment.”

At the end of the test they may be required to further acknowledge that:

“I confirm that the responses given by me during the assessment were provided without the assistance of any other persons and without the use of any aids or devices other than those explicitly permitted by the instructions”

While such 'contracts' are not legally binding or able to guarantee that the applicant has actually abided by them, they do help provide a clear set of expectations and explain that failure to abide by these conditions could have undesirable consequences.


  • Login