Bias fairness in tests rust golombok l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 19

Bias & Fairness in Tests (Rust & Golombok) PowerPoint PPT Presentation


  • 178 Views
  • Uploaded on
  • Presentation posted in: General

Bias & Fairness in Tests (Rust & Golombok). Important to note: serious consequences follow from test results! Think about it from the client’s perspective Selection test: You don’t get the job Academic test: You lose a year’s work Clinical tests: get/don’t get help

Download Presentation

Bias & Fairness in Tests (Rust & Golombok)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Bias fairness in tests rust golombok l.jpg

Bias & Fairness in Tests(Rust & Golombok)

  • Important to note: serious consequences follow from test results!

    • Think about it from the client’s perspective

    • Selection test: You don’t get the job

    • Academic test: You lose a year’s work

    • Clinical tests: get/don’t get help

  • Important to make the right decisions based on the results


Bias and fairness l.jpg

Bias and fairness

  • How “correct” our decisions are can be thought of in terms of two properties:

    • Fairness - The social justice issues surrounding the employment of the test

    • Bias - A statistical artefact in the test which makes it respond differently to different groups


Fairness in tests l.jpg

Fairness in tests

  • This is not so much a flaw the test as of the use of the test

  • Can only consider fairness in terms of the societal norms which apply

  • We can examine the pattern of decisions which have been made based on that test


Fairness and justice l.jpg

Fairness and Justice

  • Depending on the focus on authority in a society, tests might be applied more or less strictly

  • An “unfair test” is one whose consequences do not match the value system of the society


Fairness an example l.jpg

Fairness: an example

  • Imagine we set a super hard exam for 206F

  • Two thirds of the class fails the test

  • I can do two things:

    • Accept the marks as they are and see you next year

    • Adjust the test marks to increase the pass rate

  • Which is the fair thing to do?


Fairness an example6 l.jpg

Fairness: an example

  • If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do

  • If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do


Bias in tests l.jpg

Bias in tests

  • A bias exists in a test if gives different results for different populations

Example: Army Alpha

Soldiers almost always

scored lower than

officers


Bias good or bad l.jpg

Bias: good or bad?

  • Good: Bias can be used to identify which population a client belongs to

    • Should he be officer or soldier?

  • Bad: Creates a false impression of difference between groups

    • Foreign language ability can “reduce” intelligence


Bias good or bad9 l.jpg

Bias: Good or Bad?

  • Irrelevant: If the test only really gets used by one population, who cares?

  • Decide on the importance of bias based on the situation

  • Bias is never an issue of “right” or “wrong” - it is a purely statistical concept


Forms of bias l.jpg

Forms of bias

  • Bias can appear in three forms:

  • Item bias

  • Intrinsic test bias

  • Extrinsic test bias

  • We can examine each of these sources of bias separately and address each individually


Item bias l.jpg

Item bias

  • The bias exists in individual questions

    • eg. a questions about dollars, quarters and dimes would be biased

  • Linguistic bias (idioms, slang, etc)

    • common interracial bias

  • This type of item is common in IQ tests (!)


Identifying item bias l.jpg

Identifying item bias

  • Do item analysis

    • I.e. check out each item of the test separately

  • Identify possible relevant subgroups

  • Work out the “facility value” of each question for each group

    • the proportion of people who get it right


Item bias example l.jpg

Item bias: example

  • Imagine we have a test with 3 questions

  • We think it might be language biased

  • Look at the groups: native english speakers, others

    • Work out facility value

  • Native: A: 0.68B: 0.96C: 0.57

  • Other:A: 0.59B: 0.24C: 0.59

  • Big difference in item B, so it is biased


Item offensiveness l.jpg

Item offensiveness

  • Eg: Shown an engineer and a psychologist “Which is smarter?”

  • Related to item bias

    • offensive items not necessarily biased

  • Offensive items should be removed

    • May interfere with subsequent items


Intrinsic test bias l.jpg

Intrinsic test bias

  • The test has different mean scores for different groups

  • Does not exist in specific questions (Item bias), but rather a general phenomenon

  • Common in language groups

  • It is a matter of degree


Causes of intrinsic bias l.jpg

Causes of intrinsic bias

  • Tests created with a specific group in mind are biased in this way

    • Other groups perform worse

    • The more different the group, the bigger the difference


Extrinsic test bias l.jpg

Extrinsic test bias

  • When the test is unbiased, but decisions made using the test are biased

  • Eg. a test finds a true difference, and this leads to one group getting selected more than another

  • Extrinsic bias is the overlap between fairness and bias


Extrinsic bias example l.jpg

Extrinsic bias: example

  • On the SAT, poorer children tend to perform worse than richer children

    • This is a real difference - poorer children have less access to the requirements to academic success

  • The SAT was used to select university applicants

    • poorer children were selected far less frequently


Extrinsic bias and ideology l.jpg

Extrinsic bias and ideology

  • Do you believe in “true differences”

    • Or do you rather believe in “unexplored potrential”?

  • It is a fact that poorer kids did worse at university (in the USA)

    • Do we use this as a basis for not selecting them?

  • No way to take a “scientific” ideology-free standpoint on extrinsic bias


  • Login