bias fairness in tests rust golombok
Skip this Video
Download Presentation
Bias & Fairness in Tests (Rust & Golombok)

Loading in 2 Seconds...

play fullscreen
1 / 19

Bias & Fairness in Tests (Rust & Golombok) - PowerPoint PPT Presentation

  • Uploaded on

Bias & Fairness in Tests (Rust & Golombok). Important to note: serious consequences follow from test results! Think about it from the client’s perspective Selection test: You don’t get the job Academic test: You lose a year’s work Clinical tests: get/don’t get help

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Bias & Fairness in Tests (Rust & Golombok)' - wayne

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
bias fairness in tests rust golombok
Bias & Fairness in Tests(Rust & Golombok)
  • Important to note: serious consequences follow from test results!
    • Think about it from the client’s perspective
    • Selection test: You don’t get the job
    • Academic test: You lose a year’s work
    • Clinical tests: get/don’t get help
  • Important to make the right decisions based on the results
bias and fairness
Bias and fairness
  • How “correct” our decisions are can be thought of in terms of two properties:
    • Fairness - The social justice issues surrounding the employment of the test
    • Bias - A statistical artefact in the test which makes it respond differently to different groups
fairness in tests
Fairness in tests
  • This is not so much a flaw the test as of the use of the test
  • Can only consider fairness in terms of the societal norms which apply
  • We can examine the pattern of decisions which have been made based on that test
fairness and justice
Fairness and Justice
  • Depending on the focus on authority in a society, tests might be applied more or less strictly
  • An “unfair test” is one whose consequences do not match the value system of the society
fairness an example
Fairness: an example
  • Imagine we set a super hard exam for 206F
  • Two thirds of the class fails the test
  • I can do two things:
    • Accept the marks as they are and see you next year
    • Adjust the test marks to increase the pass rate
  • Which is the fair thing to do?
fairness an example6
Fairness: an example
  • If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do
  • If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do
bias in tests
Bias in tests
  • A bias exists in a test if gives different results for different populations

Example: Army Alpha

Soldiers almost always

scored lower than


bias good or bad
Bias: good or bad?
  • Good: Bias can be used to identify which population a client belongs to
    • Should he be officer or soldier?
  • Bad: Creates a false impression of difference between groups
    • Foreign language ability can “reduce” intelligence
bias good or bad9
Bias: Good or Bad?
  • Irrelevant: If the test only really gets used by one population, who cares?
  • Decide on the importance of bias based on the situation
  • Bias is never an issue of “right” or “wrong” - it is a purely statistical concept
forms of bias
Forms of bias
  • Bias can appear in three forms:
  • Item bias
  • Intrinsic test bias
  • Extrinsic test bias
  • We can examine each of these sources of bias separately and address each individually
item bias
Item bias
  • The bias exists in individual questions
    • eg. a questions about dollars, quarters and dimes would be biased
  • Linguistic bias (idioms, slang, etc)
    • common interracial bias
  • This type of item is common in IQ tests (!)
identifying item bias
Identifying item bias
  • Do item analysis
    • I.e. check out each item of the test separately
  • Identify possible relevant subgroups
  • Work out the “facility value” of each question for each group
    • the proportion of people who get it right
item bias example
Item bias: example
  • Imagine we have a test with 3 questions
  • We think it might be language biased
  • Look at the groups: native english speakers, others
    • Work out facility value
  • Native: A: 0.68 B: 0.96 C: 0.57
  • Other: A: 0.59 B: 0.24 C: 0.59
  • Big difference in item B, so it is biased
item offensiveness
Item offensiveness
  • Eg: Shown an engineer and a psychologist “Which is smarter?”
  • Related to item bias
    • offensive items not necessarily biased
  • Offensive items should be removed
    • May interfere with subsequent items
intrinsic test bias
Intrinsic test bias
  • The test has different mean scores for different groups
  • Does not exist in specific questions (Item bias), but rather a general phenomenon
  • Common in language groups
  • It is a matter of degree
causes of intrinsic bias
Causes of intrinsic bias
  • Tests created with a specific group in mind are biased in this way
    • Other groups perform worse
    • The more different the group, the bigger the difference
extrinsic test bias
Extrinsic test bias
  • When the test is unbiased, but decisions made using the test are biased
  • Eg. a test finds a true difference, and this leads to one group getting selected more than another
  • Extrinsic bias is the overlap between fairness and bias
extrinsic bias example
Extrinsic bias: example
  • On the SAT, poorer children tend to perform worse than richer children
    • This is a real difference - poorer children have less access to the requirements to academic success
  • The SAT was used to select university applicants
    • poorer children were selected far less frequently
extrinsic bias and ideology
Extrinsic bias and ideology
  • Do you believe in “true differences”
    • Or do you rather believe in “unexplored potrential”?
  • It is a fact that poorer kids did worse at university (in the USA)
    • Do we use this as a basis for not selecting them?
  • No way to take a “scientific” ideology-free standpoint on extrinsic bias