Bias fairness in tests rust golombok
1 / 19

Bias Fairness in Tests Rust Golombok - PowerPoint PPT Presentation

  • Updated On :

Bias & Fairness in Tests (Rust & Golombok). Important to note: serious consequences follow from test results! Think about it from the client’s perspective Selection test: You don’t get the job Academic test: You lose a year’s work Clinical tests: get/don’t get help

Related searches for Bias Fairness in Tests Rust Golombok

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Bias Fairness in Tests Rust Golombok' - wayne

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Bias fairness in tests rust golombok l.jpg
Bias & Fairness in Tests(Rust & Golombok)

  • Important to note: serious consequences follow from test results!

    • Think about it from the client’s perspective

    • Selection test: You don’t get the job

    • Academic test: You lose a year’s work

    • Clinical tests: get/don’t get help

  • Important to make the right decisions based on the results

Bias and fairness l.jpg
Bias and fairness

  • How “correct” our decisions are can be thought of in terms of two properties:

    • Fairness - The social justice issues surrounding the employment of the test

    • Bias - A statistical artefact in the test which makes it respond differently to different groups

Fairness in tests l.jpg
Fairness in tests

  • This is not so much a flaw the test as of the use of the test

  • Can only consider fairness in terms of the societal norms which apply

  • We can examine the pattern of decisions which have been made based on that test

Fairness and justice l.jpg
Fairness and Justice

  • Depending on the focus on authority in a society, tests might be applied more or less strictly

  • An “unfair test” is one whose consequences do not match the value system of the society

Fairness an example l.jpg
Fairness: an example

  • Imagine we set a super hard exam for 206F

  • Two thirds of the class fails the test

  • I can do two things:

    • Accept the marks as they are and see you next year

    • Adjust the test marks to increase the pass rate

  • Which is the fair thing to do?

Fairness an example6 l.jpg
Fairness: an example

  • If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do

  • If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do

Bias in tests l.jpg
Bias in tests

  • A bias exists in a test if gives different results for different populations

Example: Army Alpha

Soldiers almost always

scored lower than


Bias good or bad l.jpg
Bias: good or bad?

  • Good: Bias can be used to identify which population a client belongs to

    • Should he be officer or soldier?

  • Bad: Creates a false impression of difference between groups

    • Foreign language ability can “reduce” intelligence

Bias good or bad9 l.jpg
Bias: Good or Bad?

  • Irrelevant: If the test only really gets used by one population, who cares?

  • Decide on the importance of bias based on the situation

  • Bias is never an issue of “right” or “wrong” - it is a purely statistical concept

Forms of bias l.jpg
Forms of bias

  • Bias can appear in three forms:

  • Item bias

  • Intrinsic test bias

  • Extrinsic test bias

  • We can examine each of these sources of bias separately and address each individually

Item bias l.jpg
Item bias

  • The bias exists in individual questions

    • eg. a questions about dollars, quarters and dimes would be biased

  • Linguistic bias (idioms, slang, etc)

    • common interracial bias

  • This type of item is common in IQ tests (!)

Identifying item bias l.jpg
Identifying item bias

  • Do item analysis

    • I.e. check out each item of the test separately

  • Identify possible relevant subgroups

  • Work out the “facility value” of each question for each group

    • the proportion of people who get it right

Item bias example l.jpg
Item bias: example

  • Imagine we have a test with 3 questions

  • We think it might be language biased

  • Look at the groups: native english speakers, others

    • Work out facility value

  • Native: A: 0.68 B: 0.96 C: 0.57

  • Other: A: 0.59 B: 0.24 C: 0.59

  • Big difference in item B, so it is biased

Item offensiveness l.jpg
Item offensiveness

  • Eg: Shown an engineer and a psychologist “Which is smarter?”

  • Related to item bias

    • offensive items not necessarily biased

  • Offensive items should be removed

    • May interfere with subsequent items

Intrinsic test bias l.jpg
Intrinsic test bias

  • The test has different mean scores for different groups

  • Does not exist in specific questions (Item bias), but rather a general phenomenon

  • Common in language groups

  • It is a matter of degree

Causes of intrinsic bias l.jpg
Causes of intrinsic bias

  • Tests created with a specific group in mind are biased in this way

    • Other groups perform worse

    • The more different the group, the bigger the difference

Extrinsic test bias l.jpg
Extrinsic test bias

  • When the test is unbiased, but decisions made using the test are biased

  • Eg. a test finds a true difference, and this leads to one group getting selected more than another

  • Extrinsic bias is the overlap between fairness and bias

Extrinsic bias example l.jpg
Extrinsic bias: example

  • On the SAT, poorer children tend to perform worse than richer children

    • This is a real difference - poorer children have less access to the requirements to academic success

  • The SAT was used to select university applicants

    • poorer children were selected far less frequently

Extrinsic bias and ideology l.jpg
Extrinsic bias and ideology

  • Do you believe in “true differences”

    • Or do you rather believe in “unexplored potrential”?

  • It is a fact that poorer kids did worse at university (in the USA)

    • Do we use this as a basis for not selecting them?

  • No way to take a “scientific” ideology-free standpoint on extrinsic bias