Bias & Fairness in Tests (Rust & Golombok)

Bias & Fairness in Tests(Rust & Golombok) • Important to note: serious consequences follow from test results! • Think about it from the client’s perspective • Selection test: You don’t get the job • Academic test: You lose a year’s work • Clinical tests: get/don’t get help • Important to make the right decisions based on the results

Bias and fairness • How “correct” our decisions are can be thought of in terms of two properties: • Fairness - The social justice issues surrounding the employment of the test • Bias - A statistical artefact in the test which makes it respond differently to different groups

Fairness in tests • This is not so much a flaw the test as of the use of the test • Can only consider fairness in terms of the societal norms which apply • We can examine the pattern of decisions which have been made based on that test

Fairness and Justice • Depending on the focus on authority in a society, tests might be applied more or less strictly • An “unfair test” is one whose consequences do not match the value system of the society

Fairness: an example • Imagine we set a super hard exam for 206F • Two thirds of the class fails the test • I can do two things: • Accept the marks as they are and see you next year • Adjust the test marks to increase the pass rate • Which is the fair thing to do?

Fairness: an example • If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do • If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do

Bias in tests • A bias exists in a test if gives different results for different populations Example: Army Alpha Soldiers almost always scored lower than officers

Bias: good or bad? • Good: Bias can be used to identify which population a client belongs to • Should he be officer or soldier? • Bad: Creates a false impression of difference between groups • Foreign language ability can “reduce” intelligence

Bias: Good or Bad? • Irrelevant: If the test only really gets used by one population, who cares? • Decide on the importance of bias based on the situation • Bias is never an issue of “right” or “wrong” - it is a purely statistical concept

Forms of bias • Bias can appear in three forms: • Item bias • Intrinsic test bias • Extrinsic test bias • We can examine each of these sources of bias separately and address each individually

Item bias • The bias exists in individual questions • eg. a questions about dollars, quarters and dimes would be biased • Linguistic bias (idioms, slang, etc) • common interracial bias • This type of item is common in IQ tests (!)

Identifying item bias • Do item analysis • I.e. check out each item of the test separately • Identify possible relevant subgroups • Work out the “facility value” of each question for each group • the proportion of people who get it right

Item bias: example • Imagine we have a test with 3 questions • We think it might be language biased • Look at the groups: native english speakers, others • Work out facility value • Native: A: 0.68 B: 0.96 C: 0.57 • Other: A: 0.59 B: 0.24 C: 0.59 • Big difference in item B, so it is biased

Item offensiveness • Eg: Shown an engineer and a psychologist “Which is smarter?” • Related to item bias • offensive items not necessarily biased • Offensive items should be removed • May interfere with subsequent items

Intrinsic test bias • The test has different mean scores for different groups • Does not exist in specific questions (Item bias), but rather a general phenomenon • Common in language groups • It is a matter of degree

Causes of intrinsic bias • Tests created with a specific group in mind are biased in this way • Other groups perform worse • The more different the group, the bigger the difference

Extrinsic test bias • When the test is unbiased, but decisions made using the test are biased • Eg. a test finds a true difference, and this leads to one group getting selected more than another • Extrinsic bias is the overlap between fairness and bias

Extrinsic bias: example • On the SAT, poorer children tend to perform worse than richer children • This is a real difference - poorer children have less access to the requirements to academic success • The SAT was used to select university applicants • poorer children were selected far less frequently

Extrinsic bias and ideology • Do you believe in “true differences” • Or do you rather believe in “unexplored potrential”? • It is a fact that poorer kids did worse at university (in the USA) • Do we use this as a basis for not selecting them? • No way to take a “scientific” ideology-free standpoint on extrinsic bias

Bias & Fairness in Tests (Rust & Golombok)