Factors to Consider in Evaluating Research

1. Factors to Consider in Evaluating Research

2. Effects Related to the Research Situation • The Hawthrone Effect: a situation in which the experimental conditions are such that a mere fact that the subject is aware of participating in an experiment, is aware of the hypothesis, or is receiving special attention tends to improve performance. • The John Henry Effect: a situation found in educational research in which a control group performs above its usual average when placed in competition with an experimental group that is using a new method or procedure that threatens to replace the control procedure. • The Pygmalion Effect: changes the subject’s behavior that are brought about by the experimenter’s expectations. • Demand Characteristics: Describes all the cues available to the subject regarding the nature of research.

3. Experimenter and Statistical Contamination • Contamination: Refers to any situation in which data that should be kept independent to satisfy the requirements for sounds research have in some way become interrelated. • Experimenter Contamination: Usually arises when the research worker has knowledge of the subject’s performance on the independent variable, and this knowledge influences the observation of the behavior of the subject on the dependent variable. • Statistical Contamination: Occurs when data that have in some way become related are treated as being independent in the statistical analysis. • Critical Evaluation of Statistical Analysis: -Statistical analysis not clearly presented -Incorrect statistical methods were used to analyze data

4. Factors to Consider in Evaluating Research Formulation of the Research Hypothesis or Objective • Examine critically whether the hypothesis or objective was developed specifically from theory and previous research findings. Deliberate Bias or Distortion • Is there more interest in obtaining evidence to support a particular viewpoint than in discovering truth? • Should not be so much be an attempt to prove a point, but to get an answer • Deliberate attempts to mislead a reader: through percentages, decimal places, surprising or newsworthy findings.

5. Factors to Consider in Evaluating Research cont. Nondeliberate Bias: • Biases influencing results without researcher’s awareness. • Does the phraseology used suggest that the research worker is inclined to favor one side of the question? • Is emotional or intemperate language of either a favorable or an unfavorable nature employed? • Does the person hold a theoretical position or have a stake in a particular point of view, or does the researcher belong to a groups (racial, vocational, religious, political, ethnic, etc.) that would predispose the person in a given direction about the subject of the research?

6. Factors to Consider in Evaluating Research cont. Sampling Bias • Did the study use volunteers? • Have subjects been lost? • In an effort to get subjects who differ in the variable being studied, have groups been selected that also differ in other important respects? • Are subjects extremely nonrepresentative of the population?

7. Factors to Consider in Evaluating Research cont. • Critical Evaluation of Measurement Techniques • What reliability data are available? • Similar results for the same subjects at different times or under different conditions • What validity evidence is available? • Can’t accept standardized educational measures at face value and assume validity • Is the measure appropriate for the sample? • Are test norms appropriate? • Comparability between the research group and the test norm group should be checked • Observer Bias • Human beings have a disturbing tendency to see what they want to see, hear what the want to hear, and remember what they want to remember.