Shannon N. Whitten, Psychology Steve Fiore , Philosophy Valerie Sims, Psychology

1. Do Students Alter their Writing Style when they are being Evaluated by an Automated Essay Grader? Shannon N. Whitten, Psychology Steve Fiore, Philosophy Valerie Sims, Psychology University of Central Florida Funded by UCF Information Fluency Initiative

2. Why use Automatic Essay Graders(AEGs)? • Encourage instructors of large classes to use essay assessments when appropriate • May be used to provide feedback or a grade • Holistic • Analytic

3. Why be concerned with student reactions? • AEGs are likely to become more prevalent • If getting a grade or feedback from AEG, students may adjust their approach to composing the essay, • As a result, the assessment may not be valid.

4. Present Study • Investigate whether or not (and how) students’ composition of essays change if they know they will be assessed by an AEG. • It is possible that students modify their approach to writing an essay when they believe that it will be graded by a computer instead of a human • More keywords • Less attention to style and structure • More focus on semantic content versus pragmatic aspects of language

5. Hypotheses:Experiment 1 • When students know they are being graded by an AEG rather than a professor, they will alter their writing styles in the following ways: • H1: Anaphoric References will decrease in AEG condition. • H2: Proportion of Emotion words will decrease in AEG condition. • H3: Higher proportion of clauses will tie directly to content in AEG condition. • H4: Content that establishes common-ground will be reduced in the AEG condition, such as words or phrases about school (when written for a professor).

6. Procedure:Experiment 1 • 43 students in Cognitive Psychology • Randomly assigned to 1 of 2 conditions • Instructions: Write a short essay about one specific area of Cognitive Psychology (such as implicit memory, pragmatics of language or the Gestalt principles of Perception). Include in your essay how what you have learned about this topic can apply that to something in your life: your job, your schoolwork, your future, your family, anything at all. You will be given 1 hour to complete the essay. You should be informed that this essay will be graded using an automatic computer essay grader and not by the TA or the professor. / You should be informed that this essay will be graded by the professor and not the TA for this course.

7. Analysis of Essays • Anaphoric References (H1) were measured using Coh-Metrix • McNamara, Louwerse, Cai, & Graesser, 2005 • Emotion (H2) and Occupation/School (H4) related words were measured using Linguistic Inquiry and Word Count (LIWC, 2001) • Pennebaker, Francis, & Booth, 2001 • Content Graded by Instructor (H3)

8. Results:Experiment 1 • Hypotheses 1 – 3 were not supported • nsd for anaphoric references, emotion words, or proportion of content • Hypothesis 4 was supported • Students in the Professor condition included more words associated with school: • Professor: 2.03% (SD = 1.26) • AEG: 0.90 % (SD = .71) • Students in the Professor condition included more words associated with occupations: • Professor: 3.66 (SD = 1.79) • AEG: 2.58 (SD = 1.23)

9. Experiment 2 • Does the type of essay (Persuasive vs. Informative) interact with instructions (AEG or Professor) to influence student writing style? • Hypothesis: Students don’t alter their style for informative essays but they do for persuasive essays • Same DVs: anaphor reference, emotion words, school words, and proportion of content clauses. • Same Design for 53 students in Perception • New Essay Instructions

10. Instructions: Experiment 2 • For this last assignment, you will need to answer 2 essay questions. You will be given unlimited time to complete these essays, but you must complete the assignment once you start it. You will need to use the internet to answer the questions. You may want to write it in your Word processing program and then copy and paste it here. You should be informed that this essay will be graded using an automatic computer essay grader and not by the TA / the professor and not the TA for this course.

11. Essay Questions:Experiment 2 • Describe the physiological process of vision starting with the receptor cells to the cortex. Be specific about the structures involved (Informative) • In Chapter 14, we learned about how cochlear implants can allow some deaf individuals to hear. However, not everyone agrees that this is the best option for everyone. Research this issue on the internet. Write 1 – 2 paragraphs about this issue. Do you believe that cochlear implants should be recommended to individuals who have nerve deafness? Why or why not? (Persuasive)

12. Results for Experiment 2: School Words • There was an interaction such Instructions did not matter when the essay was Informative: • AEG: 0.16 (SD = .29) • Professor: 0.10 (SD = .18) • However, instructions did matter when the essay was Persuasive. Students used more school-related words when they wrote a persuasive essay. • AEG: 0.15 (SD = .32) • Professor: 37 (SD = .41) • Difference significant with Bonferroni adjustment

13. Results: School WordsExperiment 2, Cont.

14. Results for Experiment 2: Occupation Words • There was an interaction such Instructions did not matter when the essay was Informative: • AEG: 1.16 (SD = .62) • Professor: 0.94 (SD = .64) • However, instructions did matter when the essay was Persuasive. Students used more words to describe careers when they wrote a persuasive essay. • AEG: 1.17 (SD = .65) • Professor: 1.77 (SD = .1.21) • Difference approaches significance with Bonferroni

15. Results: Occupation WordsExperiment 2, Cont.

16. Conclusions • Students in this sample did not dramatically alter their writing style when graded by an AEG. • Students do seem to be sensitive to the fact that they are being graded by an AEG • This awareness does not influence the coherence, emotional expression, or substantive content of their essays.  • The only difference in writing style may be at the level of pragmatics, specifically in establishing common ground with the audience (professor)

17. Discussion • It should be noted the difference in occupation words was only 1% of the words in the essays. • Number of words in this category may or may not reflect an attempt to establish common ground. • Sometimes an attempt to establish common ground, but sometimes not • Issue of plagiarism; had access to the internet • Audience awareness is important for effective writing • Creators of AEGs might consider adding a component that gives feedback on this.

18. Analyses in Progress (and Predictions) • Flow Variables (Grammar, Coherence, etc.): • Number of Grammatical errors (will increase in Automatic Essay Grader condition) • Number of Transitions (will decrease in Automatic Essay Grader condition) • Sentence length (will be shorter in Automatic Essay Grader condition) • Argument Overlap (will decrease in Automatic Essay Grader condition) OPPOSITE, AEG > Professor • more repetitive content? • Zest variables: • Number of Opinion Statements (will decrease in Automatic Essay Grader condition) • Number of Examples (will decrease in Automatic Essay Grader condition) • Use of Figurative Language (will decrease in Automatic Essay Grader condition) • Content variables: • In addition to proportion of content, overall grade • Plagiarism of content

19. Thank you! Special thanks to RAs for scoring data: DeAnnCoillins Ashley Harper Alicia Glass Kevin Palmer Lauren Maleski