Roy b clariana pennsylvania state university usa rclariana@psu edu
Download
1 / 25

Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu - PowerPoint PPT Presentation


  • 211 Views
  • Uploaded on

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference. Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu. Patricia E. Wallace The College of New Jersey, USA PWallace@tcnj.edu.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Roy B. Clariana Pennsylvania State University, USA RClariana@psu.edu' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Roy b clariana pennsylvania state university usa rclariana@psu edu

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference

Roy B. Clariana

Pennsylvania State University, USA

RClariana@psu.edu

Patricia E. Wallace

The College of New Jersey, USA

PWallace@tcnj.edu

Veronica M. Godshalk

University of South Carolina, USA

Godshalk@gwm.sc.edu

Available at: http://www.personal.psu.edu/rbc4/CELDA.ppt


How do pronouns influence computer based text analysis
How do pronouns influence computer-based text analysis? computer-based analysis of essay questions:

  • Participants in an undergraduate business course (N = 49, 4 missing data, final N = 45) completed an essay as part of the course final examination and investigators manually edited every occurrence of pronouns in these essays to their antecedents.

  • The original unedited essays (with pronouns) and the edited essays (no pronouns) were processed by ALA-Reader software using linear aggregate and sentence aggregate methods. These data were then analyzed using a Pathfinder network (PFNET) approach.


Ala reader t ext analysis
ALA-Reader computer-based analysis of essay questions: text analysis

ALA-Reader – free software, converts text to a network representation (open with CMAP tools from IHMC) and also outputs relationship data in the prx format that can be analyzed by KNOT Pathfinder network software.

( http://www.personal.psu.edu/rbc4)


Ala reader t ext analysis1
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Example sentence from a student’s essay

Key words are underlined

Pronouns are shown in red

ALA-Reader has two approaches,

either linear aggregate or sentence aggregate


Ala reader t ext analysis2
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

job

empowered

job

humanist

satisfaction

employee

productivity

productivity

Here is the sentence aggregate run shown in black.

Historical basis…an excel spreadsheet to do sentence aggregates


Ala reader t ext analysis3

humanist computer-based analysis of essay questions:

job

satisfaction

productivity

employee

empowered

ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.

Here is the linear aggregate run shown in black,

so far so good,

but what happens on the 2nd occurrence of ‘job’


Ala reader t ext analysis4
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more.

humanist

satisfaction

job

.

productivity

employee

.

empowered


Ala reader t ext analysis5
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more.

satisfaction

humanist

job

The force-directed

graph begins to

fold back on itself

productivity

empowered

.

employee


Ala reader t ext analysis6
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

The force-directed

graph begins to

fold back on itself


Ala reader t ext analysis7
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Unedited essay – 7 links

(pronouns present but invisible)

Edited essay – 8 links

(pronouns converted to referents)


Ala reader t ext analysis8
ALA-Reader computer-based analysis of essay questions: text analysis

When pronouns are changed to their referent, the term Humanist (right figure) takes on central importance in the network (has the most links).

Unedited essay – 7 links

(pronouns present but invisible)

Edited essay – 8 links

(pronouns converted to referents)


Ala reader data output

½ array = prx file computer-based analysis of essay questions:

29 terms used here

ALA-Reader data output


Ala reader t ext analysis9
ALA-Reader computer-based analysis of essay questions: text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

job

empowered

job

humanist

satisfaction

employee

productivity

productivity

Here is the sentence aggregate run shown in black.

Sparse sentences under specified and dense sentences over specified, pronouns always under specify representations in the sentence aggregate approach.


Deriving and measuring group knowledge structure via computer

ALA-Reader computer-based analysis of essay questions: Sentence and linear aggregate output

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Linear aggregate

prx output

Sentence aggregate

prx output


The essay writing prompt and 29 key terms
The essay writing prompt computer-based analysis of essay questions: and 29 key terms

“Describe and contrast in an essay of 300 words or less the following management theories: Classical/Scientific Management, Humanistic/Human Resources, Contingency, and Total Quality Management. Please use the terms below in your essay: administrative principles, benchmarking, bureaucratic organizations, contingency, continuous improvement, customers, customer focus, efficiency, employee, empowerment, feelings, Hawthorne studies, human relations, humanistic, leadership, management (i.e., bosses), Management by Objectives, motivate, needs, organization (i.e., corporation), plan, product, quality, relationship, scientific management (classical), service, situation (or environment), TQM, and work (or job, task).”


Descriptive data regarding the essays
Descriptive data regarding the essays computer-based analysis of essay questions:

  • The 45 student essays averaged 301 words per essay (range from 170 to 479 words, standard deviation of 70.1).

  • The fifteen most common words account for 31% of all of the text and include in order the (6.2% of all words), and (3.3%), of (3.3%), to (3.1%), management (2.4%), a (1.9%), is (1.8%), in (1.6%), that (1.4%), employees (1.4%), this (1.1%), on (1.1%), are (0.9%), as (0.9%), and their (0.9%).

  • The three most common pronouns are their (ranked 15th. 0.9% of all words), it (18th, 0.8%), and they (19th, 0.8%). Students use of ‘their’ and ‘they’ in their essays indicates that they were writing from the perspective of either a manager or of an employee and not always as an independent observer.


Prx output for student essays b07 and b10 29 key terms
prx output for student essays b07 and b10 (29 key terms) computer-based analysis of essay questions:

KNOT software reads prx files in this format


Example prx average of student essays b07 and b10
Example: prx computer-based analysis of essay questions: average of student essays b07 and b10

It is possible (even easy) to use KNOT software to average together any number of individual student prx (essay) files in order to represent a team or group composite network representation; and then compare group-to-group and individual-to-group, etc. We compared top (n=15) and bottom (n=15) students.


Pathfinder knot
Pathfinder KNOT computer-based analysis of essay questions:

  • KNOT analyzes prx data

  • Based on graph theory, KNOT uses an algorithm to determine a least-weighted path that links all of the terms. The resulting PFNET (force-directed graph) is based on a data reduction approach that is purported to represent the most salient relationships in the raw proximity data.


Linear aggregate pfnet group average representation of the top 15 student essays
Linear aggregate computer-based analysis of essay questions: PFNET group average representation of the top 15 student essays

29 common links

Unedited essay – 31 links

(pronouns are not used)

Edited essay – 34 links

(pronouns converted to referents)


Deriving and measuring group knowledge structure via computer

Unedited essay computer-based analysis of essay questions:

Edited essay

It is fascinating to us to compare the PFNET visual depictions of the averaged essays. For example, the terms ‘management’, ‘employee’, ‘organization’, ‘customer’, and ‘product’ are well connected and so are central terms in both representations. Three of the four super-ordinate management theory categories in the essay writing prompt, ‘scientific management’, ‘contingency’, and ‘TQM’, were all associated with ‘management’ while the fourth category, ‘humanistic’, was associated with both ‘management’ and ‘employee’. Emotion-related terms such as ‘feelings’, ‘needs’, ‘relationship’, and ‘motivation’ were all associated with ‘employee’ in both PFNET representations as were the action words ‘benchmarking’, ‘work’, and ‘product’.


Comparing unedited and edited group representations
Comparing unedited and edited group representations computer-based analysis of essay questions:

Pearson correlations of the averaged proximity raw data (above the diagonal) and PFNETsimilarity data (below the diagonal in red) for the top group (n = 15) and the bottom group (n = 15) for both unedited essays and edited essays.


Individual essay scores
Individual essay scores computer-based analysis of essay questions:

  • To calculate students’ essay scores, we used KNOT to compare the students PFNET to an expert PFNET. Student’s score consisted of the number of links in common with the expert.

  • A “universal” essay score for each student was established for comparison purposes using the SPSS factor score combination of the three human essay scores plus the ALA-Reader linear aggregate unedited essay score


Individual essay scores1
Individual essay scores computer-based analysis of essay questions:

Correlations with ALA-Reader individual essay scores are shown in red. The length-adjusted ALA-Reader score correlation with the universal score (r = .84) is almost as good as that of the human raters.


Summary conclusions
Summary/Conclusions computer-based analysis of essay questions:

  • The ALA-Reader sentence aggregate approach was not discussed much in this presentation, it didn’t perform well, but it is described in the manuscript

  • For the ALA-Reader linear aggregate approach, there was little difference between the PFNETs obtained for edited and unedited essays. Apparently the linear approach is relatively robust to pronoun interference and is generally superior to the sentence aggregate method for the narrow purposes of average group knowledge representation and for scoring individual essays.