roy b clariana pennsylvania state university usa rclariana@psu edu
Download
Skip this Video
Download Presentation
Roy B. Clariana Pennsylvania State University, USA [email protected]

Loading in 2 Seconds...

play fullscreen
1 / 25

deriving and measuring group knowledge structure via computer ... - PowerPoint PPT Presentation


  • 210 Views
  • Uploaded on

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference. Roy B. Clariana Pennsylvania State University, USA [email protected] Patricia E. Wallace The College of New Jersey, USA [email protected]

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'deriving and measuring group knowledge structure via computer ...' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
roy b clariana pennsylvania state university usa rclariana@psu edu

Deriving and measuring group knowledge structure via computer-based analysis of essay questions: The effects of controlling anaphoric reference

Roy B. Clariana

Pennsylvania State University, USA

[email protected]

Patricia E. Wallace

The College of New Jersey, USA

[email protected]

Veronica M. Godshalk

University of South Carolina, USA

[email protected]

Available at: http://www.personal.psu.edu/rbc4/CELDA.ppt

how do pronouns influence computer based text analysis
How do pronouns influence computer-based text analysis?
  • Participants in an undergraduate business course (N = 49, 4 missing data, final N = 45) completed an essay as part of the course final examination and investigators manually edited every occurrence of pronouns in these essays to their antecedents.
  • The original unedited essays (with pronouns) and the edited essays (no pronouns) were processed by ALA-Reader software using linear aggregate and sentence aggregate methods. These data were then analyzed using a Pathfinder network (PFNET) approach.
ala reader t ext analysis
ALA-Reader text analysis

ALA-Reader – free software, converts text to a network representation (open with CMAP tools from IHMC) and also outputs relationship data in the prx format that can be analyzed by KNOT Pathfinder network software.

( http://www.personal.psu.edu/rbc4)

ala reader t ext analysis1
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Example sentence from a student’s essay

Key words are underlined

Pronouns are shown in red

ALA-Reader has two approaches,

either linear aggregate or sentence aggregate

ala reader t ext analysis2
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

job

empowered

job

humanist

satisfaction

employee

productivity

productivity

Here is the sentence aggregate run shown in black.

Historical basis…an excel spreadsheet to do sentence aggregates

ala reader t ext analysis3

humanist

job

satisfaction

productivity

employee

empowered

ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in their jobs, then they produced more.

Here is the linear aggregate run shown in black,

so far so good,

but what happens on the 2nd occurrence of ‘job’

ala reader t ext analysis4
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more.

humanist

satisfaction

job

.

productivity

employee

.

empowered

ala reader t ext analysis5
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then they produced more.

satisfaction

humanist

job

The force-directed

graph begins to

fold back on itself

productivity

empowered

.

employee

ala reader t ext analysis6
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

The force-directed

graph begins to

fold back on itself

ala reader t ext analysis7
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Unedited essay – 7 links

(pronouns present but invisible)

Edited essay – 8 links

(pronouns converted to referents)

ala reader t ext analysis8
ALA-Reader text analysis

When pronouns are changed to their referent, the term Humanist (right figure) takes on central importance in the network (has the most links).

Unedited essay – 7 links

(pronouns present but invisible)

Edited essay – 8 links

(pronouns converted to referents)

ala reader t ext analysis9
ALA-Reader text analysis

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

job

empowered

job

humanist

satisfaction

employee

productivity

productivity

Here is the sentence aggregate run shown in black.

Sparse sentences under specified and dense sentences over specified, pronouns always under specify representations in the sentence aggregate approach.

slide14

ALA-Reader Sentence and linear aggregate output

Humanists believed that jobsatisfaction was related to productivity. They found that if employees were given more freedom and power in theirjobs, then theyproduced more.

Linear aggregate

prx output

Sentence aggregate

prx output

the essay writing prompt and 29 key terms
The essay writing promptand 29 key terms

“Describe and contrast in an essay of 300 words or less the following management theories: Classical/Scientific Management, Humanistic/Human Resources, Contingency, and Total Quality Management. Please use the terms below in your essay: administrative principles, benchmarking, bureaucratic organizations, contingency, continuous improvement, customers, customer focus, efficiency, employee, empowerment, feelings, Hawthorne studies, human relations, humanistic, leadership, management (i.e., bosses), Management by Objectives, motivate, needs, organization (i.e., corporation), plan, product, quality, relationship, scientific management (classical), service, situation (or environment), TQM, and work (or job, task).”

descriptive data regarding the essays
Descriptive data regarding the essays
  • The 45 student essays averaged 301 words per essay (range from 170 to 479 words, standard deviation of 70.1).
  • The fifteen most common words account for 31% of all of the text and include in order the (6.2% of all words), and (3.3%), of (3.3%), to (3.1%), management (2.4%), a (1.9%), is (1.8%), in (1.6%), that (1.4%), employees (1.4%), this (1.1%), on (1.1%), are (0.9%), as (0.9%), and their (0.9%).
  • The three most common pronouns are their (ranked 15th. 0.9% of all words), it (18th, 0.8%), and they (19th, 0.8%). Students use of ‘their’ and ‘they’ in their essays indicates that they were writing from the perspective of either a manager or of an employee and not always as an independent observer.
prx output for student essays b07 and b10 29 key terms
prx output for student essays b07 and b10 (29 key terms)

KNOT software reads prx files in this format

example prx average of student essays b07 and b10
Example: prx average of student essays b07 and b10

It is possible (even easy) to use KNOT software to average together any number of individual student prx (essay) files in order to represent a team or group composite network representation; and then compare group-to-group and individual-to-group, etc. We compared top (n=15) and bottom (n=15) students.

pathfinder knot
Pathfinder KNOT
  • KNOT analyzes prx data
  • Based on graph theory, KNOT uses an algorithm to determine a least-weighted path that links all of the terms. The resulting PFNET (force-directed graph) is based on a data reduction approach that is purported to represent the most salient relationships in the raw proximity data.
linear aggregate pfnet group average representation of the top 15 student essays
Linear aggregate PFNET group average representation of the top 15 student essays

29 common links

Unedited essay – 31 links

(pronouns are not used)

Edited essay – 34 links

(pronouns converted to referents)

slide21

Unedited essay

Edited essay

It is fascinating to us to compare the PFNET visual depictions of the averaged essays. For example, the terms ‘management’, ‘employee’, ‘organization’, ‘customer’, and ‘product’ are well connected and so are central terms in both representations. Three of the four super-ordinate management theory categories in the essay writing prompt, ‘scientific management’, ‘contingency’, and ‘TQM’, were all associated with ‘management’ while the fourth category, ‘humanistic’, was associated with both ‘management’ and ‘employee’. Emotion-related terms such as ‘feelings’, ‘needs’, ‘relationship’, and ‘motivation’ were all associated with ‘employee’ in both PFNET representations as were the action words ‘benchmarking’, ‘work’, and ‘product’.

comparing unedited and edited group representations
Comparing unedited and edited group representations

Pearson correlations of the averaged proximity raw data (above the diagonal) and PFNETsimilarity data (below the diagonal in red) for the top group (n = 15) and the bottom group (n = 15) for both unedited essays and edited essays.

individual essay scores
Individual essay scores
  • To calculate students’ essay scores, we used KNOT to compare the students PFNET to an expert PFNET. Student’s score consisted of the number of links in common with the expert.
  • A “universal” essay score for each student was established for comparison purposes using the SPSS factor score combination of the three human essay scores plus the ALA-Reader linear aggregate unedited essay score
individual essay scores1
Individual essay scores

Correlations with ALA-Reader individual essay scores are shown in red. The length-adjusted ALA-Reader score correlation with the universal score (r = .84) is almost as good as that of the human raters.

summary conclusions
Summary/Conclusions
  • The ALA-Reader sentence aggregate approach was not discussed much in this presentation, it didn’t perform well, but it is described in the manuscript
  • For the ALA-Reader linear aggregate approach, there was little difference between the PFNETs obtained for edited and unedited essays. Apparently the linear approach is relatively robust to pronoun interference and is generally superior to the sentence aggregate method for the narrow purposes of average group knowledge representation and for scoring individual essays.
ad