An automatic text mining framework for knowledge discovery on the web
This presentation is the property of its rightful owner.
Sponsored Links
1 / 62

An Automatic Text Mining Framework for Knowledge Discovery on the Web PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on
  • Presentation posted in: General

An Automatic Text Mining Framework for Knowledge Discovery on the Web. Wingyan Chung The University of Arizona March 30, 2004. Acknowledgments. NSF and NIJ Grants Dr. Hsinchun Chen, Dr. Jay F. Nunamaker , Dr. J. Leon Zhao, Dr. Richard T. Snodgrass, Dr. D. Terence Langendoen, Dr. Olivia Sheng

Download Presentation

An Automatic Text Mining Framework for Knowledge Discovery on the Web

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


An automatic text mining framework for knowledge discovery on the web

An Automatic Text Mining Framework for Knowledge Discovery on the Web

Wingyan Chung

The University of Arizona

March 30, 2004


Acknowledgments

Acknowledgments

  • NSF and NIJ Grants

  • Dr. Hsinchun Chen, Dr. Jay F. Nunamaker , Dr. J. Leon Zhao, Dr. Richard T. Snodgrass, Dr. D. Terence Langendoen, Dr. Olivia Sheng

  • Dept. of MIS, U. of Arizona

  • Artificial Intelligence Lab, U. of Arizona


Outline

Outline

  • Introduction

  • Literature Review

  • Research Formulation and Approach

  • Empirical Studies on Business Intelligence Applications

    • Previous Work

      • Building a BI Search Portal for Integrated Analysis on Heterogeneous Information

      • Using Visualization Techniques to Discover BI

    • Automating Business Stakeholder Analysis

  • Conclusions, Limitations and Future Directions


Introduction

Introduction


The internet

The Internet

  • Advances in electronic network and IT support ubiquitous access to and convenient storage of information

    • They have changed human lives fundamentally (Negroponte, 2003)

    • The role of global electronic network

    • Facilitation in communication and transaction

  • The Internet emerges as the largest global electronic network

    • Rapid growth (Lyman & Varian, 2000)

    • Advantages in information storage and retrieval, but …


Problems of the internet

Problems of the Internet

To effectively and efficientlydiscover knowledge (business intelligence) from vast amount of textual information on the Web

Challenges

Information Overload

Convenient storage hasmade information exploration difficult

???

Information is unreliable

Heterogeneity and unmonitored qualityof information on the Web

Hard to know all stakeholders

Interconnected nature of the Web complicates understanding of relationships


Research questions

Research Questions

How can we develop an automatic text mining approach to address the problems of knowledge discovery on the Web?

How effective and efficient does such an approach assist human beings in discovering knowledge on the Web?

What lessons can be learned from applying such an approach in the context of human-computer interaction (HCI)?


Literature review

Literature Review

Knowledge and Knowledge Management

Human-Computer Interaction

Text Mining for Web Analysis


Knowledge

Knowledge

Views

Classifications

  • Hierarchical view (Nunamaker et al., 2001)

  • Reversed hierarchy (Tuomi, 1999)

  • As a state of mind, an object, a process, access to information, and a capability (Alavi and Leidner, 2001)

  • Resource-based theory (Barney, 1991; Penrose, 1959; Wernerfelt, 1984; Drucker, 1995)

  • Tacit and explicit dimensions (Polanyi, 1965)

  • Individual vs. collective knowledge

  • Declarative vs. procedural knowledge

  • Causal, conditional, relational and pragmatic knowledge

  • Revealed underlying assumptions in KM

  • Implied different roles of knowledge in organizations

  • Textual knowledge - Most efficient way to store, retrieve, and transfer vast amount of information

  • Advanced processing needed to obtain knowledge

    • Traditionally done by humans

    • It is useful to review the discipline of Human-Computer Interaction to understand human analysis needs


Human analysis needs

Human Analysis Needs

  • Satisfied when the problem in information seeking is solved (Kuhlthau, 1993; Kuhlthau, Spink and Cool 1992; Saracevic, Kantor, Chamis and Trivison, 1988; Choo et al., 2000)

  • Involve value-adding processes:

    • Information seeking: locating useful information from large amount of data

    • Intelligence generation: acquisition, interpretation, collation, assessment, and exploitation of the information obtained (Davis, 2002)

    • Relationship extraction: deriving patterns and relationships from data and information

Knowledge Discovery


Need automating kd processes

Need Automating KD Processes

  • Human beings can undertake KD processes by applying their experience and knowledge

    • But inefficient and not scalable

  • Text mining has been identified as a set of technologies that can automate the knowledge discovery process(Trybula, 1999)

    • Stages: information acquisition, extraction, mining, presentation

      • Need more preprocessing when considering KD on the Web (more noisy, voluminous, heterogeneous sources): Collection building, conversion, extraction

    • Evolved from work in automatic text processing


Text mining technologies

Text Mining Technologies

  • For Web KD:

    • Web mining techniques: resource discovery on the Web, information extraction from Web resources, and uncovering general patterns (Etzioni, 1996)

      • Pattern extraction, meta searching, spidering

    • Web page summarization (Hearst, 1994; McDonald & Chen, 2002)

    • Web page classification (Glover et al., 2002; Lee et al., 2002; Kwon & Lee, 2003)

    • Web page clustering (Roussinov & Chen, 2001; Chen et al., 1998; Jain & Dube, 1988)

    • Web page visualization (Yang et al., 2003; Spence, 2001; Shneiderman, 1996)

  • These techniques and approaches can be used to automate important parts of human analyses


Summary

Summary

  • Human analyses are precise but not efficient and not scalable to the growth of the Web

  • A number of text mining techniques exist but there has not been a comprehensive approach to addressing problems of knowledge discovery on the Web, namely,

    • Information overload

    • Heterogeneity and unmonitored quality of information

    • Difficulties of identifying relationships on the Web

  • The HCI aspects of using a text mining approach to knowledge discovery on the Web have not been widely explored


Research formulation and approach

Research Formulation and Approach


Methodology

Methodology

  • System Development (Nunamaker et al., 1991)

    • A Multi-methodological Approach

    • Conceptual frameworks, Mathematical models

    • Observation, Experimentation

  • Validation

    • Effectiveness (accuracy, precision, recall), efficiency (time)

    • Information quality (Wang & Strong, 1996)

    • User satisfaction (subjective ratings and comments)


Domain of study

Domain of Study

  • Business intelligence applications

    • BI is increasingly becoming an important practice in today's organizations

      • More than 40% surveyed individuals by Fuld & Co. have organized BI efforts (Fuld et al., 2002)

    • Collecting and analyzing BI have become a profession

      • SCIP has over 50 chapters worldwide

      • A new journal called Journal of Competitive Intelligence and Management was launched in 2003

    • Vibrant growth of e-commerce calls for better approaches to knowledge discovery on the Web (Morgan-Stanley, 2003)

      • Businesses use the Web to share and disseminate information

      • Many companies are conducting business using the Internet platform (e.g., Amazon.com, EBay.com)

        • Our focus is on the first category


Empirical studies on business intelligence applications

Empirical Studies on Business Intelligence Applications


Previous work 1

Previous Work (1)

  • Building a BI search portal for integrated analysis on heterogeneous information

    • The portal provides post-retrieval analysis (summarization, categorization, meta-searching)

    • Conducted a systematic evaluation to test CBizPort's ability to assist human analysis of Chinese BI

    • Results:

      • Searching and browsing performance comparable to regional Chinese SEs

      • CBizPort could significantly augment existing SEs

      • Subjects strongly favored analysis capability of CBizPort summarizer and categorizer


Previous work 2

Previous Work (2)

  • Applying Web page visualization techniques to discovering BI

    • Two browsing methods (Web community and Knowledge map) were developed to help visualize the landscape of search engine results

      • WC uses a genetic algorithm; KM uses MDS

    • The methods were empirically compared against a graphical search engine (Kartoo) and a textual result list (RL) display

    • Results: KM > Kartoo (in terms of effectiveness, efficiency, and users' ratings on point placement); WC > RL (in terms of effectiveness, efficiency, and user satisfaction)


Using web page classification techniques to automate business stakeholder analysis

Using Web Page Classification Techniques to Automate Business Stakeholder Analysis


Current business environment

Current Business Environment

  • Networked business environment facilitates information sharing and collaboration (Applegate, 2003)

  • Collaborative commerce: automating business processes by electronic sharing of information

  • Knowledge sharing about stakeholder relationships through companies’ Web sites and pages

    • Textual content or annotated hyperlinks


Problems

Problems

  • Knowledge hidden in interconnected Web resources

    • Posing challenges to identifying and classifying various business stakeholders

      • e.g., A company’s manager may not know who are using their company’s Web resources

  • Need better approaches to uncovering such knowledge

    • Enhance understanding of business stakeholders and competitive environments


Related work

Related Work

  • Stakeholder theories have evolved over time while the view of firm changes

    • Production view (19th century): Suppliers and Customers

    • Managerial view (20th century): + Owners, Employees

    • Stakeholder view (1960-80s) (Freeman, 1984): + Competitors, Governments, News Media, Environmentalists, …

    • E-commerce view (1990s - now): + International partners, Online communities, Multinational employees, …


An automatic text mining framework for knowledge discovery on the web

P = Partners/suppliers, E = Employees/Unions, C = Customers,

S = Shareholders/investors, U = Education/research institutions, M=Media/Portals,

G = Public/government, R = Recruiters, V = Reviewers, O = Competitors,

T = Trade associations, F = Financial institutions, I = Political groups,

N = SIG/Communities

Ordered by their relevance to stakeholder types appearing on the Web

*


Stakeholder research and bi

Stakeholder Research and BI

  • Previous research rarely considers the many opportunities offered by the Web for stakeholder analysis, e.g.,

    • Business intelligence, obtained from the business environment, is likely to help in stakeholder analysis

  • Tools and techniques have been developed to exploit business intelligence on the Web

    • PageRank (Brin & Page 1998), HITS (Kleinberg 1999), Web IF (Ingwersen 1998)

      • External links mirror social communication phenomena (e.g., stakeholder relationships)

    • Ong et al. 2001; Tan et al. 2002; Reiterer et al. 2000; Chung et al. 2003; Reid 2003; Byrne 2003

      • Lack stakeholder analysis capability


Existing bi tools and techniques

Existing BI Tools and Techniques

  • Exploit structural and textual content

  • But commercial BI tools lack analysis capability (Fuld et al. 2003)

  • Need to automate stakeholder classification, a primary step in stakeholder analysis

    • Automatic classification of Web pages is a promising way to alleviate the problem


Web page classification

Web Page Classification

  • The process of assigning pages to predefined categories

    • Helps to classify business stakeholders’ Web pages and enables companies to understand the competitive environment better

  • Major approaches: k-nearest neighbor, neural network, Support Vector Machines, and Naïve Bayesian network (Chen & Chau 2004)

  • Previous work

    • Kwon and Lee 2003; Mladenic 1998; Furnkranz 1999; Lee et al. 2002; Glover et al. 2002

    • NN and SVM achieved good performance


Feature selection in web page classification

Feature selection in Web Page Classification

  • Features considered

    • Page textual content: full text, page title, headings

    • Link related textual content: anchor text, extended anchor text, URL strings

    • Page structural information: #words, #page out-links, inbound outlinks (i.e., links that point to its own company), outbound outlinks (i.e., links that point to external Web sites)

  • Methods for selection

    • Human judgment / Use of domain lexicon

    • Feature ratios and thresholding

    • Frequency counting / MI


Research gaps

Research Gaps

  • Stakeholder research provides rich theoretical background but rarely considers the tremendous opportunities offered by the Web for stakeholder analysis

    • Conclusions drawn from old data may not reflect rapid development in e-commerce

  • Existing BI tools lack stakeholder analysis capability

  • Automatic Web page classification techniques are well developed but have not yet been applied to business stakeholder classification


Research questions1

Research Questions

  • How can we apply our automatic text mining approach to business stakeholder analysis on the Web?

  • How can Web page textual content and structural information be used in such an approach?

  • What are the effectiveness (measured by accuracy) and efficiency (measured by time requirement) of such an approach for business stakeholder classification on the Web?


Application of the approach

Application of the Approach

  • Purpose: To automatically identify and classify the stakeholders of businesses on the Web in order to facilitate stakeholder analysis

  • Rationale

    • Business stakeholders’ Web pages should contain identifiable clues that can be used to distinguish their types

    • Web textual and structural content information is important for understanding the clues for stakeholder classification

  • Two generic steps:

    • Creation of a domain lexicon that contains key textual attributes for identifying stakeholders

    • Automatic classification of Web pages (stakeholders) linking to selected companies based on textual and structural content of Web pages


Building a research testbed

Building a Research Testbed

  • Business stakeholders of the KM World top 100 KM companies (McKellar 2003)

  • Used backlink search function of the Google search engine to search for Web pages having hyperlinks pointing to the companies’ Web sites (e.g., “link:www.siebel.com”)

  • For each host company, we considered only the first 100 results returned

    • Removed self links and extra links from same sites

    • After filtering, we obtained 3,713 results in total

    • Randomly selected the results of 9 companies as training examples (414  283 pages stored in DB)


Creation of a domain lexicon

Creation of a Domain Lexicon

  • Manually read through all the Web pages of the nine companies’ business stakeholders to identify one-, two-, and three-word terms that were indicative of business stakeholder types (Thanks to Edna Reid)

  • Extracted a total of 329 terms (67 one-word terms, 84 two-word terms, and 178 three-word terms), e.g.,


Automatic stakeholder classification

Automatic Stakeholder Classification

  • Three steps:

Manual Tagging

Feature selection

Automatic classification


Manual tagging

Manual tagging

Feature selection

Automatic classification

Manual Tagging

  • Manually classified each of the stakeholder pages of the nine selected companies into one of the 11 stakeholder types (based on our literature review) (thanks Edna again)


Feature selection

Manual tagging

Feature selection

Automatic classification

Feature Selection

  • Structural content features: binary variables indicating whether certain lexicon terms are present in the structural content

    • A term could be a one-, two-, or three-word long

    • Considered occurrences in title, extended anchor text, and full text (Lee et al. 2002)

  • Textual content features: frequencies of occurrences of the extracted features (see next slide)

    • The first set of features was selected based on human knowledge, while the second was selected based on statistical aggregation (Glover et al. 2002), thereby combining both kinds of knowledge


Feature selection textual content

Manual tagging

Feature selection

Automatic classification

Feature Selection (Textual Content)


An example a media stakeholder type

An Example(A media stakeholder type)

Link to the host company (ClearForest)

<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

<title>David Schatsky: Search and Discovery in the Post-Cold War Era</title> ...

<p>I just saw a demo by <a href = "http://www.clearforest.com"> ClearForest, </a> a company that provides tools for analyzing unstructured textual information. It's truly amazing, and truly the search tool for the post-Cold War era. ... </p> ...

</body>

</html>

HTML hyperlink and extended anchor text


Automatic classification

Manual tagging

Feature selection

Automatic classification

Automatic Classification

  • A feedforward/backpropagation neural network (Lippman 1987) and SVM (Joachims, 1998) were used due to their robustness in automatic classification

    • Train the algorithms using the stakeholder pages of the 9 training companies and obtain a model or sets of weights for classification

    • Test the algorithms on sets of stakeholder pages of 10 companies different from training examples


Evaluation methodology

Evaluation Methodology

  • Motivation: to know effectiveness and efficiency of the approach

  • Consisted of algorithm comparison, feature comparison, and a user evaluation study

    • Compared the performance of neural network (NN), SVM, baseline method (random classification), human judgment

    • Compared structural content features, textual content features, and a combination of the two sets of features

    • 36 Univ. of Arizona business school students performed manual stakeholder classification and provided comments on the approach


Performance measures

Performance Measures

Effectiveness:

Efficiency: time used (in minutes)

User subjective ratings and comments


User study

User Study

  • Each subject was introduced to stakeholder analysis and was asked to use our system named “Business Stakeholder Analyzer (BSA)” to browse companies’ stakeholder lists

  • We randomly selected three companies (Intelliseek, Siebel, and WebMethods) from testing companies to be the targets of analysis


An automatic text mining framework for knowledge discovery on the web

Business stakeholders of Siebel

Definitions of business stakeholders


Hypotheses 1

Hypotheses (1)

  • H1: NN and SVM would achieve similar effectiveness when the same set of features was used

    • Both techniques were robust

    • Procedure: created 30 sets of stakeholder pages by randomly selecting groups of 5 stakeholder pages of each of the 10 testing companies


Hypotheses 2

Hypotheses (2)

  • H2: NN and SVM would perform better than the baseline method

    • Incorporated human knowledge and machine learning capability into the classification

  • H3: Human judgment in stakeholder classification would achieve effectiveness similar to that of machine learning, but that the former is less efficient

    • They could make use of the Web page’s textual and structural content in classifying stakeholders

    • Humans might spend more time on it


Hypotheses 3

Hypotheses (3)

  • H4 & H5 examined the use of different types of features in automatic stakeholder classification

    • H4: structural = textual

    • H5: combined > structural or textual alone


Experimental results

Experimental Results

Algorithm Comparison

  • H1 not confirmed

  • NN performed significantly differently than SVM when the same set of features was used

    • NN performed significantly better than SVM when structural content features were used

    • SVM performed significantly better than NN when textual content features or a combination of both feature sets were used

    • More studies would be needed to identify optimal feature sets for each algorithm


Effectiveness of the approach

Effectiveness of the Approach

  • H2 confirmed

  • The use of any combination of features and techniques in automatic stakeholder classification outperformed the baseline method significantly

    • Our approach has integrated human knowledge with machine-learned information related to stakeholder types …

    • and was significantly better than a random conjecture


Comparing with human judgment

Comparing with Human Judgment

  • H3b and H3d (efficiency) confirmed

    • Human: 22 minutes (average), varied

    • Algorithms: 1 – 30 seconds (average)

    • Showing high efficiency of using the automatic approach to facilitate stakeholder analysis

  • H3a and H3c (effectiveness) not confirmed

    • Humans were significantly more effective than NN or SVM

    • Could rely on more clues in performing classification

    • Experience in Internet browsing and searching helped narrow down choices


An automatic text mining framework for knowledge discovery on the web

However, the algorithms achieved better within-class accuracies than humans in frequently occurring types …


Use of features

Use of Features

  • To our surprise, hypotheses H4a-b, H5a-b, and H5d were not confirmed

    • Different feature sets yielded different performances of the algorithms

      • Structural features enabled NN to achieve better effectiveness than textual ones

      • Textual and combined features enabled SVM to achieve better effectiveness than structural ones

    • Do not know exactly why

    • Future research: studying the effect of features and the nature of algorithms

  • H5c was confirmed: structural content feature did not add value to the performance of SVM


Subjects comments

Subjects’ Comments

  • Overwhelmingly positive

  • “It would be very helpful!”

  • “That’s cool!”

  • “I want to use it.”


Conclusions limitations and future directions

Conclusions, Limitations and Future Directions


Conclusions

Conclusions

  • General conclusion: our approach helped alleviate information overload and enhance human analysis on the Web

  • Conclusions related to this presentation:

    • Showed how our approach could be applied to business stakeholder analysis on the Web

      • Integrated Human expert knowledge + machine-learned knowledge

      • Promising in terms of effectiveness and efficiency

    • Could potentially facilitate business analysts’ interaction with automated stakeholder analysis systems in today’s networked enterprises


Contributions

Contributions

  • Developing and validating a useful and comprehensive approach to knowledge discovery on the Web

  • New integration and application of techniques together with appropriate human intervention

  • Contributions related to this presentation:

    • Helps BI analysts to understand business stakeholders more efficiently

    • The feature selection approach can be used as a way of knowledge acquisition

    • Extends current stakeholder research by providing a new perspective for automated analysis


Limitations

Limitations

  • Technical limitations (e.g., efficiency)

  • Lab experiment limits external validity

  • Limitations in the presented study:

    • Limited data provided by Google

    • The use of business school students in our study reduces external validity

    • Limitation in identifying stakeholder relationships (only rely on hyperlinks)

    • Limited domain knowledge


An automatic text mining framework for knowledge discovery on the web

Using Web Page Classification for Business Stakeholder Analysis

Building a BI Search Portal

Applying Web Page Visualization to Exploring BI

Problems

Information overload

Unreliable information

Complicated relationships

Contributions

Generic applicability

Enhance knowledge discovery on the Web

Better understanding in HCI


Future directions

Future Directions

  • Related to the presented study:

    • Automate next steps of business stakeholder analysis

      • Type-specific stakeholder analysis

      • Strategic management

    • Cross-regional issues

  • Other domains (e.g., terrorism)

  • New text mining and visualization techniques, and related HCI issues

  • Collaborative commerce topics

    • Integration of the approach with business process logics, collaborative technologies


  • Login