relational evaluation techniques l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Relational Evaluation Techniques PowerPoint Presentation
Download Presentation
Relational Evaluation Techniques

Loading in 2 Seconds...

play fullscreen
1 / 30

Relational Evaluation Techniques - PowerPoint PPT Presentation


  • 144 Views
  • Uploaded on

Relational Evaluation Techniques. Daniel McEnnis. Outline. Definition Component Overview Existing Approaches Descriptions of the Components Applications and Examples. Relational Evaluation Techniques Definition.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Relational Evaluation Techniques' - quasar


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outline
Outline
  • Definition
  • Component Overview
  • Existing Approaches
  • Descriptions of the Components
  • Applications and Examples
relational evaluation techniques definition
Relational Evaluation Techniques Definition
  • Experimental setup for evaluating the performance of algorithms that use data that span more than one table or instance vector
  • Can use either relational algebra or hypergraph-based descriptions
components
Components
  • Data Acquisition
  • Ground Truth Acquisition
  • Cross-Validation Technique
  • Query Type
  • Scoring Metric
  • Significance Test
existing approaches
Existing Approaches
  • Machine Learning
  • Relational Machine Learning
  • TREC
  • Collaborative Filtering
  • ISMIR
  • Social Network Analysis
machine learning
Machine Learning
  • Predetermined flat data, no sampling
  • Predetermined ground truth
  • Typically simple queries
  • Sophisticated cross-validation
  • Basic set based metrics
  • No significance tests
relational machine learning
Relational Machine Learning
  • Predetermined relational data
  • Predetermined ground truth
  • Predefined simple query
  • Sophisticated cross-validation
  • Basic set-based metrics
  • No significance tests
slide8
TREC
  • Predetermined flat data
  • Sophisticated ground truth sampling.
  • Sophisticated queries
  • Machine-learning cross-validation
  • Ranked set-of-sets scoring
  • Simple significance tests
collaborative filtering
Collaborative Filtering
  • Predetermined flat/relational data
  • Predetermined ground truth
  • Simple, predefined query
  • No cross-validation
  • Sophisticated Scoring metrics
  • No significance tests
ismir
ISMIR
  • Sampled flat data
  • Predetermined ground truth
  • Sophisticated queries
  • Machine-learning cross validation
  • Simple set based scoring metrics
  • Sophisticated significance tests
social network analysis
Social Network Analysis
  • Sophisticated data sampling
  • Sophisticated statistical techniques
sequences of choices
Sequences of Choices
  • Plug ‘n play an experiment
  • Different aspects are evaluated
  • Some algorithms simply don’t work
  • Extensive algorithm rewrites sometimes needed
data acquisition
Data Acquisition
  • Data structure
  • Where is it?
  • What sampling technique to use
    • Random Access
    • Snowball
    • Hypergraph Snowball
  • How much data is needed?
ground truth acquisition
Ground Truth Acquisition
  • What is being tested?
  • TREC extended ground truth sampling
  • Structure of the output
cross validation
Cross-Validation
  • Actor Based
  • Link Based
  • Graph Based
  • No Cross Validation
graph notation
Graph Notation
  • Actor definition
  • Link definition
  • Graph definition
  • Database table / instance vector equivalence
  • Foreign key / link equivelance
actor cross validation
Actor Cross-Validation
  • Traditional Machine Learning approach
  • Divisions by database table
  • Folds usually random assignment
  • Works well on flat data
  • Trouble with relational data
link cross validation
Link Cross Validation
  • Rare machine learning approach
  • Divisions by foreign key reference
  • Less statistical independence than actor
  • Works for collaborative filtering
  • Usually random assignment
graph cross validation
Graph Cross Validation
  • Relational Machine Learning
  • Divisions by predetermined discrete graphs
  • Statistical independence
  • Non-learning based approaches
  • Clustering based fold generation
no cross validation
No Cross Validation
  • Standard over fitting problems
  • Useful after implied cross-validation
query type
Query Type
  • Information Need definition
  • Actor based query
  • Set or List based query
  • Conditional queries
scoring metrics
Scoring Metrics
  • Comparisons against ground truth
  • Set based metrics
  • Ranked based metrics
  • List based metrics
set based metrics
Set Based Metrics
  • Recall and Precision
  • F-Measure
  • Mean Average Performance
ranked list metrics
Ranked List Metrics
  • Pearson Correlation
  • Spearmans Correlation
  • Mean Absolute Error
  • Linear Algebra Distance Metrics
  • Serendipity
ordered list metrics
Ordered List Metrics
  • Half Life
  • Kendall Tau
  • NDPM
  • Sequence Alignment Algorithms
  • Hamming Distance
significance tests
Significance Tests
  • Pairwise student t-test
  • ANOVA
  • ANOVA/Tukey-Kramer statistical test
evaluation questions
Evaluation Questions
  • Does the data contain time (global ordered sequence)
  • Actor-, Link-, Graph-, or Set-based queries
  • List, Set, or Set-of-Lists output
  • Contextual question or absolute
  • Statistical purity versus maximum information
music recommendation
Music Recommendation
  • Example - Personalized Dynamic Tag Radio
  • LastFM profile data
  • LastFM tag data
  • Semantic Web data
  • Next-week-data ground truth
  • Conditional query
  • Graph cross-validation
  • Kendall Tau scoring metric
  • ANOVA/Tukey-Kramer statistical analysis
conclusions
Conclusions
  • No one-size-fits-all
  • Data and ground-truth set the framework
  • Question determines the final structure
  • Each discipline has a piece of the answer
  • Graph-RAT 0.5
future work
Future Work
  • Finish exploring Social Network Analysis significance tests
  • Fully explore set-of-sets evaluation metrics
  • Debugging of Graph-RAT cross-validation schedulers
  • Ease of use improvements to Graph-RAT