1 / 9

Evaluating the Effectiveness of Rank Topology vs. Traditional FLT Techniques

In the quest to improve fault localization techniques (FLT), this study compares rank topology against random ranked lists. By examining how developers interact with ranked items, we implement a proposed approach that incorporates evaluation measures focused on the likelihood of finding bug fix locations. Utilizing textual information and structural hops, we establish a rank topology metric to gauge developer effectiveness. Our preliminary findings indicate that rank topology outperforms traditional methods in identifying relevant results, emphasizing the need for a reassessment of FLT evaluation metrics.

justis
Download Presentation

Evaluating the Effectiveness of Rank Topology vs. Traditional FLT Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Which Feature Location Technique is Better? Emily Hill, Alberto Bacchelli, Dave Binkley, BogdanDit, Dawn Lawrie, Rocco Oliveto

  2. Motivation: Differentiating FLTs Precision = 0.20 Precision = 0.20 Totally unrelated In vicinity

  3. Example • Developer works down ranked list • At each item can explore or not • When exploring structure, can bail at any time

  4. Proposed Approach: Rank Topology • Use evaluation measures that consider the likelihood of a developer finding fix locations • Use textual information to approximate developer’s interest (i.e., likelihood) of following “trail” in structural topology, starting from ranked list • Rank topology = inverse of the number of hops in topology

  5. Example • 3rd rank result + 4 structural hops = 7 total hops • Rank topology metric = 1 / 7 • Developer works down ranked list • At each item can explore or not

  6. How “smart” is the user? • Omniscient: makes no wrong choices, exploring only those ranks and structural hops that lead to a bug • No discrimination: explores everything • Semi-intelligent: only follows a structural hop if the next method exhibits textual clues • Rank topology uses VSM cosine similarity (tf-idf) • Structural edge added if both methods > median scores for query • Supported by user studies of information foraging theory [Lawrance, et al TSE 2013]

  7. Preliminary Study: Distinguish QLM from Random Ranked list of results all have same bug fixes at exactly the same ranks

  8. Conclusion • Rank topology differentiates between randomly ordered lists and a state of the art IR technique (QLM) with relevant results at the exact same ranks • Future work • How well does rank topology mimic developer behavior in practice? • How closely can/should we model user behavior? • Our question: Does the research community need to revise how we evaluate FLTs?

  9. Preliminary Study • Effect of program structure on the rank topology metric for each JabRef bug used in the case study.

More Related