Managing Large RDF Graphs (Infinite Graph) Vaibhav Khadilkar Department of Computer Science,

1. Managing Large RDF Graphs (Infinite Graph) Vaibhav Khadilkar Department of Computer Science, The University of Texas at Dallas FEARLESSengineering

2. Managing Large RDF Graphs Agenda Motivation behind the project Semantic web technologies overview Proposed architecture Performance metrics FEARLESSengineering

3. Managing Large RDF Graphs Motivation - Current Problems Jena’s in-memory model does not scale Jena’s RDB and SDB models cannot handle large result sets Hinders ability to do reasoning and large graph processing Current work focuses on load balancing and fault tolerance Current systems can be broken with even 100,000 triples We work on load balancing and polynomial reasoning but memory management breaks systems before any other problems can be addressed FEARLESSengineering

4. Managing Large RDF Graphs Motivation - Relevance of the problem This is an unsolved problem Critical in handling terabytes of data relevant in today’s times Move the problem from memory space to disk space FEARLESSengineering

5. Extension Managing Large RDF Graphs In-memory RDB SDB ARQ Reasoning Jena FEARLESSengineering

6. Managing Large RDF Graphs Semantic web technologies overview - Jena Jena is a Java based framework that allows building Semantic web applications Jena provides a programmatic environment for RDF, RDFS, OWL, SPARQL and includes a rule based inference engine Jena allows the creation and manipulation of in-memory or relational database backed (RDB and SDB) RDF graphs FEARLESSengineering

7. Managing Large RDF Graphs Semantic web technologies overview - Lucene Lucene is a Java based text indexing and searching tool The smallest unit of text that Lucene indexes and searches is a Document A Document contains different fields and a corresponding value for each field The different fields are the indexes that can be used as keywords during a search FEARLESSengineering

8. Managing Large RDF Graphs Problems with In-memory Jena Model Ability to handle medium sized graphs As nodes are added memory fills up As more nodes are added, the program crashes with an out of memory exception We want to solve this out of memory problem FEARLESSengineering

9. Managing Large RDF Graphs 5. Continue adding triples 1. Add triples In-memory triple store + buffer 2. Added triples = Threshold Buffer Management Strategy 3. Buffer sorted based on memory management algorithm 4. Write triples based on sorted buffer while triples left > x of Threshold Lucene triple store FEARLESSengineering

10. Managing Large RDF Graphs 1. Query model 4. Return result In-memory triple store 2. If result not in memory query Lucene triple store 3. Return result Lucene triple store FEARLESSengineering

11. Managing Large RDF Graphs Choice of Algorithm Memory management algorithms such as LRU, MRU, FIFO, and LIFO Social network analysis measures such as degree centrality and individual clustering coefficient Combination of memory management algorithm with degree centrality and individual clustering coefficient FEARLESSengineering

12. Managing Large RDF Graphs FEARLESSengineering

13. Managing Large RDF Graphs Choice of buffer and persistence strategy Buffer can be created based on the subject, predicate, object or a combination of them Map Jena’s subject, predicate and object indexes to Lucene indexes directly Create Lucene indexes as needed taking into account the nature of SPARQL queries and Jena’s implementation FEARLESSengineering

14. Managing Large RDF Graphs FEARLESSengineering

15. Managing Large RDF Graphs Conclusions from the in-memory model Degree centrality is the best algorithm to choose a node to be persisted to disk Creating Lucene indexes as needed is a better choice for the persistence strategy than creating all indexes at the same time FEARLESSengineering

16. Managing Large RDF Graphs Problems with RDB Jena model The RDB Jena model can add any number of triples to the relational database When a query asking for a large number of triples is executed, the result set returned fills up memory causing the program to crash with an out of memory exception We want to solve this out of memory problem We leverage the previous in-memory extension to solve this problem FEARLESSengineering

17. Managing Large RDF Graphs Memory management algorithm Algorithm We use the LIMIT and OFFSET clauses in SQL to get only a part of the results at a time The retrieved triples are added to the extended in-memory Jena model Thus we use the memory management algorithm from the in-memory model Since the revised in-memory model never runs out of memory this RDB solution never runs out of memory FEARLESSengineering

18. Managing Large RDF Graphs Conclusions Conclusions from the extended RDB model Model creation times are similar to the original RDB Jena model Query times vary based on the threshold value in the in-memory solution General conclusions Implemented an in-memory cache based memory management algorithm Solves the memory problem for the in-memory and RDB Jena models by creating an impression of infinite memory for the user Moves the memory problem to disk space FEARLESSengineering

19. Managing Large RDF Graphs Problems with SDB Jena Model The SDB Jena model can add any number of triples to the relational database When a query asking for a large number of triples is executed, the result set returned fills up memory causing the program to crash with an out of memory exception We want to solve this out of memory problem The SDB solution does not depend on the in-memory or RDB extensions FEARLESSengineering

20. Managing Large RDF Graphs Memory management algorithm Algorithm We use the LIMIT and OFFSET clauses in SQL to get only a part of the results at a time The retrieved triples are returned as a separate iterator to the executing program FEARLESSengineering

21. Managing Large RDF Graphs Inferencing in Semantic Web Ontology specification - TBox Instance creation - ABox Inference - Generating new triples based on instances in the Abox backed by the TBox FEARLESSengineering

22. Managing Large RDF Graphs Problems in inferencing with this extension How do you do reasoning when the graph is divided between memory and disk ?? Scalability FEARLESSengineering

23. Continue adding triples Managing Large RDF Graphs Add triples Triple store Is triple a part of TBox?? No Yes 1. Added triples = Threshold In-memory triple store In-memory triple store + buffer Buffer Management Strategy 2. Buffer sorted based on memory management algorithm 3. Write triples based on sorted buffer while triples left > x of Threshold Lucene triple store FEARLESSengineering

24. Managing Large RDF Graphs 1. Query 7. Return result In-memory triple store 2. Get TBox triples Pellet Reasoner 6. Return result 3. Query for ABox triples In-memory triple store 4. If result not in memory query Lucene triple store 5. Return result Lucene triple store FEARLESSengineering

25. Managing Large RDF Graphs Choice of Algorithm Memory management algorithms such as LRU, MRU, FIFO, and LIFO Social network analysis measures such as degree centrality and individual clustering coefficient Combination of memory management algorithm with degree centrality and individual clustering coefficient FEARLESSengineering

26. Managing Large RDF Graphs FEARLESSengineering

27. Managing Large RDF Graphs Choice of buffer and persistence strategy Buffer can be created based on the subject, predicate, object or a combination of them Map Jena’s subject, predicate and object indexes to Lucene indexes directly Create Lucene indexes as needed taking into account the nature of SPARQL queries and Jena’s implementation FEARLESSengineering

28. Managing Large RDF Graphs FEARLESSengineering

29. Managing Large RDF Graphs Conclusions from the inference model RANDOM is the best algorithm to choose a node to be persisted to disk Creating all Lucene indexes at the same time is a better choice for the persistence strategy than creating the indexes one at a time FEARLESSengineering

30. Managing Large RDF Graphs Future Work Test all models with benchmark data Generalize the algorithm to be able to handle multiple incarnations of nodes over time Improve the efficiency of all algorithms Try other algorithms for selecting the candidate node to be written to disk FEARLESSengineering