Download
threat connect a visualized cyber threats entity reporting system backed with h adoop ecosystem n.
Skip this Video
Loading SlideShow in 5 Seconds..
Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao PowerPoint Presentation
Download Presentation
Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao

Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao

230 Views Download Presentation
Download Presentation

Scott Miao, Trend Micro s cott_miao@trend.tw @ takeshi.miao

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoopecosystem Scott Miao, Trend Micro scott_miao@trend.com.tw @takeshi.miao

  2. Who am I • RD, SPN, Trend Micro • 3+ years for Hadoop eco system • Expertise in HDFS/MR/HBase • @takeshi.miao

  3. Agenda • Threat intelligence problem • Challenges and Solutions • Summary

  4. “I want to quickly get an overview of the incident, including itsscope, timeline, and impact.” Threat intelligence problem

  5. Threat Connect • A Web Service for Threat Information Report • RESTful Interface to access • Integrated with TM Deep Discovery products • Relevantand ActionableIntelligence

  6. Process and correlates different data sources … IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc. Product 1 Product 2 Product 3 Most relevant threat report with actionable intelligenceon a single portal

  7. Challenges and Solutions

  8. Graph Problem Process & Correlate Moving Big Data Storing Real Time Access Pick Your right tool

  9. Moving

  10. Accumulate small files FBS FBS FBS Event Logs Hadoop Feed Back log Service Dear users/services

  11. Storing

  12. Process & Correlate

  13. Time • Batch • Performance • Store • Pig/MR • HDFS • Hbase • Solr • RDB • UDFs • MRs for special cases

  14. Real Time Access

  15. Free form search • Solr Cloud • Real Time Access • EX. Sandbox Reports • Random Access • HBase • EX. Threat Detection DBs

  16. Graph Model

  17. Active community ? Massive scalable ? Analyzable ?

  18. We use HBase as a Graph Storage • Google BigTableand PageRank • HBaseCon2012

  19. HGraph https://github.com/tinkerpop/blueprints/wiki

  20. Pick right tool

  21. Pick right tool for right usecases • Silver bullet ? • No one project fits all • One problem may has several choices http://www.neevtech.com/blog/2013/03/18/hadoop-ecosystem-at-a-glance/

  22. Summary

  23. Small files • Namenodefsimage would explore the memory • Too many map tasks to run for a job FBS FBS FBS

  24. Store your data anyway • Store all the raw data on the HDFS • Break invisible isolation from different data sources • Archive your data with deduced easy to use FileFormat • Trenvi, RC file, ORC file

  25. Know MR more • Even you are the pig developer • Deal with MR issues • Write better pig-latin • Sometimes you can only use MR

  26. Know your data & usecases • Realtime ? Batch ? • Access Pattern ? • Therefore, you can pick right tool

  27. Thank you guys