1 / 83

Agentless Cloud-wide Monitoring of Virtual Disk State

Agentless Cloud-wide Monitoring of Virtual Disk State. Wolfgang Richter wolf@cs.cmu.edu. Cloud Customers. Monitoring Services. Cloud Providers. [frost2013]. What is an agent ?. An agent is a process performing administrative tasks that generally runs in the background.

loe
Download Presentation

Agentless Cloud-wide Monitoring of Virtual Disk State

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Agentless Cloud-wide Monitoring of Virtual Disk State Wolfgang Richter wolf@cs.cmu.edu

  2. Cloud Customers Monitoring Services Cloud Providers [frost2013] Thesis Proposal – Wolfgang Richter

  3. What is an agent? An agent is a process performing administrative tasks that generally runs in the background. Loggly– log collection and analytics ClamAV – virus scanning Dropbox – file backup and sync Windows Update – OS / system update Tripwire – file-based intrusion detection Thesis Proposal – Wolfgang Richter

  4. What is a monitoring agent? A monitoring agent is a process performing administrative tasks that generally runs in the background and can not modify state. Loggly– log collection and analytics ClamAV – virus scanning Dropbox – file backup and sync Windows Update – OS / system update Tripwire – file-based intrusion detection Thesis Proposal – Wolfgang Richter

  5. What is meant by cloud? Users manage isolated VM(s) VM Customers Real Server Infrastructure Small local virtual HD Thesis Proposal – Wolfgang Richter

  6. Best Practice Monitoring Today • Agents run inside the monitored system • Per-OS type • Per-Application type • Per-System configuration • Per-System update + patch • Sometimes globally aware [kufel2013] Thesis Proposal – Wolfgang Richter

  7. Reimagining Monitoring General OS and application agnostic Independent Misconfiguration and Compromise Scalable Globally aware Thesis Proposal – Wolfgang Richter

  8. Independent Monitoring Resources Thesis Proposal – Wolfgang Richter

  9. Leverage Global Knowledge [wei2009] Thesis Proposal – Wolfgang Richter

  10. ✗Not General ✗Not Independent Agents VM Customers Real Server Infrastructure VMM Observable ✓ Generalizable ✓Independent Agentless [garfinkel2003] Thesis Proposal – Wolfgang Richter

  11. Agentless cloud-wide disk state monitoring is feasible, efficient and scalable. Thesis Proposal – Wolfgang Richter

  12. Outline • Challenges • Completed • Distributed Streaming Virtual Machine Introspection • /cloud • cloud-inotify • Retrospection • Remaining • File-level deduplication • Applications + scalability measurements • /cloud-history • Timeline and Summary Thesis Proposal – Wolfgang Richter

  13. The Semantic Gap Semantic Gap Interpret on-disk layout 1 0 0 1 0 1 0 1 1 0 Thesis Proposal – Wolfgang Richter

  14. Temporal Gap Crashing would cause data corruption t2 t1 t0 Data Block Data Block File Size Increase File Size Increase Data Block Data Block Must buffer until metadata update Thesis Proposal – Wolfgang Richter

  15. Generality NTFS File-system-specific backend Windows Parser ext4 Linux Normalized Parser Monitors zfs Parser Solaris Thesis Proposal – Wolfgang Richter

  16. Scalability • Support 10,000+ monitored systems • Overall latency ~10 minutes • Reasonable network bandwidth overhead • Maximize monitored VMs per host • Minimize decrease in consolidation [cohen2010] Thesis Proposal – Wolfgang Richter

  17. Bounded Overhead • Latency-completeness-performance tradeoff • Capturing every write is costly • Too much buffering hurts latency • Work within queue size budget • Must tolerate loss of writes • Extreme: detaching and re-attaching Thesis Proposal – Wolfgang Richter

  18. Select Related Work Thesis Proposal – Wolfgang Richter

  19. Outline • Challenges • Completed • Distributed Streaming Virtual Machine Introspection • /cloud • cloud-inotify • Retrospection • Remaining • File-level deduplication • Applications + scalability measurements • /cloud-history • Summary and Timeline Thesis Proposal – Wolfgang Richter

  20. /cloud cloud-inotify Retrospection Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  21. /cloud cloud-inotify Retrospection Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  22. DS-VMI Redis VMM Async.Queuer n Scale out + coord. Minimize IO Impact File system specific parsing Inference Engine Thesis Proposal – Wolfgang Richter

  23. Tapping the Disk Write Stream VM ~50 line patch drive-backup (nbd) QEMU Thesis Proposal – Wolfgang Richter

  24. Bootstrapping NTFS Normalized BSON Disk Crawler ext4 Normalized BSON Disk Crawler Thesis Proposal – Wolfgang Richter

  25. DS-VMI Overhead on Running VM Relative Overhead [richter2014] Thesis Proposal – Wolfgang Richter

  26. 5.2x Relative Overhead Thesis Proposal – Wolfgang Richter

  27. /cloud cloud-inotify Retrospection Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  28. /cloud Eventual consistency Legacy FS interface Batch-based Legacy/batch-based apps: /cloud/host/vm/path On all hosts check permissionsof /libinside every VM instance. find /cloud/*/*/lib \ -maxdepth 0 \ -not \ -perm 755 Thesis Proposal – Wolfgang Richter

  29. /cloudArchitecture Redis FUSE Driver Apps n Virtual Disk Inference Engine Thesis Proposal – Wolfgang Richter

  30. Latency – Guest Syncs Every Second [richter2014] Thesis Proposal – Wolfgang Richter

  31. /cloud cloud-inotify Retrospection Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  32. cloud-inotify Strong consistency Publish-subscribe Event-driven Subscription format: <host>:<VM>:<path> gs9671:bg1:/var/log/* On host gs9671 monitor all files under file system subtree /var/log/ in all VM’s in group bg1. Thesis Proposal – Wolfgang Richter

  33. cloud-inotifyArchitecture Loggly * : * : /var/log/* Redis Subscriber <host2>:<vm2>:<path2> n Publish <host>:<vm>:<path> Inference Engine Thesis Proposal – Wolfgang Richter

  34. /cloud cloud-inotify Retrospection Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  35. Introspection vs. Retrospection Examine active state of VM during execution Examine historical state of VMs and their snapshots VM Instance A A' A1 A2 B' B1 B2 ... Examine live logs Examine all historic logs A* [richter2011] Thesis Proposal – Wolfgang Richter

  36. File-level Deduplication [satya2010] Thesis Proposal – Wolfgang Richter

  37. Outline • Challenges • Completed • Distributed Streaming Virtual Machine Introspection • /cloud • cloud-inotify • Retrospection • Remaining • File-level deduplication • Applications + scalability measurements • /cloud-history • Summary and Timeline Thesis Proposal – Wolfgang Richter

  38. Applications stressing end-to-end performance and scalability /cloud cloud-inotify /cloud-history File-level deduplication Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  39. Applications stressing end-to-end performance and scalability /cloud cloud-inotify /cloud-history File-level deduplication Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  40. Why? find /cloud/*/*/lib \ -maxdepth 0 \ -not \ -perm 755 [satya2010] Thesis Proposal – Wolfgang Richter

  41. Desired Hash Properties Quick to re-compute for random writes DS-VMI works with a stream of writes No extra bytes from disk required Can’t rely on virtual disk Collision Resistant For correctness Compact Network synchronization Thesis Proposal – Wolfgang Richter

  42. Traditional Hashing? Supportsrapid recomputation of whole-file hashfor append-only operations Normal C API (SHA-3, NIST): [nist] Thesis Proposal – Wolfgang Richter

  43. Merkle-Damgård [wikipedia, damgård1990] Thesis Proposal – Wolfgang Richter

  44. Incremental Hashing Incremental Efficient random updates Collision-free Cryptographically secure Parallelizable Faster than sequential [bellare1997] Thesis Proposal – Wolfgang Richter

  45. Thesis Proposal – Wolfgang Richter

  46. Thesis Proposal – Wolfgang Richter

  47. Applications stressing end-to-end performance and scalability /cloud cloud-inotify /cloud-history File-level deduplication Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

  48. Applications /cloud Virus Scanning (ClamAV) Log Collection (Splunk) cloud-inotify Continuous Compliance Monitoring /cloud-history File Recovery Unindexed Search Thesis Proposal – Wolfgang Richter

  49. Planned Measurements • Latency-completeness-overhead • Vary queue sizes and flush parameters • Analyze metadata vs data • Re-attachment time • In-VM performance vs Agentless • Scalability in number of monitored systems • Number of monitored systems per host • Wikibench Thesis Proposal – Wolfgang Richter

  50. Applications stressing end-to-end performance and scalability /cloud cloud-inotify /cloud-history File-level deduplication Distributed Streaming Virtual Machine Introspection (DS-VMI) Thesis Proposal – Wolfgang Richter

More Related