1 / 15

HBase Tracing

HBase Tracing. Elliott Clark. Who Am I. Elliott Clark eclark@apache.org HBase PMC HBase Committer HTrace Mantainer Cloudera Engineer CDH Distribution. What’s Going on During Failovers. Motivation. Want to improve MTTR First need to know what’s going on Google’s Dapper. Options.

alice
Download Presentation

HBase Tracing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HBase Tracing Elliott Clark

  2. Who Am I • Elliott Clark • eclark@apache.org • HBase PMC • HBase Committer • HTrace Mantainer • Cloudera Engineer • CDH Distribution

  3. What’s Going on During Failovers

  4. Motivation • Want to improve MTTR • First need to know what’s going on • Google’s Dapper

  5. Options • Log all the things • We already do that. • Our logs are way too verbose as it is. • Hard to combine multiple logs • HRegionServer • HMaster • Client • Guess • Zipkin • Not Invented Here

  6. Spoiler: • Guessing Didn’t Win

  7. About Zipkin • Created by Twitter • Thanks • Modeled after Dapper • Scala • Finagle • Functional • Netty • Non-blocking • Thrift • Cool UI • Pluggable Storage Backend

  8. Dapper’s Terminology • Span • Segment of a remote call • Contains Annotations • Has a parent trace • Can have multiple children spans • Can have multiple annotations • Annotations • Trace • Grouping of spans

  9. Can’t use Cassandra and look at myself in the mirror

  10. Implementation Hooking up Zipkin, HTrace, and HBase

  11. Zipkin HBase Storage • Replaces need for Cassandra • Functional • Created a non-blocking client shim • Fixed Key Length • ID’s are long • Mapping table • ID -> Name • Name -> ID • Inspiration from OpenTSDB • Open source on Github

  12. HTrace • Way to instrument Java Code • Used In HBase trunk • Coming to an install near you in 0.96.0 • Pluggable where it emits Spans • To a file • To sys.out

  13. HTrace and Zipkin • Modularize HTrace • Create htrace-zipkin • Create ZipkinSpanReceiver • Receives spans from HTrace instrumented code • Converts to Thrift objects • Sends to Zipkin • Background Flush • Batching

  14. Demo Let’s hope this works.

More Related