1 / 9

Anomaly Detection in the Elasticsearch Service

Anomaly Detection in the Elasticsearch Service. Lightning Talk. 15/08/2019. Jennifer Andersson. Introduction. Elasticsearch Service:. Project goal:. Search- & analytics engine 30 clusters & 160 use cases. Detect service issues before they cause problems.

lesparza
Download Presentation

Anomaly Detection in the Elasticsearch Service

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Anomaly Detection in the Elasticsearch Service Lightning Talk 15/08/2019 Jennifer Andersson

  2. Introduction Elasticsearch Service: Project goal: • Search- & analytics engine • 30 clusters & 160 use cases Detect service issues before they cause problems

  3. Anomaly Detection & Degradation Prediction Deep autoencoder Classifier

  4. Project Challenges • Better preprocessing of the input data required • Evolution of cluster characteristics over time requires frequent retraining • Convergence issues makes retraining hard (vanishing gradient problem) Apache 300 return codes Time [two years]

  5. Data Preprocessing Raw data Preprocessed data

  6. Long Short-Term Memory Neural Networks ht+1 ht-1 ht Xt+1 Xt Xt-1 x x x + ᶴ ᶴ ᶴ ᶴ ᶴ Colah’s blog post: Understanding LSTM Networks

  7. Long Short-Term Memory Neural Networks ht+1 ht-1 ht Xt Xt+1 Xt-1 x x x + ᶴ ᶴ ᶴ ᶴ ᶴ Colah’s blog post: Understanding LSTM Networks

  8. Model Evaluation • Ongoing work: • Compare to • non-ML methods, e.g. moving average • other ML-methods , e.g. (Extended) isolation forests • Index anomaly scores to Elasticsearch and create a dashboard • Future work: • Classification of anomalous events • Apply the method to other services • Achievement: • Much better convergence than the previous model

  9. Thank you for listening Contact me: Jennifer Andersson jennifer.r.andersson@gmail.com

More Related