Atsipra šau...

2. Atsiprašau... ... bet ši skaidrė bus vienintelė skaidrė lietuvių kalba

3. When things break It’s not big deal! Resilience and Remediation

4. Agenda • About me • Intro to resilience • Credits to authors • Let it crash! • Distributed resilience • Remediation patterns • QA

5. About me • BS & MS in Program engineering KTU • Developer for 10+ years… • …and team-lead, tech-lead, consultant, agile pioneer, certified scrum master, architect • Backend, distributed systems developer for last 4+ years

7. hard.core team in BI department • Data warehouse and business intelligence • Tens of TB of information • 500 million transactions per day • Various availability requirements • Always online, always consistent, always up to date

8. Reliability • a.k.a. availability • Most critical component has two nines (.99) • Gmail had 0.99984 @ 2010

9. QCon London 2011 • My goal • Bring new ideas • Leave comfort zone • Evolve • Visit it! • InfoQ.com • Credits to authors

10. Things break, live with it

11. Let it crash! Why should we?

12. Let it crash! Erlang is a programming language used to build massively scalable real-time systems with requirements on high availability

13. Let it Crash! Idea behind • The world is concurrent • Things in the world don’t share data • Things communicate with messages • Things fail

14. Let it crash! Erlang • Runtime system with functional language • Simple threading • Message passing (no locks!) • CouchDB, Riak, Facebook chat (ejabberd), RabbitMQ, GitHub • We don’t use it :)

15. Let it Crash! Defensive programming • How to do it - write code to… • solve a problem • check all possible inputs • handle all possible errors/exceptions public void DoStuff(string message, Action<string> action) { if (String.IsNullOrEmpty(message)) throw new ArgumentNullException("message", "The message is null or empty."); if (action == null) throw new ArgumentNullException("action", "The action is null."); try { action(message); } catch (BoringMessageException) { return; } catch (OffensiveMessageException exc) { this.ReportOffender(SystemState.CurrentUser); return; } catch (ActionMonkeyIsBusyException exc) { throw new RetryLaterException(exc, DateTime.Now.AddYears(1), action, message); } catch { throw; } } public void DoStuff(string message, Action<string> action) { if (String.IsNullOrEmpty(message)) throw new ArgumentNullException("message", "The message is null or empty."); if (action == null) throw new ArgumentNullException("action", "The action is null."); action(message); } public void DoStuff(string message, Action<string> action ) { action(message); } public void DoStuff(string message,Action<string> action ) { action(message); }

16. Let it Crash! Defensive programming • How to do it - write code to… • solve a problem • check all possible inputs • handle all possible errors/exceptions • Result • More code, more bugs • Obscure code, unclear logic • Error handling is poorly tested • It is very hard to defend against everything

17. Let it Crash! Erlang way of managing applications

18. Let it Crash! According to Joe Armstrong • Do not code defensively • If you can’t do what you want, die! • Nobody should stop you from crashing • Let other process do recovery But be reasonable…

19. Story time! Legacycache

20. Let it crash! Couple of hints

21. Let it Crash! My takeaways • “Let it crash” is not • a design crutch • an excuse to lose vital data • Principle of Least Surprise • Handle what you can • Let someone else do the rest

22. Distributed Systems, Databases and Resilience … or just Distributed Resilience

23. Distributed Resilience Plan your failures • You can usually prevent full system crash • But how will it behave on partial failure? • Plan and understand… • …before the users tell you • You think you know what will break… • … you’re probably wrong

24. Distributed Resilience Failure is not bad • Best way to avoid failures is to fail constantly! • Netflix “Chaos Monkey” • Navigates in infrastructure • Kills random processes • Monitors how system recovers

25. Distributed Resilience Harvest and yield • Harvest is your data • Yield is possibility to retrieve data

26. Distributed Resilience It’s just a flesh wound! Problems • Finding single point of failures • Security • Configuration • Administration, Shared, Central • Removing • Hard to avoid, even harder to remove • Minimization is the target

27. Distributed Resilience Share nothing - sharding • Pros • Index sizes • Security (OotB)? • Leverages HW risk • Cons • More HW • Harder than looks • Auto is even harder • Shared info • Complex releases Proxy Config

28. Distributed resilience Load balancing • Pros • Rather easy • Lots of products • Cons • Session is killer • Configuration Service Service Service Service

29. Distributed Resilience Mirrors & Replication • Pros – Various! • Lots of products • Very secure? • Very fast? • Cons – Various! • Hardware • Slow? • Inconsistent?

30. Distributed resilience NoSQL CouchDB Hbase • Not (always) ACID • Atomicity • Consistency • Isolation • Durability • BASE • Basically Available • Soft state • Eventual consistency • CAP theorem • a.k.a. Brewer's theorem • Consistency • Availability • Partition tolerance Memcached Cassandra Redis MarkLogic BigTable riak mongoDB SimpleDB

31. Next slide :) Sample time!

32. Samples from adform • Sensitive statistics • Real time bidding Google User sees Node2 Node1 Bid Service1 Bid Service2

33. Remediation patterns For better releases

34. Remediation patterns Vocabulary • Remediation • Recovery to known state after a failed release • Recovery • Returning system to working state • “Fixing sh*t when it breaks” • It’s all about… • Prevention • Patterns of low risk release • Patterns of incremental delivery Yea!

35. Remediation patterns Background • Release is risky operation • The best way to fail release is to do it once • Don’t touch it if it works! • Agile • Time to market • Continuous deployment • 20 releases in 2 weeks @ adform

36. Remediation patterns Prevention

37. Remediation patterns Prevention

38. Remediation patterns Problems • The hard bits: • Testing on production environment • Create maintainable acceptance tests • Testing cross-functional requirements

39. Remediation patterns Reducing risk • Canary releasing • Partial release • Observe effects • Release for the rest

40. Remediation patterns Reducing risk • Dark launch • Release to invisible infrastructure • Direct some of real load to dark-side

41. Remediation patterns Reducing risk • Feature toggles • Develop on trunk, or else… • Feature toggle / branch by abstraction

42. Monitoring Separate slide, cause it is so damn important! • Monitoring is essential • Remote watchdogs • Watchdogs for watchdogs • Business metrics are essential • Root cause analysis • The game: why? why? why? • Root cause graph

43. Next slide :) Sample time!

44. No tests in front Testing identified 2 fires Ownership problems Delayed checklist No review on checklists Different knowledge levels Only one person writing checklist Developers don’t use it

45. Final word Troll appears Things break Learn to bend (one way or another)

46. Almost done Shameless ads

47. We are recruiting! • 100+ in Lithuania • 60+ in development • Architects • Analytics • Programmers • QA • http://www.adform.com/site/company/careers/