Key Technical Challenges Faced by Faz.net from 2000 to 2009

1. Faz.net technical challenges 2000 - 2009

2. what i won’t show • business numbers, € • business plans

3. what to do (from the point of view of lowly programmer) • invent a business plan • define, plan, and implement desired functionality • build a robust, secure, scalable system • keep that system up 100% of the time • allow changes to support new business initiatives

4. a brief history • launched 8 Jan 2001 • “stable” by Spring of 2001 • unstable on 11 Sep 2001 • better able to handle (many classes of) huge load increases by Spring 2002

6. how big is faz.net? • much smaller than google  • smaller than spiegel

8. how big is faz.net? • much smaller than google  • smaller than spiegel • comparable to other newspapers

13. a sketch of our system layout (a couple years old) • load balancer • web servers • DB and file servers • other application servers • client machines, e.g. newsroom • external partners, e.g. freemail provider

16. an ancient attempt to show how changing an article affects site • editor publishes a new article (or a new version of an existing article) • a DB trigger fires • cached DB result sets get updated • cached HTML gets updated or deleted • (we do things differently now)

19. how to measure what is happening / whether things are ok • measure throughput at / between various points

20. (partial) network traffic over time • MRTG (http://oss.oetiker.ch/mrtg/) • Standard, free (Gnu) • Useful at a glance info

21. how to measure what is happening / whether things are ok • measure throughput at / between various points • measure, e.g. cpu load on web servers (NB: load is pretty low, max would be 12*100)

25. long term trend: one year

27. Some errors can be seen by any user with a browser who runs into them • Server error • Wrong contents • Broken HTML, images, or …

28. some page ailments complain helpfully

31. digested log files can be helpful • web sites tend to have many log files, several kinds of log files; and some kinds can be *huge* • they tend to be straight ascii files in some format you might have little control over • various kinds of statistics might sometimes interest you • in particular error statistics

32. ad hoc (logfiles | awk ... excel

33. ad hoc: mrtg + paint o.ä.

34. ad hoc

35. ad hoc: DB query -> Excel

36. external, hired monitoring service • hire someone outside your site to watch certain pages on your site (load them periodically) and keep statistics about timing, sizes, errors • ideally get regular reports showing everything is groovy • support “drilling” to get more details when necessary

38. using logarithms to graph events with highly varying scales

39. some 50x errors showed up on utility servers one day

41. a primitive top 10 most popular pictures (PIs per hour)

42. What does this mean? (irregular load distribution, strange peeks)

43. somewhat subtle: unexpected cpu load spikes on utility machines

44. diagnosis: software release increases cpu load

45. Confirmation: rollback helps

47. a little bit about development methods • requirements gathering (quality varies) • planning • implementation

48. our implementation languages • browser • html (+ „furniture graphics“) • javascript • css (fairly recently) • public javascript libraries (fairly recently): MooTools, JQuery

49. One view of a web application pattern (Application Architecture Guide 2.0: Designing Applications on the .NET Platform

50. our implementation languages • webserver („front end“) • ASP (JScript, mainly 2000 – 2005) • DotNet 2 (2005 – 2007) • DotNet 3 (2007 - )