Central Web Services at Fermilab

1. Central Web Services at Fermilab Presented by Jim Fromm October 27,2006

2. Presentation Overview • Why we have a central web farm • Configuration of farm • Hardware • Software • Automated tools for administration • Monitoring of web farm • Log processing • Futures

3. Why a Central Webserver Farm? • Eliminate experiments from worrying about configuration/maintaining their own web server • Keep on top of security issues • Maintain up-to-date versions of apache/perl/python etc • Maintain valid ssl certs • Leverage our expertise to provide basic consulting for cgi scripts, web page development etc.. The web managers are responsible ultimately, but if they ask nice… • Manage file system space (AFS has a quota mechanism, we keep track of when an area is getting full).

4. Overview of Central Webserver Farm • 5 Computers – 1 Load balancing switch. • 84 vhosts (and rising fast) • 53 Additional web areas (conferences, projects, computing…) • 1300 Web Content Managers • Web hits/year: > 220 million • Staff: • Basically 0.5 of one person right now.

5. Hardware Configuration Alteon www01 www02 www03 www04 (cgi) www05 (cgi) AFS

6. Hardware Configuration - Details • Alteon Load Balancing Switch • Configuration allows for traffic to be directed depending on type. • CGI scripts only executable on www04/05. • Alteon AD3 switch, 8x 100Base-T ports with a single GigE uplink • Web Servers • Sun Netra X1 • AFS file system

7. Alteon Configuration 51: 131.225.70.46, enabled, name www05-vhosts, weight 1, timeout 10 mins, maxcon 200000 backup none, inter 2, retry 4, restr 8 remote disabled, proxy enabled, submac disabled handle URL cookie: disabled exclusionary string matching: disabled 1: any 2: /cgi-bin 7: .php real ports: http: vport http, group 11 HTTP Application: urlslb virtual server: 4, 131.225.70.20, enabled http: vport 8875, group 11 virtual server: 4, 131.225.70.20, enabled http: vport http, group 11 HTTP Application: urlslb virtual server: 5, 131.225.70.52, enabled https: vport https, group 12 virtual server: 4, 131.225.70.20, enabled 4443: vport https, group 99, pbind sslid

8. Software • Apache v1_3 • Mid-range plans to upgrade to v2_0 as security requires. • Perl • Python • PHP • Wiki support on Plone server (on separate set of servers outside of the webfarm)

9. Automated Tools • No way we can keep on top of everything the old fashioned way. • Tools (perl scripts mostly) written to automate routine tasks • Create vhosts • Symlink check • Password check • File perms check • File space check

10. Automated Tools(cont) • Symlink check • Run 1x week • Check for symlinks to areas that are sharing data that should not be shared • Check for symlinks pointing to non-existent data • Check for circular links • Sends email to web admin team

11. Automated Tools (cont) • Permission check • 1x per week • Scan for vulnerable cgi scripts, weak permissions on files and directories • This can be any variation that leave security holes or fit the profile of known exploits: • wide read-access permissions to area where passwords are stored • write-access to cgi area • wide read-access permissions to top level directories

12. Automated Tools(cont) • Passwords (1x week) • Password files, although encrypted, should not be shared • algorithm is not particularly smart: looks for variations on the word "PASSWORD" in file name and reports these if file permissions or locations are problematic

13. Web Server Monitoring • NGOP (FNAL developed monitoring system) remotely monitors and alerts on the following: • Ping of Alteon switch • Ping of each web host machine • Ping of each web server IP address • Fetch pages for each web server and virtual host • Fetching pages for commonly used services (telephone,stock) and checking correct results • Verifying that the httpd for each server is running on at least one webserver host

14. Web Server Monitoring (cont) • Verifying that "fs examine" succeeds on the main web volume for each server (/afs/fnal/files/expwww/*), and any specific other volumes associated with it • Watching each webserver's error log on each web host, and reporting important error messages (i.e. "out of memory") • Generally an error will cut a helpdesk ticket, and page the primary.

15. Log Processing with Urchin • Urchin v5 • This product provides accurate web site analytics which supplies the executives, marketers, webmasters, and the web designers at your firm with the vital up-to-date information they need to make informed business decisions. Blah blah blah. • Recently taken over by google. • Mixed opinions about the product…

16. Urchin (cont) • Urchin likes to just stop processing without notification. • Lots of pretty pictures and gizmos. Overkill for what we need, but if you are running an e-commerce business…. • We bought Urchin to remove dependence on a home grown log processing package that was very difficult to maintain. • Urchin is easier to maintain when it works.

17. Security Requirements • Run Nessus Scans 2x year. • Nessus comes with a library of known exploits scanning profiles. • Scan various newsgroups looking for announcements. • Rely on our security team to alert us. • Follow defined baseline of apache webservers developed at Fermilab.

18. Future Plans • We are looking to upgrade to using the Cisco load balancer switch • SunFire V240 servers • Considered moving to Linux, but… • Not confident of support model. Great for general purpose farms, not as confident for server level service. • Lot’s of work to convert. Software installs, possibly broken cgi scripts etc… • Apache 2.0 • Conscious of user communities in content management systems. CMS are hot items.

19. Thanks for your attention! • That’s all • One more thing before I go….

20. GO CARDS!!!!