Living Behind Administrative Firewalls at Stanford: A Survival Guide August 5, 2005 2-4 pm, Turing Auditorium

1. Living Behind AdministrativeFirewalls at Stanford:A Survival GuideAugust 5, 20052-4 pm, Turing Auditorium Sunia Yang sunia@networking.stanford.edu Networking Systems

2. Topics • Security by protocol stack • What's a firewall? • What's a vpn? • How secure are we? • Stanford's deployment of firewalls and vpns • How to get behind a firewall • How to add/change/delete rules • Troubleshooting • Requesting rules - exercise (50 min)

3. Firewalls at Stanford • Everyone, especially sys admins and application folks, behind a firewall has to learn more about networking • Networking folks have to learn more about systems and applications • More process required- audit/authorization trail

4. Living with Firewalls- Mantras • "Know Thy Network Traffic" • If you don't know it now, you're going to learn it the hard way • "Know Thy Servers" • ditto

5. Security by Protocol Stack • Firewalls and VPNs are just part of a total security approach • Firewall would not have caught bugbear-b virus • Perimeter firewall or user vpn client would not have prevented Windows RPC, Agobot, Sasser

6. Physical Layer Security • "If you can touch it, you can hack it" • Lock up servers, network closets • Wireless- major physical security risk • firewall defeated if wireless behind firewall • allowing unencrypted wireless session through firewall defeats data security

7. Data layer • Switches as security device • isolates conversations- sniffer protection • may misbehave and "leak" • block by hardware address • not possible in all switches • hardcode hw address to port- tedious, unscalable

8. Network/Transport Layers • Filter traffic by IP addresses and ports • Router ACLs (may be leaky) • Firewalls (hardware, software) • Host IP filters • Require secure (not clear text) protocols or vpn • data encrypted (ssl, ssh) • encrypted data could still be virus or worm

9. Perimeter vs Server vs Dept "Firewalls" • Perimeter- at SUNet boundaries • Protect entire network at boundary • SUNet uses Routers and Packetshapers • SUNet leaves most ports open, only blocking particularly vulnerable ports (Windows, backdoors) • Guard mostly against worms/viruses/script kiddies • Does not guard against internal hacked machines • Server- protect essential data/services • Block all unused ports • Departmental- at the network boundary • still working out what we can support

10. Application Layer • Very critical layer for security- starts in the design • good architecture- 3 tier (separate web, app, db) • no group accounts • good passwords • secure transports- no cleartext data or passwords • critical data only on secured hosts (workstations!!!) • separate production and development (data, too!) • Good sys admins • patch, antivirus-software • turnoff unused services • Monitoring- • how to detect compromise

11. User/Political Layer • The most important security layer • Political • Policy and mandate • Funding • Enforcement • User • # 1 risk- disgruntled employees • data on workstations • bad passwords • usage - laptops, wireless, patching, AV

12. How secure are we and why do we care? • Computer security is among top financial risks to Stanford • loss of data and reputation • cost of cleaning hacked machines • legal liability- Hipaa (medical), Ferpa (student) • Audit reports are grim • many security issues at application layer

13. What is a firewall? • Network/transport layer • Passes traffic based on these basic criteria: • source IP • destination IP • destination port • session status- initiation, timeout • Generally, default is block everything

14. Why firewalls/vpns? • Physical and data layer security is critical • mostly implemented already (except wireless) • Too many badly architected apps on market • many assume perimeter and server firewalls • Often best return of security for given staff, time and money

15. Firewall Issues • No security on opened ports • Manageable rule set vs. many exceptions • Hard to secure port-hopping apps- VPN instead • Session timeout limits • Hosts are unprotected from hacked hosts behind firewall ( want personal firewall!!)

16. What is a vpn? • Network/transport layer • Establish session between two devices • vpn client and vpn server/concentrator • two firewalls • Encrypt traffic- secure cleartext traffic • Another layer of authentication/authorization

17. VPN Pros • With limited staff time and money, may get some application layer security • Sometimes can be used to enforce patch level of client operating systems

18. VPN Cons • May break IP dependent services • Inconvenient- processing overhead • Most vpn clients incompatible with each other • Incomplete security- allows encrypted path for hacker • Cost of vpn client support

19. VPN - Split Tunnel • only traffic to specific servers is encrypted • pros- performance • less encryption overhead • less traffic to central VPN concentrator • cons- security • if client host is hacked, hacker can control VPN session • no split tunnel allowed for admin apps

20. Stanford Public Vpn • Allows split tunnel • Two main uses: • access Windows svrs from outside • get Stanford IP for authorization VPN Client google.com su-vpn Library Resources Windows svr SUNet

21. Stanford Admin Vpn No split tunnel allowed Mainly to access firewalled servers VPN Client google www.stanford.edu server firewall firewall vpnap

22. General Steps For Firewall Design • Design topology • Firewall Rules • Enforce rules • Monitor, document, audit (not in this class) • Troubleshooting

23. Firewall Process at Stanford • Process tries to make things easier but auditable • Contact firewall team (firewall-team@lists ASAP • Fill out form + application layout • Meet and design topology • Order hardware, cables • Move behind firewall • Rule requests and approval • VPN access • SLA and charges • https://www.stanford.edu/group/networking/fwmaps/service_site/

24. New Firewall Form • form gives list of the process and required information • application diagram • hosts and required ports • transport between hosts • where's the data? • what trying to protect- hosts, data, service

25. A p p l i c a t i o n A r c h i t e c t u r e D i a g r a m - d r a s t i c a l l y s i m p l i f i e d D e v e l o p e W e b 1 W e b 2 P r o d u c t i o n W e b P r o d u c t i o n W e b D e v e l o p m e n t W e b S e r v i c e - P o r t 4 4 3 S e r v i c e - P o r t 4 4 3 S e r v i c e - P o r t 8 8 A p p 1 A p p 2 P r o d u c t i o n A p p S e r v i c e - P r o d u c t i o n A p p S e r v i c e - D e v e l o p m e n t A p p S e r v i c e - P o r t 4 5 7 2 P o r t 4 5 7 2 P o r t 4 5 7 3 D B 1 P r o d u c t i o n D a t a b a s e - T r a i n i n g D e v e l o p m e n t P o r t 1 5 2 1 D a t a b a s e - P o r t 1 5 5 0 D a t a b a s e - P o r t 1 5 6 0 D B 2 C i t r i x U s e r s O d d U s e r s S t a n f o r d u s e r - a n y w h e r e S t a n f o r d u s e r - a n y w h e r e r s - v p n c l i e n t O d d 1 O d d A p p l i c a t i o n P o r t 8 5 7 2 C i t r i x A p p E v e r y A r r o w P o i n t i s A n o t h e r R u l e ! C i t r i x 1

26. Meeting and Design • meet with someone from firewall team and information security • review form, application diagram, transports, data • propose initial firewall topology and schedule install

27. Laying out Firewall Topology • Group servers by • Sensitivity and type of data • Security level (don't put petty cash in the safe) • Production vs development • Especially as projects are out-sourced, don't want our data somewhere else in the world • Separate web from app/db layers • Sharing switches • Generally, databases or servers actually holding data should be on separate switch (no VLANs)

28. Basic Firewall Topology FW = firewall SW = switch S = server Firewall can only filter between zones by IP address and port Applications often use a well-known port Zone 1 FW1 Zone 2 Ex. Web Servers Zone 3 Ex. App Servers Zone 4 Ex. Database Servers SW1 vlan 20 vlan 30 SW2 S1 S2 S3 S4 S5 S6 S7 S8 S9

29. Routing vs Transparent FWs • Routing FWs • extra security because of layer 3 separation • standard for server firewalls • Transparent FWs • standard for departmental firewalls • acts like a switch

30. Ordering • firewalls- currently using Juniper/Netscreen • switches • cabling if in machine room • fwteam assigns switch ports for servers • sysadmin requests cabling from machine room infrastructure support through HelpSU

31. Server Moves • usually firewall is in routing mode • fw team notifies project owner that firewall ready • sys admin moves server behind firewall by unplugging old cable and putting in new cable. Sys admin also takes care of any needed NetDB changes if renumbering server

32. Rule Request Form • Ideally request initial rules before move • Template rules = set of commonly requested rules • backup • windows administration • solaris administration • linux administration • database administration • Rule request form • https://tools.stanford.edu/cgi-bin/firewall-request • reminder of required fields • archived for audit purposes • Requesters/approvers designated by app owner

33. Rule requires the following pieces: Action: Permit, Deny Source IPs: Client, VPN Client, Admin Destination IPs: Servers Destination Port: 80(web), 25(smtp), etc. Port Type: tcp, udp Firewall Rules- Part 1

34. Firewall Rules- Part 2 Examples: Allow 10.0.1.5 to 171.64.7.77 on udp port 53 (DNS) Allow 10.0.1.0/24 to 10.0.2.10 on tcp port 25 (SMTP) Deny 10.0.1.0/24 to any on tcp port 25 (SMTP) Sources: servers, clients, vpn clients, hackers (remember the last one when you are writing rules!!!!)

35. Types of Rules - Part 1 • Basic host functions - in template • DNS, NTP, ping • Remote host admin- mostly in template Monitoring – snmp, email, icmp Remote session - ssh, citrix Authentication - sident, kerberos, MS auth Maintenance - upgrades, virus, rebuilds, backup, file transfer Central systems –Microsoft domains/AD, afs, nfs

36. Types of Rules - Part 2 • Application specific Client: Web services, front end ports Server to server: db sharing, file transfer, app to db • Development Environments- training, development, etc Server to server: db sharing, file transfer, app to db Application build Developer access- in-house, remote

37. VPN access • traffic from sysadmin or application folks to servers should be over secure transports or over vpn (unencrypted SQL, etc) • authorization set by workgroups • app owner sets membership with Workgroup Manager (http://workgroup.stanford.edu) • currently, delegated vpn approver must send email to firewall-team to activate vpn account

38. Troubleshooting • A can't reach B which is behind firewall • Try ping first (allowed by default at Stanford on FWs) • If fails, check IP addr, physical connection • Try telnet to desired port • If okay, then not a firewall issue- probably app layer • Message like "Connected to B" • If fails, depends on message: • "Connection closed by foreign host" or "Connection refused" means B rejects A • Hangs with message "Trying B", finally getting message like "Unable to connect to remote host: timed out" means that port is not reachable- possibly firewall • Run "netstat" on B to see if ports are open • Ask us to log- see if traffic is coming thru or blocked

39. Common Problems • ~80% requests to check firewall show that firewall is not the problem • ~10% of time, previously unknown traffic ("know thy app") has no appropriate rule • Typos, miscommunication • Extra filtering on host itself • Host IP changes, thus breaking rule

40. Exercise Goals • understand firewall terms • understand how to construct a rule • understand the types of rules • understand how topology affects rule requests

41. A p p l i c a t i o n A r c h i t e c t u r e D i a g r a m - d r a s t i c a l l y s i m p l i f i e d D e v e l o p e W e b 1 W e b 2 P r o d u c t i o n W e b P r o d u c t i o n W e b D e v e l o p m e n t W e b S e r v i c e - P o r t 4 4 3 S e r v i c e - P o r t 4 4 3 S e r v i c e - P o r t 8 8 A p p 1 A p p 2 P r o d u c t i o n A p p S e r v i c e - P r o d u c t i o n A p p S e r v i c e - D e v e l o p m e n t A p p S e r v i c e - P o r t 4 5 7 2 P o r t 4 5 7 2 P o r t 4 5 7 3 D B 1 P r o d u c t i o n D a t a b a s e - T r a i n i n g D e v e l o p m e n t P o r t 1 5 2 1 D a t a b a s e - P o r t 1 5 5 0 D a t a b a s e - P o r t 1 5 6 0 D B 2 16 C i t r i x U s e r s O d d U s e r s S t a n f o r d u s e r - a n y w h e r e S t a n f o r d u s e r - a n y w h e r e r s - v p n c l i e n t 4 13 14 15 17 1 2 O d d 1 8 9 O d d A p p l i c a t i o n 5 18 19 P o r t 8 5 7 2 3 6 20 10 7 11 12 C i t r i x A p p E v e r y A r r o w P o i n t i s A n o t h e r R u l e ! C i t r i x 1

42. S y s t e m A d m i n i s t r a t i o n & D e v e l o p m e n t D i a g r a m ( g r e a t l y s i m p l i f i e d ) M a i l S e r v e r S y s A d m i n s W i n d o w s P D C s - 2 0 0 . 0 . 1 . 5 2 0 0 . 0 . 0 . 2 0 s s h - p o r t 2 0 0 . 0 . 0 . 3 0 2 2 s m t p - p o r t s m t p - p o r t 2 5 w i n d o w s p o r t s - 1 3 5 , 2 5 1 3 7 , 1 3 8 , 1 3 9 , 4 4 5 W e b 1 W e b 2 A p p 2 A p p 1 D N S s v r s O d d 1 C i t r i x 1 T i m e s v r s D B 1 D B 2 2 0 0 . 0 . 2 . 9 p i n g - p i n g - I C M P I C M P M o n i t o r i n g S e r v e r 2 0 0 . 0 . 3 . 1 1 A n y h o s p i n g - I C M P S w i t c h 1 S w i t c h 2 t e l n e t / s s h F w A d m i n s S w i t c h 3 S y s A d m i n s 32 t e r m i n a l s v c s s s h - p o r t 2 2 B a s t i o n 21 22 23 24 33 25 26 U n i x S e r v e r s W i n d o w s S e r v e r s 31 29 p i n g - p i n g - 27 28 35 I C M P I C M P 30 t A n y h o s t a f s m o u n t s a f s s v r s 2 0 0 . 0 . 5 . 0 / 2 4 37 E v e r y A r r o w P o i n t i s A n o t h e r R u l e ! N e t w o r k D e v i c e s

43. N e t w o r k T o p o l o g y H a c k e r H a c k e r U s e r V P N C l i e n t s W i r e d U s e r 1 0 . 0 . 1 0 0 . 6 4 - 1 0 . 0 . 1 0 0 . 1 2 6 S U N e t I n t e r n e t D N S / N T P R o u t e r 1 0 . 0 . 0 . 1 / 2 4 1 0 . 0 . 0 . 2 / 2 4 F i r e w a l l 1 0 . 0 . 1 . 1 / 2 4 1 0 . 0 . 2 . 1 / 2 4 Z o n e = D B 1 0 . 0 . 3 . 1 / 2 4 Z o n e = W e b Z o n e = A p p S w i t c h 1 S w i t c h 2 1 0 . 0 . 1 . 2 / 2 4 1 0 . 0 . 2 . 2 / 2 4 S w i t c h 3 1 0 . 0 . 3 . 2 / 2 4 W e b 1 D B 1 D B 2 A p p 1 A p p 2 W e b 2 C i t r i x 1 O d d 1 1 0 . 0 . 1 . 3 / 2 4 1 0 . 0 . 1 . 4 / 2 4 1 0 . 0 . 2 . 3 / 2 4 1 0 . 0 . 2 . 4 / 2 4 1 0 . 0 . 3 . 4 / 2 4 1 0 . 0 . 3 . 3 / 2 4 1 0 . 0 . 3 . 5 / 2 4 1 0 . 0 . 3 W i r e l e s s U s e r D e v e l o p e r D e v e l o p e r S y s A d m i n S y s A d m i n 2 0 0 . 0 . 2 . 9 M o n i t o r i n g 2 0 0 . 0 . 3 . 1 1 a f s / n f s s v r s 2 0 0 . 0 . 5 . 0 / 2 4 2 0 0 . 0 . 0 . 2 0 P D C s M a i l S v r s 2 0 0 . 0 . 1 . 5 2 0 0 . 0 . 0 . 3 0 B a s t i o n H o s t 1 0 . 0 . 4 . 2 / 2 4 1 0 . 0 . 4 . 1 / 2 4 . 6 / 2 4