Building Defensive Architectures Using Backdoors

1. Building Defensive Architectures Using Backdoors Liviu Iftode Department Of Computer Science Rutgers University (iftode@rutgers.edu)

3. Rebooting Not Always Acceptable Service.com

4. Traditional (Eager) Recovery Approaches • Checkpointing (process, virtual machine) • Hot machine backups (state machine replication, primary-backup) • Limitation: time-consuming and costly • Intrusive during failure-free execution • Require dedicated machines • Require stable storage available after the failure

5. Lazy State Recovery? • OS and application state still in system memory Operating System CPU Mem NIC Disk TTY … • Can we recover or repair the state lazily (after the failure occurred)?

6. How to Access the State? CPU and OS resources not available

7. The World Today Internet Attacks Failure 10:00pm EST 3:00am GMT 8:30am IST • Computer maintenance requires human intervention: slow and expensive • Emails and phone calls do not scale • Worse at planetary scale: different time zones and language barriers

8. Our Vision : Defensive Architectures Internet Attacks Failure 8:30 AM IST 3:00am GMT 10:00pm EST • Computer Systems perform self-defensive management cooperatively • Access to memory possible even in presence of failures and attacks • Operating System not involved

9. Outline • Motivation • Backdoor Architecture • Components • Remote Repair using Backdoors • Lazy State Recovery using Backdoors • Defensive Architectures using Backdoors • Future Work and Conclusions

10. Backdoor Backdoor: a hidden software or hardware mechanism, usually created for testing and troubleshooting --American National Standard for Telecommunications

11. The Backdoor Idea “Front door” CPU Mem NIC Backdoor Management Infrastructure • Backdoor provides an alternative access to system resources • Aprogrammable I/O device for physical platforms • Avirtual machine for virtual platforms over a virtual machine monitor (VMM) • Backdoors can be connected over a private network, specialized interconnect, or even a cellular link

12. Backdoor Architecture Hardware Programmable device with processor and memory Software OS extensions for remote healing Firmware extensions for BD programming Design Principles Availability Failure of target OS must not impair BD Nonintrusiveness BD operations must not involve processors of the target system Access control monitor and target systems negotiate access permissions Integrity target system cannot modify the result of a BD operation Responsiveness target system cannot block BD operation Target System Monitor System CPU CPU BD BD Mem Mem I/O I/O

13. Backdoor Implementation Target Monitor MONITOR (Remote-R) Memory Memory CPU CPU RECOVER/REPAIR (Remote-R/W) BD BD NIC CPU

14. Nonintrusive Remote Healing • Three components • Detection, Diagnosis, and Action • Performed nonintrusively from a remote machine • Zero-cycle monitoring and failure detection fortarget system • Remote extraction of useful state from a “hung” system for diagnosis and recovery • In-placerepair of OS state of a “damaged” system

15. Backdoor Software Architecture • Monitoring and Failure Detection • Sensor Box: system health indicators (sensors) provided by the target OS in its memory • Sensor: <UniqueID, Update Deadline , Value> • Repair of damaged OS State • Externalized State: OS state data that the BD can read • Remote Access Hooks: OS control data that the BD can write to perform repairing actions • Recovery of light-weight state • Continuation Box: fine-grain OS and application state that the BD can transfer between systems to migrate running applications

16. Outline • Motivation • Backdoor Architecture • Components • Remote Repair using Backdoors • Lazy State Recovery using Backdoors • Defensive Architectures using Backdoors • Future Work and Conclusions

17. Failure Model • Computer system freeze • OS bug: hang, crash, deadlock, etc. • OS damage: resource exhaustion, DoS attack • hardware: peripheral device stops responding • Fail-stop (no erratic behavior) • memory not wiped out during failure

18. Monitoring and Failure Detection Backdoor Target OS Monitor Sensor Box <Timer interrupts> <Context switches> <NIC interrupts> … • Target OS updates progress sensors in Sensor Box • Monitor BD reads Sensor Box periodically, checks counters • Failure = counter stalled beyond its deadline

19. Diagnosis and Repair • Diagnosis • Inspect live OS data structures in target’s memory through( the externalized state) • Identify damaged OS state e.g. resource exhaustion due to memory hogging processes • Repair • Modify target OS memory (remote access hooks) to correct damaged state (e.g. remove memory hogging processes by “injecting” a kill signal in its process control block)

20. Backdoor Prototype • Myrinet LanaiX NIC • Modified firmware and low level GM library • Modified FreeBSD 4.8 kernel • Sensor Box • Externalized State and Remote Memory Hooks • Two resource exhaustion case studies • Memory exhaustion • CPU starvation • Experimental setup • Dell Poweredge 2600 servers with 2.4 GHz dual Intel Xeon, 1GB RAM, 2GB swap, Myrinet Lanai X NIC • Benchmark: a simple CPU-bound application

21. Effectiveness of Remote Repair

22. Repair Timeline

23. Outline • Motivation • Backdoor Architecture • Components • Remote Repair using Backdoors • Lazy State Recovery using Backdoors • Defensive Architectures using Backdoors • Future Work and Conclusions

24. Internet Services and Servers C1 Server 1 C2 servers Internet

25. Internet Services and Servers C1 Server 1 C2 service servers Internet Server 2

26. Service Continuation (SC) C1 Streaming Server 1 GET “96.3FM” C2 SC2 = {“96.3FM”, “2nd song”} Streaming Server 2

27. Service Continuation (SC) C1 Streaming Server 1 C2 Streaming Server 2 SC2 = {“96.3FM”, “2nd song”}

28. Service Continuation (SC) C1 Streaming Server 1 C2 Streaming Server 2 SC2 = {“96.3FM”, “2nd song”}

29. Continuation Box (CB) • Idea • extract “essential” state • pass it to similar application on a healthy machine • CB encapsulates fine-grained server state associated with a client session • OS data, e.g., data in transit through IPC channels • Application data • Application may need to cooperate with the OS!

30. Lazy Extraction of Continuation Box Recovered State Continuation Box Memory Memory CPU OS BD BD Recovery machine (healthy) Victim machine (crashed)

31. Recovery: What and How? Victim machine Recovery machine 1 recv 2 Backdoor recv 3 recv CB = ??? Time

32. Solution: Continuation Box API Victim machine Recovery machine 1 recv 2 Backdoor recv 3 export() recv log import() 3 CB recv Time

33. Service Continuation Structure SC_APP SC_IO Front-end server process Back-end server process Client 1 TCP/IP IPC SC1 Client 2 SC2

34. Service Continuation API • export SC_APP • import SC_APP • create_sc for a client session • associate I/O channel with the SC • open_sc from an I/O channel

35. A Server with Service Continuation while (cid = accept()) { scid = create_sc(cid) if (import(scid, &{file_name, offset}) == NULL) { receive(cid, file_name) offset = 0 } fd=open(file_name) seek(fd, offset) while (read(fd, block, size) != EOF) { send(cid, block, size) offset += size export(scid, {file_name, offset}) } }

36. Case Study: Multi-tier Internet Service Front-End (FE) Apache web server Middle Tier (MT) JBoss app. server Back-End MySQL DB server

37. Recoverable Service

38. Recoverable Service • Experimental setup • Dell PowerEdge 2600 servers, 2.4 GHz dual Intel Xeon, 1GB RAM, 1Gb Ethernet • Workload modeled after TPC-W • Fault injection in FE and MT nodes • synthetic freeze, emulated freeze by remote OS locking, bugs inserted in network drivers • Evaluation • Low cost, low overhead under load • Fast recovery

39. Low Cost • Monitoring • < 1% CPU @ 100 ms sampling period (100 sensors) • Continuation Box • API export/import < 30 us • Extraction 358 us for 10 KB CB

40. Low Overhead under Load

41. Recovery Timeline

42. Outline • Motivation • Backdoor Architecture • Components • Remote repair using Backdoors • Lazy State Recovery using Backdoors • Defensive Architectures using Backdoors • Future work and Conclusions

43. Defensive Architectures Using Backdoors • Autonomous Backdoors • BDs are programmed to execute defensive activities during bootstrap, then “sealed” • Tamper-resistant during normal execution: OS cannot alter or stop BD execution • BDs communicate among themselves to execute certain defensive activities cooperatively • Hierarchical Defensive Architectures • Defensive Computer Architecture (DCA) : Single computer system equipped with BD • Defensive Network Architecture (DNA) : Cluster nodes equipped with BD connected over high-speed private network • Defensive Inter-Network Architecture (DINA) : Loosely coupled DNA clusters over a Wide Area Network

44. Applications of Defensive Architectures • Smart Watchdog for DCA • Continuously monitor the system memory • Identify and enforce OS invariants in the host memory or the I/O system • Search for virus/worm signatures • Continuous Remote Logging and Integrity Verification over DNA • Continuously retrieve logged data from system memory • Send it to another node in the DNA • Cooperative OS integrity verification • Defensive News Agency over DINA • A global secure information network • Critical system controllers (routers, GRID control nodes, PlanetLab peers, etc.) subscribe to it • Multiple DNAs publish information to the system • System propagates information of interest about the Internet, individual networks, or hosts

45. Future Work • Virtual Backdoor • Realization of the BD for virtual environments e.g. VMMs and PlanetLab • Enables planetary-scale system monitoring and management • Orion : Holistic Approach to System Monitoring • Continuous non-intrusive physical memory inspection over BD • Identify memory modification patterns and correlate them to predict unstable system states • BD Language • BD can execute basic building blocks of defensive actions • Express complex defensive actions using the basic building blocks • Security • Prevent malicious users from using BDs to perform remote attacks • Authenticate and verify actions before performing them • BD over the Phone • Use cellular link to access the BD for system management operations

46. People Behind Backdoors • Florin Sultan • Aniruddha Bohra • Pascal Gallard (INRIA/IRISA, France) • Iulian Neamtiu (University of Maryland) • Stephen Smaldone • Yufei Pan • Arati Baliga • Tzvika Chumash