On the Difficulty of Scalably Detecting Network Attacks

1. On the Difficulty of Scalably Detecting Network Attacks Kirill Levchenko with Ramamohan Paturi and George Varghese

2. Detecting Attacks • Which Intrusion Detection problems are hard (require per-flow state)? • Counting distinct flows? Approximately? • Detecting SYN Flooding? Port Scans? • Are there efficient algorithms we don’t know yet? • How to tell?

3. Theoretical Techniques • Abstract problem formulation • Communication Complexity • Source for NIDS problem reductions • Establishes worst-case lower bounds • Easy and fun to use

4. Outline • Problem Motivation and Setting • The Set Disjointness problem • Application to SYN Flooding • Implications • Conclusion

5. Idea: move the NIDS into the network, protect more systems! Problem Setting

6. Problem Setting • Protect more hosts • Single device to administer • More informed position

7. Problem Setting • Challenges: • More hosts to protect • Higher bandwidth links • How do resource requirements scale? • Are there fundamental limits?

8. Problem Setting Outbound • Two packet sequences: traffic entering and leaving the network • Goal: detect attack using packet sequences Inbound

9. Algorithms & Reductions • Formalize the algorithmic problem • An algorithm on packet sequences • Reduce one problem to another • Show solving one problem solves the other

10. Our Reduction Set Disjointness → NIDS Problem • Solving the NIDS problem solves the Set Disjointness problem • Use strong bounds on Set Disjointness • Establishes space lower bound for the NIDS problem

11. Disjoint? Set Disjointness Alice Bob

12. Set Disjointness Communication: Alice Bob (even randomized)

13. Let’s Try It!(on SYN Flooding)

14. SYN Flooding Normal Attack Goal: detect unclosed connections.

15. SYN Flooding • Creates packet sequence corresponding to her set with SYN flag • Runs NIDS algorithm on input sequence • Suspends it after reading the last item • Sends the state of the algorithm to Bob

16. SYN Flooding • Creates packet sequence corresponding to elements not in his set with FIN flag • Resumes NIDS algorithm on input sequence (using state sent by Alice) • Result indicates if sets intersect or not Yes/No

17. SYN Flooding Algorithm sees: • If A and B intersect, there is a SYN packet not followed by a FIN • If A and B are disjoint, every SYN packet has a matching FIN

18. SYN Flooding • Alice sent Bob the state of the NIDS algorithm: • Set Disjointness requires Ω(n) bits • The state of the algorithm is Ω(n) bits no. of flows

19. Other NIDS Problems • Port Scans • per-host state required • TCP connection hijacking • per-flow state required • Evasion by fragmentation • re-assembly required

20. Implications • Sometimes per-flow state is required, but: • Examples are artificial (may not occur in practice) • Problem semantics may be a great help • System can fail gracefully when out of memory • Additional information may be available!

21. Implications • Additional information can be useful: • Count outgoing SYN+ACK and FIN packets • Works if protected network can be trusted • Hop-count filtering or other fingerprinting • Small UDP fragments are unusual

22. Conclusion • Set Disjointness is a useful reduction source • Even hard under randomization • Permits relaxations (see paper) • Exposes and formalizes the hardness of some NIDS problems • Guides practical algorithm design to consider hard cases and failure modes