Protomatching Network Traffic for High Throughput Network Intrusion Detection

1. Protomatching Network Traffic for High Throughput Network Intrusion Detection

2. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. NIDS Attacker Network Signature database

3. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. GET <URL>/cmd.exe HTTP/1.1\n NIDS Attacker Network • “cmd.exe” is the attack pattern Signature database cmd.exe

4. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. Be aware of the “cmd.exe” attack NIDS Shai Network • “cmd.exe” is the attack pattern Signature database cmd.exe

5. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. GET <URL>/cmd.exe HTTP/1.1\n NIDS Attacker Network • “cmd.exe” is the attack pattern, • but only if it is part of a URL Signature database cmd.exe

6. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. POST <URL>/cmd.exe HTTP/1.1\n NIDS Attacker Network • “cmd.exe” is the attack pattern, • but only if it is part of a URL, • and the HTTP method is GET Signature database cmd.exe

7. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. GET <URL>/CMD.exe HTTP/1.1\n NIDS Attacker Network • “cmd.exe” is the attack pattern, • but only if it is part of a URL, • and the HTTP method is GET, • and takes into account upper-lower case characters, Signature database cmd.exe

8. Signature evolution • Informally, a signature is usually defined as “a characteristic pattern of the attack”. GET <URL>/%43MD.exe HTTP/1.1\n NIDS Attacker Network • “cmd.exe” is the attack pattern, • but only if it is part of a URL, • and the HTTP method is GET, • and takes into account upper-lower case characters, • and takes into account HTTP encodings Signature database cmd.exe

9. Problem in This Talk TCP streams • What we specify: a traditional signature that exposes: • false negatives • false positives cmd attack A traditional signature Goal: Develop a signature that is cheaper to enforce What we enforce: a signature that inherently fits the attack. TCP streams cmd.exe attack A traditional signature

10. Contributions • Conceptual: Protomatching signature • Practical: Superset Protomatcher • Real world impact: 25% improvement in Snort performance

11. Protomatching Signature • It is a regular expression with two properties: • Ensures that the characteristics pattern of an attack appears in the context that is necessary for the attack to succeed. • Second, a protomatching signature matches both normalized and encoded versions of an attack.

12. Superset protomatcher • It recognizes a superset of the traffic matched by a full-coverage protomatcher. • Three properties: • A superset protomatcher consumes less memory. • Traffic that matches the superset protomatcher may do not match any NIDS signatures • Traffic that does not match the superset protomatcher also does not match any signature in the NIDS database.

13. Related work • Protocol analysis and traffic normalization • Modern NIDS are based on the ANM methodology. • Ptacek and Newsham were the first to recognize that a NIDS that does not perform normalization is susceptible to evasion. • The problem of alternate encodings is particularly painful for HTTP traffic.

14. Related Work II • Fast pattern matching for NIDS • Previous work does not solve encodings problem, and does not consider protocol analysis in matching algorithm • Researchers have proposed using regular expression matching • To match regular expressions, Sommer and Paxson used a DFA. However, they performed matching on already-normalized traffic.

15. Related Work III • Dealing with high-speed links. • To deal with high-speed links, researchers have suggested a distributed NIDS that balances the network traffic such that each sensor monitors a different portion of the protected network • Our work focuses on the performance of a single sensor. It can perform better with cooperating distributed design.

16. Analyze-normalize-match (ANM) approach • First, a NIDS encodes its signatures in a normalized form • During runtime, NIDS parses the traffic according to the protocol the attack uses and normalizes the traffic • Last, the NIDS matches the normalized traffic against its normalized signatures.

17. Current conversion and signature matching • Naively, each phase requires traversing the input • In practice (e.g., Snort) two traversals: • Protocol analysis + normalization • Matching • Notice that all traffic, benign and malicious, requires all three phases GET <…>/%43MD.exe HTTP/1.1\n Protocol analysis Method = GET URL = <…>/%43MD.exe Version = HTTP/1.1 Normalization Sig=CMD.EXE URL=CMD.EXE String matching No Yes Benign Malicious

18. Protomatching GET <…>/%43MD.exe HTTP/1.1\n GET <…>/%43MD.exe HTTP/1.1\n Protocol analysis Sig=???? Method = GET URL = <…>/%43MD.exe Version = HTTP/1.1 • Goal: • Single traversal on the input • Protomatching= • Protocol analysis+ Normalization+ • Matching Normalization Sig=CMD.EXE URL=CMD.EXE Pattern matching No No Yes Yes Benign Malicious Benign Malicious

19. Protomatching GET <…>/%43MD.exe HTTP/1.1\n GET <…>/%43MD.exe HTTP/1.1\n Protocol analysis Sig=Regular expression Method = GET URL = <…>/%43MD.exe Version = HTTP/1.1 Single pass implies: use a Deterministic Finite State Machine Normalization Sig=CMD.EXE URL=CMD.EXE Pattern matching No No Yes Yes Benign Malicious Benign Malicious

20. Converting a traditional signature into a protomatching signature • Let S be a traditional signature • Expand S to conform to the protocol specification

21. Traditional signature • *[c|C][m|M][d|D].[e|E][x|X][e|E] • 8 states • size = 8*256=2048 bytes

22. Add a little bit of context • *”GET”*[c|C][m|M][d|D].[e|E][x|X][e|E] • 12 states • size = 12*256=3072 bytes

23. And even more context • (*\n\n)*”GET”[SP]+(PN)*[c|C][m|M][d|D].[e|E][x|X][e|E] • 18 states • size = 18*256=4608 bytes • SP denotes white space characters, and PN denotes characters • that can appear in a URL according to the HTTP specification • (e.g., ‘\n’ cannot appear in a URL).

24. Converting a traditional signature into a protomatching signature • Let S be a traditional signature • Expand S to conform to the protocol specification, obtaining S’ • Expand S’ to account for all possible encodings, obtaining S’’

25. Representing encodings The character c can be represented as: C, c, %43, %63, %U0043, %U0063, %u0043, %u0063 Replace every instance of the small machine with the large machine

26. And even more context • (*\n\n)*”GET”[SP]+(PN)*[c|C][m|M][d|D].[e|E][x|X][e|E] • 18 states • size = 18*256=4608 bytes

27. *\n\n”GET”[SP]+(PN)*[c-C][m-M][d-D].[e-E][x-X][e-E]and HEX encoding and Uencoding • 53 states • size = 53*256=13,568 bytes

28. Building a protomatcher • Let S be a traditional signature • Expand S to conform to the protocol specification, obtaining S’ • Expand S’ to account for all possible encodings, obtaining S’’ • Perform 1-3 for every traditional signature in your database, obtaining S1’’, S2’’,…,Sn’’ • Build the protomatcher: an FSM that identifies S1’’S2’’,…,Sn’’ Problem: we increased each signature by factor of 7 (at least). A full protomatcher does not fit into 2GB (or 4GB) of memory

29. Superset protomatching signature • Assumption: the majority of the benign traffic is not only benign, but also not even similar to malicious traffic. • For example, most benign traffic not only does not contain “cmd.exe”, but also does not contain “cmd.” • Note that is a request does not contain “cmd.”, then it also does not contains “cmd.exe” • “cmd.” is a superset signature because it matches the attack and more

30. Full protomatching signature for cmd.exe • *\n\n”GET”[SP]+(PN)*[c-C][m-M][d-D].[e-E][x-X][e-E]and HEX encoding and Uencoding • 53 states • size = 53*256=13,568 bytes

31. Superset protomatching signature for cmd.exe • *\n\n”GET”[SP]+(PN)*[c-C][m-M][d-D].[e-E][x-X][e-E]and HEX encoding and Uencoding • 29 states • size = 29*256=7,424 bytes

32. Building a superset protomatcher • Let S be a traditional signature • Trim S into a superset signature (e.g., “cmd.exe” into “cmd.”) obtaining S’ • Expand S to conform to the protocol specification, obtaining S’’ • Expand S’’ to account for all possible encodings, obtaining S’’’ • Perform 1-3 for every traditional signature in your database, obtaining S1’’’, S2’’’,…,Sn’’’ • Build the protomatcher: an FSM that identifies S1’’’S2’’’,…,Sn’’’

33. Superset Protomatching GET <…>/%43MD.exe HTTP/1.1\n GET <…>/%43MD.exe HTTP/1.1\n Sig=superset protomatching signature Protocol analysis Method = GET URL = <…>/%43MD.exe Version = HTTP/1.1 Superset Protomatcher: match a superset protomatching signature Yes Normalization Sig=CMD.EXE URL=CMD.EXE Pattern matching No No Yes Yes Benign Malicious Benign Malicious

34. Implementation • Implemented a compiler that converts a traditional signature into a protomatching signature • The compiler also builds the protomatcher • Incorporated the protomatcher into Snort • Used traditional Snort as the second phase of a superset protomatcher

35. Two ways to implement Protomatcher • Using a deterministic FSM. That is what we do in the examples used. • Using a hierarchical FSM. It has two parts: a matcher and a normalizer. • The matcher is responsible for protocol analysis and pattern matching. • The normalizer is responsible for processing multiple encodings. • Unlike ANM which first normalizes the whole http request, it uses the normalizer only when necessary. • Can help reduce memory needed.

36. Performance improvement ApPPT: Average per Packet Processing Time (cycles)

37. Comparison between Protomachers memory size

38. Sensitivity to Cache Poisoning Attack • We assumed that the attack would have a larger effect on a protomatcher-based Snort than on vanilla Snort. • But the result contradicts the assumption. There might be two reasons for this result: • First, the attack was ineffective in increasing the number of cache misses. It means that a more sophisticated cache poisoning attack is needed. • Second, the attack was effective, but cache performance is only a minor component of the ApPPT.

39. Conclusion • Optimize for the common case is a known method • In this talk we presented develop a technique that uses this method to improve matching efficiency • Our technique is based on formal methods • These methods enable automation, therefore efficiency, and facilitates accuracy

40. Discussion on shortcomings • Failure due to Cache-poisoning attacks • Converting a Protomatching signature to a superset signature should be done manually. Better methods?