Reporter: 高嘉男 Advisor: Chin-Laung Lei 2010/3/15

1. Active Botnet Probing to Identify Obscure Command and Control ChannelsG Gu, V Yegneswaran, P Porras, J Stoll, and W Lee - on Annual Computer Security Applications Conference 2009(ACSAC 2009) Reporter: 高嘉男 Advisor: Chin-Laung Lei 2010/3/15 1

2. Outline • Introduction • Problem statement & assumptions • Active botnet probing: architecture & algorithms • Experiments with BotProbe • Conclusion

3. Introduction • Botnet C&C channel: existing protocols • IRC, HTTP & P2P • Botnet detection: passive • Signature-based detection • Honeypot-based detection • Behavior-based botnet detection • Contemporary IRC botnet • Obfuscated IRC messages • Small sizes • Infrequent C&C interactions

4. Active Method • Collect evidence actively • Assume there is only one round of (obscure) chat-like botnet C&C interaction from one bot, can we still detect the bot with a high probability?

5. Key Observations • Botnet C&C interaction has a clear command-responsepattern • A bot will behave deterministically to replayed commands • Bots are preprogrammedto respond to the set of commands they receive • Bots have limited tolerance for typographical errors in conversations

6. Adversary Assumption • A bot should respond when it receives a predefined command in a reasonable time • Message response • IRC PRIVMSG message • Activity response • Scan response • Third-party response • Spam response

7. Architecture Design

8. Active Probing Techniques

9. Active Probing Techniques (Cont’d) • P0 (Explicit-Challenge-Response) • Reverse Turing test • Request the user to visit a website to read and translate a CAPTCHA • P1 (Session-Replay-Probing) • Replay the same application command to the client several times

10. Active Probing Techniques (Cont’d) • P2 (Session-Byte-Probing) • The BotProbe monitor randomly permutes certain bytes of the application command • P3 (Client-Replay-Probing) • Register a new user into the channel • Send the observed command(s) to the selected client • P4 (Man-In-The-Middle-Probing) • Intercept the newcommand and launch a man-in-the-middle-like chat message injection

11. Turing-Test-Hypothesis Algorithm • Perform one or more rounds of P0 probing • H1: the hypothesis “botnet C&C” • H0: the hypothesis “normal chat” • Binary random variable D: whether or not we observe a wrongreply for a challenge from the client (D = 1: an incorrect reply) • θ1 = Pr( D=1 | H1), θ0 = Pr( D=1 | H0) • θ1 ≒ 1, θ0 ≒ 0 • α: false positive rate, β: false negative rate • n : rounds of probing • Define

12. Turing-Test-Hypothesis Algorithm (cont’d) • Threshold random walk (TRW) • Walk starts from origin(0) • Walk goes up with length ln(θ1/θ0)if Di = 1 • Walk goes down with length ln(1-θ1/1-θ0)if Di = 0 • After n rounds • If Λn > ln(1-β/α): H1 is true, it is a botnet C&C • If Λn < ln(β/1-α): H0 is true, it is a normal IRC dialog • If else: additional rounds of testing

13. Single-Binary-Response-Hypothesis Algorithm • Perform one or more rounds of P1 probing • D: whether or not a response from the client is observed • Iterate the TRW process at different scales depending on the responses • Multiple different types of responses corresponding to the same command • Choose the one that provides highest confidence (walks a largest step)

14. Interleaved-Binary-Response-Hypothesis Algorithm • Perform one or more rounds of interleaved P1 and P2 probing • D = 1: the observation of a response from the replayed packets and no response from modified packets • Bots • Respond to replayed packets reliably • Donot recognize the modified command • Human • Respond to a message with typographical error • How normal users may respond to two replayed IRC messages?

15. Evaluating User Disturbance • The degree of disturbance • The number of rounds (packets modified/replayed) • To produce a botnet C&C declaration • To produce a human user IRC channel declaration

16. Evaluating User Disturbance (cont’d)

17. BotProbe: an Active Botnet Probing System

18. Test the False Negative Rate • How many bot C&Cs are missed by BotProbe? • Execute the bot in Windows XP (VMware) • Monitor with BotProbe on Linux • Three classes of real-world IRC bots • Open-source bots with obfuscated communication • Spybot • Bot binaries with cleartext communication • Phatbot, Rbot, Rxbot, Sdbot • Bot binaries with obfuscated communication • W32.Wargbot, Trojan.Dropper.Sramler.C

19. Test the False Negative Rate (cont’d) • Parameters of testing algorithm • θ1 =0.99, θ0 =0.15, α(FP)=0.001, β(FN)=0.01 • θ0scan=0.01, θ03rd-party-access=0.02

20. Test the False Negative Rate (cont’d) • W32.Wargbot • Put an encrypted command in the IRC TOPIC message for bots to execute • Trojan.Dropper.Sramler.C

21. Test the False Positive Rate • How frequently could normal chatting sessions be mislabeled as botnet C&C • Study design • Human users periodically sent messages that simulate the effect of botnet probing to real users at diverse channels • Test on two different platforms • IRC & mebbo.com

22. Test the False Positive Rate (cont’d) • Study design • Design six different questions to test 123 different users • Questions • “what’s up” “nice weather” “you like red?” “how may I help you?” “English only! I play nice fun” • Modified questions • “waat’s up” “noce weather” “aou like red?” “Bow may I help you?” “Eaglish only! I play nice fun” • Turing test messages • “what’s 3+6=?”

23. Test the False Positive Rate (cont’d)

24. Conclusion • The first feasibility study of the use of active techniques in botnet detection • Collect evidence actively • Shorten the detection time • A hypothesis testing framework & a prototype system implementation • Separates deterministic botnet communication from humanconversations effectively

25. Reference • G Gu, V Yegneswaran, P Porras, J Stoll, and W Lee, “Active Botnet Probing to Identify Obscure Command and Control Channels.” in Annual Computer Security Applications Conference, 2009.