UTM-1 Troubleshooting and advanced configuration

1. UTM-1 Troubleshootingand advanced configuration Boaz Anin, Mail Security Team

2. Agenda • Web Filtering – how does it work? • Anti Virus • Messaging Security • Kernel / Usermode separation • Kernel debug output • Usermode Messaging Security processes • Usermode debug output • MS Daemons • MS Troubleshooting

3. Web Filtering • in.httpd security server – 500 concurrent connections per process • Communicates with in.aciufpd daemon. • in.aciufpd downloads incremental updates of categorized URLs from OEM’s site • First time download is 128MB • List is kept in memory and on disk • Update size is < 1MB • Default update interval is 120 minutes • HTTP GET requests are parsed in in.httpd and a categorization request is sent to in.aciufpd. • According to the matched categories, the configured action is taken. • A URL can match several categories • Allowed URL list / Blocked URL list are also ‘categories’.

4. Web Filtering – scaling • 500 concurrent connections per process can sometimes be a limitation • Number of process can be configured using GuiDBedit tool • C:\Program Files\CheckPoint\SmartConsole\NGX R65 with Messaging Security\PROGRAM\GuiDBedit.exe • Other->content_security->Global_security_server_settings->http_process_num • Default value is 6, maximum is 12 • Note that in previous versions this was configured in fwauthd.conf and required cpstop/cpstart • In a future version there will be no security servers • Virtually no limitation for concurrent connections • Parsing will be done in the kernel • Requests will be sent to the usermode using traps • Local cache in the kernel, however, will match 92% of requests

5. Anti Virus • Works with HTTP, FTP, SMTP and POP3. Uses the security servers architecture. • Performance was improved dramatically in R65 • The raise_trigger (thread-related function) queue was filled quickly under load. • Updates are downloaded to the management and distributed between the GWs.. • Future plans: Streaming AV from the kernel – signature based • Will achieve amazing performance

6. MS: Activities in the kernel • Anti Spam connection matching (and AV, UF matching separately) • Whether to enforce Anti Spam features on this session • By IP or by direction • Exceptions • Both SMTP and POP3 • In IP Allow list  No match • IP Block List • SMTP only (in POP3 the IP is the mail server) • Performed after connection is matched for Anti Spam enforcement • IP Reputation • SMTP only • Communicates with usermode daemon (in.msd)

7. Activities in the usermode • IP Reputation query • Done from the in.msd process which queries the ctipd daemon. • Gets query request from the kernel and returns the result to the kernel. • Sender-address Allow/Block lists • in.emaild.smtp / in.emaild.pop3 security servers (processes). • Both SMTP and POP3 • Mail Anti Virus • Content Anti Spam engine • in.emaild.smtp / in.emaild.pop3 • Communicate with ctasd daemon which queries the detection center • Zero-Hour protection (an AV feature) actually done the same way

8. Content Security Updates • Unlike SmartDefense, updates are done from GW, not from GUI • AV and URL filtering are using the same mechanism • Make sure GW has Internet Connectivity as well as DNS • You can define a proxy to get the updates: • Use GuiDBedit • other->content_security->global_AV_settings->signature_updates->proxy_address

9. Debugging the kernel – connection matching • Use fw ctl zdebug + content mail • On the enforcing module (expert mode) • ‘content’ and ‘mail’ are debug flags • IP addresses appear in hexadecimal representation • Folding: The connection will pass through a security server • Transparent proxy. in.emaild.smtp or in.emaild.pop3

10. Debugging the kernel – connection matching

11. Debugging the kernel – connection matching • There was no connection match here. • Folding is done because AV is on • ifs_types is (interface) types: 1 – external, 4 – internal, 2 – dmz. • If -1, there’s a topology configuration error • CI also includes AV and UF match information.

12. Debugging the kernel – IP reputation • Source IP was not in local cache • An IP reputation query was sent to the usermode (trap) • Usermode updates local cache • Returns to the kernel with ACCEPT (IP is OK)

13. Debugging the kernel – IP reputation • Here IP reputation is off. • RDNS and DNSBL are advanced undocumented features • Enabled from $FWDIR/conf/mail_security_config file. • Not recommended to use them • Local cache is cleaned during policy installation

14. Debugging the kernel – Block list • Source IP was in the Block List • Connection is rejected without querying for IP reputation • The reject SMTP error message which is sent to the client is also displayed

15. Usermode processes – Mail Security • This is OK • Several daemon threads of ctipd (IP Reputation) and ctasd (Content Anti Spam). Up according to configuration. • Several (configurable number) of security servers (in.emaild.*) • in.msd should always be up (except after cpstop…) • Performs additional tasks (e.g. periodic Anti Spam license verification with the user center) • in.emaild.* processes begin to run only upon connection which matches ASPAM usermode activities or Mail AV. They stay up until cpstop. • If an SMTP resource is defined and matched, in.asmtpd and mdq processes will be up instead of in.emaild.smtp. ps -A

16. Debugging the usermode • Use: [Expert@cpmodule]# ps -A | grep in.emaild.smtp 2849 ? 00:00:01 in.emaild.smtp 2879 ? 00:00:01 in.emaild.smtp [Expert@cpmodule]# fw debug 2849 on TDERROR_ALL_MAIL_SECURITY=5 [Expert@cpmodule]# fw debug 2879 on TDERROR_ALL_MAIL_SECURITY=5 • The debug output will be in the file:$FWDIR/log/emaild.smtp.elg (emaild.pop3.elg) • Will contain a lot of information • Search for the words “error” or “fail” to find problems • Can also search for email addresses

17. Debugging the usermode

18. Tracker logs • The control analysis string is actually CT’s ref-id for the pattern extracted from the email. • In case of FP this is sent to them. • Email Session ID is important • “Follow Email Session ID…” Tracker feature • Flags by country (source IP checked locally)

19. Additional content \ enhancements • Above is an example of a spam session • Non Spam tracking was enabled • Mail was rejected because it had spam content • UTM-1 Clusters Logs: • Cluster Logs are send to primary member by default • Send to Secondary when Primary is unreachable

20. Troubleshooting • All Content Anti Spam logs are bypass logs • Internet connectivity problem? Check DNS is working… • In this case IP reputation logs should also be bypass • A CT daemon problem, kill the ctasd.bin processes. They will go back up within one minute. • Use: kill -9 `pgrep ctasd` • No Anti Spam logs appear • Make sure the directional match / IP matched is configured correctly • Verify Anti Spam is enabled on the gateway properties • Use Non Spam tracking to see the more logs

21. Monitoring and Reporting • Express reports: • Anti Spam • AV + URL Filtering support • Express Reports License

22. Additional Issues • When email is rejected (all 6 dimensions) • Receiver will not get the email • Sender will get an NDR (Non-Delivery report) containing the Email Session ID and the headers of the original email. • The hop before UTM-1 is responsible for generating the NDR • This is according to the SMTP RFC • When using SMTP Resource the FW will generate and deliver the NDR. • Bridge Mode – Fully supported, except cluster mode • Same limitations as FW-1 Bridge Mode

23. Q&A ?