Design and Management of Email Service

1. Design and Management ofEmail Service Source :homepage.ntu.edu.tw/~jsc/2005-mail.ppt

2. Outline • Introduction to the architecture and operation of SMTP • Design of a suitable email system • Webmail solutions • Postfix and simple configuration samples • Spam and virus filtering • Conclusion

3. Overview • Electronic mail service has already evolved into one of the major Internet applications. • It is not only fundamental, but also a must. • Users may become impatient when mails were delayed, not to mention failed to access their emails. • Imagine we meet the situation of power failure or cut of water supply

4. Architecture of a Simple Mail System • Consists of the following components • MTA - Mail transfer agent • Sending and forwarding emails • Server end • MDA - Mail delivery agent • Delivering emails to recipients’ mailbox • Server end • Pop3/Imap4 Daemons • For users to download their mailboxs • Server end • MUA - Mail user agent • Reading and composing emails • Client end

5. Architecture of a Simple Mail System

6. Protocols Used for Mail System • Protocols • For computer programs to communicate with each other • Similar to languages that human beings speak • SMTP • Simple Mail Transfer Protocol • Too simple to provide any “advanced features” • Authentication • Authorization • POP3 • Post Office Protocol version 3 • Simple • IMAP4 • Internet Message Access Protocol version 4 • Fully compatible with internet message standards, e.g. MIME. • Allow messages to be accessed from more than one computer. • Provide support for online, offline, and disconnected modes. • Multiple and share folders.

7. Mail Forwarding Between Servers

8. How to Find the Way to the Destination? • How do we find the way tojsc@ntu.edu.tw?

9. DNS: The Key to All Internet Services • Query DNS server by the address part of email address.(jsc@ntu.edu.tw) • Query for MX records • Query for A records • Give up! • Not required to exist a real machine hosting the address.(ntu.edu.tw) • Adding MX records to improve query performance and to reduce unnecessary query overhead.

10. DNS: Query Results • Precedence value of MX records • Lower valueswith higher priority

11. DNS: Query Results • When MX record is absent, second query is required to issue.

12. Features Required By Modern Mail Systems • This is what end users care. • Ease of use interface • IMAP4 • Webmail • Security • Authentication • Encrypted transmission • Spam and virus filtering • The only effective way to prevent from virus infection is to remind and force your users to install antivirus softwares. • Stop using M$ systems is alternative solution.

13. Make the Right Decision For the Promise of Performance • MTA • Postfix • Our suggestion! • Sendmail • Not scalable! • Lack of new features. • Not efficiency in resource management. • Rich of documents and tutorials • Qmail • Termination of development. • Exim • Immature. • M$ Exchange • Seeking trouble for yourself? • POP3/IMAP4 • UW-IMAP • Courier-IMAP • Supports Maildir format. • Dovecot • Supports both mbox and Maildir format. • Still under development. • Cyrus • Proprietary spool format. • Not easy to convert between other formats.

14. Webmail: The Trend of Browser Based Mail Reader • Your determination depends on the system architecture. • IMAP4 based • Access the mailbox via IMAP4 protocol locally or remotely. • Flexible, scalable • Speed may be limited by protocol overhead and network transmission. • Horde IMP project • Locally direct access • Access the mailbox locally, proprietary protocol. • Could only run on local mail storage server. • NFS could be applied when deploying multiple servers • Extremely fast!! iff system not overloaded. • Not scale well when you want to expand. • Openwebmail • Excellent project developed by NCKU, Taiwan • Marvelous support to Chinese encodings • Only support mbox format currently

15. Webmail Architecture – IMAP4 based

16. Webmail Architecture – Locally Direct Access

17. Postfix: The Modern and Advance MTA • http://www.postfix.org • Developed by Wietse Venema • Features • Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases • Human readable configuration file • Multiple small programs with limited execution privilege • Multiple queues to avoid block of normal mails • Backend database lookup table supported • Better support for virtual host • Virtual user system supported • Content filtering interface • Simple spam check/block mechanism

18. Postfix - the Big Picture • http://www.postfix.org/big-picture.html • Yellow ellipsoids are mail programs. • Yellow boxes are mail queues or files. • Blue boxes are lookup tables.

19. Postfix Installation • Debian Linux • apt-get install postfix-tls libsasl7 libsasl-modules-plain courier-imap • Redhat/Fedora Linux • rpm -ivh postfix-2.2.x.i386.rpm • rpm -ivh cyrus-sasl-2.1.21.i386.rpm • BSD • Use the ports system • /usr/ports/mail/postfix • /usr/ports/security/cyrus-sasl2 • /usr/ports/mail/courier-imap

20. Post Installation • Postfix Configuration • master.cf • Similar to inetd.conf • Control the behavior of small programs • In contrast against sendmail, with one binary and one config file • main.cf • The main configuration of the mail system • In general cases, no modification is required for a simple setup. • Commands to control the postfix program • postfix start • postfix stop • postfix reload

21. Test For Receiving Email • Send an email to yourself. • Check system logs • /var/log/maillog • An simple test from jason@dolphin.cc.ntu.edu.tw to jason@freebsd.csie.nctu.edu.tw

22. myhostname = mail.cc.ntu.edu.tw mydomain = cc.ntu.edu.tw myorigin = $mydomain mydestination = $myhostname, localhost.$mydomain On a mail domain gateway,you should also include $mydomain. mynetworks_style = host relay_domains = $mydestination home_mailbox = Maildir/ message_size_limit = 20971520 smtpd_recipient_limit = 1000 Frequently Used Configuration Options In main.cf

23. SMTP/Authentication • Traditional mail servers could only determine their relay policy based on connected ip addresses. • Insufficient in face of the modern network • DHCP, NAT • ADSL and dialup users • Roaming users • Mobile, wireless • SMTP Authentication • Username/password authentication for each mail delivery. • Reduce the chance to relay spam mails for anonymous.

24. Email Headers • Envelope header • Negotiated between mail servers • Typical tricks that SPAM mail plays • Why do I receive mails that are not destined to me? • Content Text • Mail servers will not examine the content in general cases.

25. Real World Snail Mail Example • Postman could only deliver the mail by envelope information. • Sender address is not verified when email delivery. • Trick of SPAM mails. • Content may be irrelevant or conflict with envelope.

26. Virus and Spam Mail • Virus mail • Contain virus infected attachment • How do we determine if the mail is virus infected? • Simple and easy. Base on the inspection and judgment of antivirus software. • Spam mail • UCE/UBE • Unsolicited Commercial Email • Unsolicited Bulk Email • How do we determine if the mail is a Spam mail? • Difficult. It’s not a yes/no problem. • One person’s meat may be another’s poison.

27. Issues When Dealing With SPAM • Do not block or discard users’ mail unless you have the authorization and delegation. • The only safe and acceptable assistant you can provide is to score and tag each mail and let your users to do the rest themselves. • To warn or not to warn the senders/recipients is a dilemma! • Virus and spam senders nowadays always fake the from address. • System notification mails may be another kind of SPAM. The result will be blamed by your users and blocked by other mail servers.

28. Greylisting • Recently proposed (2003) method to block spam and virus mails. • RFC requires MTA to be capable to support queue and retry of temporarily undeliverable mails. • Spam and virus always fire and forget. • For efficiency concern, usually they do not retry. • How it works? • In brief, block every delivery at its first try. • For each mail delivery, check if the the triplet(source ip, sender, recipient) was seen recently. • If yes, pass the mail. • If not, reject the delivery with “450 Please retry later.”

29. Greylisting • Drawback • Mail delivery will suffer a delay ranging from several minutes to hours. • Depending on the remote MTA configuration. • Risk • Sites that do not retry their mail delivery. • Some bank notifications were observed to try only once. • Mail sending programs that process the SMTP transaction themselves without calling the “sendmail” MTA interface. • Suggestion • Whitelisted known hosts to prevent from delay and loss of mail.

30. Greylisting • Benefit • Effectively reduce the volume of spam and virus. • Effectively reduce both cpu and disk load of servers.

31. Softwares To Block/Filter Spam and Virus • Postfix • Content filter interface • Postgrey • Greylisting policy server • Amavisd-new • http://www.ijs.si/software/amavisd • http://www.ijs.si/software/amavisd/README.postfix • Clam Antivirus • http://www.clamav.net • A GPL virus scanner • SpamAssassin • http://spamassassin.apache.org

32. Software Architecture for Filtering

33. Amavisd-new • A high performance interface between MTA and content checkers. • Calling external antivirus programs to do virus scanning. • Calling external spamassassin program to do spam level determination. • CPU intensive workloads. • Can be flexibly configured to pass, discard, or quarantine mails based on user defined policy. • Pass spam mails with score > 10 with subject prepended the *** SPAM *** keyword. • Quarantine spam mails with score > 20. • Discard spam mails with score > 30. • Quarantine virus mails.

34. Spamassassin • Spam level scoring software. • Rich set of tests to identify various spam signatures. • Keywords, bad headers, encodings • Use bayesian analysis to help scoring. • Training the bayesian database using know spam and ham mails. • Default to enable the auto-learn feature. • Calling external programs to check if the mail was a known spam. • Use hash of mail content as the query key. • Razor, DCC, Pyzor.

35. Spamassassin • RBL(realtime black list) look up based on sender ip address. • RBL may contains too many ill-administrated sites. • Use the result as an addition of spam score. • Do not block remote sites depend solely on RBL. • SURBL(Spam URI realtime black list) look up based on the URIs within the content of mail. • Spammers may keep changing their sending IP addresses. • The URIs in the content may be the final destination the advertisement want people to visit.

36. Make Use of the Filter Rules on your MUA • Outlook Express • Can only filter based on limited headers. • Subject, Sender, etc. • Configure to move mails having subjects beginning with *** SPAM *** to an SPAM folder. • Outlook, Netscape, Thunderbird • You can filter mails based on any headers. • Determine your own spam level threshold referring the X-Spam-Level: ******

37. Conclusion • Make use of public domain, open source software to construct your own mail system. • Official web sites, mailing lists, and google are always your best friends. • Feedback and contribute as possible as you can.

38. Reference : • homepage.ntu.edu.tw/~jsc/2005-mail.ppt