Using greylisting and spamassassin
This presentation is the property of its rightful owner.
Sponsored Links
1 / 19

Spam Reduction Techniques PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on
  • Presentation posted in: General

Using greylisting and SpamAssassin. Spam Reduction Techniques. The problem. The vast majority of email today is Spam Some current statistics indicate over 90% of email Spam This matches my experience. Botnets. Vast majority of Spam comes from Botnets compromised home PCs

Download Presentation

Spam Reduction Techniques

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Using greylisting and spamassassin

Using greylisting and SpamAssassin

Spam Reduction Techniques


The problem

The problem

  • The vast majority of email today is Spam

  • Some current statistics indicate over 90% of email Spam

  • This matches my experience


Botnets

Botnets

  • Vast majority of Spam comes from Botnets

  • compromised home PCs

  • hundreds of thousands to millions, or even tens of millions of machines in a heard

  • Controlled by the owner of the heard via a centralised command and control structure

  • Typically don't have a “real” smtp server to actually send the email


Spam reduction with greylisting and spamassassin

Spam Reduction with Greylisting and SpamAssassin

  • Currently > 99% effective (closer to 99.8%)‏

  • In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.


Spam reduction techniques

Spam statistics as of: 16/09/2007

Total spam: 5459

Total greylisted: 4457(90.8%)‏

Total emails accepted (both spam and legitimate): 451 (9.2)%

Total identified spam through to end users: 1002 (20.4%)‏

Emails greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)‏

emails via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam

Spam statistics as of: 23/09/2007

Total spam: 5167

Total greylisted: 4928(90.8%)‏

Total emails accepted (both spam and legitimate): 499 (9.2)%

Total identified spam through to end users: 239 (4.4%)‏

Emails greylist_delayed: 99 (1.8%), marked as spam 98 (97.0%), NOT marked as spam 3 (3.0%)‏

emails via backup mx: 151 (2.8%), marked as spam 138 (90.2%), NOT marked as spam 15 (9.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.7%. 18 out of 5427 not marked as spam

Spam statistics as of: 30/09/2007

Total spam: 6216

Total greylisted: 5950(91.2%)‏

Total emails accepted (both spam and legitimate): 573 (8.8)%

Total identified spam through to end users: 266 (4.1%)‏

Emails greylist_delayed: 141 (2.2%), marked as spam 135 (95.1%), NOT marked as spam 7 (4.9%)‏

emails via backup mx: 151 (2.3%), marked as spam 128 (84.2%), NOT marked as spam 24 (15.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.5%. 31 out of 6523 not marked as spam

Spam statistics as of: 07/10/2007

Total spam: 7901

Total greylisted: 7712(93.0%)‏

Total emails accepted (both spam and legitimate): 581 (7.0)%

Total identified spam through to end users: 189 (2.3%)‏

Emails greylist_delayed: 135 (1.6%), marked as spam 134 (97.8%), NOT marked as spam 3 (2.2%)‏

emails via backup mx: 62 (0.7%), marked as spam 55 (87.3%), NOT marked as spam 8 (12.7%)‏

Effectiveness of Greylisting / SpamAssassin: 99.8%. 11 out of 7901 not marked as spam

  • Greylisting removes > 90% of incomming Spam

  • SpamAssassin catches > 90% of received spam

  • Total effectiveness > 99.5%


Greylisting

Greylisting

  • Relies on Spammers not using a “proper” mail server. They just fire-and-forget

  • Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will


Which messages to challenge

Which messages to challenge

  • Look at (all of):

    • From address

    • To Address

    • IP of sending machine

  • If not seen before:

    • give temporary failure

    • record this “tuple” + time


Spam reduction techniques

  • If seen before:

    • check if it is now past a “start time” (time + time to go live)‏

      • time to live is typically a parameter passed to greylisting server.

      • many recommend 60 minutes

      • I use 60 seconds

    • OK – let through

      • record the time

    • Not OK

      • reject again

  • Any subsequent communication is let straight through


Potential issues

Potential issues

  • Some delay first time someone new contacts you

  • Small chance of non delivery of some messages.

    • non compliant mail servers

    • ISPs with rotary pool of mail servers may get continually greylisted

    • email from web forms that doesn't go through a real mail server


Risk minimisation

Risk minimisation

  • Can have various white lists

    • add mail server details for all regular / potential contacts to a white list

      • these emails are coming from a real mail server, so we don't need to use this test on them.

      • grep you mail server logs to determine who does conatct you. eg:egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog*| awk '{print $7}' | awk -F = '{print $2}' | awk -F [ '{print $1}' | sort | uniq -u

      • can use regex in these whitelists


Examples of server whitelist

Examples of server whitelist

/^.*\.ebay\.com$/

/.*\.emailebay\.com$/

/^.*\.mx\.bigpond\.com$/

/^.*\.dell\.com\.au$/

/^.*\.mailguard\.com\.au$/

/^mailout.*\.pacific\.net\.au$/

/^mail-out.*\.netspace\.net\.au$/

/^mx.*\.phx\.paypal\.com$/

/^smtp.*\.bis\.ap\.blackberry\.com$/

/^.*\.server-mail\.com$/

/^vscan.*\.westnet\.com\.au$/

/^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/


Implementations

Implementations

  • Available for many popular mail servers including MS Exchange


Spamassassin

SpamAssassin

  • Categorises email as either Spam or Ham (good stuff, not Spam), based on a number of tests

  • Each test may add to the overall score for this email

  • If the total score exceeds a (configurable) limit, it is marked as Spam

  • Highly configurable

    • personal limits, tests, scoring etc


Tests

Tests

  • Tests to find words that look like viagra etc

  • Is the sender in a RBL

  • Does the sender match the SPF record

    • v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all

  • Does the body look like spam

  • The ratio of text to images

  • Bayesian analysis of the content

  • Many more tests

  • see: http://spamassassin.apache.org/tests_3_2_x.html for the full list


Spam ham folders

Spam / Ham folders

  • can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from. As a large proportion of email is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.


Implementations1

Implementations

  • Available for many popular mail servers including MS Exchange

    • Exchange implementations tend to be commercial offerings


Smtp conversation

SMTP Conversation


Greet pause

Greet - Pause

  • When the sender connects, delay the greeting

  • If the sender tries to continue the conversation, before the appropriate response, the conversation is stopped by the smtp server.

  • A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test.

  • About 10% of Spam can be eliminated this way


Components in my system

Components (in my system)‏

  • Postfix mta (postfix-2.3.3-2)http://www.postfix.org

  • postgrey greylisting server (v 1.30)http://postgrey.schweikert.ch/

    • See also http://www.greylisting.org/

  • SpamAssassin (spamassassin-3.2.2-1.el5.rf)http://spamassasin.apache.org/


  • Login