using greylisting and spamassassin
Download
Skip this Video
Download Presentation
Spam Reduction Techniques

Loading in 2 Seconds...

play fullscreen
1 / 19

Spam Reduction Techniques - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on

Using greylisting and SpamAssassin. Spam Reduction Techniques. The problem. The vast majority of email today is Spam Some current statistics indicate over 90% of email Spam This matches my experience. Botnets. Vast majority of Spam comes from Botnets compromised home PCs

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Spam Reduction Techniques' - bat


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the problem
The problem
  • The vast majority of email today is Spam
  • Some current statistics indicate over 90% of email Spam
  • This matches my experience
botnets
Botnets
  • Vast majority of Spam comes from Botnets
  • compromised home PCs
  • hundreds of thousands to millions, or even tens of millions of machines in a heard
  • Controlled by the owner of the heard via a centralised command and control structure
  • Typically don\'t have a “real” smtp server to actually send the email
spam reduction with greylisting and spamassassin
Spam Reduction with Greylisting and SpamAssassin
  • Currently > 99% effective (closer to 99.8%)‏
  • In a recent week, only 11 out of 8,000 Spam messages made it through to the end user without being stopped or marked.
slide5
Spam statistics as of: 16/09/2007

Total spam: 5459

Total greylisted: 4457(90.8%)‏

Total emails accepted (both spam and legitimate): 451 (9.2)%

Total identified spam through to end users: 1002 (20.4%)‏

Emails greylist_delayed: 58 (1.2%), marked as spam 57 (96.6%), NOT marked as spam 2 (3.4%)‏

emails via backup mx: 991 (20.2%), marked as spam 944 (95.2%), NOT marked as spam 48 (4.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.0%. 50 out of 4908 not marked as spam

Spam statistics as of: 23/09/2007

Total spam: 5167

Total greylisted: 4928(90.8%)‏

Total emails accepted (both spam and legitimate): 499 (9.2)%

Total identified spam through to end users: 239 (4.4%)‏

Emails greylist_delayed: 99 (1.8%), marked as spam 98 (97.0%), NOT marked as spam 3 (3.0%)‏

emails via backup mx: 151 (2.8%), marked as spam 138 (90.2%), NOT marked as spam 15 (9.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.7%. 18 out of 5427 not marked as spam

Spam statistics as of: 30/09/2007

Total spam: 6216

Total greylisted: 5950(91.2%)‏

Total emails accepted (both spam and legitimate): 573 (8.8)%

Total identified spam through to end users: 266 (4.1%)‏

Emails greylist_delayed: 141 (2.2%), marked as spam 135 (95.1%), NOT marked as spam 7 (4.9%)‏

emails via backup mx: 151 (2.3%), marked as spam 128 (84.2%), NOT marked as spam 24 (15.8%)‏

Effectiveness of Greylisting / SpamAssassin: 99.5%. 31 out of 6523 not marked as spam

Spam statistics as of: 07/10/2007

Total spam: 7901

Total greylisted: 7712(93.0%)‏

Total emails accepted (both spam and legitimate): 581 (7.0)%

Total identified spam through to end users: 189 (2.3%)‏

Emails greylist_delayed: 135 (1.6%), marked as spam 134 (97.8%), NOT marked as spam 3 (2.2%)‏

emails via backup mx: 62 (0.7%), marked as spam 55 (87.3%), NOT marked as spam 8 (12.7%)‏

Effectiveness of Greylisting / SpamAssassin: 99.8%. 11 out of 7901 not marked as spam

  • Greylisting removes > 90% of incomming Spam
  • SpamAssassin catches > 90% of received spam
  • Total effectiveness > 99.5%
greylisting
Greylisting
  • Relies on Spammers not using a “proper” mail server. They just fire-and-forget
  • Give a temporary failure to any “suspect” messages. Spammers will not retry, but a mail server will
which messages to challenge
Which messages to challenge
  • Look at (all of):
    • From address
    • To Address
    • IP of sending machine
  • If not seen before:
    • give temporary failure
    • record this “tuple” + time
slide8

If seen before:

    • check if it is now past a “start time” (time + time to go live)‏
      • time to live is typically a parameter passed to greylisting server.
      • many recommend 60 minutes
      • I use 60 seconds
    • OK – let through
      • record the time
    • Not OK
      • reject again
  • Any subsequent communication is let straight through
potential issues
Potential issues
  • Some delay first time someone new contacts you
  • Small chance of non delivery of some messages.
    • non compliant mail servers
    • ISPs with rotary pool of mail servers may get continually greylisted
    • email from web forms that doesn\'t go through a real mail server
risk minimisation
Risk minimisation
  • Can have various white lists
    • add mail server details for all regular / potential contacts to a white list
      • these emails are coming from a real mail server, so we don\'t need to use this test on them.
      • grep you mail server logs to determine who does conatct you. eg:egrep "client=.*mail.*|client=.*mx.*|client=.*smtp.*" /var/log/maillog*| awk \'{print $7}\' | awk -F = \'{print $2}\' | awk -F [ \'{print $1}\' | sort | uniq -u
      • can use regex in these whitelists
examples of server whitelist
Examples of server whitelist

/^.*\.ebay\.com$/

/.*\.emailebay\.com$/

/^.*\.mx\.bigpond\.com$/

/^.*\.dell\.com\.au$/

/^.*\.mailguard\.com\.au$/

/^mailout.*\.pacific\.net\.au$/

/^mail-out.*\.netspace\.net\.au$/

/^mx.*\.phx\.paypal\.com$/

/^smtp.*\.bis\.ap\.blackberry\.com$/

/^.*\.server-mail\.com$/

/^vscan.*\.westnet\.com\.au$/

/^ihug-mail\.icp-qv1-irony?\.iinet\.net\.au$/

implementations
Implementations
  • Available for many popular mail servers including MS Exchange
spamassassin
SpamAssassin
  • Categorises email as either Spam or Ham (good stuff, not Spam), based on a number of tests
  • Each test may add to the overall score for this email
  • If the total score exceeds a (configurable) limit, it is marked as Spam
  • Highly configurable
    • personal limits, tests, scoring etc
tests
Tests
  • Tests to find words that look like viagra etc
  • Is the sender in a RBL
  • Does the sender match the SPF record
    • v=spf1 a mx mx:westnet.com.au include:westnet.com.au ~all
  • Does the body look like spam
  • The ratio of text to images
  • Bayesian analysis of the content
  • Many more tests
  • see: http://spamassassin.apache.org/tests_3_2_x.html for the full list
spam ham folders
Spam / Ham folders
  • can also set up folders containing Spam and Ham (non Spam) for SpamAssassin to learn from. As a large proportion of email is actually spam (if you are not using greylisting), doing this may not be a good idea, as eventually the Bayesian filter gets poisoned and everything ends up looking like spam.
implementations1
Implementations
  • Available for many popular mail servers including MS Exchange
    • Exchange implementations tend to be commercial offerings
greet pause
Greet - Pause
  • When the sender connects, delay the greeting
  • If the sender tries to continue the conversation, before the appropriate response, the conversation is stopped by the smtp server.
  • A “proper” smtp server will handle this, a Spam bot may just have a sequential script and fail this test.
  • About 10% of Spam can be eliminated this way
components in my system
Components (in my system)‏
  • Postfix mta (postfix-2.3.3-2)http://www.postfix.org
  • postgrey greylisting server (v 1.30)http://postgrey.schweikert.ch/
    • See also http://www.greylisting.org/
  • SpamAssassin (spamassassin-3.2.2-1.el5.rf)http://spamassasin.apache.org/
ad