Suremail notification overlay for email reliability
Download
1 / 32

SureMail - PowerPoint PPT Presentation


  • 282 Views
  • Updated On :

SureMail Notification Overlay for Email Reliability Sharad Agarwal Venkat Padmanabhan Dilip A. Joseph 8 March 2006 Outline Email loss problem Design philosophy SureMail design SureMail robustness to security attacks SureMail implementation What is Email Loss?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'SureMail' - Jimmy


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Suremail notification overlay for email reliability l.jpg

SureMailNotification Overlay for Email Reliability

Sharad Agarwal

Venkat Padmanabhan

Dilip A. Joseph

8 March 2006


Outline l.jpg
Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


What is email loss l.jpg
What is Email Loss?

  • Email loss : sent email not received

  • Silent email loss

    • Loss w/o notification (no bounceback / DSN)

  • Why?

    • Aggressive spam filters

      • 90% corp. emails thrown away (blacklist)

      • AOL’s strict whitelist rules (must send 100/day)

      • Bouncebacks contribute to spam

    • Complex mail architecture upgrades / failures

      • SMTP reliability is per hop, not end-to-end


How much email loss l.jpg
How Much Email Loss?

  • Even loss of 1 email / user / year is bad

    • If it’s an important email

  • To really measure loss

    • Monitor many users’ send & receive habits

    • Count how many sent emails not received

    • Count how many bouncebacks received

    • Difficult to find enough willing participants that email each other across multiple domains


Prior work l.jpg
Prior Work

  • “The State of the Email Address”

    • Afergan & Beverly, ACM CCR 01.2005

      • Rely on bouncebacks; similar to “dictionary” attack

      • 25% of tested domains send bouncebacks

      • 1 sender

      • 0.1% to 5% loss, across 1468 servers, 571 domains

  • “Email dependability”

    • Lang, UNSW B.E. thesis 11.2004

      • 40 accounts, 16 domains receive emails from 1 sender

      • Empty body, sequence number as subject

      • 0.69% silent loss


Our email loss study l.jpg
Our Email Loss Study

  • Methodology

    • Controller composes email, sends

    • Our code for SMTP sending

    • Outlook for receiving (both inbox & junk mail)

    • Parse sent and received emails into SQL DB

    • Match on {sender,receiver,subject,attachment}

    • Heuristics for parsing bouncebacks

  • Want

    • Many sending, receiving accounts

    • Real email content


Experiment details l.jpg
Experiment Details

  • Email accounts

    • 36 send, 42 receive

    • Junk filters off if possible

  • Email subject & body

    • Enron corpus subset

    • 1266 emails w/o spam

  • Email attachment

    • 70% no attachment

    • jpg,gif,ppt,doc,pdf,zip,htm

    • marketing,technical,funny



Loss rates by account l.jpg
Loss Rates by Account

  • Loss rate 1.82% to 0.82%


Loss rates by attachment l.jpg
Loss Rates by Attachment

  • Nothing stands out


Loss rates by subject body l.jpg
Loss Rates by Subject/Body

  • ~50-250 emails sent per subject

  • Without 35% case : loss rate 1.82% to 1.79%


Summary of findings l.jpg
Summary of Findings

  • Email loss rates are high

    • 1.82% loss

    • 0.71% conservative silent loss ( 1 / 140 )

  • Difficult to disambiguate cause of loss

    • Difference between domains (filters or servers?)

    • No difference between mailboxes

    • No difference between attachments

    • Only 1 body had abnormally high loss


Outline13 l.jpg
Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


We found email loss now what l.jpg
We Found Email Loss; Now What?

  • Can try to fix email architecture, but

    • Hard to know exactly what is problem

    • Spam filters continually evolve; not perfect

    • Some architectures are very complicated

    • How many email systems are out there?

    • The current system mostly works


Fixing the architecture l.jpg
Fixing the Architecture

  • Improve email delivery infrastructure

    • more reliable servers

      • e.g., cluster-based (Porcupine [Saito ’00])

    • server-less systems

      • e.g., DHT-based (POST [Mislove ’03])

    • total switchover might be risky

  • “Smarter” spam filtering

    • moving target  mistakes inevitable

    • non-content-based filtering still needed to cope with spam load


Email notifications l.jpg
Email Notifications

  • DSN / bouncebacks

    • Most spam filters don’t generate DSN on drop

    • Bogus DSNs due to spam w/ bogus sender

    • Some MTAs block DSN for privacy

    • MTA crash may not generate DSN

    • No DSN for loss between MTA and MUA

  • MDN / read receipts

    • Expose private info (when read, when online)

    • Can help spammers


Notification design requirements l.jpg
Notification Design Requirements

  • Cause minimal MTA/MUA disruption

  • Cause minimal user disruption

  • Preserve asynchronous operation

  • Preserve user privacy

  • Preserve repudiability

  • Maintain spam and virus defenses

  • Minimize traffic overhead


Outline18 l.jpg
Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


Suremail design requirements l.jpg
SureMail Design Requirements

  • Cause minimal MTA/MUA disruption

    • No MTA modification; no Outlook modification

  • Cause minimal user disruption

    • User notified only on loss

  • Preserve asynchronous operation

  • Preserve user privacy

    • Only receiver is notified of loss

  • Preserve repudiability

    • No PKI / authentication

  • Maintain spam and virus defenses

    • Emails not modified

  • Minimize traffic overhead

    • 85 byte notification per email


  • Basic operation l.jpg
    Basic Operation

    • Sender S sends email to receiver R

      • S also posts notification to overlay

    • R periodically downloads new email

      • R also downloads notifications from overlay

    • Notification without matching email  loss

      • delay : median 26s, mean 276s, max 36.6 hrs


    Suremail overview l.jpg

    You’ve Lost Mail!

    H1(Mnew),

    H1(Mold),

    T,

    MAC([T,H1(Mnew)]

    ,H2(Mold))

    GetNotifications

    Request lost message

    Register

    Verify

    SureMail Overview

    Recipient R

    Sender S

    Dnot=H1(R)

    Dreg=H2(R)


    Suremail overview22 l.jpg
    SureMail Overview

    • Emails, MTAs, MUAs unmodified

    • Parallel notification overlay system

      • Decentralized; limited collusion

      • Agnostic to actual implementation

        • end-host-based (e.g., always-on user desktops)

        • infrastructure-based (e.g., “NX servers”)

    • Prevent notification snooping & spam

      • Email based registration

      • Reply based shared secret


    Email based registration l.jpg
    Email-Based Registration

    • Goal: prevent hijacking of R’s notifications

      • Only R can receive emails sent to R

      • Limited collusion among notification nodes

    • One-time operation for initial registration

      • R sends registration request to H2(R), H3(R)

      • H2(R), H3(R) email registration secrets to R

    • To retrieve notifications at H1(R)

      • R uses registration secrets with H1(R); H1(R) verifies with H2(R) H3(R), sends back notifications

      • Neither H1(R), H2(R), H3(R) can associate notifications with R, unless they collude


    Reply based shared secret l.jpg
    Reply-Based Shared Secret

    • Goal: prevent notification spoofing & spam

      • Only R & S know their email conversations

      • S rarely converses with spammers

    • Reply detection

      • S sends Mold to R, R replies with M’old

      • S uses H(Mold) to “prove” identity to R in future

    • Notification for Mnew from S to R

      • H1(Mnew),H1(Mold),T,MAC([T,H1(Mnew)],H2(Mold))

      • Only R can identify S

      • Shared secret can be continually refreshed


    Attacks defeated by design l.jpg
    Attacks Defeated by Design

    • X cannot retrieve H1(R) notifications

    • H1(R) cannot identify R

    • H2(R), H3(R) cannot see R’s notifications

      • If they don’t collude; can increase to 3 nodes

    • X, H(R) cannot identify S

    • X, H(R) cannot learn Mnew, Mold

    • X cannot annoy R with bogus notifications

    • X cannot masquerade post to H1(R) as S


    First time sender l.jpg
    First Time Sender

    • What if FTS email is lost?

    • FTS & spammer generally indistinguishable

    • But perhaps FTS knows I who knows R

      • Email networks have small world properties

      • I makes shared secret SI with all known parties

      • FTS sends email to R

        • Posts multiple notifications

        • One for every SI it has learned


    Other issues l.jpg
    Other Issues

    • Reply-detection:

      • “in-reply-to” header may not always help

      • indirect checks based on text similarity

    • Reducing overhead:

      • post notifications only for “important” emails

      • delay posting in hope of receiving implicit ACK (reply) or NACK (bounce-back)

    • Mobility:

      • reply-based shared secret can be regenerated

      • web-mail

    • Can support mailing lists


    Outline28 l.jpg
    Outline

    • Email loss problem

    • Design philosophy

    • SureMail design

    • SureMail robustness to security attacks

    • SureMail implementation


    Suremail implementation l.jpg
    SureMail Implementation

    • Reply detection heuristic for shared secret

    • Notification service

      • Centralized server running

      • Chord based DHT running

    • Notification posting, retrieving

      • Grab in/out bound email via Outlook MAPI call

        • No modification to Outlook binaries

      • XML notification put/get commands

      • Simple Win32 GUI


    Suremail gui l.jpg

    Lost!

    Not lost

    SureMail GUI

    • Client UI will see emails, will post & retrieve notifications

    • E.g. running on two machines

      [email protected] and [email protected]



    Summary l.jpg
    Summary

    • Email does get lost!

      • ~40 accounts, 158000 emails, 0.71%-0.91% silent loss

    • SureMail

      • Client based – unmodified email, servers, clients; no PKI

      • User intervention only on lost email

      • Keeps repudiability, privacy, asynchronous, spam & virus defense

    • Separate notification overlay robust

      • Simple, small message format

      • No virus, malware, spam filters needed

      • Provides failure independence

    • Status

      • ACM Hotnets 05; ACM Sigcomm 06 submission

      • Prototype implementation


    ad