Suremail notification overlay for email reliability
Download
1 / 32

SureMail Notification Overlay for Email Reliability - PowerPoint PPT Presentation


  • 271 Views
  • Updated On :
  • Presentation posted in: Internet / Web

SureMail Notification Overlay for Email Reliability Sharad Agarwal Venkat Padmanabhan Dilip A. Joseph 8 March 2006 Outline Email loss problem Design philosophy SureMail design SureMail robustness to security attacks SureMail implementation What is Email Loss?

Related searches for SureMail Notification Overlay for Email Reliability

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentationdownload

SureMail Notification Overlay for Email Reliability

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Suremail notification overlay for email reliability l.jpg

SureMailNotification Overlay for Email Reliability

Sharad Agarwal

Venkat Padmanabhan

Dilip A. Joseph

8 March 2006


Outline l.jpg

Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


What is email loss l.jpg

What is Email Loss?

  • Email loss : sent email not received

  • Silent email loss

    • Loss w/o notification (no bounceback / DSN)

  • Why?

    • Aggressive spam filters

      • 90% corp. emails thrown away (blacklist)

      • AOL’s strict whitelist rules (must send 100/day)

      • Bouncebacks contribute to spam

    • Complex mail architecture upgrades / failures

      • SMTP reliability is per hop, not end-to-end


How much email loss l.jpg

How Much Email Loss?

  • Even loss of 1 email / user / year is bad

    • If it’s an important email

  • To really measure loss

    • Monitor many users’ send & receive habits

    • Count how many sent emails not received

    • Count how many bouncebacks received

    • Difficult to find enough willing participants that email each other across multiple domains


Prior work l.jpg

Prior Work

  • “The State of the Email Address”

    • Afergan & Beverly, ACM CCR 01.2005

      • Rely on bouncebacks; similar to “dictionary” attack

      • 25% of tested domains send bouncebacks

      • 1 sender

      • 0.1% to 5% loss, across 1468 servers, 571 domains

  • “Email dependability”

    • Lang, UNSW B.E. thesis 11.2004

      • 40 accounts, 16 domains receive emails from 1 sender

      • Empty body, sequence number as subject

      • 0.69% silent loss


Our email loss study l.jpg

Our Email Loss Study

  • Methodology

    • Controller composes email, sends

    • Our code for SMTP sending

    • Outlook for receiving (both inbox & junk mail)

    • Parse sent and received emails into SQL DB

    • Match on {sender,receiver,subject,attachment}

    • Heuristics for parsing bouncebacks

  • Want

    • Many sending, receiving accounts

    • Real email content


Experiment details l.jpg

Experiment Details

  • Email accounts

    • 36 send, 42 receive

    • Junk filters off if possible

  • Email subject & body

    • Enron corpus subset

    • 1266 emails w/o spam

  • Email attachment

    • 70% no attachment

    • jpg,gif,ppt,doc,pdf,zip,htm

    • marketing,technical,funny


Email loss results l.jpg

Email Loss Results


Loss rates by account l.jpg

Loss Rates by Account

  • Loss rate 1.82% to 0.82%


Loss rates by attachment l.jpg

Loss Rates by Attachment

  • Nothing stands out


Loss rates by subject body l.jpg

Loss Rates by Subject/Body

  • ~50-250 emails sent per subject

  • Without 35% case : loss rate 1.82% to 1.79%


Summary of findings l.jpg

Summary of Findings

  • Email loss rates are high

    • 1.82% loss

    • 0.71% conservative silent loss ( 1 / 140 )

  • Difficult to disambiguate cause of loss

    • Difference between domains (filters or servers?)

    • No difference between mailboxes

    • No difference between attachments

    • Only 1 body had abnormally high loss


Outline13 l.jpg

Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


We found email loss now what l.jpg

We Found Email Loss; Now What?

  • Can try to fix email architecture, but

    • Hard to know exactly what is problem

    • Spam filters continually evolve; not perfect

    • Some architectures are very complicated

    • How many email systems are out there?

    • The current system mostly works


Fixing the architecture l.jpg

Fixing the Architecture

  • Improve email delivery infrastructure

    • more reliable servers

      • e.g., cluster-based (Porcupine [Saito ’00])

    • server-less systems

      • e.g., DHT-based (POST [Mislove ’03])

    • total switchover might be risky

  • “Smarter” spam filtering

    • moving target  mistakes inevitable

    • non-content-based filtering still needed to cope with spam load


Email notifications l.jpg

Email Notifications

  • DSN / bouncebacks

    • Most spam filters don’t generate DSN on drop

    • Bogus DSNs due to spam w/ bogus sender

    • Some MTAs block DSN for privacy

    • MTA crash may not generate DSN

    • No DSN for loss between MTA and MUA

  • MDN / read receipts

    • Expose private info (when read, when online)

    • Can help spammers


Notification design requirements l.jpg

Notification Design Requirements

  • Cause minimal MTA/MUA disruption

  • Cause minimal user disruption

  • Preserve asynchronous operation

  • Preserve user privacy

  • Preserve repudiability

  • Maintain spam and virus defenses

  • Minimize traffic overhead


Outline18 l.jpg

Outline

  • Email loss problem

  • Design philosophy

  • SureMail design

  • SureMail robustness to security attacks

  • SureMail implementation


Suremail design requirements l.jpg

SureMail Design Requirements

  • Cause minimal MTA/MUA disruption

    • No MTA modification; no Outlook modification

  • Cause minimal user disruption

    • User notified only on loss

  • Preserve asynchronous operation

  • Preserve user privacy

    • Only receiver is notified of loss

  • Preserve repudiability

    • No PKI / authentication

  • Maintain spam and virus defenses

    • Emails not modified

  • Minimize traffic overhead

    • 85 byte notification per email


  • Basic operation l.jpg

    Basic Operation

    • Sender S sends email to receiver R

      • S also posts notification to overlay

    • R periodically downloads new email

      • R also downloads notifications from overlay

    • Notification without matching email  loss

      • delay : median 26s, mean 276s, max 36.6 hrs


    Suremail overview l.jpg

    You’ve Lost Mail!

    H1(Mnew),

    H1(Mold),

    T,

    MAC([T,H1(Mnew)]

    ,H2(Mold))

    GetNotifications

    Request lost message

    Register

    Verify

    SureMail Overview

    Recipient R

    Sender S

    Dnot=H1(R)

    Dreg=H2(R)


    Suremail overview22 l.jpg

    SureMail Overview

    • Emails, MTAs, MUAs unmodified

    • Parallel notification overlay system

      • Decentralized; limited collusion

      • Agnostic to actual implementation

        • end-host-based (e.g., always-on user desktops)

        • infrastructure-based (e.g., “NX servers”)

    • Prevent notification snooping & spam

      • Email based registration

      • Reply based shared secret


    Email based registration l.jpg

    Email-Based Registration

    • Goal: prevent hijacking of R’s notifications

      • Only R can receive emails sent to R

      • Limited collusion among notification nodes

    • One-time operation for initial registration

      • R sends registration request to H2(R), H3(R)

      • H2(R), H3(R) email registration secrets to R

    • To retrieve notifications at H1(R)

      • R uses registration secrets with H1(R); H1(R) verifies with H2(R) H3(R), sends back notifications

      • Neither H1(R), H2(R), H3(R) can associate notifications with R, unless they collude


    Reply based shared secret l.jpg

    Reply-Based Shared Secret

    • Goal: prevent notification spoofing & spam

      • Only R & S know their email conversations

      • S rarely converses with spammers

    • Reply detection

      • S sends Mold to R, R replies with M’old

      • S uses H(Mold) to “prove” identity to R in future

    • Notification for Mnew from S to R

      • H1(Mnew),H1(Mold),T,MAC([T,H1(Mnew)],H2(Mold))

      • Only R can identify S

      • Shared secret can be continually refreshed


    Attacks defeated by design l.jpg

    Attacks Defeated by Design

    • X cannot retrieve H1(R) notifications

    • H1(R) cannot identify R

    • H2(R), H3(R) cannot see R’s notifications

      • If they don’t collude; can increase to 3 nodes

    • X, H(R) cannot identify S

    • X, H(R) cannot learn Mnew, Mold

    • X cannot annoy R with bogus notifications

    • X cannot masquerade post to H1(R) as S


    First time sender l.jpg

    First Time Sender

    • What if FTS email is lost?

    • FTS & spammer generally indistinguishable

    • But perhaps FTS knows I who knows R

      • Email networks have small world properties

      • I makes shared secret SI with all known parties

      • FTS sends email to R

        • Posts multiple notifications

        • One for every SI it has learned


    Other issues l.jpg

    Other Issues

    • Reply-detection:

      • “in-reply-to” header may not always help

      • indirect checks based on text similarity

    • Reducing overhead:

      • post notifications only for “important” emails

      • delay posting in hope of receiving implicit ACK (reply) or NACK (bounce-back)

    • Mobility:

      • reply-based shared secret can be regenerated

      • web-mail

    • Can support mailing lists


    Outline28 l.jpg

    Outline

    • Email loss problem

    • Design philosophy

    • SureMail design

    • SureMail robustness to security attacks

    • SureMail implementation


    Suremail implementation l.jpg

    SureMail Implementation

    • Reply detection heuristic for shared secret

    • Notification service

      • Centralized server running

      • Chord based DHT running

    • Notification posting, retrieving

      • Grab in/out bound email via Outlook MAPI call

        • No modification to Outlook binaries

      • XML notification put/get commands

      • Simple Win32 GUI


    Suremail gui l.jpg

    Lost!

    Not lost

    SureMail GUI

    • Client UI will see emails, will post & retrieve notifications

    • E.g. running on two machines

      netprofa@microsoft.com and netprofa@gmail.com


    Notification results l.jpg

    Notification Results


    Summary l.jpg

    Summary

    • Email does get lost!

      • ~40 accounts, 158000 emails, 0.71%-0.91% silent loss

    • SureMail

      • Client based – unmodified email, servers, clients; no PKI

      • User intervention only on lost email

      • Keeps repudiability, privacy, asynchronous, spam & virus defense

    • Separate notification overlay robust

      • Simple, small message format

      • No virus, malware, spam filters needed

      • Provides failure independence

    • Status

      • ACM Hotnets 05; ACM Sigcomm 06 submission

      • Prototype implementation


  • Login