1 / 44

Academic Advisory Committee of Computing Management Board June 11, 2003 UTORdial

Academic Advisory Committee of Computing Management Board June 11, 2003 UTORdial UTORmail Re-Architecture Spam Update Eugene Siciunas Computing & Networking Services. UTORdial Costs. UTORdial Revenue. 2002/03 2001/02. UTORdial Surplus. UTORdial History.

reuel
Download Presentation

Academic Advisory Committee of Computing Management Board June 11, 2003 UTORdial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Academic Advisory Committee of Computing Management Board June 11, 2003 UTORdial UTORmail Re-Architecture Spam Update Eugene Siciunas Computing & Networking Services

  2. UTORdial Costs

  3. UTORdial Revenue 2002/03 2001/02

  4. UTORdial Surplus

  5. UTORdialHistory

  6. Pause for Discussion UTORmail Re-Architecture next

  7. UTORmail Re-Architecture • UTORmail has scaled from 1995 one small server to today 13 servers. From 2 GB to 3/4 TB. • We asked whether existing architecture is suitable to meet future demands. • Some of the issues addressed follow ...

  8. UTORmail Re-Architecture: servers • Examined single server vs multiple servers. • Concluded existing multi-server is effective. • Concluded lack of infrastructure for many more servers so will consolidate to fewer bigger ones.

  9. UTORmail Re-Architecture: Linux • Currently using Sun servers:examined Sun versus Linux Intel servers • Tested Sun & Intel models considering buying. • After heavy discounting Sun was 2x the price • Linux was 2.5x faster encrypting, up to 12x faster in some post office functions. • Linux Intel wins.

  10. UTORmail Re-Architecture: Storage • Storage use doubles every year • High performance requirement. • Management nightmare: • A disk failure a month • Only protection is nightly backups • A restore from backups could cost hours of down time and loss of a day worth of new messages • Difficult to balance load across disks • Uneven distribution causing problems for backups

  11. De-couple Server Upgrades from Disks for Storing Mail • UTORmail server upgrades need to happen before September. • Requirements and proposals for disk storage for storing mail will be brought forward at another time. (SAN, disk array, JBOD, RAID, etc.) • Disk storage solution cannot be put inbefore September

  12. IMAP SMTP UTORmail Architecture Message Stores External SMTP Netscape Messenger Outlook Express Internal SMTP Webmail

  13. Message Store: Normal Growth • In past added 1-2 servers per year normal growth • Increase in number of customers • Increase in transactions per customer • Increase in size of messages • Increase in use of http://my.utoronto.ca webmailwhich pounds the message stores • Regular Alarms now: trivial transactions taking too long (>10seconds). Regularly peaks of 15-20 seconds. Maximum peaks up to 45 seconds.

  14. Message Store: UTORauth • Current message store connections in clear:spy can see everything including passwords. Must encrypt for UTORauth. • Encryption (SSL is standard) requires huge amounts of CPU power • Existing Message Store CPUs too slow. • Looked at alternatives: encryption co-processor vs new servers, latter more cost-effective • Conclusion: Need to upgrade all message stores

  15. Message Store: not disks • We propose to upgrade the server but leave the existing disks (will copy all message files from Solaris to Linux format using same disks) • Will bring forth proposal for alternative to existing disks at a later point in time.

  16. Message Store: Inaction Consequence • Can't accommodate normal growth in accounts(let alone double cohort) • Passwords flow in clear: forget UTORauth • Existing customers will see >1minute delays when clicking to view messages

  17. External SMTP: Normal Growth • Have had 2 old slow servers for years;this year added a 3rd to keep up with load. • This year 60% growth in message numbers compared to 30% in previous years. • Servers no longer keeping up: regular periods when queues build resulting in long delays. • Spam episodes result in multi-hour delays because servers so close to the edge. • Conclusion: Need to add more horsepower.

  18. External SMTP: Inaction Consequence • By September will not be able to keep up with normal number of messages from other post offices. • Queues which keep growing in size. • Mail indefinitely delayed. • Complete meltdown of service(messages arrive faster than can be delivered.)

  19. Number of Messages per Day* • Messages 339,000 12,000MBRecipients 429,000 18,000MBInternal SMTP Server UTORmail to UTORmail 96,000 5,000MB UTORmail to external 109,000 6,000MBExternal SMTP Server external to UTORmail 223,000 7,000MB* Stats for May 27, 2003

  20. Internal SMTP: Normal Growth • Have used the same old slow 2 servers for years. • Currently at capacity. • Before September, must increase number of Internal SMTP servers; intend to use other UTORmail servers coming out of production. • May need to put in faster SMTP serverslater in the year.

  21. Internal SMTP: Inaction Consequence • By September, long delays when the “send” button is pushed by customers. • By mid-fall, delays in delivering messages. • Depending on continued internal growth in usage, possible meltdown (messages arriving faster than they can be delivered, resulting in service collapse.)

  22. Webmail • Single Sun (v880) server with 4 CPUs builtin • Predicted would need one board (2 CPUs) mid-last year but was able to grow with existing CPUs • Now need this board (2 CPUs) before September

  23. Summary TypeNumber Message Store Servers [1] 5 External SMTP Servers 2 Internal SMTP Servers 2 [2] Webmail CPU boards 1 (2 CPUs) [1] Message Store Disks stay asis for September; will bring forth proposal later. [2] Stop gap temporarily use removed UTORmail servers.

  24. UTORmail Re-Architecture Costs • 2 X External SMTP $17,720 • 2 X Standby/Deployment $16,084 • 5 X IMAP Message Stores $46,398 • 2 X Internal SMTP $17,720 • WebMail CPUs (2 chips) $26,018 • SCSI Cards (for IMAP Servers) $3,816 • Racks for servers $4,000 • Disks for IMAP $12,000 • Total, including Taxes $158,578

  25. Pause for Discussion Spam Update Next

  26. Spam Update • TechKnowFile.03 Opportunity for feedback • Spam problem seems more pervasive • Selected four areas to attempt quantification • Dentistry, Law, SIS, AMS to be surveyed • Asked to measure spam for a week to quantify and examine impact on users

  27. Dentistry • Users reported spending 5-10 min/day on Spam • Worse for those in field using slow WebMail • 10 Min becomes 30min/day in Internet Café at $$ • Disincentive to publish contact info for collaboration, as it attracts Spam • May impair growth of digital publishing

  28. Law

  29. SIS • 28 Staff and Help Line monitored for a week • Spam ranged from low of 1 to high of 203/wk • Daily Average ranged from 4.7 to12 • Average for all was 38.6/person/week • See scatter chart

  30. AMS • G.Kemp monitored all his mail for May • 1,237 emails received in Total • 584, or >47% were Spam • If deletion took10 secs each, amounts to 97 min. • Some have to be opened to verify Spammness, so even more time consumed • AMS staff results to be presented

  31. Nature of Complaints • Increasing frustration with junk: Viagra, cheap mortgages, get rich quick schemes, free digital cable, penis/breast enlargement pills • Sexually explicit offers becoming much more offensive • Some complain it amounts to sexual harassement • Could lead to unsafe workplace problems • Growing ANGER, not just nuisance

  32. More Complaints • 150 Span messages after 4 day absence!!! • “I don’t care if you’re too stupid to fix it, just let me KNOW if you are!!” • Some offices expect mail from outside strangers, so many messages must be opened in order to verify nature • Now some viruses generate Spam • Increasing # users are demanding action

  33. Costs to Remedy • Spam Assassin to mark suspected Spam for user to file separately or just delete if trusted • PureMessage License cost is C$5K/cpu oto • Anti-Spam Def’ns + Support = C$3.2k/yr/cpu • CPUs approx $8k oto,so $13Koto + $3.2K/yr/cpu • 6 CPU = $78K + $19.2K/yr + 1/4 FTE ($20K)

  34. Spam Filtering • Proposed costs for UTORmail load • Dept’l mail systems doing own thing • No knowledge of dept’l mail systems or load • Technically feasible to funnel all mail through Spam filter and then back to dept’l systems • Deploy on UTORmail, incrementally add Dept’l until need more processors

  35. PureMessage Testimonials • U.Wash has 120k mailboxes, 800k msgs/day • Run 13 servers, 5 out, rest anti-span and virus • 40% of inbound mail is Spam • Tag messages as Spam • Stanford has 22k user accounts, 500K msgs/day • Run 7 Dual-cpu SUN Netra 20 servers, Solaris • Spam not blocked but tagged

  36. PureMessage Testimonials • Alabama ISP has 11k Users, 1M msgs/day • Claim “Incredible Accuracy” • Cuts 40% of their traffic by blocking • IndianaU CS get 14K emails/day, 3.5K Spam • Run PM on 1 Dual 1.4GHz Athlon, 1GB RAM • Identify close to 100% of Spam

  37. Pause for Discussion (Also end of Presentation)

More Related