1 / 30

Creating an Email Archiving Service with Amazon S3

analiese
Download Presentation

Creating an Email Archiving Service with Amazon S3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Creating an Email Archiving Service with Amazon S3

    3. Who are we?

    4. Who are we? an email hosting company Blacksburg, VA founded in 1999 by two Virginia Tech students 54 employees 47,261 customers 476,130 email accounts 200 resellers

    5. Amazon Web Services (AWS)? Infrastructure: S3 = Simple Storage Service EC2 = Elastic Compute Cloud (virtual servers) SQS = Simple Queue Service E-Commerce & Data: ECS = E-commerce Service Historical Pricing Mechanical Turk Alexa

    6. Example Uses Data backup (S3) - Altexa, JungleDisk Content Delivery (S3) - Microsoft (MSDN Student Download program)? Live Application Data (S3) - 37signals Image Repository (S3) - SmugMug

    7. Example Uses Audio/Video Streaming (EC2+S3) – Jamglue, GigaVox Web Indexing (EC2 + S3) - Powerset Development (EC2 + S3) - UC Santa Barbara

    8. Our Use... Backing up Email Data

    9. Backing up the old way (tape)? Not smart - file system diffs - but... maildir filenames change - wasteful (needless I/O, duplicates = $$$)? Does not scale - 100s of servers = slow backups - needed more tape systems... egh Hard to add on to - we like to build stuff

    10. Possible solutions Commercial Storage Systems - e.g. Isilon, Netapp... $$$ Clustered File Systems - e.g. Lustre, Red Hat GFS Distributed Storage Systems - e.g. MogileFS, Hadoop Build it ourselves - again, we like to build stuff

    11. Possible solutions These all require a lot of development work, and we needed a solution quickly...

    12. Amazon S3 to the rescue In Spring 2006, Amazon released a new storage API: Put, Get, List, Delete

    13. Amazon S3 to the rescue

    14. Backing up the new way (S3)? Smart - because we wrote the client - maildir filename changes are OK - everything is incremental Scales - no longer our concern... Amazon's concern - all servers backup in parallel Cheap - old cost = $180K per year - new cost = $36K per year

    15. Backing up the new way (S3)? And look what else we can build now! - web-based restore tool for customers - custom retention policies - real-time archiving

    16. The backup client Two processes run nightly on each mail server: 1. Figure out what to back up - take a snapshot of file list per maildir - compare to previous night's snapshot - create list of new files 2. Send it to S3 - compress each file and send - 1 email = 1 file = 1 S3 object (for now) - send state information too (flags, status)

    17. The backup client

    18. The restore client Command line utility: Get list of backup snapshots for a given mailbox Get number of emails contained in a given snapshot Get list of folders contained in a given snapshot Restore a folder or set of folders

    19. Repetitive, manual work 3-4 restore requests per day Must charge $100 Only one-in-four customers go through with it

    20. Repetitive, manual work We want happy customers

    21. Web-based Restore Tool In customer control panel Full control (list backups, list emails/folders, restore)? Free

    22. Web-based Restore Tool Behind the scenes Control panel does not talk to S3 directly Calls our custom REST API hosted on EC2 servers EC2 servers talk to S3 Inserts restore jobs into a queue Mail servers pop restore jobs from the queue

    23. Deleting old data from S3 Distributed workload via EC2 and SQS Thousands of cleanup jobs inserted into SQS Many worker EC2 servers are spawned which pop jobs out of the SQS queue Job = set of mailboxes to cleanup Workers check retention policies and delete old data EC2 servers killed when work is complete

    24. AWS Customer Support Forums

    25. Things to watch out for Internal Server Errors are frequent, but manageable - Work around it Request Overhead can really slow things down - Batch your small files into larger S3 objects - Hit 'em too hard and they'll ask you to throttle - PUT and LIST requests are much slower than GET

    26. Batch your files

    27. No really... batch your files! Requests are expensive. New pricing will force everyone to play nice. effective June 1st, 2007: Requests (new): $0.01 per 1,000 PUT or LIST requests $0.01 per 10,000 GET and all other requests* * No charge for delete requests Storage: same as before ($0.15/GB-month)? Bandwidth: slightly cheaper (was $0.20/GB)?

    28. Things we're working on Batching files :)? Real-time archiving - Send data to S3 as it arrives using transaction log - Transaction log already used for live mirroring

    29. My Amazon Wishlist SLA ability to modify S3 meta data static IP option for EC2 load balancing for EC2 monitoring/management tool for EC2

    30. Please fill out your session evaluation form and return it to the box at the entrance to the room. Thank you!

More Related