using windows azure storage n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Using Windows Azure Storage PowerPoint Presentation
Download Presentation
Using Windows Azure Storage

Loading in 2 Seconds...

play fullscreen
1 / 48

Using Windows Azure Storage - PowerPoint PPT Presentation


  • 123 Views
  • Uploaded on

COS304. Using Windows Azure Storage . Jai Haridas Senior Development Lead Microsoft Corporation. Agenda. Windows Azure Storage Blobs , Drives, Tables , Queues Partitioning & Scalability Q&A. What is Windows Azure Storage?. Cloud Storage System Scalable Durable Highly Available

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Using Windows Azure Storage' - ira


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using windows azure storage
COS304

Using Windows Azure Storage

Jai Haridas

Senior Development Lead

Microsoft Corporation

agenda
Agenda
  • Windows Azure Storage
    • Blobs, Drives, Tables, Queues
  • Partitioning & Scalability
  • Q&A
what is windows azure storage
What is Windows Azure Storage?
  • Cloud Storage System
    • Scalable
    • Durable
    • Highly Available
  • Pay for what you use
abstractions
Abstractions
  • Blobs– File system in the cloud
  • Drives–Durable NTFS volumes for Windows Azure applications
  • Tables– Massively scalable structured storage.
  • Queues– Reliable storage and delivery of messages
windows azure data storage concepts
Windows Azure Data Storage Concepts

Container

Blobs

Account

  • Table

Entities

https://<account>.blob.core.windows.net/<container>

Queue

Messages

https://<account>.table.core.windows.net/<table>

https://<account>.queue.core.windows.net/<queue>

windows azure blobs
Windows Azure Blobs
  • A highly scalable and durable file system in the cloud
    • An account can create many containers
    • No limit on number of blobs in a container
  • Associate metadata with blobs
    • Share
  • Share your blobs
    • Public containers, Shared Access Signature (SAS)
  • Upload/Download Blobs
    • Allows range reads
    • Conditional operations – If-Match, If-Not-Modified-Since etc.
how to upload download files

How to Upload/Download files?

  • Goals
    • It is simple!
    • Upload your files

demo

windows azure blobs types block blobs
Windows Azure Blobs Types – Block Blobs
  • Targeted at streaming workloads
  • Each blob consists of a sequence of blocks
    • 2 Phase commit: Blocks are uploaded and then separately committed
    • Efficient continuation and retry
    • Random range reads possible
  • Size limit 200GB per blob
block blobs

Block 1

Block 2

Block 3

Block 4

Block 1

Block blobs
  • File has variable sized blocks
  • Upload blocks in parallel using PutBlock
  • Retry failed blocks
  • Commit the blob using PutBlockList

Local file

Block 5

Block 2

Block 1

Block 3

Cloud blob

windows azure blobs types page blobs
Windows Azure Blobs Types – Page Blobs
  • Targeted at random write workloads
  • Each blob consists of an array of pages
  • Size limit 1TB per blob
  • Page
    • Each page range write is committed on PUT
    • Page is 512 bytes in size
    • Write boundary aligned at multiple of 512 byte
    • Range reads possible
    • Pages that do not have data are zeroed out
page blobs
Page blobs
  • Write 5K bytes – PutPage
  • Clear 2K bytes starting at offset 1K – ClearPage
  • Overwrite 2K bytes starting at 2K – PutPage
  • Truncate blob to 3K - SetMaxBlobSize

5120

5120

5120

0

0

0

0

3072

3072

1024

2048

4096

sharing your files
Sharing Your Files
  • Every blob request must be signed with the account owner’s key
  • Share your files
    • The container must be public
    • Shared Access Signature (SAS) – share pre-authenticated URLs with users
  • SAS
    • Use container level access as it allows access to be easily revoked
how to share your files

How to Share your files?

  • Goals
    • By default – authentication required
    • Public containers – everyone has read access
    • SAS – share url with subset without sharing your key!

demo

what are snapshots
What Are Snapshots?
  • Create a point in time read-only copy of a blob
  • Every snapshot creates a new read only point in time copy
  • Restore snapshots using copy blob
  • Cleanup your snapshots
  • Charged only for unique blocks or pages i.e. reuse blocks or pages
    • For reuse, use WritePages or PutBlock& PutBlockList
what does unique mean
What does unique mean?
  • Add 2 blocks {1, 2}
  • Commit the blob
  • Take a snapshot
  • Charged for
    • Base blob’s blocks {1, 2}

ID=1

ID=2

ID=1

ID=2

A

BB

A

BB

Base blob = alphabets.txt

#1 snapshot=2011-04-10T19:26:24.8690267Z

Snapshot #1

what does unique mean1
What does unique mean?
  • Add a new block Id=3
  • Commit the base blob
  • Charged for
    • Base blob’s blocks {1, 2, 3}
  • Snapshot to create the second snapshot
  • Charged for
    • Base blob’s blocks {1, 2, 3}

ID=3

ID=2

ID=2

ID=1

ID=1

ID=2

ID=2

ID=1

ID=1

ID=3

CCC

CCC

A

BB

A

A

A

BB

BB

BB

Base blob = alphabets.txt

#1 snapshot=2011-04-10T19:26:24.8690267Z

Snapshot #2

Base blob = alphabets.txt

#2 snapshot=2011-05-10T19:26:24.8690267Z

what does unique mean2
What does unique mean?
  • Update block id 1 to now contain the same data “A”
  • Commit the blob
  • Charged for:
    • Base blob’s blocks {1, 2, 3}
    • Snapshot #1’s blocks {1}
  • Take a snapshot
  • Charged for:
    • Base blob’s blocks {1, 2, 3}
    • Snapshot #1’s blocks {1}

ID=2

ID=1

ID=2

ID=3

ID=2

ID=3

ID=1

ID=2

ID=3

ID=1

ID=1

A

BB

CCC

BB

A

A

BB

BB

CCC

A

CCC

#1 snapshot=2011-04-10T19:26:24.8690267Z

Snapshot #3

A

#2 snapshot=2011-05-10T19:26:24.8690267Z

Base blob = alphabets.txt

#3 snapshot=2011-05-10T19:28:24.8690267Z

blobs best practices tips
Blobs – Best Practices & Tips
  • Client Library uses a default of 90s timeout – use size based timeout
  • Use parallel block upload to reduce latency
  • Snapshots – For block or page reuse, use PutBlock-PutBlockList/PutPage to update
  • Shared Access Signature
    • Use container level policy as it allows revoking permissions
    • Share SAS URLs using https
blobs best practices tips1
Blobs – Best Practices & Tips
  • Create new containers for blobs like log files that have retention period
    • Delete logs after 1 month - create new containers every month.
  • Container recreation
    • Garbage collection can take time until which time container with same name cannot be created.
    • Use unique names for containers
slide20

WINDOWS AZURE DRIVE

  • Provides a durable NTFS volume for Windows Azure applications
    • Use existing NTFS APIs
    • Easy migration path to the cloud
    • Durability and survival of data on application failover or hardware failure
      • All flushed and un-buffered writes to drive are made durable
slide21

WINDOWS AZURE DRIVE

  • A Windows Azure Drive is a Page Blob
  • Mounts Page Blob as an NTFS drive
  • Mounted by one VM at a time for read/write
  • A VM can dynamically mount up to 16 drives
  • Drives can be up to 1 TB
windows azure tables
Windows Azure Tables
  • Massively Scalable and Durable Structured Storage
    • Provides flexible schema
    • A storage account can contain many tables
    • No limit on number of entities (aka rows) in each table
  • Familiar and Easy to use API
    • OData Protocol
    • WCF Data Services - .NET classes and LINQ
simple structured storage how to build it

Simple structured storage… how to build it?

  • Goals
    • It is simple! Declare your POCO
    • Show Add/Update/Delete
    • Compare various types of queries
    • How to handle multiple types in a single table?

demo

tables best practices tips
Tables – Best Practices & Tips
  • Use clustered index in queries for performance
  • Limit large scans
  • Expect continuation tokens for queries that scan
    • Split “OR” on keys as individual queries
  • Point query throws an exception if resource does not exist.
    • DataServiceContext.IgnoreResourceNotFoundException = true
  • For “read only” scenarios turn off tracking
    • DataServiceContext.MergeOption = MergeOption.NoTracking
tables best practices tips1
Tables – Best Practices & Tips
  • Entity Group Transaction
    • Get transaction semantics
    • Reduce costs
  • Do not reuse DataServiceContext across multiple logical operations and discard on failures
  • AddObject/AttachTo can throw exception if entity is already being tracked
windows azure queues
Windows Azure Queues
  • Highly scalable, available and provides reliable message delivery
  • A storage account can create any number of queues
  • Access is provided via REST
windows azure queue message
Windows Azure Queue Message
  • 8K message size limit and default expiry of 7 days
  • Programming semantics – Ensures that a message can be processed at least once
    • Get message to make the message invisible
    • Delete message to remove the message
simple queue demo how to build it

Simple Queue Demo… how to build it?

  • Goals
    • Show Put/Get messages
    • Remove Poison Messages

demo

queues best practices tips
Queues – Best Practices & Tips
  • Make message processing idempotent
  • Do not rely on order – invisibility time can result in out of order
  • Messages > 8KB
    • Use blobs or tables to store and message has a reference
  • Use dequeue count to detect problems
    • Visibility expiry time needs to increase
    • Poison messages
queues best practices tips1
Queues – Best Practices & Tips
  • Use message count to dynamically increase/reduce workers.
    • Retain one instance that polls once every X time period
    • One instance polling every second results in 2,678,400 calls per month
    • Costs around $2.67
    • Spawn more instances when you detect backlog
agenda1
Agenda
  • Windows Azure Storage
    • Blobs, Drives, Tables and Queues
  • Partitioning & Scalability
  • Q&A
know the scalability targets
Know The Scalability Targets
  • Single Blob Partition
  • Throughput up to 60 MB/s
  • Single Queue/Table Partition
  • Up to 500 transactions (entities or messages) per second
  • Storage Account
  • SLA – 99.9% Availability
  • Capacity – Up to 100 TBs
  • Transactions – Up to 5000 entities per second
  • Bandwidth – Up to 3 gigabits per second
how to scale beyond known scalability targets
How To Scale Beyond Known Scalability Targets?
  • Distribute traffic to multiple partitions
  • Use multiple storage accounts
  • When limit is hit, app may see frequent ‘503 server busy’; applications should implement exponential back-off
storage partitioning how to scale
Storage Partitioning – How to Scale?

Understanding partitioning is key to understanding scale

automatic load balancing assignment
Automatic Load Balancing - Assignment

VIP

Legend

- Partition

FE

FE

FE

- Server Load

Master System

Master System

Master System

Reassign Partitions

Offload Partitions

BE1

BE2

BE3

BE4

Distributed File System

  • Time between offload to reload is in the order of seconds
  • Time to decide to load balance is in the order of minutes
  • Goal is to only reassign a partition only if the system has to
windows azure storage takeaways
Windows Azure Storage Takeaways
  • Scalable data abstractions to build your applications
    • Blobs– Files and large objects
    • Drives– Durable NTFS volume for Windows Azure applications
    • Tables– Massively scalable structured storage
    • Queues– Reliable delivery of messages
  • Easy to use via the Storage Client Library
  • More info on Windows Azure Storage at:

http://blogs.msdn.com/windowsazurestorage/

track resources
Track Resources

Don’t forget to visit the Cloud Power area within the TLC (Blue Section) to see product demos and speak with experts about the Server & Cloud Platform solutions that help drive your business forward.

You can also find the latest information about our products at the following links:

  • Cloud Power - http://www.microsoft.com/cloud/
  • Private Cloud - http://www.microsoft.com/privatecloud/
  • Windows Server - http://www.microsoft.com/windowsserver/
  • Windows Azure - http://www.microsoft.com/windowsazure/
  • Microsoft System Center - http://www.microsoft.com/systemcenter/
  • Microsoft Forefront - http://www.microsoft.com/forefront/
resources
Resources
  • Connect. Share. Discuss.

http://northamerica.msteched.com

Learning

  • Sessions On-Demand & Community
  • Microsoft Certification & Training Resources

www.microsoft.com/teched

www.microsoft.com/learning

  • Resources for IT Professionals
  • Resources for Developers

http://microsoft.com/technet

http://microsoft.com/msdn

slide47

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.