Windows Azure for Research
1 / 80

Windows Azure for Research Roger Barga Jared Jackson Contributors include Nelson Araujo, Dennis Gannon and Wei Lu Cloud - PowerPoint PPT Presentation

  • Uploaded on

Windows Azure for Research Roger Barga & Jared Jackson Contributors include Nelson Araujo, Dennis Gannon and Wei Lu Cloud Computing Futures Group, Microsoft Research. Presentation Outline. Microsoft and Cloud Computing             [10 minutes] Introduction to Windows Azure      [35 minutes]

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Windows Azure for Research Roger Barga Jared Jackson Contributors include Nelson Araujo, Dennis Gannon and Wei Lu Cloud' - sandra_john

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Windows Azure for ResearchRoger Barga & Jared JacksonContributors include Nelson Araujo, Dennis Gannon and Wei LuCloud Computing Futures Group, Microsoft Research

Slide2 l.jpg

Presentation Outline

Microsoft and Cloud Computing            [10 minutes]

Introduction to Windows Azure     [35 minutes]

Research Applications on Azure, demos[10 minutes]

How They Were Built                                                 [15 minutes]

A Closer Look at Azure [15 minutes]

Cloud Research Engagement Initiative[ 5 minutes]

Q&A   [ * ]

Microsoft and c loud computing l.jpg

Microsoft and Cloud Computing

Science 2020 l.jpg
Science 2020

“In the last two decades advances in computing technology, from processing speed to network capacity and the Internet, have revolutionized the way scientists work.

From sequencing genomes to monitoring the Earth's climate, many recent scientific advances would not have been possible without a parallel increase in computing power - and with revolutionary technologies such as the quantum computer edging towards reality, what will the relationship between computing and science bring us over the next 15 years?”

Sapir whorf context and research l.jpg
Sapir–Whorf: Context and Research

Sapir–Whorf Hypothesis (SWH)

  • Language influences the habitual thought of its speakers

    Scientific computing analog

  • Available systems shape research agendas

    Consider some past examples

  • Cray-1 and vector computing

  • VAX 11/780 and UNIX

  • Workstations and Ethernet

  • PCs and web

  • Inexpensive clusters and Grids

    Today’s examples

  • multicore, sensors, clouds and services …

    What lessons can we draw?

The pull of economics l.jpg
The Pull of Economics …

Moore’s “Law” favored consumer commodities

  • Economics drove enormous improvements

  • Specialized processors and mainframes faltered

  • The commodity software industry was born

    Today’s economics

  • Manycore processors/accelerators

  • Software as a service/cloud computing

  • Multidisciplinary data analysis and fusion

    This is driving change in research and technical computing

  • Just as did “killer micros” and inexpensive clusters













1 MB

1 MB







1 MB








1 MB

1 MB









1 MB








1 MB

1 MB















Clouds are built on data centers l.jpg
Clouds are built on Data Centers

  • Range in size from “edge” facilities to megascale.

  • Economies of scale

    • Approximate costs for a small size center (1000 servers) and a larger, 100K server center.

Each data center is

11.5 times

the size of a football field

Microsoft advances in dc deployment l.jpg
Microsoft Advances in DC Deployment

Conquering complexity

  • Building racks of servers & complex cooling systems all separately is not efficient.

  • Package and deploy into bigger units, JITD

Data centers and hpc clusters select comparisons l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

  • Communication fabric

  • Storage systems

  • Reliability and resilience

  • Programming model and services

Data centers and hpc clusters select comparisons11 l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

    • Node architectures are indistinguishable – Intel Nehalem, AMD Barcelona or Shanghai, multiple processors, big chunk of memory on the nodes

  • Communication fabric

  • Storage systems

  • Reliability and resilience

  • Programming model and services

Data centers and hpc clusters select comparisons12 l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

  • Communication fabric

Data centers and hpc clusters select comparisons13 l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

  • Communication fabric

  • Storage systems

    • HPC: local scratch or non-existent, secondary is SAN or PFS, PB tertiary storage

    • DC: TB local storage, secondary is JBOD, tertiary is non-existent

Data centers and hpc clusters select comparisons14 l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

  • Communication fabric

  • Storage systems

  • Reliability and resilience

    • HPC: periodic checkpoints, rollback and resume in response to failures, MTBF approaching zero, checkpoint frequency increasing, I/O demand intolerable.

    • DC: loosely consistent models, designed to transparently recover from failures

Data centers and hpc clusters select comparisons15 l.jpg
Data Centers and HPC Clusters – select comparisons

  • Node and system architectures

  • Communication fabric

  • Storage systems

  • Reliability and resilience

  • Programming model and services

A bunch of machines in a data center l.jpg
A bunch of machines in a data center

Azure FC Owns this Hardware


Fabric Controller (FC)

Fc t hen installs the guest vm l.jpg
FC then Installs the Guest VM

Each vm has l.jpg
Each VM Has…

At Minimum

CPU: 1.5-1.7 GHz x64

Memory: 1.7GB

Network: 100+ Mbps

Local Storage: 500GB

Up to

CPU: 8 Cores

Memory: 14.2 GB

Local Storage: 2+ TB

Fc then installs the azure platform l.jpg
FC Then Installs the Azure Platform



Windows azure compute service a closer look l.jpg
Windows Azure Compute Service A closer look

Web Role

Worker Role


{ … }


ASP.NET, WCF, etc.


Load Balancer





Suggested application model using queues for reliable messaging l.jpg
Suggested Application ModelUsing queues for reliable messaging

To scale, add more of either


{ … }

Worker Role

Web Role

1) Receive work

4) Do work

ASP.NET, WCF, etc.

2) Put work in queue

3) Get work from queue


Scalable fault tolerant applications l.jpg
Scalable, Fault Tolerant Applications

  • Queues are the application glue

    • Decouple parts of application, easier to scale independently;

    • Resource allocation, different priority queues and backend servers

    • Mask faults in worker roles (reliable messaging).

    • Use Inter-role communication for performance

    • TCP communication between role instances

    • Define your ports in the service models

Storage l.jpg






Load Balancer

Azure storage service a closer look l.jpg
Azure Storage ServiceA closer look










Windows azure storage points of interest l.jpg
Windows Azure StoragePoints of interest

Storage types

  • Blobs: Simple interface for storing named files along with metadata for the file

  • Drives – Durable NTFS volumes

  • Tables: entity-based storage

    Not relational – entities, which contain a set of properties

  • Queues: reliable message-based communication


  • Data is exposed via .NET and RESTful interfaces

  • Data can be accessed by:

    • Windows Azure apps

    • Other on-premise applications or cloud applications

Development environment l.jpg
Development Environment



Development Fabric






Development Storage





Application Works Locally

In the cloud l.jpg
In the Cloud

Application Works Locally

Application Works

In Staging


Windows azure platform basics what the value add l.jpg
Windows Azure Platform Basics What the ‘Value Add’ ?

Provide a platform that is scalable and available

  • Services are always running, rolling upgrades/downgrades

  • Failure of any node is expected, state has to be replicated

  • Failure of a role (app code) is expected, automatic recovery

  • Services can grow to be large, provide state management that scales automatically

  • Handle dynamic configuration changes due to load or failure

  • Manage data center hardware: from CPU cores, nodes, rack, to network infrastructure and load balancers.

Windows azure compute fabric fabric controller l.jpg
Windows Azure Compute FabricFabric Controller

  • Owns all data center hardware

  • Uses inventory to host services

  • Deploys applications to free resources

  • Maintains the health of those applications

  • Maintains health of hardware

  • Manages the service life cycle starting from bare metal

Windows azure compute fabric fault domains l.jpg
Windows Azure Compute FabricFault Domains

Purpose: Avoid single points of failures

Fault domains

  • Unit of a failure

    • Examples: Compute node, a rack of machines

  • System considers fault domains when allocating service roles

  • Service owner assigns number required by each role

    • Example: 10 front-ends, across 2 fault domains

Allocation is across fault domains

Windows azure compute fabric update domains l.jpg
Windows Azure Compute FabricUpdate Domains

Purpose: ensure the service stays up while undergoing an update

  • Unit of software/configuration update

    • Example: set of nodes to update

  • Used when rolling forward or backward

  • Developer assigns number required by each role

    • Example: 10 front-ends, across 5 update domains

Update domains

Allocation is across update domains

Windows azure compute fabric push button deployment l.jpg
Windows Azure Compute FabricPush-button Deployment

Step 1: Allocate nodes

  • Across fault domains

  • Across update domains

    Step 2: Place OS and role images on nodes

    Step 3: Configure settings

    Step 4: Start Roles

    Step 5: Configure load-balancers

    Step 6: Maintain desired number of roles

  • Failed roles automatically restarted

  • Node failure results in new nodes automatically allocated

Allocation across fault and update domains


Windows azure compute fabric the fc keeps your service running l.jpg
Windows Azure Compute FabricThe FC Keeps Your Service Running

Windows Azure FC monitors the health of roles

  • FC detects if a role dies

  • A role can indicate it is unhealthy

    • Current state of the node is updated appropriately

    • State machine kicks in again to drive us back into goals state

      Windows Azure FC monitors the health of host

  • If the node goes offline, FC will try to recover it

    If a failed node can’t be recovered, FC migrates role instances to a new node

  • A suitable replacement location is found

  • Existing role instances are notified of change

Windows azure key takeaways l.jpg
Windows AzureKey takeaways

Cloud services have specific design considerations

  • Always on, distributed state, large scale, fault tolerance

  • Scalable infrastructure demands a scalable architecture

    • Stateless roles and durable queues

      Windows Azure frees service developers from many platform issues

      Windows Azure manages both services and servers

Azure applications l.jpg
Azure Applications

Demonstrating Scientific Research Applications in the Cloud

AzureBLAST- Finding similarities in genetic sequences

Azure Ocean- Rich client visualization with cloud based data computation

Azure MODIS- Imagery analysis from satellite photos

PhyloD- Finding relationships in phylogenetic trees

Azureblast demonstration l.jpg
AzureBLAST Demonstration

Azure modis overview l.jpg
Azure MODIS Overview

Two satellites:

  • Terra,“EOS AM” , launched 12/1999,descending, equator crossing at 10:30 AM

  • Aqua, “EOS PM”, launched 5/2002,ascending, equator crossing at 1:30 PM

    Near polar orbits, day/night mode, ~2300 KM swath

    L0 (raw) and L1 (calibrated) data held at Goddard DAAC

    L2 and L3 products made by a collection of different algorithms provided by a number of different researchers

Azure modis l.jpg

. . .

Research Results



Analysis Reduction Stage


Service Web Role Portal

Data Collection Stage


Derivation ReductionStage

Phylod overview l.jpg
PhyloD Overview

  • Statistical tool used to analyze DNA of HIV from large studies of infected patients

  • PhyloD was developed by Microsoft Research and has been highly impactful

  • Small but important group of researchers

    • 100’s of HIV and HepC researchers actively use it

    • 1000’s of research communities rely on results

Cover of PLoS Biology

November 2008

  • Typical job, 10 – 20 CPU hours, extreme jobs require 1K – 2K CPU hours

    • Requires a large number of test runs for a given job (1 – 10M tests)

    • Highly compressed data per job ( ~100 KB per job)

Azureblast looking deeper l.jpg
AzureBLAST – Looking Deeper

Step 1. Staging

  • Compress required data

  • Upload to Azure Store

  • Deploy Worker Roles- Init() function downloads and decompresses data to the local disk


Sequence Database



Azure Storage




Azureblast looking deeper48 l.jpg
AzureBLAST – Looking Deeper

Step 2. Partitioning a Job

User Input

Input Partition

Azure Storage

Queue Message

Web Role

Single Partitioning

Worker Role

Azureblast looking deeper49 l.jpg
AzureBLAST – Looking Deeper

Step 3. Doing the Work

User Input

Input Partition

BLAST Output

Azure Storage

Queue Message


Web Role

Single Partitioning

Worker Role

BLAST ready Worker Roles

Azureblast some good results l.jpg
AzureBLAST – Some good results

Azureblast some interesting results l.jpg
AzureBLAST – Some interesting results

Azureblast lessons learned l.jpg
AzureBLAST – Lessons Learned

  • Always design with failure in mind- On large jobs it will happen, and it can happen anywhere

  • Factoring work into optimal sizes has large performance impacts- The optimal size may change depending on the scope of the job

  • Test runs are your friend- Blowing $20,000 of computation is not a good idea

  • Make ample use of logging features- When failure does happen, it’s good to know where

  • Cutting 10 years of computation down to 1 week is great!!- Little Cloud development headaches are probably worth it

Phylod illustrating scalability l.jpg
PhyloD – Illustrating Scalability

Time-Space fungibility in the Cloud





Azureocean architecture l.jpg
AzureOcean – Architecture

  • Utilizes a general jobs based task manager

    which registers jobs and their resulting data

Data Products

Metadata cataloged in Registry

Binary reference and versioning


Submit Application Package (binaries + definitions)

Azure ocean l.jpg
Azure Ocean

Client Visualization / Cloud Data and Computation

  • The Cloud is not a Jack-of-All-Trades

  • Client side tools are particularly appropriate for

    • Applications using periphery devices

    • Applications with heavy graphics requirements

    • Legacy user interfaces that would be difficult to port

  • Our goal then:

    • Make best use of the capabilities of client and cloud computing

    • Often by making the cloud invisible to the end user

Developing on azure l.jpg
Developing on Azure

  • A deeper dive into Windows Azure’s inner workings

    - Focus on Azure Storage internals

  • A sampling of best practices

Windows azure storage l.jpg
Windows Azure Storage

  • Rich Data Abstractions

    • Large user data items: blobs

    • Service state: tables

    • Service workflow: queues

    • Existing NTFS service migration: drives

  • Simple and Familiar Programming Interfaces

    • REST: HTTP and HTTPS

    • Supported Storage Client Library: .Net APIs

    • NTFS:Azure Drive

Windows azure storage account l.jpg
Windows Azure Storage Account

  • User creates a globally unique storage account name

    • Can choose geo-location to host storage account

      • “US Anywhere”, “US North Central”, “US South Central”,

    • Can co-locate storage account with compute account

    • Receive a 256 bit secret key when creating account

  • Storage Account Capacity

    • Each storage account can store up to 100 TB

    • Default limit of 5 storage accounts per subscription

Blob storage basics l.jpg



  • Blob

Blob Storage Basics

  • PIC01.JPG

  • images

  • jared

  • PIC02.JPG

  • movies

  • MOV1.AVI

Blob containers l.jpg
Blob Containers

  • Number of Blob Containers

    • Can have has many Blob Containers as will fit within the storage account limit

  • Blob Container

    • A container holds a set of blobs

    • Set access policies at the container level

      • Private or Public accessible

    • Associate Metadata with Container

      • Metadata are <name, value> pairs

      • Up to 8KB per container

The two types of blobs block and page l.jpg
The Two Types of Blobs – Block and Page

  • Block Blob

    • Targeted at streaming workloads

    • Each blob consists of a sequence of blocks

      • Each block is identified by a Block ID

    • Size limit 200GB per blob

  • Page Blob

    • Targeted at random read/write workloads

    • Each blob consists of an array of pages

      • Each page is identified by its offset from the start of the blob

    • Size limit 1TB per blob

Blob storage basics blocks and pages l.jpg



  • Blob

Blob Storage Basics – Blocks and Pages

Block or Page

  • PIC01.JPG

  • images

Block orPage 1

  • jared

  • PIC02.JPG

Block orPage 2

  • movies

Block orPage 3

  • MOV1.AVI

Uploading a block blob l.jpg
Uploading a Block Blob

blobName = “TheBlob.wmv”;

PutBlock(blobName, blockId1, block1Bits);

PutBlock(blobName, blockId2, block2Bits);


PutBlock(blobName, blockIdN, blockNBits);



10 GB Movie

Block Id 2

Block Id 1

Block Id 3

Block Id N

Windows Azure Storage

  • Benefit

    • Efficient continuation and retry

    • Parallel and out of order upload of blocks



Block blob details l.jpg
Block Blob Details

  • Block can be up to 4MB each

    • Each block can be variable size

    • Each block has a 64 byte ID

      • Scoped by blob name and stored with the blob

  • Block operation

    • PutBlock

      • Puts an uncommitted block defined by the block ID for the blob

  • Block List Operations

    • PutBlockList

      • Provide the list of blocks to comprise the readable version of the blob

        • Can use blocks from uncommitted or committed list to update blob

    • GetBlockList

      • Returns the list of blocks, committed or uncommitted for a blob

        • Block ID and Size of Block is returned for each block

Page blobs l.jpg
Page Blobs


  • Create MyBlob

    • Specify Blob Size = 10 GBytes

  • Fixed Page Size = 512 bytes

  • Random Access Operations

    • PutPage[512, 2048)

    • PutPage[0, 1024)

    • ClearPage[512, 1536)

    • PutPage[2048,2560)

  • GetPageRange[0, 4096) returns valid data ranges:

    • [0,512) , [1536,2560)

  • GetBlob[1000, 2048) returns

    • All 0 for first 536 bytes

    • Next 512 bytes are data stored in [1536,2048





10 GB Address Space


10 GB

Choosing between block and page l.jpg
Choosing Between Block and Page

  • Block Blob

    • Targeted at streaming workloads

    • Update semantics

      • Upload a bunch of blocks. Then commit change.

      • Concurrency: ETag Checks

  • Page Blob

    • Targeted at random read/write workloads

    • Update Semantics

      • Immediate update

      • Concurrency: Leases

Snapshots l.jpg

  • All writes applied to base blob name

  • Only delta changes are maintained across snapshots

  • Restore to a prior version via snapshot promotion

  • Can use ListBlobs to enumerate the snapshots for a blob

Snapshot 2

Snapshot 1


MyBlob, 2009/11/03 08:31:30.15

MyBlob, 2009/11/03 08:40:20.32


Azure drive l.jpg
Azure Drive

  • A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD)

    • Drives can be up to 1TB

  • A VM can dynamically mount up to 8 drives

  • A Page Blob can only be mounted by one VM at a time for read/write

  • Remote Access via Page Blob

    • Can upload the VHD to its Page Blob using the blob interface, and then mount it as a Drive

    • Can download the Drive through the Page Blob interface

Azure tables l.jpg
Azure Tables

  • Provides Structured Storage

    • Massively Scalable Tables

      • Billions of entities (rows) and TBs of data

      • Can use thousands of servers as traffic grows

    • Highly Available & Durable

      • Data is replicated several times

  • Familiar and Easy to use API

    • ADO.NET Data Services – .NET 3.5 SP1

      • .NET classes and LINQ

      • REST – with any platform or language

Azure table data model l.jpg
Azure Table Data Model

  • Table

    • A storage account can create many tables

    • Table name is scoped by account

    • Set of entities (i.e. rows)

  • Entity

    • Set of properties (columns)

    • Required properties

      • PartitionKey, RowKey and Timestamp

Best practices l.jpg
Best Practices

Design and Planning

  • Design your workers to execute a task only once

  • Optimize against storage transactions as well as data size

  • Use Azure Drive for distributing existing non-Azure applications

    Azure Storage

  • Remember Azure tables only index on partition and row keys

  • Batch multiple small tasks into a single queue message

  • Use snapshots when you need read only access to a blob

  • Use batch updates to all of your data stores

Best practices72 l.jpg
Best Practices

Network Communication

  • Increasing VM size will also increase your network throughput

  • Use node-to-node communication to save on message latency costs- Note that you lose durable messaging when you do this

    Testing & Development

  • Include retry logic in all instances where you are accessing data

  • Use built-in logging and performance measurement APIs

  • Use multiple worker nodes to add tasks to the message queue

  • Use ‘heartbeat’ mechanisms when debugging your applications

Best practices references l.jpg
Best Practices - References



Slide75 l.jpg
Cloud Computing Research Engagement InitiativeDennis Gannon, Roger Barga, Jared Jackson, Nelson Araujo, and Wei Lu

  • International Engagements. Offer cloud resources to academic and research communities worldwide, back up this offering with a technical engagements team. Lower barrier to entry through tutorials, accelerators, developer best practices. Support policy change in government funding agencies.

  • Data. Provide select reference data sets on Azure to enable communities of researchers. Invest in services and applications to easily upload data and samples that can be repurposed. Let the community use these to host own data sets.

  • Services for Research. Provide applications and core services for research, as coherent solution accelerators. Pull through MS products and MSR technologies, partner with ISVs, make these technologies discoverable and usable.

  • Ask the question, what does it take to catalyze a community of researchers, what are the core services, key products to pull through to support research.

The cloud could c hang e the way we do research offers the promise of broadening participation l.jpg
The cloud could change the way we do researchOffers the promise of broadening participation

The Rest of Us

  • Use laptops.

  • Got data, now what?

  • And it is really is about data, not the FLOPS…

    • Our data collections are not as big as we wished.

    • When data collection does grow large, not able to analyze.

      Paradigm shift for research

  • The ability to marshal needed resources on demand.

    • Without caring or knowing how it gets done…

  • Funding agencies can request grantees to archive research data.

  • The cloud can support very large numbers of users or communities.

F ocus c loud cloud for research l.jpg
Focus Cloud + Cloud for Research

Seamless interaction

  • Cloud is the lens that magnifies the power of desktop;

  • Persist and share data from client in the cloud;

  • Analyze data initially captured in client tools, such as Excel;

    • Analysis as a service (think SQL, Map-Reduce, R/MatLab).

    • Data visualization generated in the cloud, display on client;

    • Provenance, collaboration, other ‘core’ services…

Microsoft research nsf engagement l.jpg
Microsoft Research NSF Engagement

Access to a substantial Windows Azure resources

  • Available over a three year period

  • To be allocated by NSF with new NSF awards

    Coupled with

  • Access to a research-oriented technical team

    Azure resource offering

  • 20 million core hours per year

  • 200 terabytes of triply replicated storage

  • 1 terabyte/day/project of aggregate ingress/egress bandwidth

  • Tier one support

    International program, discussions underway…

Questions l.jpg

http:[email protected]