Large scale machine learning using dryadlinq
Download
1 / 17

Large-scale Machine Learning using DryadLINQ - PowerPoint PPT Presentation


  • 144 Views
  • Uploaded on

Large-scale Machine Learning using DryadLINQ. Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019. Goal of DryadLINQ. Software Stack. Applications. . Net + LINQ. DryadLINQ.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Large-scale Machine Learning using DryadLINQ' - randi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Large scale machine learning using dryadlinq

Large-scale Machine Learning using DryadLINQ

Mihai Budiu

Microsoft Research, Silicon Valley

Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop

Stanford, June 11, 2019



Software stack
Software Stack

Applications

.Net + LINQ

DryadLINQ

Dryad

Cluster storage

Cluster services

Windows Server

Windows Server

Windows Server

Windows Server


Dryad execution layer
Dryad = Execution Layer

Job (application)

Pipeline

Dryad

Unix Shell

Cluster

Machine


Linq data model
LINQ Data Model

.NET objects of type T

Collection


Linq language summary
LINQ Language Summary

Input

Where (filter)

Select (map)

GroupBy

OrderBy (sort)

Aggregate (fold)

Join


LINQ

=> DryadLINQ

Dryad


Dryadlinq data model
DryadLINQ Data Model

.Net objects

Partition

Collection


Dryadlinq linq dryad
DryadLINQ = LINQ + Dryad

Collection<T> collection;

static boolIsLegal(Key c);

var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

Code

Dryad job

Data

collection

C#

C#

C#

C#

results



Natal problem
Natal Problem

  • Recognize players from depth map

  • At frame rate

  • Low resource usage


Learn from data
Learn from Data

Rasterize

Training examples

Motion Capture

(ground truth)

Machine learning

Classifier



Cluster based training
Cluster-based training

Classifier

Training examples

Machine learning

DryadLINQ

Dryad


You can have it
You can have it!

  • Dryad+DryadLINQ available for download

    • Academic license

    • Commercial evaluation license

  • Runs on Windows HPC platform

  • Dryad is in binary form, DryadLINQ in source

  • Requires signing a 3-page licensing agreement

  • http://connect.microsoft.com/site/sitehome.aspx?SiteID=891