Large scale machine learning using dryadlinq
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Large-scale Machine Learning using DryadLINQ PowerPoint PPT Presentation


  • 106 Views
  • Uploaded on
  • Presentation posted in: General

Large-scale Machine Learning using DryadLINQ. Mihai Budiu Microsoft Research, Silicon Valley Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop Stanford, June 11, 2019. Goal of DryadLINQ. Software Stack. Applications. . Net + LINQ. DryadLINQ.

Download Presentation

Large-scale Machine Learning using DryadLINQ

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Large scale machine learning using dryadlinq

Large-scale Machine Learning using DryadLINQ

Mihai Budiu

Microsoft Research, Silicon Valley

Ambient Intelligence: From Sensor Networks to Smart Environments and Social Media Workshop

Stanford, June 11, 2019


Goal of dryadlinq

Goal of DryadLINQ


Software stack

Software Stack

Applications

.Net + LINQ

DryadLINQ

Dryad

Cluster storage

Cluster services

Windows Server

Windows Server

Windows Server

Windows Server


Dryad execution layer

Dryad = Execution Layer

Job (application)

Pipeline

≈

Dryad

Unix Shell

Cluster

Machine


Linq data model

LINQ Data Model

.NET objects of type T

Collection


Linq language summary

LINQ Language Summary

Input

Where (filter)

Select (map)

GroupBy

OrderBy (sort)

Aggregate (fold)

Join


Large scale machine learning using dryadlinq

LINQ

=> DryadLINQ

Dryad


Dryadlinq data model

DryadLINQ Data Model

.Net objects

Partition

Collection


Dryadlinq linq dryad

DryadLINQ = LINQ + Dryad

Collection<T> collection;

static boolIsLegal(Key c);

var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

Code

Dryad job

Data

collection

C#

C#

C#

C#

results


Example natal training

Example: Natal Training


Natal problem

Natal Problem

  • Recognize players from depth map

  • At frame rate

  • Low resource usage


Learn from data

Learn from Data

Rasterize

Training examples

Motion Capture

(ground truth)

Machine learning

Classifier


Running on xbox

Running on Xbox


Cluster based training

Cluster-based training

Classifier

Training examples

Machine learning

DryadLINQ

Dryad


You can have it

You can have it!

  • Dryad+DryadLINQ available for download

    • Academic license

    • Commercial evaluation license

  • Runs on Windows HPC platform

  • Dryad is in binary form, DryadLINQ in source

  • Requires signing a 3-page licensing agreement

  • http://connect.microsoft.com/site/sitehome.aspx?SiteID=891


Conclusions

Conclusions

=

17


  • Login