An architectural approach to managing data in transit
Download
1 / 15

An Architectural Approach to Managing Data in Transit - PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on
  • Presentation posted in: General

An Architectural Approach to Managing Data in Transit. Micah Beck Director & Associate Professor Logistical Computing and Internetworking Lab Computer Science Department University of Tennessee DOE Data Management Workshop 3/17/2004. “Data in Transit”.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentationdownload

An Architectural Approach to Managing Data in Transit

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


An architectural approach to managing data in transit

An Architectural Approach to Managing Data in Transit

Micah Beck

Director & Associate Professor

Logistical Computing and Internetworking Lab

Computer Science Department

University of Tennessee

DOE Data Management Workshop 3/17/2004


Data in transit

“Data in Transit”

  • After being generated by an instrument or supercomputer

  • Not stored in a permanent archive

  • Serving the diverse purposes of a community of users and applications

  • Being transferred, processed and stored to meet changing and unanticipated needs

    • Visualization

    • Data Mining

    • Collaboration

    • Distributed Computing


Interoperability via a common interface

Interoperability via a Common Interface

  • Span heterogeneous physical resources, operating systems, local management schemes

  • Serve changing and unexpected application requirements; enable application autonomy

  • We measure success in terms of infrastructuredeployment scalability

    • In networks and distributed systems, this means number, distribution, global reach, spanning administrative domains…

    • The Internet is the gold standard of infrastructure deployment scalability


Layering as an architectural approach

Layering as An Architectural Approach

  • Abstractions at each layer can hide differences at lower layers

  • Exposed approaches avoid creating overly complex mechanisms at lower layers

  • The E2E Principle: Attributes of lower layers implemented on shared infrastructure enable deployment scalability

    • Generality: Serve diverse application needs, model diverse lower layer resources

    • Weak semantics: Don’t give too much away at one time!


The ip network stack

The IP Network Stack

Application

Transport

common interface (IP)

Network

Link

Physical


Ip s failure of scalability

IP’s Failure of Scalability

  • Today, IP is failing as a common interface

  • The design of IP is out of date

    • Application communities are more diverse

    • Link layer technologies violate IP assumptions

  • Application communities are defining their own common interfaces for general resource sharing, deploying their own infrastructure (e.g. the Grid)

  • Some networking communities have abandoned interoperability at the network layer between widely divergent link layer technologies (e.g. optical switching & IP)


The transit layer a new location for interoperability

The Transit Layer: A New Location for Interoperability

  • Expand the link layer to a local layer to model transfer, storage and processing resources

  • Insert a new transit layer between the local and network layers to implement a common interface to diverse technologies at the local layer

  • Adopt a highly general common interface at the transit layer, providing a uniform view of all of the resources of the network node

  • Build diverse network services on top of this common interface to model diverse application requirements

  • “Locating Interoperability in the Network Stack”, Micah Beck & Terry Moore, UT-CS-04-520, Univ. of TN CS Dept Tech Rpt


The transit network stack

The Transit Network Stack

Application

Transport

Network

common interface

Transit

Local

Physical

transfer

storage

processing


Transit networking a unified view

Transit Networking: A Unified View

“… memory locations … are just wires turned sideways in time”

Dan Hillis, 1982,Why Computer Science is No Good


Logistical networking an overlay implementation of the transit layer

Logistical Networking: An Overlay Implementation of the Transit Layer

  • Logistical Networking is an overlay implementation of transit layer functionality built on top of the IP network

  • The Internet Backplane Protocol is the common transit layer interface for Logistical Networking

  • Network nodes are IBP “depots” that run as user level processes, communicate using TCP/IP as well as other link and network layer protocols

  • Depots also serve storage and processing resources to Logistical Networking clients


Ln tools and deployment

LN Tools and Deployment

  • The Logistical Runtime System (LoRS) is a set of tools based on IBP that enable users to take advantage of the resources of IBP depots

  • Logistical Distribution Network (LoDN) is a data directory, monitoring and management system

  • The Logistical Backbone is a Resources Discovery service and global experimental IBP testbed

    • Over 35 TB of storage available

    • Over 300 depots in 21 countries

    • Leverages the resources of PlanetLab

  • Additional depots deployed at ORNL & NERSC


L bone august 2003 20tb

L-Bone: August 2003 (20TB)


Example ln applications

Example LN Applications

  • Astrophysics: Terascale Supernova Initiative (A. Mezzacappa, ORNL; J. Blondin, NCSU)

    • Management of massive datasets

  • Fusion Energy Research (S. Klasky, PPPL)

    • Streaming of simulation data during generation

  • Viewset-Based Visualization

    • Prestaging & caching of distant data

  • Content Distribution

    • Heroic data distribution problems (Linux ISOs)

  • Multimedia Networking

    • Creation, mgt & delivery of high value content


Ln futures and directions

LN Futures and Directions

  • Storage

    • Implementation of file system services

    • Moving data through firewalls at line speed

    • QoS in highly controlled environments

  • Networking

    • Interoperability at ultrascale

    • Advanced services (e.g. multicast)

  • Computation

    • Offloading visualization to IBP depots

    • Developing sets of operations to support application communities


An architectural approach to managing data in transit

Thank you!

mbeck@cs.utk.edu

http://loci.cs.utk.edu


ad
  • Login