neesgrid data overview n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
NEESgrid Data Overview PowerPoint Presentation
Download Presentation
NEESgrid Data Overview

Loading in 2 Seconds...

play fullscreen
1 / 39

NEESgrid Data Overview - PowerPoint PPT Presentation


  • 135 Views
  • Uploaded on

NEESgrid Data Overview. Charles Severance. Goals. Operates both as local and central (curated) repository using same toolset Object oriented - referential integrity - data and metadata are supported together Data migration and replication transparently including security, and metadata

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'NEESgrid Data Overview' - angelina


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
neesgrid data overview

NEESgrid Data Overview

Charles Severance

goals
Goals
  • Operates both as local and central (curated) repository using same toolset
  • Object oriented - referential integrity - data and metadata are supported together
  • Data migration and replication transparently including security, and metadata
  • Uses Grid Services throughout (OGSA/GT3)
  • Easily used access controls - Grid as “single sign on”
  • Data provenance
  • Data in support of research publication
  • Support for repeatable experiments
  • Data oriented research computation support
  • Support for workflow
vision data on the grid

Data

Meta Data

Gathering

Extracting

Repository

Mapping

Vision: data on the Grid

studies

neesgrid data core elements
NEESgrid Data – Core Elements
  • Local Repository
  • Central Repository
  • JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services
    • NEES File Management Services
    • NEES Meta Data Services
  • Data Viewers
    • Streaming (numeric, X/Y graph)
    • Stored (X/Y graph, 2-D structure, video)
core elements
Core Elements

Data Acquisition

NEESpop

Data/MD

Ingest

Tools

API

Local

Repository

Grid and Web Services

Data

Teamlets

API

Workstation

NEESdata

Data tools

Central

Repository

Data

Teamlets

API

Data viewers

neesgrid data technologies
NEESgrid Data – Technologies
  • Grid
    • GRIDFTP is used for data transport
    • Grid Web Services are used to insure security and provide access control between systems over the Internet – also provide for credential passthrough
    • Grid credentials are used as part of login providing a single sign-on framework
  • CHEF
    • Provides a flexible mechanism for deploying GUI tools like the data viewers and data browsers.
a simple experimental scenario

Researcher System

A Simple Experimental Scenario

Developer System

DAQ System

Test Specimen

Labview

Glue

a simulation scenario

Simulation System

Simulation System

Simulation System

Code

Code

Code

A Simulation Scenario

Developer System

the most scenario
The MOST Scenario
  • Part of the run-up to NEESPop 2.0
    • Used Beta of NEESpop and Beta of CHEF
    • Tested the data ingestion
    • Tested the metadata capabilities
    • Developed sample metadata
    • Tested mapping capabilities
  • System still available at

https://cee-nees.cee.uiuc.edu/chef/

slide10

Series of files

Ftp

Plug In

Complete file (aggregated)

NTCP

NSDS

Wires

File I/O

Site / Location

NFMS/NMDS

Computer

Process

UIUC/Newmark

Colorado

Test Specimen

Test Specimen

MatLab Host

And Real-Time

Target Control

System

LabView DAQ

Shore-Western

Sim Controller

LabView DAQ

UIUC

Matlab

NEESPop (1.1)

NSDS

NTCP

Incoming

FTP

CO

NEESPop (2.0)

NTCP

Ingest

Repository

NSDS

CO

UIUC

NCSA

NEESPop (1.1)

Incoming

FTP

Ingest

NCSA

Meta

NTCP

UIUC

NEESMost (Win XP)

Matlab

Computational

Model

NCSA

Ingest

MOST Data Flows

slide11

Time

ch01

ch02

0.00

0.01

3

6

4

8

0 2 3

5 3 3

5 3 4

0 3 4

0 6 8

The experiment was viewed using the standard NEES stored viewer with synchronized video and data and the ability to move back and forth

DAQ

NEESpop 2.0 Alpha

Metadata

slide13

DAQ

0 3 4

0 6 8

<experiment>

<blah>

<public-view>

</experiment>

University ofMinnesota

0 3 4

0 6 8

<experiment>

<blah>

<public-view>

</experiment>

nees markup language neesml
NEES Markup Language (NEESML)
  • Provides an RDF-like structure capable of representing semantic information
    • XML is the syntax which is used
    • Logic is more “object oriented”
      • Can define objects
      • Can create objects
      • Can reference objects
rdf xml versus neesml
RDF/XML Versus NEESML
  • NEESml is topologically equivalent to RDF but more straightforward to use
    • A compromise between usability and functionality
    • Focused on solving the problems of ingesting types and data – rather than “cross-server ontology webs”
    • Used to build a reference set of ingestion tools
    • RDF is a moving target
  • Repository does not store either RDF or NEESML – It is an relational database tuned to store “three-tuples”
the slide

Data

Metadata

The Slide

There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second.

Where we make a viewer capable of viewing a certain type of object.

This is where we build things which make use of knowledge.

This layer will never be complete but it is a large focus of the coming months.

Data Ingestors

Search

Data Mappers

Concepts

Data Viewers

strategies
Strategies
  • Take Existing Data Models and Adopt
  • Build tools and data models at the same time
  • Find existing tools that produce types of data we find interesting
  • Build meta data extractors for important file formats
  • Build converters for things like Excel
why start with the orst model
Why start with the ORST Model?
  • It provides good coverage of a well-understood scope
  • It has consensus across multiple sites
  • A prototype toolset exists around the model which is an important validation
  • http://nees.orst.edu/IT/data.model/
  • We can adopt the core elements and extend as necessary
slide21

o:project

specialCondions

title

startDate

o:experiment[s]

o:role[s]

o:acknowledgement[s]

o:facility

title

shortDescription

longDescription

<type id="o:project" title=”Project">

<specialConditions title=”Special Conditions"/>

<title title=”Title"/>

<startDate title=”Starting Date"/>

<o:experiment allow="o:experiment” max="unbounded" />

<o:role allow="o:role” max="unbounded" />

<o:acknowledgements allow=“o:acknowledgement” max=“unbounded” />

</type>

<type id="o:experiment" title=”Experiment">

<status title=”Status"/>

<title title=”Title"/>

<shortDescription title=”Description"/>

<o:facility allow="o:facility” />

</type>

<type id="o:experiment" title=”Experiment">

<title title=”Title"/>

<shortDescription title=”Description"/>

<longDescription title=”Description"/>

</type>

o:experiment

status

title

shortDescription

o:facility

go forward core elements
Go Forward – Core Elements
  • Implement Access Control
  • Implement Replication
  • Investigate RDF and its relationship to NEESML
  • Investigate provenance – would like to adopt from another project
  • Investigate mapping – Would like to adopt from another project
go forward tools
Go Forward - Tools
  • Evaluate the ORST interface and use it to implement experiment-based interface to meta data repository
  • Investigate tools to represent structural data (like SAC data)
  • Extend and improve viewers – publish API so that sites can extend the viewers
  • Improve notebook
    • Single signon using CHEF/Grid credentials
    • Integration with Metadata
    • Smother integration with CHEF
  • Explore automated synchronized video and data capture and after-experiment replay of synchronized video and data (ORST UMinn)
  • Explore the capture of high quality still images as data (UMinn)
  • Investigate adopting a data-editing tool (XMLSpy)
go forward data models
Go Forward – Data Models
  • Analyze the ORST model, determine core, convert to NEESML, pre-populate repositories with types, and develop usage documentation
  • Form core group between SI, ES, and CS to push data model issues forward – once groundwork is better defined – we can disperse into distributed teams
  • Use experiment based deployment to help us encounter new data needs over time
how to prioritize model exploration and development
How to prioritize model exploration and development
  • Focus on the following areas:
    • Areas where we have or are building tools
    • Areas where we already have incoming data in some format
    • Build the model through experiment based deployment - solve real problems in an open way and see if (with some adaptation) the solutions apply more broadly (i.e. Minnesota )
what is in release 2 0 october 7
Groovy look and feel

Local Data Repository

Repository Browser in CHEF

Browse

Create objects

Upload / download data

API documentation

NEESML User Documentation

Extensible data mapping in Java

Data Viewer in CHEF

Improved visually

Configured by XML

Can read data from repository or from urls

Pre-populated with sample video and data formats

Local Repositories pre-populated with

SAC Data

MOST Data

ORST data model (subset)

What is in Release 2.0?October 7
neesml
NEESML

Table 1: Primitive types in NEESML

configuring events
Configuring Events

<event id="oregon" desc="Oregon Large Tank Test September 8, 2003"

host=“/chef/org.nees.repo.data/retrieve-data?lfn=nacse_sample_01.txt&amp;

static=yes&amp;mapping=nacse-" type="stored">

<channel id="00" desc="time" unit="Seconds" url="t" />

<channel id="01" desc="Offshore Wave Gauge" unit="" url="02" />

<channel id="02" desc="Wave gauge at the front face of the cylinder" unit="" url="03" />

<channel id="03" desc="Wave gauge at the back face of the cylinder " unit="" url="04" />

<channel id="04" desc="Pressure at the front face of the cylinder " unit="" url="05" />

<video id="01" desc="Video of cylinder"

url=“/chef/retrieve-data/static/nacse_sample_01.avi" />

</event>

   <type id="nees:storedEvent">     <data allow="io:file"/>     <url/>     <parseOptions/>

     <storedTimeChannelsmin="1" max=“1" allow="nees:storedTimeChannel"/>     <storedDataChannels min=“0" max="unbounded" allow="nees:storedDataChannel"/>

     <storedVideoChannels min=“0" max="unbounded" allow="nees:storedVideoChannel"/>

     <storedStruct2DChannels min=“0" max="unbounded" allow="nees: stored2DStructureChannel "/>     <fileChannels max="unbounded" allow="nees:fileChannel"/>   </type>

We may be able to get a patch out to switch this to NEESml and provide a simple entry tool.

mappings and the data viewer
Mappings and the Data Viewer
  • NSDS (ISO 8601 Time channel)
  • Column data with time recorded as a column
  • Column – generate time
  • Column – generate time – trigger filter

Channel units: g,g,in,kip

Time ATL1 ATT1

2002-11-13T15:48:55.26499 -0.006409 0.004272

2002-11-13T15:48:55.36499 -0.005798 -0.003662

100.000

0.435 0.161 -1.016 -0.981

0.430 0.161 -1.016 -0.977

0.435 0.161 -1.016 -0.977

public class NEESDataMap

{

public static boolean repoMap(File mainFile,

File mappingFile, String mapping)

{

// Code here

}

}

release 2 1 data aspects december 2003
Release 2.1 Data AspectsDecember 2003
  • NEESpop
    • Notebook to metadata repository connection made
    • Closer integration of notebook into CHEF
    • First release of experiment tool (based on ORST)
    • Retool data viewers to be completely driven by Metadata objects rather than their own objects
    • More fine grained access control
    • Enhanced data models
  • Tools
    • Ingestion tools released
    • A limited set of pre-release video/image tools
further releases
Further releases
  • Release 2.2 – March 04
    • Driven by your needs as we encounter them
    • Perhaps some “nice to haves” from the SI team
  • Release 3.0 – June 04
    • Very limited new functionality – maybe almost nothing new in the core components of the NEESpop