neesgrid data technologies n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
NEESgrid Data Technologies PowerPoint Presentation
Download Presentation
NEESgrid Data Technologies

Loading in 2 Seconds...

play fullscreen
1 / 24

NEESgrid Data Technologies - PowerPoint PPT Presentation


  • 84 Views
  • Uploaded on

NEESgrid Data Technologies. Charles Severance January 8, 2004 NSF Site Visit. NEESgrid Data - Value Proposition. An RDF like store – Referential integrity long-term flexibility Seamless data and meta data transport Smooth integration of data with meta data Extensible tools

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'NEESgrid Data Technologies' - lexi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
neesgrid data technologies

NEESgrid Data Technologies

Charles Severance

January 8, 2004

NSF Site Visit

neesgrid data value proposition
NEESgrid Data - Value Proposition
  • An RDF like store – Referential integrity long-term flexibility
  • Seamless data and meta data transport
  • Smooth integration of data with meta data
  • Extensible tools
  • Involved with sites through Experiment Based Deployment
neesgrid data current elements
NEESgrid Data – Current Elements
  • Local Repository
  • Central Repository
  • JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services
    • NEES File Management Services
    • NEES Meta Data Services
    • NEES Data Mapping Services
  • Data Viewers
    • Streaming (numeric, X/Y graph)
    • Stored (X/Y graph, 2-D structure, video)
current elements
Current Elements

Data Acquisition

NEESpop

Local

Repository

Grid and Web Services

Data/MD

Ingest

Tools

API

Data

Teamlets

API

Workstation

NEESdata

Data

Servlets

API

Data tools

Mapping

Central

Repository

Data

Teamlets

API

Data viewers

a simple experimental scenario
A Simple Experimental Scenario

Developer System

MD

MD

M

M

Data

DAQ System

Test Specimen

API

Labview

Glue

Researcher System

Data

Data

mappings and the data viewer
Mappings and the Data Viewer
  • NSDS (ISO 8601 Time channel)
  • Column data with time recorded as a column
  • Column – generate time
  • Column – generate time – trigger filter

Channel units: g,g,in,kip

Time ATL1 ATT1

2002-11-13T15:48:55.26499 -0.006409 0.004272

2002-11-13T15:48:55.36499 -0.005798 -0.003662

100.000

0.435 0.161 -1.016 -0.981

0.430 0.161 -1.016 -0.977

0.435 0.161 -1.016 -0.977

public class NEESDataMap

{

public static boolean repoMap(File mainFile,

File mappingFile, String mapping)

{

// Code here

}

}

nees metadata representation
NEES Metadata Representation
  • NEES Markup Language (NEESML)
    • Provides an RDF-like structure capable of representing semantic information
    • XML is the syntax which is used
    • Logic is more “object oriented”
      • Can define objects
      • Can create objects
      • Can reference objects
  • Meta data is many different things….
  • Goal if we EVER want to build reusable data tools, we have to represent the semantics inside the meta data rather than just the information
neesml
NEESML

Table 1: Primitive types in NEESML

java apis
JAVA APIs
  • http://neesbox.ncsa.uiuc.edu/chef/doc/repo/api/
remaining work
Remaining Work
  • Second Generation Repository API
  • Project Browser
  • Electronic Notebook
  • Data Turbine
  • Video as data
  • Schema/XML Ingestion
  • RDF Model/Data Ingestion
  • Curation Tools ***
repository api
Repository API
  • The NFMS and NMDS APIs are being combined into a single Repository API
    • Rich support for access control
  • Access control will use Community Authentication Service (CAS) from the Grid technology
  • Aligning with JSR-170 Java Content Repository
  • http://www.jcp.org/en/jsr/detail?id=170
project browser
Project Browser
  • Joint effort between the NEESgrid SI team and Oregon State technical developers
  • Based on a project browser prototype at Oregon State University
  • Provides a user friendly interface to Metadata elements - compliment to the project browser and electronic notebook
  • http://nees.orst.edu/prototype_4/
electronic notebook
Electronic Notebook
  • Collaborative effort with the DOE SciDAC
    • Electronic notebook - metadata entry
    • Data mapping
    • Data provenance
    • Data display
    • Slide data/metadata jakarta.apache.org/slide/
  • Ultimate integration will be via JSR-170
  • www.scidac.org/SAM/
  • collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html
data turbine
Data Turbine
  • Commercial, free data streaming toolkit
  • Developedby NASA
data turbine cont
Data Turbine (cont)
  • Existing data viewers will be adapted to access and display data from data turbine
  • Data acquisition software will be adapted to place information in Data Turbine Channels
  • Metadata elements will be developed to represent data turbine live, stored, and derived channels
  • New efforts (video as data) will be developed from the ground up using Data Turbine
  • outlet.creare.com/rbnb/
video as data
Video as Data
  • Follow on to initial demonstration at ORST
  • Experiment based development: Minnesota
  • Design phase nearly complete
  • Joint effort, NEESGrid SI, ORST, Minnesota, UC Davis, Texas, and others as design solidifies
video as data user views
Video as data: User Views

Still Image / Camera Control

Still Image Viewer

^

<

>

^

~

<

>

<

>

Thumbnail + Audio + Data

DT Main System

Data Viewer

<

>

+

Camera Control

Gateway

schema xml ingestion
Schema/XML Ingestion
  • Several data efforts are expressing their data/models in Schema/XML (Cosmos, etc)
  • We are developing capabilities to parse XML and automatically extract relevant metadata for the repository
  • The entire XML file will be stored as data
  • This allows data developers to use tools like XMLSpy to develop their models.
rdf integration
RDF Integration
  • Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF.
  • RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF
  • We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository
curated data tools
Curated Data Tools ***
  • Still evolving fine-grain requirements with community
    • Sites
    • Consortium DSAC has this as its focus
  • Some expected minimum requirements
    • Transfer between repositories
    • Workflow - implemented as ACLs (incoming, in-progress, published)
    • Will be extensions to Repository browser as well as a simple workflow tool
conclusion
Conclusion
  • We are focusing on both core elements and the application of those elements
  • We are engaging the sites increasingly in the going forward development process
  • We have a lot of work - some of these efforts will continue post-transition with the sites taking an increasing role in the development