Grids for geosensors geoscience and geoscientists
1 / 42

Grids for GeoSensors, GeoScience and GeoScientists - PowerPoint PPT Presentation

  • Updated On :

EarthScope CSIT Workshop March 25 2002 Grids for GeoSensors, GeoScience and GeoScientists PTLIU Laboratory for Community Grids Geoffrey Fox Computer Science, Informatics, Physics

Related searches for Grids for GeoSensors, GeoScience and GeoScientists

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Grids for GeoSensors, GeoScience and GeoScientists' - albert

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Grids for geosensors geoscience and geoscientists l.jpg

EarthScope CSIT Workshop March 25 2002

Grids for GeoSensors, GeoScience and GeoScientists

PTLIU Laboratory for Community Grids

Geoffrey Fox

Computer Science, Informatics, Physics

Indiana University, Bloomington IN 47404

uri="" email=""

Trends of importance l.jpg
Trends of Importance

  • Resources of increasing performance or functionality

    • Computers (ASCI, Earth Simulator to TeraGrid), storage, sensors, networks, PDA’s

  • Applications of increasing sophistication

    • Size, multi-scales, multi-disciplines

  • New algorithms and mathematical techniques

  • Computer science

    • Compilers, Parallelism, Objects, Components

  • Grid and Internet Concepts and Technologies

    • Enabling new applications

    • XML, Web Services,Portals, Collaboration

uri="" email=""

Projected top 500 until year 2009 l.jpg
Projected Top 500 Until Year 2009

  • First, Tenth, 100th, 500th, SUM of all 500 Projected in Time

Earth Simulator from Japan

uri="" email=""

Paci 13 6 tf linux teragrid l.jpg







= 32x 1GbE

32 quad-processor McKinley Servers

(128p @ 4GF, 8GB memory/server)

PACI 13.6 TF Linux TeraGrid

574p IA-32 Chiba City


256p HP




64 Nodes

1 TF

0.25 TB Memory

25 TB disk




32 Nodes

0.5 TF

0.4 TB Memory

86 TB disk

128p Origin



128p HP



HR Display & VR Facilities






92p IA-32










Black Diamond


Chicago & LA DTF Core Switch/Routers

Cisco 65xx Catalyst Switch (256 Gb/s Crossbar)








Juniper M160


500 Nodes

8 TF, 4 TB Memory

240 TB disk


256 Nodes

4.1 TF, 2 TB Memory

225 TB disk

Juniper M40

Juniper M40











Myrinet Clos Spine









Myrinet Clos Spine


1024p IA-32

320p IA-64

1176p IBM SP

Blue Horizon



= 64x Myrinet


= 32x Myrinet

1500p Origin

Sun E10K

= 32x FibreChannel

= 8x FibreChannel

10 GbE

32 quad-processor McKinley Servers

(128p @ 4GF, 12GB memory/server)

Fibre Channel Switch

16 quad-processor McKinley Servers

(64p @ 4GF, 8GB memory/server)

IA-32 nodes

Cisco 6509 Catalyst Switch/Router

uri="" email=""

The hpcc track l.jpg
The HPCC Track

  • The 1990 HPCC 10 year initiative was largely aimed at enabling large scale simulations for a broad range of computational science and engineering problems

  • It was in many ways a success and we have methods and machines that can (begin to) tackle most 3D simulations

    • ASCI simulations particularly impressive

    • DoE still putting substantial resources into basic software and algorithms from adaptive meshes to PDE solver libraries

  • Machines are still increasing in performance exponentially and should achieve petaflops in next 7-10 years

  • EarthScope community needs to harness these capabilities

    • Japan’s Earth Simulator activity major effort with large hardware and software (GEOFEM) efforts

uri="" email=""

Some hpcc advice to earthscope l.jpg
Some HPCC Advice to EarthScope

  • Important to build Sustainable modular software

  • Use MPI and openMP if needed for performance on shared memory nodes

  • Adaptive Meshes

  • Load Balancing

  • PDE Solvers including fast multipoles

  • Particle dynamics

  • Other areas such as datamining, visualization and data assimilation quite advanced but still significant research


Are well understoodto get high performanceparallel simulationsUse broad communityexpertise

uri="" email=""

Use of object technologies l.jpg
Use of Object Technologies

  • There is emerging HPCC component architecture allowing production of more modern libraries (integration Infrastructure)

    • DoE has very large CCA – Common Component Architecture – effort

    • Package software (“system and applications”) as distributed objects – not as traditional libraries

  • CORBA Java and Web Services are not naturally high performance as component models but OK for coarse grain objects (“full programs”)

  • As a language, C++ can be high performance but Java is not uniformly so (it is improving)

    • Fortran (including Fortran90) will continue to decline in importance and interest – the community should prefer not to use it

  • Not essential to write modules in object oriented language

    • It is essential to package modules in object framework

uri="" email=""

What is a web service i l.jpg
What is a Web Service I

  • A web service is a computer program running on either the local or remote machine with a set of well defined interfaces (ports) specified in XML (WSDL)

  • In principle, computer program can be in any language (Fortran .. Java .. Perl .. Python) and the interfaces can be implemented in any way what so ever

    • Interfaces can be method calls, Java RMI Messages, CGI Web invocations, totally compiled away (inlining) but

  • The simplest implementations involve XML messages (SOAP) and programs written in net friendly languages like Java and Python

  • Web Services separate the meaning of a port (message) interface from its implementation so CAN get high performance in spite of voluminous XML format

  • Enhances/Enables re-usable component model of ANY electronic resource

uri="" email=""

What is a web service ii l.jpg

PaymentCredit Card

WSDL interfaces





WSDL interfaces

What is a Web Service II

  • Web Services have important implication that ALL interfaces are XML messages based. In contrast

  • Web Services in some sense replace distributed object paradigms such as CORBA and Java but can wrap these other technologies as Web Services

    • We wrapped our CORBA + Java Computing Portal Gateway as Web services straightforwardly

uri="" email=""

Slide10 l.jpg

Web Service (WS)








Raw Data

Raw Data

(Virtual) XML Data Interface




XML WS to WS Interfaces

(Virtual) XML Knowledge (User) Interface

Render to XML Display Format

(Virtual) XML Rendering Interface


uri="" email=""

Classic grid architecture l.jpg



Classic Grid Architecture


Content Access


Middle TierBrokers Service Providers





Middle Tier becomes Web Services


Users and Devices

uri="" email=""

Examples of system web services i l.jpg
Examples of System Web Services I

  • OGSA (Open Grid Service Architecture)

    • Integrate Web Service and Grid Concepts and allows Globus to be implemented as Web Services

  • Audio-Video Conferencing as a Web Service

    • Integrates H323, SIP, JXTA (etc.) protocols by mapping to single XML Interface

    • Provides VRVS reflector model from Messaging Web Service

  • Messaging or Event Web Service provides intelligent routing and buffering of messages

  • Computing as a Web service

    • Job submittal, status, composition, data services, visualization

    • Performance WS allows access to distributed monitoring data, analysis, models, and final benchmarks with interoperable XML interfaces

uri="" email=""

Earthscope peer to peer grid community l.jpg
EarthScope Peer to Peer Grid Community


connected byXML messages

Distributed Scientists usingCollaboration Web Serviceto access/use Application Web Services

uri="" email=""

Gateway and web services l.jpg
Gateway and Web Services

  • We can use the Gateway Computing Portal as an example (

    • It is largely built using CORBA with a Java Server Pages front end

  • Several capabilities have been interfaced using WSDL

    • Job Submission (11 Methods including execute local and remote command, copy files etc. as well as Submit Job)

    • Manage WebFlow Session (67 Methods)

    • Generate Batch Script (just 1 method but two implementations developed – one at SDSC and one at Indiana – with UDDI to manage)

    • Each is one service – could have used finer grain services

    • Sample files are at

uri="" email=""

Wsdl abstractions l.jpg
WSDL Abstractions

  • WSDL abstracts a program as an entity that does something given one or more inputs with its results defined by streams on one or more outputs.

  • Functions are defined by method name and parametersmethodname(parm1,parm2, … parmN)

    • Where parameters are “Input” “Output” or both

  • In WSDL, we will have a Web Service which like a (Java or CORBA Program) can be thought of as a (distributed) object with many methods

    • Instead of a function call, the “calling routine” sends an XML message to the Web Service specifying methodname and values of the parameters

    • Note name of function is just another parameter

uri="" email=""

Wsdl message example l.jpg
WSDL Message Example

<message name="submitRequest">

<part name="xmljob" type="xsd:string"/>



<part name="response" type="xsd:string"/>


  • For the batch script service, we pass the XML description of the job as a string and get back the script as a string.

  • In general, any XML primitive or complex types can be used in messages.

    • We could improve our service by defining a BatchScript complex type.

uri="" email=""

Soap and gateway portal i l.jpg
SOAP and Gateway Portal I

  • Having specified service in WSDL, the run-time is implemented in SOAP which is “just” an XML header (info needed by transport – empty here) and body

  • Here is SOAP transported by HTTP message

  • This is execLocalCommand WSDL operation to run one particular command (ls) on current WebFlow directory

HTTP Header

Argument of operation

Specify ls as

SOAP Envelope and body

uri="" email=""

Examples of system web services ii l.jpg
Examples of System Web Services II

  • Education as a Web Service

    • One of easiest to do as object standards well defined (IMS) and little performance issues

    • Grading, Homework submission, registration, assessment etc.

  • Universal Access and Web Services

    • As Web Services allow multiple implementation of a particular interface, one can adjust to needs of particular clients (PDA v. versus, impaired sight etc.)

    • Can build custom implementations of certain web services for particular communities but re-use others

  • Collaborative Web Services

    • As interfaces all message based, much easier to share Web Services than other applications (PowerPoint interface is NOT message based and harder to share than server app)

uri="" email=""

Education as a web service l.jpg
Education as a Web Service

  • Can link to Science as a Web Service and substitute educational modules

  • “Learning Object” XML standards already exist from IMS/ADL – need to update architecture

  • Web Services for virtual university include:

  • Registration

  • Performance (grading)

  • Authoring of Curriculum

  • Online laboratories for real and virtual instruments

  • Homework submission

  • Quizzesof various types (multiple choice, random parameters)

  • Assessment data access and analysis

  • Synchronous Delivery of Curricula

  • Scheduling of courses and mentoring sessions

  • Asynchronous access, data-mining and knowledge discovery

  • Learning Plan agents to guide students and teachers

uri="" email=""

Sensor web service l.jpg
Sensor Web Service

Distributed Sensor Web Service

Out Web Service portsUniversal sensor accessfor people/computers

In Web Service portsDifferent formatSensor Data

uri="" email=""

Application web services l.jpg






Build as multiple interdisciplinaryPrograms

Build as multiple Filter Web Services

Sensor Data as a Webservice (WS)

Simulation WS

Simulation WS

Data Analysis WS

Data Analysis WS

Sensor ManagementWS

Visualization WS

Visualization WS

Application Web Services

  • Note Service model integrates sensors, sensor analysis, simulations and people

  • An Application Web Service is a capability used either by another service or by a user

    • It has input and output ports – data is from users, sensors or other services

    • Big services built hierarchically from “basic” services

SLE (space Link Extension) as a WS

uri="" email=""

E science is xml specified resources connected by xml specified messages l.jpg





e-Science is XML Specified Resourcesconnected by XML specified messages


Or Event







Implementation of resource and

connection may or may not be XML

uri="" email=""

E science is just a pile of xml l.jpg


e-Science is just a pile of XML

  • Each leaf is a piece of XML either defining a nugget of information and/or containing links to other XML or “raw resources”

uri="" email=""

Xml rss specification of information nuggets l.jpg
XML (RSS) Specification of Information Nuggets

  • <itemrdf:about="">

  • <title> Processing Inclusions with XSLT </title>

  • <link></link>

  • <description>

  • Processing document inclusions with general XML tools can be

  • problematic. This article proposes a way of preserving inclusion

  • information through SAX-based processing.

  • </description>

  • </item>

  • <item rdf:about="">

  • <title> Putting RDF to Work </title>

  • <link></link>

  • <description>

  • Tool and API support for the Resource Description Framework

  • is slowly coming of age. Edd Dumbill takes a look at RDFDB,

  • one of the most exciting new RDF toolkits.

  • </description>

  • </item>

  • </rdf:RDF>

Example of XML meta-data in the “pile”pointing to other (outside) resources

uri="" email=""

Distributed information l.jpg
Distributed Information

Actually the XML is distributed

all around in a dynamic Grid

uri="" email=""

Structured xml information l.jpg
Structured (XML) Information

Note XML specifiesboth internal andexternal nodes of tree






uri="" email=""

Matching information service providers and consumers i l.jpg
Matching Information/Service Providers and Consumers I

  • Classic Centralized Approach

  • Those with services publish information as to location – this is percolated up and down the tree of brokers

  • At simplest, publish location; better publish location and meta-data allowing easier discovery of value

  • Those wanting service, look it up using either

    • Some search of information registered with brokers

    • A search using a system like Google

    • Because they were told some key

  • Like using an encyclopedia; very reliable and fast for well established information

uri="" email=""

Unstructured and structured xml l.jpg

Hoosier National Forest showingstructured trees and a Gallimaufry

of unstructured leaves (fall 2001)

Unstructured and Structured XML






“mess” can be multiple levels of tree

uri="" email=""

Peer to peer grid l.jpg





Integrate P2P

and Grid/WS

Peer to Peer Grid


Web Service Interfaces

Web Service Interfaces


Peer to Peer Grid

uri="" email=""

Matching information service providers and consumers ii l.jpg
Matching Information/Service Providers and Consumers II

  • Peer-to-peer Approach (or how to search the “mess”)

  • Those with services publish XML advertisements to their friends; their friends may forward it to other friends

  • Those wanting a service, publish an XML request to a chosen set of friends

  • Friends use their personal idiosyncratic approach to matching requests with advertisements and to choosing who else should be asked

  • Analogous to way communities exchange information as in a meeting like this

  • Uncertain reliability but scales well (communities intra-exchange information independently) and supports rapidly varying information (Web Services)

  • Allows many different approaches – EarthScope imposes interfaces NOT analysis methods

uri="" email=""

Grid p2p use of internet i l.jpg

Rival Estimate

MainlyDigital Video


Grid/P2P Use of Internet I


Global Grid Forum Toronto Feb 18 2002

uri="" email=""

Grid p2p use of internet ii l.jpg

S2S Server to Server

Digital Video“on demand”

Grid/P2P Use of Internet II

uri="" email=""

Semantic grid digital brilliance i l.jpg
Semantic Grid & Digital Brilliance I

  • The (XML) advertisement-request matching provides a publish-subscribe linkage between resources – these are people, computers and raw/processed data

  • The richer the meta-data, the more precise the linkage

    • This is spirit of Semantic Web – RDF/DAML/OIL metadata enables meaningful linkage

  • In a physics analogy, resources can be thought of as spins and the meta-data induced linkage as forces or interactions

  • Phase transitions will occur when “enough” resources are linked – one will get associated spins to align in the direction of new knowledge

    • Term this digital brilliance

uri="" email=""

Semantic grid digital brilliance ii l.jpg
Semantic Grid & Digital Brilliance II

  • This suggests ways of quantifying value of metadata induced linkages and ways of identifying where one “should” add more resource specifications

  • Note that related resources are not necessarily directly connected but rather messages are forwarded through friends

  • Study of Peer to Peer networks teach us that we can build “small worlds” where distance between resources is logarithmic in number of nodes

  • This physics based picture provides an interesting underlying formalism to give a theory of e-Science ….

    • All you need to do is to build a lot of XML Meta-data specification wizards

uri="" email=""

Semantic grid digital brilliance iii l.jpg
Semantic Grid & Digital Brilliance III

  • EarthScope Collaboratory consists of a set of connected “spins” (being a physicist; resources if I was W3C)

  • Resources are anything with a digital signature

    • Raw data, Analysers, Simulators, Simulations, Processed Information, Extracted Knowledge, Scientists ….

  • The linkage of Earthquake Fault Simulator Web Service to the Greens Function Solver Web Service is as program to subroutine; must have agreement on both syntax and Semantics

  • The linkage of Granular Physics model to (my) remark that Los Alamos has interesting new simulation technology is less precise

  • So linkages with very precise ontologies and those which are more qualitative are both part of Semantic Grid

uri="" email=""

Portals and web services l.jpg
Portals and Web Services

  • Web Services allow us to build a component model (see CCA) for resources.

  • Each resource naturally has a user interface (which might be customized for user)

  • Web Service <--> Portlet

  • Natural to use a component model for portal building displayed web page from collection of portlets

    • So can customize each portlet and customize which portlets you want

  • Apache Jetspeed seems good open source technology supporting this model

    • JSP model is better than say a client-side Java integration in that also message based so this is “Portal as a Web Service”

uri="" email=""

Jetspeed computing portal choose portlets l.jpg

4 available portletslinking to Web ServicesI choose two

Jetspeed Computing Portal: Choose Portlets

uri="" email=""

Choose portlet layout l.jpg
Choose Portlet Layout

Choose 1-column Layout

Original 2-column Layout

uri="" email=""

Two computing portlets l.jpg
Two Computing Portlets

uri="" email=""

Earthscope csit strategy l.jpg
EarthScope CSIT Strategy

  • Make a list of resources with a hierarchical arrangement

    • People, Places, Results (Publications, meeting archives, Simulation Output), Activities, Sensors (Instruments), Data (raw and processed), Earth features, Computers, Software

  • Decide on component (Web Service) model and URI labeling (earthscope://devices/satellites/year/label …)

    • Respect performance requirements

    • Design so modules can be re-used, re-arranged and replaced for outreach (education)

  • Study related CSIT architectures of other fields

    • Grid Forum, PACI, ASCI for computing issues

    • W3C Web Consortium for basic IT infrastructure

    • openGIS XMML for related fields

    • IMS for Education

uri="" email=""

Earthscope hpcc strategy l.jpg
EarthScope HPCC Strategy

  • Decide what services are well enough understood and useful enough to be encapsulated as application Web Services

    • Parallel FEM Solvers

    • Visualization

    • Parallel Particle Dynamics

    • Access to Sensor Data

    • Image Processing

  • Make services as small as possible – smaller is simpler and more sustainable but with higher communication needs

    • Compose large services from smaller ones

  • Design Portals and portal components that allow one to manipulate services – set parameters, compose, invoke

  • Install chosen System Web Services (job submit, performance, queue) on central machines and local clusters

    • Make certain infrastructure supports compute, data, middleware needs

    • Set necessary hardware/software meta-data

uri="" email=""

Earthscope it strategy l.jpg
EarthScope IT Strategy

  • Design an internal EIF (EarthScope Internal Framework) defining architecture and interface standards of internal Web Services and data structures

  • Design EEF (EarthScope External Framework) which maps external raw data into sensor web services

  • Support diverse set of explorations as many new approaches to Earth Science enabled by EarthScope

  • Choose some appropriate (mix of) middleware frameworks

    • .net, IBM, BEA, Sun, Oracle

  • Look at special requirements for key system services

    • Hardware/Data systems (new and legacy issues)

    • Security

    • Collaboration including Audio/Video conferencing

    • Peer-to-peer networking

  • Develop necessary meta-data wizards

uri="" email=""