January 23 2006 ilkay altintas
Download
1 / 17

January, 23, 2006 Ilkay Altintas - PowerPoint PPT Presentation


  • 133 Views
  • Updated On :

New Developments in Kepler . January, 23, 2006 Ilkay Altintas. Kepler System Architecture. Authentication. GUI. …Kepler GUI Extensions…. Vergil. Documentation. Smart Re-run / Failure Recovery. Provenance Framework. Kepler Object Manager. SMS. Type System Ext. Actor&Data

Related searches for January, 23, 2006 Ilkay Altintas

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'January, 23, 2006 Ilkay Altintas' - taite


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
January 23 2006 ilkay altintas l.jpg

New Developments in Kepler

January, 23, 2006

Ilkay Altintas


Kepler system architecture l.jpg
Kepler System Architecture

Authentication

GUI

…Kepler GUI Extensions…

Vergil

Documentation

Smart

Re-run /

Failure

Recovery

Provenance

Framework

Kepler

Object

Manager

SMS

Type

System

Ext

Actor&Data

SEARCH

Kepler

Core

Extensions

Ptolemy


Joint authentication framework l.jpg
Joint Authentication Framework

  • Requirements:

    • Coordinating between the different security architectures

      • GEON uses GAMA which requires a single certificate authority.

      • SEEK uses LDAP with has a centralized certificate authority with distributed subordinate Cas

    • To connect LDAP with GAMA

    • Coordinating between 2 different GAMA servers

    • Single sign-on/authentication at the initialize step of the run for multiple actors that are using authentication

      • This has issues related to single GAMA repository vs multiple, and requires users to have accounts on all servers.

      • Kepler needs to be able to handle expired certificates for long-running workflows and/or for users who use it for a long time.

      • A trust relation between the different GAMA servers must be established in order to allow for single authentication.


Functional prototype completed l.jpg
Functional Prototype Completed

  • APIs and tests cases in place

  • More work required on certificate renewal and multiple server access


Vergil is the gui for kepler l.jpg
Vergil is the GUI for Kepler

Actor Search

Data Search

  • Actor ontology and semantic search for actors

  • Search -> Drag and drop -> Link via ports

  • Metadata-based search for datasets


Actor search l.jpg
Actor Search

  • Challenges:

    • Building/searching a repository …

    • Making changes to MoML (see KAR)

    • GUI changes

    • Ontology management

  • Kepler Actor Ontology

    • Used in searching actors and creating conceptual views (= folders)

  • Currently 160 Kepler actors added!


Data search and usage of results l.jpg
Data Search and Usage of Results

  • Kepler DataGrid

    • Discovery of data resources through local and remote services

      • SRB,

      • Grid and Web Services,

      • Db connections

    • Registry of datasets on the fly using workflows


Vergil updates l.jpg
Vergil Updates

  • To make it more useful to the user

    • Updated actor icons

    • Menu redesign

  • Improve readability

  • Develop cohesive visual language

  • Follow standard HF principles

  • Improve organization

Composite

DB Query

Computation or Operation

Transformation

Filter

File Operation

Web Service


Kepler archives l.jpg
Kepler Archives

  • Purpose: Encapsulate WF data and actors in an archive file

    • … inlined or by reference

    • … version control

      • More robust workflow exchange

      • Easy management of semantic annotations

      • Plug-in architecture (Drop in and use)

      • Easy documentation updates

  • A jar-like archive file (.kar) including a manifest

  • All entities have unique ids (LSID)

  • Custom object manager and class loader

  • UI and API to create, define, search and load .kar files


Kar file example l.jpg
KAR File Example

<entity name="Multiply or Divide" class="ptolemy.kernel.ComponentEntity">

<property name="entityId" value="urn:lsid:localhost:actor:80:1" class="org.kepler.moml.NamedObjId"/>

<property name="documentation" class="org.kepler.moml.DocumentationAttribute"></property>

<property name="class" value="ptolemy.actor.lib.MultiplyDivide" class="ptolemy.kernel.util.StringAttribute">

<property name="id" value="urn:lsid:localhost:class:955:1" class="ptolemy.kernel.util.StringAttribute"/></property>

<property name="multiply" class="org.kepler.moml.PortAttribute">

<property name="direction" value="input" class="ptolemy.kernel.util.StringAttribute"/>

<property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>

<property name="isMultiport" value="true" class="ptolemy.kernel.util.StringAttribute"/></property>

<property name="divide" class="org.kepler.moml.PortAttribute">

<property name="direction" value="input" class="ptolemy.kernel.util.StringAttribute"/>

<property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>

<property name="isMultiport" value="true" class="ptolemy.kernel.util.StringAttribute"/>

</property>

<property name="output" class="org.kepler.moml.PortAttribute">

<property name="direction" value="output" class="ptolemy.kernel.util.StringAttribute"/>

<property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>

<property name="isMultiport" value="false" class="ptolemy.kernel.util.StringAttribute"/></property>

<property name="semanticType00" value="http://seek.ecoinformatics.org/ontology#ArithmeticMathOperationActor" class="org.kepler.sms.SemanticType"/>

</entity>


Kepler object manager l.jpg
Kepler Object Manager

  • Designed to access local and distributed objects

  • Objects: data, metadata, annotations, actor classes, supporting libraries, native libraries, etc. archived in kar files

  • Advantages:

    • Reduce the size of Kepler distribution

      • Only ship the core set of generic actors and domains

    • Easy exchange of full or partial workflows for collaborations

    • Publish full workflows with their bound data

      • Becomes a provenance system for derived data objects

        => Separate workflow repository and distributions easily


Initial work on provenance framework l.jpg
Initial Work on Provenance Framework

  • Provenance

    • Track origin and derivation information about scientific workflows, their runs and derived information (datasets, metadata…)

  • Need for Provenance

    • Association of process and results

    • reproduce results

    • “explain & debug” results (via lineage tracing, parameter settings, …)

    • optimize: “Smart Re-Runs”

  • Types of Provenance Information:

    • Data provenance

      • Intermediate and end results including files and db references

    • Process (=workflow instance) provenance

      • Keep the wf definition with data and parameters used in the run

    • Error and execution logs

    • Workflow design provenance (quite different)

      • WF design is a (little supported) process (art, magic, …)

      • for free via cvs: edit history

      • need more “structure” (e.g. templates) for individual & collaborative workflow design


Kepler provenance recording utility l.jpg
Kepler Provenance Recording Utility

  • Parametric and customizable

    • Different report formats

    • Variable levels of detail

      • Verbose-all, verbose-some, medium, on error

    • Multiple cache destinations

  • Saves information on

    • User name, Date, Run, etc…


Provenance possible next steps l.jpg
Provenance: Possible Next Steps

  • Provenance Meeting: Last week at SDSC

    • Deciding on terms and definitions

    • .kar file generation, registration and search for provenance information

    • Possible data/metadata formats

    • Automatic report generation from accumulated data

    • A GUI to keep track of the changes

    • Adding provenance repositories

    • A relational schema for the provenance info in addition to the existing XML


What other system functions does provenance relate to l.jpg
What other system functions does provenance relate to?

  • Failure recovery

  • Smart re-runs

  • Semantic extensions

  • Kepler Data Grid

  • Reporting and Documentation

  • Authentication

  • Data registration

Re-run only the updated/failed parts

Guided documentation generation an updates


Hot topics in kepler l.jpg
Hot Topics in Kepler

http://kepler-project.org/Wiki.jsp?page=HotTopics