A Quick Update on the TSI and PIW workflows
Download
1 / 12

Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, - PowerPoint PPT Presentation


  • 89 Views
  • Uploaded on

A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 https://www-casc.llnl.gov/sdm/. Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, Lots of Others…. UCRL-PRES-209990. Example workflows available for download

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, ' - hovan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

A Quick Update on the TSI and PIW workflowsSDM All Hands March 2-3, 2005https://www-casc.llnl.gov/sdm/

Terence Critchlow,

Xiaowen Xin,

Bertram Ludaescher, Ilkay Altintas

Mladen Vouk, Zengang Cheng,

Lots of Others….

UCRL-PRES-209990


Web page is up and running

Example workflows available for download

Documentation exists and is improving

List of (somewhat dated) publications

https://www-casc.llnl.gov/sdm/

Web page is up and running


Tsi 1 workflow

Update notebook

Update notebook

Update web page

Update web page

Extract

Update notebook

Get Variables

Create neutrino vars

Create Chem vars

Update web page

Remap coordinates

Derive other vars

Write diagnostic file

Generate plots

Update notebook

Tool-1

Tool-2

Update web page

Tool-3

Tool-4

TSI-1 Workflow

Error notification

Web interface for Simulation configuration submission

Identify config files

Move files to correct locations

Submit batch request at NERSC

Are files consistent

Delay

Generate directory listing

Are there new files?

Identify complete files

Job still running?

Delay

Check on job status

Empty list

Count files

Transfer completed correctly

Delete file

Transfer files to HPSS

Transfer completed correctly

Transfer files to SB

Generate movie

Generate thumbnails


Update notebook

Update notebook

Update web page

Update web page

Extract

Update notebook

Get Variables

Create neutrino vars

Create Chem vars

Update web page

Remap coordinates

Derive other vars

Write diagnostic file

Generate plots

Update notebook

Tool-1

Tool-2

Update web page

Tool-3

Tool-4

TSI-1 Workflow

Remote job submission - done

Error notification

Web interface for Simulation configuration submission

Identify config files

Move files to correct locations

Submit batch request at NERSC

Are files consistent

Delay

Generate directory listing

Are there new files?

Identify complete files

Job still running?

Delay

Check on job status

Empty list

Count files

Transfer completed correctly

Delete file

Transfer files to HPSS

Transfer completed correctly

Transfer files to SB

Generate movie

Generate thumbnails


Update notebook

Update notebook

Update web page

Update web page

Extract

Update notebook

Get Variables

Create neutrino vars

Create Chem vars

Update web page

Remap coordinates

Derive other vars

Write diagnostic file

Generate plots

Update notebook

Tool-1

Tool-2

Update web page

Tool-3

Tool-4

TSI-1 Workflow

Real-time file transfer - done

Error notification

Web interface for Simulation configuration submission

Identify config files

Move files to correct locations

Submit batch request at NERSC

Are files consistent

Delay

Generate directory listing

Are there new files?

Identify complete files

Job still running?

Delay

Check on job status

Empty list

Count files

Transfer completed correctly

Delete file

Transfer files to HPSS

Transfer completed correctly

Transfer files to SB

Generate movie

Generate thumbnails


Update notebook

Update notebook

Update web page

Update web page

Extract

Update notebook

Get Variables

Create neutrino vars

Create Chem vars

Update web page

Remap coordinates

Derive other vars

Write diagnostic file

Generate plots

Update notebook

Tool-1

Tool-2

Update web page

Tool-3

Tool-4

TSI-1 Workflow

Data analysis – not yet started

Error notification

Web interface for Simulation configuration submission

Identify config files

Move files to correct locations

Submit batch request at NERSC

Are files consistent

Delay

Generate directory listing

Are there new files?

Identify complete files

Job still running?

Delay

Check on job status

Empty list

Count files

Transfer completed correctly

Delete file

Transfer files to HPSS

Transfer completed correctly

Transfer files to SB

Generate movie

Generate thumbnails


Tsi 1 workflow1
TSI-1 Workflow

  • Developed modular, configurable, actors that can be used elsewhere

    • File filtering

      • Use regular expressions to filter file list

    • File transfer

      • User specified command transfers files between two remote locations

    • SSH

      • Executes remote command, catches errors

    • Retry

      • Repeat an action until successful or until user defined limit is reached



Submit actor
Submit actor

Debug actor lets you test without actually submitting job

Submit job using llsubmit

Return the job number


Piw workflow

Added logging

Modularized workflow a bit more

Added debugging and testing code

New visualization actor

PIW Workflow


Next steps
Next Steps

  • Working through deadlock / race condition problems in Ptolemy PN director

    • Discovered because of test suite / debug actor

  • Need to be able to “merge” multiple streams

  • Data analysis section of TSI-1 workflow

  • More documentation


Next talks
Next talks

  • Steve Parker

    • SCIRun and SPA integration status

  • Mladen Vouk

    • On Large Data Flow Scientific Workflows: An Astropysics Case Study

  • Bertram Ludaescher

    • Kepler / SPA Extensions for Scientific Workflows – Now and Upcoming


ad