1 / 15

Supported by EU projects

Open Data in Agriculture. Hands-on with data infrastructures that can power your agricultural data products. 12/12/2013 Athens, Greece. Supported by EU projects. Tutorial on how to use from external tools or build new complex data processing workflows ( example: data aggregation).

alma
Download Presentation

Supported by EU projects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Open Data in Agriculture Hands-on with data infrastructures that can power your agricultural data products 12/12/2013 Athens, Greece Supported by EU projects

  2. Tutorial on how to use from external tools or build new complex data processing workflows (example: data aggregation) Robert Lovas MTA SZTAKI, Hungary

  3. Intro: user scenario • a user (information manager) wants to create a data set on-the-fly that will include specific filtered subsets of existing data sources, e.g. CIARD RING • E.g. covering a particular geographical region, time period, type of resources • use agINFRA-powered RING panel (Drupal module) to select desired data sources and relevant metadata properties, facets, formats • aggregation workflow executed when grid resources are available, notifying user to collect produced data set when ready • an advanced user wants to develop further the existing workflows or create new complex data processing workflows

  4. Objectives This presentation aims to provide • illustration how an agINFRA powered data aggregation works with simple but powerful interface to • select targets, execute complex workflow, and provide the downloadable outputs (data sets on the fly) • overview on designing workflows for data processing with user access and documentation details

  5. Targets from CIARD RING vs. manual settings

  6. Checking progress (with details)

  7. Executionfinished recordscan be downloaded

  8. The background Steps: • Configuration • Starting aggregation (workflowsubmissionthroughRemote API ) • Enactingtheworkflow (Initharvests,etc.doingone-by-one) • As a step of theworkflow, uploadingtheharvesteddatato Grid Storage LFN(using Robot certificate) • RegisteringthedatasetinagCrouchDB • Retrievingthe HTTP location • Provide the downloadable datasets

  9. Aggregation workflow

  10. Workflow support with gUSE (www.guse.hu)

  11. agINFRA Science Gateway for workflows Liferay-basedgUSE/WS-PGRADE for creating and running workflows  http://aginfra-portal.lpds.sztaki.hu/liferay-portal-6.0.5/ • or on-demand deployment • agINFRA VO is accessible (ca. 3500 CPU cores, 900 TB storage)

  12. Example: REST-specific featuresin job settings

  13. Documentation of gUSE For workflows Used from the GUI

  14. New! Request form for a new instance of a workflow engine (+ workflows and other components) on the Cloud  http://www.desktopgrid.hu/oc-public-aginfra

  15. Thank you! Robert Lovas MTA SZTAKI Robert.Lovas@sztaki.mta.hu

More Related