1 / 24

DILIGENT: from documents into computers to content across eInfrastructures

DILIGENT: from documents into computers to content across eInfrastructures. Andrea Manieri EGEE-UF – Manchester, 11 th May 2007. New ways of doing research. eScience (and eArts, and eHumanities) is changing the way research is done:

aricin
Download Presentation

DILIGENT: from documents into computers to content across eInfrastructures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DILIGENT: from documents into computers to content across eInfrastructures Andrea Manieri EGEE-UF – Manchester, 11th May 2007

  2. New ways of doing research • eScience (and eArts, and eHumanities) is changing the way research is done: • Soon will complete the digitisation transition and any type of data will be referred as DIGITAL OBJECT • Research is even more becoming Data-centric • Theories, experimentations, and simulations all based on availability and analysis of data • Data captured by instruments • Data generated by simulations • Data generated by sensor networks • But also collections and repositories of images, videos, etc. • All this data needs to be stored, handled, indexed, retrieved, annotated, preserved and shared … in a dynamic, secure and transparent way! Users in the wider Grid community - from science to business

  3. copy Digital Libraries Today • Many organizations that demand a Digital Library (DL) are small, distributed & dynamic • DLs perceived as a instrument for collaboration within a community • DLs often used to support temporary activities: • Courses, Exhibitions, Projects, … • Construction and management of a DL • Requires high investments and specialized personnel • Content production is very expensive • Multimedia handling requires powerful computational resources Users in the wider Grid community - from science to business

  4. The DILIGENT objective • Develop a DL Test-bed Infrastructure that allows dynamic virtual research organizations to create on-demand DLs • Shared computing • Shared storage • Multimedia, multi-type content • Application resources • With DILIGENT, communities are able to setup their own DL • Even if they do not have the financial and human resources • When and for the time they need it • With DL services empowered by underline Grid infrastructure Users in the wider Grid community - from science to business

  5. Key Concepts of DILIGENT • Service-oriented architecture • reusable components for other (eScience) applications • Integrating DL services on top of the EGEE infrastructure • Enhances existing Grid services with the functionality to • build, • operate and • maintain • transient virtual digital libraries Users in the wider Grid community - from science to business

  6. DILIGENT in a Nutshell From documents into computers to content across the eInfrastructure Users in the wider Grid community - from science to business

  7. EGEE EGEE-II EGEE-III gLite early adopter Users in the wider Grid community - from science to business

  8. The User Perspective… • Two scenarios: • IMPeCT: on Environmental Science • ARTE : on Humanities • Three relevant Users • ESA/ESRIN: • Scuola Normale Superiore di Pisa • Rai Educational • A nice four-minutes video… Users in the wider Grid community - from science to business

  9. How achieve all this? • An application frameworkfor the development of WSRF services which can be automatically deployed into a grid infrastructure • A service-based grid middleware for hosting and manage Web Services on grid nodes • A runtime environment • Allowing Federation of structured and unstructured data, by mean of • Providing a set of WSRF services implementing basic digital libraries functionalities • Opened to adding new WSRF services Users in the wider Grid community - from science to business

  10. The DILIGENT system ARTE IMPeCT DILIGENT EGEE PPS Users in the wider Grid community - from science to business

  11. How DILIGENT works: the user perspective Users in the wider Grid community - from science to business

  12. Community requirements • A Community needs to: • Access a certain amount (in size and number) of Content Repositories • Grant the access to a (large) number of users • Some DL functionalities • …for a limited period of time Users in the wider Grid community - from science to business

  13. Setting a new VDL • A Community Leader uses DILIGENT portal to create a new VDL • Authenticates himself (X.509 required) • Selects available contents or/and DL functionalities • Imports external Content granting access, indexing metadata, etc… • Adds other functionalities (implemented as WSRF services) Users in the wider Grid community - from science to business

  14. Selecting content, services… • The DILIGENT system • Selects the services implementing the functionalities requested • Deploys them (an adequate number of occurrences) on DILIGENT nodes • Keeps them alive and running for the required timeline • Instantiates a portal for the community Users in the wider Grid community - from science to business

  15. Administration of a new VDL • The Community Leader (now called VDL Admin) • Registers users • Assigns grants • Monitors the system • Uses workflow interface to create new compound services Users in the wider Grid community - from science to business

  16. What capabilities needed? • Federation of any type of data, metadata, etc. • Computing intensive methods for extracting features and metadata, indexing, etc • Ability to set up services, maintain for a certain time and then remove them • Ensure data security (for certain type of data) through watermarking and content-level access grants • Interoperability with existing grid-infrastructures (AuthN&AuthZ, job submission, file storage) Users in the wider Grid community - from science to business

  17. Useful features • Process Execution • A dynamic distributed workflow management system using WSRF services • Univ of Basel, dept of Computer Science • Managing transient services • A set of grid services – called Collective layer enabling the registration, deployment and administration of WSRF services, in a secure way • CNR Pisa, ISTI & Engineering Ingegneria Informatica s.p.a Users in the wider Grid community - from science to business

  18. startProcess process Execution • Process execution is realized in a distributed way by dedicated services [PES-process execution service] • Control dynamically shared by a set of PES instances • Services that can be invoked are • DILIGENT services • gLite jobs (via generic gLite Job wrapper) • External Web services Non-DILIGENT service DILIGENT service (with PES locally deployed) DILIGENT service (without local PES) gLite job gLite job wrapper Users in the wider Grid community - from science to business

  19. … … WS WS State State WS WS Managing transient services DHN Package Repository HW Dynamic deployment Users in the wider Grid community - from science to business

  20. … … DHN DHN State State State State HW HW WS WS WS WS Managing transient services Failure Recovery DHN HW Service provision continuity Users in the wider Grid community - from science to business

  21. Dynamic Load Balancing Dynamic Load Balancing State State State State State State State State State State WS WS WS WS WS WS WS WS WS WS CPU Usage CPU Usage CPU Usage CPU Usage 30% 90% 30% 90% Balancing utilization with head room Balancing utilization with head room DHN DHN DHN DHN DHN DHN DHN DHN DHN DHN HW HW HW HW HW HW HW HW HW HW Dynamic Load Balancing Dynamic Load Balancing Dynamic Load Balancing State State State State State State State State State State WS WS WS WS WS WS WS WS WS WS CPU Usage CPU Usage CPU Usage CPU Usage CPU Usage CPU Usage 30% 30% 30% 90% 90% 90% Balancing utilization with head room Balancing utilization with head room Balancing utilization with head room Managing transient services Users in the wider Grid community - from science to business

  22. Contributions to DL Community • A data model allowing the management of data, objects, complex objects, collections, metadata on such objects •  Univ. of Strathclyde Glasgow & CNR Pisa, ISTI • Transparent replication and partitioning of content using grid storage (on-going) •  Univ of Basel, dept of Computer Science • Distributed, grid-aware search service •  Univ. of Athens, FAST Search • Any further question to info@diligentproject.org Users in the wider Grid community - from science to business

  23. grid for content The Knowledge Infrastructure grid for services grid for computing Conclusions • DILIGENT is paving the way for new ways of doing research, demonstrating the availability of tools and technologies Users in the wider Grid community - from science to business

  24. Thanks! www.diligentproject.org Acknowledgements: - Luigi Fusco (ESA) - Donatella Castelli, Pasquale Pagano (CNR) - Roberta Faggian (CERN) - Heiko Schuldt (UNIBAS) Users in the wider Grid community - from science to business

More Related