1 / 38

GridLab Applications

GridLab Applications. Gabrielle Allen, Zhou Lei, Archit Kulshrestha Center for Computation & Technology, LSU. Modeling Black Holes. Einstein’s equations of General Relativity: G ab = 0 One of the most complex set of equations in physics Mixed hyperbolic-elliptic initial value problem

duante
Download Presentation

GridLab Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GridLab Applications Gabrielle Allen, Zhou Lei, Archit Kulshrestha Center for Computation & Technology, LSU

  2. Modeling Black Holes • Einstein’s equations of General Relativity: Gab = 0 • One of the most complex set of equations in physics • Mixed hyperbolic-elliptic initial value problem • 3 Dimensional, multiple scales • 12 or more equations, with thousands of terms • Require diverse distributed collaborations & many large scale simulations: • Parallelization, optimization • Parallel IO, data management, viz • Checkpointing, steering, interaction Cactuscode.org t=100 t=0

  3. Application Scenarios • Intelligent Automatic Parameter Surveys • Dynamic Migration: faster/cheaper/bigger machine • Multiple Universe: create job clone to investigate steered parameter • Automatic Component Discovery and Loading: Needs of job change • Automatic Convergence Testing • Look Ahead: Spawn off and run coarser resolution to predict likely future • Spawn Independent/Asynchronous Tasks • send to cheaper machine, main simulation carries on • Routine Profiling • best machine/queue, choose resolution parameters based on queue • Dynamic Load Balancing and Optimization: inhomogeneous loads, multiple grids, network and cache hierarchies.

  4. Application Scenarios • Automatically record details about simulations and results. • Store locations of output data files, along with searchable metadata. • Ontologies for describing application area: use for meta data, connection to journal publications etc. • All done in a community accessible way

  5. SCOOP: SURA Coastal Ocean Observing Program (twiki.sura.org) NOAA/ONR Data coming in from sensors all over Gulf: Realtime Operational Grid Feeds in to models on LSU SuperMike and other Grid sites Indication of strengthening storm changing direction: Must Act! Multiple simulations, expertise, and data connected together: FL, LA, notification, steering sensors Early warning, preparation: Insurance industry, petroleum industry, local economy all save millions, lives saved Hurricanes on the Grid LSU, GoMOOS, Texas A&M, UaH, UNC/MCNC, U. Florida, U. Miami, VIMS

  6. Ubiquitous Computing and Monitoring System (UCoMS) • $2.4M DOE/BOR project • ULL, LSU, Southern, Landmark • Petroleum Engineering, CS, ULL • Deploy sensor networks across Gulf • Data collected to provide input to simulations, tasks farmed out • Results collected (transmitted back) • http://www.ucoms.org

  7. Dynamic Grid Computing LRZ

  8. Application Tools for Grids • Application developers need the right kinds of tools and environments to use Grids • APIs, toolkits, debuggers, … • Focused on the application requirements (MPI, Fortran, legacy, ….) • Hide the complexities of the Grid • Work in a familiar way (avoid new paradigms) • Functioning and reliable heterogeneous testbed environments for developing and testing • Support new and dynamic application scenarios, as generic as possible.

  9. GridLab Project • EU Funded (5th Framework): Jan 2002 • Partners: PSNC, AEI, ZIB, MASARYK, VU, SZTAKI, ISUFI, Cardiff, NTUA, Chicago, ISI, Wisconsin, Sun, Compaq • GridSphere Portal • Applications • Numerical Relativity (Cactus) • Gravitational Waves (Triana) • Mobile Users • Europe-wide Test Bed (Linux, SGI, Hitachi, …) • Grid Application Toolkit (GAT) • Range of Grid Services • http://www.gridlab.org

  10. Grid Application Toolkit (GAT) • Application oriented access to Grid through a standard API • GATFile_Move • GATResource_Find • GATFile_Logical • Independent of Grid infrastructure and available services. • C, C++, Java, Python, [Perl/Fortran] • All platforms: Unix, Windows • API driver for GGF SAGA-RG

  11. GAT Motivation • Use of Grids by applications is lagging behind the development of Grid technologies, big jump from prototypes and demonstrations to real production use of Grids needed by applications. • Problems: • Missing or immature grid services. • Changing environment, new software versions. • Different and evolving interfaces to the “grid”. • “Functioning” Grid is needed to develop applications. • Interfaces are not simple for scientific application developers.

  12. Example: Remote File Copying Application “Copy my file from there to there ..” SOAP WSDL Corba OGSA Other Monitoring Profiling Information Logging Security Notification Resource Management Application Manager Migration Data Management GLOBUS Other Grid Infrastructure?

  13. Copy a File: Globus GASS int RemoteFile::GetFile (char const* source, if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP || char const* target) { source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_url_t source_url; globus_ftp_client_operationattr_init (&source_ftp_attr); globus_io_handle_t dest_io_handle; globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, globus_ftp_client_operationattr_t source_ftp_attr; &source_ftp_attr); globus_result_t result; } globus_gass_transfer_requestattr_t source_gass_attr; else { globus_gass_copy_attr_t source_gass_copy_attr; globus_gass_transfer_requestattr_init (&source_gass_attr, globus_gass_copy_handle_t gass_copy_handle; source_url.scheme); globus_gass_copy_handleattr_t gass_copy_handleattr; globus_gass_copy_attr_set_gass(&source_gass_copy_attr, globus_ftp_client_handleattr_t ftp_handleattr; &source_gass_attr); globus_io_attr_t io_attr; } int output_file = -1; output_file = globus_libc_open ((char*) target, if ( globus_url_parse (source_URL, &source_url) != GLOBUS_SUCCESS ) { O_WRONLY | O_TRUNC | O_CREAT, printf ("can not parse source_URL \"%s\"\n", source_URL); S_IRUSR | S_IWUSR | S_IRGRP | return (-1); S_IWGRP); } if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); if ( source_url.scheme_type != GLOBUS_URL_SCHEME_GSIFTP && return (-1); source_url.scheme_type != GLOBUS_URL_SCHEME_FTP && } source_url.scheme_type != GLOBUS_URL_SCHEME_HTTP && /* convert stdout to be a globus_io_handle */ source_url.scheme_type != GLOBUS_URL_SCHEME_HTTPS ) { if ( globus_io_file_posix_convert (output_file, 0, printf ("can not copy from %s - wrong prot\n", source_URL); &dest_io_handle) return (-1); != GLOBUS_SUCCESS) { } printf ("Error converting the file handle\n"); globus_gass_copy_handleattr_init (&gass_copy_handleattr); return (-1); globus_gass_copy_attr_init (&source_gass_copy_attr); } globus_ftp_client_handleattr_init (&ftp_handleattr); result = globus_gass_copy_register_url_to_handle ( globus_io_fileattr_init (&io_attr); &gass_copy_handle, (char*)source_URL, &source_gass_copy_attr, &dest_io_handle, globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); my_callback, NULL); &io_attr); if ( result != GLOBUS_SUCCESS ) { globus_gass_copy_handleattr_set_ftp_attr printf ("error: %s\n", globus_object_printable_to_string (&gass_copy_handleattr, (globus_error_get (result))); &ftp_handleattr); return (-1); globus_gass_copy_handle_init (&gass_copy_handle, } &gass_copy_handleattr); globus_url_destroy (&source_url); return (0); }

  14. API layer between applications and the grid infrastructure: Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs Insulate against Rapid evolution of grid infrastructure State of Grid deployment Choose between different grid infrastructures Application developers use and develop for the grid independent of the state of deployment of the grid infrastructure Service developers can make their software available to many different applications. GAT Solution

  15. Copy a File: GAT/C++ #include <GAT++.hpp> GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS; }

  16. Example: Remote File Copy Application GAT:File.Copy gridftp cp scp gsiscp RFT wget sftp newest Project Specific GridLab GridJIT

  17. Grid Application Toolkit • Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services • Implements the GAT-API • Used by applications • GAT Adaptors • Connect to capabilities/services • Implements well defined CPI (mirrors GAT-API) • Interchangeable adaptors can be loaded/switched at runtime • GAT Engine • Provides the function bindings for the GAT-API

  18. GridLab Architecture

  19. GAT API Scope • Files • Resources • Events • Information exchange • Utility classes (error handling, security, preferences)

  20. GAT Adaptors • Default Adaptors: • Basic functionality which will work on a single isolated machine (e.g. cp, fork/exec) • GridLab Adaptors: • Connect to services developed through the GridLab project (GRMS, Mercury, Delphoi, iGrid) • Globus Adaptors: • Core Globus functionality: GRAM, MDS, GT-RLS, GridFTP • Others: DRMAA, Condor, SGE, gsissh, …

  21. GGF SAGA-RG • Simple API for Grid Applications • Global Grid Forum Research Group • Motivated by GAT-API and other efforts to develop a real common standard • DRMAA, Java-COG, GAT-API, … • Meeting at LSU Dec04 will provide an initial draft • A standard API, implemented and supported by vendors, will greatly advance the deployment and adaoption of Grid applications

  22. Simple Examples Many short examples and “one liners” included in GAT distribution: • Read from a remote file • Read from a logical file • Move a file • Spawn a subtask to a remote machine • Information about remote/logical file (e.g. file size, permissions)

  23. Example: Spawn a Subtask GAT::Table sdt; sdt.add ("location", "/bin/date"); GAT::Table hdt; hdt.add ("machine.type", "i686"); GAT::SoftwareDescriptionsd (sdt); GAT::HardwareResourceDescriptionhrd (hdt); GAT::JobDescription jd (context, sd, hrd); GAT::ResourceBroker rb (context, prefs); GAT::Job j = rb.submit (jd);

  24. Task Farming on the Grid Task farming infrastructure implemented in Cactus TFM GAT used for starting remote TFMs TFM TFM Designed for the Grid TFM TFM Tasks can be Anything (MPI, Single proc)

  25. Black Holes on the Grid • Task farm small Cactus black hole simulations across GGTC Grid • Parameter survey: black hole corotation parameter • Results steer a large production black hole simulation

  26. Job Migration

  27. Migration of Legacy Apps • Using e.g. wrapper scripts the GAT can be used to help Grid-enable “legacy” applications. Use a GAT wrapper to start Legacy App Send a checkpoint request to the GAT Wrapper Legacy Application writes checkpoint file GAT Wrapper Exits – Kills Legacy App GAT Wrapper registers checkpoint file to logical file Legacy Application is started within GAT Wrapper GRMS starts the GAT Wrapper on target host GRMS Replicates logical File on target Host

  28. Notification and Information Replica Catalog “The Grid” GridSphere Portal SMS Server Mail Server

  29. GridSphere Portal • Portlet based: JSR 168 • Collaboration focal point • Interact, share data • Start, monitor remote jobs • Move/browse files • Track and monitor announced jobs • Application specific portlets • Cactus, UCoMS, SCOOP

  30. Portlets • JSR-168 lays the foundation for a new open-standard for Web portal development frameworks. • Multiple portlets can be composed in a portal page. • Portlets define how to construct and deliver Web content as modular components. • Portlets can be “maximized” or “minimized”. • Portlets support various modes (View, Edit, Help, Configure) • Users can subscribe to portlets.

  31. GridSphere Feature List • Portlets compatible with IBM's WebSphere 4.2. • Support for easy integration of "third-party portlets" • Higher-level model for building complex portlets. • Customized layouts. • Built-in support for Role Based Access Control (RBAC) separating users into guests, users, admins and super users. • Sophisticated portlet service model. • Documentation (DocBook) HTML/PDF guides and tutorials. • GridSphere core portlets offer base functionality including login, logout, user and access control management. • Grid Portlets add general Grid Capabilities.

  32. So what does it look like?

  33. Grid Portlets • Resource Registry Portlet (Grid/Administration Tab) • Administrators can customize for particular VO • Resource Browser Portlet (Grid/Resources Tab) • Resource information from MDS/iGrid etc • Credential Retrieval Portlet (Grid/Credentials Tab) • MyProxy • Job Submission Portlet (Grid/Jobs Tab) • GRAM, GRMS • File Browser Portlet (Grid/Files Tab) • GridFTP • …and many more coming… • Implements generic Grid Portlet API (similar to GAT)

  34. Resource Browsing

  35. Job Submission Portlet

  36. Job History & Output

  37. Links • GridLab: • http://www.gridlab.org • Grid Application Toolkit • http://www.gridlab.org/GAT • GridSphere: • http://www.gridsphere.org

  38. Mardi Gras Conference Frontiers of Grid Applications and Technologies Louisiana State University Baton Rouge February 3rd to 5th, 2005 http://www.mardigrasconference.org

More Related