1 / 22

Grid eXplorer (GdX) An Instrument for eXploring the GRID

Grid eXplorer (GdX) An Instrument for eXploring the GRID. F. Cappello, O. Richard, P. Sens ( fci@lri.fr , olivier.richard@imag.fr , pierre.sens@lip6.fr ). LRI, ID-Imag, Lip6, etc. What we know about Grid/P2P?. Infrastructure features: Large scale distributed systems

cairo-walker
Download Presentation

Grid eXplorer (GdX) An Instrument for eXploring the GRID

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid eXplorer (GdX)AnInstrumentfor eXploring the GRID F. Cappello, O. Richard, P. Sens (fci@lri.fr, olivier.richard@imag.fr, pierre.sens@lip6.fr) LRI, ID-Imag, Lip6, etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  2. What we know about Grid/P2P? Infrastructure features: • Large scale distributed systems • Dynamic infrastructures (network, nodes) • Dynamic workloads • Heterogeneous (parallel & uniproc. nodes) • High Computation/Communication perf. Ratio F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  3. What we know about Grid/P2P? Application features: • Large data sets (data bases, physics instruments, generation of very large results) • Huge computations (in terms of #operations) • Long run times • Large number of data operations (movements, transactions, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  4. What we know about Grid/P2P? Utilization: • A large number of users (researcher, industry, individual, etc.) • A large set of applications (research, business) • High throughput (file download, multi-parameters, computation, etc.) • Real time (Sensors, Entertainment, HPC, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  5. What are open issues? • Security (GSI, CAS) • Data Storage/consultation/movement • Multi users/ Multi applications scheduling • Coordination (virtual, ephemeral infrastructure) • Programming • Fault Tolerance! • Scalability • Performance • Easy/efficient deployment techniques • Application characterization techniques • Etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  6. What is Grid today? • Middleware: Globus, Legion, Netsolve, Unicore, DIET, Condor, XtremWeb, Boinc, NWS  they are actually working! • Testbed: DataGRID, TeraGRID, Etoile, Grads, XW, Boinc  Difficult to build (debug, human factor, etc.) • Dedicated applications (SETI, Kazaa, Jabber, etc.)  they are working already (at large scale!),  BUT they address much less issues! F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  7. Why Grid is so difficult? • Grids are very complex systems (we still have problems with large scale parallel computers!, we have less control on Grid resources! ) • Many issues are addressed simultaneously  We need a methodology enabling the study of Grid issues, independently but realistically. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  8. What are the current approaches? • Simulators: SimGRID, MicroGRID, etc.  they have strong limitations (scalability, != than execution of real codes, validation) • Experimental testbed (is there any?)  Most testbed are for production, each testeb is specific, representativeness? • We have no way to test: ideas independently, at a significant scale, with realistic parameters and behaviors! F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  9. What is missing? • A full fledge scientific environment (reproducible realistic experimental conditions) • Probes measuring the performance of real resources and networks (Ganglia, NWS, la grenouille) • Fully experimental testbed, (GRID 5000 would remove this lack)  Not enough, we need instruments with parametrisable reproducible experimental conditions F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  10. Methodology • When physicists can’t measure a phenomenon because a) the item to measure is not reachable or b) the phenomenon is hidden by others, • They build instruments! • And they observe… F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  11. Grid eXplorer A tool set for conducting experiments & measurements An experimental conditions database or generation An experimental hardware platform F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  12. Grid eXplorer:Analogy with physic instruments A set of sensors Inside real platform A tool set for Observation Results Analysis A Hardware Platform for Emulation Simulation A database of Real platform Validation on real platforms F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  13. Grid eXplorer (GdX): Instrument for Grid exploration: • An instrument for exploring Grid middleware, algorithms, performance and applications under reproducible experimental conditions • A tool set for emulation/simulation of large scale distributed systems • 1K CPU clusters + configurable network & OS • For Grid, P2P, etc. • Potentially connected to Grid testbeds F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  14. Grid eXplorer (GdX): eXperimental conditions database • A set of sensors (Nodes, Networks):  Academic Networks (x K nodes, GRID 5000)  ADSL (la grenouille  60 K nodes) • A common format for traces • A tool set for accessing, managing traces • Tools for trace analysis F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  15. Grid eXplorer (GdX): Hardware platform for eXperiments: • 1K CPU clusters • configurable network • configurable OS • Multi-users • Potentially located/managed by IDRIS or CEA F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  16. Grid eXplorer (GdX): Tool set: • Emulators (folding 10K virtual nodes on 1K nodes) • Parallel simulators (difficult!) • Virtual GRID environment (1k virtual nodes on 1k nodes) • Measurement tools, • Visualization tools. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  17. Grid eXplorer (GdX): eXperiments: • Real scale pseudo-emulation (1:1 scale) • Large scale emulation (10:1 scale) • Large scale simulation (100:1 scale) • Application test (bottleneck discovery, performance evaluation, optimization) • Infrastructure test (bottleneck discovery, performance evaluation, optimization) • Connection to Grid 5000, International Grids, etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  18. Grid eXplorer (GdX): Instrument related projects: • Instrument projects • Cluster management (ID Imag), • Sensor event data base (LRI-fci, la grenouille), • Event analysis (ID Imag), • Emulators (LRI-fci, etc.), • Experimental conditions injection, • Result visualization (???), • Result analysis, • Validation methodology, • User validation (physicists, biologists, chemists, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  19. Grid eXplorer (GdX): Grid/P2P research projects: • Grid • DataGRID technology (???) • GRIDRPC (LIP, Desprez?) • Code coupling (IRISA?) • P2P • Large scale desktop Grids (LRI-fci) • Large scale storage (Laria, Irisa?) • Large scale data consistency (Lip6-P. sens) • Large scale fault tolerance (LRI-Beauquier) • Network • Emulation of latency, packet loss, bandwidth • Emulation of topology F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  20. Grid eXplorer (GdX): User applications: • Bio Grid (Genomining, IBBMC, etc.) • Medi GRID (???) • Geo GRID (IFP, CGG) • Power GRID (EDF) • Industry GRID (EADS, Alcatel Space) • Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.) • Industry Partners (GridXpert???, IBM???) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  21. Grid eXplorer (GdX)Project organisation • A Virtual laboratory • A director (or a group), • A scientific director, an administrative director • A scientific council (with physicists), • A Researcher group • A group of engineers, • A user group • A set of scientific events F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

  22. Grid eXplorer GdX • A long term effort • A medium term milestone: 4 years  A fully functional prototype • Many scientific issues (large scale emulation, experimental conditions injection, distance to reality, etc.) • A tool for Grid users or potential users • A tool for Grid/P2P developers F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

More Related