qcdgrid progress n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
QCDGrid Progress PowerPoint Presentation
Download Presentation
QCDGrid Progress

Loading in 2 Seconds...

play fullscreen
1 / 15
cameron-mays

QCDGrid Progress - PowerPoint PPT Presentation

105 Views
Download Presentation
QCDGrid Progress
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh

  2. Contents • Background • DataGrid: hardware status • DataGrid: software status and future development • Job Submission status and future development • International Collaboration

  3. Background • Aim: To develop a distributed data storage and compute grid for quantum chromodynamics in the UK • Currently a 4-site data grid • provides reliable distributed data storage, including a searchable metadata catalogue • A job submission system is under development • Key technologies used • Globus 2.0 • European Data Grid • eXist XML database

  4. The Data Grid: Hardware • The data storage grid has been up and running for several months • currently managing a few hundred gigabytes of data • Consists of storage elements at 4 sites • Edinburgh, Liverpool, RAL and Swansea • may soon expand to Glasgow and Southampton • QCDOC comes on-line 2004 • expect to store around 30 Tbytes of data • requested further space from GridPP • further disk space expected at Edinburgh

  5. The Data Grid: Software • Replication of files is currently managed by custom written software • built on Globus 2.0 • will consider replacing / integrating with existing software in the future • Central control thread • ensures there are always at least two copies of each file, stored at different sites • Replica catalogue • maps logical filenames to actual physical locations

  6. The Metadata Catalogue • Each file on the datagrid has an associated XML metadata document • These documents are stored in the eXist open source XML database, where they can be searched using the XPath query language • Java GUI provides a user friendly interface • Metadata catalogue is itself stored on the datagrid

  7. DataGrid: Future Development • Review current available technologies for possible integration • after phase 1 of the job submission work • Control Node • currently backed up on the data grid • develop second control node that allows read access to the datagrid automatically • allow full control node to be started manually on another system • Integrate EDG software for virtual organisation management and security • already running on test bed system • currently being rolled out to main data grid system

  8. Job Submission: Requirements • Aim: To allow data generation/analysis jobs to be easily submitted to grid machines • Integration with the existing datagrid is desirable • Resource broking not essential • users normally know in advance on which machine a job should run • is however desirable – especially for analysis jobs • Real-time job status monitoring important • Must integrate with a diverse range of machines from normal (Linux) PCs to QCDOC • User-friendly GUI or web portal a desirable

  9. Job Submission: Technology • As with the datagrid, requirements dictate a combination of existing software and purpose-built middleware • Globus toolkit to be used for low level access to grid resources and data • European Data Grid software to be used for virtual organisation management and security • Batch systems such as PBS will be integrated into the system • QCDgrid job submission software will build on these components, providing the interface and features that users need

  10. Job Submission: Status • Job submission system is being developed on a test grid independent from the main datagrid • Jobs can now be submitted to grid resources using a command line tool • Input files can be fetched automatically from the datagrid • Job output and input can be streamed to and from the user’s console • allows jobs to be monitored • allows interactive jobs to run on grid resources • All output files generated by the job are automatically brought back to the user’s local machine

  11. Job Submission: Future Development • First ‘users’ to test the software this month on test system • Integrate with batch queuing systems such as PBS, loadleveler, Sun grid engine • Improve the user interface • Provide more advanced interaction with the datagrid • metadata operations • storing output automatically • Deploy and test on “real” grid

  12. International Collaboration: ILDG • ILDG (International Lattice DataGrid) • A collaboration of scientists involved in lattice QCD from all over the world (UK, Japan, USA, France, Germany, Australia and other countries) • Working on standards to allow national datagrids to interoperate, for easier data sharing • Two working groups looking at different aspects of this goal: metadata and middleware

  13. ILDG Working Groups • ILDG metadata working group is developing a standard schema for describing QCD metadata • Likely to be based on UKQCD’s schema which is extensible and can be applied to various different data types • ILDG middleware working group is developing a standard interface to grid data storage systems • Based on the SRM (Storage Resource Manager) specification • A basic implementation of SRM to interface with UKQCD’s datagrid has been written • Together these standards will enable international data sharing

  14. Summary • Aim is to develop a grid for use by the UKQCD collaboration for storing data and performing computations • A datagrid, including searchable metadata catalogue, is already in use • A job submission system is in development • Software based on Globus toolkit and European Data Grid middleware • International efforts to develop standards for lattice QCD grids and data are ongoing

  15. References • QCDgrid Web Site • http://www.gridpp.ac.uk/qcdgrid • ILDG Web Site • http://www.lqcd.org/ildg • European Data Grid Project • http://www.eu-datagrid.org/ • GridPP Web site • http://www.gridpp.ac.uk