QCDgrid User Interfaces

1. QCDgrid User Interfaces James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh

2. QCDgrid Summary • QCDgrid project is developing a data and compute grid for scientists in the UKQCD collaboration • data storage grid has been up and running for some months now • job submission system is in early stages of development • software developed is released as open source • builds on Globus 2.0, eXist XML database and various other technologies • For more information on the project in general, see Lorna’s talk this afternoon!

3. User Requirements • User interface is naturally driven by users’ requirements • most QCDgrid users have a good understanding of computers • for them, advanced scripting capabilities are more important than user-friendly GUIs • powerful command line interface is top priority for QCDgrid software • GUIs also useful for some operations • for example, searching and browsing the metadata catalogue • C and Java APIs facilitate integration with software in many different programming languages

4. Datagrid Interfaces • Data management aspect of grid consists of two distinct parts • Low level data replication deals with files themselves • files at this level are just blocks of binary data – they could contain anything • Globus replica catalogue maps logical filenames to actual physical locations • Metadata catalogue associates some meaningful, structured information with each file • allows users to search for data more easily • maps interesting characteristics of data (structured as XML) to logical filenames

5. Low Level Data Grid Interface • Low level operations provided by command line tools and C API • Java interface using JNI also available • SRM-compliant interface to some functionality • Fairly small set of basic operations • put a file or directory on the grid • get a file or directory from the grid • delete a file or directory • list files on grid • register interest in a file or directory • User must have a valid Globus proxy initialised

6. Data Grid Example Commands • Some example data grid commands: put-file-on-qcdgrid /home/username/myfile gridfile • puts the local file ‘myfile’ onto the grid under logical name ‘gridfile’ • replication software will take care of deciding where to store the file, adding replica catalogue entries, etc. get-file-from-qcdgrid -R griddir /tmp/mydir • gets directory ‘griddir’ from the grid, storing it in local directory /tmp/mydir • ‘-R’ switch means ‘recursive’, works with most QCDgrid commands

7. More Example Commands • More example commands... qcdgrid-list • lists all files on grid by logical name i-like-this-file interestingdata • registers interest in the file with logical name ‘interestingdata’ • replication system takes this into account, tries to store files close to where they are most often wanted qcdgrid-delete olddata • removes all copies of the file ‘olddata’ from grid

8. Data Grid APIs • APIs provide similar functionality • Example: QCDgridClient grid = QCDgridClient.getClient(true); String logicalFile = “gridfile”; File physicalFile = new File(“localfile”); grid.getFile(logicalFile, physicalFile);

9. Metadata • Problem: logical file names may not be meaningful • users may have trouble finding data • Solution: metadata catalogue • associate some meaningful information with each file on the grid • including date produced, machine used, code used, actual physical parameters • users can then search on these fields • metadata is XML, stored in eXist XML database • queried using XPath query language • Command line, GUI and Java interfaces (via standard XMLDB API) available

10. Metadata Interface: Commands • Command line functionality currently limited to 3 operations • submit • remove • update schema • Examples: java QCDgridMetadataClient localhost:8080/exist \ updateSchema newschema.xsd java QCDgridMetadataClient localhost:8080/exist \ submit newfile.xml newdocumentid

11. Metadata Interface: GUI • Metadata browser GUI allows users to easily search for the data they want • XPath queries can be built using simple graphical input methods • GUI generated automatically from current schema • when schema is updated, GUI updates itself • matching data can be easily retrieved from the grid

12. Searching MDC, Step 1 • Main browser window gives a list of saved queries • these are stored in the user’s profile • support for ‘libraries’ of queries is planned

13. Searching MDC, Step 2 • Creating a new query • first a node in the XML document structure must be selected from the tree • tree is automatically generated from schema when browser starts up • e.g. to find all the data produced on a certain date, user should select the ‘date’ node

14. Searching MDC, Step 3 • Once node has been selected, predicate must be specified • this is just an XPath term for criteria for matching node data • predicate can be entered as raw XPath if desired • most users will want to make use of form to simplify process

15. Searching MDC, Step 3 cont... • More complex queries can be created relatively easily • in this example, the query is extended to search for data from 2 years • for most queries, knowledge of XPath is not required

16. Searching MDC, Step 4 • New query now appears on list • from here, queries can be managed • queries can be combined together • or can be submitted to the database backend

17. Searching MDC, Step 5 • Matching metadata documents are displayed • XML is parsed into easy-to-read, expandable tree format • corresponding data files can be fetched from grid at the press of a button

18. Job Submission • QCDgrid job submission still in very early stages • As with data management, users require command interface that can be used from scripts • integration with data grid will simplify user interface • unlike plain Globus, job input, output and error streams can be redirected to and from the user’s console • this allows for interactive jobs on the grid – useful for debugging etc. • GUI or web portal interface may be added later

19. Example Commands • Early prototype of job submission software is up and running • Syntax quite similar to globus-job-run • Example commands: qcdgrid-job-submit qcdtest.epcc.ed.ac.uk \ /bin/date qcdgrid-job-submit doorstopper.epcc.ed.ac.uk \ /usr/bin/program arg1 arg2 arg3 \ --fetch-from-qcdgrid gridName localName

20. Administrator Interface • Previous slides have focussed on normal end users’ experience • QCDgrid software also provides tools to aid in administration • commands to add and remove grid nodes, and change the state of existing nodes • commands for building and maintaining the Globus replica catalogue • commands for maintaining directory of grid users • Admin GUI to integrate many of these functions is a possibility

21. Some Admin Commands • Administrators are identified by their certificate subjects • Must have a valid proxy with subject listed in the config file before executing these commands add-qcdgrid-node newnode.ed.ac.uk Edinburgh \ /home/qcdgrid disable-qcdgrid-node notworking.ed.ac.uk verify-qcdgrid-rc setup-security.sh adduser James Perry \ /O=certificate/O=subject/CN=jtp \ jamesp@epcc.ed.ac.uk

22. Interface Summary • Low level data grid has command line interface and APIs • Metadata catalogue mainly accessed through browser GUI • this also integrates with low level data grid • Job submission currently usable from command line only • possible GUI/web portal in future • Various admin tools exist or are in development • Better integration of the different parts of the project is planned