1 / 16

SURAgrid Accounting Status September 2007

SURAgrid Accounting Status September 2007. SURAgrid Accounting Status September 2007. Accounting WG - Phase 1. Call for a working group at SURAgrid All-Hands March 07, to develop (at least) a recommendation for implementing accounting on SURAgrid

ronan-austin
Download Presentation

SURAgrid Accounting Status September 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SURAgrid Accounting StatusSeptember 2007

  2. SURAgrid Accounting StatusSeptember 2007

  3. Accounting WG - Phase 1 • Call for a working group at SURAgrid All-Hands March 07, to develop (at least) a recommendation for implementing accounting on SURAgrid • Group formed with desire to be short-term but contribute a tangible immediately useful first step • Volunteers included: Victor Bolet, Steve Brandt, Mahantesh Halappanvar, Steve Johnson, Shawn McKee, Jerry Perez, John-Paul Robinson, Bin Zhou

  4. Accounting WG Phase 1 • Goals of initial SURAgrid accounting: • To develop and implement the capability to analyze SURAgrid usage in support of setting implementation priorities, planning, and input to proposals. • To monitor allocations, however these are prescribed by the resource owner at this time. • Create a system that can be implemented with current resources but also incorporated into, or expanded by, more complex system later on if needed - also tracking well with standards and best practices in this area.

  5. Phase 1 Implementation Plan • Determine set of common data to be collected from SURAgrid resources • Document desired data collection in the form of a SURAgrid Usage Record • Based on job Usage Record standard currently progressing through the OGF User Record Working Group (http://forge.ggf.org/sf/projects/ur-wg). • Enable data collection through local schedulers • Store data in standard format (XML) and central database for access now, also later availability to another accounting package if one is selected.

  6. Snapshot Status Phase 1 Work • Some still to discuss but…SURAgrid Usage Record essentially complete! • WG members piloting with all major local schedulers & documenting for others to follow • Data from pilots is being stored in centralized database w/querying capability, hosted by ODU • Will need to see if initial database & reporting package provide enough utility, can scale, etc. • “Accounting WG Phase 2” to address if not…

  7. With Special Thanks to… This was a real working working group and extra appreciation goes to the following SURAgridr’s for producing the components you see today: • Victor Bolet • Steve Brandt • Mahantesh Halappanvar • Steve Johnson • Shawn McKee • John-Paul Robinson

  8. SURAgrid Usage Record See handout…

  9. Status of Local Scheduler Pilots • PBS: Conversion to XML complete, script provided, incorporated into workflow • LoadLeveler: Conversion to XML complete, script provided • SGE: Basic data mapping defined, needs conversion to xml • Condor: Conversion to XML complete, script provided, OSG Gratia project • GridWay: Initial investigation suggests data available, needs code to extra from data store • LSF?

  10. Working Group Action Items • GSU: work on Loadlever scripts with LSU • ODU, TAMU, LSU: Loadlever and pbspro upload workflow • TAMU: hammer out PBS to URWG conversion and will look at user mapping • ODU, UAB, TAMI: will working on running 1.4.3 xindice (parallel development path with work-flow effort) • UAB: SGE and Gridway acct data record generation, project page updates, move wiki to more formal home to facilitate future collaboration • ALL: solidify project documentation with ideally step-by-step instructions

  11. Status of User Documentation • Currently in a Wiki, hosted by UAB • http://webapp.lab.ac.uab.edu/projects/jpr/wiki/SURAgridAccounting • Moving to UABgrid collaboration environment: http://projects.uabgrid.uab.edu/sg-accounting • Working Group members still adding detail to this resource • Still need to determine best flow for information • Eventually move to other location or format? • Can be hosted by UABgrid indefinitely

  12. From PBS to Central Data Store • Resource Manager: OpenPBS/PBS-Pro/Torque • XML Conversion Tool: PBS-XML Accounting Toolkit • XML Database: Xindice • Report Generator: DarkSlide Figure 1. A Basic Flow Chart for Collecting SURAgrid Usage Records.

  13. PBS to XML • Site: install PBS-XML Accounting Toolkit • Requires Java (1.4) and Ant • Export code from CVS @ Sourceforge • Build and Install RPM % ant buildrpm % rpm –ivh pbsxmlaccounting.rpm • Set up user environment • Run command • Central: install Darkslide and Xindice

  14. Status of Central Data Collection • Update from Mahantesh

  15. Additional considerations • Need to revisit the “for further discussion” fields in the SURAgrid Usage Record • Need a way to map local user to grid user (dn) • On account naming conventions: Thinking at this time is to leave it to sites and work on mapping. • It's up to local system to filter out non-suragrid users. Sites could leave all accounting data in place and use SURAgrid summary to generate local reports. Might be a good way to generate a community around this resource.

  16. Next Steps • Need longer term Accounting WG for Phase 2 • Insure and support implementation of SURAgrid Usage Record • Evolve record as necessary • Evaluate use and utility of the basic system and consider/select other options if needed • Participants: • From the community: Mahantesh & Steve B. so far committed. Other volunteers? • From SURA: Dali Wang, MFY if/as needed

More Related