1 / 8

Local Resource Utilization for Program P on Files Subset from Collection C

Develop and test a Request Executor (ReqEx) to apply Program P to a subset of files meeting constraint Q from Collection C using local processing and storage resources. Test with Query Estimator (QE), Meta Catalog (MCAT), and file staging.

mimis
Download Presentation

Local Resource Utilization for Program P on Files Subset from Collection C

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Request Execution: a Tier 3 Perspective

  2. Request Use local resources (processing and storage) to apply program P to a subset of files from collection C that meets constraint Q • Files must be staged locally • No output • Fixed amount of local disk space • Has to compete for CPU power

  3. Expectations • No need to know location of C • Reliable • Error Reporting • Efficient • Progress Reporting

  4. Method Develop a Request Executor (ReqEx) and test its end-to-end functionality in a testbed with the following components: • Query Estimator (QE) at LBL • Meta Catalog (MCAT) provided by SRB • Collection replicated at PPDG sites (LBL,SDSC, ...) • File staging via SRB • Condor pool and disk cache in Wisconsin

  5. Execution Steps • Receive C and Q from user • Locate QE for C • Send Q to QE and get list of files • Obtain file replication information from MCAT • For each file • Select storage location • Allocate local space and stage file • Apply F to file • Remove file and free local space

  6. Building the Testbed • Install SRB and QE related software in Wisconsin • Obtain SRB account • Install SRB software in LBL • Register LBL PPDG collection in MCAT • Replicate (and register) PPDG collection at SDSC. • Establish disk cache at Wisconsin

  7. ReqEx prototype • Second version built with software components from Condor • Prototype works • Only uses the Sget command of SRB • Does not support replication • QE location hard coded.

  8. Main Lessons • Installation of software components can not be done by “the locals” • Lack of support for inter-component exception handling a major obstacle • Need for a “development” testbed and a “production” testbed • Software upgrades a problem.

More Related