Particle Physics Data Grid Request Management working group

1. Particle Physics Data GridRequest Managementworking group Arie Shoshani

2. Applications Applications Applications tape system tape system Site Manager parallel disk system Site Manager Network Site Manager parallel disk system parallel disk system Schematic of the distributed data system Arie Shoshani

3. Request Interpreter Request Manager Cache Manager Arie Shoshani

4. Scenarios Simple: All files for a local application are cached to a local disk-cache General: Files for an application can be cached to any disk-cache on any site Arie Shoshani

5. Local Site Manager Remote Services logical request (property predicates / event set) Properties, Events, Files Index Request Interpreter files to be retrieved {file:events} Request Manager File Replica Catalog Request to move files {file: from,to} Request to reserve space {cache_location: # bytes} Storage Reservation service Storage Access service File Access service Cache Manager Local Resource Manager Logical Index service Matchmaking Service Application (data request) Client (file request) Resource Planner Cache Manager GLOBUS Services Layer To Network Fig 1: Architecture for the general scenario - API boundaries Arie Shoshani

6. Functionality of modules (1) • Request Interpreter • accepts “estimate” requests based on “property predicates” • accepts “execute” requests for: • property predicates • set of events • set of files • sends request inquiry to Logical Index Service • consults Matchmaker + Resource Planner for estimation • passes entire “logical file request” to Request Manager Arie Shoshani

7. Functionality of modules (2) • Request Manager • accepts “logical file request” and notifies Application module “ready to process files” • consults Matchmaker + Resource Planner to generate “request plan” • accepts “get next events (files)” requests from Client module • coordinates multi-application requests to share files • issues “cache file(s)” request to Cache Manager • notifies Client module when a (bundle of) files is available in some cache (can be on remote sites) • notifies Cache Manager when a Client is finished with files Arie Shoshani

8. Functionality of modules (3) • Cache Manager • accepts “cache a file” requests from the local Request Manager • accepts “reserve space” requests from remote reservation calls • allocates space on local cache as well as tape resourcesaccording to a local resource management policy • issues “lock a file” service requests to remote sites • issues “reserve space” service requests to remote sites • issues “move a file” service request • issues “release a file” service request to remote sites • notifies Matchmaker of any local cache changes Arie Shoshani

9. Functionality of modules (4) • Local Resource Manager • accepts requests for multi-file caching • Determines the best way to allocate local tape and disk cache resources • notifies cache manager to reserve space • notifies tape resource manager to reserve tape drives • ... Arie Shoshani

10. Local Site Manager Remote Services Storage Reservation service Storage Access service File Access service Cache Manager Local Resource Manager Logical Index service Matchmaking Service Application (data request) Client (file request) (Resource Planner) Cache Manager logical request (property predicates / event set) Properties, Events, Files Index Request Interpreter files to be retrieved {file:events} Request Manager File Replica Catalog Request to move files {file: from,to} Request to reserve space {cache_location: # bytes} To Local Cache GLOBUS Services Layer To Network Fig 1: Architecture for the simple scenario Arie Shoshani

11. Local Site Manager Remote Services logical request (property predicates / event set) Properties, Events, Files Index Request Interpreter files to be retrieved {file:events} 4 2 6 5 7 1 9 3 8 Request Manager File Replica Catalog Request to move files {file: from,to} 11 10 12 13 Request to reserve space {cache_location: # bytes} Storage Reservation service Storage Access service File Access service Cache Manager Local Resource Manager Logical Index service Matchmaking Service Application (data request) Client (file request) Resource Planner Cache Manager GLOBUS Services Layer To Network Fig 1: Architecture for the general scenario - APIs use order Arie Shoshani

12. Sequence of API calls 1. Application issues an estimate 2. Consult Logical index Service (properties-events-files) for estimate 3. Consult Matchmaker + Resource Planner for estimate 4. Issue “execute” request command 5. Consult Logical index Service for precise request evaluation 6. Sent {files:events} request to Request Planner 7. Client issues a request for next events to process (files are implied) 8. Request Planner Consults Matchmaker + Resource Planner 9. Request Planner issues “get file(s), source, destination” request 10. Cache Manager issues “reserve space” request if “destination” is remote site 11. Cache Manager issues “move file(s)” request 12. Cache Manager notifies Matchmaker of any changes to local cache 13. Client issues commands to read events out of files Arie Shoshani

13. People assigned to define API interfaces(first name responsible to contact others) 1 + 2 + 4 +5 -- Alex, Dave, Vicky (Igor) 3 + 8 -- Vicky (Igor) to coordinate with Miron’s working group 6 -- Alex, Arie 7 -- Dave, Vicky (Igor), Andy 9 -- Luis, Andy (for bundle at a time request, not just a single file) 10 + 11 -- Luis, Andy 12 -- Andy, Luis 13 -- Andy, Dave Arie Shoshani

14. Local Site Manager Remote Services Storage Reservation service Storage Access service File Access service Cache Manager Local Resource Manager Logical Index service Matchmaking Service Application (data request) Client (file request) (Resource Planner) Cache Manager logical request (property predicates / event set) Properties, Events, Files Index Request Interpreter Objectivity files to be retrieved {file:events} AMS Request Manager File Replica Catalog Event (File) in Cache? Request to move files {file: from,to} Request to reserve space {cache_location: # bytes} OOFS Reserve / make space Move file HPSS-DISK To Local Cache GLOBUS Services Layer Where OOFS functionality corresponds to architecture components To Network Fig 3: OOFS integration Arie Shoshani