SAMGrid Developments – Part I

1. SAMGrid Developments – Part I Siniša Veseli CD/D0CA S. Veseli - SAM Project Status

2. Completed Projects • DB Schema Redesign • Python API • Batch Adapter • D0 Online dCache TDP • Monitoring Tools • Data Dimensions Grammar (Rob K.) S. Veseli - SAM Project Status

3. Ongoing Projects • DB Server • C++ API • Configuration Management • Test Harness • Grid-JIM Developments (Rob K.) • Chains and Links (Rob K.) • CDF DFC Migration (Rob K.) • CDF (H Stream) Reprocessing (Rob K.) • Station Developments (Rob K.) • Linux Deployment of Centralized Services (Rob K.) • Subproject Assessments (Rob K.) S. Veseli - SAM Project Status

4. DB Schema Redesign • Provided support for the unified (D0/CDF) DB schema • Wyatt and Adam were driving the efforts, many others have been involved • Description of schema changes can be found onhttp://www-d0.fnal.gov/~lyon/samSchema/ • Completed in May 2003, few minor changes introduced later • One of the major reasons for the DB Server rewrite S. Veseli - SAM Project Status

5. Python API • Most of the work done by Lauri • Replacement for the old sam_user code • Benefits: • Real Python API • Self-documenting interfaces • Maintainable code • Improved performance • Status: • Deployed in production since June 2003 • Maintenance load is very low • Ongoing work on replacing the functionality relevant to file metadata S. Veseli - SAM Project Status

6. Batch Adapter • Replacement for the old station batch adapters • Features: • Flexible/configurable, supports any batch system • Non-restrictive (i.e., does not require specific batch system configuration) • Support for multiple user processes, grid job submission interface • Status: • Deployed in production since June 2003 • Low maintenance load • Possible future work: • ability to export/import adapter configuration in XML format S. Veseli - SAM Project Status

7. D0 Online dCache TDP (Tapeless Data Path) • Work done by Lauri/Siniša • Provides functionality of storing files via dCache write pools for the D0 online system: • Reduce resource contention (tape drives) • Files are immediately available for use, even before they are stored on tape • D0 requirement: ability to choose between encp and dccp for different streams – this led to sam_cp redesign S. Veseli - SAM Project Status

8. D0 Online dCache TDP • New sam_cp features: • Refined interface: ability to perform tasks before and after invoking of the actual file transfer • New file transfer protocol management: ability to plug in an arbitrary algorithm for determining which protocol to choose • Status: • Used by D0 Online in production since November 2003 • Operations: load problems with online machines • Possible future sam_cp work: • XML-based configuration • Server which handles handshakes between sites S. Veseli - SAM Project Status

9. Monitoring Tools • Excellent monitoring tools developed in the last year by Adam and Robert • SAMTV: shows what is happening with stations and projects (http://d0db-dev.fnal.gov/sam/samTV/samTV.html) • CAB monitoring: automatically reconfigure the CAB station depending on the problems in the cluster S. Veseli - SAM Project Status

10. DB Server • Work started in June 2003 after agreement on the new DB schema was reached • Developers involved: Lauri, Carmenita, Steve, Siniša – none of us worked full time the project • We’ve got lots of help from DBAs (Julie, Anil): • mini-SAM developer databases • Benefits: • Code base and the number of IDL interfaces were significantly reduced and cleaned up • New CORBA infrastructure: wrapper classes around IDL structs and methods, exception handling – lead to code reuse, easier development, more maintainable code S. Veseli - SAM Project Status

11. DB Server • Benefits: • Removal of non-atomic methods • Multithreading capability • Replaced fnorb with omniorb: better performance, less bugs • New file metadata classes: more flexibility, easier maintenance • New set of administrative interfaces • Merging of the MC and Farm Request systems • Improved autodestinations S. Veseli - SAM Project Status

12. DB Server • Status: • Basic functionality is ready (metadata classes, station/project interfaces) • We are getting ready to upgrade the D0 development DB schema, start running test harness • To be completed: MC/Farm Request system merge, autodestinations, client code • Deployment plan: • minimize effect on users, do not force immediate client upgrades • introduce new servers and upgrade clients gradually as they become ready S. Veseli - SAM Project Status

13. DB Server • Milestones: • Early March: upgrade D0 integration DB to the latest schema, start integration testing • Early April: upgrade D0 production DB to the latest schema, start deploying new db servers - CDF will follow • Next project assessment: mid-March S. Veseli - SAM Project Status

14. C++ API • Provide unified support for both D0 and CDF SAM C++ clients • Code reuse, common CORBA infrastructure • Faster development, lower maintenance load • New features: • File deliveries via CORBA callbacks • SAMTV Ready! (i.e., has complete monitoring event infrastructure) S. Veseli - SAM Project Status

15. C++ API • Planned features: • Support for the multiple naming services • Ability to influence client behavior at runtime (e.g., direct the client to stop requesting files) • Support for staging files • Ability to define datasets, start projects (needed for CDF) • Status: • Initial version released, D0 C++ client (sam_manager) using it (sam_manager code reduced to 30% of its original size) • CDF (Fedor) is looking into adapting AC++ to use the API • Mid-March milestone: add CDF-requested functionality • Next project assessment: mid-March S. Veseli - SAM Project Status

16. Configuration Management • Goal: simplify existing (complex) installation/configuration, eliminate code compatibility problems, etc. • Work done by Art, recently joined by Adam on the D0 side • Status: • Station installation scripts ready, used by CDF in production • Adapting procedure for D0 in progress • We need similar scripts for client code • We have to revisit upgrade/downgrade procedures S. Veseli - SAM Project Status

17. Configuration Management • Late February milestone: both experiments should be using the scripts for their standard station installations • Next project assessment: mid-March S. Veseli - SAM Project Status

18. Test Harness • Old test harness: • Has not been developed in couple of years, did not keep up with the SAM code development • Hard to adapt for CDF usage • Work on the new test framework started in December 2003 (Matthew, Stefan, Adam) • Key features: • Easier, XML-based configuration • Written in python only, uses new API, easier to maintain • Cleaner output, easier monitoring • More flexibility in creating/adding tests S. Veseli - SAM Project Status

19. Test Harness • Status: • Basic infrastructure completed • Work on creating tests is in progress • Mid-March milestone: demonstrate basic functionality (tests involving file delivery/storage) • Next project assessment: early April S. Veseli - SAM Project Status

20. Concluding Remarks • To be continued by Rob… S. Veseli - SAM Project Status