1 / 28

Status of the US-CMS “Core Applications Software” Project

Status of the US-CMS “Core Applications Software” Project. Ian Fisk UCSD Acting Deputy Level 2 Project Manager US-CMS FNAL Oversight Panel October 24, 2000. Introduction to CAS. Overview of CAS CAS Currently Consists of 4 Sub-Projects in the WBS 2.1 CMS Software Architecture

marv
Download Presentation

Status of the US-CMS “Core Applications Software” Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Status of the US-CMS“Core Applications Software”Project Ian Fisk UCSD Acting Deputy Level 2 Project Manager US-CMS FNAL Oversight Panel October 24, 2000

  2. Introduction to CAS Overview of CAS CAS Currently Consists of 4 Sub-Projects in the WBS • 2.1 CMS Software Architecture • 2.2 Interactive User Analysis (IGUANA) • 2.3 Distributed Data Management and Processing • 2.4 Support Core Applications Software

  3. CMS Architecture 2.1 • It is the intention of CMS to deploy a coherent architecture for all aspects of physics data processing, including simulation, higher-level triggering, reconstruction and selection, physics analysis and visualization. The software architecture project seeks to facilitate this by ensuring that the problems we are solving are well researched and by ensuring that the core on which the software will be built on is well engineered and understood by the community. • 2.1.1 CAFE • 2.1.1.1 Architecture Document Publishing Tools • 2.1.1.2 Domain Analysis • 2.1.1.3 Document Project Management • 2.1.1.4 Architectural Views Documentation • 2.1.1.5 Design Evolution • 2.1.2 Overall Architecture and Framework • 2.1.3 Subsystem Architecture • 2.1.3.1 Detector Geometry Description • 2.1.3.2 Simulation Sub-Architecture • 2.1.3.3 Reconstruction Sub-Architecture Core Applications Software

  4. CMS Architecture 2.1.1 CAFE • In order to have a forum to discuss CMS architecture and take input from the Collaboration about issues of design, the CAFE project was formed. • The CAFE project covers the documentation side of the architecture. It provides tools for the document project, analyses the problem domain, manages the documentation, documents the different aspects of the architecture and feeds input back to architecture to evolve it. • CAFE has been developing the infrastructure necessary to produce the entire architectural documentation. • They have been researching to develop an understanding of the problems we are solve. • Gathering input from the Collaboration and propagating the new ideas back into the architectural implementation. • This process is being handled by a CAS engineer. Core Applications Software

  5. CMS Architecture 2.1.3.1 Detector Geometry Description • This task will provide an environment for creating, manipulating, and using the parameters describing the CMS detector in a consistent manner. In particular, it covers the geometrical description of the detector elements at various levels (full engineering detail, full GEANT detail, fast simulation, trigger tower geometries, etc.), associated material properties, magnetic field map, etc. The Detector Description sub-system will serve a number of clients, including OSCAR, the Fast Simulation, ORCA, the Calibration, and the User Analysis Environment. • The prototype will use a generic interface/viewpoint framework, based on XML and mediators, which is being developed for CMS-wide data access. Core Applications Software

  6. CMS Architecture 2.1 Future Plans • Complete documentation and analysis of architecture in the CAFE project. • Complete functional prototype for Generic Detector Geometry description sub-project • Develop Architecture and Framework for use in some of the Reconstruction Architecture. NOT physics reconstruction code, but the underlying framework for fast efficient physics code. • General CARF (CMS Architecture and Framework) Development. Core Applications Software

  7. IGUANA 2.2 • The IGUANA software project addresses interactive visualization software needs for three domains: • 2.2.1 graphical user interfaces (GUI's) • 2.2.2 interactive detector and event visualization • 2.2.3 interactive data analysis and presentation for use in a variety of areas such as offline simulation and reconstruction, data analysis, and test beams. • Tasks include the assessment of use-cases and requirements and the evaluation, integration, adaptation, verification, deployment, and support in the CMS environment of visualization software from HEP, academia, the public domain, and the commercial sector. • Pre-existing software is exploited as much as possible to optimize the use of the resources available. Core Applications Software

  8. IGUANA 2.2 General Strategy • Focus on a sustainable medium to long-term strategy • Provide a general set of tools (not single application) • for a wide variety of applications and environments • for both experienced developers and non-expert users • Strong emphasis on modularity and use of standards • to address the issues of scalability, maintenance, deployment, support, long-term evolution,… • Pro-actively exploit software developed elsewhere • For example... • HEP (e.g. HTL, HepODBMS, HEPVis, Lizard,… • Public domain (e.g. Qt GUI + extensions, MESA OpenGL,…) • Commercial (e.g. OpenInventor, NAG_C, …) • IGUANA work involves evaluation, integration, and support as well as developing extensions and CMS-specific software Core Applications Software

  9. LHC++ or HEP Public- domain IGUANA Commercial IGUANA & Related Software Modules Event Display Plotting Fitting and Statistical analysis Graphical User Interfaces Histograms, persistent tags Core Applications Software

  10. IGUANA Functional Prototype Milestone • Milestone was delayed from June 2000 to October 2000 • Shortage of professional software engineering manpower • Alignment with 1st release of CERN Lizard interactive analysis • This milestone is satisfied (imminent release of IGUANA 2.2.0) • Documented Requirements • Software Infrastructure • Set of Software Prototypes, Packages, and Documentation • Now is the time for reflection and future planning • Document what was learned and current software • Broad discussion of where we are going, priorities,input from the Collaboration Core Applications Software

  11. IGUANA 2.2.1 Graphical User Interface • IGUANA evaluation and prototyping with emphasis on • OO design, C++ API, and standards (facilitates integration) • Functionality, extendibility, widespread adoption, support,… Qt library has all the usual widgets Many other special purpose widgets also available Core Applications Software

  12. IGUANA 2.2.2 Detector and Event Visualization • Usual emphasis on OO design, C++ API, and standards , functionality, extendibility, widespread adoption, support,…. • Build extensions for event display based on HEP / public-domain / commercial software • Development of specific CMS program (Cmscan with CARF & ORCA) Some Details: • Basic graphics technologies • X11, Qt, OpenGL, OpenInventor, SoQt and QGL • IGUANA Viewers • performant 3D rendering • rotations, translations, zoom, slicing, visibility control,... • Interfaces to GEANT3 (CMSIM) & OpenInventor • Full CMS GEANT3 detector could be displayed (very slow!) • IGUANA implements sensible choices of volumes to display • Event Visualization • Implemented within context of ORCA • Currently (ORCA 4.3.0) about half of the obvious reconstructed objects you would like to see are visualizable (work in progress) Core Applications Software

  13. IGUANA 2.2.2Detector and Event Visualization IGUANA Viewer Scene Controller Core Applications Software

  14. IGUANA 2.2.3 Interactive Data Analysis and Presentation • Statistical / Numerical Analysis • Responsibility of CERN/IT API group • Key issue is long-term support for minimization engine • NAG_C engine works in parallel to MINUIT w/ same C++ API • Histograms / Tags • Responsibility of CERN/IT/API (HTL & HepODBMS products) • Extensions /variations on tags…see other talks • Plotting • 1 year ago, not well covered, hence IGUANA developed a plotter • Same basis now adopted by CERN/IT/API for their “Qplotter” • General Purpose Analysis • Many IGUANA prototypes; future emphasis on data interface • CERN/IT/API “Lizard” interactive analysis tool: Oct 2000 Core Applications Software

  15. IGUANA 2.2.3 Many Prototypes HTL Browser & Plotter HTL Browser & Plotter Tag Browser Fitting Core Applications Software

  16. IGUANA 2.2 Future Directions Functional prototype phase finished for all of CMS software—re-examining and prioritising use cases • Graphical User Interface • Continuing support “only” a matter of manpower • Detector and Event Visualisation • Basic functionality in place with ORCA • Committed to steady development • Interactive Data Analysis and Presentation • Very large range of prototypes and products that could be developed • Choosing a few key directions not covered by others Core Applications Software

  17. IGUANA 2.2.3 Interactive Data Analysis and Presentation • Continue work with CERN/IT division to utilize Lizard, which includes a paw-like front-end to existing back-end packages (HTL, HepODBMS, Qplotter, HepFitting) • Move towards closer integrationwith data • We can do much more and better than just a N-tuple today • CMS has made a big jump in data accessibility • A widely used, very powerful object model • Our user interfaces to the new data model need to catch up • Focus on issues of integration: a user-friendly and productive environment built from existing software • Federation management, browsing databases, running different types of analysis jobs—in addition to “mere” presentation • There is a great range of possibilities for user tools, exciting things. Core Applications Software

  18. Distributed Data Management and Processing 2.3 • The Distributed Data Management and Processing software project addresses issues of database replication and access and process management in four sections: • 2.3.1 Distributed Task Management • 2.3.2 Distributed Database Management • 2.3.3 Distributed Production Tools • 2.3.4 System Simulation • Distributed Data Management and Processing will provide tools for the CMS distributed computing model. Allowing submission, monitoring, and control of processes on remote sites; automatic replication and synchronization of databases between region centers; and tools to facilitate production, database access, and system monitoring. • This task aims to provide tools needed for use in production today as it develops more advanced tools for the future. Core Applications Software

  19. DDMP 2.3.1 Distributed Task Scheduling • A system to efficiently handle process management over the computing grid. • Distributed Task Scheduling sub-project has a working prototype. • The service allows clients to submit monitor and terminate jobs as a set. • A scheduling mechanism that allows selection of processors based on processor type, load, and availability of dataset. • Replicated states are maintained so that computations will not be lost if a server fails. • System has been tested on 32 processors with the ORCA production software. • System scaled to 64 processors • Documentation is in progress. Core Applications Software

  20. DDMP 2.3.2 Distributed Database Management • Developing tools to replicate and synchronize databases easily between regional centers. • First Tool came as an investigative prototype written in Perl. • The Second Prototype is based on Grid Data Management Pilot (GDMP) Tool Development, which is a Globus Middleware based toolkit, for database replication and synchronization. Catalogues of the contents in the database are distributed to subscribing sites. Database files are staged if necessary from the HPSS, transferred to the site, and automatically attached to the local federation. • The tools include the ability to resume transfer from checkpoints in the case of network failure. • Web publishing of transfer progress. • Ready for use in the fall ORCA production. Core Applications Software

  21. Integration into the CMS environment Site A CMS environment Physics software CheckDB script GDMP system Write DB DB completeness check CMS/GDMP interface Production federation catalog Site B Purge file Copy file to MSS Stage & Purge scripts Copy file to MSS Stage & Purge scripts MSS MSS Transfer & attach Update catalog Purge file User federation User federation catalog catalog wan Stage file (opt) trigger trigger trigger read GDMP export catalog GDMP import catalog Subscriber’s list Replicate files write Generate import catalog Publish new catalog Generate new catalog GDMP server Core Applications Software

  22. DDMP 2.3.4 MONARC and System Simulation • The MONARC collaboration began in 1998 with the charge of developing a toolkit to use for in modeling large scale computing systems for LHC experiments. • The MONARC toolkit was used to model the CMS spring production using ORCA • As a reminder the spring production was the first large enough to attempt to accurately simulate pile-up events. Combining previously simulated pile-up events with signal events. This involves lot of reading a writing from database servers. • The quality of simulation was only made possible by entering accurate and detailed performance measurements from the production system into the simulation. Considerable work has gone into system monitoring tools. • As an indication of the maturity of the simulation, it has begun used to evaluate Task Scheduling Systems on models of proposed future CMS production facilities: using standard techniques and self configuring neural networks. Core Applications Software

  23. DDMP 2.3.4MONARC and System Simulation • Simulation GUI Core Applications Software

  24. Muon <0.90> Jet <0.52> DDMP 2.3.4 MONARC and System Simulation • Below are simulation examples of network traffic and CPU efficiency Measurement Simulation Core Applications Software

  25. DDMP 2.3 Future Plans • Productizing and Integrating Task Scheduler and Database Replicator to create tools to support the CMS distributed computing model in which jobs are submit to data and data is moved to jobs to maximize efficiency. • Implementation of Request Redirection Protocol in Objectivity on the Federated Database Servers for the Fall production. Should dramatically increase the availability of the database servers. • Continued work on MONARC simulation: simulation of fall production with additional input parameters and a detailed study of tape access to determine modes and guidelines to determine what will be the likely quality of service. Core Applications Software

  26. Support 2.4 • One of the primary goals of hiring dedicated software professionals to work within the CMS Software and Computing project was to offer support to US users and developers so that they would be able to participate fully in CMS physics. The Support Software Project is broken into three areas. • 2.4.1 Developer Support • 2.4.2 User Support • 2.4.3 Software Support Tools • Developer Support Seeks to provide off-project developers with help from professional software engineers: designing, integrating, optimizing, and porting their code. • User Support ensures that documentation is available for publicly released CMS software packages, that current and useful examples exist for CMS software packages, and user have been trained on the CMS software necessary for physics success. Core Applications Software

  27. Support 2.4.2 User Support • Software Configuration Support ensures that publicly released CMS software is properly versioned, is distributable using the agreed upon tools (cvs), and can be configured and built (SCRAM). • A few examples: Properly documented and versioned IGUANA release • All 7 CAS engineers participated in the recent ORCA Tutorial at Fermilab, which over 60 people attended. Core Applications Software

  28. Summary • After hiring the 7 CAS engineers and seeing their skills we have modified the project plan to be coherent. • We have a lot of ideas for the future, but we also have a number of tools that work today. • We are poised to make a significant contribution to the general CMS architecture and design. • We are developing tools that will improve the ways people do physics analysis. • We are developing tools to support the data grid and the distributed computing model. • We have reached a critical mass to offer user and developer software support to CMS physicists Core Applications Software

More Related