Cms status requirements
This presentation is the property of its rightful owner.
Sponsored Links
1 / 20

CMS Status & Requirements PowerPoint PPT Presentation


  • 64 Views
  • Uploaded on
  • Presentation posted in: General

CMS Status & Requirements. Topics covered: CMS Grid Status CMSUK approach to Grid work First stab at high-level requirements Proposed UK work areas Other sources of info: Our EB/TAG submission – this talk summarises it. CMS Note 2001/037 (Holtmann et al)

Download Presentation

CMS Status & Requirements

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Cms status requirements

CMS Status & Requirements

  • Topics covered:

    • CMS Grid Status

    • CMSUK approach to Grid work

    • First stab at high-level requirements

    • Proposed UK work areas

  • Other sources of info:

    • Our EB/TAG submission – this talk summarises it.

    • CMS Note 2001/037 (Holtmann et al)

      • Do NOT read as a fixed statement of CMS approach or priorities

      • Our approach has already evolved since then…

    • DataGrid WP8 validation proposal – public at end of this week.

  • DMN at CERN for CMS software week – apologies sent.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms grid status

CMS Grid Status

  • Many active Grid groups within CMS

    • US (Caltech, FNAL, UCSD, …); INFN; CERN

  • Some Grid technologies already used ‘in production’

    • Data replication via GDMP (for 12 months)

    • Data movement across WAN using bbcp, and now GridFTP

  • …and some have been tried in large-scale tests

    • CONDOR / DAGman

  • Formal CMS Grid Project just started

    • Coordinator: C. Grandi (Bologna)

    • Oriented toward coordination of existing work areas + manpower

    • Many of the ‘obvious’ work areas already well covered

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

28 Pages

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

CMS is developing a detailed vision of the Grid.

CMS Overview and Requirements Document

“The document contains a snapshot, taken in 2001, of the vision of the intended software capabilities of the production system in 2003, and the expected scaling towards 2007”

Document admits that the “vision” will evolve….

But contains architectural choices and constraints.

There is a detailed “Division of labour” between tasks for

grid-projects components and tasks for CMS components.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

2003 CMS Grid-Job (Vision)

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

Pythia

CMSIM

(Geant3)

C)

Pythia

C)

HEPEVT

Ntuples

Zebra files

with Hits

DataBase

ORCA

Add Pile-up

ORCA

ooHit

Formatter

CMS has considerable experience already of distributed computing on large scales.

ORCA

User

Analysis

Histograms

Ntuples

Root files...

CMS Software Today

Pythia

Production presently at about 10 regional centres - UK one of the largest - has produced 10TB of data (15TB by eoy)

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms production status

CMS Production Status

  • MC Production aims:

    • Satisfy (increasing) requests for samples by physics/HLT groups; will get worse as HLT + Physics TDRs approach

    • Investigate scalability of hardware, software, people (!)

    • Put in place the correct communication mechanisms + policies

  • Status:

    • We are now/soon at the 10’s of TB scale

    • Production runs quasi-continuously, worldwide.

    • We are continually find (sometimes even solve!) the expected scaling, communication problems…

    • CMS physicists:

      • Gripe a lot

      • Demand lots of support + a high level of reliability

      • Are highly clued-up, and provide very useful input + steerage

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

Can these milestones contribute to the success of GridPP,

and vice versa?

CMS Milestones

Dec-01:2x10*7 MC events (reduced processing chain).

Jun-02:2x10*7 MC events (full chain and storage).

Late-02:Specification of final CMS software system for LHC era.

Dec-02:Higher-level-trigger TDR completed.

Dec-03: Computing TDR completed; includes 5% challenge.

Late-04:20% data challenge.

Dec-04:Physics TDR completed.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

CMS in the UK

Main UK contribution to date has been the development of MC production software and its exploitation to meet ongoing simulation requirements.

The UK has provide over 25% of the data to date.

Plan is to maintain this high profile (much greater than our size).

Plan to contribute via GridPP in two ways:

Assist with the Grid-enabling of our current world-wide

MC production system.

Deploy test and operate production software at prototype Tier-1, 2, 3 centres in the UK to test the scalability and performance of the developing software.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Requirements protot1

Requirements: protoT1

  • What do we need?

    • A rapidly-deployed protoT1 at RAL

    • Scale attained should be ~0.5 of a Hoffmann T1 centre by end 2003 (all available to CMS for short periods)

    • Good + responsive hardware and operations support / expertise

    • Direct support for the tools we need to use there

      • Large, fast database systems

      • Commercial software installations (e.g. LHC++, Objy, Oracle)

  • Is this a ‘technical requirement’? Yes (IMHO).

    • There is more to building / operating a T1 than stacking boxes (i.e. we need a real programme of R&D)

    • Industry does NOT yet know how to do this in a cost-effective way

    • CMS has valuable experience + real plans for this area.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Requirements middleware

Requirements: Middleware

  • Middleware / uppermiddleware req’d by CMS includes:

    • Data management

      • Covered (for flat files) by GDMP

    • Workload management

      • Lots of work being done in US, DataGrid projects, INFN

    • Monitoring / control

      • A UK contribution?

  • How to integrate middleware?

    • No plans to ‘Gridify’ our core applications (OSCAR, ORCA, etc)

    • We take a ‘toolkit’ approach for our production (-> analysis) framework; will start to introduce Grid middleware under the toolkit, where it makes life easier.

    • Integration + testing will be a primary technical task in the UK.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Requirements oostuff i

Requirements: ooStuff (I)

  • Object data management (from a non-expert):

    • Traditionally, HEP experiments have used flat-file storage

    • Flat files are ‘easy’ to deal with, but often inefficient, reliant upon huge and complex metadata system, not amenable to ‘virtual data’ approach.

    • CMS plans to use a true object-based data model, and a corresponding persistent object store.

  • The problem:

    • OS / Mass storage only knows about flat files

    • Worse: data replication tools only know about flat files

    • Vendor-supplied data management tools are inadequate / insecure for use on the WAN

    • Files will get very big soon (remember: 2GB limit will not exist for much longer) – no longer a good unit of data transfer.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Requirements oostuff ii

Requirements: ooStuff (II)

  • A short-term solution:

    • CMS will need to implement an object clustering/declustering layer

    • Stuffs object collections into flat files when required for: Archiving; Data Movement; Creation of compact ‘final stage’ datasets from tagged collections

    • Destuffs them for efficiency when required.

  • Long-term solution:

    • Our data management / metadata tools need to understand object granularity (or at least, object collections and their properties).

    • This is not a well-understood problem (to say the least); lots of interesting computer science here

    • Presumably (?) very interesting to industry.

  • A role for GridPP / UK ?

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Requirements interoperability

Requirements: Interoperability

  • We have middleware coming from all over the place

    • Probably a more visible problem within CMS than anywhere else right now.

  • Please please please:

    • Coordinate at a high level between projects

    • Make sure the tools work together wherever possible

    • Avoid architectural mismatches.

  • Otherwise:

    • CMS will have to ‘make a choice’

    • GridPP / UK T1 will have to support it.

    • We will all have to test / integrate twice as much stuff

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Cms status requirements

CMS-UK: Areas of interest

DataGrid deliverables within the WP8 programme (installation,

testing, organisation and support of data challenges).

MC Grid development: Introducing more grid-elements into the CMS data challenges.

Development and integration of object database into regional centres.

Monitoring tools for CMS applications and T1 hardware.

Object-level replication tool development.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Uk proposed work areas i

UK Proposed Work Areas (I)

  • NOT in any priority order

  • protoT1 database management/support

    • Primary goal: to enable and support use of last, fast, modern object database systems at the protoT1

    • Some ‘operations’ tasks

    • Some technical development tasks (which can be monitored / managed): e.g. integration with Grid StorageElement.

    • Also a very important liaison role with

      • CERN / SLAC / whoever

      • Vendors

    • This is a necessary condition for use of the Tier-A/protoT1 by CMS and BaBar

      • We intend to collaborate in this area.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Uk proposed work areas ii

UK Proposed Work Areas (II)

  • Production / analysis Grid framework:

    • The enhancement of our existing MC production toolkit (IMPALA) using Grid technology

    • This project is starting NOW

    • Integrates with DataGrid WP8 work to some extent

    • Will lead directly to Grid analysis framework development (no acronym yet assigned!)

  • UK contribution:

    • Use our experience to guide detailed requirements defn

    • Deploy and test first tools at our protoT1/T2/T3 sites

    • Investigate possible collaborations with other expts

    • Also: provide input from working physicists who need the tools!

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Uk proposed work areas iii

UK Proposed Work Areas (III)

  • Monitoring / T1 management + simulation

    • Development of tools to allow effective:

      • Monitoring of workflow

      • Monitoring of application performance

      • Analysis of bottlenecks + computing system performance

      • Comparison of T1 performance with simulation

      • Infra-T1 workflow scheduling and work pattern optimisation

    • Builds upon UK expertise in monitoring, info services, data publication

    • Builds upon MONARC work towards ‘real’ simulation of a local or worldwide computing system

    • Builds upon UK interest (incl. computer science) in network performance, application optimisation, workflow management.

    • First tools required (Real Soon) for optimisation of work at T1.

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Uk proposed work areas iv

UK Proposed Work Areas (IV)

  • Object data management / replication:

    • Provision of short-term solutions to this problem

      • Provision of a ‘clustering layer’

      • A good example of an indirect contribution by Grid projects to solve a real ‘core software’ problem

    • Investigation of possible long-term approaches

    • Lots of potential collaboration with

      • US projects

      • Replication tool developers in DataGrid (etc)

      • Computer scientists

      • Industry

    • This work should not be tied to one DB product or approach

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


Summary

Summary

  • CMS and the Grid:

    • We have considerable experience already. We are already “pushing-the-envelope” with our MC productions.

    • We have uncovered many problems, issues, impedance mismatches already. CMS is developing a grid plan.

  • The UK contribution:

    • Build upon our practical experience / expertise.

    • Make use of the large protoT1 we will have

    • Focus upon:

      • integration / deployment / testing

      • interesting development projects needed by CMS

  • There are real, interesting, CMS-related things to do

    • GridPP can make a big contribution to CMS

    • CMS UK data challenges can make a big contribution to GridPP

D. Newbold / D. BrittonGridPP Collaboration Meeting, 5/11/2001


  • Login