Developing a data management system for the atlas experiment l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 19

Developing a Data Management System for the ATLAS Experiment PowerPoint PPT Presentation


Developing a Data Management System for the ATLAS Experiment. September 20, 2005 Miguel Branco [email protected] Outline. ‘Data Challenges 2’ and ‘Rome Production’ Lessons Learned DQ2 Design Implementation Data model Services Conclusion. DC2 and Rome Production.

Download Presentation

Developing a Data Management System for the ATLAS Experiment

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Developing a data management system for the atlas experiment l.jpg

Developing a Data Management System for the ATLAS Experiment

September 20, 2005

Miguel Branco

[email protected]

ATLAS DDM


Outline l.jpg

Outline

  • ‘Data Challenges 2’ and ‘Rome Production’

  • Lessons Learned

  • DQ2

    • Design

    • Implementation

    • Data model

    • Services

  • Conclusion

ATLAS DDM


Dc2 and rome production l.jpg

DC2 and Rome Production

  • Production started Spring 2004 and finished recently

  • “ProdSys”

  • Data Management (DQ): high-level service that interacted with all ATLAS Grid catalogs and storages

    • File-based: relied on backend RLS (Globus RLS, EDG RLS)

    • Also implemented a simple reliable file transfer (FIFO queue)

Supervisors: collect jobs from production database dispatch to executors

Executors (per ‘Grid’): translate physics definition to a Grid job and launch it

DQ: All components interacted with data management


A few of the lessons learned l.jpg

(a few of the) Lessons learned

  • Catalogs were provided by Grid providers and used “as-is”

  • Granularity: file-level. No datasets, no “file collections”

    • No scoping of queries (difficult to find data, slow)

  • No bulk operations for most operations

  • No managed and transparent data access from grid m/w

    • GridFTP, SRM unreliable (error reporting);

      • Problems with mass storage. Situation has since improved

    • Difficult to handle multiple conditions from Grid ‘abstraction’

      • Dynamically mapping users to mass storage stagers?

  • Metadata support not usable; too slow

    • Logical Collection Name as metadata ‘string’: /datafiles/rome/…

  • Catalogs not always geographically distributed

    • Single point of failure (middleware, people/timezones)

  • No “ATLAS resources information system” (with known/negotiated QoS)

    … and unreliable information systems from Grid providers, when available

  • Operational problems

    • Timezones, lack of people, experience, communication

ATLAS DDM


Dq2 design rationale l.jpg

DQ2 Design rationale

  • Evolve from past experience

  • Interoperability

  • Grid m/w components: Start with:

    • Replica Catalog, Storage Management, Reliable File Transfer

  • Global != Site != Worker Node != Client Tools

  • Handle Production and User Analysis

    • Well-defined data flows and chaotic data movement

  • Security

  • Datasets, not files…

  • Bulk requests native to the design and interfaces

  • Scalability

    • Administrative, Geographical, Load

ATLAS DDM


Slide6 l.jpg

DQ2

  • Moves from a file based system to one based on datasets

    • Hides file level granularity from users

    • A hierarchical structure makes cataloging more manageable

    • However file level access is still possible

  • Scalable global data discovery and access via a catalog hierarchy

  • No global physical file replica catalog

    • but global dataset replica catalog and global dataset location catalog

Files

Files

Datasets

Sites

Files

Files

Dataset

Files


Catalog architecture and interactions l.jpg

Catalog architecture and interactions

ATLAS DDM


Global catalogs l.jpg

‘Global’ catalogs

Dataset

Repository

Holds all dataset names and unique IDs (+ system metadata)

Dataset

Hierarchy

Maintains versioning information and information on

‘container datasets’, datasets consisting of other datasets

Dataset

Content

Catalog

Maps each dataset to its constituent files

This one holds info on every logical file so must be highly scalable,

however it can be highly partitioned using metadata etc..

Dataset

Location

Catalog

Stores locations of each dataset

All logically global but may be distributed physically

ATLAS DDM


Local catalogs l.jpg

‘Local’ Catalogs

Local

Replica

Catalog

Per grid/site/tier providing logical to physical file name mapping.

Implementations of this catalog are Grid specific but

must use a standard interface.

Claims

Catalog

Per site storage, keeping user claims on datasets.

Claims are used to manage stage lifetime, resources and

provide accounting.

Currently all ‘Local’ catalogs are deployed per ATLAS site

ATLAS DDM


Datablocks l.jpg

Datablocks

  • Datablocks are defined as immutable and unbreakable collections of files

    • They are a special case of datasets

    • A site cannot hold partial datablocks

    • There are no versions for datablocks

  • Used to aggregate files for convenient distribution

    • Files grouped together by physics properties, run number etc..

    • Much more scalable than file level distribution

    • Useful for provenance: immutable sets of data

  • The principal means of data distribution and data discovery

    • immutability avoids consistency problems when distributing data

    • moving data in blocks improves data distribution (eg. datablock can map to bulk SRM copy or stage request)

ATLAS DDM


Subscriptions l.jpg

Subscriptions

  • A site can subscribe to data

    • When a new version is available, this latest version of the dataset is automatically made available through site-local specific services carrying out the required replication - Automated movement

  • Subscriptions can be made to datasets (for file distribution) or container datasets (for datablock distribution)

  • Use cases:

    • Automatic distribution of datasets holding a variable collection of datablocks (container datasets)

    • Automatic replication of files by subscribing to a mutable dataset (eg file-based calibration data distribution)

Site ‘X’:

Dataset ‘A’

Subscriptions:

File1

File2

Dataset ‘A’ | Site ‘X’

Dataset ‘B’ | Site ‘Y’

(Container) Dataset ‘B’

Site ‘Y’:

Data block1

Data block2


Subscriptions12 l.jpg

File1

DS1

File2

File3

DB1

CDS1

Subscribes

to CDS1

DB2

DB3

DB1

“Subscribes” (temporarily)

to DB1

Subscriptions

  • Various data movement use cases…

    • Datasets:

      • latest version of a dataset (triggers automatic updates whenever a new version appears)

    • Container Datasets:

      • which in turn contain datablocks or datasets

      • supports subscriptions to the latest version of a container dataset (automatically triggers updates whenever e.g. the set of datablocks making up the container dataset changes)

    • Datablocks (single copy of immutable set of files)

    • Databuckets (diagram next slide)

      • replication of a set of files using notification model (whenever new content appears on the databucket, the replication is triggered)

Subscribes

to DS1

Dataset Location

Catalog updated


Data buckets l.jpg

Remote Site

File 2

File 1

File 1

Data “bucket”

(file-based data bucket)

Data buckets

  • Data must be replicated (quickly) not by the appearance of a new version but by new content

    • alternative would be constantly defining new versions of datasets!

  • Will use notification model:

    • Whenever new content appears on a data bucket, sites subscribing to it are notified and data is moved accordingly

  • Data buckets can contain files

  • Data buckets can contain datablocks

ATLAS DDM


Claims l.jpg

Claims

  • Claims catalog manages the usage of datasets

    • User requests have a lifetime

    • Claim is assigned

    • User may add claims on existing datasets

    • Claim owner may (should) release claim when done

    • Claim owner may extend lifetime of claim

    • Automatically handled by user client tools

  • Behavior

    • Each claim has an expiration time (now plus lifetime)

    • Claim is active until released or expired

    • Datasets may have multiple active claims for different users

  • Claims is a generic approach to handle usage of data

    • Other tools use it for cache-turnover (expired claims), accounting, policy enforcement and even for dealing with mass storage (claim triggers SRM stage request)

ATLAS DDM


Implementation l.jpg

Implementation

  • Architectural Style

    • REST-style (not entirely RESTful)

    • Communication: intend to migrate non performance-critical payload (monitoring, real-time status reporting) to XML soon

      • vocabularies will emerge from experience of running system

  • Development

    • First usable prototype deployed 47 days after project started

  • Technology choices

    • HTTP text/plain; HTTP text/xml

    • Python; servers hosted on Apache (mod_python, mod_gridsite); clients using PyCurl

    • POOL File Catalog interface gives us choice of back-end for catalogs

      • Globus RLS, LCG FC, MySQL, Oracle, XML

    • File movement: SRM (v1.1), GridFTP, gLite FTS, HTTP, dccp, cp

  • Security

    • Use HTTPS (with Globus proxy certs) for POST/PUT/DELETE and HTTP for GETs, ie world-readable data, best performance (can be made secure to ATLAS VO if required)


Api http l.jpg

API (HTTP)

GET http://atl02dq.cern.ch/dq2/repository/dataset?lfn=some_atlas_dataset

GET http://atl02dq.cern.ch/dq2/content/dataset?lfn=some_atlas_dataset

POST http://atl02dq.cern.ch/dq2/repository/dataset PAYLOAD “dsn=DSN”

POST http://atl02dq.cern.ch/dq2/content/dataset PAYLOAD “[guids] [files]”

ATLAS DDM


Detail on subscriptions l.jpg

Detail on Subscriptions

Function

Agents

State Machine

Fetcher

Finds incomplete datasets

unknownSURL

ReplicaResolver

Finds remote SURL

knownSURL

MoverPartitioner

Assigns Mover agents

assigned

Mover

Moves file

toValidate

ReplicaVerifier

Verifies local replica

(usable standalone)

validated

BlockVerifier

Verifies whole dataset complete

done

List of software required to handle subscriptions.

Requires minimal deployment effort (laptop support!)


Conclusion l.jpg

Conclusion

  • Evolve the model based on past experience

    • based on proven technologies

  • Appears to scale so far

    … although it remains to be proven on mass-scale production

    • load: datasets; subscriptions running per site; multiple catalogs

    • geographic: local site information stays local (eg. SURLs); site services

    • very important: administrative scalability: ATLAS users can manage their site installation

  • It is running now across some US ATLAS and LCG sites

    • Ramping up (starting now!) to the full set of LCG and US ATLAS resources.

      http://uimon.cern.ch/twiki/bin/view/Atlas/DDM


Summary of services l.jpg

Summary of Services

  • Global services

    • Dataset catalogs

    • Requirements: grid environment, database, Apache services

  • Site services

    • Subscriptions, Databuckets, Claims (plus daemons running per storage) and minimal information system (monitoring, real-time reporting)

    • Requirements: grid environment, database, Apache services, DQ2 agents and daemons for moving data & claims & validation, grid-specific data movement clients, Python, PyCURL, grid (host) certificate

  • Local worker node client

    • Contact local LRC, get and put data to local Storage

    • Requirements: grid environment, (grid-specific data movement clients?)

  • Clients

    • Define datasets and datablocks, subscribe them to sites

    • Associate files with new dataset versions

    • Query dataset definition, contents, location

    • Requirements: Python, PyCURL, grid certificate (for ‘writing’)


  • Login