catalogue synchronization acl propagation n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Catalogue synchronization & ACL propagation PowerPoint Presentation
Download Presentation
Catalogue synchronization & ACL propagation

Loading in 2 Seconds...

play fullscreen
1 / 15

Catalogue synchronization & ACL propagation - PowerPoint PPT Presentation


  • 109 Views
  • Uploaded on

Catalogue synchronization & ACL propagation. Fabrizio Furano (CERN IT-GT). Outline. The problem The Working Group Goal and milestones Message brokers Communication protocols and guidelines Conclusions. The problem. Various catalogues keep information that is related

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Catalogue synchronization & ACL propagation' - kaoru


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
catalogue synchronization acl propagation

Catalogue synchronization&ACL propagation

Fabrizio Furano (CERN IT-GT)

outline
Outline
  • The problem
  • The Working Group
  • Goal and milestones
  • Message brokers
  • Communication protocols and guidelines
  • Conclusions

F.Furano - Catalogue Synchronization & ACL propagation

the problem
The problem
  • Various catalogues keep information that is related
    • E.g. LFC keeps info about the content of remote SEs, each one with its catalogue
    • A change in the permissions of a file in LFC is not automatically reflected by the peripheric catalogue
    • If a SE looses a file, the LFC des not know
  • Keeping them in sync is a very hard problem

F.Furano - Catalogue Synchronization & ACL propagation

the problem1
The problem
  • Actually, the updates to the various catalogues are performed as external actions
    • By the jobs, as long as they do not die/crash in the wrong moment
    • As scheduled rounds of massive synchronizations

F.Furano - Catalogue Synchronization & ACL propagation

the problem2
The problem
  • Current consistency model is not resilient to failures
    • Storage failures lead to dangling entries to be cleaned up manually. Catalogue failures lead to orphaned files.
  • Namespace scanning for diffs is an expensive workaround

F.Furano - Catalogue Synchronization & ACL propagation

the proposal
The proposal
  • Make the various catalogues/SE able to talk to each other
    • In order to exchange messages that keep them synchronized
    • 2 ways:
      • LFC->SE to propagate changes in the permissions
      • SE->LFC to propagate info about lost files
    • These are just starting examples, what can be done can be even much more interesting

F.Furano - Catalogue Synchronization & ACL propagation

the proposal1
The proposal
  • In other words:
    • Disasters and misalignments will still happen
      • Nobody can prevent a file from disappearing
    • Let’s design a mechanism that ruggedizes the catalogues by means of notifications
    • Let’s design a mechanism to propagate the changes in the LFC to all the interested SEs

F.Furano - Catalogue Synchronization & ACL propagation

the working group
The working group
  • The kickoff mail was sent a few days ago in emi-jra1-data
  • Working group:
    • Jean-Philippe B.
    • Paul M.
    • Riccardo Z.
    • Michele D.
    • Fabrizio F. (Me)

F.Furano - Catalogue Synchronization & ACL propagation

milestone
Milestone
  • Proposed demonstrators to use reliable message (i.e. industry standard MQ) as backbone of the reliability
    • All interested catalogues can “subscribe” for permissions that changed in the LFC
    • Lost files can be broadcast on the “lost” topic to interested catalogues
      • Somehow possible also for corrupted “bad” files (e.g. not readable)

F.Furano - Catalogue Synchronization & ACL propagation

communication
Communication

SE Sends to the appropriate topics (e.g. “Changes”)

LFC Subscribes to the relevant topics (e.g. “Lost”)

LFC

SE sends to the appropriate topics (e.g. “Lost”)

Looking for good ways to reliably communicate and cooperate

Broker(s)

ATLAS

catalogue

SE or exp. catalogue subscribes to the relevant topics (e.g. “Changes”)

SE1

SE2

SEn

F.Furano - Catalogue Synchronization & ACL propagation

message brokers
Message brokers
  • Industry standard MQ brokers, like ActiveMQ will constitute the backbone
  • Need to:
    • Agree on the needed messages (content+semantics)
    • Decide the implementation guidelines

F.Furano - Catalogue Synchronization & ACL propagation

apis and protocols
APIs and protocols
  • Several APIs are available, usable from nearly every language
  • Each API talks to the broker using a protocol. Mostly, in our case:
    • OpenWire (binary)
    • STOMP (XML based)
  • The broker speaks them all and assures the interoperability
  • The idea is not to enforce one protocol, just give usage guidelines

F.Furano - Catalogue Synchronization & ACL propagation

apis and protocols1
APIs and protocols
  • Although compatible, each API/Protocol has its own idiosyncrasies. E.g.:
      • The recent release of the C++ library is simply broken. I had to do all the tests with (their) trunk, with very good results.
      • The reference STOMP implementation that I tested does not support recovery. Any glitch silently makes a zombie client until manual restart.
      • Probably many others.
    • Need to sort out the needed features

F.Furano - Catalogue Synchronization & ACL propagation

conclusions
Conclusions
  • Many aspects have still to be sorted out (E.g. security requirements), but at least we started well.
  • Making catalogues and SEs interact seems a good way to solve the consistency problem
  • The messaging broker tech isnot new,at least we know that it works
  • The performance seems good, eventually it should also be sufficiently scalable
  • Next goal: demo for December 2010

F.Furano - Catalogue Synchronization & ACL propagation

thank you
Thank you

EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611

F.Furano - Catalogue Synchronization & ACL propagation