Panda Mover

1. Panda Mover • Normal pilot-server model • Pilot runs a script (dq2_cr) on WN as a transfer agent • Panda server schedules pilots • New job category (prodSourceLabel=‘ddm’) • dq2_cr • takes GUIDs, LFNs,src/destDQ2ID,dest dir • Scans dest LRC/LFC, transfers missing files and register them to dest LRC/LFC • Runs three transfer threads be default • Transfer tools • SRM : glite-srm-copy • srmcp requires too much runtime memory • GSIFTP : glite-url-copy • understands SURL • Prestaging : dccp • 3rd party transfers for SRM/GSIFTP • 6 dedicated WNs at BNL. 20~30 slots per WN

2. Workflow for dispatch datasets Panda Panda DQ2 DB DB Eowyn Eowyn submit submit insert insert subscription instantiate mover job insert mover transfer pilot callback activate get mover transfer Dispatch datasets are no longer DQ2 datasets listFilesInDataset (to get GUIDs)  registerNewDataset  registerDatasetLocation  registerDatasetSubscription callback activate

3. Fare-share and Scheduling • Each job defines prodSourceLabel • Movers : ddm • Production jobs : managed • Analysis jobs : user • Each mover defines transferType • For dispatch DS : dis • For destination DS : sub • For general replication : ddm • Pilots send requests with prodSourceLabel=XYZ and pickup high-priority jobs • XYZ is used for match-making • A transfer channel is defined as (sourceSite,destinationSite,transferType)

4. Fare-share and Scheduling (cntd.) • The number jobs to be released to pilots can be configured for each channel • 10 jobs for each dis channel by default • 30 = 103 files are transfered in pararel • 50 jobs for prestaging (BNL,BNL,dis) • Retry for mover • Twice for one channel • Jobs are reassigned five times • The mover is assigned to new channel  10 attempts • Once one attempt fails, the mover sleeps for 3 hours