WP2/WP7 Demonstration
Download
1 / 19

WP2/WP7 Demonstration - PowerPoint PPT Presentation


  • 96 Views
  • Uploaded on

WP2/WP7 Demonstration. WP7 High Throughput Data transfers WP2/WP7 Replica Selection based on Network Cost Functions WP2 Replica Location Service. High Throughput Data Transfers. Richard Hughes-Jones Jules Wolfrat. NIKHEF. CERN. Demo Setup.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'WP2/WP7 Demonstration' - lou


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

WP2/WP7 Demonstration

WP7 High Throughput Data transfersWP2/WP7 Replica Selection based on Network Cost FunctionsWP2 Replica Location Service


High throughput data transfers

High Throughput Data Transfers

Richard Hughes-Jones

Jules Wolfrat


Demo setup

NIKHEF

CERN

Demo Setup

  • We will show data transfers from Mass Storage system at CERN to Mass Storage system at NIKHEF/SARA

  • 2 systems at CERN, Geneva with datasets from experiment LHCb

  • 4 Linux systems at NIKHEF/SARA, Amsterdam to where data has to be transferred; each with a disk sub-system I/O bandwidth of ~70 MB/s

  • All systems have Gigabit Ethernet connectivity

  • Use GridFTP and Measure disk to disk performance

GEANT

SurfNet


Demo consists of

GridFTP

GridFTP

Demo Consists of:

Data over TCP Streams

Raid0

Disk

Raid0

Disk

Dante Monitoring

Site Monitoring

Node Monitoring


European topology nrns geant sites
European Topology: NRNs, Geant, Sites

Sara & NIKHEF

SURFnet

SuperJANET4

CERN


Some measurements of throughput cern sara
Some Measurements of Throughput CERN -SARA

  • Using the GÉANT Backup Link

    • 1 GByte file transfers

  • Standard TCP

    • Average Throughput 167 Mbit/s

    • Users see 5 - 50 Mbit/s!

  • High-Speed TCP

    • Average Throughput 345 Mbit/s

  • Scalable TCP

    • Average Throughput 340 Mbit/s


Wp7 high throughput achievements
WP7 High Throughput Achievements

  • Close Collaboration with Dante

  • “Low” layer QOS testing over GEANT

    • LBE

    • IP premium

  • iGrid 2002 and ER 2002 : UDP with LBE

    • Network performances evaluation

  • EU Review 2003 : application level transfer with real data between EDG sites

    • proof of concept


Conclusions
Conclusions

  • More research on the TCP stacks and its implementation is needed

  • Continued the collaboration with Dante to:

    • Understand the behavior of GEANT backbone

    • Learn the benefits of QoS deployment

  • WP7 is taking the “Computer Science” research and knowledge of the TCP protocol & implementation and applying it to the network for real Grid users

  • Enabling Knowledge Transfer to sysadmins and end users

    • EDG release 1.4.x has configuration scripts for TCP parameters for SE and CE

    • Firewalls rules recommendations

    • Network tutorials for end users

    • Work with users – focus on 1 or 2 sites to try to get improvements


Wp2 wp7 replica selection based on network cost functions
WP2/WP7 Replica Selection based on Network Cost Functions

Franck Bonnassieux (WP7)

Kurt Stockinger (WP2)


Networkcost functionality

CERN

RAL

NIKHEF

IN2P3

CNAF

CERN

46,75

77,78

44,87

35,44

RAL

7,46

2,44

7,12

4,35

NIKHEF

11,13

3,25

11,86

2,66

IN2P3

5,03

10,38

6,24

7,08

CNAF

4,5

6,53

4,04

13,08

CERN

RAL

NIKHEF

IN2P3

CNAF

CERN

RAL

NIKHEF

IN2P3

CNAF

NetworkCost functionality

getNetworkCost

FileSize = 10 MB

Results = time to transfer (sec.)


Networkcost architecture

NetworkCost

R-GMA

Globus MDS

Archive

Raw

Distributed Data Collector

PCP

PingEr

IPerf

UDPmon

GridFTP

NetworkCost Architecture

Processing

Collect

And

Storage

Measure


Networkcost model
NetworkCost model

  • The current cost model is designed for data intensive computing and especially large files transfers

    • The most relevant metric for that cost model is available throughput

  • Implementation

    • Iperf Measurements (current)

    • GridFTP Logs (future)

    • Other metrics (future) : UDP, RTT, Jitter, ...

    • Synchronisation (PCP)


Replica management services
Replica Management Services

VO Membership Service

Replica Management Services

Replica Manager Client

Optimization

Information Service

Replica Metadata

File Transfer: GridFTP

Replica Location Service RLS


Testbed sites replica manager commands
Testbed Sites & Replica Manager Commands

edg-rm copyAndRegisterFile -l lfn:higgsCERN  LYON

edg-rm listReplicas -l lfn:higgs

edg-rm replicateFile -l lfn:higgs NIKHEF

edg-rm listBestFile -l lfn:higgs

 CERN

edg-rm getAccessCost -l lfn:higgs

CERN NIKHEF LYON

edg-rm getBestFile -l lfn:higgs

 CERN

edg-rm deleteFile -l lfn:higgs LYON

edg-rm listBestFile -l lfn:higgs

 CERN


Wp2 replica location service

WP2Replica Location Service

Peter Kunszt

WP2 – Data Management


Replica location service rls
Replica Location Service RLS

  • Local Catalogs hold the actual name mappings

  • Remote Indices redirect inquiries to LRCs actually having the file

  • LRCs are configured to send index updates to any number of RLIs

  • Indexes are Bloom Filters



Rls demo topology today
RLS Demo Topology Today

CERN

lxshare0344.cern.ch

Glasgow

grid03.ph.gla.ac.uk

California

dc-n4.isi.edu

Melbourne

wombat.unimelb.edu.au

Replica Location Index

CERN

lxshare0342.cern.ch

Glasgow

grid01.ph.gla.ac.uk

California

dc-n2.isi.edu

Melbourne

koala.unimelb.edu.au

LocalReplicaCatalog


Summary
SUMMARY

  • Replica Optimization

    • WP7 Network cost functions are integrated into the Replica Management functionality providing an essential functionality that was missing up to now.

    • This gives us the necessary framework to start work on high-level optimization algorithms.

  • Replica Location Service

    • Scalable distributed catalog as a much-needed replacement for the current Replica Catalog.

    • Addresses all issues brought up by the experiments. Tests have been conducted with very large catalogs

      • The lookup time for an entry is independent of the number of catalog. Tested for up to 108 entries.

      • The catalog withstands simultaneous user queries of over 1000 queries or inserts per second.


ad