Juxmem an adaptive supportive platform for data sharing on the grid
This presentation is the property of its rightful owner.
Sponsored Links
1 / 27

JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid PowerPoint PPT Presentation


  • 58 Views
  • Uploaded on
  • Presentation posted in: General

JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid. Gabriel Antoniu, Luc Bougé, Mathieu Jan IRISA / INRIA & ENS Cachan , France. Grid Data Service (GDS) meeting, Rennes, September 22 th 2003. Context: Data Management on the Grid.

Download Presentation

JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Juxmem an adaptive supportive platform for data sharing on the grid

JuxMem: An Adaptive Supportive Platform for Data Sharing on the Grid

Gabriel Antoniu, Luc Bougé, Mathieu Jan

IRISA / INRIA & ENS Cachan, France

Grid Data Service (GDS) meeting, Rennes, September 22th 2003


Context data management on the grid

Context: Data Management on the Grid

  • Distributed numerical simulations (code coupling)

    • Needs data sharing

    • No many functional systems

Designing a satellite

Solid mechanics

Optics

Dynamics

Thermodynamics


Existing data management systems

Existing Data Management Systems

  • Non-transparent large scale data management

    • GridFTP (Globus) and MPI-IO

    • Internet Backplane Protocol (IBP)

    • Explicit transfert

    • No consistency ensured

  • Transparent small-scale data management

    • Distributed shared memory (DSM)

    • Consistency models and protocols

    • Transparent access

    • Small-scale, static and homogeneous architecture

  • TreadMarksTM


    Another approache peer to peer systems

    Another approache: peer-to-peer systems

    • Peer-to-peer systems (P2P)

      • Distributed (large-scale)

      • Volatile peers

      • Peers have the same capacities and responsabilities

  • Sharing of immutable data

    • Centralized (Napster)

    • Flooding (Gnutella, KaZaA)

    • Distributed hash table (CFS, PAST)

  • Sharing of mutable data

    • One writer per data + static architecture (OceanStore)

    • Conflicts have to be manually resolved (Ivy)


  • Design principles

    Design principles

    • Proposition: data sharing service for the grid

      • DSM systems: consistency and transparent access

      • P2P systems: scalability and high dynamicity


    A data sharing service for the grid

    A Data Sharing Service for the Grid

    Persistent storage

    Transparency of localization

    Internet

    ?

    Data transfert


    A data sharing service for the grid1

    A Data Sharing Service for the Grid

    Optimization of access

    Data consistency

    Internet

    Internet

    Data transfert


    A data sharing service for the grid2

    A Data Sharing Service for the Grid

    Internet

    Internet

    Scalability of the architecture


    A data sharing service for the grid3

    A Data Sharing Service for the Grid

    Internet

    Internet

    Handling volatility


    Jxta a framework for p2p

    JXTA: a framework for P2P

    • Open-source platform for programming P2P applications

      • Specify a set of protocols

  • A peer

    • Uniquely identified (ID)

    • Address independent of the physical location

    • Several network access point (TCP, HTTP, etc)

  • Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer

    Peer

    Peer

    TCP/IP

    Peer

    Peer

    Peer

    Peer

    Peer

    Peer

    Firewall

    Firewall

    Peer

    Peer

    Peer

    Peer

    Peer

    HTTP


    Jxta peer groups

    JXTA: peer groups

    • Set of peers that share a common set of interests

      • Scope of communications

      • Specific policy of management

      • Peer group services

    NetPeerGroup

    PeerGroupA

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    Peer ID

    PeerGroupB


    Jxta advertisements

    JXTA: Advertisements

    • Every resource is described by an advertisement

      • Peers

      • Peer groups

      • Communication

        channels

      • Services

    PeerGroup Advertisement:

    <?xml version="1.0"?>

    <!DOCTYPE jxta:PGA>

    <jxta:PGA>

    <GID>

    urn:jxta: uuid- BCBCDEABDBBBABEABBBABA000000

    </GID>

    <MSID>

    urn:jxta:uuid-BFEFDEDFBABAFRUDBACE00000001

    </MSID>

    <Name>

    My Group

    </Name>

    <Desc>

    This group is to be used for my own testing

    </Desc>

    </jxta:PGA>


    Juxmem a prototype

    JuxMem: a prototype

    Peer group juxmem

    Peer group data

    Peer group cluster A

    Peer group cluster C

    Peer group cluster B

    Logical architecture

    Physical architecture


    Api of juxmem

    API of JuxMem

    • Alloc (size, attribs)

    • Map (id, attribs)

    • Put (id, value)

    • Get (id)

    • Lock (id)

    • Unlock (id)


    Managing memory resources

    Managing Memory Resources

    • Advertisement of type provider: peer group cluster

    • Advertisement of type cluster: peer group juxmem

    Peer group cluster

    Memory provided

    Peer group juxmem

    Size 10

    Size 10


    Managing shared data blocks

    Managing Shared Data Blocks

    • Allocation of a memory area = creation of a peer group data

      • Data blocks identified by the ID of the peer group

      • Advertisement published in the peer group juxmem

      • Shared access for clients by knowing the ID

  • Consistency

    • Data blocks are replicated on providers

    • Updated simultaneously (logical multicast)

    • Clients are not notified of updates

  • Synchronization

    • Lock for each data block


  • Handling the volatility of peers

    Handling the Volatility of Peers

    • A manager by peer group (cluster and data)

      • Dynamic monitoring of available peers

      • Data blocks are automatically replicated (data)

      • Updates advertisement of type cluster (cluster)

  • Volatility of managers

    • Periodic exchange of hearbeats

    • Dynamic replication of managers if needed on other peers


  • Implementation of juxmem

    Implementation of JuxMem

    • JuxMem

      • JXTA service

      • + 5000 lines

      • Graphical tool


    Preliminary evaluations

    Preliminary Evaluations

    • Cluster

      • PentiumII: 450 Mhz and 256 Mb of RAM

      • Network used: FastEthernet 100 Mb/s

      • Number of nodes used: 20

  • Experiments

    • Overhead memory consumption

      • Low: @ 6 % with respect to the underlying JXTA peers used

    • Study of the volatility of providers


  • Study of the volatility of providers 1

    Study of the Volatility of Providers (1)

    Data of one byte

    Degree of redundancy = 3

    Data manager is not killed

    1 client: 100 iterations lock-put-unlock

    16 providers

    Peer group juxmem

    Peer group data

    Peer group cluster


    Study of the volatility of providers 11

    Study of the Volatility of Providers (1)

    Data of one byte

    Degree of redundancy = 3

    Data manager is not killed

    1 client: 100 iterations lock-put-unlock

    16 providers

    Peer group juxmem

    Peer group data

    Peer group cluster


    Study of the volatility of providers 2

    Study of the Volatility of Providers (2)

    • Internal lock when replicating

      • Ensure the consistency of the new replica

      • Client is blocked

    Peer group juxmem

    Peer group data

    Peer group cluster


    Study of the volatility of providers 3

    Study of the Volatility of Providers (3)

    • JXTA and Java

    • Time-out detection adapted for the WAN

    • Independent of the data size

    Reconfiguration time @

    11 seconds

    Targeted volatility is weaker ( >> 80 seconds)


    Conclusion

    Conclusion

    • Defines an hierarchical architecture for a data sharing service for the grid

      • Specific policy for each cluster

      • Scalability

  • Storage and transparent access to data blocks

    • Application level: data block ID

    • Persistency

    • Shared consistency

  • Handling volatility of peers

    • Actively taken into account


  • Future work

    Future Work

    • Transparent mobility of data blocks

      • Unavaibility of nodes or even clusters

      • Affinity data – computations

      • Affinity data – data

  • Parameterizable consistency

    • Specific for each client

  • Hierarchical synchronization


  • Allocation request

    Allocation Request

    6

    4

    3b

    1

    2

    5

    3a

    3b

    3a


  • Login