Object based storage
Download
1 / 48

Object-based Storage - PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on

Object-based Storage. Long Liu 2010-10-23. Outline. Why do we need object based storage? What is object based storage? How to take advantage of it? What's the status of object based storage? What can we do about it?. Outline. Why do we need object based storage?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Object-based Storage' - heidi-charles


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Object based storage

Object-based Storage

Long Liu

2010-10-23


Outline
Outline

  • Why do we need object based storage?

  • What is object based storage?

  • How to take advantage of it?

  • What's the status of object based storage?

  • What can we do about it?


Outline1
Outline

  • Why do we need object based storage?

  • What is object based storage?

  • How to take advantage of it?

  • What's the status of object based storage?

  • What can we do about it?


Background
Background

  • Existing enterprise storage infrastructures are feeling the strain

  • the volume of data generated by many network-based applications continues to escalate


Background1
Background

Two Technologies:

  • Files:

    • Flexible data sharing

    • Secure

  • Blocks:

    • High performance

    • Scalable

Files

Blocks


Comparison
Comparison

Objects can be regarded as the convergence

of two technologies: files and blocks

Operations:

Create object

Delete object

Read object

Write object

Addressing:

[object, byte range]

Operations: Read block Write block

Addressing: Block range

Object Based Disk

Block Based Disk


Comparison1
Comparison

Traditional Storage

Object-based Storage

Applications

Applications

File System

File System

Object Interface

Storage component

Logical Block Interface

Hard Drive

Object-based Storage Device (OSD)


Comparison2
Comparison

CPU

Applications

System Call Interface

File System

User Component

File System

Storage Component

Block I/O Manager

Storage Device

(a)

Traditional model

(b)

Object storage

model

  • Two changes :

  • Object-based storage offloads the storage component to the storage device

  • The device interface changes from blocks to objects

  • CPU

  • Applications

  • System Call Interface

  • File System

  • User Component

  • Object Interface

Block Interface

File System

Storage Component

Block I/O Manager

Storage Device


Motivation
Motivation

Improved device and data sharing

– Platform-dependent metadata moved to device

• Improved scalability & security

– Devices directly handle client requests

– Object security

• Improved performance

– Data types can be differentiated at the device

• Improved storage management

– Self-managed, policy-driven storage

– Storage devices become more autonomous

Objects


Outline2
Outline

  • Why do we need object based storage?

  • What is object based storage?

  • How to take advantage of it?

  • What's the status of object based storage?

  • What can we do about it?


Object based storage1
Object-based Storage

  • Object

  • OSD(Object-based Storage Device)

  • MDS(Metadata Server)


Object based storage2
Object-based Storage

  • Object

  • OSD(Object-based Storage Device)

  • MDS(Metadata Server)


Object
Object

  • An object is a logical unit of storage

    • ID (Identification)

    • Application data

    • Metadata which includes block allocation and length

    • Attributes that is accessible by users

  • Objects have file-like methods

    • open, close, read, write


  • Object1
    Object

    • The root object -- The OSD itself

    • User object -- Created by SCSI commands from the application or client

    • Collection object -- A group of user objects, such as all .mp3

    • Partition object -- Containers that share common security and space managementcharacteristics


    Object2
    Object

    P4

    P3

    P2

    OSD

    P1

    Root Object

    (one per device)

    Partition Objects

    U1

    User Data

    Collection Objects

    Metadata

    Attributes

    • User Objects(for user data)

    Object ID


    Object based storage3
    Object-based Storage

    • Object

    • OSD(Object-based Storage Device)

    • MDS(Metadata Server)



    Object Storage Devices

    CPU

    Applications

    System Call Interface

    File System

    User Component

    File System

    Storage Component

    Block I/O Manager

    Storage Device

    (a)

    Traditional model

    (b)

    Object storage

    model

    • Two changes :

    • Object-based storage offloads the storage component to the storage device

    • The device interface changes from blocks to objects

    • CPU

    • Applications

    • System Call Interface

    • File System

    • User Component

    • Object Interface

    Interface

    File System

    Storage Component

    Block I/O Manager

    Storage Device


    Object storage devices1
    Object Storage Devices

    Expect wide variety of Object Storage Devices

    Disk array subsystem

    2 SATA disks – 240/500 GB

    Highly integrated, single disk

    4 Gbps per

    shelf to cluster

    Orchestrates system activity

    Balances objects across OSDs

    Stores up to 5 TBs per shelf


    Object based storage4
    Object-based Storage

    • Object

    • OSD(Object-based Storage Device)

    • MDS(Metadata Server)



    Client

    Storage component

    Metadata Servers (MDS)

    Object Storage Devices (OSDs)

    Object-based Storage Device (OSD)

    Dataflow of Metadata

    Applications

    Client

    File System

    Metadata Manager


    Outline3
    Outline

    • Why do we need object based storage?

    • What is object based storage?

    • How to take advantage of it?

    • What's the status of object based storage?

    • What can we do about it?


    T10 osd commands face

    Interface

    OSD Client

    OSD Interface

    T10 OSD Commands(face)

    CREATE/REMOVE

    GET/SET ATTR

    READ/WRITE

    Opaque

    attributes

    (stored only)

    Shared

    attributes

    (stored &

    processed)

    Attribute pages

    OSD Target

    10001110101

    10000001110

    11001110111

    10001111000

    ..…

    User Data

    Object


    Osd commands interface
    OSD Commands(Interface)

    • Security

    – Authorization

    – Integrity

    – SET KEY

    – SET MASTER KEY

    • Groups

    – CREATE COLLECTION

    – REMOVE COLLECTION

    – LIST COLLECTION

    • Management

    – CREATE PARTITION

    – REMOVE PARTITION

    – FLUSH PARTITION

    – PERFORM SCSI COMMAND

    – PERFORM TASK MGMT

    • Basic Protocol

    – READ

    – WRITE

    – CREATE

    – REMOVE

    – GET ATTR

    – SET ATTR

    • Specialized

    – APPEND

    – CREATE & WRITE

    – FLUSH

    – LIST

    very basic

    shared secrets

    space mgmt

    attributes

    • opaque

    • internal

    • shared


    Storage technology today
    Storage Technology Today

    • Direct attached storage (DAS)

    • Fabric Attached Storage (FAS)

      -Network Attached Storage (NAS)

      -Storage Area Networks (SAN)


    Direct attached storage
    Direct Attached Storage

    LAN

    Windows

    UNIX

    Windows

    A traditional Direct Attached Storage model

    RAID

    BACKUP SERVER

    RAID

    RAID

    RAID

    TAPE


    Fabric attached storage
    Fabric Attached Storage

    C/S

    Fabric Attached Storage

    SERVER

    DATA

    CLIENTS


    Network attached storage
    Network Attached Storage

    This figure illustrates NAS being used to share files among a number

    of clients.

    Clients

    File I/O

    IP network

    Storage area network

    Block I/O

    The files themselves may be stored on a fast SAN

    File server

    Block storage


    Storage area networks
    Storage Area Networks

    Clients

    This figure illustrates a SAN file system

    The files themselves are stored on a fast storage Area to which the clients are also attached.

    Servers

    Metadata

    Storage area network

    Data

    Management

    Block-based storage devices


    Security
    Security

    CLIENTS

    METADATA SERVERS

    Attribute

    Capability

    NETWORK

    Attribute

    Capability

    Data

    Management

    OBJECT-BASED STORAGE DEVICE


    Outline4
    Outline

    • Why do we need object based storage?

    • What is object based storage?

    • How to take advantage of it?

    • What's the status of object based storage?

    • What can we do about it?


    Status
    Status

    • Industrial

      • Lustre

      • Panasas

    • Academic


    Status1
    Status

    • Industrial

      • Lustre

      • Panasas

    • Academic


    Lustre
    Lustre

    • First open sourced system with object storage

    • High-performance parallel file system

    • Consist of clients, MDS and OST(Object Storage Targets)


    Lustre1
    Lustre

    Metadata

    Metadata

    METADATA SERVER

    CLUSTER

    Data

    Management

    NETWORK

    CLIENTS

    Management

    Data

    OBJECT STORAGE TARGETS(OST)


    Panasas
    Panasas

    • High-performance file system

    • Consist of OSD, Panasas File System, MDS


    Panasas1
    Panasas

    KeyObjectStorageFeatures

    Intelligent space management in storage layer

    ßMedia geometry aware placement

    ßData aware prefetching, caching & recovery

    Encapsulation of data and attributes

    ßNative object interface, good programming model

    ßStorage interpreted attributes for per file properties

    KeyObjectStorageAdvantages

    ßRobust, shared access by many clients

    ßScalable performance via an offloaded data path

    ßStrong fine-grained end-to-end security


    Panasas2
    Panasas

    • Clients are from Energy, Government, Finance, Manufacturing and Higher Education


    Status2
    Status

    • Industrial

      • Lustre

      • Panasas

    • Academic


    A design of metadata server cluster in large distributed object based storage
    A Design of Metadata Server Cluster In Large Distributed Object-based Storage

    Motivation:

    • Metadata server cluster maybe the bottleneck

    • Frequent metadata access and movement

    • Terrible load balance management


    A design of metadata server cluster in large distributed object based storage1
    A Design of Metadata Server Cluster In Large Distributed Object-based Storage

    Application Server Cluster

    VoD

    Server

    Web

    Server

    E-mail

    Server

    File

    Server

    Database

    Server

    Metadata

    Data

    Storage Network

    (Fibre Channel)

    MDS

    Cluster

    Security

    Object-based Storage Device Cluster

    Object-based Storage System Architecture


    A design of metadata server cluster in large distributed object based storage2
    A Design of Metadata Server Cluster In Large Distributed Object-based Storage

    Application Servers

    Application

    Hashing Partition:

    File Hashing Manager

    Mapping Manager

    Metadata Server Cluster

    • A total solution for

      • File hashing

      • Metadata partitioning

      • Metadata storage

    Hashing Partition

    Logical Partition Manager

    Metadata Server Backend

    Common Storage Space

    Figure 3. Hashing Partition


    A design of metadata server cluster in large distributed object based storage3
    A Design of Metadata Server Cluster In Large Distributed Object-based Storage

    Pathname: /Dir1/Dir2/filename

    1

    4

    ①.Filename hashing

    ②.Selecting MDS through Mapping Manager

    ③ .Accessing metadata by pathname hashing result

    ④.Returning metadata to application server

    Pathname Hashing

    Result (i)

    Pathname

    Mapping Manager

    Metadata

    &

    etc

    2

    Pathname Hashing

    Result (i+1)

    Pathname

    Metadata Server Cluster

    Metadata

    &

    etc

    3

    Logical Partitions

    Figure 4. Metadata Access Pattern


    A design of metadata server cluster in large distributed object based storage4
    A Design of Metadata Server Cluster In Large Distributed Object-based Storage

    Hashing Partition

    Mapping Manager

    2

    1

    3

    Metadata Server Cluster

    Logical Partitions

    4

    Common Storage Space

    Figure 5. MDS cluster failover procedure


    Outline5
    Outline Object-based Storage

    • Why do we need object based storage?

    • What is object based storage?

    • How to take advantage of it?

    • What's the status of object based storage?

    • What can we do about it?


    From our perspective
    From Our Perspective Object-based Storage


    The end
    The End Object-based Storage

    Thank you


    ad