object based storage
Download
Skip this Video
Download Presentation
Object-based Storage

Loading in 2 Seconds...

play fullscreen
1 / 48

Object-based Storage - PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on

Object-based Storage. Long Liu 2010-10-23. Outline. Why do we need object based storage? What is object based storage? How to take advantage of it? What\'s the status of object based storage? What can we do about it?. Outline. Why do we need object based storage?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Object-based Storage' - heidi-charles


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
object based storage

Object-based Storage

Long Liu

2010-10-23

outline
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
outline1
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
background
Background
  • Existing enterprise storage infrastructures are feeling the strain
  • the volume of data generated by many network-based applications continues to escalate
background1
Background

Two Technologies:

  • Files:
    • Flexible data sharing
    • Secure
  • Blocks:
    • High performance
    • Scalable

Files

Blocks

comparison
Comparison

Objects can be regarded as the convergence

of two technologies: files and blocks

Operations:

Create object

Delete object

Read object

Write object

Addressing:

[object, byte range]

Operations: Read block Write block

Addressing: Block range

Object Based Disk

Block Based Disk

comparison1
Comparison

Traditional Storage

Object-based Storage

Applications

Applications

File System

File System

Object Interface

Storage component

Logical Block Interface

Hard Drive

Object-based Storage Device (OSD)

comparison2
Comparison

CPU

Applications

System Call Interface

File System

User Component

File System

Storage Component

Block I/O Manager

Storage Device

(a)

Traditional model

(b)

Object storage

model

  • Two changes :
  • Object-based storage offloads the storage component to the storage device
  • The device interface changes from blocks to objects
  • CPU
  • Applications
  • System Call Interface
  • File System
  • User Component
  • Object Interface

Block Interface

File System

Storage Component

Block I/O Manager

Storage Device

motivation
Motivation

Improved device and data sharing

– Platform-dependent metadata moved to device

• Improved scalability & security

– Devices directly handle client requests

– Object security

• Improved performance

– Data types can be differentiated at the device

• Improved storage management

– Self-managed, policy-driven storage

– Storage devices become more autonomous

Objects

outline2
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
object based storage1
Object-based Storage
  • Object
  • OSD(Object-based Storage Device)
  • MDS(Metadata Server)
object based storage2
Object-based Storage
  • Object
  • OSD(Object-based Storage Device)
  • MDS(Metadata Server)
object
Object
  • An object is a logical unit of storage
      • ID (Identification)
      • Application data
      • Metadata which includes block allocation and length
      • Attributes that is accessible by users
  • Objects have file-like methods
      • open, close, read, write
object1
Object
  • The root object -- The OSD itself
  • User object -- Created by SCSI commands from the application or client
  • Collection object -- A group of user objects, such as all .mp3
  • Partition object -- Containers that share common security and space managementcharacteristics
object2
Object

P4

P3

P2

OSD

P1

Root Object

(one per device)

Partition Objects

U1

User Data

Collection Objects

Metadata

Attributes

  • User Objects(for user data)

Object ID

object based storage3
Object-based Storage
  • Object
  • OSD(Object-based Storage Device)
  • MDS(Metadata Server)
slide18

Object Storage Devices

CPU

Applications

System Call Interface

File System

User Component

File System

Storage Component

Block I/O Manager

Storage Device

(a)

Traditional model

(b)

Object storage

model

  • Two changes :
  • Object-based storage offloads the storage component to the storage device
  • The device interface changes from blocks to objects
  • CPU
  • Applications
  • System Call Interface
  • File System
  • User Component
  • Object Interface

Interface

File System

Storage Component

Block I/O Manager

Storage Device

object storage devices1
Object Storage Devices

Expect wide variety of Object Storage Devices

Disk array subsystem

2 SATA disks – 240/500 GB

Highly integrated, single disk

4 Gbps per

shelf to cluster

Orchestrates system activity

Balances objects across OSDs

Stores up to 5 TBs per shelf

object based storage4
Object-based Storage
  • Object
  • OSD(Object-based Storage Device)
  • MDS(Metadata Server)
slide22

Client

Storage component

Metadata Servers (MDS)

Object Storage Devices (OSDs)

Object-based Storage Device (OSD)

Dataflow of Metadata

Applications

Client

File System

Metadata Manager

outline3
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
t10 osd commands face

Interface

OSD Client

OSD Interface

T10 OSD Commands(face)

CREATE/REMOVE

GET/SET ATTR

READ/WRITE

Opaque

attributes

(stored only)

Shared

attributes

(stored &

processed)

Attribute pages

OSD Target

10001110101

10000001110

11001110111

10001111000

..…

User Data

Object

osd commands interface
OSD Commands(Interface)

• Security

– Authorization

– Integrity

– SET KEY

– SET MASTER KEY

• Groups

– CREATE COLLECTION

– REMOVE COLLECTION

– LIST COLLECTION

• Management

– CREATE PARTITION

– REMOVE PARTITION

– FLUSH PARTITION

– PERFORM SCSI COMMAND

– PERFORM TASK MGMT

• Basic Protocol

– READ

– WRITE

– CREATE

– REMOVE

– GET ATTR

– SET ATTR

• Specialized

– APPEND

– CREATE & WRITE

– FLUSH

– LIST

very basic

shared secrets

space mgmt

attributes

• opaque

• internal

• shared

storage technology today
Storage Technology Today
  • Direct attached storage (DAS)
  • Fabric Attached Storage (FAS)

-Network Attached Storage (NAS)

-Storage Area Networks (SAN)

direct attached storage
Direct Attached Storage

LAN

Windows

UNIX

Windows

A traditional Direct Attached Storage model

RAID

BACKUP SERVER

RAID

RAID

RAID

TAPE

fabric attached storage
Fabric Attached Storage

C/S

Fabric Attached Storage

SERVER

DATA

CLIENTS

network attached storage
Network Attached Storage

This figure illustrates NAS being used to share files among a number

of clients.

Clients

File I/O

IP network

Storage area network

Block I/O

The files themselves may be stored on a fast SAN

File server

Block storage

storage area networks
Storage Area Networks

Clients

This figure illustrates a SAN file system

The files themselves are stored on a fast storage Area to which the clients are also attached.

Servers

Metadata

Storage area network

Data

Management

Block-based storage devices

security
Security

CLIENTS

METADATA SERVERS

Attribute

Capability

NETWORK

Attribute

Capability

Data

Management

OBJECT-BASED STORAGE DEVICE

outline4
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
status
Status
  • Industrial
    • Lustre
    • Panasas
  • Academic
status1
Status
  • Industrial
    • Lustre
    • Panasas
  • Academic
lustre
Lustre
  • First open sourced system with object storage
  • High-performance parallel file system
  • Consist of clients, MDS and OST(Object Storage Targets)
lustre1
Lustre

Metadata

Metadata

METADATA SERVER

CLUSTER

Data

Management

NETWORK

CLIENTS

Management

Data

OBJECT STORAGE TARGETS(OST)

panasas
Panasas
  • High-performance file system
  • Consist of OSD, Panasas File System, MDS
panasas1
Panasas

KeyObjectStorageFeatures

Intelligent space management in storage layer

ßMedia geometry aware placement

ßData aware prefetching, caching & recovery

Encapsulation of data and attributes

ßNative object interface, good programming model

ßStorage interpreted attributes for per file properties

KeyObjectStorageAdvantages

ßRobust, shared access by many clients

ßScalable performance via an offloaded data path

ßStrong fine-grained end-to-end security

panasas2
Panasas
  • Clients are from Energy, Government, Finance, Manufacturing and Higher Education
status2
Status
  • Industrial
    • Lustre
    • Panasas
  • Academic
a design of metadata server cluster in large distributed object based storage
A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Motivation:

  • Metadata server cluster maybe the bottleneck
  • Frequent metadata access and movement
  • Terrible load balance management
a design of metadata server cluster in large distributed object based storage1
A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Application Server Cluster

VoD

Server

Web

Server

E-mail

Server

File

Server

Database

Server

Metadata

Data

Storage Network

(Fibre Channel)

MDS

Cluster

Security

Object-based Storage Device Cluster

Object-based Storage System Architecture

a design of metadata server cluster in large distributed object based storage2
A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Application Servers

Application

Hashing Partition:

File Hashing Manager

Mapping Manager

Metadata Server Cluster

  • A total solution for
    • File hashing
    • Metadata partitioning
    • Metadata storage

Hashing Partition

Logical Partition Manager

Metadata Server Backend

Common Storage Space

Figure 3. Hashing Partition

a design of metadata server cluster in large distributed object based storage3
A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Pathname: /Dir1/Dir2/filename

1

4

①.Filename hashing

②.Selecting MDS through Mapping Manager

③ .Accessing metadata by pathname hashing result

④.Returning metadata to application server

Pathname Hashing

Result (i)

Pathname

Mapping Manager

Metadata

&

etc

2

Pathname Hashing

Result (i+1)

Pathname

Metadata Server Cluster

Metadata

&

etc

3

Logical Partitions

Figure 4. Metadata Access Pattern

a design of metadata server cluster in large distributed object based storage4
A Design of Metadata Server Cluster In Large Distributed Object-based Storage

Hashing Partition

Mapping Manager

2

1

3

Metadata Server Cluster

Logical Partitions

4

Common Storage Space

Figure 5. MDS cluster failover procedure

outline5
Outline
  • Why do we need object based storage?
  • What is object based storage?
  • How to take advantage of it?
  • What\'s the status of object based storage?
  • What can we do about it?
the end
The End

Thank you

ad