A Label-Based File System

1. A Label-Based File System Shang Rong Tsai Department of Information Management Chang-Jung University

2. Outlines • Background and Introduction • Network Storage/SAN/NAS • Object-Based Storage • A Label-Based File System • Epilogue

3. The Epoch of Data Explosion • Electronic data are continuously growing • Data and storage managements are becoming more and more important • Continuous service and data availability • Data backup and restore • Storage and data sharing • Efficiency in data and storage management • Expecting using less man power • Flexibility in system configuration • Efficiently expanding storage capacity

4. Personal Data • Capacity of disks is growing • Huge number of files on a single disk • Documents got from Internet • Photos by Digital Cameras • Sometimes we have problems to find a file • More ways to locate files are needed • Giving labels to files for locating

5. Network Storage Systems • Storages are moving to networks for sharing and efficient management • Such demands push the emerge of SAN (Storage Area Network) • NAS (Network Attached Storage) has been used for longer time • Object-Based Storage System • Both SAN and NAS have their own merits and drawbacks. • SAN supports data access at block level, not good for applications in data sharing • NAS imposes bottleneck on servers and has bad scalability • Object-Based approach potentially eliminates their drawbacks • Network storage technologies offer a new platform for networking and storage people to play a new game.

6. Conceptual Model of SAN client client client client Clients Network to connect clients and hosts Hosts Linux Windows SunOS SAN to connect hosts and storages Storages tape Disk Disk Disk

7. What is a SAN • SAN is a high speed network (traditionally Fiber Channel) to connect storages to servers (hosts). The network is basically used as a replacement of storage bus in traditional shared bus storage systems, thus enhances the possible sharing scale and possible fail over, etc. • Access to SAN storage is at block level

8. Why SANs (1/2) • From the communication aspect, the SAN can bypass the possible communication bottleneck. It enables communication between • Server-to-server • Server-to-storage (typical model) • Storage-to-storage (e.g. for backup without servers’ intervention)

9. Why SANs (2/2) • Improvements to application availability: Storage is accessible through multiple data paths for better reliability, availability, and serviceability. • Higher application performance: Storage processing is off-loaded from servers • Virtualized storage: Storages on SAN can be flexibly configured as a logical volume of any size, thus easily sharing storage space at configuration • Data backup to remote sites: enabled for disaster protection • Simplified centralized management: Single image of storage systems simplifies management.

10. Storage Virtualization (1/2) • SNIA defines storage virtualization as: “The act of integrating one or more (back end) services or functions with additional (front end) functionality for the purpose of providing useful abstractions. Typically, virtualization hides some of the back-end complexity, or adds or integrates new functionality with existing back end services. Examples of virtualization are the aggregation of multiple instances of a service into one virtualized service, or to add security to an otherwise insecure service. Virtualization can be nested or applied to multiple layers of a system.”

11. Storage Virtualization (2/2) • To be more practical, storage virtualization is the aggregation of physical storage from multiple network storage devices into a single logical storage device that is managed and used by a central host. • Logical Volume Manager (LVM) is basically the concept of storage virtualization which has been used for many years. • By storage virtualization we can easily and flexibly configure the size of a logical disk. • For some applications or sites, the amount of storage required grows at unprecedented rates. Try to think about if a disk partition is becoming full.

12. SAN connectivity • Traditionally, SAN used Fibre Channel technology to implement the storage networks • Fibre Channel SANs support high bandwidth storage traffic at 200 MB/s and enhancements to 10 Gb/s in the near future. This will be mostly used for inter switch links (ISL) between switches. • iSCSI SAN (SCSI over TCP/IP) is a relatively new approach for storage networks. • Fibre Channel over IP (FCIP) • FCIP can be used as WAN bridging transport for both FC and iSCSI based campus SANs

13. iSCSI • iSCSI stands for Internet Small Computer System Interconnect • iSCSI is a protocol for encapsulating SCSI commands on a TCP/IP network • The iSCSI protocol enables universal access to storage devices and Storage Area Networks over TCP/IP network

14. SCSI Protocol Layers • SCSI command layer • Generic commands (for all devices) • Device specific commands • Transport Layer • Physical Layer (Connectivity Layer)

15. SCSI Protocol Layer SCSI application (e.g. File systems) SCSI commands for other types of devices SCSI block commands SCSI stream commands SCSI commands layer SCSI Generic (Primary) commands SCSI transport layer iSCSI over TCP/IP Parallel SCSI SCSI over Fiber Channel TCP IP Fiber Channel Physical layer Layer 2 (Ethernet) Parallel SCSI Bus

16. iSCSI PDU (from SNIA)

17. Overview of iSCSI • iSCSI provides initiators and targets with unique names and a discovery method. • The iSCSI protocol establishes communication sessions between initiators and targets, and provides methods for them to authenticate one another. • An iSCSI session may contain one or more TCP connections and provides recovery in case of connection failures. • SCSI CDBs (Command Descriptor Block) are passed from the SCSI command layer to the iSCSI transport layer. The iSCSI transport layer encapsulates the SCSI CDB into an iSCSI Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer. • iSCSI provides the SCSI command layer with a reliable transport.

18. What is NAS • Network Attached Storage (NAS) is basically a LAN attached file server that provides shared file access using a file sharing protocol such as Network File System (NFS) or Common Internet File System (CIFS) • Access to NAS is at file level.

19. Why NAS • NAS technology has been used for decades to share files, thus to save storage space and to keep data consistency. (in contrast to file copy, like FTP) • Data sharing in NAS is at file level, which matches the semantic for applications. In contrast, sharing in SAN is at block level which is very difficult (if not impossible) for applications to R/W data sharing. (Applications recognize files, not blocks)

20. What is an Object-Based Storage • Storage devices that operate at object-level • Traditional storage devices (Such as DAS, SAN) operate at block-level • Objects are typically files which match the semantic level that applications manipulate data. • In traditional systems, files are mapped to blocks before they are stored in storages.

21. Why Object-Based Storage (1/2) • Drawbacks for NAS • Most of the processing on file access are on the file server => Poor scalability • Difficult to distribute users’ files to multiple servers while preserve a single global file namespace. • We may delegate the management of the subset of the whole file namespace to a file server, however, the load distributed to each server may be very uneven.

22. Why Object-Based Storage (2/2) • Drawbacks of SAN • SANs operate at block-level, so sharing data between applications may still need upper layer software. • file read/write sharing • Record/file locking • Object-Based Storage can potentially eliminate both the drawbacks of NAS and SAN

23. The Value of Objects (from SNIA) • Better security via capabilities • Each object can have its own security domain • All I/O is authorized by the device • Easier to share data • Files and records can be stored as objects • Low-level metadata managed by device • Opportunities for intelligence • Attribute-based learning for resource allocation • Better caching, pre-fetching and staging of data • Self-configuring storage w/ continuous reorganization • Layout objects to best serve client requests

24. Two Basic Components in File Service • Directory Service • Providing the global file name space visible to users and applications • Mapping file pathname to unique file id • May need the flat file system to access directories • Flat File System • Given the file id, returning the file contents • Storing file attributes • Managing file allocation on disks

25. Typical Handling in File Access • File pathname lookup : open /a/b/c • Symbolic file name => file id (inode in Unix) • For each file path component, get the file id, get the file contents until getting file id of the target • Get the attributes (including the location of the file on disks) of the file • Ready for W/R

26. How to Restructure the file system to fit the Object-Based Storage Architecture ? • We may (as Lustre did) partition the functions into • MDS (Meta Data Server) • OSD (Object-Storage Device) • What is the total system architecture? • What are the data and management functions delegated to MDS? • What are the data and management functions delegated to OSDs? • Important observation: Typically, among the file processing in systems, 90% processing is by OSDs, 10% by MDS => To balance the loads

27. Traditional structure vs. Object-Based structure

28. Lustre • A system developed at CMU, a very early working system demonstrate the concept of object-based storage systems. • Modifying the Linux kernel • A few number of MDSs • A large number of OSDs • Majorly targeting at the applications in large scale computing. • Large number of users • File service for high performance computing • Good Scalability

29. OST 1 MDS 1 (active) MDS 2 (failover) OST 2 OST 3 OST 4 OST 5 OST 6 OST 7 Lustre Architecture (from CFS Inc.) Linux OST Servers with disk arrays QSW Net SAN Lustre Clients (1,000 Lustre Lite) Up to 10,000’s GigE 3rd party OST Appliances Lustre Object Storage Targets (OST)

30. Lustre Components and Functions (1/2) • Basic Components • Client Filesystems • Interfacing to local file system (VFS) on clients • As MDS client • As OST client • MetaData Servers • All the meta-data operations - creating new directories, files, symbolic links, or acquiring and updating inodes, are handled by the MDS. • Object Storage Target • All the file I/O related operations are directed to the OST’s.

31. Lustre Components and Functions (2/2) • The role of the client filesystem is to provide a directory tree, subdivided into filesets, which provides cluster-wide Unix file sharing semantics. • The client filesystem interacts with the meta-data servers for meta-data handling, i.e. for the acquisition and updates of inodes and directory information. • File I/O, including the allocation of blocks, striping, and security enforcement, is contained in the protocol between the client filesystem and the object storage targets. • A third protocol exists between the OST and the MDS, largely for pre-allocation and recovery purposes.

32. SNIA T10 Specification • A specification developed for Object-based Disk • The SCSI command set defined to provide efficient peer-to-peer operation of input/output logical units that manage the allocation, placement, and accessing of variable-size data-storage containers, called objects.

33. Some Operations defined in T10 • Format OSD -defining OSD structure on device • Create Object Group -defining a set in which to create objects • Create object -creating an object, returning the object ID fid • Open fid • Read (fid,starting byte,length) • Write (fid,starting byte,length) • Close fid • Get attributes of an object

34. Possible Intelligent functions • OSDs serve and manage data at object (most often representing files) level. This enables OSDs to smartly enforce management on a per-object basis • Automatic replication/backup • QoS • Caching/Prefetching • Optimal layout • Preprocessing/Postpressing (such as data compression/decompression, data filtering)

35. How to support the intelligent functions ? • Extensible file attributes would be a good way • Users can define their own file attributes. A special ‘code’ attribute can be assigned to a particular file. Setting the code attribute to a file is in effect installing the code to the OSD to execute the intelligent functions within the OSD. • How to set the code attribute in practice ? • Consideration for platform (OSD) dependence • Sharing the code on OSD • How to command the OSDs to execute the intelligent functions automatically

36. Label-Based File System(LBFS) • A project cooperated with Industrial Technology Research Institute • A file system using object-based structure • Features • Capability to give labels to files for easy locating • Flexibility in restructuring the namespace • object-based structure

37. Why LBFS ? • Number of files are growing on a single disk • Traditional directory-based file systems can not satisfy the requirements in locating files • Sometimes we forget in which directories the files reside • Content-based file search takes times • Typical file attributes are not enough • The namespace of the whole file system can not be reorganized flexibly • Can each user have his own file namespace given a single set of files ?

38. Overview of LBFS (1/2) Client MDS CategoryA CategoryA CategoryB Namespace and meta data service Network Object access service OSD

39. Overview of LBFS (2/2) • MDS providing meta data and namespace services • MDS providing a browsing structure called Category • OSD providing object storage • System abstraction • General Objects (GO) • Attributes of file objects • Labels • Collections • Categories

40. General Object and Label (1/2) • GO is the basic unit to store a file object • We plan a higher level object called (XO) XML objects for encapsulating an XML information unit • Label is basically a keyword attached to a GO by users • Searching a file by labels is much faster than by content matching

41. 01 . jpg General Object and Label (2/2) File attributes + Labels General attributes: Object Name、Object Size、Time Stamp … attributes GO Indexing attributes: Label1、Label2 … data File contents

42. GO Metadata typical file metadata GO extended metadata

43. Collection (1/3) • Collection is logically an objects container • A collection logically contains a set of symbolic links, each of which links to a file object • Each collection has an expression defined for it to specify the set of objects contained in the collection Collection Picture Music Doc i-expression : Picture Picture Doc Picture Picture

44. Collection (2/3) • When users create an object and give it a label, the object will be “included” in the corresponding collection automatically Collection A Music Creating objects and labeling User i-expression : Picture Picture Collection B Picture Music i-expression : Music

45. Collection (3/3) • In traditional file systems, a directory and 和files under it are bound in a fixed way • In an LBFS, a collection and file objects in it are bound in a flexible way, by matching the expression assigned to the collection and the labels attached to the objects • The expression , called i-expression • AND label-name • OR label-name • NOT label-name

46. Category (1/3) • An Category is logically a namespace of file objects, it is organized as a tree structure, where leaf nodes are objects and non-leaf nodes are collections Category GOs

47. Category ???(2/3) Music Category i-expression :Null 音樂 i-expression : 音樂+流行 i-expression :Null i-expression : 音樂+精選 i-expression: 音樂+西洋+古典 古典 流行 精選 i-expression: 音樂+中國+古典 中國 西洋 音樂 音樂 文件 文件 照片 音樂 照片 照片 音樂 音樂 音樂 音樂 GOs 流行 流行 風景 中國古典 人物 風景 中國古典 西洋 西洋古典 西洋 論文 投影片 精選 OSD 林志玲 阿里山 OSD 精選 精選

48. Family Mary Family George Photo Photo Photo Mary Photo Family George Photo Family George Photo Photo Family Photo Photo Category (3/3) • Multiple Categories can be organized to bind a single set of objects Category Category Category Mary Photo Mary

49. Overview of the prototype • Networking：與Client使用Metadata Service Protocol，與OSD使用iSCSI Protocol • MDS：提供MDS對外的服務 • VP Manager：負責管理各個VP，包含根據Storage Policy決定物件的配置 • OSD Manager：負責管理各OSD • Security Manager：負責T10 OSD Security Model中所提供的功能，包含credential的發放 • GO Metadata Service：負責管理GO Metadata • Metadata Storage Pool：真正儲存metadata的地方

50. Metadata Storage Pool • 測試用的雛型系統使用資料庫儲存Category、Collection、Label等相關資訊。 • Metadata Storage Pool介面明確(metadata storage API)，方便未來更換不同的儲存媒體。