1 / 24

An Introduction to Sector/Sphere

Sector & Sphere. An Introduction to Sector/Sphere. Yunhong Gu Univ. of Illinois at Chicago and VeryCloud LLC @CHUG, June 22, 2010. What is Sector/Sphere?. Sector: Distributed File System Sphere: Simplified Parallel Data Processing Framework Goal: handling big data on commodity clusters

belva
Download Presentation

An Introduction to Sector/Sphere

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sector & Sphere An Introduction to Sector/Sphere Yunhong Gu Univ. of Illinois at Chicago and VeryCloud LLC @CHUG, June 22, 2010

  2. What is Sector/Sphere? • Sector: Distributed File System • Sphere: Simplified Parallel Data Processing Framework • Goal: handling big data on commodity clusters • Open source software, BSD license, written in C++. • Started since 2006, current version 2.3 • http://sector.sf.net

  3. Motivation: Data Locality Super-computer model: Expensive, data IO bottleneck Sector/Sphere model: Inexpensive, parallel data IO, data locality

  4. Motivation: Simplified Programming Parallel/Distributed Programming with MPI, etc.: Flexible and powerful. very complicated application development Sector/Sphere model (cloud model): Clusters regarded as a single entity to the developer, simplified programming interface. Limited to certain data parallel applications.

  5. Motivation: Global-scale System Systems for single data centers: Requires additional effort to locate and move data. Sector/Sphere model: Support wide-area data collection and distribution.

  6. Sector Distributed File System • DFS designed to work on commodity hardware • racks of computers with internal hard disks and high speed network connections • File system level fault tolerance via replication • Support wide area networks • Can be used for data collection and distribution • Not POSIX-compatible yet

  7. Sector Distributed File System User account Data protection System Security Metadata Scheduling Service provider System access tools App. Programming Interfaces Security Server Masters Clients SSL SSL Data UDT Encryption optional slaves slaves Storage and Processing

  8. Security Server • User accounts, permission, IP access control lists • Use independent accounts, but connect to existing account database via a simple “driver”, e.g., Linux accounts, LDAP, etc. • Single security server, system continue to run when security server is down, but new users cannot login

  9. Master Servers • Maintain file system metadata • Metadata is a customizable module, currently there are two implementations, one in-memory and one on disk • Authenticate users, slaves, and other masters (via security server) • Maintain and manage file replication, data IO and data processing requests • Topology aware • Multiple active masters can dynamically join and leave; load balancing between masters

  10. Slave Nodes • Store Sector files • Sector file is not split into blocks • One Sector file is stored on the “native” file system (e.g., EXT, XFS, etc.) of one or more slave nodes • Process Sector data • Data is processed on the same storage node, or nearest storage node possible • Input and output are Sector files

  11. Clients • Sector file system client API • Access Sector files in applications using the C++ API • Sector system tools • File system access tools • FUSE • Mount Sector file system as a local directory • Sphere programming API • Develop parallel data processing applications to process Sector data with a set of simple API • The client communicate with slave directly for data IO, via UDT

  12. UDT: UDP-based Data Transfer • http://udt.sf.net • Open source UDP based data transfer protocol • With reliability control and congestion control • Fast, firewall friendly, easy to use • Already used in many commercial and research systems for large data transfer

  13. Application-aware File System • Files are not split into blocks • Users are responsible to use proper sized files • Directory and File Family • Sector will keep related files together during upload and replication • In-memory object

  14. Sphere: Simplified Data Processing • Data parallel applications • Data is processed at where it resides, or on the nearest possible node (locality) • Same user defined functions (UDF) are applied on all elements (records, blocks, files, or directories) • Processing output can be written to Sector files or sent back to the client • Transparent load balancing and fault tolerance

  15. Sphere: Simplified Data Processing for each file F in (SDSS datasets) for each image I in F findBrownDwarf(I, …); SphereStream sdss; sdss.init("sdss files"); SphereProcess myproc; myproc->run(sdss,"findBrownDwarf", …); findBrownDwarf(char* image, int isize, char* result, int rsize);

  16. Sphere: Data Movement • Slave -> Slave Local • Slave -> Slaves (Hash/Buckets) • Each output record is assigned an ID; all records with the same ID are sent to the same “bucket” file • Slave -> Client

  17. What does a Sphere program like? • A client application • Specify input, output, and name of UDF • Inputs and outputs are usually Sector directories or collection of files • May have multiple round of computation if necessary (iterative/combinative processing) • A UDF • A C++ function following the Sphere specification (parameters and return value) • Compiled into a dynamic library

  18. Sphere/UDF vs. MapReduce • Map = UDF • MapReduce = 2x UDF • First UDF generates bucket files and second processes the bucket files.

  19. Sphere/UDF vs. MapReduce • Sphere is more flexible and efficient • UDF can be applied directly on records, blocks, files, and even directories • Support multiple inputs/outputs with better data locality, including certain legacy applications that process files and directories • Native binary data support w/ permanent index files • Sorting is required by Reduce, but it is optional in Sphere • Output locality allows Sphere to combine multiple operations more efficiently

  20. Sphere Benchmarks • Terasort: sort 1TB data over distributed servers • Malstone: detect malware website from billions of transactions • Graph processing: analyze very large social networks at billions of vertices (BFS and enumerating cliques) • Genome pipeline: analyze genome sequences • Satellite image processing: compare satellite images of different time, for disaster relief • Sphere is about 2 – 4 times faster than Hadoop

  21. Open Cloud Testbed • 15 Racks in Baltimore (JHU), Chicago (StarLight and UIC), and San Diego (Calit2) • 10Gb/s inter-site connection on CiscoWave • 1 - 2Gb/s inter-rack connection • Two dual-core AMD CPU, 8 - 16GB RAM, 1-4TB RAID-0 disk

  22. Open Cloud Testbed

  23. Development Status • Current version 2.3, all core functions ready, still working on to improve code quality and details for certain modules. • Partly funded by NSF for NCDM/UIC • Commercial support via VeryCloud LLC • Next step: support column-based data tables (similar to BigTable) • Open source contributors are welcome

  24. More Information • Sector Website: http://sector.sourceforge.net • Email: gu@lac.uic.edu

More Related