Download
microterabyte n.
Skip this Video
Loading SlideShow in 5 Seconds..
MicroTerabyte PowerPoint Presentation
Download Presentation
MicroTerabyte

MicroTerabyte

203 Views Download Presentation
Download Presentation

MicroTerabyte

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. MicroTerabyte Leveraging InfiniBand to Build a Powerful, Scalable Oracle Database and Application Platform Brian Dougherty Chief Architect, CMA

  2. Background • Exploding data volumes are presenting new challenges to small and medium sized organizations. • These organizations need a new generation of technology that delivers powerful analytical capability with reduced cost and complexity. • CMA, in partnership with Dell, QLogic, Oracle, and EMC, has developed a sophisticated solution to address this growing need.

  3. MicroTerabyte PlatformWhat is It ? • A pre-configured, integrated, fault tolerant, high performance, commodity hardware and software based runtime environment • Leverages the power of the Oracle database running on commodity Linux servers, and a unified InfiniBand fabric over QLogic multi-protocol fabric directors • Provides a scalable, reliable, lower cost platform for Business Intelligence, Custom Software, or Commercial Software

  4. MicroTerabyte PlatformFeatures • High Performance • Scalable • Fault Tolerant • Commodity Hardware • Lower Cost • Smaller Footprint • Reduced Power Consumption

  5. MicroTerabyte PlatformAttributes • Red Hat Linux Operating System • Commodity Hardware (Dell / EMC CX / QLogic) • 42U Rack Footprint with Fault Tolerant 42U rack • Clustered Oracle Database • Clustered Storage Provisioning Layer • Unified Storage and Interconnect Fabric

  6. MicroTerabyte Single Rack Configuration • Sample Hardware Components (per rack) • (2) Dell 1950 Database Servers • 32GB RAM Each • 16 Processor Cores • (1) Dell 1950 ETL Server • 16GB RAM • 8 Processor Cores • (1) Dell 1950 Business Intelligence Server • 16GB RAM • 8 Processor Cores • (1) QLogic 9020 InfiniBand Fabric Director • (2) FVIC Modules • (2) EMC CX3-40 Storage Arrays • 2TB – 5TB Storage • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console Physical Data Guard

  7. MicroTerabyte Mid-Range Configuration • Hardware components (two racks) • (4) Dell 1950 Database Servers • 32GB RAM Each • 32 Processor Cores • (1) Dell 1950 ETL Server • 32GB RAM • 8 Processor Cores • (1) Dell 1950 Business Intelligence Server • 32GB RAM • 8 Processor Cores • (1) QLogic 9040 InfiniBand Fabric Director • (2) FVIC Modules • (2) EMC CX3-40 Storage Arrays • 8TB – 15TB Storage • (2) EMC SATA 1TB Drives (for backup) • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console

  8. MicroTerabyte Large Scale Configuration • Hardware components (three racks) • (8) Dell 1950 Database Servers • 32GB RAM Each • 64 Processor Cores • (2) Dell 1950 ETL Server • 32GB RAM Each • 8 Processor Cores • (2) Dell 1950 Business Intelligence Server • 32GB RAM Each • 8 Processor Cores • (1) QLogic 9040 InfiniBand Fabric Director • (4) FVIC Modules • (4) EMC CX3-40 Storage Arrays • 24TB – 32TB Storage • (2) EMC SATA 1TB Drives (for backup) • (1) Dell PowerConnect 48 Port GigE Switch • (1) Belkin KVM • (1) Dell Console

  9. Compute Nodes Unified Fabric Layer Storage Nodes MicroTerabyte ArchitectureMid-Level Diagram Dell 1950 PowerEdge Servers / RHEL v5 QLogic Multi-Protocol Fabric Director EMC CX3 / CX4 Flare OS / Navisphere

  10. ERP Application Package ISV Third Party Software CMA BI Suite Business Intelligence MicroTerabyte ArchitectureDetailed Diagram Applications Oracle Grid Control 11g MT Clustered Database / Storage Provisioning Oracle EE 11g w/ RAC and Partitioning Oracle ASM 11g MicroTerabyte Platform QLogic OFED InfiniBand Drivers Oracle Clusterware 11g MT O/S Binding Red Hat Linux O/S Oracle ASMlib Public Networking Dell PowerConnect 6248 Server Compute Dell 1950/2950 + QLogic 7104 HCA MT Core QLogic 9020/9040 Multi-Protocol Director Unified Fabric Storage Infrastructure EMC CX3/CX4

  11. MicroTerabyte PlatformOracle Software

  12. MicroTerabyte Compute NodesDell 1950/2950 • MicroTerabyte solution consists of 2/4/8 RAC Database nodes • 2 ETL and BI Nodes • Each node is a Dell 1950/2950 consisting of: • Processor: One/Two quad-core Intel Xeon X5355 @ 2.66GHz • Memory: 16-64 GB • Hard drives: 218GB-3TB Internal Storage • RAID Controller: PERC 5/i • 1 DDR InfiniBand HCA • Network interface cards: Dual gigabit NICs (100baseTx-FD) • Power supply: 670W, optional hot-plug redundant power (1+1) • Operating system: RedHat Enterprise Linux v5

  13. Front-End Host Connectivity • Two storage processors per CX4 • Each processor has: • Four 4Gb Fibre Channel optical ports • FCP SCSI-3 protocol • Back-End Disk Connectivity • Each processor has 4Gb Fibre Channel arbitrated loops. • Multiple RAID groups may be distributed across redundant loops • Supports a maximum of 480 disk drives • System Memory • Each processor has 8GB of Memory • Power Consumption (Processor Chassis) • 355 VA (290W max) MicroTerabyte Storage NodesEMC CX4 Model 480 • Power Consumption (Disk Expansion Chassis) • 440 VA (425W max)

  14. MicroTerabyte Unified Fabric LayerWhat is it? • QLogic 9020 & 9040 Multi-Protocol Fabric Directors • 9020 with two (2) FVIC IB-FC Virtual I/O Controllers • 9040 with up to four (4) FVIC IB-FC Virtual I/O Controllers • each FVIC provides 10 DDR (20Gb) IB ports & 8 4Gb FC ports • supports up to 128 Virtual HBA ports per module • automatic sensing 1/2/4 Gb/s • load balancing • automatic port and module fail-over • LUN mapping and masking features • QLogic 7104 Host Channel Adapters • Dual Port, DDR • IPoIB, RDS, SRP

  15. MicroTerabyte Unified Fabric LayerGeneral Benefits • Managing one fabric • Reduced footprint • Compact implementation • Fewer host components needed to support I/O and interconnect • Increased bandwidth and reduced latency • Reduced host resources (1 HCA vs. several HBAs) • Path failover through SRP protocol • Well positioned to take advantage of advances in InfiniBand technology

  16. MicroTerabyte Unified Fabric LayerOracle RAC Benefits • Scalable platform to support Oracle RAC • More predictable response times • Capability to drive more Oracle I/O through fewer compute nodes • Ability to exploit storage capability at a lower cost • Reduced Oracle messaging latency via RDS/IB

  17. MicroTerabyte ORION Benchmarks CONFIGURATION A mid-size MicroTerabyte configuration, including: Servers: Four (4) Dell 1950 Intel quad core servers. Each 1950 includes 8 cores, 16GB memory and 1 dual channel HCA. The server is running Red Hat Linux 5 update 1 (2.6.18-53 kernel). Storage: Two (2) EMC CX3-40 storage arrays. Each storage array includes 8 4Gbps front-end fiber channel connections for a total of 16 4Gbps FE adapters -- approx 7.5TB usable storage configured in 4+1 RAID sets. Unified Interconnect and Storage IB fabric: One (1) QLogic 9040 Multi-Protocol Director with 2 FVIC modules HARDWARE COST Approximate total cost (market): $500,000 TEST METHOD AND RESULTS Testing simulator: Oracle ORION -- Oracle I/O Numbers Calibration Tool for Linux >> Random I/O Test Results: Test Type: random I/O I/O size: 8K Rate observation 1: 9,600 sustained IOPS @ 1.87 ms per I/O avg node latency Rate observation 2: 30,566 sustained IOPS @ 2.96 ms per I/O avg node latency Rate observation 3: 42,615 sustained IOPS @ 2.96 ms per I/O avg node latency >> Sequential I/O Test Results: Test Type: sequential I/O I/O size: 1MB Rate observation 1: 677MB/sec sustained I/O seq. throughput @ 26.48 ms per 1MB I/O avg node latency Rate observation 2: 2.098GB/sec sustained I/O seq. throughput @ 91.48 ms per 1MB I/O avg node latency

  18. MicroTerabyte Oracle Benchmarks Hardware Servers: (4) Dell 1950 Dual Socket Quad Core 16 GB RAM per server Single dual port IB HCA Unified I/O Fabric: QLogic 9040 (2) FVIC Modules Storage: (2) EMC CX-3 Model 40 (16) Fibre Channel ports Software O/S: Red Hat Linux v 5.1 Kernel 2.6.18-51 UDEV Unified I/O Fabric: QLogic IB drivers v 4.2.0.0.39 IB/RDS IB/SRP IPoIB Database: Oracle RAC 10gR2 Oracle Clusterware 10gR2 Oracle ASM Oracle RMAN Oracle DataGuard Storage: Flare O/S EMC Navisphere Backup: EMC Networker

  19. MicroTerabyte Benchmark Comparisons

  20. Summary As demonstrated in CMA’s MicroTerabyte platform, Infiniband can provide an extremely capable transport mechanism for unifying interconnect and i/o traffic Increases bandwidth and reduces latency Reduces Oracle messaging latency via RDS/IB Provides more predictable response times Reduces host resource requirements (i/o processing workload off-loaded to HCA card) Consistent with Oracle’s strategic technology direction

  21. For More Information Brian Dougherty Chief Architect, CMA bdougherty@cma.com