1 / 45

Contributed: Ronen Kat, Doron Chen, George Goldberg, Dmitry Sotnikov

Storage Advanced Management Controller for Energy Efficiency ( SAMCEE – a Part of the GAMES EU Project). Ealan Henis November 5, 2012. Contributed: Ronen Kat, Doron Chen, George Goldberg, Dmitry Sotnikov. Agenda. Background and Motivation Formal Problem Statement Technical Challenges

terris
Download Presentation

Contributed: Ronen Kat, Doron Chen, George Goldberg, Dmitry Sotnikov

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Storage Advanced Management Controller for Energy Efficiency (SAMCEE – a Part of the GAMES EU Project) Ealan Henis November 5, 2012 Contributed: Ronen Kat, Doron Chen, George Goldberg, Dmitry Sotnikov

  2. Agenda • Background and Motivation • Formal Problem Statement • Technical Challenges • Technical Solution Details • Experimental data • Results and Publications • Conclusions

  3. Background and Motivation • Energy efficiency becomes important, not only performance, hence the EU funded project • Partners: ENG, POLIMI, TUC, HLRS, Christmann, ENERGOECO, ENELSi • Need energy aware management of storage • Scope is the datacenter • Most savings in storage may be attained by shutting down storage components • However, the MAID approach is not acceptable • Performance (delays caused by spinning up devices) • Data safety (data loss when trying to spin up) • Need a different approach

  4. GAMES DASHBOARD • Editing • - initialization • Migrationplanexecution • SemiautomaticScheduling • GPI mgmt Data Mining Energy Practice Knowledge Base • Raw data • Patterns (e.g., file usage, file dependencies) • Requirements modifications, e.g., thresholds, data dependencies • BP annotation Energy Efficiency Assessment tool ESMI intf • Provenance GAMES-enabledApplicationmetadata • GPIs, KPIs • System Usage info • BP annotation • GPI evaluation • Critical components • Enabled actions • Notifications • Controllers rules • FilesForAppication • Performed actions • Results • Migrationplan Violations AggregateddB Migration Controller Acoustic Mode Controller Server data monitoring IT Infrastructure Data Storage Server storage performance monitoring

  5. Formal Problem Statement • Goal • Reduce energy consumption in the storage system • Means • Energy aware storage management • Via control of data placement and disk (acoustic) modes • Input data for control obtained from • Dedicated storage monitoring mechanisms • GAMES ESMI and databases • Application level annotations, application-to-file mapping

  6. Technical Challenges • How do we save energy yet observe performance • What are the guiding principles for controller design. E.g., • Client/server architecture • Separate control of data placement and disk acoustic modes • Spin down disks, but not ones that are used by any application • What are the control policies. E.g., • Ranking based data placement • Usage centric efficiency metrics • File splitting into smaller chunks for finer granularity

  7. Technical Solution Details • Device ranking based on usage centric metrics • Space/power, IOPS/power, MBPS/Power • Similar to potential values in form, but actual usage is the focus • Placement policy based on ranking (select high ranks) • Files are split into chunks, each chunk is placed separately • Round robin chunk placement (top ranks) for performance • Storage tiers used implicitly (via ranking) • Separate control of data placement and disk acoustic modes • Separate spin up/down controller, of unused disks • keep spares ready for intact performance • NFS NAS server with standard NFS client access • Online new chunk allocation and modes • Offline chunk migration (usage statistics available) • Fuzzy control for modes, rules are based on previous study

  8. Architecture – Client Side View NAS Server Client Host NFS Server Application1 Application2 annotate file1 File Mapper annotate file2 Client Host File System annotate file1 annotate file2 Migration and Disk Mode Controller Data Mover Backend Storage Access file1 NFS Client Access file2 See server details in next slide

  9. SAMCEE NAS - Server Side View File access Annotations NFS Server File Mapper/splitter File1 split into file1a, file1b Annotations Placement Controller Current map file1a Tier1 Disk3 file1b Tier1 Disk5 file1a Tier1 Disk3 file1b Tier3 Disk2 Desired map Data Mover Execute migration plan Monitored storage Backend Storage Set mode Disk mode Controller Tier1 SSD Tier2 SAS Tier3 SATA

  10. List of SAMCEE Components • Basic level (5 modules) • Linux kernel extension • Three Service agents: performance and power low level data collection and store into mysql DB • Fuse user level file splitter and new chunks placement • Nagios storage system reporting • Mid Level (5 modules) • Application to file mapping, annotations and EPKB access • Device power modeling • Application power modeling and reporting • Events handling: metrics updates and migration request • Spin up/down of device scripts (system dependent) • High level (4 modules) • Device ranking based on usage centric efficiency metrics • Spin up/down controller and logic of spare disks • Disk acoustic mode control logic (fuzzy) and actuation • Migration planning and execution

  11. Integration with GAMES Components SAMCEE receiving (via PTIEvent) Change usage centric metrics weights Enable/disable disk acoustic mode control Migration requests Application annotations, app-to-file mapping Load configuration file from EPKB SAMCEE sending System and application power via NAGIOS NRPE plug-in Critical conditions PTIEvent

  12. Potential Savings Estimates • Energy savings depend on the performance/runtime effects • Performance hit will be minor via NFS • Savings estimated at 50% of power difference between all disks down and all disks up • IBM testbed • 120 vs. 150  10% w/o SAMCEE server • 360 vs. 390  4% with SAMCEE server • HLRS testbed • 168 vs. 226  13% with SAMCEE server • ENG testbed • 350 vs. 410  7% w/o SAMCEE controller • 600 vs. 660  4% with SAMCEE controller • No disk mode control available • Spinning off of 3 disk pairs out of 6 also limits energy savings

  13. Storage Reference Platform • FUSE was chosen for simplicity of implementation • In a production system, the file splitter implementation should be a kernel implementation • GAMES performance and energy benchmarks Should use NFS with FUSE storage and compare: • SAMCEE enabled vs. SAMCEE disabled • We are testing SAMCEE’s control not FUSE efficiency

  14. Disk Acoustic Mode Control • Fuzzy Inference System (FIS) interpolates well between extreme cases with known solutions • Inputs to the disk mode controller are • Portion of disk sequential accesses, disk queue length, IO read/write response times, disk average capacity used, disk average usage - IOPS and Throughput • Extreme cases are based on lessons from previous investigation of disks acoustic modes • Examples of mode selection policy • For sequential IOs prefer Normal Mode, else (random IOs) • Small or Medium IOPS and RT prefer Quiet Mode • High IOPS and medium or high RT prefer Normal mode • Modes control in SAMCEE is repeatedly activated every 10 seconds

  15. Easy Changes to FIS via XML init File • <FuzzyRules> • <Rule> • <RuleName>zeroRule</RuleName> • <SimpleAntecedent> • <InputName> DiskSequentialAccessRatio </InputName> • <LinguisticVar> Non-sequential </LinguisticVar> • </SimpleAntecedent> • <SimpleAntecedent> • <InputName> DiskIOPSRate </InputName> • <LinguisticVar> Small </LinguisticVar> • </SimpleAntecedent> • <SimpleAntecedent> • <InputName> DiskResponseTime </InputName> • <LinguisticVar> Low </LinguisticVar> • </SimpleAntecedent> • <SimpleAntecedent> • <InputName> DiskInMode </InputName> • <LinguisticVar> NormalInMode </LinguisticVar> • </SimpleAntecedent> • <OutputLinguisticVar> DiskQuietMode </OutputLinguisticVar> • </Rule>

  16. Chunk Placement Control • Based on storage device ranking • Ranks calculated based on usage centric energy efficiency metrics • Need to consider actual (usage) energy efficiency for the standard metrics capacity, I/O and TP per Watt energy efficiency metrics. • Usage metrics directly linked to user applications • Derived on the basis of dynamically collected storage usage and power statistics • Device ranks are periodically recalculated • Using a weighted sum of the 3 metrics • Storage tiers are used implicitly through ranks • Data consolidation automatically obtained by high weight for capacity metric • Device performance is treated as a constraint • Round robin size for top ranking -> performance

  17. Demo and Experimental data • 10 minutes demo of basic SAMCEE operations • Disk acoustic mode control • Automatic Spin up and down of unused / used disk • Followed by experimental data from 3 testbeds

  18. IBM Testbed Disk Modes Data For 5000 IOs, 16 threads, requested IO rate above 150 IOPS For 5000 IOs, 16 threads, requested IO rate of 80 IOPS SAMCEE automatically selects the optimal acoustic mode Heuristic algorithm and fuzzy inference system Preliminary tests use stable unvarying mini-benchmarks Best mode was selected for the entire run For improving test accuracy same workload on all disks

  19. IBM Testbed Spin Down Data Synthetically generated micro-benchmarks run of the HRL testbed Uniform fixed work runs (preset total amount of IOs, concurrency and IO rate) Energy consumption data (1 JBOD, 8 SATA disks) Measured JBOD Power values (in Watts +- 2) for Measured JBOD Power values (in Watts +- 2) at idle with Each idle disk contributes approximately 10 W Additional 2.5/5 W (for 150 random IOPS load in Quiet/Normal acoustic modes) Depending on the characteristics of the workload, energy may be saved by selecting an appropriate Q/N mode Depends on the power saved vs. the runtime prolongation ratio

  20. Load per Disk is Low for RR=8 Locally copy 200GB of 10MB files, 20 concurrent threads Average power=157 W Runtime 120 min Energy=157 * 120 = 18840 Jouls*60

  21. Load per Disk is High RR=1 Locally copy 200GB of 10MB files, 20 concurrent threads Average power=130 Runtime = 170 min Energy=130*170=22100 Jouls*60 Note the spin-up/down effects Note the prolonged runtime

  22. Load with RR=3 Allows Optimization Locally copy 200GB of 10MB files, 20 concurrent threads Average power=158 Runtime = 100 min (shorter than for RR=8) Energy=158*100=15800 Jouls*60 (lower than RR=8)

  23. Load with RR=3 Allows Optimization Locally copy 200GB of 10MB files 20 concurrent threads Average power=137 (-13%, -5% with server 240W) Runtime = 100 min (similar runtime as w/o SAMCEE) Energy=137*100=13770 Jouls*60 (-13%, -5%)

  24. IBM Testbed Migration Results

  25. HLRS Testbed Results • HPC Simulation (1), storage power only

  26. HLRS Testbed Results • HPC Simulation (2), storage power only

  27. HLRS Testbed Results • HPC eBusiness GLC (3), storage power only The global server controller saves a lot of energy (e.g., 20-30%) by turning off servers. A strong performance penalty is introduced by GLC when the Virtual Machines are booted

  28. Storage Power Reference Run Devices not turned off Power consumed according to load

  29. Storage Power with SAMCEE only Some devices turned off, no performance hit Some 5.4% in energy savings

  30. Storage Power with Controllers Some devices turned off, some performance hit No (-0.5%) energy savings

  31. HLRS Testbed Results • HPC eBusiness GLC (4), storage power and app E The global server controller saves a lot of energy (e.g., 30-40%) by turning off servers. A strong performance penalty is introduced by GLC when the it boots new Virtual Machines

  32. HLRS Testbed Migration Results

  33. ENG Benchmark reference Run Average power=604.8 Runtime = 109 min Energy=3966 KJouls

  34. ENG Benchmark with SAMCEE Average power=573.8 Runtime = 105 min Energy=3609 KJouls (8.7% savings)

  35. ENG Testbed Migration Results

  36. Results • Most savings from spinning down unused disks • Most important metric is the capacity usage efficiency • For the dynamic scenario • Potential saving of 0-13% depending on data access pattern • For the static scenario (data migration) • Most application data after run is static • Re-placing the data chunk has potential savings of 0-25% depending on initial over-provisioning of space • Tiers and data consolidation with improvements • Augmented with energy considerations • Done at the data center level • Trade off (weighted sum) energy and performance via usage energy efficiency metrics • Interaction with other controllers (GAMES and/or HSM) is important and merits further investigation • Access parallelism (rr_size) is (again) important

  37. Publications • "Usage Centric Green Performance Indicators", GreenMetrics 2011 workshop, ACM SIGMETRICS conference, June 7, 2011 San Jose, USA. • To be adopted as standard by Green Grid consortium • "ADSC: Application-Driven Storage Control for Energy Efficiency", the 1st International Conference on ICT as Key Technology for the Fight against Global Warming - ICT-GLOW 2011, August 29 - September 2, 2011, Toulouse, France • “Metrics for energy efficient storage”, GAMES whitepaper available at http://www.green-datacenters.eu/ under Public docs, GAMES, StorageMetrics.IBM.WP6.V1.1.pdf, 2011. • "Setting energy-efficiency goals in data centres: the GAMES approach", Proc. E3DC workshop, within E-Energy 2012, Energy Efficient Datacentres, The 1st International Workshop on Energy Efficient Datacentres, May 8-11, Madrid, Spain.

  38. Conclusions • Novel energy aware management of storage • Major design decisions • Separate control of data placement and disk modes • File level data granularity with file splitting • Spin down of only unused disks • Ranking based data placement: usage centric efficiency metrics • Most savings obtained from spinning down disks • Data consolidation (data centers are over-provisioned) • Keep spare disks for performance • Tests show 0-13% and 0-25% for dynamic/static • No known comparable solutions in the market today • Cross controller interactions need further research

  39. Backup Slides • Backups follow

  40. HLRS Experimental System Storage Tiers • Tier1 storage • SSD (low capacity, high performance, low energy, very expensive) • 2x Intel SSD, 80GB • Tier2 • HDD (low capacity, high performance, high energy, high cost) SAS disks • 8x SAS, 147GB, 15k rpm • Tier3 • low tier disk (high capacity, low performance, low energy, low cost) SATA disks • 4x SATA 1.5TB, 7200rpm

  41. HLRSStorage Server Architecture Energy sensor x86 server Scientific Linux (SL) 4x2GB RAM Intel Xeon DP L5630 SATA NFS SSD 2x Intel SSD, 80GB SAS SAS 8x SAS, 147GB, 15k rpm SAS 1Gbit ETH SATA SATA SATA 4x SATA 1.5TB, 7200rpm SATA

  42. ENGStorage Server Architecture Energy sensor Energy sensor x86 server Scientific Linux (SL) 16GB RAM Intel Xeon E5620 NFS SAS RAID of SAS 450GB, 15k rpm SAS SATA 1Gbit ETH 8Gbit FC SATA RAID of SATA 1TB, 7200rpm SATA SATA

  43. Development Storage Server Architecture (IBM) Energy sensor Energy sensor x86 server Scientific Linux (SL) NFS SAS 6Gbit SAS 8x SAS 300GB, 15k rpm SAS 1Gbit ETH SATA SATA 6Gbit SAS 8x SATA 2TB, 7200 rpm SATA SATA Xyratex JBODs

  44. Simulation Results Disk mode After migration GAMES Palermo 20-1-2010 44

  45. Data Center Storage Efficiency Metrics • IBM Haifa is actively involved in the Green Grid DCsE Task Force, driving the introduction of Data Center Storage Efficiency (DCsE) metrics • The DCsE Task Force has acknowledged the storage metrics developed as part of GAMES and builtupon these metrics DRAFT

More Related