1 / 53

Columbia University’s Advanced Concept Data Center Pilot Project

Columbia University’s Advanced Concept Data Center Pilot Project. April 30, 2010. Agenda. 11:30am Data Center Tour 12:00pm Welcome and Opening Remarks Bryan P. Berry, NYSERDA Project Review Alan Crosswell, Chief Technologist, CUIT Richard Hall, Project Manager, CUIT

devon
Download Presentation

Columbia University’s Advanced Concept Data Center Pilot Project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Columbia University’s Advanced Concept Data Center Pilot Project April 30, 2010

  2. Agenda 11:30amData Center Tour 12:00pm Welcome and Opening Remarks Bryan P. Berry, NYSERDA Project Review Alan Crosswell, Chief Technologist, CUIT Richard Hall, Project Manager, CUIT 1:00pmInstallation Summary of Monitoring and Measurement Tools Ian Katz, Manager, CUIT Data Center 1:45pmMeasurement Plan and Initial Results Raj Bose and Peter Crosta, Research Computing Services, CUIT 2:45pm Closing Comments Alan Crosswell, Chief Technologist, CUIT Victoria Hamilton, Research Initiatives Coordinator 3:00pm Meeting Adjourned 2

  3. Welcome and Opening RemarksBryan P. Berry, NYSERDA

  4. Faculty House • Recently renovated Faculty House has been awarded the prestigious Leadership in Energy and Environmental Design (LEED) Gold Certification by the United States Green Building Council. • http://www.environment.columbia.edu/blog/facultyhouseawardedleedgold • First LEED Gold Certified building on Morningside campus. • The first McKim, Mead & White building in the country to be given this designation.  • To learn about Columbia University’s other “green initatives” please visit Columbia University’s Environmental Stewardship webpage • http://www.environment.columbia.edu/initiatives

  5. Advanced Concepts Data Center Pilot Project - ReviewAlan Crosswell, Assoc VP and Chief Technologist, CUIT

  6. The opportunities Data centers consume 3% of all electricity in New York State (1.5% nationally as of 2007). That’s 4.5 billion kWh annually. Use of IT systems especially for research high performance computing (HPC) is growing. We need space for academic purposes such as wet labs, especially in our constrained urban location. Columbia’s commitment to Mayor Bloomberg’s PlaNYC 30% carbon footprint reduction by 2017. NYS Gov. Paterson’s 15x15 15% electrical demand reduction by 2015 goal. National Save Energy Now 25% energy intensity reduction in 10 yrs.

  7. Green data center best practices • Measure and validate • You can’t manage what you don’t measure. • Power and cooling infrastructure efficiency • Best Practices for Datacom Facility Energy Efficiency. ASHRAE (ISBN 978-1-933742-27-4) • IT equipment efficiency • Moore’s Law performance improvements • Energy Star/EPEAT • Server consolidation and virtualization • BIOS, OS and Application tuning

  8. Measuring infrastructure efficiency • The most common measure is Power Use Effectiveness (PUE) or its reciprocal, Data Center Infrastructure Efficiency (DcIE). [Total Datacenter Electrical Load] PUE = [Datacenter IT Equip. Electrical Load] • PUE only measures efficiency of the electrical and cooling infrastructure. • Chasing a good PUE can lead to bizarre results since heavily-loaded facilities usually use their cooling systems more efficiently.

  9. LBNL Average PUE for 12 Data Centers Power Use Effectiveness (PUE) =2.17

  10. Making the server slice bigger, the pie smaller and green. • Reduce the PUE ratio by improving electrical & mechanical efficiency. • Google claims a PUE of 1.2 • Consolidate data centers (server rooms) • Claimed more efficient when larger (prove it!) • Free up valuable space for wet labs, offices, classrooms. • Reduce the overall IT load through • Server efficiency (newer, more efficient hardware) • Server consolidation & sharing • Virtualization • Shared research clusters • Move servers to a zero-carbon data center

  11. Data center electrical best practices • 95% efficient 480V room UPS • Basement UPS room vs. wasting 40% of rack space • Flywheels or batteries? • 480V distribution to PDUs at ends of rack rows • Transformed to 208/120V at PDU • Reduces copper needed, transmission losses • 208V power to servers vs. 120V • More efficient (how much?) • Variable Frequency Drives for cooling fans and pumps • Motor power consumption increases as the cube of the speed. • Generator backup

  12. Data center mechanical best practices • Air flow – reduce mixing, increase delta-T • Hot/cold or double hot aisle separation • 24-36” under floor plenum • Plug up leaks in floor and in racks (blanking panels) • Duct CRAC returns to an overhead plenum if possible • Perform CFD modeling • Alternative cooling technique: In-row or in-rack cooling • Reduces or eliminates hot/cold air mixing • More efficient transfer of heat (how much?) • Supports much higher power density • Water-cooled servers are making a comeback

  13. Measuring IT systems efficiency • A complementary measure to PUE is the amount of useful work being performed by the IT equipment. What should the metric be? • MIPS per KwH? • kilobits per MWh (an early NSFNet node benchmark:-) • SPECpower benchmark

  14. Highly recommended reading • Surveys 261 higher ed institutions • Cites Columbia’s environmental sustainability efforts and NYSERDA’s role

  15. Our NYSERDA project • New York State Energy Research & Development Authority is a public benefit corporation funded by NYS electric utility customers. http://www.nyserda.org • Columbia competed for and was awarded an “Advanced Concepts Datacenter demonstration project”. 18 months starting April 2009 • ~ $1.2M ($447K Direct costs from NYSERDA) • Goals: • Learn about and test some industry best practices in a “real world” datacenter • Measure and verify claimed energy efficiency improvements. • Share lessons learned with our peers.

  16. External Visiting Committee • Laurie Kerr, Senior Policy Advisor for Energy and Green Buildings, NYC Mayor’s Office of Long-Term Planning and Sustainability • Vace Kundakci, Assistant Vice President for Information Technology and Chief Information Officer, City College of New York/City University of New York • Timothy Lance, President and Board Chair, New York State Education and Research Network (NYSERNet) • Marilyn McMillan, Associate Provost and Chief Information Technology Officer, New York University Role: The External Visiting Committee is charged with maintaining skepticism about our plans and our accomplishments from the perspective of external institutions. To the extent we demonstrate feasibility and potential impact, they will serve as powerful ambassadors to other institutions. 16

  17. Internal Advisory Group Wilmouth A. Elmes, Associate Vice President of Engineering/Technical Services, Manhattanville Development Project Doug McKean, Director, Capital Project Management Arthur M. Langer, School of Engineering and Applied Science, Senior Director of the Center for Technology, Innovation, and Community Engagement and Faculty & Assoc Director, Executive Masters of Science in Technology Management at the School of Continuing Education Nilda Mesa, Assistant Vice President, Environmental Stewardship Scott W. Norum, Chief Administrative Officer for Arts and Sciences and Vice President, Office of the Vice President for Arts and Sciences Leonard Peters, School of Business, Associate Dean and Chief Information Officer – Information Technology Role: The Internal Advisory Groupserves two primary functions. Not only do we anticipate they will prove a valuable sounding board for the execution of the pilot (implementation guidance/expert judgment), but we expect them to pose the hard questions about issues surrounding scale-up(feedback). 17 17

  18. Research Faculty User Group Liam Paninski, Associate Professor, Dept. of Statistics Kathryn V. Johnston, Associate Professor, Dept. of Astronomy Mary Putman, Clare Boothe Luce Associate Professor of Astronomy Greg L. Bryan, Assistant Professor, Dept. of Astronomy Role: Responsible for vetting the results of the combined HPC cluster for the Department of Astronomy and the Department of Statistics 18 18

  19. Working Group Co-Principal Investigators Alan Crosswell, Associate Vice President and Chief Technologist, CUIT Victoria Hamilton, Director of the Office for Research Initiatives (ORI) Richard Hall, Project Manager Anthony Cirillo, AVP Systems and Operations Peter Miner, Senior Director, Systems Engineering Ian Katz. Data Center Facilities Manager Melissa Metz, Director of Systems Engineering Fran Ovios, Director of Systems Engineering Stew Feuerstein, Director and Chief Architect Joseph Rini, Director of Network Engineering Megan Andersen, Research Systems Administrator Ed McArthur, CU Facilities Project Manager Raj Bose, Manager, Research Computing Services Don Lanini, Senior Systems Engineer Role: Day-to-day responsibility for planning, executing and measuring required by the proposal. The Columbia University in-house team will perform much of the work. 19 19

  20. CU Data Center Improvement Program • Begun with an assessment and recommendation performed by Bruns-Pak, Inc. in 2009 ($50K). • CUF Operations HVAC study by Horizon Engineering. • Generator overload mitigation study by Rowland Engineering ($32K). • JB&B, Gensler & Structuretone developed a master plan ($152K) based on the Bruns-Pak work which was used to develop: • DOE ARRA grant application for HVAC improvements (not awarded). • NIH ARRA grant application for electrical improvements (awarded 4/15/10 $10M Core Research Computing Facility). • Will be used for future funding opportunities • Informing portions of the NYSERDA project (chilled racks).

  21. Advanced Concepts Data Center Pilot Project - Scope ReviewRichard Hall, Project Manager, CUIT

  22. Scope of Work Review

  23. Scope of Work – Detail To achieve the overall project objectives, CUIT has broken down the project into major tasks. Inventory (task 2) – COMPLETE Create detailed physical inventory of existing in-scope server Instrument server power consumption (task 3) – COMPLETE Install network monitored power monitors for each server Perform data collection at 5-min intervals Instrument server input air temperature and overall DC chilled water (task 4) - COMPLETE Install server input ambient air temperature for each server Install BTU metering on data center supply and return lines Perform data collection at 5-min intervals Establish overall Data Center profile (task 5) - September 2010 Utilize equipment load results to establish baselines Develop PUE ratio for entire data center & inventoried servers Implement 9 racks of high power density in-row cooling (task 6) – September 2010 Install 9 server racks outfitted for high power density Provide in-row cooling subsystem for 9 server racks 23 23

  24. Scope of Work Detail Cont’d Replace 30 “old” servers and measure efficiency improvement (task 7) – September 2010 Consolidate the replacement servers into high density racks and re-implement the same IT services Take measurements of before-and-after power consumption Document expected and actual efficiency improvement Compare old and new research clusters (task 8) – COMPLETE Benchmark applications on new Astronomy/Statistics HPC cluster Implement server power management (task 9) - September 2010 Install server BIOS/high level power management feature upgrades to servers (identified in task 2) Increase chilled water set point and measure (task 10) - May 2010 Document measured before-and-after energy consumption Communicate results (task 11) – On-going Share results with key stakeholders 24 24

  25. Installation Summary of Monitoring and Measurement ToolsIan Katz, Data Center Facilities Manager, CUIT

  26. Progress so far… Installed power meters throughout Data Center Established overall data center power usage ~ 290kW Installed metered PDU’s and plugged in inventoried hosts Ready for Idle benchmarking and service group power measurements Installed chilled water flow meters Established overall data center heat load ~ 120tons Estimated CU Data Center PUE (Power Use Effectiveness) Other Data Center Improvements

  27. Selected Metering Products • Power Panel Metering • WattNode Meter • Babel Buster SPX (ModBus to SNMP translator) • Server Level Metering • Raritan PDU • Chilled Water Metering • Flexim – Fluxus ADM 7407

  28. Power Meter Installation Installed WattNodes in 20 Power Panels 17 Panels in Data Center 3 Main Feeder Panels in Mechanical room ATS 2 & 3 - HVAC Load ATS 4 - IT Load • 290kW IT load read from PP1,2,3,4,5,6,16,26,27 • 120kW HVAC load read from ATS 2 & 3

  29. Chilled Water Meter Installation • Flexim meters installed in Mechanical Room • Sensors installed to measure flow rate and temperature • Result is Heat Flow Rate in tons • HF (tons) = Vol Flow (gpm) * ∆T / 24 • Sensors installed in 3 locations • Liebert CRACs 1 – 6 • AC 1 & 2 • Dry Coolers • Meters tied into same Modbus network as Wattnodes

  30. Server Level Metering Meter many different hardware types with Raritan PDU’s Sun: NetraT1, V100, V210, V240, 280R, V880, T2000 HP: DL360G4p, DL360G5p, DL380G5 28 servers Identified to: Establish Active/Idle Benchmark Investigate service usage comparisons Blade chassis (HP c7000) and blade servers (HP BL460c) metered with built in tools.

  31. Wattnode meters Campus Level power panel CRAC unit Data Center: 200 Level Flexim meters chilled water pipes Mechanical Room: 100 Level Main IT power feed (ATS4) server rack Raritan Power Distribution Units (PDUs) and Uninterruptible Power Supplies (UPSs)

  32. CU Data Center PUE Pie Chart

  33. More Data Center Improvements • Overhead Cable Trays • Will allow us to clean up under the raised floor • New Data Center Layout • Hot Aisle / Cold Aisle Format • *Future* -- Duct CRAC units & use ceiling as plenum • To return hot air from hot aisles to CRACs

  34. Measurement Plan and Initial ResultsRaj Bose and Peter Crosta, Research Computing Services, CUIT

  35. Overview: Tasks 7, 8, 9 • Task 7: Comparing power consumption of old and new(er) hardware • Task 8: High performance computing (HPC) cluster power consumption comparison • Task 9: Power management and tuning

  36. Task 7: Out with the old, in with the new • If we replace old servers with new servers, how will power consumption change? IBM 7090 in University Computer Center, 1966 Microsoft’s Chicago data center, 2009

  37. Task 7: Power measurement plan • Inventory servers • Determine comparison groups • Two-tiered power measurement approach 1) pre/post migration comparison 2) SPECpower benchmark

  38. Service: E-mail Hardware: new Time: 1 week Energy: ?? kWh Service: E-mail Hardware: old Time: 1 week Energy: 68 kWh E.g. Task 7: Pre/post migration comparisons • Power consumption of same IT services on different hardware Migration Old server New server Time

  39. Task 7: SPECpower benchmark • Industry standard benchmark to evaluate performance and power • Addresses the performance of server side Java • Finds maximum ssj_ops (server side Java operations per second) • With simultaneous power measurement, allows calculation of ssj_ops / watt (performance per watt)

  40. Task 7: SPECpower results – HP DL360G4p • On some systems, power consumption decreases as load decreases.

  41. Task 7: SPECpower results – HP DL360G4p

  42. Task 7: SPECpower results – Sun Sunfire V880 • On other systems, constant power consumption regardless of load.

  43. Task 7: Summary • Designed plan to measure old and new server power consumption in multiple ways • Energy consumed while running the same IT services • Performance per watt of power used • Pre-migration measurement period is complete; awaiting migrations and follow-up measurements • Benchmarking has begun and nears completion

  44. Task 8: Cluster comparison • Can a new, larger research cluster be more energy efficient than an older, smaller research cluster? Beehive Hotfoot

  45. Beehive Built in 2005 16 cores 8 servers Dual-core 2.2 GHz AMD Operton 2 to 8 GB RAM 10 TB SATA storage OpenPBS scheduler Theoretical Peak GFlops: 61.6 IDLE POWER IN WATTS: 2.7 kW Hotfoot Built in 2009 256 cores 16 high-density blades (2 servers each) Dual quad-core 2.66 GHz Intel Xenon 16 GB RAM 30 TB SATA storage Condor scheduler Theoretical Peak GFLops: 1361.9 IDLE POWER IN WATTS: 4.1 kW Task 8: The clusters

  46. Task 8: Comparison plan • Power use in active idle state • Beehive = 2.7 kW • Hotfoot = 4.1 kW • Energy consumption at load • Counting to one billion • Summing primes from 2 to 2 million (MPI) • Summing primes from 2 to 15 million (MPI)

  47. Task 8: Energy use while running jobs New cluster uses less energy to run research jobs than old cluster.

  48. Task 8: Summary • Older cluster consumes less power and uses less energy at baseline • Advantages of newer cluster are evident as utilization increases

  49. Task 9: Power tuning • Implement server-, BIOS-, OS-level power tuning and power management • Re-run benchmarks and service group comparisons to collect additional power usage data

  50. Summary of Tasks 7, 8, and 9 • Task 7: In progress • Task 8: Completed – newer cluster proves to use less energy than older cluster for research intensive work • Task 9: In progress

More Related