Computer hardware and procurement at cern
Download
1 / 31

Computer Hardware and Procurement at CERN - PowerPoint PPT Presentation


  • 158 Views
  • Uploaded on

Computer Hardware and Procurement at CERN. Helge Meinhard (at) cern ch HEPiX fall 2005 @ SLAC. Outline. Procedures Hardware (being) procured Power measurements Observations. Procedures. Constraints (1). CERN is an international organisation with strict administrative rules

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Computer Hardware and Procurement at CERN' - paul


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Computer hardware and procurement at cern

Computer Hardware and Procurement at CERN

Helge Meinhard (at) cern ch

HEPiX fall 2005 @ SLAC


Outline
Outline

  • Procedures

  • Hardware (being) procured

  • Power measurements

  • Observations

[email protected]: Hardware procurement at CERN



Constraints 1
Constraints (1)

  • CERN is an international organisation with strict administrative rules

    • Competitive tendering required covering (at least) member states

      • No way to avoid for commodity equipment

    • Lowest compliant bid wins

      • No negotiations about added value of higher offers

[email protected]: Hardware procurement at CERN


Constraints 2
Constraints (2)

  • Different procedures depending on expected volume

    • < 10’000 CHF: IT seeks 3 offers

    • < 200’000 CHF: Formal price enquiry by purchasing service. Four weeks response time

    • < 750’000 CHF: Formal call for tender preceded by market survey. Six weeks response time

    • > 750’000 CHF: As < 750’000 CHF, plus approval by CERN’s Finance Committee (5 sessions/year, papers ready two months in advance)

      (1 CHF = 0.78 USD = 0.65 EUR)

[email protected]: Hardware procurement at CERN


Our problems
Our problems

  • Procedures badly adapted to quickly evolving computing market

  • Difficult to give preference to “good”, reliable equipment

[email protected]: Hardware procurement at CERN


Our choices 1
Our choices (1)

  • For significant purchases (> 100 kCHF) we require (a) sample system(s)

    • with the tender for big tenders

    • on CERN’s request for small tenders

  • Tenders include 3 years on-site warranty for hardware

    • Typical requirements:

      • 4 working hours response / 12 working hours repair for critical machines

      • 3 working days response / 5 working days repair for farm nodes

    • Supplier can subcontract on-site warranty

[email protected]: Hardware procurement at CERN


Our choices 2
Our choices (2)

  • Payment within 30 days after provisional acceptance on receipt of bank guarantee of 5% of purchase sum valid until end of warranty period

  • Delivery within 6 weeks, penalty for late delivery: 2% of purchase sum per complete week, max. 10%

[email protected]: Hardware procurement at CERN


Our choices 3
Our choices (3)

  • If more than 10% systems fail during acceptance or during first month after: right to return the whole batch

  • If a system fails 3 or more times during any 6 months’ period, right to request complete replacement of system

  • If more than 20% of any component fail during any 6 months’ period, right to request complete replacement of this component across batch

  • If CERN adds third-party devices, no impact on warranty obligations for system as delivered

[email protected]: Hardware procurement at CERN


Our choices 4
Our choices (4)

  • If justified by volume, procure from two suppliers (lowest and second-lowest compliant)

    • Better protection if one delivers crap or nothing at all

    • Better chance for companies to win an order

    • Increased workload on our part

[email protected]: Hardware procurement at CERN


Example of a procurement
Example of a procurement

  • Procurement of equipment worth < 750 kCHF

    • Approval by Finance Committee not needed

  • Market survey already done

    • Market survey can cover different types of equipment

    • Valid for 1 year

    • If not done yet, add ~ 16 weeks

[email protected]: Hardware procurement at CERN


Steps 1
Steps (1)

  • Fix scope 2 w

  • Write technical, commercial docs 3 w

  • IT-internal review

  • Revise technical, commercial docs 2 w

  • Specification meeting

  • Revise technical, commercial docs 1 w

  • Tender out

  • Deadline for replies 6 w

  • Opening of replies 1 w

    (Total so far: 15 weeks, at best compressible to 12 weeks)

Typical case

[email protected]: Hardware procurement at CERN


Steps 2
Steps (2)

(Total from previous slide: 15 w, min. 12 w)

  • Technical analysis of replies 1 w

  • Visual inspection, mounting 1 w

  • Benchmarks, reports 3 w

  • Technical clarifications 1 w

  • Purchase request, order 2 w

  • Delivery 7 w

  • Preliminary acceptance 6 w

    Total: 36 weeks, compressible to 30 weeks

Typical case

[email protected]: Hardware procurement at CERN



Objectives
Objectives

  • Cover existing needs with as few different models and as few procurement procedures as possible

  • Closely follow technology and market evolution and satisfy requirements with modern hardware at low cost

contradiction

[email protected]: Hardware procurement at CERN


Fabric infrastructure and operations 1

From CERN site report 2005/10/11

Fabric Infrastructure and Operations (1)

  • RedHat 7.3 phased out on public services

    • Campaign on storage nodes far advanced

  • New in machine room since Karlsruhe:

    • 200 farm PCs (dual Nocona): in production

    • 116 disk servers (> 5 TB usable each, total of 900 TB gross capacity): part in production, part under acceptance test

    • 112 “midrange servers”: under acceptance test

    • 32-node Infiniband-based cluster for Theory

  • Refurbishment of machine room proceeding

    • LHS being populated, but power remains limited

Talk

[email protected]: Hardware procurement at CERN


Hardware being procured 1
Hardware being procured (1)

  • Large volumes – several times < 750 kCHF per year

    • “Farm PCs” – non-redundant, cheap dual-processor work horses

    • “Disk servers” – storage-in-a-box systems with many SATA disks for streaming applications

[email protected]: Hardware procurement at CERN


Hardware being procured 2
Hardware being procured (2)

  • Medium-size volumes – once < 750 kCHF per year or once or several times < 200 kCHF per year

    • “Midrange servers” – redundant building blocks for specific applications

    • “Tape servers” – midrange servers with an FC interface

    • “Disk arrays” – autonomous RAID units with FC uplinks

    • SAN infrastructure (most notably FC switches)

    • Head nodes for serial console infrastructure

    • “Small disk servers”, somewhere between disk servers and midrange servers

    • Miscellaneous

[email protected]: Hardware procurement at CERN


Specifications farm pcs 1
Specifications: Farm PCs (1)

  • 2 boxed Intel Noconas of 2.8 GHz

  • Mainboard:

    • BMC (IPMI 1.5 or higher)

    • PXE, USB boot

    • BBS menu

    • Console redirection

    • Configurable to stay off on AC power loss

  • 2 GB ECC memory

    • From mainboard manuf. approved list

    • Upgradable to 4 GB without removing modules

[email protected]: Hardware procurement at CERN


Specifications farm pcs 2
Specifications: Farm PCs (2)

  • 1 disk > 140 GB, IDE not permitted

    • Certified for 24/7, 3 y warranty by disk manuf.

  • 1 GigE providing PXE and IPMI access

  • 19” chassis max. 4 U, with rails

    • Power, reset button

    • Power, disk activity LED

  • Power supply supporting machine + 50 W

    • Active PFC

    • C13 to C14 LSZH power cord

  • Guaranteed to run under RHEL 3 (i386 and x86_64)

  • Delivery within 6 weeks from dispatch of order

[email protected]: Hardware procurement at CERN


Specifications disk server 1
Specifications: Disk server (1)

  • 1 or 2 boxed Intel Xeon with EM64T

  • Mainboard as for Farm PCs

    • Now adding support for memory mirroring

  • Memory as for Farm PCs

  • General requirements for disks etc.

    • ≥ 7200 rpm, no EIDE, 3 y warranty, certified for 24/7 by manufacturer

    • Metallic hot-swap trays certified by chassis manuf.

    • Indicators for power and activity for each tray

    • PCB backplanes for disks, multilane cabling

    • “Intelligent” RAID controllers

[email protected]: Hardware procurement at CERN


Specifications disk server 2
Specifications: Disk server (2)

  • System disks: 2 x ≥ 140 GB mirrored

  • Data disks: all identical

    • Redundant RAIDs with hot spares (min. 1/15)

    • Total usable capacity per system above 5 TB

    • Battery buffer if controller with active cache

  • 1 GigE providing required performance, PXE, IPMI access

  • 19” chassis rack-mountable with rails

    • Min. 40 TB usable in 42 U high rack

  • Power supply: N+1 redundant, active PFC

  • Guaranteed to run under RHEL 3 (i386 and x86_64)

  • Delivery within 6 weeks from dispatch of order

[email protected]: Hardware procurement at CERN


Specifications disk server 3
Specifications: Disk server (3)

  • Performance: memory to disk: iozone with 16 GB files and 256 kb record size

    • Single stream: 40 MB/s write, 40 MB/s read

    • Multi-stream (at least 10): 115 MB/s write, 170 MB/s read (*)

  • Memory to network: iperf

    • Single stream: 100 MB/s write, 100 MB/s read

    • Two streams: 110 MB/s write, 110 MB/s read

    • Two streams in, two streams out: 145 MB/s

[email protected]: Hardware procurement at CERN


Specifications disk server 4
Specifications: Disk server (4)

  • Global (disk to network) performance: At least 10 clients transferring 2 GB files via rfio

    • Reading from system: 95 MB/s (*)

    • Writing to system: 90 MB/s (*)

      (*): Requirements scale linearly with usable capacity, numbers for 5000 GB usable

[email protected]: Hardware procurement at CERN


Power measurements

Power measurements

Done by

Andras Horvath, CERN


Power measurements1
Power measurements

http://ahorvath.home.cern.ch/ahorvath/power

[email protected]: Hardware procurement at CERN



Observations 1
Observations (1)

  • Profile of winning companies

    • Tier-1 suppliers competing with large integrators

    • Small ‘round the corner companies eliminated at Market Survey stage

    • Almost always the integrators win

      • Specially tailored solutions responding to our specifications

      • Prices of Tier-1s rather high in Europe

[email protected]: Hardware procurement at CERN


Observations 2
Observations (2)

  • Stress test as (important) part of the acceptance test

    • Introduced ~ 2 years ago (triggered by presentations from SLAC and FNAL at HEPiX)

    • Very useful

    • Based on va-ctcs

      • No longer sufficiently actively maintained

      • Large number of false positives

      • Looking for a replacement

[email protected]: Hardware procurement at CERN


Observations 3
Observations (3)

  • Pushing these procedures through requires dedicated (and knowledgeable) person power

  • Not obvious to run multiple procedures in parallel

    • In particular, if things go wrong, e.g. stress test fails

[email protected]: Hardware procurement at CERN


Summary
Summary

  • Computer hardware procurement is an excellent experimental confirmation of two fundamental laws of human nature

    • Murphy: “Everything that can go wrong will go wrong.”

    • Hoffstaedter: “Things always take longer than you think, even if you take into account Hoffstaedter’s law.”

[email protected]: Hardware procurement at CERN


ad