Computing in be co
1 / 37

Computing in BE-CO - PowerPoint PPT Presentation

  • Uploaded on

Computing in BE-CO. Vito Baggiolini BE-CO-DO IT Technical Forum 4-May-2012. A few words about myself. Senior member of BE-CO Software Engineer Technical Coordinator of BE-CO Section Leader of the new “ DevOps ” section. Outline. Overview of Accelerator Controls

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Computing in BE-CO' - delano

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Computing in be co

Computing in BE-CO

Vito Baggiolini BE-CO-DO

IT Technical Forum 4-May-2012

A few words about myself
A few words about myself

  • Senior member of BE-CO

  • Software Engineer

  • Technical Coordinator of BE-CO

  • Section Leader of the new “DevOps” section


  • Overview of Accelerator Controls

  • Services used from IT department

  • BE-CO and the new DevOps section

  • 3 initiatives to further improve quality and availability of the control system

V. Baggiolini, ITTF 4-May-2012

Computing in be co

“The beam is our master”

Vito Baggiolini, ITTF, 4-May-2012

Computing in be co

V. Baggiolini, TC 12-Mar-2010

“Operations are our main client”

Controls hw infrastructure
Controls HW infrastructure







Fixed Displays

Other operational


Presenta-tion tier

Appl Servers

PVSS Servers

File Servers


Middle tier



Real-time Linux LynxOS Computers








Resource tier






Beam instrum. KickersMachine Protection, …

Magnet Control, RF

Quench Protection, …

Cryo, Vacuum, …

Computing in be co

Fixed Displays

Operator Consoles

IEEE-NSS-MIC - CERN Beams Controls Group

Be co related software
BE-CO-related Software

TCP/IP communication services

OP Specific GUIs

OP Specific GUIs

DB Access


Fixed Displays Frame

Expert GUIs

Controls Middleware


Business Layer

Alarms (LASER)

LHC Software Architecture Core










Settings &



Settings &



Settings &



Timing Management

TCP/IP communication services




Software InterlockSystem



Controls Middleware


Role Based Access Control RBAC

Front-End FESA servers

Front End Layer

Front-End FESA servers

TCP/IP communication services

Front-End FESA servers

device servers

RT Lynx/OS

VME Front Ends


Front Ends


Using BE-CO Frameworks

Developed by BE-CO

Computing in be co

Java Applications

Vito Baggiolini, ITTF, 4-May-2012

Some software releated numbers
Some Software-Releated Numbers

  • GUI and Middle-tier (Java)

    • ~8 million lines of production code

    • > 1000 jar files in production

    • Combined to 400 different GUIs and 200 server programs

    • Up to 1000 processes running on 550 machines

    • Developed by 80 people from 10 different groups

  • Front-End Layer (C/C++)

    • 550 different device types

    • 70’000 device instances on 1000 different front-end machines

    • Developed by80 people from 8 different groups

Be co has a preference for open solutions
BE-CO has a preference for “open” solutions

  • Open standards and protocols

    • E.g. CORBA, JMS for middleware

  • Open source where possible

    • Linux as the recommended operational OS

    • Open-source 3rd party for C, C++ and Java

  • Open hardware initiative

    • The VHDL of our electronics is on the Internet (




  • Overview of Accelerator Controls

  • Services used from IT department

  • BE-CO The new DevOps section

  • 3 initiatives to further improve quality and availability of the control system

V. Baggiolini, ITTF 4-May-2012

It services needed 24 7 for beam
IT services needed 24/7 for beam

  • Network:

    • At the basis of our (distributed) system

  • Databases:

    • Oracle used online in core applications.

  • Authentication/Authorization: (ldap, active directory)

    • Used for Role-based access control in operational applications

  • DFS home directories

    • For some operational accounts (e.g. cryo)

  • Web Services

    • Used for accelerator status pages watched (also) by high management

  • Generally we’re very happy! We have good collaborations and get good services and support.

It services not directly needed for beam
IT services not directly needed for beam

  • HyperV (Linux and Window) for SW development on the Technical Network (TN)

  • Windows TS for expert access from home

  • DFS and AFS in our daily work

  • SVN for all sources and part of configuration

  • Yum (Linuxsoft) and CMF for software installations

  • ADSM (Tivoli) for automatic backups

  • CASTOR for long-term backup

  • Security services (CNIC, security scans)

  • IT Printer service

  • We heavily rely on your services and expertise!


  • Overview of Accelerator Controls

  • Services used from IT department

  • BE-CO The new DevOps section

  • 3 initiatives to further improve quality and availability of the control system

V. Baggiolini, ITTF 4-May-2012

The be co group in context
The BE-CO group in context

  • “The beam is our master, BE-OP our main client”

    • Direct and personal support

  • Equipment groups

    • Use our real-time embedded computers and software frameworks

    • Beam Instrumentation (BE-BI) Radio Frequency (BE-RF), Power Converters (TE-EPC), Kickers, transfer lines (TE-ABT), Machine Protection (TE-MPE), Motorizations (TN-STI)

  • Industrial controls (EN-ICE)

    • Use machines provided by BE-CO

  • IT department as our main solution and service provider

    • BE-CO often acts as a “value added reseller”

Co mandate and internal structure
CO mandate and internal structure

  • Group Mandate: contribute to producing the beam

    • Controls infrastructure for accelerator complex

    • Core controls sub-systems and applications

    • General purpose building blocks and frameworks

    • Operational support: 24/7 (piquet for HW, best effort for SW)

    • Development tools and advice

  • Sections:

Be co sections and their mandate
BE-CO sections and their mandate

  • Hardware and Timing: (Javier Serrano)

    • General purpose electronic modules,

    • drivers + driver generation framework

    • Timing system (generation and distribution)

  • Front-end computing: (Marc vanden Eynden)

    • Real-time embedded computer platforms (racks, crates, CPUs)

    • FESA (Front-end SW Architecture - Framework to integrate devices)

    • OASIS (software-based, remote oscilloscopes)

  • Data and applications: (Katarina Sigerud)

    • All data management activities (mainly Oracle)

    • Core controls applications (Java)

    • SW frameworks and components (GUIs)

Infrastructure section pierre charrue
Infrastructure Section (Pierre Charrue)

  • HW procurement, installation and maintenance

  • File servers and backup service

  • User management, mapping from users to roles

  • Windows system admin

  • Window Terminal Servers

  • HyperV PCs, server-side virtualization (upcoming)

  • Software projects

    • Controls Middleware (CMW)

    • Role-based access control (RBAC)

    • Diagnostics and Monitoring for OP (DIAMON)

  • Heavily based on IT products, services and expertise

Vito Baggiolini, ITTF, 4-May-2012

Dev ops


Dev Ops

  • Opera-tions

  • Develop-ment

  • Quality assurance


Bridging the gap between Development

and IT Operations

by using agile techniques

Do section members
DO section members



















Development tools dev
Development Tools (“Dev”)

  • “Individual” SW development tools

    • JDK, GCC, 3rd party libraries on Linux

    • Tailor-made Eclipse distribution (on Linux and Windows)

  • Issue tracking and configuration management tools

    • Atlassian suite (JIRA, Confluence, Fisheye, Crucible)

  • Repositories

    • SVN: “added value”: admin, Eclipse integration, best practices

    • NFS and Maven Repository (Nexus) for binary artifacts

  • Build/release tools and servers

    • Java: old (ANT) and new (Maven or Gradle)

    • C/C++: new build tool with dependency management

  • Testing and Quality Assurance tools

    • Continuous Integration servers (Atlassian Bamboo)

    • Testbed (miniature accelerator controls system)

Plans for dev tools in 2012
Plans for dev tools in 2012:

  • New, tailor-made Eclipse distribution (3rd party tools?)

  • Dependency analysis pluginfor Eclipse

  • New build and release tools for Java and C/C++

  • Continuous integration also for C/C++

  • Testbed becomes official part of release workflow for Control System core

  • Instrumentation / Usage data collection of applications, jars, Eclipse plugins, JIRA projects, etc.

Computing in be co

Continuous Integration

Triggered by changes in a dependency

SIP -- The Software Improvement Process - K.Sigerud, 13th October

Courtesy K. Sigerud

Computing in be co

Test Coverage

Red = not covered

Green = covered

Courtesy K. Sigerud

SIP -- The Software Improvement Process - K.Sigerud, 13th October

Operational unix platforms ops
Operational Unix Platforms (“Ops”)

  • Operational Linux and LynxOSplatforms

    • On Front-ends, middle-tier servers, operational consoles,boot servers

    • Selection, configuration and validation of new OS versions and Kernels from IT

    • OS installation, system administration + upgrades

    • OS expert user support + debugging

    • IT Security administration (in collaboration with S. Lueders)

  • Monitoring and Auditing of our systems

    • With BE-CO tool (DIAMON), Lemon or other tools

  • Deployment of operational controls software

    • Deployment scripts

    • Console Manager to launch GUIs

    • Wreboot / transfer.ref for process management

  • Heavily relying on IT services and support!

Plans for ops system admin in 2012
Plans for Ops (system admin) in 2012

  • New SLC6-based Kernel with MRG Realtime

  • Discuss and implement selected security improvements (in collaboration with S>>L)

  • Better/more monitoring, auditing, instrumentation

  • New deployment tools (Puppet/GLU?)

  • Prepare for Long Shutdown 1

    • Document dependencies between operational machines

    • Isolate a limited set of machines/services to be run during LS1

    • Study possibilities of geographical redundancy after LS1

    • Prepare for legacy clean-up (e.g. LynxOS)

    • Prepare for other changes (e.g. IP multicast, IPv6)

Challenges for support
Challenges for Support

  • Dev Tools user community grew from ~40 people to the whole accelerator sector

    • Exploding JIRA: 500 users, 210 projects

    • Exploding SVN: 1200 active SVN projects, 200 committers

  • Rapidly growing number of machines (for LHC)

    • From 250 to 1000 embedded real-time computers

    • From 50 to 300 middle-tier servers (Linux + Windows)

    • From 100 to 450 consoles (Linux + Windows)

  • More support with same/fewer people

    • More (planned) maintenance work in general

    • More support requests

    • Too many interruptions

Strategy for support
Strategy for support

  • Reduce diversity

    • Few official solutions, no/few special cases

    • Support only for official solutions

    • Consequent phasing out of old solutions

  • Optimize solutions for lighter support load

    • High quality, well-tested solutions that just work

    • Automation instead of documentation and human help

    • Documentation

    • Better tools for remote administration and diagnostics

    • Use APIs to IT services (e.g. programmatically create/delete NetOps entries, SVN repositories, e-groups, …)

  • “Delegate”

    • 1st line support people within user teams

    • Tell users to directly use IT solutions (e.g. JIRA)




  • Overview of Accelerator Controls

  • Services used from IT department

  • BE-CO The new DevOps section

  • 3 initiatives to further improve quality and availability of the control system

    • Software Improvement Process

    • Accelerator Controls Exploitation Tools

    • Smooth Upgrades WG

V. Baggiolini, ITTF 4-May-2012

Software improvement process sip
Software Improvement Process (“SIP”)

  • Started in 2009 for Java (now also C++)

  • Objectives

    • Introduce quality assurance as an integral part of the everyday development work

    • Leverage tools to automate the process as much as possible

    • Establish guidelines and metrics to measure quality

  • Recommended/Obligatory Activities (“rules & tools”)

    • Interactive design reviews across team boundaries

    • Automatic code analysis (Eclipse Warnings, Findbugs)

    • Interactive code reviews of critical parts (Atlassian Crucible)

    • Unit tests with > 30% coverage (Atlassian Clover)

    • Continuous integration

  • 4 dedicated SIP days a year

  • Top/Flop lists as motivation ;-)

Computing in be co

Automatic code analysis

The ‘bug’ line indicated

A list of ‘bugs’

‘Bug’ explained

SIP -- The Software Improvement Process - K.Sigerud, 13th October

Computing in be co

TOP / FLOP lists

SIP -- The Software Improvement Process - K.Sigerud, 13th October

Accelerator controls exploitation tools acet
Accelerator Controls Exploitation Tools (ACET)

  • Better Troubleshooting and diagnostics tools

    • “Deepen and broaden” diagnostic scope

    • Enable operators to do first-line diagnostic (deepen)

    • Enable CO experts to look into each other’s systems (broaden)

  • Objectives

    • Documentation portal

    • Leverage and integrate existing diagnostic tools

    • Collect and use dependency information

    • Early-warning system based on our monitoring tools (DIAMON)

    • Collect and analyze logfiles centrally (Splunk?)

  • Timeline

    • Proof of concepts this year

    • Fully operational after LS1

Smooth upgrades wg
Smooth Upgrades WG

  • We have to upgrade a running controls system

    • Low-risk, small or peripheral changes: anytime

    • Riskier, large-scale or core changes: during Technical Stops

Smooth upgrades wg1
Smooth Upgrades WG

  • We have to upgrade a running controls system

    • Low-risk, small or peripheral changes: anytime

    • Riskier, large-scale or core changes: during Technical Stops

  • Sector-wide working group (with all our clients and partners) lead by me

    • Elaborate a workflow and spread good habits

  • Smooth Upgrades Workflow

    • Good preparation (analysis of impact, Plan B, e.g. roll-back)

    • Announcements to all relevant people, without spamming

    • Preference for backward compatible changes

    • Good testing (continuous integration, testbed)

    • Deployment with roll-back if problems

    • Follow-up and learning

Conclusions and next steps
Conclusions and next steps

  • I hope you got an insight into what we do and what IT services we use

    • BE-CO and its new DevOps section

    • Good clients and “value-added resellers” of IT services

  • I hope this laid the basis for good future collaboration with the new DevOps section

    • I’ll come and talk to you individually

  • Please share ideas, feedback, etc. with us!

    • Possible collaborations

    • IT Services we don’t yet use

Vito Baggiolini, ITTF, 4-May-2012