LHC Cryogenics Schneider PLCs: Radiation tests at H4IRRAD Mitigation measures

1. LHC Cryogenics Schneider PLCs: Radiation tests at H4IRRAD Mitigation measures M. Calviani, M. Brugger, G. Spiezia (EN/STI) E. Blanco, J.M. Beckers, D. Willeman, P. Durand, Ph. Gayet (EN/ICE)

2. Outline • Schneider PLCs issues on the LHC CRYO Control system • H4IRRAD test campaign results • Proposed mitigation measures M. Calviani, E. Blanco - EN Department

3. LHC downtime due to the Cryo Schneider PLC issues PLC Schneider Premium LHC Cryo • Is radiation the cause? • Potential SEUs: 7 All underground! M. Calviani, E. Blanco - EN Department

4. LHC refrigerators control architecture LHCA SCADA Data Servers Local & Central Control Rooms LHCCA LHCB LHCCB Return Module SL & SR Comp 1.8K LN2 Buffer Comp 4.5K Main Dryer Comp 4.5K Comp 1.8K Surface Main Dryer RM QSCCA QSAA QSCA QSDN QSDN QSCB QSCCB QSAB UCB 4.5K Cold Box 4.5K QSKA QSRA QSRB Shaft US85: ✓ US45: ✓ US65: ✓ UX85: ✓✓✓ QURA Cavern CB 1.8K UX45: ✓ CB 1.8K QURCA QUI QURCB Connection Box Profibus DP PA WorldFIP Alcoves Sector R (3.3 Km) Sector L (3.3 Km) RMSR Tunnel RMSL 4 M. Calviani, E. Blanco - EN Department

5. Radiation tests 1 • Objectives • Confirm PLC radiation sensitivity: as shown in the LHC cryogenics control system during 2011 operation • Not for testing PLC radiation hardness • Validate potential solutions • Architectures to test: • PLC Premium CRYO-like • LHC Cryogenics like sample but in two different configurations: • P575634M with memory card • P576634M without memory card • PLC Premium with remote I/O backplane • PLC CPU in safearea (UL) and a passive bus X connection to the remote I/O backplane located in radiation area. • PLC Quantum Redundant architecture • Possible solution to increase availability (should be seen decoupled of the radiation issue) 2 3 M. Calviani, E. Blanco - EN Department

6. H4IRRAD • H4IRRAD can reach 1 year of nominal LHC in ~1 week (underground zones) • LHC RadMon for monitoring – agreement with FLUKA simulations within 20-30% Production target SPS beam Internal irrad zone (LHC tunnel) External irrad zone (LHC shielded zones) Access is possible only from the top shielding (removal of 160 cm Fe blocks) M. Calviani, E. Blanco - EN Department

7. Radiation sensitivity • The PLCs have been tested during a period of low beam intensity due to their significant radiation sensitivity • Crashes observed after ~5-10 minutes at H4IRRAD nominal beam (~5*105 HEH/cm2/min) • Clear correlation between PLCs CPU crash and radiation • Cross-section estimated from a period with ~5*104 HEH/cm2/min (US85 ~ 3.5*102 HEH/cm2/min) ~4.5*105 HEH/cm2/min ~2*105 HEH/cm2/min ~5*104 HEH/cm2/min M. Calviani, E. Blanco - EN Department

8. Sensitivity results 1 2 3 • PLC Premium cross-section in agreement with LHC operation data • Radiation sensitivity appears to be dependent to the user application memory occupancy (~ 2.5 times) • Crash with no remote access possible with PLC Premium! • No failure observed or induced by the remote I/O backplane M. Calviani, E. Blanco - EN Department

9. LHC radiation levels in P4/P6/P8 • P8: LHCb luminosity • Values ≤4*107 HEH/cm2/2011 (in US85) • Evolution will directly depend on the cumulated LHCb lumi (1-2 fb-1 2011/2) – same value expected for 2012 • P4/P6: beam-gas due to pressure spikes in the cavities (P4) and in the extraction septa (P6) • Values ~106 HEH/cm2/2011 in UX • 2012 levels should be similar as this year • After LS1 values will depend on the vacuum behaviour with 25ns operation (situation not yet clarified) • P2: ALICE luminosity  levels insignificant in US25 M. Calviani, E. Blanco - EN Department

10. Conclusions of the test campaign • What we confirmed: • High sensitivity to radiation (SEUs) of Schneider Premium PLCs • SEU induced failure cross-section in agreement with LHC operation data during 2011 • What we observed: • Radiation sensitivity dependent on the user application memory occupancy (to be confirmed with Schneider) • Radiation robustness of the remote I/O backplane and I/O cards (including CPU reliability with that architecture) – no failure observed • Equal sensitivity with or without flash memory card • Redundancy solution offers limited radiation sensitivity robustness M. Calviani, E. Blanco - EN Department

11. Mitigation actions(winter shutdown) • Improve availability : The radiation campaign in H4IRRAD showed remote I/O low radiation sensitivity and, therefore, the PLC CPU relocation to a safe area (UL) is the solution retained. • Feasibility discussed already within the R2E activities (EN/ICE, EN/EL, TE/CRG) • Action prepared for the winter shutdown (including ordering, cabling, electricity) • Affecting P4, P6 and P8 (in that precise order). P2 not needed • Other improvements (once the PLCs are in a safe area) • Remote accessibility - Additional Ethernet card to be installed in the relocated PLCs. (21) - Allows remote access via IP reducing time and resources during interventions • Robustness - Eliminate flash memory cards in some PLCs (9, awaiting Schneider validation) - Already done in 2011 for the P8 QURA PLC • Diagnostics - Enrich the onboard PLC diagnostics (collaboration with Schneider Electric) 1 2 3 M. Calviani, E. Blanco - EN Department

12. Mitigation actions(Long Term) • Long-term actions (LS1) • Awaiting 2012 LHC operation to decide the next move (reliability of Schneider PLCs) • But in parallel: • Continue radiation testing on H4IRRAD facility on equipments like the ones remaining in radiation areas (PLC Quantum, Siemens on turbines,…) and also in other architectures (redundancy) • Study a possible full relocation of all active electronics in P4 and P6 as done in P8 • Validate PLC redundant architectures for a possible migration of some equipments during LS1This is independent of radiation issues and will increase the availability of the cryogenics control system in view of the large downtime induced by an issue in the control system (only if the new situation does not satisfy the required availability!) M. Calviani, E. Blanco - EN Department

13. Acknowledgments • Many thanks to: • EN/MEF for the precious support during construction and during each installation phase • The H4IRRAD team and EN/STI for the follow-up of the test area operation (radiation and beam monitoring) • EN/ICE for availability and reactivity • Thanks a lot for your attention M. Calviani, E. Blanco - EN Department

14. M. Calviani, E. Blanco - EN Department

15. H4IRRAD external zone EN/EL p+ beam EN/ICE PLCs M. Calviani, E. Blanco - EN Department

16. H4IRRAD configuration Production target SPS beam Internal irrad zone (LHC tunnel) External irrad zone (LHC shielded zones) Access is possible only from the top (removal of 160 cm Fe blocks) M. Calviani, E. Blanco - EN Department

17. Passive vs. active bus X solutions • The active bus X includes two electronic pieces in both ends of the cable that can be a source of problems under radiation environment. M. Calviani, E. Blanco - EN Department

18. H4IRRAD test area overview • What is it? • Test area to test equipment in LHC-like particle spectra • H4IRRAD can reach 1 year of nominal LHC in ~1 week (tunnel and alcoves) • Radiation monitoring • Radiation monitoring based on the LHC RadMon system • FLUKA simulations employed to cross-check and understand the radiation field (agreement within 20-30%) M. Calviani, E. Blanco - EN Department