Outline

1. SLAC LCLS First Year in Operation andControls Lessons LearnedDebbie RogindSLAC Controls Software

2. Outline • LCLS Status • Commissioning Success • Enhancements in Progress • Controls Lessons Learned • LCLS Future Plans • LCLS II • R & D

3. Linac Coherent Light Source at SLAC Injector (35º) at 2-km point Existing 1/3 Linac (1 km) (with modifications) New e- Transfer Line (340 m) Undulator (130 m) X-rayFEL uses last 1-km of existing 3-km linac Electron Beam Dump

4. SLAC/LCLS Main Control Center (MCC) Gun brightness beats LCLS Goals Max x-ray pulse energy of 4.5 mJ surpasses design of 2.0 mJ (6.4 Gev) FEL is fully tunable in rep rate, pulse length, photon energy, peak power,…

5. Linac-to-Undulator (340m)

6. Undulator Hall (130m) - 33 Undulator Segments Beam saturates in 60m rather than 90m; 25 of 33 undulators are installed

7. Beam Dump

8. Most BPMs & Magnets EPICS LCLS Commissioning Success (2007-2010) Challenging combination of Legacy and EPICS First e-through linac First e- from gun Firste-to dump First Light in FEE Install Undulators First lasing Install BC2 First Users CD-4 Review First Light in FEH Today X-Rays in NEH PEP-II ends J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J M A M J J A S O N D A S O D A D M 2007 2008 2009 2010 Down Down Down All BPMs and Magnets are EPICS, Linac Upgrade in Progress Most BPMs and Magnets are EPICS Mix of Legacy and EPICS new devices EPICS new devices Injector/BC1 Linac/BC2 NEH LTU/Undulator FEL/FEE Drive Laser Installation (Aug-Nov, 2006) First Year Of Operations ~ 4 years Gradual migration from Legacy

9. LCLS - all EPICS early 2011 • Linac Upgrade underway (T Himel) • Move all critical functionality off of legacy VMS system and onto Linux • Legacy micros replaced with VME IOCs • Move existing CAMAC control to VME module • RF control and timing control challenging • 100% EPICS early 2011! Status

10. Linac Upgrade Status

11. 120Hz Operation Jan 2011 fb03 core fb02 fb01 Controller IOCs IN20 Li22 LI23 LI24 Li25 LI26 LI27 Li28 LI29 LI30 BSY LTU UND DMP LI21 LTU0 mg01 LTU1 mg01 DMP bp01 BSY bp01 BSY bp02 bp02 rf01 rf01 bp02 bl01 5 4 LTU1 bp01 LTU1 bp02 8 UND bpo1 UND bp02 9 10 27 10 4 1 7 bp01 bp02 bl01 4 bp01 LTU0 bp01 LTU1 bp03 LTU1 bp04 UND bp03 UND bp04 7 6 1 RF EIOCs 7 4 6 9 9 120Hz Fast Feedback (D Fairley) • Dedicated FNET multicast network; FCOM protocol (T. Straumann) • Pattern Aware Unit (PAU); RF Control (K Kim) • Replaces MATLAB feedbacks 120Hz Beam Synch Acquisitionplus MPS work Status

12. 465 Total IOCs

13. LCLS Controls Success • Major commitment to EPICS based control • Sept 2010: 465 Total IOCs • 165 VME IOCs, 218 EIOCs, 82 Soft IOCs • Total PVs: 697,330, and growing with linac upgrade • 115,000 PVs archived, and growing with linacupgrade • Control system ready on time • Great collaboration with operations and physicists • Reliability/Availability is unusually high Status

14. LCLS Controls Availability – Last User Run • Controls Dept Goal: 98.8% Status

15. Controls Lessons Learned EPICS and Legacy System Integration Physicist Contributions Controls HLA development IOC Development Controls System Monitoring Policy Lessons

16. EPICS and Legacy • Move off of legacy system sooner • Incorrectly assumed existing legacy system physics applications would dominate • Years of effort expended to develop complexEPICS-based bridge thatprovides legacy system with EPICS data not worth the large effort • Underestimated the utility of AIDA (Accelerator Independent Data Access, G White) • Provides legacy data to EPICS (more lucrative direction) • MATLAB + AIDA + VMS based CAS + labCAbridges the gap between legacy and EPICS. Physicists can rule! Lessons

17. Harvest the Power of Physicists • Harvest the Power of Physicists • Provide MATLAB / EPICS infrastructure sooner • Look for synergy between physicists and Controls - not competition Lessons

18. MATLAB High Level Physics Apps …

19. MatlabSupportPVs • Soft IOC framework for physicists - • Banks of user-configurable PV records for physics data (Mike Z)

20. Harvest the Power of Physicists • PROS for physicists’ MATLAB GUIs • Makes use of a huge resource of physicists; • Domain knowledge; no need to translate requirements to engineers • Rapid prototyping – agile methodologies • Automatic access to machine time / shifts • GUIs are intuitive • CONS for physicists’ MATLAB GUIs • No structured requirements, design reviews; limited documentation • No code reviews; code can be difficult to follow • Reliability issues • Limited support beyond original author • What happens to the maintenance after the physicists transfer to another project? Lessons

21. Controls Java-based HLA • Java-based apps built from common jars with GUIs are too heavy-weight and feature rich (complex) • JVM tuning is challenging • Ops desire performance and reliabilty over feature rich • Evaluate service based architecture • XAL Model roll-out took more effort than expected • “Like fitting a square peg into a round hole”, Mark Woodley • Physicists went ahead and developed their own MATLAB LEM and Model Lessons

22. Controls Model Based Apps - LEM P Chu, G White et al

23. Controls Model Based Apps– Model Manager P Chu, et al

24. Controls HLA - MPS S Chevstov, S Norum et al

25. IOC development Place requirements, design review and documentation in all project schedules from beginning Lack of code reviews at all levels (driver,…,DB) cause cleanup work later; can cause downtime EPICS IOC configuration and deployment standards needed day 1 Naming standards verification tool needed to enforce naming convention Tight schedules trump above! Lessons

26. IOC development • Underestimated RTEMS skill set required • Train RTEMS personnel earlier for personnel load balancing • Drivers, Board support • Strong OSI policy needed day 1 for device support Lessons

27. IOCInterfaces • Require “QA” IOC external interface checklist prior to IOC deployment – gets left undone • Publish IOC APIs ! • Establish archiver policy - rates, PV fields (.EGU, .LOLO, etc ) • Complete configs/integration for RDB, SCORE, IRMIS, ALH, EDM, autosave, archiver, message logging, Help System • “Boot Fests” were required to ensure IOC restores to desired state • invalid PV, autosave parameters plagued us • Manual and disparate client configurations above should be automated • Archive Engine reliability /scaleability a surprise • Forced to establish rules for PVs that should/should not be archived Lessons

28. Control Systems Monitoring Engineers need a more thorough monitoring and alert mechanism to inform them of errant / alarmed IOC or linux system process prior to being contacted by Operations Lessons

29. Policy All software, no matter how small the modification, needs a code review prior to release into production Every subsystem engineer needs a trained backup Comprehensive test plan for any software release is required. All test plans are reviewed by physicists and operations for approval prior to release into production. Requires lots of time! Every test plan requires a backout plan Controls Deputy (Controls SW engineers rotate weekly) coordinates test plan /approvals /scheduling/ oversight. Lessons

30. Policy • All software should be scheduled via one of the following mechanisms: • CATER (trouble report from user) • Software Request Database • An approved Project schedule • Supervisor request • NOT in the “hallway” from a user Lessons

31. Policy • Invest in test stands. • Needed, given the high demand for machine time. Competition especially keen after user runs started. • For new project development, relieving engineers of their support and maintenance dutiesgives the schedule more determinism • Worked well for LCLS! Lessons

32. Future Looks Bright! LCLS II R & D Future

33. Future is bright: LCLS II Builds on the success of and demand for LCLS experimental time The LCLS II will essentially be a clone of the LCLS, 10 sectors upstream, and with a challenge of supporting multiple beam lines (~$400M project) Goal is to support more users with reliable high-quality beam CDR work commencing Construction start ~2012 Expected lifetime is 20-30 years Lots of R&D before that (new platform, remove CAMAC) Future

34. LCLS II Layout 3-7 GeV undulator RF gun-2 enclosure exists at sector 10 3-7 GeV bypass line L0 RF gun-1 X L1 BC1 L2 BC2 L3 L0 FACET wall undulator X L1 BC1 L2 BC2 L3 3-15 GeV LCLS existing sector-11 sector-14 sector-21 sector-24 und-hall Future

35. Current R&D Direction • Move to a micro-TCA platform • Research already underway • Design and test controls for all items in a sector • RF (funds for major upgrade existnow) • BPM, Timing, Magnets, Toroids, Movers, etc • Propose modern upgrade for LCLS II and later LCLS as well • Team: R Larsen, Q Yang, T Himel, and many others Future

36. Conclusion • EPICS is now assumed by everyone at SLAC for new projects • Many lessons have been and continue to be learned by SLAC Controls. • Use to establish future direction for LCLS II • SLAC Controls is actively pushing for a viable long-term controls hardware solution. • Lots of fun, challenging, work ahead! Future

37. The End Thank you for listening.