Controls Strategy

1. Controls Strategy Joel N.Butler Talk to Department of Energy Assessment of the Run II Luminosity Plan at the Fermilab Tevatron Accelerator Integration Session Feb. 24, 2004 Controls Strategy- Lehman Review – Feb 2004

2. What is the Concern? • The Control System works well but could work better and contains obsolete components • The character of the Run 2 upgrades puts a lot of pressure on the system. We are also for the first time running Fixed Target and Collider simultaneously. The system must not only control and monitor the complex but must provide data acquisition and archiving, support data analysis and trending, provide more automated control and feedback, and support complex accelerator studies and calculations • It must support/facilitate development of new applications • It must operate efficiently and be easy to maintain • It must continue to do these tasks for many years There are needs that are not optimally addressed by the current system, so process improvement is worth pursuing. We are undertaking a comprehensive review of the system Controls Strategy- Lehman Review – Feb 2004

3. Goal of the Current Review • We want to understand the requirements of the system and support departments in order to optimize the long term strategy • We aren’t trying to break things • We aren’t trying to make extra work for people • We are very aware of the need to maintain the ability to operate even while we undertake improvements. Controls Strategy- Lehman Review – Feb 2004

4. Constraints • No long shutdowns to replace large blocks of functionality • Precludes fundamental redesign of the system • Parts of new system must be plug compatible and communicate with old • Must be able to easily switch new things in and back out and/or run them in parallel • No big pile of money to replace everything • Base on inexpensive commodity computing equipment • Many small PCs/unix boxes rather than few large systems • No big influx of new people for a parallel effort • Work must be done by existing lab personnel. • No long break to learn a new system • New languages, tools etc. should have modest learning curve • A lot of code written by non-Computing Professionals • System department people are busy with accelerators Controls Strategy- Lehman Review – Feb 2004

5. Control System Overview Applications Applications Applications ….. CentralServices CentralServices ….. ethernet Front-Ends Front-Ends Front-Ends ….. field bus Field Hardware ….. Controls Strategy- Lehman Review – Feb 2004

6. Applications Applications Datapool Manager Datapool Manager Front-End Front-End Overview (VMS) >500 “Console Applications”, those with user interfaces, written in C and FORTRAN. These do much of the work in support of operations X Windows Display(PC) X Windows Display(PC) … Ethernet VAXes Computer Room Or elsewhere The VAX 4090’s are obsolete and unsupported ……………… Shared Memory ……….. Mature libraries and services to facilitate applications development. System is written in C (assembler) and relies heavily of VMS operating system ACNET/Ethernet ……………… Front Ends (anywhere) Controls Strategy- Lehman Review – Feb 2004

7. Application Web Application DSE----------------OAC/Servlet DSE----------------OAC/Servlet DAE DAE Front-End Front-End Overview (Java) ……………… PCs (anywhere) RMI http Sun Netras (computer room) ……….. From here, same protocol and database tables as VAXs ACNET (16 consolidators) ACNET Front Ends (anywhere) ……………… This system could be a complete replacement on a more modern, operating system independent, platform. It is used for many non-interactive task (OACs), for SDA, Autotune for MiniBooNe and Ecool, and a few interactive applications. Controls Strategy- Lehman Review – Feb 2004

8. Issues • VAXes • obsolete and constraining • Too little CPU power, memory, disk storage • There are many workarounds to solve problems • Have the most complete application support, but not easily portable • Familiar environment. AD people know how to program in C and FORTRAN. The applications are mostly portable if the new system’s infrastructure was made to “look like the VAX”. • JAVA • Modern computers and programming language with plenty of resources and easy expansion capability • Incomplete support for applications • Unfamiliar programming paradigm and language. “Porting”in the usual sense of the word is not possible • Migration was assumed to involve a rewrite • Because of the nature of Java, has some performance and tuning issues as well Controls Strategy- Lehman Review – Feb 2004

9. Approach: Controls Working Group • We need to find out • What we are running now and what problems there are • What we would like to be running now if only we had the resources (especially programmers) • What new applications we need to develop in the future and what resources it will take to acquire them and run them to full advantage • What specialized and embedded systems we support, how they are supported and how they integrate into operations • What specialized systems and capabilities we imagine supporting in the future We set up a group consisting of two representatives from each department and a few at large members to answer these questions – a.k.a the Controls Working Group (CWG). Controls Strategy- Lehman Review – Feb 2004

10. Schedule of CWG Talks Controls Strategy- Lehman Review – Feb 2004

11. Future Needs A major challenge for the Booster is to achieve and MAINTAIN peak performance … Protons/pulse Day to day and even hour to hour scatter is large! Maximum p/hr allowed by BLM’s Controls Strategy- Lehman Review – Feb 2004

12. Monitoring Changes • Need to continuously monitor and record changes in the machine’s behavior • Complicated by the rapid cycling nature of the Booster • Would like to monitor hundreds of devices on minute time scales but are limited by bandwidth issues • … lots of potential for improvement.. • Can provide alarm capability for snapshot devices • Change analyses • Auto tune Controls Strategy- Lehman Review – Feb 2004

13. Key General Themes • MORE BANDWIDTH! • MORE DATA STORAGE! • We are storing more and more stuff in the data-loggers • We would like it to stay around for a much longer time (years not months) • MORE AUTOMATION • “It’s the 21st Century. Why not think more about smart systems.” • Automated troubleshooting • Automated beam tuning These, together with the desire for a more convenient and easier application development environment were the key general themes we heard from the CWG Controls Strategy- Lehman Review – Feb 2004

14. Migration to JAVA • We did a survey of VMS Console Applications: • Not counting obsolete or redundant programs • Estimating what could be gained by application consolidation either within departments or across departments • We were pleased to learn that there was a fairly high degree of consolidation, even across departments • We have now established that there is a base of VMS console applications that is greater than 500 programs • Most people claimed that they did not have time to learn a new programming language. Nor did they have time to “rewrite” their programs in this new language • There has not been an aggressive program to teach people JAVA nor were there good tutorials anchored in the actual tasks people were trying to do • There are pieces of the JAVA support infrastructure missing. Many needs are too pressing to wait for JAVA to catch up so even some new APPS are written for the VAX Controls Strategy- Lehman Review – Feb 2004

15. The Emergence of a New Proposal • The Controls Group has proposed to attempt to “port” the VAX/VMS CONSOLE INFRASTRUCTURE to a LINUX/INTEL platform. • Then, most of the Console Applications should simply port • The much more modern, powerful processors should solve a number of problems and applications can be extended in a familiar language. • This proposal makes sense ONLY if the Console Infrastructure can be rewritten QUICKLY • The goal is for enough work to be done to port some applications by the summer and • To have most of the Console Applications migrated by Oct 2005 • These are just goals at present – planning/scheduling is just beginning • There will be a review of the proposal, including its implementation, in March. A charge has been written. Controls Strategy- Lehman Review – Feb 2004

16. What Happens to JAVA? • The JAVA environment has proven to be very useful and many modern Accelerator controls systems are based on it. In the end, it is believed to provide the best path to an easier application development environment • We will try to overcome some of the”acceptance” problems by • Developing JAVA tutorials • Producing examples of how to write common types of applications • Completing missing infrastructure • Improving documentation • Establishing a JAVA applications writer group Controls Strategy- Lehman Review – Feb 2004

17. How will it come together? • The JAVA system already runs on LINUX. Then, we will have one platform that can run C and JAVA • Somewhat farther down the road, one could see that a single set of common services could support either C or JAVA-based applications • The Controls Department will be supporting at least two systems for some time. The Department will have to devote resources to develop the infrastructure and some of the applications support will have to revert to physicists and others This requires the Department to play more of a leadership role and less of a support role. They will have to guide people to use the most efficient and appropriate tools for their applications. This will require management and the Department to work together to get the correct result. Priorities and rules will need to be set and enforced. Controls Strategy- Lehman Review – Feb 2004

18. Other Lessons from the CWG • Communication on controls issues is not great (for many good reasons) • Consequently, “best practices” are not uniformly applied across the Division • Knowledge of the capabilities and support available from the current system is not good • Needs and problems are not well-communicated • Priorities and schedules are not always well-established • We will continue the CWG (perhaps biweekly) and turn it into a forum for addressing controls issues across all departments. Examples are • hardware initialization • Save and Restore • Writing of non-interactive tasks (OACS) Controls Strategy- Lehman Review – Feb 2004

19. Conclusion • The Control System is not broke, but can work better to serve Fermilab’s goals. • We will work towards improving it • We hope that it will lead to easier operations and maintenance, more consistency, more automation, and more efficient studies and in that way contribute to the Run 2 luminosity improvements. Controls Strategy- Lehman Review – Feb 2004

20. Conclusion The Control System is not broke, but can work better to serve Fermilab’s goals. We will work towards improving it. We hope that it will lead to easier operations and maintenance, more consistency, more automation, and more efficient studies and in that way contribute to the Run 2 luminosity improvements. Controls Strategy- Lehman Review – Feb 2004