LDAQ – the New Lujan Center Data Acquisition Application

1. LDAQ – the New Lujan Center Data Acquisition Application Frans Trouw, Gary Cooper and Paul Lewis Lujan Center, Los Alamos National Laboratory

2. Outline • Brief Overview of Lujan Center • Description of hardware • Existing data acquisition application (LOS) • Issues with LOS (reminder, see Paul’s talk) • Guiding principles for new application (LDAQ) • Overview of existing new LDAQ application • Experiences with LDAQ (good!) • Summary

3. Data Acquisition at the Lujan Center - hardware • Detectors are all 3He gas tubes or area detectors • Pre-amps at detectors generate a differential signal • Custom “Time-of-Flight” boards take differential input and convert into events in a buffer • VME (VXI) crate with single board computer collects events from TOF units

4. T0 from accelerator T0 from accelerator VME Crate VME Crate Processors Processors CPU ROC TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF VxWorks VxWorks Remote access CPU CPU CPU ROC ROC ROC TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF "Public" LANL "Public" LANL Private DAQ Private DAQ Network Network Network Network 100 100 Mbit Mbit /s /s 100 100 Mbit Mbit /s /s LANSCE Facility LANSCE Facility EPICS Gateway EPICS Gateway CPU CPU CPU ROC ROC ROC TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF Chopper Chopper EPICS IOC EPICS IOC DAQ Host PC DAQ Host PC Windows Windows XP CPU CPU CPU ROC ROC ROC TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF TOF Data Archive Data Archive VXI Crates VXI Crates Controls PC Sample Environments Detector & Preamp PC PC Panels Overview of Hardware Configuration

5. Crate contains SBC, ROC module, and 1-11 TOF units ROC module TOF unit SBC To Detectors

6. “Model compiler”: templates generation scripts (rules) Source code generator Imported software “Model compiler”: run-time library Build Source file Source file Source file Source file Source file Source file Source file Build script Domain: class diagram, state charts, action language Domain: class diagram, state charts, action language Domain X: class diagram, state charts, action language Executable Executable Executable LOS Development Methodology

7. xtUML is in principle attractive • High level description generates consistent code and less errors • Extending or changing model generates new code that takes into account interdependencies • Less hand-coding => less manpower • Quality is consistent and assured

8. xtUML has Significant Drawbacks • Requires the desire to think at a very high level of abstraction – requires expert • Not all of the code can be generated • Highly dependent on message passing • The community using xtUML is very small (< 100 at SMUG) - “early adopters” • Only available for Windows • Debugging next to impossible!

9. xtUML – communication risky? • Message passing (MP) is critical to xtUML • MP is done using Windows message queues or sockets • A single communication failure kills the whole system • VxWorks 5 network stack obsolete • Windows network stack documentation limited (being generous here)

10. LOS move to Linux – MP problems! • As Windows is so opaque and a single user environment, xtUML system libraries were ported to Linux • Communication was implemented using a typical client server model (TCP/IP sockets) • Linux and VxWorks 5 network stacks are incompatible

11. LDAQ “design principles” • VXI crate LDAQ processes should be independent entities – fault tolerant • No “host” computer controlling task • Minimal code base • Homogenous and state of the art operating system environment • Open source preferred – transparency, quality & cost

12. What is LDAQ? • Threw out all of LOS except hand coded pieces that do VME communication • Moved VXI CPU’s from VxWorks to Linux • “Host” computer is Linux • Wrote a minimal application that integrates the VME calls – about 1 month of effort • LDAQ runs independently as a user application on each VXI crate

13. LDAQ visual description Local Network Computer LANL 20 Hz Start Stop MySQL (on host) LDAQ ROC module Ready Data Trigger TOF units

14. LDAQ benefits versus LOS • Code can be changed and recompiled in seconds using Eclipse (> 1/2h for LOS) • LDAQ failure on any VXI crate does not affect others – can be restarted • Remote cross-debugging using gdb successful • Linux is a true multi-user system (security) • Open source – no licensing costs

15. Single Event and Histogram • LDAQ does single event “out of the box” – temporary NFS limitations • Single event -> large amount of data, use only when useful • LOS histograms implemented in LDAQ • “Live data” transfer call implemented • Defined “time-focusing” histogram – e.g. live GSAS coming very soon!

16. Experience with LDAQ • Pharos has run LDAQ for most of this year – only failures were power cuts! • HIPD and more recently HIPPO are successfully using LDAQ with legacy scripting • LDAQ is a user application; changing SBC’s is straightforward (e.g. PPC -> Intel) – cost, effort • Restart takes seconds • Infrequent/development VME bus “hangs” now limiting reliability

17. Why is LDAQ successful? • Each VXI crate is an independent agent – does not know about others • No controlling task • Minimal features, implement bells and whistles externally (e.g. run timing) • Linux is robust and transparent • Keep it simple!

18. LDAQ development - observations • Exclusively using Linux - economical (up front and manpower) – cultural challenges • Does require significant Linux expertise • MySQL straightforward, robust & free • Agile development, develop features as needed – requires IS collaboration • Close collaboration between physical and computer scientists

19. LDAQ - future • LDAQ development is essentially done! • Documentation • Needs external setup & control scripts • Scripting – leave to IS, provide, or port existing control scripts? • Integration with slow controls – EPICS • Visualization toolkit • Real time analysis