Multi-threaded ROOT

1. Multi-threaded ROOT Once upon a time in a mail... On Thu, 13 Mar 1997 12:15:08 -0500 (EST) Martin Purschke wrote: Dear Rene, as you know, we -- that is Phenix for me now :-) -- are evaluating root's capabilities for our online monitoring. In this case you want to have the prompt (or some other user interface) active all the time, while the event loop or other process is running in the background. So without stopping that, you want to bring up different histograms on your canvas, or make fits, or whatever. I looked through the manuals and tried all the GUI examples, but they are single-threaded, that is, inactive until the currently executing macro or function terminates. Are there any multi-threaded capabilities in root, or is this just not available? Thanks for any hints. Cheers, Martin

2. Why multi-threading? • When you work offline (batch-style analysis, etc) you fill your histograms, and write them out when the data file is processed. You look at them later, fit them, analyze them, etc. Easy. • For online monitoring, you want to work with your histograms, display them at will, clear them, fit them, store them, whatever, while they are being filled in the background by your monitoring task. • At any time, you want your prompt (or a GUI) active so that you can give commands. • Until recently, ROOT couldn’t do that easily. You get the prompt back only after the processing has finished or has been suspended • Unless you built commands into your process loop to actually display histograms, you can see them only after processing has stopped or has been suspended (Timer). • “Stop processing” is something you don’t like too much in online monitoring -- you don’t want to miss events and miss the detector problem. • That’s where threads come in -- one thread fills, another waits for your command.

3. Now, how does it work? Compile a function into a shared lib. This function can be “forked off” as a thread. Both the “main thread” (your command prompt thread) and the background thread can manipulate the same objects [with some limitations]. Example: The background thread can fill histograms, at the prompt, you can work with your histograms, display them at will, clear them, fit them, store them, whatever -- you get the idea.

4. The pMonitor framework • Given that basic principle, we have made a framework. • It loads libraries and sets up root for threads • it provides you with a handful of commands to steer the analysis • at well-defined points, it calls your routines (which you provide) • at your command, it will start the analysis background thread until it terminates or you stop it. • From the command prompt, you get access to all histos, etc.

5. A minimal pMonitor • In its simplest form, you provide 2 functions (copy from template) • int pinit() -- initialize, get histo factory, book histos, open files, etc. • int process_event (Event * evt) -- process an event, which is delivered by the framework. • This is the top-level entry to your analysis -- you are free to call or make whatever function or whatever object is needed for the analysis. You can use nodes, the histo factory, the toolbox, all the goodies.

6. Example root [0] gSystem->Load("$PMON/lib/libpmonitor.so") root [1] ptestopen(); root [2] pstatus(); Not running Stream open: -- testEventiterator (standard) root [3] pstart(); Thread Main ID=1025 requested root [4] pstatus(); Running at Event 68 Stream open: -- testEventiterator (standard) root [5] 0 - load the library 1- open a test stream (testEventiterator) 2- get status (”Not running”) 3- start processing 4- get status (“running at event 68”)

7. User source code example #include <Event.h> #include <PhHistogramFactory.hh> #include <TH1.h> #include <time.h> #include <pmonitor.h> PhHistogramFactory *hf; int init_done = 0; PhH1 *h; // the one histogram int pinit() { if (init_done) return 1; init_done = 1; hf = PhHistogramFactory::instance(); if (hf) { h = hf->CreateH1F("H1", "Channel 0", 101,-49.5,49.5); return 0; } else return -1; } int process_event (Event * e) { // static int i = 0; Packet *p = e->getPacket(1003); if (p) { h->Fill (p->iValue(0)); delete p; } nsleep(100); // 100 milliseconds return 0; }

8. Standard pmonitor “commands” int pddopen (const char * ddname); int pfileopen (const char * filename); int ptestopen (); int pclose (); int pstatus (); int pstart (); int pstart (const int nevents); int pstop (); int pidentify (); int pidentify (const int n); int pclearidentify (); int pcontrol (); int pcontrol (const int seconds); open a dd pool as input open a file as input open a test stream close the stream print status information start processing start, but stop after n events stop processing have events id themselves have the next n events id themselves clear the “id” flag, no more id bring up status GUI (doesn’t work yet) update GUI each seconds (ditto)

9. How complicated is it? Not very. We have recipes, tools, and templates for... • running your code as a CINT macro to debug, compile the unmodified code as a shared lib - mandatory for threading • Generating the Linkdef files for rootcint automatically (you only need to give some “hints”) • Actually running the thing as a thread • Adding custom commands to your project (e.g., clear all histograms) • The Templates give you all that, the histogram factory, the toolbox, the PhNodes if you need them, ...

10. Added comfort: Linkdef files on the fly Manually writing those linkdef files for rootcint is a PITA. With a new utility “gencintlinkdef.pl”, they actually become Makefile targets. The idea is that you tag functions and classes which you want to be accessible from CINT (typically, only very few of many), with a comment (“//++CINT”) which the perl script can search for. [purschke@phoncsa]$ gencintlinkdef.pl dummy.h #ifdef __CINT__ #pragma link off all globals; #pragma link off all classes; #pragma link off all functions; #pragma link C++ class rootcintclass; #pragma link C++ function cint_function; #endif /* __CINT__ */[ // only to show the recipe class rootcintclass { //++CINT make it known to cint public: int i; }; class internal_class { //this one not public: int i; }; int cint_function (const int ii); //++CINT this one will be known int function (const int ii); // this one not. $(LINKFILE): $(HDRFILES) $(CINTGEN) $(HDRFILES) > $@

11. Limitations Not everything works with threads yet. The biggest limitation is that only one thread can talk to X windows. A background thread cannot bring up any displays so far (that’s a serious constraint) Some thread-unsafe stuff will need mutex (mutual exclusion) locks -- we believe we know what, but I’m sure we’ll find more places. Some stuff simply doesn’t work yet - you can’t do thread->Stop() without a crash yet, for example Ctrl-C in a runaway macro will kill the whole thing - hmm. Careful with macros.

12. Whishlist I believe “compiled with thread support” is the default - make sure it stays this way Link root exe with -lpthread already (so you don’t need to link all your libs with it) Mutexed/thread-safe global objects - this might go deep into the guts of root clean up some printout clutter complete the doc’s (some classes are still missing) • For threads-in-ROOT in general (not necessarily the RDT): • get all the thread functionality right (thread-stop()) • find all places that need locks, or make them thread-safe (often hard to do) • find a way to let threads bring up X displays • improve the robustness

13. The cast Marc Hemberger, GSI -- did all the work, pestered Rene, forced RDT to add changes to root distribution, made latest changes available to me Hans Essel, GSI -- got GSI to be a real player in the ROOT arena, adds political muscle to all that Victor Perevoztchikov, BNL -- implemented first go at threads way back when, helped Marc and me to sort out things MLP -- field-tested, changed stuff, tried things, fixed bugs, claims victory On a personal note, this is -- after having been a GSI employee from 1991-1996 -- the first collaboration with someone from the GSI computing folks ever.