1 / 21

PHENIX Raw Data utilities

PHENIX Raw Data utilities. This talk is the summary of many questions I answered over the past couple of weeks. I’d like to create something like a FAQ. What are the “event handling libraries”? How to I get at them, and who sets my account up in a way so I use the “best” distribution?

ray-ramsey
Download Presentation

PHENIX Raw Data utilities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PHENIX Raw Data utilities This talk is the summary of many questions I answered over the past couple of weeks. I’d like to create something like a FAQ. • What are the “event handling libraries”? • How to I get at them, and who sets my account up in a way so I use the “best” distribution? • Where is the manual? • What are the ready-made utilities? • What is a “generic” dump? • How can I sift through a data stream (a file, a DD stream,…)? • Who assigns those packet id’s? • What is the story about the “even 1000” id, 1000, 2000, …? • Example directories to start with • Where do I look up the APIs?

  2. What are the Event handling Libraries? • They are the collection of functions and C++ classes which allow you get at the data with minimum hassle. They contain • the lower-level rawFormat library (the bare-bones raw format internals used in the event builder and the DCM’s) • a C++ class library which provides easy-to-use classes for essentially all aspects of the data handling (such as Eventiterator, Event, oBuffer, etc), built on top of rawFormat • some stuff from the message library • All that is bundled in the libEvent.so shared lib for ROOT (the only library needed for data handling).

  3. How to I get at them, and who sets my account up in a way so I use the “best” distribution? • Around here, Dave et al/ONCS does. All this centers around the famous /opt/phenix/bin/phenix_setup.csh script. If that’s the only line in your .login, you will always get the installation which [we think] is the most accurate and most stable. • Outside, your sysadmin may do it, or if that’s you, come see me. • To temporarily use another distribution, do, in your .login: • setenv ONLINE_MAIN <the distribution top dir> • echo “Reminder: I use a non-standard distribution!” • source /opt/phenix/bin/phenix_setup.csh • You can also check out your own from CVS.

  4. Where are the docs? The DOC++-generated, most up-to-date documentation

  5. What are the ready-made utilities? dlist, ddump, and dpipe. dlist lists the packets contained in a given event. ddump dumps a given packet from an event out to your screen. dpipe copies (whole) events from any source to any destination. All utilities have many options so that you can use them in many different ways (especially dpipe). Let’s start with dpipe... $ dpipe -s <source type> -d <destination type> source_name destination_name

  6. ddEventiterator ddEventiterator DD Pool dpipe (1) Dual-CPU machine This is what we showed many times: distribute one datafile evenly for the analysis on a dual-CPU machine $ dpipe -s f -d dd /export/data1/dcm_data/run_00001.evt cw_test Source is a file destination is a dd pool And this is how you would get the data into the pool.

  7. dpipe (2) dpipe -s f -d f /export/data1/dcm_data/run_00001.evt xx.evt “pipe” from a file into a file? Isn’t that just “cp”? No -- what happens here is that the output file has the data in the native endianess of the copying machine. If you plan to analyze a file many times over, this saves a lot of time. dpipe -s f -d f -i /export/data1/dcm_data/run_00001.evt xx.evt “identify the Events one by one” -- handy to see what’s going on. -- Event 1 Run: 1 length: 8 frames: 0 type: 9 (Begin Run Event) -- Event 2 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 3 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 4 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 5 Run: 1 length: 8 frames: 0 type: 1 (Data Event) …lines deleted... -- Event 100 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 101 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 102 Run: 1 length: 8 frames: 0 type: 12 (End Run Event)

  8. dpipe (3) dpipe -s f -d n -i /export/data1/dcm_data/run_00001.evt “destination is nowhere/null/nothing” (with -i, just sift through the events in the file). -- Event 1 Run: 1 length: 8 frames: 0 type: 9 (Begin Run Event) -- Event 2 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 3 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 4 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 5 Run: 1 length: 8 frames: 0 type: 1 (Data Event) …lines deleted... -- Event 100 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 101 Run: 1 length: 8 frames: 0 type: 1 (Data Event) -- Event 102 Run: 1 length: 8 frames: 0 type: 12 (End Run Event) dpipe -s dd -d f cw_test xx.evt A simple data-logger: source is a dd pool, destination is a file. Write events from the pool to the file.

  9. more options -s <source type> can be dd, file, Teststream (d,f,T will suffice) -d <destination type> can be dd, file, nothing (d,f,n) -w <seconds> wait so many seconds between events (throttle data flow) -n <number> stop after so many events -v be a bit more verbose -i identify each event as it comes -h a few lines of help text

  10. dlist From the help text [dlist -h]: “dlist lists the packets contained in a given event. The event can come from any of the standard sources, DD pool, file, or test stream. The default is to get the next available event from a DD pool.” Example: $ dlist -f /export/data1/dcm_data/emc/emc_run_33333.evt Packet 8001 985 0 (Unformatted) 808 (IDPBSC_DCM32) Packet 8002 985 0 (Unformatted) 808 (IDPBSC_DCM32) Packet id length in DWORDS structure hitformat

  11. The “MLP test” Say I have just installed a new version of the “online_distribution” somewhere (as recently on Roy’s machines at Stony Brook). Does it work? “Test stream” $ dlist -T Packet 1001 26 0 (Unformatted) 30006 (ID4EVT) Packet 1002 16 0 (Unformatted) 30005 (ID2EVT) Packet 1003 10 0 (Unformatted) 30006 (ID4EVT) The output is predictable, because an event from a test stream (the “testEventiterator”) has a known composition, always. This command tests a large portion of the installation, because it uses a lot of the classes and functions found in the libraries, including those which assemble events (normally used in the event builder mostly).

  12. dlist options -f stream is a file -T stream is a Test stream -i identify the “picked” event in addition -e <event number> search for this event number -t <event type> search for event of this type (data, begin-run, etc) -h a few lines of help text Defaults: stream is DD pool, get first event of type “DATA” In the making: -n <number> repeat for so many events -c <number> get nth event (different from -e) -t BEGRUN accept mnemonics for event type (numerical now) Note: point -n, -c have been implemented.

  13. dlist Let’s look at a DCH file… $ dpipe -s f -d n -n 5 -i /export/data1/dcm_data/dc/dc_run_00011.evt -- Event 1 Run: 11 length: 8 frames: 0 type: 9 (Begin Run Event) -- Event 2 Run: 11 length: 11780 frames: 6 type: 1 (Data Event) -- Event 3 Run: 11 length: 11780 frames: 6 type: 1 (Data Event) -- Event 4 Run: 11 length: 11780 frames: 6 type: 1 (Data Event) -- Event 5 Run: 11 length: 11780 frames: 6 type: 1 (Data Event) $ dlist -f -i /export/data1/dcm_data/dc/dc_run_00011.evt -- Event 2 Run: 11 length: 11780 frames: 6 type: 1 (Data Event) Packet 3001 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3002 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3003 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3004 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3005 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3006 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3007 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3008 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3009 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3010 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3011 977 0 (Unformatted) 403 (IDDCH_DCM0) Packet 3012 977 0 (Unformatted) 403 (IDDCH_DCM0) You can see that we get the first event of type “data”.

  14. Tricks Do we really have 12 Drift chamber packets in our events? $ dlist -f /export/data1/dcm_data/dc/dc_run_00011.evt | grep IDDCH | wc -l 12 Yep. Also in the event number 15?: dlist -f -e 15 /export/data1/dcm_data/dc/dc_run_00011.evt | grep IDDCH | wc -l 12 How much data does the DCH produce?: $ dlist -f -e 15 /export/data1/dcm_data/dc/dc_run_00011.evt | grep IDDCH | awk '{print $3}' | perl -e 'while(<STDIN>){$i+=$_;} print $i,"\n";’ 11724 And what fraction of the event is this? (well, you can get carried away): $ dlist -i -f -e 15 /export/data1/dcm_data/dc/dc_run_00011.evt | perl -e 'split(/ +/,<STDIN>);$l=$_[7];while(<STDIN>) {if (/IDDCH/){split(); $i+=$_[2];}} print $i*100/$l," %\n";' 99.5246179966044 %

  15. “generic” packet dumps A “generic” dump is essentially a straight hex (or other base) dump of the packet’s contents. It doesn’t rely on the packet data’s integrity, and is for you if you like hex dumps. Very little in the way between you and the data that can be wrong. “generic” $ ddump -g -p 8003 -f /export/data1/dcm_data/emc/linearity_run_00013.evt | more Packet 8003 985 0 (Unformatted) 808 (IDPBSC_DCM32) 0 | ffff 4 1f6c 25 4 | 0 8e 26 23 8 | 29 1467 2111 3bd9 c | 4f79 5f84 146d 2121 10 | 3b71 4f74 5f7f 1444 14 | 2122 3d12 4f64 5f78 18 | 1467 2124 3ca8 4f60 1c | 5f7b 1452 2121 3b85 For “newer” packet formats, there is no specialized dump function available yet, and you will get the generic dump.

  16. ddump - “non-generic” (Emcal) ddump -p 8003 -f /export/data1/dcm_data/emc/linearity_run_00013.evt | more Packet 8003 985 0 (Unformatted) 808 (IDPBSC_DCM32) Start marker word : ffff Detector id: 4 Event number: 1f6c Module address: 25 Flag Word: 0 Beam Clock Counter: 8e AMU cell 1: 26 AMU cell 2: 23 AMU cell 3: 29 Userword 0 5a00 Userword 1 5a00 Userword 2 5a00 Userword 3 5a00 Userword 4 5a00 Userword 5 5a00 Userword 6 5a00 Userword 7 5a00 Long. Parity word ed06 0 | 1467 2111 3bd9 4f79 5f84 1 | 2111 3bd9 4f79 5f84 146d 2 | 3bd9 4f79 5f84 146d 2121

  17. ddump options (same as for dlist, + some) -f stream is a file -T stream is a Test stream -i identify the “picked” event in addition -e <event number> search for this event number -t <event type> search for event of this type (data, begin-run, etc) -h a few lines of help text -d generic decimal dump Defaults: stream is DD pool, get first event of type “DATA” In the making: -n <number> repeat for so many events -c <number> get nth event (different from -e) -t BEGRUN accept mnemonics for event type (numerical now) (not giving a packet id) -- dump whole event (remains to be seen) Note: point -n, -c have been implemented.

  18. Packet Id’s The formula is “detector number” * 1000 +1,2,3,4,5… e.g. for Emcal (number 8) 8001,8002, 8003, etc What’s the story about id n*1000? • The idea is that a given Id covers some (fixed) “footprint” on the detector, a number of channels, keystone, what not. This assignment id->”footprint” (mapping) is external to the PRDF, that’s in some database. • In principle that assignment is never to be changed, e.g. the upper-left corner of Emcal sector 1 is ALWAYS to be found in packet id, say, 8015. • How the DAQ stores the data inside a packet is completely orthogonal to the mapping. That can change at any time transparently. That’s how we do zero-suppressing, compressing, all that, and it could even change from event to event and from packet to packet. • This assumes that the data of each FEM ends up in a packet of its own. PRDF foresees the “collapsing” of packets, so that a DCM with 2 FEMs gives only one packet, that a DCB (the 5th DCM) collapses the packets of its 4 other DCMs, the Event builder rearranges, etc. • We reserve the id n*1000 to describe the “collapsing” scheme, so an analysis program would query that packet how to locate another detector channel in the data stream.

  19. …for example (simple-minded) channels Packet id today later 8001 0-143 0 - 575 8002 144-287 8003 288-431 8004 432-575 id 8000 0 143 8001 0 144 287 8002 0 288 431 8003 0 432 575 8004 0 ... id 8000 0 143 8001 0 144 287 8001 144 288 431 8001 288 432 575 8001 432 ... In this way we could direct the software to the packet which contains the channel.

  20. Advertisement: /examples and /rootexamples If you want to start a project, look in $EVT_MAIN/examples/ (compiled standalone examples) $EVT_MAIN/rootexamples (CINT macros) which get you started quickly. Among others: the examples from the manual reading from any of the input streams getting at channel values (iValue) getting all channel values at once (fillIntArray) getting at raw data much more… In the making: How do I write my CINT macros so that I can convert them into a shared lib with zero hassle? (Cint is ok to debug, but slow to run). Already in beta with EmCal(Henner/Stefan) paving the road.

  21. Where do I look up the API? Well, in the manual… look at the examples to see how it works. Some recurring things (p is a pointer to some packet, say, from Emcal): Stuff from the FEM header int event_number = p->iValue(0,"EVTNR"); int beamclock = p->iValue(0, "BCLK"); int beamclock = p->iValue(0, "MODULE"); int beamclock = p->iValue(0, "PARITY"); amu_pre = p->iValue(1,"AMU"); int rawArray[2000]; int nw; p->fillIntArray(rawArray,2000,&nw, "RAW"); //or p->fillIntArray(rawArray,2000,&nw, "DATA"); Get the raw packet data (RAW = with packet header, “DATA” just the data payload)

More Related