35 Ton Run 2 Hardware and Operational experiences

1. 35 Ton Run 2 Hardware and Operational experiences Alan Hahn Dune Friday Phone Meeting

2. Organization • Overall Run • Cryo • Detector “Hardware” Categories • TPC • Including HV • Cold Electronics • Photon Detectors • DAQ • Systems Dune Friday Phone Meeting

3. Run: 35T Cooldown and Filling Membrane Temperature sensors Top-Off from TRAILER XFer from LAPD GAr xfer from LAPD (over weekend) ~1.5 tons LAPD Pump trip Dune Friday Phone Meeting

4. Run: e- Lifetime PrM2 & 3, both long PrMs midway heights in Cryostat(these lifetimes may drop a bit (~10%) using PrM0 as a constraint) Bad News Period 3 ms Sitewide Power Outage Pump Start Tubing Break LN2 Cooling Loss Dune Friday Phone Meeting

5. Run: Purity Stratification near start of Purification PrM 1 (near Cryostat bottom) PrM 4 (near surface) O2 Gas Analyzer, ~(~80 cm from bottom) Dune Friday Phone Meeting

6. Geometry of 35 Ton Cryostat(from E.Voirin recent 35 Ton FEA calculations Dune DocDb 1156) Pump outflow 4 purity monitors in this corner (Geometry Not in CFD model) PrM1 Pump intake Dune Friday Phone Meeting

7. New constraint Discussion of Results / Lessons LearnedPhase separator cools inlet flow to approach saturation temperature, which is below the “bulk temperature” of the argon. This cooler, more dense layer of argon tends to stay at the bottom, and not mix well with the rest of the argon. Back pressure control at the phase separator could allow us to flow in warmer argon, which would then mix well with the bulk. Pump Discharge Pipe with Phase Separator on Top, LAr flows down, and impinges on cryostat floor. Pump Suction RTD measurements previous slide verifies this temperature stratification Dune Friday Phone Meeting

8. Run: Stratification ExplanationPrM1&4 with 3 Precision RTDs Opportunity provided by Power Outage Pump is off Pump is on 3 RTD offsets made to match Barometric Pressure Uppermost RTD 12 hrs after Outage PrM 1&4 show same lifetime PrM 1&4 lifetimes diverge Lowest RTD Dune Friday Phone Meeting

9. Run: Beware the Ides of March • Three consecutive “events” ~one week apart and all on weekends • Site-wide power outage stopped our recirculation and dropped the LAr Purity (3/4—Friday Night) • See the loss of purity due to outage on previous slides. • Loss of LN2 Cooling (broken valve) caused a loss of LAr (~1/2 ton through boil off) (3/12--Saturday) • Amount of loss because Autodialer Alarm had not been reset since the previous week’s Power Outage—Doh! • Only noticed on Sunday and fixed in afternoon • Dropped LAr level below desired HV-precaution level • Unrecoverable Contamination of LAr (3/17-Saturday 2 AM) • Small tube attached to GAr Bellows Pump that was being used to transfer GAr between LAPD and 35Ton broke- pulling in air and injecting it into the LAr stream. • Estimate it would take ~20 weeks of filtering/filter regeneration to remove just the O2, N2 would not be removed. • Went from 100 ppt to 30 ppm O2 in about 3/4 hr. Dune Friday Phone Meeting

10. This broken tubing is on the suction side of pump that was pulling GAr out of LAPD, compressing it and injecting it into the LAr coming from 35T just before the purification filters. This was how we were raising the LAr level in the Cryostat Bellows Pump Dune Friday Phone Meeting

11. Cryo Issues • Pumping • Had problems establishing the 35Ton Internal LAr Pumping • Pumps had been rebuilt after Phase 1, but had never been run in. • Finally putting 3 phase 208 AC on pump motor broke it free. • Then normal AC Drive worked fine. • Lesson learned—nothing goes into cryostat without a functionality test • Even if it worked fine last time (e.g. Phase 1). • Need a better interface into Controls and Alarms • Autodialer not working cost us LAr level when LN2 valve stopped functioning. • Did it help lead us a bit quicker into the Tubing break?? (Speculative) • Painful to use the Cryo Interface at time • incredible slow downs. • Discourages one from looking at trends if it takes forever to get up and run. • Massive Contamination of LAr • I believe there will be a review of exactly what happened and that will suggest several lessons to be learned from • How did we get into this situation in the first place? • Recognition of how fast you can completely contaminate a sizeable amount of LAr • How to identify other similar modes that might cause same problem. • Perhaps a way to quickly and automatically respond to an event since there is almost no time for people to recognize what is going on. Dune Friday Phone Meeting

12. TPC • Detector Installation in Cryostat was a well planned and effectively carried out enterprise • Thanks Jack Fowler, Dan Wenman, Bo Yu, Lee Scott, Bob Kubinski • Demonstrates that careful planning actually works! • Field Cage, Cathode, HV • 60 kV (@Cathode) in “clean” LAr • Wanted to get track measurements made at this lower field before “risking” raising the HV higher and potentially breaking something. • Did see small spikes, and did have two apparent breakdowns (one while ramping back up after first breakdown). • These breakdowns happened just ~12 hrsbefore the LAr contamination • Large disappointment with LAr contamination is that we weren’t able to test Cathode HV at 120 kV in pure LAr • However we did continue the tests in what we had • Held 120 kV for 1 week without breakdown • Held 135 kV for 1 day with breakdown. • A summary of the HV studies will be made. • Most info is in elog Dune Friday Phone Meeting

13. HV History over Run 150kV 150kV These are Voltages at Power Supply , ~10% higher than at cathode. 135kV 133kV breakdowns 67kV PowerOutage Cooling Loss contamination Dune Friday Phone Meeting

14. Cold Electronics • As is well known, the FEMBs + the APAs have been plagued by relatively high noise levels • These swamp the original specs that were based on thermal noise figures, as measured with capacitive loads • Clearly see noise influences that correlate to the individual wire planes Dune Friday Phone Meeting

15. Recent data from Cold Run17366 3/16) Test done by Bo Y. and Matt W. at BNL with 3 Bd APA. DocDB 9439, 7/2014 Data from 4 Bd taken at DAB (7/15) Dune Friday Phone Meeting

16. Noise (Continued) • Our “low noise” state includes this correlated noise. • In addition, we are occasionally/often triggered into an extremely high noise state that makes it impossible to use the data for anything. • Much work is being done now by Marvin Johnson, Brian Kirby, Linda Bagby, Steve Chappa… in trying to characterize this noise • Hope that understanding can be fed into the electronics/APA design/??? for ProtoDune Dune Friday Phone Meeting

17. Online Event Display -Collection PlaneTop view of APA Very High Noise State “Normal” Noise State Long drift APA Short drift Dune Friday Phone Meeting

18. Photon Detectors • Detectors • One Detector left unplugged during cabling • Observed Low noise level for those channels wasn’t recognized as the serious problem it was. • PD Signals seem to be impacted by noise—not positive if this is noise being generated by the APAs • SSPs • After fixing an issue with the NOvA timing chain (back at DAB), these have been reliable • We have had very good response from ANL when we have had any problems • Calibration Module • Working fine, able to calibrate the PDs Dune Friday Phone Meeting

19. DAQ • Flange Board • All signals/electrical services to APA/PD pass through this Board. • Had serious issues with initial version • Second version much better • Issue with Bias lines—leakage currents quite high, and very non-linear with voltage • Eventually we bypassed the FB and went directly to SHV bulkhead connectors on own flanges. • PennTrigger Board • Slow start, but eventually it wsa providing our triggers • Cosmic Ray Counters • Allowed us to trigger on tracks that are useful for looking at the APA features. • Fiber Links • Very problematic at DAB and early at PC4. Often had to “reseat” the optical transceivers, or reset the links. • Settled down at some point and were less of a problem for rest of run. • RCE systems • Once things calmed down, they appear to be working fine. • Spares were an issue, since a failure would take down ½ the TPC APAs Dune Friday Phone Meeting

20. DAQ (continued) • NEMA 12 Rack • Turned out to be one of the highest risk items for the 35 T run (until Tubing at least). • Contains DAQ Servers, RCE Systems, Network Switches in a closed Air Condition Rack • The power for the Air Conditioning was put on an interlock that was tied into the Rack Smoke Detection Module and was not plugged into the rack power. Also an internal power switch for Air Conditioning could be switched off. • Three“near miss” incidents • Two were early in the DAQ installation at PC4 where the air conditioner could or actually was disabled, with the rack doors closed. However these were caught before anything happened. • 3rd event was much more advanced when it was caught—rack temperatures were in > 60C range as reported by Network switch in rack • Fortunately M. Convery came by PC4 from ROC-W (@7PM because he noticed the RCE readout wasn’t working at ROC-W.He opened the doors, things cooled off. • After this last episode, we established • a management control signage to be used whenever rack was opened to be worked on. • Added an external light that was lit as long as power was available to the AC. • Recently a 50C klixon (thermal switch) was added that will drop rack power in case the Air Conditioning fails. • Recommendation that one needs to do a careful risk analysis on a system like this. • Operation • For better or worse, the DAQ servers were not setup as Fermi Standard, so we had several cases where we were halted because the experts were not available to get the systems running. • DAQ Running. • Due to event size we were typically limited to event rates ~1 Hz • Reliability had vastly improved near the end of the run. Dune Friday Phone Meeting

21. Systems—or how things play together • Grounding • The 35 T cryostat was never designed with grounding in mind. • There is no way that the internal membrane could be kept isolated from contact with the concrete structure that encased the Cryostat. • Reason was that a carbon steel vapor barrier was bolted to the concrete, and that was then welded to the A&B plates (which were welded to the stainless steel membrane). • Due to this inherent connection, it was never obvious to those in PC4 as to what the impedance between detector and building grounds should be, since the concrete impedance was sensitive to moisture, and that varied a lot from the installation to the running period. • Then when we went to 35T installation, our grounding crew disappeared to uBoone. • Rest of us didn’t really know what to do but to turn down the “honking horn” on the ground fault device, since it could change on/off with nothing being done. Dune Friday Phone Meeting

22. Inside the Concrete Shell—note the iron Carbon Steel Vapor barrier Dune Friday Phone Meeting

23. Systems—or how things play together • System Integration • Desperately need a real low noise Test Bed setup with a reasonable APA ---before one has committed to a final FEMB design. • These things can’t be thought of only in isolation, they impact each other • Need Low noise Setup so one can beat the noise down knowing it should be lower, rather than blaming external factors as causing the noise. • Need engineers during this period of time to see how their equipment is working (or not). • Installation is too late to be doing this debugging. • It is important that Groups that are producing parts actually participate in the debugging, installation, and running of their products. • We had too many MIA groups for the 35 T. Dune Friday Phone Meeting

24. Xtra slides Dune Friday Phone Meeting

25. Loss of N2 Cooling 3/12/16 RTD on flat of top Membrane. LAr Level Drop ~ 3.8 cm ~1/2 ton Cryostat Pressure RTD on tip of Corrugation Dune Friday Phone Meeting

26. LAr Level Concern “Corrugations” APA’s Cathode (up to 135 kV) Dune Friday Phone Meeting

27. Tubing Break--3/19 Dune Friday Phone Meeting