40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log  Not logged in ELOG logo
Entry  Wed May 22 22:50:13 2019, gautam, Update, VAC, Pumpdown prep 
    Reply  Thu May 23 15:30:56 2019, gautam, Update, VAC, Pumpdown underway - so far so good! pumpdownPres.png
       Reply  Fri May 24 11:47:15 2019, gautam, Update, VAC, IFO is almost at nominal vacuum pumpdown_20190524.png
          Reply  Mon May 27 11:37:13 2019, gautam, Update, VAC, c1vac is unresponsive 
             Reply  Tue May 28 09:51:33 2019, gautam, Update, VAC, c1vac hard-rebooted Screen_Shot_2019-05-31_at_12.44.54_PM.png
Message ID: 14641     Entry time: Tue May 28 09:51:33 2019     In reply to: 14640
Author: gautam 
Type: Update 
Category: VAC 
Subject: c1vac hard-rebooted 

The vacuum itself was fine - CC1 gauge reported a pressure of 1.3e-5 torr. Note to self: the C1:Vac-CC1_HORNET_PRESSURE channel, which is the analog readback of the Hornet gauge and which is hooked up to an Acromag ADC in the c1auxex chassis, is independent of the status of the c1vac machine, and so can serve as a diagnostic.

However, I was unable to interact with c1vac in any way, the monitor hooked up directly to it was showing a frozen display. So I hard-rebooted the system. It took a few minutes to come back online - but even after 10 minutes of waiting, still no display. In the process of the reboot, several valves were closed off - when the EPICS processes restart, there are momentary instances where the readback channels get an "undefined" value, which prompts the main interlock process to transition to a "SAFE" state. 

Running df -h, I saw that the /var partition was completely full. Maybe this was somehow interfering with the machine running smoothly? Two files in particular, daemon.log and daemon.log.1 were ~1GB each. The contents of these files seemed to be just the readbacks for the caget and caput commands. So I cleared both these files, and now the /var partition usage is only 26%. I also got the display back up and running on the physical monitor hooked up to the c1vac machine's VGA port. Let's see if this has improved the stability situation. The CPU load is still high (~6-7), with most of this coming from the modbus process. Why is this so high? c1susaux has more Acromag units but claims a much lower load of 0.71. Is the CPU of the c1vac machine somehow inferior?

In the meantime, I ssh-ed into c1vac and restored the "Vacuum normal" valve config. During this little escapade, the main volume pressure rose to ~6e-5 torr. It's coming back down smoothly.

Unrelated to this work: we had turned the RGA off for the vent, I powered it back on and re-initialized it this morning.

Attachment 1: Screen_Shot_2019-05-31_at_12.44.54_PM.png  1.035 MB  Uploaded Fri May 31 13:45:22 2019  | Hide | Hide all
ELOG V3.1.3-