40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
 40m Log, Page 242 of 339 Not logged in
ID Date Author Type Category Subject
7712   Wed Nov 14 13:58:18 2012 JenneUpdateIOOTurn off MCL path before doing MC spot measurement

Quote:

 Quote: We have discovered that the MCL loop squishes the length fluctuations that result from the MC spot measurement angular dither.  This is good, in that the MCL is doing what it ought to do.  However, we need to turn it off before doing a spot measurement.

This is totally non-sensical statement, of course.

We might also say that the DARM loop totally squishes the GW signal and so its better not to have any feedback in the interferometer.

Hmmm, indeed.  To keep MC_L on, we should be looking at the control signal rather than the error signal.  I think Den has MC_L running nicely such that it always comes on when the MC locks, so I can switch it.

8617   Wed May 22 15:48:56 2013 JamieUpdateSUSTurn off zero padding in DAC outputs

After the results of the analysis in the #8598 thread, I have added the "no_zero_pad=1" flag to the cdsParameters block of all SUS models:

• c1mcs
• c1sus
• c1scx
• c1scy

The upsampling to the 64 kHz DAC output will now be done with sample-holds, instead of zero-pads.  This should reduce the 32 kHz lines we were noticing in the analog DAC output.

I note, though, that Brian Lantz points out that this might actually introduce a delay of about a half sample.  We will continue to investigate.

In any event, I have rebuilt and installed all models listed above.  I will restart as soon as opportunity allows.

8625   Thu May 23 10:20:33 2013 JamieUpdateSUSTurn off zero padding in DAC outputs

 Quote: After the results of the analysis in the #8598 thread, I have added the "no_zero_pad=1" flag to the cdsParameters block of all SUS models: c1mcs c1sus c1scx c1scy The upsampling to the 64 kHz DAC output will now be done with sample-holds, instead of zero-pads.  This should reduce the 32 kHz lines we were noticing in the analog DAC output. I note, though, that Brian Lantz points out that this might actually introduce a delay of about a half sample.  We will continue to investigate. In any event, I have rebuilt and installed all models listed above.  I will restart as soon as opportunity allows.

I have restarted all the suspension models with the new no_zero_pad flag for the DAC upsampling.  Everything came up fine and all optics are damped as expected (except for concerns about c1scy which I'll note in a followup).

8426   Tue Apr 9 00:32:39 2013 ranaConfigurationIOOTurn on MCL

Listening to the free swinging Y-arm, its clear that the fringe velocity is higher with the MCL path off, than on. I checked that the default gain of -300 was too high and changed it to -100 in the mcup script. With the higher gain value, there was clear gain peaking at just under 60 Hz (where the 60 Hz comb filter starts). We basically want the UGF to be between 20 and 60 Hz so that we can have the Bounce mode RG at 16.25 Hz and also the notch filter at 60 Hz.

Den is measuring and setting the UGF to be ~45 Hz.

With the MCL path on there is a high frequency horn-like noise in the Yarm when it locks. Its reduced a little by reducing the loop gain, but clearly this is just noise introduced by the MCL loop as Den noted before (and also tonight). He is cleaning up the whole MCL MEDM situation so that its useable by us. At the moment I've re-enabled it in the mcup.

My belief is that the frequency noise from the unstabilized MC is making the PRC locking harder. This will be investigated by tuning the shape of the MCL/MCF crossover so that we can turn it on without ruining the arm cavity spectra. Since the PRC length is ~2x smaller than the MC, we would expect it to be less sensitive to the MC frequency noise. But, since there is some common mode rejection in there, this may not be true. We'll only know by measuring PRC control signal with MCL on/off.

8434   Wed Apr 10 03:59:41 2013 DenConfigurationIOOTurn on MCL

 Quote: My belief is that the frequency noise from the unstabilized MC is making the PRC locking harder. This will be investigated by tuning the shape of the MCL/MCF crossover so that we can turn it on without ruining the arm cavity spectra. Since the PRC length is ~2x smaller than the MC, we would expect it to be less sensitive to the MC frequency noise. But, since there is some common mode rejection in there, this may not be true. We'll only know by measuring PRC control signal with MCL on/off.

I think if we make MCL UGF higher then 20 Hz, arm cavity spectra will feel it. It might be possible to use a combination of feedback and feedforward control from ground seismometers. I made MCL UGF at 3 Hz to reduce 1 Hz motion of the pendulum; feedforward OAF subtracted the stack at 3.3 Hz. Once OAF converged, I blocked adaptation and the filter became static FIR. MC length RMS was reduced by a factor of 10 and arm cavity spectra was not affected at frequencies >20 and became better at low frequencies. We'll see if this enough.

On the attached plot red color shows MC_F with MC_L OFF, blue - MC_L is ON, green - MC_L and OAF are ON.

Then I locked PRCL (using AS_Q and REFL55_I) to carrier and aligned the cavity. Power RIN was 50-70% and 00 beam on the POP camera was moving significantly. BS oplev was shaking the optics at 5 Hz. I fixed it, but there should be something else as RIN was still high.

Attachment 1: MCL.pdf
4209   Wed Jan 26 14:49:48 2011 AidanUpdateEnvironmentTurned on Control room AC

80 degrees is too uncomfortable in the control room so I turned on the AC. The set point is 74F.

6905   Mon Jul 2 23:08:38 2012 YaakovUpdateSTACISTurning on STACIS

This past Friday I swapped out a burnt resistor on the spare STACIS unit I'm working with and powered it up. Here's the setup:

And here's what happened:

X an Y directions: When I switched from open to closed loop (making the internal geophones provide feedback), the STACIS started making a loud noise- it seemed like it was oscillating uncontrollably.

Z direction: The same thing happened in z until I added some weight to the top of the STACIS- then it quieted down, and seemed to work okay. The geophone signal dropped considerably compared to the open loop signal, which is expected if the feedback is working.

Then I tried driving the PZTs with a signal from the SR785 network analyzer. With an amplitude of tens of mV and frequencies from around .1 to 200 Hz, I could see the accelerometers I mounted on top of the STACIS definitely register motion, which means I was successfully driving the PZTs.

Below are transfer functions of the STACIS as I drove the PZTs from .1 to 100 Hz at 10 mV. The top graph is open loop, the second is closed loop. These were measured with the internal geophones.

In the bottom graph, "A" is closed loop and "B" is open loop, where the transfer functions were taken with the accelerometers instead of the geophones.

Attachment 2: geo_closed.GIF
4817   Tue Jun 14 16:38:31 2011 JamieUpdatePSLTweaked input pointing to PMC

The PMC trans power was a little low (0.77V or so).  I tweaked up the input pointing and now we're getting about 0.875V transmitted.

12213   Thu Jun 23 17:24:32 2016 ericqUpdateGeneralTweaks

I spent some time this afternoon reviving some of my CESAR/ESCOBAR shenanigans on the Y arm. I found it neccesary to adjust a few things.

• PMC realigned
• ETMY oplev centered
• Y End green realigned
• PSL/Y Green beat realigned

Afterwards, ALSY noise levels were good.

14170   Mon Aug 20 14:04:53 2018 johannesBureaucracyEquipment loanTwo C30642G PDs removed

EDIT: After discussing with Koji and checking the existing M2ISS PDs I put the two C30642G back and took two C30665GH (active diameter: 3mm) diodes. Only one of this type remains in storage.

I removed two C30642G photodiodes from the stash for the new M2ISS hardware and updated the wiki page accordingly.

1866   Fri Aug 7 20:43:35 2009 Clara, Jenne, Rana, JanUpdatePEMTwo Guralps plugged in, prepped for huddle test

Both Guralps and the Ranger have been placed in our nice new insulated foam box, complete with packing peanuts, in the corner between the x and y arms. The Guralp breakout box has been reinstalled and everything is plugged in in prepartion for the huddle test. However, we're having some issues with ADC channels, which will be worked out tomorrow (hopefully) so that data can be collected over the weekend.

Currently, one Guralp is plugged into the three SEIS-MC1 channels. We made new channels for the second Guralp (GUR-EW, GUR-NS, and GUR-VERT), but had issues with those. So, EW and NS have been plugged into PEM_AUDIO-MIC1 and MIC2 for the time being.

Attachment 1: Untitled.png
1868   Sat Aug 8 17:19:07 2009 ranaUpdatePEMTwo Guralps plugged in, prepped for huddle test

I found that several of the cables are unlabeled so I'm not sure what's plugged in. In the end, I found that the TEMP_2, _3, & _44 channels were working and so I plugged in anything that looked seismic into there.

TEMP_2 is now apparently the X channel of the 2nd Guralp. If someone can figure out which cable belongs to the Y channel, please plug it into TEMP_3 and then we can fix the channel names.

I also removed (gently) all of the accelerometers from MC2's chamber. This didn't break the lock, but I intentionally broke it to make sure it reacquired fine. It did and the MC TRANS QPD showed no significant shift afterwards.

Attachment 1: Untitled.png
482   Fri May 16 14:38:50 2008 josephbSummaryCamerasTwo cameras setup
I've changed the pickoff setup from yesterday for the GigE cameras to include a 33% beam splitter (first one I could find). The reflection is going to the GC650 (CCD camera) while the transimission is going to the GC750 CMOS camera. This means the CMOS camera has roughly twice the light incident as the GC650 and should be kept in mind in all comparisons. The distances from the beam splitter are approximately the same both cameras, but some more accurate positioning might be useful.

Its very easy to get the GC650 camera into a bad state where you need to go out and cycle the power (simply unplug and re-plug in the power supply either at the camera or outlet). If the ListCamera program doesn't see it, this is probably necessary.

Andrey added at 6.30PM: Actually the 650 camera keeps crashing constantly. Every time I attempt to capture an image, the camera fails.
7166   Mon Aug 13 21:47:30 2012 YaakovUpdateSTACISTwo changes to STACIS noise budget

In eLog 7148 (http://nodus.ligo.caltech.edu:8080/40m/7148), Koji pointed out that the op-amp and SR560 noise values (which I took from specs and then multiplied by the geophone calibration factor to get m/s/rtHz) were waaay too low. My error was an extra multiplication factor in the plotting script.

The other change was recalculating the ADC noise by splitting a signal into two ADC channels and subtracting the time series (then taking the PSD and converting to m/s/rtHz). It compares well to the value I got by terminating the ADC channels, which was the ADC noise line in my last eLog.

Both these changes are included in the below plot:

Attachment 1: noise_budget_8-13.bmp
424   Thu Apr 17 20:17:37 2008 AndreyUpdatePEMTwo issues with our weather station

I encountered two difficulties working with the "Weather Station".

(1) It turns out that there is no indication for "outside humidity" on the "weather monitor" (a small black box located on the north wall of the interferometer). I realized that "outside humidity" is absent in our system when I tried to see the Dataviewer trend and real-time value from the channel "C1: PEM-weather-outsideHumid". It shows impossible number 128%.

It follows from the "Davis" technical documentation that the outside sensor can be of two types: either "External Temperature Sensor" or "External Temperature/Humidity Sensor". I suspect (I do not know for sure) that we have the first type of sensor "external temperature only" and therefore we in principle cannot have information about outside humidity. I propose to Steve to climb to the roof on Friday to resolve this uncertainty looking at the sensor.

(2) I wanted to change the units of pressure from "Pascal" (force/area) to other units, "mbar" for example. For this purpose I need to edit the file "Weather.st" in the directory /cvs/cds/caltech/target/c1pem1 (this file is run on the VME processor "c1pem1"). Unfortunately, when I try to open the file with emacs, I get the message that the file exists but protected from modifications. I do not know how to unblock the file "Weather.st". I need some help with that.

I thought that switching-off the processor "c1pem1" could resolve the issue, so I switched-off the whole crate where the processor "c1pem1" is installed for about 5 minutes, turning the metallic key. As it did not make any difference for the accessibility of the file "Weather.st", I switched-on the crate after 5 minutes. There are other processors besides "c1pem1", so they were turned-off for several minutes earlier today.

Also, I created a new MEDM screen which has information about weather only, a smaller version of the "C0Checklist.adl" MEDM screen. Both screens are now located under the most top-left button "Checklist" of the main MEDM screen.
915   Wed Sep 3 18:43:19 2008 YoichiConfigurationElectronicsTwo more active probes
I found two active probes, an HP41800A and a Tektronix P6201.
Thanks John for telling me you saw them before.
Now we have three active probes, wow !
We have to find or make a power supply for P6201.
The manual of the probe is attached.
Attachment 1: Tektronix-P6201-active-probe.pdf
1523   Sun Apr 26 02:13:18 2009 YoichiUpdateLockingTwo more successes of RF CARM handoff
Tonight, the RF CARM hand off (mostly) succeeded twice.
But still, the IFO lost lock when I reduced the REFL_DC gain in the AO path to zero.

At the beginning of tonight's work, MICH DD hand off failed several times. This was because the the PD9 gains were set to zero.
I found that the offset script, which I called before starting the locking, fails to restore the gain values sometimes.
This happens when ezcaread fails to read the current gain. We have to be careful when running the LSCoffsets script.
501   Wed May 28 12:51:32 2008 josephbConfigurationComputersTwo more switches mounted
Two more Prosafe 24 port switches have been mounted in the racks, one in 1Y9 and one in 1Y6. (The first one was placed in 1X4).

The one in 1Y9 has been set to an IP address of 131.215.113.251, while the one in 1Y6 is set to 131.215.113.252, and these have been labeled as such.
10861   Wed Jan 7 02:56:15 2015 diegoUpdateLSCUGF Servo for DARM

[Jenne, Diego]

Today we began implementing the UGF Servos. Things we did:

• we updated the LSC model with both DARM and CARM servos, and moved them from after the control system to before it, at the level of the error signal;
• we updated the medm screens; new buttons are located in the main LSC screen;
• we started commissioning the DARM servo, at first using DARM for the lock of the single Y arm, then we moved on to the PRFPMI lock and the usual transition from ALS to Transmission;
• although we had several lock losses during the night, we managed to tweak the parameters of the DARM UGF servo (phases, excitation, gains), which now seems to work sufficiently fine;
• the filters added to the I and Q filter banks are a single lowpass in each, while the only filter in the main servio is a standard integrator;
• we don't have a step response at the moment, but we can say that the settling time of the servo is in the range of 10 seconds;
• we updated the ALSdown.py and ALSwatch.py scripts with a call to a new UGFdown.py script; this script, located in the scripts/PRFPMI folder, takes care of disabling the servos and putting the excitation to zero in case of a lock loss; re-enablement of such things must be done manually;
10875   Thu Jan 8 02:52:09 2015 ericqUpdateLSCUGF Servo for LSC

I added UGF servos for the DRMI DoFs, after creating a library block for the servos. I also deleted the FMs before the phase rotation, since we can just do it afterwards in other existing FMs. I've only added the MICH and PRCL buttons to the LSC screen because in the end, I feel like a dropdown is better, but I just wanted to get it running quickly tonight. The LSC model and the UGF block have been committed to the svn.

We were able to use the PRCL UGF servo successfully, as Jenne was exploring MICH offset space.

10901   Wed Jan 14 19:27:09 2015 ericqUpdateLSCUGF servo now linear again

The UGF servos have been moved to the control point, are are once again totally linear!

10902   Thu Jan 15 03:18:11 2015 diegoUpdateLSCUGF servo now linear again

The UGF servos were recommissioned today:

• suitable values of frequency, excitation, phases and gain were found;
• the phases were chosen in order to maximize the I signal and suppress the Q one;
• the servos seemed sufficiently stable when in a quiet state, but they didn't performed well in other cases;
• I also found out that DARM & CARM and MICH & PRCL are maybe too much coupled, but that could be actually due to the main loops rather than the UGF ones;
• however, after some weird rampings with no apparent reasons, and after some quite bad and glitchy step responses, I found out that the effect of the chosen phases vanished: the I and Q signals were of the same order of magnitude again, probably causing the bad performance;
• Jenne and I tried to increase che SINGAINs and COSGAINs (but keeping them equal to each other): this has the good effect of separating more the I and Q signals, but it's just a zoom effect: there still are mixing effects and, more important, some zero-crossings into negative values that cause the signal going into the servo to go crazy.

Our idea is that we need with some thinking about these servos and most of all try to figure out all this phase thing before we can start to trust the servos to be used for locking.

10911   Fri Jan 16 04:14:05 2015 diegoUpdateLSCUGF servo now linear again

UGF Servos' commissioning still going on, updates of today:

• on Rana's suggestion, we don't use anymore the Q-signal rejection at the level of Phase 1 and Phase 2; instead, a proper complex division is made between those two signals (with a check in case of zero); then the resulting signal is demodulated with a new Phase 3, which is the one used to select the I signal while zeroing the Q one; changes to the model and the screens have been made;
• a new evaluation of all the parameters for the four servos has been made; aside for the new phase, and the zeroing of the other phases (because they are not used anymore for the selection), the parameters are not dissimilar from the prevoius ones
• the PRCL and MICH servos seem sufficiently stable;
• CARM and DARM are stable only for a short amount of time; what usually happens is that one of the two starts drifting in one random direction, and usually the other one follows shortly after; it is not clear if there is a relation or if they stop being stable after a similar amount of time; I still noticed a few lowest limits appearing in the input signal, which should be avoided; I'll check the model again tomorrow;
• the weird thing about CARM and DARM is that at the same time when one of them starts drifting, its I and Q signals begin to be comparable; when the servo is shut off, they resume their normal state;
• an increase in the excitation gain improves the separation of I and Q and also reduces their variations, but a high peak in the loop due to this might not be a good idea.

10916   Fri Jan 16 20:37:52 2015 diegoUpdateLSCUGF servo now linear again

I found an error in the model of the UGF servos, I have now corrected it; for future reference, now the division between TEST2 and TEST1 is properly done with complex math: given

$TEST1 = a + i b\hspace{.5cm},\hspace{.5cm}}TEST2 = c + id$

we have that TEST3:

$TEST3 = \frac{TEST2}{TEST1} = \left(\frac{ac+bd}{a^2+b^2}\right) + \left(\frac{ad-bc}{a^2+b^2} \right)i$

TEST3 is the actual signal that is now phase rotated to select only the I signal while rejecting the Q one.

All the updates to the model, the screens and the script have been SVNed.

5645   Mon Oct 10 16:32:18 2011 steveUpdateSUSUL sensor of ETMY is recovered

I lost UL osem voltage this morning when I was checking the actual connection at rack ETMY

This after noon I disconnected  the 64 pins IDE connector from satelite amp at the rack, and the two 25 pins Dsubs at this juction board.

UL OSEM recovered after reconnecting these three connectors.

Attachment 1: ETMY_UL.png
Attachment 2: ETMY_OSEM_UL.png
Attachment 3: noisyETMY_UL.png
2692   Mon Mar 22 02:03:57 2010 ranaSummaryElectronicsUPDH Box #17: Ready

It took too long to get this box ready for action. I implemented all of the changes that I made on the previous one (#1437). In addition, since this one is to be used for phase locking, I also made it have a ~flat transfer function. With the Boost ON, the TF magnitude will go up like 1/f below ~1 kHz.

The main trouble that I had was with the -12V regulator. The output noise level was ~500 nV/rHz, but there was a large oscillation at its output at ~65 kHz. This was showing up in the output noise spectrum of U1 (the first op-amp after the mixer). Since the PSRR of the OP27 is only ~40 dB at such a high frequency, it is not strange to see the power supply noise showing up (the input referred noise of the OP27 is 3.5 nV/rHz, so any PS noise above ~350 nV/rHz becomes relavent).

I was able to tame this by putting a 10 uF tantalum cap on the output of the regulator. However, when I replaced the regulator with a LM7912 from the blue box, it showed an output noise that went up like 1/f below 50 kHz !! I replaced it a couple more times with no benefit. It seems that something on the board must now be damaged. I checked another of the UPDH boxes, and it has the same high frequency oscillation but not so much excess voltage noise. I found that removing the protection diode on the output of the regulator decreased the noise by a factor of ~2. I also tried replacing all of the 1 uF caps that are around the regulator. No luck.

Both of the +12 V regulators seem fine: normal noise levels of ~200 nV/rHz and no oscillations.

Its clear that the regulator is not functioning well and my only guess is that its a layout issue on the board or else there's a busted component somewhere that I can't find. In any case, it seems to be functioning now and can be used for the phase locking and PZT response measurements.

2693   Mon Mar 22 10:07:30 2010 KojiSummaryElectronicsUPDH Box #17: Ready

Attachment 1: 15V_power_supply.pdf
1763   Mon Jul 20 10:35:06 2009 steveUpdateVACUPS batteries replaced

APC Smart-UPS (uninterruptible power supply) batteries RBC12 replaced at 1Y8 vacuum rack.

Their life span were 22 months.

13234   Mon Aug 21 16:35:48 2017 gautamUpdateVACUPS checkup

[steve, gautam]

At Rolf/Rich Abbott's request, we performed a check of the UPS today.

Steve believed that the UPS was functioning as it should, and the recent accidental vent was because the UPS batteries were insufficiently charged when the test was performed. Today, we decided to try testing the UPS.

We first closed V1, VM1 and VA6 using the MEDM screen. We prepared to pull power on all these valves by loosening the power connections (but not detaching them). [During this process, I lost the screw holding the power cord fixed to the gate valve V1 - we are looking for a replacement right now but it seems to be an odd size. It is cable tied for now.]

The battery charge indicator LEDs on the UPS indicated that the batteries were fully charged.

Next, we hit the "Test" button on the UPS - it has to be held down for ~3 seconds for the test to be actually initiated, seems to be a safety feature of the UPS. Once the test is underway, the LED indicators on the UPS will indicate that the loading is on the UPS batteries. The test itself lasts for ~5seconds, after which the UPS automatically reverts to the nominal configuration of supplying power from the main line (no additional user input is required).

In this test, one of the five battery charge indicator LEDs went off (5 ON LEDs indicate full charge).

So on the basis of this test, it would seem that the UPS is functioning as expected. It remains to be investigated if the various hardware/software interlocks in place will initiate the right sequence of valve closures when required.

 Quote: Never hit O on the Vacuum UPS ! Note: the " all off " configuration should be all valves closed ! This should be fixed now. In case of  emergency  you can close V1 with disconnecting it's actuating power as shown on Atm3 if you have peumatic pressure 60 PSI

1858   Fri Aug 7 16:14:57 2009 robOmnistructureVACUPS failed

Steve, Rana, Ben, Jenne, Alberto, Rob

UPS in the vacuum rack failed this afternoon, cutting off power to the vacuum control system.  After plugging all the stuff that had been plugged into the UPS into the wall, everything came back up.  It appears that V1 closed appropriately, TP1 spun down gracefully on its own battery, and the pressure did not rise much above 3e-6 torr.

The UPS fizzed and smelled burnt.  Rana will order a new, bigger, better, faster one.

1864   Fri Aug 7 19:34:40 2009 steveSummaryVACUPS failed

The Maglev is running on single phase 220V and that voltage  was not interrupted. TP1 was running undisturbed with V1 and V4 closed.

It is independent of the UPS 120V.

1912   Sat Aug 15 18:57:48 2009 ranaUpdateVACUPS failed

As Rob noted last Friday, the UPS which powers the Vacuum rack failed. When we were trying to move the plugs around to debug it, it made a sizzling sound and a pop. Bad smells came out of it.

Ben came over this week and measured the quiescent power consumption. The low power draw level was 11.9 A and during the reboot its 12.2 A. He measured this by ??? (Rob inserts method here).

So what we want is a 120 V * 12.2 A  ~ 1.4 kVA UPS with ~30-50% margin. We look for this on the APC-UPS site:

On Monday, we will order the SUA2200 from APC. It should last for ~25 minutes during an outage. Its $1300. The next step down is$200 cheaper and gives 10 minutes less uptime.

15721   Wed Dec 9 20:14:49 2020 gautamUpdateVACUPS failure

Summary:

1. The (120V) UPS at the vacuum rack is faulty.
2. The drypump backing TP2 is faulty.
3. Current status of vacuum system:
• The old UPS is now powering the rack again. Sometime ago, I noticed the "replace battery" indicator light on this unit was on. But it is no longer on. So I judged this is the best course of action. At least this UPS hasn't randomly failed before...
• main vol is being pumped by TP1, backed by TP3.
• TP2 remains off.
• The annular volumes are isolated for now while we figure out what's up with TP2.
• The pressure went up to ~1 mtorr (c.f. ~600utorr that is the nominal value with the stuck RV2) during the whole episode but is coming back down now.
4. Steve seems to have taken the reliability of the vacuum system with him.

Details:

Around 7pm, the UPS at the vacuum rack seems to have failed. Don't ask me why I decided to check the vacuum screen 10 mins after the failure happened, but the point is, this was a silent failure so the protocols need to be looked into.

Going to the rack, I saw (unsurprisingly) that the 120V UPS was off.

• Pushed the power on button - the LCD screen would briefly light up, say the line voltage was 120 V, and then turned itself off. Not great.
• I traced the power connection to the UPS itself to a power strip under the rack - then I moved the plug from one port to another. Now the UPS stays on. okay...
• but after ~3 mins while I'm hunting for a VGA cable, I hear an incessant beeping. The UPS display has the "Fault" indicator lit up.
• I decided to shift everything back to the old UPS. After the change was made, I was able to boot up the c1vac machine again, and began the recovery process.
• When I tried to start TP2, the drypump was unusually noisy, and I noticed PTP2 bottomed out at ~500 torr (yes torr). So clearly something is not right here. This pump supposedly had its tip-seal replaced by Jordan just 3 months ago. This is not a normal lifetime for the tip seal - we need to investigate more in detail what's going on here...
• Decided that an acceptable config is to pump the main volume (so that we can continue working on other parts of the IFO). The annuli are all <10mtorr and holding, so that's just fine I think.

Questions:

1. Are the failures of TP2 drypump and UPS related? Or coincidence? Who is the chicken and who is the egg?
2. What's up with the short tip seal lifetime?
3. Why did all of this happen without any of our systems catching it and sending an alert??? I have left the UPS connected to the USB/ethernet interface in case anyone wants to remotely debug this.

For now, I think this is a safe state to leave the system in. Unless I hear otherwise, I will leave it so - I will be in the lab another hour tonight (~10pm).

Some photos and a screen-cap of the Vac medm screen attached.

Attachment 1: rackBeforenAfter.pdf
Attachment 2: IMG_0008.jpg
Attachment 3: IMG_0009.jpg
Attachment 4: vacStatus.png
15724   Thu Dec 10 13:05:52 2020 JonUpdateVACUPS failure

I've investigated the vacuum controls failure that occurred last night. Here's what I believe happened.

From looking at the system logs, it's clear that there was a sudden loss of power to the control computer (c1vac). Also, the system was actually down for several hours. The syslog shows normal EPICS channel writes (pressure readback updates, etc., and many of them per minute) which suddenly stop at 4:12 pm. There are no error or shutdown messages in the syslog or in the interlock log. The next activity is the normal start-up messaging at 7:39 pm. So this is all consistent with the UPS suddenly failing.

According to the Tripp Lite manual, the FAULT icon indicates "the battery-supported outlets are overloaded." The failure of the TP2 dry pump appears to have caused this. After the dry pump failure, the rising pressure in the TP2 foreline caused TP2's current draw to increase way above its normal operating range. Attachment 1 shows anomalously high TP2 current and foreline pressure in the minutes just before the failure. The critical system-wide failure is that this overloaded the UPS before overloading TP2's internal protection circuitry, which would have shut down the pump, triggering interlocks and auto-notifications.

### Preventing this in the future:

First, there are too many electronics on the 1 kVA UPS. The reason I asked us to buy a dual 208/120V UPS (which we did buy) is to relieve the smaller 120V UPS. I envision moving the turbo pumps, gauge controllers, etc. all to the 5 kVA unit and reserving the smaller 1 kVA unit for the c1vac computer and its peripherals. We now have the dual 208/120V UPS in hand. We should make it a priority to get that installed.

Second, there are 1 Hz "blinker" channels exposed for c1vac and all the slow controls machines, each reporting the machine's alive status. I don't think they're being monitored by any auto-notification program (running on a central machine), but they could be. Maybe there already exists code that could be co-opted for this purpose? There is an MEDM screen displaying the slow machine statuses at Sitemap > CDS > SLOW CONTROLS STATUS, pictured in Attachment 2. This is the only way I know to catch sudden failures of the control computer itself.

Attachment 1: TP2_time_history.png
Attachment 2: slow_controls_monitors.png
15725   Thu Dec 10 14:29:26 2020 gautamUpdateVACUPS failure

I don't buy this story - P2 only briefly burped around GPStime 1291608000 which is around 8pm local time, which is when I was recovering the system.

Today. Jordan talked to Jon Feicht - apparently there is some kind of valve in the TP2 forepump, which only opens ~15-20 seconds after turning the pump on. So the loud sound I was hearing yesterday was just some transient phenomenon. So today morning at ~9am, we turned on TP2. Once again, PTP2 pressure hovered around 500 torr for about 15-20 seconds. Then it started to drop, although both Jordan and I felt that the time it took for the pressure to drop in the range 5 mtorr - 1 mtorr was unusually long. Jordan suspects some "soft-start" feature of the Turbo Pumps, which maybe spins up the pump in a more controlled way than usual after an event like a power failure. Maybe that explains why the pressure dropped so slowly? One thing is for sure - the TP2 controller displayed "TOO HIGH LOAD" yesterday when I tried the first restart (before migrating everything to the older UPS unit). This is what led me to interpret the loud sound on startup of TP2 to indicate some issue with the forepump - as it turns out, this is just the internal valve not being opened.

Anyway, we left TP2 on for a few hours, pumping only on the little volume between it and V4, and PTP2 remained stable at 20 mtorr. So we judged it's okay to open V4. For today, we will leave the system with both TP2 and TP3 backing TP1. Given the lack of any real evidence of a failure from TP2, I have no reason to believe there is elevated risk.

As for prioritising UPS swap - my opinion is that it's better to just replace the batteries in the UPS that has worked for years. We can run a parallel reliability test of the new UPS and once it has demonstrated stability for some reasonable time (>4 months), we can do the swap.

I was able to clear the FAULT indicator on the new UPS by running a "self-test". pressing and holding the "mute" button on the front panel initiates this test according to the manual, and if all is well, it will clear the FAULT indicator, which it did. I'm still not trusting this unit and have left all units powered by the old UPS.

Update 1100 Dec 11: The config remained stable overnight so today I reverted to the nominal config of TP3 pumping the annuli and TP2 backing TP1 which pumps the main volume (through the partially open RV2).

 Quote: According to the Tripp Lite manual, the FAULT icon indicates "the battery-supported outlets are overloaded." The failure of the TP2 dry pump appears to have caused this. After the dry pump failure, the rising pressure in the TP2 foreline caused TP2's current draw to increase way above its normal operating range. Attachment 1 shows anomalously high TP2 current and foreline pressure in the minutes just before the failure. The critical system-wide failure is that this overloaded the UPS before overloading TP2's internal protection circuitry, which would have shut down the pump, triggering interlocks and auto-notifications.
Attachment 1: vacDiag1.png
15722   Thu Dec 10 11:07:24 2020 ChubUpdateVACUPS fault

Is that a fault code that you can decipher in the manual, or just a light telling you nothing but your UPS is dead?

15723   Thu Dec 10 11:17:50 2020 ChubUpdateVACUPS fault

I can't find anything in the manual that describes the nature of the FAULT message.  In fact, it's not mentioned at all.  If the unit detects a fault at its output, I would expect a bit more information.  This unit does a programmable level of input error protection, too, usually set at 100%.  Still, there is no indication in the manual whether an input issue would be described as a fault; that usually means a short or lifted ground at the output.

 Quote: Is that a fault code that you can decipher in the manual, or just a light telling you nothing but your UPS is dead?
15560   Sun Sep 6 13:15:44 2020 JonUpdateDAQUPS for framebuilder

Now that the old APC Smart-UPS 2200 is no longer in use by the vacuum system, I looked into whether it can be repurposed for the framebuilder machine. Yes, it can. The max power consumption of the framebuilder (a SunFire X4600) is 1.137kW. With fresh batteries, I estimate this UPS can power the framebuilder for >10 min. and possibly as long as 30 min., depending on the exact load.

@Chub/Jordan, this UPS is ready to be moved to rack 1X6/1X7. It just has to be disconnected from the wall outlet. All of the equipment it was previously powering has been moved to the new UPS. I have ordered a replacement battery (APC #RBC43) which is scheduled to arrive 9/09-11.

15537   Mon Aug 24 08:13:56 2020 JonUpdateVACUPS installation

I'm in the lab this morning to interface the two new UPS units with the digital controls system. Will be out by lunchtime. The disruptions to the vac system should be very brief this time.

15538   Mon Aug 24 11:25:07 2020 JonUpdateVACUPS installation

I'm leaving the lab shortly. We're not ready to switch over the vac equipment to the new UPS units yet.

The 120V UPS is now running and interfaced to c1vac via a USB cable. The unofficial tripplite python package is able to detect and connect to the unit, but then read queries fail with "OS Error: No data received." The firmware has a different version number from what the developers say is known to be supported.

The 230V UPS is actually not correctly installed. For input power, it has a general type C14 connector which is currently plugged into a 120V power strip. However this unit has to be powered from a 230V outlet. We'll have to identify and buy the correct adapter cable.

With the 120V unit now connected, I can continue to work on interfacing it with python remotely. The next implementation I'm going to try is item #2 of this plan [ELOG 15446].

 Quote: I'm in the lab this morning to interface the two new UPS units with the digital controls system. Will be out by lunchtime. The disruptions to the vac system should be very brief this time.
15446   Wed Jul 1 18:03:04 2020 JonConfigurationVACUPS replacements

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

• Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
• Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
• Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

15465   Thu Jul 9 18:00:35 2020 JonConfigurationVACUPS replacements

Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).

They will arrive within the next two weeks.

 Quote: ​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference: Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models... Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager. Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring. I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.
1496   Sun Apr 19 11:34:33 2009 josephbHowToCamerasUSB Frame Grabber - How to

To use the Sensoray 2250 USB frame grabber:

Ensure you have the following packages installed: build-essential, libusb-dev

Download the Linux manual and linux SDK from the Sensoray website at:

http://www.sensoray.com/products/2250data.htm

Go to the Software and Manual tab near the bottom to find the links.  The software can also be found on the 40m computers at /cvs/cds/caltech/users/josephb/sensoray/

The files are Manual2250LinuxV120.pdf and s2250_v120.tar.gz

Run the following commands in the directory where you have the files.

tar -xvf s2250_v120.tar.gz

cd s2250_v120

make

make

sudo make modules_install

cd ..

At this point plug in the 2250 frame grabber.

Now you can run the demo with

./sraydemo or ./sraydemo64

Options will show up on screen.  A simple set to start with is "encode 0", which sets the recording type, "recvid test.mpg", which starts the recording in file test.mpg, and "stop", which stops recording.  Note there is no on screen playback.  One needs an installed mpeg player to view the saved file, such as Totem (which can screen cap to .png format) or mplayer.

All these instructions are on the first few pages of the Manual2250LinuxV120 pdf.

13277   Wed Aug 30 22:15:47 2017 ranaOmnistructureComputersUSB flash drives moved

I have moved the USB flash drives from the electronics bench back into the middle drawer of the cabinet next to the AC which is west of the fridge. Drawer re-enlabeled.

9192   Thu Oct 3 02:46:58 2013 ranaMetaphysicsPEMUSGS Furlough

7749   Tue Nov 27 00:26:00 2012 jamieOmnistructureComputersUbuntu update seems to have broken html input to elog on firefox

After some system updates this evening, firefox can no longer handle the html input encoding for the elog.  I'm not sure what happened.  You can still use the "ELCode" or "plain" input encodings, but "HTML" won't work.  The problem seems to be firefox 17.  ottavia and rosalba were upgraded, while rossa and pianosa have not yet been.

I've installed chromium-browser (debranded chrome) on all the machines as a backup.  Hopefully the problem will clear itself up with the next update.  In the mean time I'll try to figure out what happened.

To use chromium: Appliations -> Internet -> Chromium

6192   Thu Jan 12 21:22:16 2012 Leo SingerConfigurationWIKI-40M UpdateUnable to create Wiki page

I can't create a new page on the 40m wiki.  The page that I was trying to create is

http://blue.ligo-wa.caltech.edu:8000/40m/Stewart

I get this message when I try to save the new page:

Page could not get locked. Unexpected error (errno=13).

6193   Thu Jan 12 23:13:42 2012 KojiConfigurationWIKI-40M UpdateUnable to create Wiki page

 Quote: I can't create a new page on the 40m wiki.  The page that I was trying to create is http://blue.ligo-wa.caltech.edu:8000/40m/Stewart I get this message when I try to save the new page: Page could not get locked. Unexpected error (errno=13).

This address for wiki is obsolete. Recently it was switched to https://wiki-40m.ligo.caltech.edu/
Jamie is working on automatic redirection from the old wiki to the new place.

The new one uses albert.einstein authentication.

11889   Thu Dec 17 01:55:16 2015 ericqUpdateLSCUncooperative AUX X

[ericq, Gautam]

We were not able to fix the excess frequency noise of the AUX X laser by the usual laser diode current song and dance. Unfortunately, this level of noise is much too high to have any realistic chance of locking.

We're leaving things back in the IR beat -> phase tracker state with free running AUX lasers, on the off chance that there may be anything interesting to see in the overnight data. This may be limited by our lack of automatic beatnote frequency control. (Gautam will soon implement this via digital frequency counter). I've upped the FINE_PHASE_OUT_HZ_DQ frame rate to 16k from 2k, so we can see more of the spectrum.

For the Y beat, there is the additional weird phenomenon that the beat amplitude slowly oscillates to zero over ~10 minutes, and then back up to its maximum. This makes it hard for the phase tracker servo to stay stable... I don't have a good explanation for this.

11892   Fri Dec 18 17:37:04 2015 ranaUpdateLSCUncooperative AUX X

Here's how we should diagnose the EX laser:

1. Compare IR RIN of laser out to 100 kHz with that of another similar NPRO.
2. Look at time series of IR beat signal with a fast scope. Are there any high frequency glitches?
3. Disconnect all of the cables to the EX laser PZT and temperature control. Does the frequency noise change?
4. Change the temperature by +/- 1 deg to move away from mode hop regions. Remeasure RIN and frequency noise and plot.
ELOG V3.1.3-