My belief is that the frequency noise from the unstabilized MC is making the PRC locking harder. This will be investigated by tuning the shape of the MCL/MCF crossover so that we can turn it on without ruining the arm cavity spectra. Since the PRC length is ~2x smaller than the MC, we would expect it to be less sensitive to the MC frequency noise. But, since there is some common mode rejection in there, this may not be true. We'll only know by measuring PRC control signal with MCL on/off.
I think if we make MCL UGF higher then 20 Hz, arm cavity spectra will feel it. It might be possible to use a combination of feedback and feedforward control from ground seismometers. I made MCL UGF at 3 Hz to reduce 1 Hz motion of the pendulum; feedforward OAF subtracted the stack at 3.3 Hz. Once OAF converged, I blocked adaptation and the filter became static FIR. MC length RMS was reduced by a factor of 10 and arm cavity spectra was not affected at frequencies >20 and became better at low frequencies. We'll see if this enough.
On the attached plot red color shows MC_F with MC_L OFF, blue - MC_L is ON, green - MC_L and OAF are ON.
Then I locked PRCL (using AS_Q and REFL55_I) to carrier and aligned the cavity. Power RIN was 50-70% and 00 beam on the POP camera was moving significantly. BS oplev was shaking the optics at 5 Hz. I fixed it, but there should be something else as RIN was still high.
80 degrees is too uncomfortable in the control room so I turned on the AC. The set point is 74F.
This past Friday I swapped out a burnt resistor on the spare STACIS unit I'm working with and powered it up. Here's the setup:
And here's what happened:
X an Y directions: When I switched from open to closed loop (making the internal geophones provide feedback), the STACIS started making a loud noise- it seemed like it was oscillating uncontrollably.
Z direction: The same thing happened in z until I added some weight to the top of the STACIS- then it quieted down, and seemed to work okay. The geophone signal dropped considerably compared to the open loop signal, which is expected if the feedback is working.
Then I tried driving the PZTs with a signal from the SR785 network analyzer. With an amplitude of tens of mV and frequencies from around .1 to 200 Hz, I could see the accelerometers I mounted on top of the STACIS definitely register motion, which means I was successfully driving the PZTs.
Below are transfer functions of the STACIS as I drove the PZTs from .1 to 100 Hz at 10 mV. The top graph is open loop, the second is closed loop. These were measured with the internal geophones.
In the bottom graph, "A" is closed loop and "B" is open loop, where the transfer functions were taken with the accelerometers instead of the geophones.
The PMC trans power was a little low (0.77V or so). I tweaked up the input pointing and now we're getting about 0.875V transmitted.
I spent some time this afternoon reviving some of my CESAR/ESCOBAR shenanigans on the Y arm. I found it neccesary to adjust a few things.
Afterwards, ALSY noise levels were good.
EDIT: After discussing with Koji and checking the existing M2ISS PDs I put the two C30642G back and took two C30665GH (active diameter: 3mm) diodes. Only one of this type remains in storage.
I removed two C30642G photodiodes from the stash for the new M2ISS hardware and updated the wiki page accordingly.
Both Guralps and the Ranger have been placed in our nice new insulated foam box, complete with packing peanuts, in the corner between the x and y arms. The Guralp breakout box has been reinstalled and everything is plugged in in prepartion for the huddle test. However, we're having some issues with ADC channels, which will be worked out tomorrow (hopefully) so that data can be collected over the weekend.
Currently, one Guralp is plugged into the three SEIS-MC1 channels. We made new channels for the second Guralp (GUR-EW, GUR-NS, and GUR-VERT), but had issues with those. So, EW and NS have been plugged into PEM_AUDIO-MIC1 and MIC2 for the time being.
I found that several of the cables are unlabeled so I'm not sure what's plugged in. In the end, I found that the TEMP_2, _3, & _44 channels were working and so I plugged in anything that looked seismic into there.
TEMP_2 is now apparently the X channel of the 2nd Guralp. If someone can figure out which cable belongs to the Y channel, please plug it into TEMP_3 and then we can fix the channel names.
I also removed (gently) all of the accelerometers from MC2's chamber. This didn't break the lock, but I intentionally broke it to make sure it reacquired fine. It did and the MC TRANS QPD showed no significant shift afterwards.
In eLog 7148 (http://nodus.ligo.caltech.edu:8080/40m/7148), Koji pointed out that the op-amp and SR560 noise values (which I took from specs and then multiplied by the geophone calibration factor to get m/s/rtHz) were waaay too low. My error was an extra multiplication factor in the plotting script.
The other change was recalculating the ADC noise by splitting a signal into two ADC channels and subtracting the time series (then taking the PSD and converting to m/s/rtHz). It compares well to the value I got by terminating the ADC channels, which was the ADC noise line in my last eLog.
Both these changes are included in the below plot:
Today we began implementing the UGF Servos. Things we did:
I added UGF servos for the DRMI DoFs, after creating a library block for the servos. I also deleted the FMs before the phase rotation, since we can just do it afterwards in other existing FMs. I've only added the MICH and PRCL buttons to the LSC screen because in the end, I feel like a dropdown is better, but I just wanted to get it running quickly tonight. The LSC model and the UGF block have been committed to the svn.
We were able to use the PRCL UGF servo successfully, as Jenne was exploring MICH offset space.
The UGF servos have been moved to the control point, are are once again totally linear!
The UGF servos were recommissioned today:
Our idea is that we need with some thinking about these servos and most of all try to figure out all this phase thing before we can start to trust the servos to be used for locking.
UGF Servos' commissioning still going on, updates of today:
I found an error in the model of the UGF servos, I have now corrected it; for future reference, now the division between TEST2 and TEST1 is properly done with complex math: given
we have that TEST3:
TEST3 is the actual signal that is now phase rotated to select only the I signal while rejecting the Q one.
All the updates to the model, the screens and the script have been SVNed.
I lost UL osem voltage this morning when I was checking the actual connection at rack ETMY
This after noon I disconnected the 64 pins IDE connector from satelite amp at the rack, and the two 25 pins Dsubs at this juction board.
UL OSEM recovered after reconnecting these three connectors.
Atm3, bad connection.........noisy UL
It took too long to get this box ready for action. I implemented all of the changes that I made on the previous one (#1437). In addition, since this one is to be used for phase locking, I also made it have a ~flat transfer function. With the Boost ON, the TF magnitude will go up like 1/f below ~1 kHz.
The main trouble that I had was with the -12V regulator. The output noise level was ~500 nV/rHz, but there was a large oscillation at its output at ~65 kHz. This was showing up in the output noise spectrum of U1 (the first op-amp after the mixer). Since the PSRR of the OP27 is only ~40 dB at such a high frequency, it is not strange to see the power supply noise showing up (the input referred noise of the OP27 is 3.5 nV/rHz, so any PS noise above ~350 nV/rHz becomes relavent).
I was able to tame this by putting a 10 uF tantalum cap on the output of the regulator. However, when I replaced the regulator with a LM7912 from the blue box, it showed an output noise that went up like 1/f below 50 kHz !! I replaced it a couple more times with no benefit. It seems that something on the board must now be damaged. I checked another of the UPDH boxes, and it has the same high frequency oscillation but not so much excess voltage noise. I found that removing the protection diode on the output of the regulator decreased the noise by a factor of ~2. I also tried replacing all of the 1 uF caps that are around the regulator. No luck.
Both of the +12 V regulators seem fine: normal noise levels of ~200 nV/rHz and no oscillations.
Its clear that the regulator is not functioning well and my only guess is that its a layout issue on the board or else there's a busted component somewhere that I can't find. In any case, it seems to be functioning now and can be used for the phase locking and PZT response measurements.
For your reference: Voltage noise of LM7815/LM7915 (with no load)
APC Smart-UPS (uninterruptible power supply) batteries RBC12 replaced at 1Y8 vacuum rack.
Their life span were 22 months.
At Rolf/Rich Abbott's request, we performed a check of the UPS today.
Steve believed that the UPS was functioning as it should, and the recent accidental vent was because the UPS batteries were insufficiently charged when the test was performed. Today, we decided to try testing the UPS.
We first closed V1, VM1 and VA6 using the MEDM screen. We prepared to pull power on all these valves by loosening the power connections (but not detaching them). [During this process, I lost the screw holding the power cord fixed to the gate valve V1 - we are looking for a replacement right now but it seems to be an odd size. It is cable tied for now.]
The battery charge indicator LEDs on the UPS indicated that the batteries were fully charged.
Next, we hit the "Test" button on the UPS - it has to be held down for ~3 seconds for the test to be actually initiated, seems to be a safety feature of the UPS. Once the test is underway, the LED indicators on the UPS will indicate that the loading is on the UPS batteries. The test itself lasts for ~5seconds, after which the UPS automatically reverts to the nominal configuration of supplying power from the main line (no additional user input is required).
In this test, one of the five battery charge indicator LEDs went off (5 ON LEDs indicate full charge).
So on the basis of this test, it would seem that the UPS is functioning as expected. It remains to be investigated if the various hardware/software interlocks in place will initiate the right sequence of valve closures when required.
Never hit O on the Vacuum UPS !
Note: the " all off " configuration should be all valves closed ! This should be fixed now.
In case of emergency you can close V1 with disconnecting it's actuating power as shown on Atm3 if you have peumatic pressure 60 PSI
Steve, Rana, Ben, Jenne, Alberto, Rob
UPS in the vacuum rack failed this afternoon, cutting off power to the vacuum control system. After plugging all the stuff that had been plugged into the UPS into the wall, everything came back up. It appears that V1 closed appropriately, TP1 spun down gracefully on its own battery, and the pressure did not rise much above 3e-6 torr.
The UPS fizzed and smelled burnt. Rana will order a new, bigger, better, faster one.
The Maglev is running on single phase 220V and that voltage was not interrupted. TP1 was running undisturbed with V1 and V4 closed.
It is independent of the UPS 120V.
As Rob noted last Friday, the UPS which powers the Vacuum rack failed. When we were trying to move the plugs around to debug it, it made a sizzling sound and a pop. Bad smells came out of it.
Ben came over this week and measured the quiescent power consumption. The low power draw level was 11.9 A and during the reboot its 12.2 A. He measured this by ??? (Rob inserts method here).
So what we want is a 120 V * 12.2 A ~ 1.4 kVA UPS with ~30-50% margin. We look for this on the APC-UPS site:
On Monday, we will order the SUA2200 from APC. It should last for ~25 minutes during an outage. Its $1300. The next step down is $200 cheaper and gives 10 minutes less uptime.
Around 7pm, the UPS at the vacuum rack seems to have failed. Don't ask me why I decided to check the vacuum screen 10 mins after the failure happened, but the point is, this was a silent failure so the protocols need to be looked into.
Going to the rack, I saw (unsurprisingly) that the 120V UPS was off.
For now, I think this is a safe state to leave the system in. Unless I hear otherwise, I will leave it so - I will be in the lab another hour tonight (~10pm).
Some photos and a screen-cap of the Vac medm screen attached.
I've investigated the vacuum controls failure that occurred last night. Here's what I believe happened.
From looking at the system logs, it's clear that there was a sudden loss of power to the control computer (c1vac). Also, the system was actually down for several hours. The syslog shows normal EPICS channel writes (pressure readback updates, etc., and many of them per minute) which suddenly stop at 4:12 pm. There are no error or shutdown messages in the syslog or in the interlock log. The next activity is the normal start-up messaging at 7:39 pm. So this is all consistent with the UPS suddenly failing.
According to the Tripp Lite manual, the FAULT icon indicates "the battery-supported outlets are overloaded." The failure of the TP2 dry pump appears to have caused this. After the dry pump failure, the rising pressure in the TP2 foreline caused TP2's current draw to increase way above its normal operating range. Attachment 1 shows anomalously high TP2 current and foreline pressure in the minutes just before the failure. The critical system-wide failure is that this overloaded the UPS before overloading TP2's internal protection circuitry, which would have shut down the pump, triggering interlocks and auto-notifications.
First, there are too many electronics on the 1 kVA UPS. The reason I asked us to buy a dual 208/120V UPS (which we did buy) is to relieve the smaller 120V UPS. I envision moving the turbo pumps, gauge controllers, etc. all to the 5 kVA unit and reserving the smaller 1 kVA unit for the c1vac computer and its peripherals. We now have the dual 208/120V UPS in hand. We should make it a priority to get that installed.
Second, there are 1 Hz "blinker" channels exposed for c1vac and all the slow controls machines, each reporting the machine's alive status. I don't think they're being monitored by any auto-notification program (running on a central machine), but they could be. Maybe there already exists code that could be co-opted for this purpose? There is an MEDM screen displaying the slow machine statuses at Sitemap > CDS > SLOW CONTROLS STATUS, pictured in Attachment 2. This is the only way I know to catch sudden failures of the control computer itself.
I don't buy this story - P2 only briefly burped around GPStime 1291608000 which is around 8pm local time, which is when I was recovering the system.
Today. Jordan talked to Jon Feicht - apparently there is some kind of valve in the TP2 forepump, which only opens ~15-20 seconds after turning the pump on. So the loud sound I was hearing yesterday was just some transient phenomenon. So today morning at ~9am, we turned on TP2. Once again, PTP2 pressure hovered around 500 torr for about 15-20 seconds. Then it started to drop, although both Jordan and I felt that the time it took for the pressure to drop in the range 5 mtorr - 1 mtorr was unusually long. Jordan suspects some "soft-start" feature of the Turbo Pumps, which maybe spins up the pump in a more controlled way than usual after an event like a power failure. Maybe that explains why the pressure dropped so slowly? One thing is for sure - the TP2 controller displayed "TOO HIGH LOAD" yesterday when I tried the first restart (before migrating everything to the older UPS unit). This is what led me to interpret the loud sound on startup of TP2 to indicate some issue with the forepump - as it turns out, this is just the internal valve not being opened.
Anyway, we left TP2 on for a few hours, pumping only on the little volume between it and V4, and PTP2 remained stable at 20 mtorr. So we judged it's okay to open V4. For today, we will leave the system with both TP2 and TP3 backing TP1. Given the lack of any real evidence of a failure from TP2, I have no reason to believe there is elevated risk.
As for prioritising UPS swap - my opinion is that it's better to just replace the batteries in the UPS that has worked for years. We can run a parallel reliability test of the new UPS and once it has demonstrated stability for some reasonable time (>4 months), we can do the swap.
I was able to clear the FAULT indicator on the new UPS by running a "self-test". pressing and holding the "mute" button on the front panel initiates this test according to the manual, and if all is well, it will clear the FAULT indicator, which it did. I'm still not trusting this unit and have left all units powered by the old UPS.
Update 1100 Dec 11: The config remained stable overnight so today I reverted to the nominal config of TP3 pumping the annuli and TP2 backing TP1 which pumps the main volume (through the partially open RV2).
Is that a fault code that you can decipher in the manual, or just a light telling you nothing but your UPS is dead?
I can't find anything in the manual that describes the nature of the FAULT message. In fact, it's not mentioned at all. If the unit detects a fault at its output, I would expect a bit more information. This unit does a programmable level of input error protection, too, usually set at 100%. Still, there is no indication in the manual whether an input issue would be described as a fault; that usually means a short or lifted ground at the output.
Now that the old APC Smart-UPS 2200 is no longer in use by the vacuum system, I looked into whether it can be repurposed for the framebuilder machine. Yes, it can. The max power consumption of the framebuilder (a SunFire X4600) is 1.137kW. With fresh batteries, I estimate this UPS can power the framebuilder for >10 min. and possibly as long as 30 min., depending on the exact load.
@Chub/Jordan, this UPS is ready to be moved to rack 1X6/1X7. It just has to be disconnected from the wall outlet. All of the equipment it was previously powering has been moved to the new UPS. I have ordered a replacement battery (APC #RBC43) which is scheduled to arrive 9/09-11.
I'm in the lab this morning to interface the two new UPS units with the digital controls system. Will be out by lunchtime. The disruptions to the vac system should be very brief this time.
I'm leaving the lab shortly. We're not ready to switch over the vac equipment to the new UPS units yet.
The 120V UPS is now running and interfaced to c1vac via a USB cable. The unofficial tripplite python package is able to detect and connect to the unit, but then read queries fail with "OS Error: No data received." The firmware has a different version number from what the developers say is known to be supported.
The 230V UPS is actually not correctly installed. For input power, it has a general type C14 connector which is currently plugged into a 120V power strip. However this unit has to be powered from a 230V outlet. We'll have to identify and buy the correct adapter cable.
With the 120V unit now connected, I can continue to work on interfacing it with python remotely. The next implementation I'm going to try is item #2 of this plan [ELOG 15446].
I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:
I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.
Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).
They will arrive within the next two weeks.
To use the Sensoray 2250 USB frame grabber:
Ensure you have the following packages installed: build-essential, libusb-dev
Download the Linux manual and linux SDK from the Sensoray website at:
Go to the Software and Manual tab near the bottom to find the links. The software can also be found on the 40m computers at /cvs/cds/caltech/users/josephb/sensoray/
The files are Manual2250LinuxV120.pdf and s2250_v120.tar.gz
Run the following commands in the directory where you have the files.
tar -xvf s2250_v120.tar.gz
sudo make modules_install
At this point plug in the 2250 frame grabber.
sudo modprobe s2250_ezloader
Now you can run the demo with
./sraydemo or ./sraydemo64
Options will show up on screen. A simple set to start with is "encode 0", which sets the recording type, "recvid test.mpg", which starts the recording in file test.mpg, and "stop", which stops recording. Note there is no on screen playback. One needs an installed mpeg player to view the saved file, such as Totem (which can screen cap to .png format) or mplayer.
All these instructions are on the first few pages of the Manual2250LinuxV120 pdf.
I have moved the USB flash drives from the electronics bench back into the middle drawer of the cabinet next to the AC which is west of the fridge. Drawer re-enlabeled.
After some system updates this evening, firefox can no longer handle the html input encoding for the elog. I'm not sure what happened. You can still use the "ELCode" or "plain" input encodings, but "HTML" won't work. The problem seems to be firefox 17. ottavia and rosalba were upgraded, while rossa and pianosa have not yet been.
I've installed chromium-browser (debranded chrome) on all the machines as a backup. Hopefully the problem will clear itself up with the next update. In the mean time I'll try to figure out what happened.
To use chromium: Appliations -> Internet -> Chromium
I can't create a new page on the 40m wiki. The page that I was trying to create is
I get this message when I try to save the new page:
Page could not get locked. Unexpected error (errno=13).
This address for wiki is obsolete. Recently it was switched to https://wiki-40m.ligo.caltech.edu/
Jamie is working on automatic redirection from the old wiki to the new place.
The new one uses albert.einstein authentication.
We were not able to fix the excess frequency noise of the AUX X laser by the usual laser diode current song and dance. Unfortunately, this level of noise is much too high to have any realistic chance of locking.
We're leaving things back in the IR beat -> phase tracker state with free running AUX lasers, on the off chance that there may be anything interesting to see in the overnight data. This may be limited by our lack of automatic beatnote frequency control. (Gautam will soon implement this via digital frequency counter). I've upped the FINE_PHASE_OUT_HZ_DQ frame rate to 16k from 2k, so we can see more of the spectrum.
For the Y beat, there is the additional weird phenomenon that the beat amplitude slowly oscillates to zero over ~10 minutes, and then back up to its maximum. This makes it hard for the phase tracker servo to stay stable... I don't have a good explanation for this.
Here's how we should diagnose the EX laser:
EDIT: Sleepy Eric doesn't understand loops. The conditions for this observation included active oplev loops. Thus, obviously, looking at the in-loop signal after the ASC signl joins the oplev signal will produce this kind of behavior.
After some talking with Rana, I set out on making an even better-er QPD loop. I made some progress on this, but a new mystery halted my progress.
I sought to have a more physical undertanding of the plant TF I had measured. Earlier, I had assumed that the 4Hz plant features I had measured for the QPD loops were coming from the oplev-modified pendulum response, but this isn't actually consistent with the loop algebra of the oplev servos. I had seen this feature in both the oplev and qpd error signals when pushing an excitation from the ASC-XARM_PIT (and so forth) FMs.
However, when exciting via the SUS-ETMX-OLPIT FMs (and so forth), this feature would not appear in either the QPD or oplev error signals. That's weird. The outputs of these two FMs should just be summed, right before the coil matrix.
I started looking at the TF from ASC-YARM_PIT_OUT to SUS-ETMY_TO_COIL_1_2, which should be a purely digital signal routing of unity, and saw it exhibit the phase shape at 4Hz that I had seen in earlier measurements. Here it is:
I am very puzzled by all of this. Needs more investigation.
The measured transimpedance of the latest POY11 PD matches my model very well up to 100 MHz. But at about ~216MHz I have a resonance that I can't really explain.
The following is a simplified illustration of the resonant circuit:
Perhaps my model misses that resonance because it doesn't include stray capacitances.
While I was tinkering with it, i noticed a couple of things:
- the frequency of that oscillation changes by grasping with finger the last inductor of the circuit (the 55n above); that is adding inductance
- the RF probe of the scope clearly shows me the oscillation only after the 0.1u series capacitor
- adding a small capacitor in parallel to the feedback resistor of the output amplifier increases the frequency of the oscilaltion
Where did you get the 55nH based notch from? I don't remember anything like that from the other LSC PD schematics. This is certainly a bad idea. You should remove it and put the notch back over by the other notch.
Why is it a bad idea?
You mean putting both the 2-omega and the 55MHz notches next to each other right after the photodiode?