All suspension damping has been restored.
All the front end machines are back up after the outage. It looks like none of the front end machines came back up once power was restored, and they all needed to be powered manually. One of the things I want to do in the next CDS upgrade is put all the front end computers in one rack, so we can control their power remotely.
c1sus was the only one that had a little trouble. It's timing was for some reason not syncing with the frame builder. Unclear why, but after restarting the models a couple of times things came back.
There's still a little red, but it mostly has to do with the fact that c1oaf is busted and not running (it actually crashes the machine when I tried to start it, so this needs to be fixed!).
There had to be a power outage. Laser and air condition turned back on. The vacuum is OK
Sorensen DC power supplies were tripped, so they were reset: at AUX OMC South 18V and 28V for RF PS and at 1X1 24V
Power Outage confirmed:
** Notification **
CALIFORNIA INSTITUTE OF TECHNOLOGY
Date: Thursday October 04,2012
This morning at 2:17 a.m. much of the City of Pasadena including our Campus experienced a electric power sag of short duration, approximately 1/10 of a second. The cause was a fault on one of Pasadena’s 17KV circuits. Some sensitive equipment have been impacted.
Contact: Mike Anchondo x-4999
We naively hoped that just replacing the fuses would fix the problem with the PZT HV drivers. Alas, this was not the case.
All of our investigations (other than visual inspections) today have been of the PZT2 module. We have not applied any electricity to any PZT1 components/modules today.
After blowing a few more fuses (not good, we know, but we really didn't know what was going on at the time and were convinced that our changes between fuse installations should prevent fuse-blowing, including removing all modules except the HV driver), we found that the YAW driver for both PZT1 and PZT2 has severe discoloration on the PCB, and several resistors and other solder joints are damaged near some high voltage regulators. Pitch on PZT1 looks a tiny bit discolored, but doesn't look totally cooked like the 2 YAW modules do. So, at least PZT1's Yaw was cooked before we started replacing fuses, since we haven't plugged it in yet today.
We then began some more methodical checks:
We bypassed the fuses by applying 10 Vpp = ~7.2 Vrms to the input side of the big transformer on the PZT2 HV driver board. (This usually sees the 120 Vrms from the wall AC, so we were looking at things with a factor ~16 attenuation from what they normally see.) We then measured things on the other side of the transformer, and made sure that they made some sense (one path for 5V stuff, one path for 15V stuff, one path for 180V stuff). One of the rectifying diode bridges (the one for HV) didn't seem to be working, and didn't seem to have all of its pins connected, as if perhaps one or more diodes inside was destroyed.
When I went home for dinner, Koji continued looking at the low voltage supply capability of the PZT2 driver. He removed the diode bridge from the HV path, and also removed the FET that lives on the output side of the HV driver board. He was then able to energize the HV driver and the non-burnt pitch module. So the +\-5 V and +\-15 V paths have been confirmed okay for PZT2's driver stuff.
What I will do tomorrow (when there is someone here to rescue me if I crispy-fry myself) is solder a wire to the now open pin of the backplane connector on the HV driver board, so that we can supply an external 180V to the pitch / yaw modules (although, obviously we won't be using the burnt yaw modules as-is). Tomorrow I'll start by applying a nice small voltage, check that things still look okay, no shorts, and then I'll slowly increase the voltage until I get to the nominal 180V.
Since the low voltage stuff on the driver board is working, once we supply an external 180V (if successful), we should be able to re-install the PZT driver and drive PZT2.
Since both Yaw modules that we have are burnt, I am proposing that we use the PZT2 HV board (which has been checked and modified this evening) with the 2 pitch modules. Since we are not actively utilizing the strain gauge sensors, the fact that the calibrations on these modules are not exactly the same (rather, that PZT1's pitch is not the same as PZT2's yaw) should not matter at all. This means that we will not be able to energize PZT1 at all, but that shouldn't be a problem. Even when PZT 2 was working, PZT1 had very, very, very limited motion through the full range of applied voltage, so having no driver connected shouldn't have an impact.
After the AOM work the beam wasn't well aligned to the PMC. The PMC REFL CCD shows large misalignment in yaw.
This is the third morning in a row that the MC2 was tripped. Would you look at it Koji?
This may be cause by the impact of crazy WFS signal after the lock loss.
The auto locker is not fast enough to shut the WFS down before the mirrors are kicked.
Jenne and I discussed the issue and agreed that this can be solved by implementing
the same triggering algorithm as the LSC triggers.
Give us a bit more time to work on this.
I made a wiki page for the active IO tip-tilts. I should have made this a long time ago.
We set start to check the performance of the AOM on the PSL table. The AOM driver spits out ~1.5W rf at 80MHz for 1V DC at its modulation input. In order to align the AOM, we reduced the input power to the AOM to ~10% using the QWP between the PBS and the laser. We touched the steering mirror before the AOM...but did not succeed in getting any appreciable first order deflection. We then released the AOM mount and moved it a few microns in and out until we obtained a significant change in power along the zero-order beam from 400mV to 100mV when the rf power was changed from 0 to ~1.5W (by changing modulation input from 0 to 1V). The AOM was clamped at this alignment and the QWP was rotated to give maximum input power.
During the course of aligning the AOM, the PMC unlocked and was restored after the alignment.
All went well without having to make any emergency calls to anyone
We will now have to think about switching the AOM on and off for ringdown measurements. This could be done by either using a high-power rf switch or by switching the modulation DC input between 0 and 1V; whichever will be more comfortable to take many many ringdown measurements.
I spotted around 4 within 30 minutes working at the PSL table even after the deathly spray. They seem to be running down from the cables on the oscilloscope rack to the table and the optics.
The SOS coil drivers (Atm2) were moved from 1X1 to 1Y2 location. Is this the best place to locate the IOO Tip-Tilt steering that will replace the PJ-PZT ?
See 40m wiki T-T
We observed one or two ants climbing over PMC optics without booties and safety glasses.
The floor was mopped with strong Bayer Home Pest Control solution in the Vertex area.
Do not work inside the 40m lab if you are sensitive to chemicals!
Den Martynov received 40m specific safety training.
Jamie and I pulled the whole PZT driver for both PZT1 and PZT2.
Koji and I found that each HV power supply (the left-most module) has 2 fuses. Both HV supplies (PZT1 and PZT2) have one blown fuse. The "T2L250A" measures low resistance for both HV supplies, but the "T250mAL250V" measures Open for both HV supplies.
I have ordered 10 pieces of each kind of fuse, Next Day shipping, from DigiKey.
We installed the AOM driver back on the PSL table this morning. To calibrate the AOM RF output we connected a 1V dc to the modulation input of the driver and we are convinced with the setup.
Before we direct the rf signal to the AOM, in order to check its diffraction efficiency, we would like to setup an rf PD at the AOM output. We think we have place for a filter and PD after the AOM (replacing a beam dump) and would like to confirm the position before we actually install them. The layout is the picture below showing sweet spots for the new pd to sit. If you think it may disturb the system in any way, let us know!
The rf PD and filter have been installed at the earlier proposed spot on the PSL table.
pzt2 mod signals matched slider vals for both pitch and yaw
pzt2 yaw mon output = 6
pzt2 pitch mon output = 11.3
From the PZT connector-converter board we determined the following pin-outs:
X=Yaw: red=1, white=14, black=3
Y=Pitch: red=2, white=15, black=16
We believe that red is signal, white/black/shield are all ground. We also believe (although this is from the PMC PZT) that the expected capacitance of the PZTs should be in the 100's of nF range.
Here are the readings from the two PZT dsub connectors:
pin 1:14 PZT1 = ".003" on 2uF scale
PZT2 = ".184"
pin 2:15 PZT1 = ".002" on 2uF scale
PZT2 = ".202"
So we think this means (given this crappy capacitance meter) that PZT2 is showing roughly 200nF, which sounds ok, but that PZT1 is indeed bad.
So next we investigate the PZT2 driver.
The PDA255 that Koji repaired is still not alright. It seems to be saturating again. I've left it in the PD cabinet where it is marked 'PDA 255'. I've asked Steve to order a fast PD at 150MHz, PDA10A because we don't seem to have any at the 40m.
The new cold cathode gauge CC1 is in place. We were at atmosphere for 28 days ......more later
cc1 = 2.3e-5 Torr at day 6 vacuum normal
Changed the list of channels to be written to frames from having the IN1 suffix to OUT. Now we can load the calibration of the channel into the filter module and the DQ channel will be calibrated.
We should do this wherever possible so that our channels will have real calibrations associated with them.
Next we should up the rate at which the model runs up to 16 kHz so that we can record the microphones at 16 kHz. FM radio has information up to 20 kHz. AM radio goes up to ~8 kHz. We should be at least as modern as AM radio. How do we make the change? How do we make sure the FOTON file stays OK?
I have made some changes to the daily summary file to compensate. New files is /users/public_html/40m-summary/share/c1_summary_page.ini.
The typical sign of a dying gas laser is that it glows for a few minutes only. The power supplies are fine.
Two new JDS - Uniphase 1103P lasers ( NT64-104 ) arriving on Monday, May 21
Yesterday I swapped in new He/Ne laser with output power 3.5 mW The return spot on qpd is large ~6mm in diameter and 20,500 counts
The spot size reduction require similar layout as ETMX oplev.
The oplev path is relayed and the spot size on the qpd is reduced. I still have to clean up and replace "Miki Mouse" lens holder.
There was no IP-ANG coming out of the chamber at this time!
I did relayed the oplev path with new f 500 mm lens
For some reason the frame builder and mx stream processes on ALL front ends were down. I restarted the frame builder and all the mx_stream processes and everything seems to be back to normal. Unclear what caused this. The CDS guys are aware of the issue with the mx_stream stability and are working on it.
I'll come back to the PZTs later, but I want to write down all the elogs I have found so far that look relevant.
Found a bug in the IOO screen: All of the 6 WFS signal indicators is liked to the same info (C1:IOO-MC1_PIT_OUTPUT).
Fix this, Jenne! Baaaaagghhhhh!
My bad. As it turns out, you can't copy and paste between MEDM instances. It is now fixed.
MC WFS was fixed. Now it is running constantly with the autolocker.
Found a bug in the IOO screen: All of the 6 WFS signal indicators is liked to the same info (C1:IOO-MC1_PIT_OUTPUT).
Fix this, Jenne! Baaaaagghhhhh!
What I did:
1. C1:IOO-MC_RFPD_DCMON indicator was saturating. "HOPR" of this entry was set to 5 by running the following command:
ezcawrite C1:IOO-MC_RFPD_DCMON.HOPR 5
2. Scan MC2 spot position by using /opt/rtcds/caltech/c1/scripts/MC/moveMC2 scripts.
or the adjustment, C1:SUS-MC2_ASCPIT_EXC and C1:SUS-MC2_ASCYAW_EXC were excited with 300cnt at 12Hz and 10Hz, respectively.
The corresponding peaks (i.e. ANgle to length coupling) in C1:IOO-MC_F were monitored on DTT and adjusted so that the peaks are approximately nulled.
3. moveMC2 scripts are not perect to keep the maximum of the transmission. So, the alignment was adjusted with MC1 and MC3.
4. Repeated 2 and 3 until the alignment converges.
5. Once I got satisfied with the MC2 spot position, I went to the MC2 table and aligned the steering mirror before the QPD.
6. As these actions above moves the REFL beam, I went to the MC REFL path and adjusted the MC REFL PD position and the MC WFS spot positions.
7. Checked if the alignment is still good. The MC REFL is 0.50~0.51. Pretty good.
8. Run /opt/rtcds/caltech/c1/scripts/MC/WFS/WFS_FilterBank_offsets to register the current WFS offset etc.
9. At this point, MC WFS started working fine. I also confirmed the autolocker worked with this setting.
Checked how the things are going in the morning. There were several unlocks. But the autolocker and WFS kept the cavity lcoked again.
Some power fluctuation of ~1% is observed in the MC trans. I checked the PMC trans and found it is also fluctuating by 1% in a coherent way.
So I judge the WFS itself is fine. (See attached)
Where is IP-ANG ? It is good practice to use two mirrors at launching and detecting the beam, so you can walk it - precisely adjust it.
The window can be replaced at ~$1,500 ea. 10 weeks as optical quality BK7 with dual AR
1. I wonder how the mode profiling/matching was considered in the new layout.
I can see the distances between the components and lenses are largely different from the old ones.
This is OK if you plan to go through a new mode matching solution with new lenses.
But it takes
a certain amount of time.
Note that we don't care the distance after the last lens as the Rayleigh range there is supposed
to be long enough to allow this kind of change.
2. The huge frustration of the green alignment in the old setup was caused by the 3D beam steering
at the last two 2" mirrors. i.e. the beam elevation on the table does not match with the beam elevation of the cavity.
In order to avoid this, I suggest you to use three 45 deg 2" mirrors instead of two. In fact these mirrors are supposed to be used at 45deg incidence!
3. The incident green beam and the transmitted IR beam should share a same path as they
share a same cavity mode.
This means that you should use a harmonic separator for the transmitted light pick-off.
4. Use the harmonic separator for the fiber path too. Get the mirror spec from Jamie.
5. Since the optical window on the chamber has a wedge angle, the beam paths are not straightforward.
The cavity beams can't be moved as they are constrained by the arm cavity.
Probably there is almost no freedom to move even for the oplev beams.
It would be safe just to follow the old positions and angles on the window.
Make sure the beam on the drawing is realistic. The angles of the oplev beams in the old setup look strange.
Is there a possibility to replace the optical window so that it has an AR for 532 and 1064 at least???
6. I wonder if the rejected beam by the Farady have a realistic angle or not. Check it with the old setup.
It is definitely better to have a steering mirrror and a lens before the refl PD.
7. The IR QPD and trans PD are intended to be used for the low and high power detection.
I forgot which is which. So check the range of them and think about the power distribution.
8. We should have separated CCDs for IR Trans and Green Refl.
We had a terrible ghost green beam on the IR trans CCD.
Thus, think about the amount of ghost reflection and consider filtering if necessary.
I am working towards redesigning the endtables. I've attached the first version of the layout. As per Steve's comment I've tried to leave a 2" empty space on all sides of the table. It still has to be updated with the whole 40m layout to be more precise about the pickoff and the ingoing beam directions.
Mike and I installed all of the telescopes and launching hardware for REFL11, REFL33, REFL55, AS55, MCRef, POX11 and POP55. On Monday afternoon Steve will work with us on the fiber routing. Steve is buying some protective covers for the fibers.
Jenne, Mike and I installed all of the post holders we could today including: REFL11, REFL33, REFL55, AS55, MCRef, POX11 and POP55. We did not install AS110, POY or REFL165 because there are interferences that will require moving stuff around. We also did not mount POP22 because it is a peely wally ThorLabs PD that will be replaced by a strong, straight and right thinking LIGO PD in the fullness of time. We did move it out of the way however which is no more than it deserves. Next step this afternoon Mike and I will install all of the telescopes and launching hardware. Then with the help of Steve we will begin routing the fibers. The 1x16 splitter module will be here by next Monday, the laser by the following Friday and then we will light up the fibers.
I'm proposing split loom tubing that would run in the cable tray to protect the fibers inside of it. This tubing diameter in the cable tray can be 1.5-2" and out of the tray 0.75"
[Koji, Steve, Den]
TT alignment is fine, yaw damping is satisfactory, pitch damping is slow. We might want to add magnets to the mirror and attach blades to the frame for pitch edge current damping.
We are moving towards electronics testing.
Atm1, TT 1.5" high adaptor base will be back from the shop in 10 days.
Atm2, There is no PITCH damping, YAW edie current damping works well at 0.5 mm gap
Atm3, Adjustable Al -disc that contains a small magnet is purely designed.
We have to come up with a solution to have damping in PITCH
Jenne, Mike and I installed all of the post holders we could today including: REFL11, REFL33, REFL55, AS55, MCRef, POX11 and POP55. We did not install AS110, POY or REFL165 because there are interferences that will require moving stuff around. We also did not mount POP22 because it is a peely wally ThorLabs PD that will be replaced by a strong, straight and right thinking LIGO PD in the fullness of time. We did move it out of the way however which is no more than it deserves. Next step this afternoon Mike and I will install all of the telescopes and launching hardware. Then with the help of Steve we will begin routing the fibers. The splitter module will be here by next Monday, the laser by the following Friday and then we will light up the fibers.
Since the MC spots are good, I put the beam back on WFS 1 and WFS 2.
Also, I changed the indicators on the LockMC screen to reflect the change in elog 7289, where we added another on/off switch for the WFS so that the ASS could be on, but the WFS off. For the last month, the WFS could be disabled, but the MC screen's indicators would suggest that we were pushing very significantly on all 3 MC mirrors. Now the MC screen reflects reality a little better.
I also uncommented the WFS lines in the mcup script. Den had commented them out, but didn't elog about it! C'mon Den, please elog stuff!!!! (He confessed out loud the other day, but it still wasn't in the elog).
I'm leaving the WFS loops disabled (even though the MC autolocker tries to turn them on, I have them manually disabled using the extra on/off switch) since they're unstable. I'm in the process of figuring out what's wrong. So far, the WFS improve the MC alignment for a minute or two, and then they totally misalign the MC. This is a work in progress.
[Whomever took away this ND filter without elogging it was BAD!!! (Jamie, when we first found IPANG coming out of the vacuum during this vent, we moved some of the mirrors on the out-of-vac table in the IPANG path. Was the ND filter removed at that time? Or has it been out for much longer, and we never noticed because IPANG wasn't coming nicely out of the vacuum / was clipping on the oplev lens?)
I do not remember removing anything from that setup. We just moved some mirrors and lenses around
[Jenne, Evan, Den]
MC REFL beam is back on the PD, and the mode cleaner locks. It looks like we're a little high on the MC Refl camera, but the MC spots were measured, and don't look like they changed from Friday (or maybe Monday?), the last time they were measured. We took this to be acceptable MC alignment, and did not touch the PSL table's pointing.
The laser power reduction optics were removed, and placed out of the way on the PSL table (where do they belong?). PSL-POS and PSL-ANG aren't quite perfectly centered, but a beam dump had been in the way of that path, so I don't know if they drifted bad, or if it was a sudden thing. The beam is still hitting the QPDs though. After removing the beam power reducing optics, we recentered the MC REFL beam on the REFL PD, still not touching any PSL alignment. MC mirrors were aligned (Den did this work while I showed Evan around, so I don't know by how much), and MC Trans was maximized (really MC Refl was minimized, making sure that the unlocked MC Refl was the usual 4.something units on the EPICS readback.
We turned on the PZT high voltage supplies for the output steering PZTs and for the input steering PZTs. We left the OMC locking PZT supplies off, since we're still not using the OMC. Sadly, the beam coming out of the AS port looks clipped somewhere. [SELF: attach the videocapture shot when you get to work tomorrow] We tried moving PZT2's sliders, but nothing happened!!! I can move BS and the ITMs to get the beam mostly unclipped, but I really need to be able to move the PZTs, or at least one of them. IPPOS and IPANG beams are hitting the QPDs (although they're not centered perfectly), so the PZTs came back mostly to the same positions, but not exactly. Evan and I sat next to the input steering PZT controllers in 1Y3, and moved the sliders around. For most of the range, nothing changes on the LCD screen for either PZT2 pitch or yaw. Yaw can make 2 small steps near the far negative side of the slider, but nothing happens for most of the slider. Pitch really doesn't do anything for any part of the slider. We ensured that the LED labeled "CL ON" was not illuminated, next to the button labeled "closed loop", for all 4 controllers (PZT1 and 2, pitch and yaw). Sad!! I don't know if the LCD screen on the front panel of the PZT controllers is a readback of signal supplied to the PZTs, or of the strain gauges. I really hope it's the controller that's not working, rather than the PZTs themselves. The PZTs were fine before we vented, and Koji and I did our centering of the PZT range check during the vent, so they were fine then. What happened??? All PZT high voltage supplies were off during the pump-down. I turned them off yesterday, and Evan and I turned them back on tonight around 9:30pm or 10pm. What else could make them bad?
Without being able to move PZT2, just using BS and / or ITMs, I was unable to completely make the beam look nice on the AS camera. I came close, but it still seems a little bit funny, and I had to move the suspended optics quite a bit to find that place. This is not good.
[Jenne, Evan Hall]
Both IPPOS and IPANG beams are (after turning on the input and output PZTs) hitting their QPDs. However IPANG was saturating. We went down to take a look, and we had ~2.8mW incident on the QPD. There was an ND filter sitting unmounted, next to the diode, and an empty fork directly in front of the diode. Since IPPOS also has an ND filter in front, we stuck this ND filter back in. Now we are no longer saturating.
We're not hitting (yet) the center of these 2 PDs, but we're at least hitting the diodes, so it shouldn't be too hard to steer the input PZTs.
Whomever took away this ND filter without elogging it was BAD!!! (Jamie, when we first found IPANG coming out of the vacuum during this vent, we moved some of the mirrors on the out-of-vac table in the IPANG path. Was the ND filter removed at that time? Or has it been out for much longer, and we never noticed because IPANG wasn't coming nicely out of the vacuum / was clipping on the oplev lens?)
IFO P1= 1mTorr, CC1 = 5e-4 Torr. The IFO is ready for work. Be free to open the shutter and turn on HV
Jenne will double check the MC -REFL path for 1W power. There is a manual block on the PSL table.
Atm1, Pirani gauge at 9 h 10 min
Atm2, Pump configuration of pd#73 at 9h 35 min
What was the reasoning / resolution of the POP forward/backward beam? Are we going to have the right beam for DRMI locking?
From Koji's email to me:
"With the backward beam you can see the returning beam even when the PRM is misaligned. That's the only difference. Once the PRM is aligned both beams have the same information."
So, we should be fine.
The IFO will be at atmosphere overnight. The annuloses are pumped down. The access connector jam nuts are tightened to 45 ft/lbs
Actual pumpdown will start tomorrow morning.
I've installed Guralp readout box back and it turned out that it does not work with voltage provided from the rack (+13.76 0 -14.94). +/-12 voltage regulators inside the box convert it to -0.9 0 -12. I've connected the box to +/-15 DC voltage supply to measure seismic motion at the ETMY table. Readout box works fine with +/- 15.
I'm not sure what the problem is here. Den and I looked at it for a few minutes, before I went back to helping with putting doors on. The Sorensons are not supplying the rack power for 1X1. There are some flat cables which go from the fuses on the side of the rack up to the cable tray, and go elsewhere. Den is going to continue looking into this, but I think it's a moderately high priority, since lots of things should be getting served by that same power.
[The 40m Family]
The access connector and all heavy doors are back on.
Jamie put the regular viton EQ stops back on PRM, since he had to adjust the distance between the EQ stops and the PRM anyway. Jamie also waved an IR card near the IPANG steering mirrors in ETMY, but it was not possible to take a good photo. Jamie certifies that the beam is centered on both of those 2 optics.
I have centered IPPOS and IPANG QPDs.
All oplevs need a little realignment, especially ETMY, which had it's lens removed (Rana has a Wall of Shame photo of this, which is why it was removed by him). Steve will look into this tomorrow, after he starts pumping.
I have turned off all PZT high voltage supplies for in-vac PZTs: The input PZTs, the output PZTs, and the OMC PZTs (which weren't on, but I confirmed they were off).
I have also prepared the 3 low-power items for high power: MC refl's path was changed back to regular BS, AS camera was moved to its nominal position, and IPPOS has its ND filters back. MC refl and the AS camera will need to be realigned once we're actually at high power tomorrow afternoon.
Long vent, but good work everyone.
The SP table was found open this morning. Please, do not make optics dirty!
I cleaned up the tops of the SP table.
Stop storing your junk, boxes, laptops, etc. on the optical tables. This includes the big SP table. Please move all of that junk into racks or shelves, etc.
I've applied LQR approach to MC_L locking. Results show that LQR does not make MC_F signal smaller below 0.3 Hz in contrast with classical locking. This might indicate that in this frequency range we see sensing noise as LQR was provided with state-space model of MC only so it tries to reduce displacement noise. It is also possible that state-space model is not accurate enough.
Seismic noise on the ETMY table measured to be a few times higher then on the floor in horizontal direction in the frequency range 50 - 200 Hz. Attached are compared spectrums of X, Y and Z motions.