I think the ETMX slow machine might be dead. All of the regular FE readbacks are fine, and the c1iscex FE computer looks fine, but the slow readbacks are all whited out.
I turned off the damping loops for ETMX, since I don't have access to the watchdog disable/enable switch. I guess checking this out will be task #1 for Monday morning.
For lack of a better idea, I keyed the crate. The computer came back up just fine, ETMX is happily damped again.
I revived the ETMX simplant model, c1spx. It's running on cpu4 on c1iscex, and interfaces with C1scx via SHMEM.
The channel names for the simplant suspensions will be SUP, so the channel from this model will C1:SUP-ETMX_.
Next I'll try to get the ITMX and LSC ("LSP") simplant models running so we can run a "full" cavity simulation.
Sasha has been working on LSP, so we should be ready to do something with that soon. In the mean time she's going to fix up the SPX MEDM screens, since some channel names have been changed since it was last run.
Koji suggested systematic investigation of the ETMX suspension electronics. The tests to be done are:
So the ETMX satellite box is unplugged now, starting 530 pm PDT.
The satellite box was reconnected and the suspension was left with watchdog off but OSEM roughly centered. We will watch for glitches over the weekend.
We have positioned and applied epoxy to one ruby standoff on ETMX, for overnight curing according to the SOS standoff gluing procedure. This included:
Instead of trying to fix up a way of gluing the guiderod with the proper alignment, we chose to be more conservative and glue the standoff today, then switch the gluing fixture's arms tomorrow to glue the guide rod with the good fixture arm.
Additionally, we chose to glue one of the more assymetric standoffs on this first side. What I mean by this is: We have 3 ruby standoffs with grooves. Two of them have the groove about 1/8th of the way along their length, and one has it about 1/4 of the way. Since the second standoff is going to be glued while suspended, after pitch balancing, we figure that we want to use the more centered groove on that side, meaning we used one of the 1/8th standoffs today.
Unfortunately, we neglected to take any pictures :/
Today I took the picture of the glued ruby stand-off. The groove has not been invaded by the epoxy!
Jamie did computer magic. I burt restored scxepics, and restored ETMX damping.
I'm not sure why or when it was tripped, but I have restored the ETMX watchdog.
ETMX is currently in the clean room, the barrel is the tiniest bit submerged in acetone that will remove a guide rod, standoff, and side OSEM.
Additional inpsection of the standoffs on the flow bench did not provide any insight, pictures are in picasa. Here is a cropped version of a picture we took:
We should look at them under a microscope.
The magnet, guide rod, and standoff came off without too much force. However, some epoxy residue remains on the barrel. I didn't really want to scrape it off, so I've opted for more soaking. Much of the acetone had evaporated already, so I put some more - just to the point where the residue is submerged.
In the evening, I went into the clean room to check how it goes.
- The air around the table is quite warm like a hell. Is this normal?
- I checked how the scattered epoxy spots look like. They were not touching the bath anymore due to evaporation.
- I scraped the spots with the tweezers there. They were easily removed. The particlates on the side barrel were wiped by a wipe with aceton. (Result: Attachment 3)
- Then looked at the other side. I poked the standoff with the tweezer. It was easily removed. I don't think the bond was too weak. Just the area of the bond was so tiny.
- Also residue was scraped by a tweezer and wiped with a cloth. (Attachment 2)
- The removed stand off is in the stainless bowl together with the parts that Eric removed.
- I didn't want to leave the optic in the aceton fume. It was placed on a metal donuts for a 3" optic. (Attachment 4)
- I couldn't find a vacant clean glass jar for the lid. So, a foil hut was built. We should be very careful not to scratch the optic when we remove the hut. (Attachment 5)
- The aceton bath was covered with the foil as it was. (Attachment 6)
We put ETMX back in its tower, and confirmed its balance. It might be pointing a teensy bit upward, but it is way less than the DC pointing offset we see when we put the OSEMs in the towers (since the PDs and LEDs have some magnetic bits to them).
Discussions are ongoing as to where the ETM should sit on its table, but we'll probably toss it into the chamber later this evening.
I took ETMY out of the magnet gluing fixture, and put it in a ring, in the foil house. It is ready to have the wire winched and get balanced at our convenience.
The updated status table:
Looking at the sensor and oplev trends over the weekend, there was only one event where the optic seems to have been macroscopically misaligned, at ~11:05:00 UTC on Oct 19 (early Saturday morning PDT). I attach a plot of the 2kHz time series data that has the mean value subtracted and a 0.6-1.2 Hz notch filter applied to remove the pendulum motion for better visualization. The y-axis calibration for the top plot assumes 1 ct ~= 1 um. This "glitch" seems to have a timescale of a few seconds, which is consistent with what we see on the CCD monitors when the cavity is locked - the alignment drifts away over a few seconds.
As usual, this tells us nothing conclusive. Anyways, I am re-enabling the watchdog and pushing on with locking activity and hope the suspension cooperates.
I took data of the ETMX SUSPOS, SUSPIT and SUSYAW channels while driving each of the 4 face coils. I manually turned off all the damping except the side.
Excitation: I used white noise bandpassed from 0.4 to 5 Hz in order to examine the responses around the resonance frequencies. To avoid ringing things up too much, I started with a very weak drive signal and gradually increased it until it seemed to have an effect on the mirror motion by looking at the oplev signals/sensor RMS values on the SUS screen; it's possible I'll need to do it again with a stronger signal if there's not enough coherence in the data.
Finding the matrix: The plan is to estimate the transfer function of the coil drive signal with the sensed degrees of freedom (specified by the already diagonalized input matrix). This transfer function can be averaged around the resonance peak for each dof to find the elements of the matrix that converts signals to dof responses, (the "response matrix", which is the inverse of the output matrix). Each column of the response matrix gets normalized so that the degrees of freedom influence the drive signals in the right ratio.
UR osem IR shield glass is pushed back. It came out of its clip holder. The magnet is free.
Atm2, UL & LL magnets centered poorly. Almost hinging on opposite sides.
UR & LR centered well. There have plenty of room to move in an earth quake.
I believe that the Xend aux laser was turned off earlier today, for Steve's work swapping out the oplev. When I went down there, the red "off" LED was illuminated, and the LCD screen was showing something. I pushed the green "on" button, and I immediately got green.
Also, I saw that the 24Hz roll mode was very rung up on ETMX. I looked at the FM5 "bounce roll" filter, and it had some old values, 12Hz and 18Hz for the resonant gains. All other optics have the proper 16Hz and 24Hz frequencies. I copied the BS oplev bounce roll filter over to ETMX pit and yaw (both were wrong), and loaded them in. The mode is starting to ring down.
I've turned the ETMX oplev servos off for the time being. (At the input side, so that no scripts will accidently turn it on).
Thus, only the local damping is being applied, let's see if we see any kicks...
During this time period, it looks like there was maybe one excursion. Here are the individual OSEM signals, which I think to be calibrated to microns.
When I'm doing other things, I'm going to intentionally leave the watchdog tripped, and see if anything happens.
During a ETMX kick that just occured, with only local damping on, the slow VMon channels didn't show any noticable change.
How many volts does it take to pitch the ETM by 5 urad?
By looking at the oplev and Vmons before and after a step of 90 counts in SUS-ETMX_PIT_OFFSET, I observe:
UL: +1.53mV / urad
UR: -1.94mV / urad
LR: -1.54mV / urad
LL: +1.92mv / urad
The random error associated with these measurements is ~0.02mV
So, the ~7urad urad shift seen in my earlier post would mean a change of around 10mV in the Vmon signals, which isn't evident in the traces. So, this is possibly a piece of evidence in favor of a real mechanical shift, rather than an electronic glitch.
I took loop measurements of ETMX pit and yaw, and set the upper UGF to be ~6Hz for both. This required a pitch gain of 25, and a yaw gain of 16.
The spectra look similar to what they were before Steve did the swap.
For some reason or another, I decided that we should see if the optical lever servos were injecting too much noise into the test masses. The ITMs are much worse than the ETMs and I am afeared that they might be making the main noise for our arms in the 20-40 Hz region. Jenne is checking up on these feedback loops to see what's up.
To estimate the actuator gains of the mirrors, I turned on 1 count drives from LSC/CAL oscillators into the LSC drives of each test mass at the frequencies shown in the plot with the resulting peaks showing up in in POX/Y with the single arm locks in red. I will leave these going permanently, but with 0.1 count ampltiudes; we need to make it so in the scripts.
I'm in the process of filling this table
I should replace ETMX He/Ne laser
The ETMX oplev returning beam is well centered on the qpd. We lost this signal 2 days ago. I will check on the qpd.
ETMX sus damping restored. c1iscex computer is down
The ETMX oplev returning beam is well centered on the qpd. We lost this signal 2 days ago. I will check on the qpd.
ETMX sus damping restored
As reported a couple days ago, the ETMX IO chassis has no timing signal. This is why we there is no QPD signal and why we turned off the ETMX watchdog. In fact, I believe that there is probably nothing coming out of the ETMX suspension controller at all.
I'm working on getting the timing board replaced. Hopefully today.
Kiwamu and Steve,
This is one of the better oplev set up we have. Single bounce from TM, no mirror on stack. One lens and one mirror on ISCT. Old Uniphase 1103P laser on heafty 3" od mounts.
Somewhat big ~5-6 mm spot on qpd in 1,300 counts.
The intensity noise effect can be seen at 1 Hz and 3-20 Hz
Oplev servo was OFF
ETMX oplev had 6 mm diameter beam on the qpd. I relayed the beam path with 2 lenses to get back 3 mm beam on the qpd
BRC 037 -100 Bi _concave lens and PCX 25 200 VIS do the job. Unfortunately the concave lens has the AR 1064.
ETMX oplev qpd gain has to be increased.
Atm3, Oplev sum read 12,000 counts when the qpd was disconnected ?
Dark qpd was zero and normal He/Ne incident on qpd was 1,730 counts.
This QPD circuit (D980325-C1 ) uses the nice OP497 Quad FET opamp as the transimpedance amplifier. It has a low enough current noise, such that we can increase the resistors (R1-4) up to 100k and still be Johnson noise limited. We should also make sure that the compensation caps (C3-6) are ~2.2 nF so as to not destabilize the opamp. f_low = 1/2/pi/R/C = 730 Hz.
I will do the swap later today unless someone else gets to it first. (note: check for oscillations w/ fast scope probe after installing)
I did these modification tonight. The slideshow of some images is attached. Instead of 100k, I used 97.6k thin film, since this seemed like an oddball size that doesn't get used otherwise. I forgot to measure the dark noise of the quadrants before doing the swap, but comparing the pit/yaw/sum before/after the swap shows that the signal is basically unchanged (since pit/yaw is normalized by SUM), but that the noise is lower by a factor of a few above 100 Hz due to being above ADC noise now. Previously, it was bottoming out at ~10 prad/rHz. Since the signal is unchanged, I guess that the calibration and therefore the loop gain should not have changed either...
And the sum went up by almost 10x as expected from the resistor change.
Coherence at 1 and 2-3 Hz only. He/Ne laser intensity noise is not an issue.
I'm still not super happy with the low power level of the ETMX oplev, so I went to investigate.
This is a 3-year plot. The first ~year in the plot, the oplev sum is ~15000 cts, and in the most recent year, it's ~1000 cts. A new HeNe was installed in May 2011 (elog 4686), with an output of 2.6mW, after the old one had died. When the new one was installed, Steve said that it was giving ~1400 counts, so maybe 1000 isn't too, too embarrassing. There is, however, a lens on the injection side, which is AR coated for 1064. The power before this lens (measured with the Ophir, set to 632nm) was ~2.6mW. The power after this lens was ~1.5mW. Now THAT is embarrassing. I'm adding replacing that lens to the to-do list (elog 6595), although I don't want to do it until such a time (maybe in an hour, maybe in a few days?) when I've got the Xarm locked / aligned, so I can nicely re-center the oplev. UPDATE: The lens is a KBC 037 (-100) lens, and the sticker on the lens mount says coated for 1064. We don't have any KBC037's in the visible lens kits, so we need to get one before I can do this replacement (PURCHASED 10pm).
There is an elog (elog 5004) from July 2011, which mentions that these channels have not been saved for a long time, so that's why there's the year-long gap.
Anodized aluminum dumps replaced by 6 razor beam dumps.
Two more razor beam dumps added this afternoon. The picture will updated tomorrow.
There are 9 razor beam dumps at ETMY-ISCT
I added two green glass absorbers. The oplev centering may need a touch up when it is well aligned.
This has been edited several times over the last several hours, as I try to change different parameters, to see if they affect the movement of ETMX. So far, I don't know what is causing the motion. If it is there, it is only present when the LSC is engaged, so I don't think it's wobbling constantly on a twisted wire.
FINAL EDIT, 9:10pm: The arm ASC was turning itself on when the arms were locked. Whelp, that was only 3 hours of confusion. Blargh.
For his penance for leaving the arm ASC engaged, Q has made a set of warning lights on the LSC screen, right next to the ASS warning lights.
ETMX might be having one of those days today, which is lame.
So far tonight, I have run the LSC offset script, set the FSS slow value to +0.2, and run the arm ASS scripts. Nothing too crazy I think.
Sometimes when I lock the single arms, the ETMs move around like crazy. Other times, not. What is going on here??? The ETMs don't move at all when they are not being actuated on with the LSC.
In this screenshot you can see the end of a POX/POY lock stretch where everything was nice and good. Then, the arms were unlocked, and they have a bit of a DC offset. After settling from that step, they continue sitting nice and still. Then, I relock the cavities on POX and POY a little before -4 minutes. ETMY takes a moment to pull itself together, but then it's steady. ETMX just wobbles around for several minutes, until I turn off the LSC enable switch (happened after the end of this plot).
I'm not going to be able to lock like this. Eeek!
This is somehow related to light being in the Xarm. This next plot was taken while the arms were held with ALS in CARM/DARM mode.
I closed and re-opened all 3 green shutters. Now (at least the last 8 arm locks in the last 6 mintues) ETMX has never gone wobbly, except for a little bit right after acquisition, to deal with whatever the DC offset it. Why is this changing?
The arms were fine for one long ~30 minute lock while I stepped out for dinner. At some point after returned, the MC lost lock. When the arms came back, ETMX was being fussy again. Then, it decided that it was done.
In this plot, at -1 minute I started the ASS. Other than that, I did not touch any buttons at all, just observed. I have no idea why at about -3 minutes the bad stuff seems to go away.
I was curious if it had to do with the DC pointing of the optics, so I unlocked the arms, put ETMX about where it was during the long good lock stretch, then reaquired lock. I had to undo a little of that so that it would lock on TEM00, but at the beginning of the lock stretch (starting at about -3) the pitch is about the same spot. But, the oscillations persist. This time it was clear that the oscillations were around 80 mHz, and they started getting bigger until they settled to an amplitude they seemed to like.
Seems pretty independent from FSS temp. There are 3 lock stretches in the next plot (easier to see by looking at the Yarm transmission, green trace). The first one, the FSS slow was at 0.35. the middle one, it was around 0.05. The last one, it was around -0.4. Other than the different DC pointings (which I don't know if they are related), I don't see anything qualitatively different in the movement of ETMX.
Brief summary, some pictures and such follow in the daytime.
The epoxy needs at least 12 hours of room temperature air curing, so no touchy until 3:30PM on Jul 28!
Attachment #1 - After multiple trials shimming the magnet gluing rig with teflon spacers, we think that we managed to find a configuration in which the side magnet edge is between 0.25 mm and 0.5 mm from the groove in the ruby wire standoff in which the wire will sit.
Attachment #2 - Zoomed in view of the side magnet.
Of course we won't know until we suspend the optic, but we believe that we have mitigated the misalignment between the side OSEM axis and side magnet.
The short term plan is to try and suspend ETMY in the end chamber and have a look at the alignment between all magnets and OSEM coils for it. Once the epoxy on ETMX is cured, we will try and suspend the optic again, this time taking extra care while tightening the wire clamps.
Unrelated to this work: Bob just informed me that we had left the air bake oven on overnight - this unfortunately melted the plastic thermocouple inside.
The epoxy arrived. Eric managed to remove the excess glue below the guiderod with a razor blade (see attachment 1). The magnet and dumbell that came apart were reglued successfully and passed the stregth test of picking up the magnet from the table by the dumbell, so the magnet was glued back on the optic and is setting in the gluing apparatus (see attachment 2).
We double checked the polarity against the side magnets on ETMY. Because of the gluing position strategy (a fixed distance toward the HR side from the groove location), the other side magnet appears slightly below the center of the gluing barrel, which after some discussion with Koji was determined to be ok.
One step forward, two steps back...
While attempting to suspend ETMX, I broke off a side magnet
It is now gluing
(This is *not* the one that was previously glued. I.e., now both ETMX side magnets have been reglued)
I have put an offset of 1000 counts to C1:SUS-ETMX_ALS_OFFSET. This actually misalign the mirror a lot.
While the offset is applied. I adjusted the balance of the coil matrix.
UL 1.580 UR 0.620
LL 0.420 LR 1.380
> ezcaread C1:SUS-ETMX_TO_COIL_0_0_GAIN
C1:SUS-ETMX_TO_COIL_0_0_GAIN = 1.58
> ezcaread C1:SUS-ETMX_TO_COIL_0_1_GAIN
C1:SUS-ETMX_TO_COIL_0_1_GAIN = 0.62
> ezcaread C1:SUS-ETMX_TO_COIL_0_2_GAIN
C1:SUS-ETMX_TO_COIL_0_2_GAIN = 0.42
> ezcaread C1:SUS-ETMX_TO_COIL_0_3_GAIN
C1:SUS-ETMX_TO_COIL_0_3_GAIN = 1.38
Now, we can keep TEM00 for green with +/-1000counts of push although the fast step of the offset make the lock lost.
It turned out that the step longitudinal input temporary misalign the mirror in pitch because the length and pitch are coupled.
I guess that we don't excite pitch if we push the mirror slowly. Eventually, we need f2p transfer function adjusted in the output matrix.
The oplev and the LSC are off.
I was trying to take a few more IR transmission scans with ALS when the ETMX got kicked again. I'm not sure how to fix this, so for the time being, I'm leaving the Oplev servo and the LSC turned off. The oplev spot looks really far off center especially in yaw, the yaw error is ~ -80.
ETMX has some periodic oscillation. It's damping was found tripped this morning.
We tracked this down to the power normalization stuff that Yoichi added over the weekend.
With a non-zero normalization factor, and a small TRX transmission, the input the XARM controller gets really big. When XARM is then triggered, a huge impulse is sent into the SUS_ETMX_LSC input, which causes the Vio2 filter in FM0 to ring like crazy. This probably also explains why Yoichi was seeing trouble locking the arm when the normalization is on
The solution, as Yoichi also mentions, is probably to trigger the normalization like we trigger the rest of the boost filters.
We have talked about the drift of ETMX sus on the Wednesday meeting.
It has stopped moving on Jan 8, 2015 and it has been reasanable stable since than.