MC2 damping restored,  MZ locked and the arms are flashing now.

  Earthquake mag 4.0 at Lennox, Ca     trips MC2 watchdogs       http://quake.usgs.gov/recenteqs/Quakes/ci10411545.html

See 40m accelerometers as they see it.

 The EQ did not change the input beam pointing. All back to normal, except MC2 wachdogs tripped again.

 Round 3 for the day of MC2 watchdogs tripping.

Summary:

The analog de-whitening filters for MC2 are different from those on the other optics (i.e. ITMs and ETMs). They have one complex pole pair @7Hz, Q~sqrt(2), one complex zero pair @50Hz, Q~sqrt(2), one real pole at 2.5kHz, and one real zero @250Hz (with a DC gain of 10dB).

Details:

I took the opportunity last night to measure all 4 de-whitening channel TFs. Measurements and overlaid LISO fits are seen in Attachment #1. 

The motivation behind this investigation was that last week, I was unable to lock the IMC to one of the arms. In the past, this has been done simply by routing the control signal of the appropriate arm filter bank (e.g. C1:LSC-YARM_OUT) to MC2 instead of ETMY via the LSC output matrix (if the matrix element to ETMY is 1, the matrix element to MC2 is -1).

Looking at the coil output filter banks on the MC2 suspension MEDM screen (see Attachment #2), the positions of filters in the filter banks is different from that on the other optics. In general, the BIO outputs of the DAC are wired such that disengaging FM9 on the MEDM screen engages the analog de-whitening path. FM10 then has the inverse of the de-whitening filter, such that the overall TF from DAC to optic is unity. But on MC2, these filters occupy FM7 and FM8, and FM9 was originally a 28Hz Elliptic Low-pass filter.

So presumably, I was unable to lock the IMC to an arm because for either configuration of FM9 (ON or OFF), the signal to the optic was being aggressively low-passed. To test this hypothesis, I simply copied the 28Hz elliptic to FM6, put a gain of 1 on FM9, left it engaged (so that the analog path TF is just flat with gain x3), and tried locking the IMC to the arm again - I was successful. See Attachment #3 for comparison of the control signal spectra of the X-arm control signal, with the IMC locked to the Y-arm cavity.

In this test, I also confirmed that toggling FM9 in the coil output filter banks actually switches the analog path on the de-whitening boards.

Since I now have the measurements for individual channels, I am going to re-configure the filter arrangement on MC2 to mirror that on the other optics. 

Unrelated to this work: the de-whitening boards used for MC1 and MC3 are D000316, as opposed to D000183 used for all other SOS optics. From the D000316 schematic, it looks like the signals from the AI board are routed to this board via the backplane. I will try squishing this backplane connector in the hope it helps with the glitching MC1 suspension.

GV Aug 13 11:45pm - I've made a DCC page for the MC2 dewhitening board. For now, it has the data from this measurement, but if/when we modify the filter shape, we can keep track of it on this page (for MC2 - for the other suspensions, there are other pages). 

I briefly checked the MC2 analog dewhitening filters.

It turned out that the switching of the dewhitening filters from epics worked correctly except for the UR path.

I couldn't get a healthy transfer function for the UR path probably because the UR monitor at the front panel on either the AI filter or the dewhitening filter maybe broken.

Need a check again.

We need to re-look at this new MC autolocker stuff, and the new MCL filters.

MC2 is getting kicked up (sometimes the watchdog trips, sometimes it just comes close) pretty regularly.  I'm not sure yet what is causing this, but we need to deal with it since it's pretty obnoxious.

Mirko and  Den are measuring MC_F, which they will report about later, but I noticed that MC2 is totally crazy right now.  It shouldn't matter that they are doing things (like unplugging the feedback to the PSL's PZT), because we actuate on the laser, not on the MC.  I disabled the MC autolocker before they started working. 

Anyhow, somehow MC2 got kicked up (whatever, that happens), but it won't re-damp.  I think it's the WFS.  The yaw output from the WFS is truely crazy. 

I have disabled the WFS output / ASC input on the MC SUS screens, and MC2 was then able to damp.  My disabling only the MC2 WFS input at first kicked up MC1 and 3, so I disabled all of the WFS stuff, and all 3 MC mirrors are again happy. 

SURESH: FIX ME!  (signed, The WFS)

The Problem

Turning off the WFS servo loops usually should be done using the 'mcwfsoff' script.  The script takes care of switching off the integrators and Clears the History.  

'mcdown' and 'mcup' scripts run the 'mcwfsoff' and 'mcwfson' scripts so when the MC unlocks the WFS servos are shutdown and restarted properly.  However if the MC autolocker script is suspended by pressing the Enable/Disable switch in the LOCKMC screen and then the MC unlocks, it results in the WFS servo integrators accumulating large values.  If these values are passed through the ASC filter banks the optic will get a pretty huge kick.

The Solution

I have added some indicators which will let us know if the WFS Servo Filter outputs are larger than +/-1000.  When engaging the WFS loops the user has to take care to Clear History in the servo filter banks if these indicators are steadily Red.   before engaging the WFS Servo loops ensure that the servo filter outputs are zeroes. 

Koji and I discussed whether it would be useful to run the 'mcwfsoff' script when the Disable button is pressed in the autolocker.  His recommendation is that we should keep  the autolocker script simple and that user has to be cautious when switching on the WFS servos and when directing the ASC outputs to the suspensions.

MC2 is having a bad day, and I'm not yet sure why.  It's to do with the damping though.  When the damping is off, after a little while it will settle to ~30mV or so on the Watchdog screen.  When I enable all of the outputs and then turn on the damping, the optic gets kicked up.  It's like there's a minus sign error somewhere, maybe in a bad burtrestore?  This has been going on since I did my morning bootfest.

It's started to sit down and play nicely now.  Is someone doing magic remotely that is fixing things that I hadn't figured out yet?

The MCL path of MC2 was in a strange state as the filters were activated as if it is in lock even though we had no lock. So I manually ran "mcdown". This reset the filters of the MCL path.

I'm running the MC2 loss map scripts on pianosa now. The camera server is throwing an error and is not grabbing snapshots :(

Update: I finished taking the readings for MC2 loss map. I couldn't get the snapshots with the script, so I manually took some 4-5 pictures.

Can you please be more specific about what the error is? Is this the usual instability with the camera server code? Or was it something new?

The camera server keeps throwing the error: failed to grab frames. Milind suggested that it might a problem with the ethernet cable, so I even unplugged it and connected it again; it still had the same issue. One more thing I noticed was, it does take snapshots sometimes with the terminal command caput C1:CAM-ETMX_SNAP 1, but produces a segmentation fault when ezca.Ezca().write(C1:CAM-ETMX_SNAP, 1) ezca.Ezca().write(CAM-ETMX_SNAP, 1) is used via ipython. When the terminal command also fails to take snapshots, I noticed that the SNAP button on the GigE medm screen remains on and doesn't switch back to OFF like it is supposed to.

I think ezca.Ezca().write() takes the string "CAM-ETMX_SNAP" as an argument and not C1:CAM-ETMX_SNAP. See this, line 47. Are you sure this is not the problem?

In my script I have used "CAM-ETMX_SNAP" only; while entering it in the elog I made a mistake, my bad!

I aligned MC2 suspension by 0.01 in pit and yaw to align the MC better to the PSL beam. Then I turned the WFS back on. The beams are not centered on the WFS heads.

Nic and Gabriele ought to send their SURF some example code (in April) for how to start redesigning the WFS telescopes so that we can order some optics in early June.

I've also turned on the MC2 TRANS path to gather some data over the weekend on how well or bad it works. Please turn it off on Monday.

 MC2_TRANS path in WFS servo turned OFF.

Summary:  I calibrated MC2 pitch and yaw offsets to spot position in mm. Here's what I did:

1. Changed the MC2 pitch and yaw offset values using  ezca.Ezca().write('IOO-MC2_TRANS_PIT_OFFSET', <pitch offset value> ) and ezca.Ezca().write('IOO-MC2_TRANS_YAW_OFFSET', <yaw offset value> )
2. Waited for ~ 700-800 sec for system to adjust to the assigned values
3. Took snapshots with the 2 GigEs I had installed - zoomed in and zoomed out. (I'll be using these to make a scatter loss map, verify the calibration results, etc)
4. Ran the mcassDecenter script, which can be found in /scripts/ASS/MC. This enters the spot position in mm in the specified text file.

Results:  In the pitch/yaw vs pitch_offset/yaw_offset graph attached,

• intercept_pitch = 6.63 (in mm) ,  slope_pitch = -0.6055 (mm/counts) 
• intercept_yaw = -4.12 (in mm) ,  slope_yaw = 4.958 (mm/counts) 

Around 6PM on the 27th, I found that the C1:IOO-MC_RFPD_DCMON had risen to about 2.5V.   I checked the trend of MC2 sensors and found that  between 2PM and 6PM, MC2 had drifted in a strange way.  And also that the alignment had grown worse over several days.   I also noticed that the spot on the MC2F camera had shifted to the left.

I attempted to correct the alignment (decrease the C1:IOO-MC_RFPD_DCMON to ~0.5V ) by just moving the MC2 and succeeded!! So it is quite likely that most of the slow MC drift is arising due to MC2 table drift.

I decided to try and close the MC2_TRANS QPD to MC2 loops separately to see if MC alignment becomes stable  But several screens needed to be fixed before we could try anything.   So I fixed C1IOO_WFS_MASTER,  C1IOO_WFS_INMATRIX and ...OUTMATRIX screens.  Deleted the older ones to avoid confusion.

In this process I noticed that the directory of $screens$/c1mcs/master contains copies of older C1SUS_MC1 , 2. and 3 screens which look very similar to the new autogenerated screens.  Some of the links in the WFS screens were pointing to the old screens.  I have redirected the links and I will delete these in a couple of days, if no one objects.

Olympus SP570 UZ - without  IR blocker, set up as Atm.3  Camera distance to MC  face ~85 cm,  IOO-MC_TRANS_SUM 16,300 counts, Lexan cover on not coated viewport.

Image mode: RAW + JPG,  M-costum,  manual focus,  Lens: Olympus 4.6 - 92 mm, f2.8 - 4.5,  Apeture: F2.8 - 8,  Image pick up device: 1/2.33" CCD (primary color filter)

Atm.1,       212k.jpg of raw 15 MB,  exp 0.025s,   apeture 2.97,  f 4.6,   iso 64,  

Atm.2,        Copied through my Cannon S100  (  3.3 MB.jpg of raw from UFraw photo shop )I will look up the original raw file for details.

I found the IMC was largely misaligned and was not locking. The WFS feedback signals were saturated and MC2 was still largely misaligned in yaw after resetting the saturation.
It seemed that the MC WFS started to put the large offset at 6:30AM~7:00AM (local).
MC2 was aligned and the lock was recovered then the MC WFS seems working for ~10min now.

This afternoon I felt like saying hello to the input mode cleaner. So I decided to center the spot on MC2.

Motivation

MC has 6 alignment dofs. 4 of them are controlled by the WFSs. Remaining 2 appears at the spot position on MC2.
If the spot on the MC2 is fixed, the beam hits the same places of three mirrors. If the mirrors are completely fixed
in terms of the incident beam, I suppose the reflected beam is also fixed. This makes the WFS spots more stable.
Then I feel better.

Today's goal is to confirm the behaviour of MC such as dithering amplitude, response of the couplings to the alignment,
behavior of the WFS, and the transmitted power.

Method

1) Turned off MC auto locker. Turned off MC WFS as the WFS servos disturbs my work.
2) Dithered MC2 in Pitch and Yaw using DTT. There looks elliptic filter (fc=28Hz) in the ASC path, I used 20Hz-ish excitations.
- C1:SUS-MC2_ASCPIT_EXC 100cnt_pk@19Hz
- C1:SUS-MC2_ASCYAW_EXC 100cnt_pk@22Hz
3) Looked at C1:SUS-MC2_MCL_OUT to find the peaks at 19Hz and 22Hz. These are caused by alignment-length coupling.
If they are minimized I assume the spot is somehow centered on MC2.
Note: This may not be the true center. The suspension response should be investigated. But this is a certain reporoducible spot position.
Note: I should use ezcademod in order to obtain the phase information of the dither result.

4) Move MC2 Pitch for certain amount (0.01cnt) by the alignment slider. Align MC1/MC3 to have max transmittion.
5) If the Pitch peak got lower, the direction of 4) was right. Go further.
5') If the Pitch peak got higher, the direction of 4) was wrong. Go the other direction.

6) Repeat 4)&5) for Yaw.

Result

After the adjustment, the couplings got lower about 10 times. (Sorry! The explanation is not so scientific.)
Next time I (or someone) should make a script to do it and evaluate the coupling by the estimated distance of the spot from the center of the mirror (the center of the rotation).
I have not seen visible change in the spectrum of C1:SUS-MC2_MLC_OUT.

MCT QPD issue

By the spot centering, I could expected to see some improvement of the transmittion. But in reality, there was no change.
In fact, the transmittion power was getting down for those weeks.

I checked WFS and MCT paths. Eventually I found that a couple of possible problems:
1) MCT Total output varies more than 10% depending on the spot position on MCT QPD.
2) Just before the QPD, there is a ND1 filter.
This may suggest that:
a) Four elemtns of the MCT QPD have different responses.
b) The ND filter is causing a fringe.

So far I aligned the ND filter to face the beam. The reflection from the filter was blocked at a farther place.
Still the output varies on the spot position. Probably I have to look at the QPD someday.

So far the spot on the QPD was defined so that I get the maximum output from the QPD. This is about 8.8.
As I touched the steering mirrors, the X and Y outputs of the QPD are no longer any reference.

For now, I closed the PSL table. The full IFO was aligned.

I made good on my threat from yesterday to convert the MC2 suspension controller to the library part.  Whatever changes were in MC2 were thrown out, although they are archived in the SVN.  Again, this kind of undocumented breaking is forbidden.

Change was committed to SVN, and c1mcs was recompiled/installed/restarted.

While Kevin is working on the MC2 electronics chain - we disconnected the output to the optic (DB15 connector on coil driver board). I decided to look at the 'free' freeswinging MC2 OSEM shadow sensor data. Attachment #1 suggests that the suspension eigenmodes are showing up in the shadow sensors, but the 0.8Hz peak seems rather small, especially compared to the result presented in this elog.

Maybe I'll kick all 3 MC optics tonight and let them ringdown overnight, may not be a bad idea to checkup on the health of the MC suspensions/satellite boxes... [MC suspensions were kicked @1207113574]. PSL shutter will remain closed overnight...

For whatever reason, the autolocker didn't turn the tickle off for several hours. Seems to work okay now. The linked plot suggests that the coil balancing on MC2 is pretty lousy.

 [Jenne, Manasa]

While we were trying to relock the MC after Jenne put back the RF box, we found there was some mysterious motion in MC2. After spending time trying to figure out where this was coming from, the source was found to be at LOCKIN2 of MC2 suspension "The MC TICKLER" that was left enabled. This was turned OFF and MC locked just fine after that.

EDIT JCD:  The Tickler should be disabled, if the autolocker is disabled.

 Sounds like this was just incidental, since the MC locked fine also with the tickler enabled for weeks.

The tickle is disabled by the down script, but there's no way to correctly handle all possible button pushes. If you want to disable the autolocker for some reason you should run mcdown before trying to lock. This will set up things with the correct settings.

 Isn't that backwards?  Shouldn't the tickler be enabled by the down script, and disabled by the up script?  Either way, the problem was that, after I acquired lock, the tickler was causing the transmitted power to fluctuate by ~20%, and often the MC would lose lock after a minute or so.  Also, the WFS definitely couldn't be enabled by hand. 

Anyhow, I'll try to keep in mind that this exists, so I'm not stymied by it again.

 You're right - down turns it on. Still, the fact that the same tickle recently causes a problem and didn't make 20% power fluctuations until now tells me that its not that the tickle amplitude is too large. Whatever changed recently is the problem.

Last night while noise hunting, Rana found that the MC2 trans beam has been left for an unknown length of time just hitting the side of the black enclosure box.  Today I put a brand-spankin'-new razor dump on the MC2 table, to dump the beam.

We noticed last week that the MC2 trans camera has pitch and yaw swapped; I rotated what I thought is the correct camera by 90 degrees clockwise (as viewed from above, like in the attachment), but I now have doubts. It's the camera on the right in the attachment.

The left one is analog and 90deg rotated.

See also: This issue tracker

Just felt a big "kerplunk" type ground-shaking, presumably from all the antics next door.  MC2's watchdog tripped as a result.  The watchdog has been reenabled.

People complained about the MC instability: If we aligned the MC, it locked nicely for a while.
Then suddenly you find that it got totally misaligned with the order of 0.2 with the alignment slider.
This misalignment usually happens for MC2, but it happend on MC3 once.

Surprisingly to me, this instability happened even without MCL and WFS, not only once but at least three times.
This suggests that the suspensions are the cause of the trouble.

I played with the MC2 suspension for a while in the afternoon. It seems that it has a hysteresis (or say, bistablity).
And the nominal alignment of MC2 seems close to the point where the transition happens. (Dunno why)

I further played with MC2 and found that a step of POS actuation by +/-10000 induces this transition go and back.
When the POS kick is in the negative direction (-10000), the MC2 seems to return to the preferrable
position. Therefore, I applied DC position force of -5000 to pull the mirror a bit from the nominal position.

I let the MC locked for ~4hours without MCL and WFS, it kept good alignment and the lock was stable
except for unlocks because of the activties by Den and Ayaka.

All of them has been done without previous monitor data as the tools were available.

The MC2 situation is not conclusive but we should check how we can prevent the bistable transition
by restricting MCL/WFS.

During a lull in Koji vs. The Arm, I switched on the MC2_TRANSQPD feedback path to check out its UGF. In the past months, when its been on, it has had a gain of ~0.03 - 0.1.

Today, I found that with the gain turned up to 11, it has a ~1 minute step response time (as you see in the above Strip chart). So its had a UGF of ~2 hours or so during the times when we thought it might be doing bad or good or magic.

I leave it on now to see if it behaves well over the next days. Let's see if Steve thinks its good or not based on his trend monitoring.

** also touched up the PMC pointing (using the PMC REFL image / please never align the beam into the PMC without this camera image)

We turned on the MC2_TRANS paths for both PIT/YAW tonight.

I turned off the BLP200 and turned on the RLP7 (RLP always are better than BLP). G_PIT = -0.111, G_YAW = 0.111. On Monday, let's let Steve look at the trends and determine if this centering servo is bad or good.

I should've included this in my Thursday night ELOG... That evening, I aligned the mode cleaner with reasonable MC1/3 spot positions, and the MC2 spots very close to centered, and recentered the WFS and MC2 Trans QPDs. The mode cleaner held up very well over the course of that evening, even when actuating CARM on MC2 with WFS engaged (which previously wasn't very stable when the WFS weren't well aligned).

The MC was showing slow but periodic alignment drifts and eventually unlocked around noon. I looked up the alignment trend (Attach: 2 day trend)

MC_TRANS_PIT_ERR and MC_TRANS_Y_ERR show that the MC_TRANS servo slowly drifted the IMC alignment causing it to lose lock from time to time (mostly in yaw).

To confirm that the drift was NOT due to off-centering in the MC2_TRANS QPD, I turned off the WFS servo, moved MC2 to recenter the trans beam on the QPD, and re-enabled WFS servo.

MC_TRANS path in WFS is still left enabled.

This is a 4-day trend. I don't see any difference here which is significant. My guess is that the MC_TRANS servo gain is so low that its not really doing anything.

I'll turn it on periodically this week and then on Monday people can look at the trend again to see if they can identify when the servo is ON and when its OFF.

[From yesterday]

The MC had not been stable lately with WFS drifting constantly. I checked the servo and found that the MC_TRANS path was still running. It turned out that the autolocker script enables the TRANS path in the locking process. I have turned the MC_TRANS path servo inputs OFF and now it is no more a part of the WFS servo.

P.S. Jenne fixed the PMC alignment mostly in pitch to bring it up to 0.81 from 0.77. But the temperature fluctuations have still not got us to the sweet spot for optimum PMC trans.

AA IN -> COIL DRIVER IN transfer function for MC3

I've provided excitation to the AA input, the same for all OSEM channels. In the digital domain coherence between C1:SUS-MC3_ULSEN_INMON / C1:SUS-MC3_ULCOIL_INMON and other channels OSEM -> COIL is 1 starting from 0.1Hz.

The only thing left to understand is why the coherence AA IN / COIL DRIVER IN measured in the analog domain is not 1 in the frequency range 0.1 - 1 Hz. It does not look like just SRS noise. I've connected Ch 1 and 2 to the source, coherence is close to 1.

Yesterday we found that MC3 OSEM LL PD did not have a sensible signal - the readback was close to zero and it was making MC move around. I disabled the PD LL so that the damping is done with just three face plus side PDs. There still no signal from MC3 LL PD today. It needs debugging.

I noticed that the MC3 LL sensor was apparently dead according to its suspension screen. Since it was only the fast ADC channel and not the SLOW PDmon, I could tell that it was just in the ADC cabling. I pushed in a few of the MC3 sensor cables on the front and back of the PD whitening board and it came back OK. According to this trend of the past 40 days and 40 nights, it started slipping on this past Wednesday morning.

Was anyone walking near MC2 or the suspension electronics racks before noon on Wednesday (Oct. 2nd)?

I did the same measurement for MC3 with one difference that OSEMs report more motion than IMC cavity length change due to it being at 45 degrees. Following are the new cts2um gain values

• UL 0.36 -> 0.39827(0.00045)
• UR 0.36 -> 0.33716(0.00038)
• LL 0.36  -> 0.34469(0.00039)
• LR 0.36 -> 0.33500(0.00038)

Balanced MC3 coil strengths using the same method.

Final coil strengths found:

C1:SUS-MC3_ULCOIL_GAIN: 0.942
C1:SUS-MC3_URCOIL_GAIN: -1.038
C1:SUS-MC3_LRCOIL_GAIN: 1.075
C1:SUS-MC3_LLCOIL_GAIN: -0.945