There looks some activity at around MC2 on Wednesday afternoon.
It caused the misalignment of MC2. Misalignment was not found in MC1/3.

It seems that the incident beam on the MC was aligned in the evening.
This increased the MC transmission but it is vibible that the spot on MC2 is shifted from the center.

We need an action on this issue tomorrow in the daytime.

I am working on fixing this.  You might some strange stuff going on in the control room screens.  Pls ignore it till I am done.

  I have realigned the MC by recentering the spots on all the MC optics.  The current spot positions (in mm) are:

MC1P     MC2P     MC3P      MC1Y      MC2Y     MC3Y

0.2245    0.3364   -0.2801   -1.8891    0.1631   -1.744

Initially the lockins 2 and 5 showed very small outputs.  This was traced to the fact that we have recently switched on a 28Hz ELP filter module in the MC2 ASC filter bank which introduces an extra phase of about 75deg..  See this elog.

When the MC ASS lockins were initially setup, the phase was set with this filter module switched off.  Since quite some time has passed since the last calibration of these phases, I readjusted the phases to minimise the  Q_OUTPUT and I also adjusted the GAINs in the SIG filter banks  of all the six lockins so that their I_OUT's drop by the calibration value of -2.65 when an offset of 0.1 is introduced into the MC suspension output matrices.  Two short scripts in the $scripts$/ASS/ directory help in setting and removing these offsets.  They are called MCxoffsetOn and MCxoffsetOff.   They have to be edited appropriately to address each DoF of the MC.

The $scripts$/ASS/mcassUp script., which sets up everything to make the MC spot decentering measurement, has been edited to set these new phases and gains.  The old settings have been commented out.

I then centered the spots on the WFS sensors and the MC_TRANS QPD.  We are now ready to make the MC WFS output matrix transfer coef measurement again, but this time with the WFS loops closed.

While chatting with Jenne I learnt that some substantial amount of work had taken place yesterday around the MC2 chamber.  This was associated with the relocating of seismometers.  ref elog. 

I reiterate what is well known for quite sometime:  MC2 table is not well isolated from the ground.  And we should not approach this chamber unless absolutely necessary. I have blocked off the area around it which we should avoid.  It is a serious waste of time and effort to realign the MC each time the MC2 table decides to settle into a new position.

Steve tells me that the mild-steel frame supporting the chamber+MC2_table sits with two legs on one concrete slab while the other two legs sit on another one.   The frame is also quite weak without sufficient gussets or cross connects.  The next time we have a major shutdown we must replace this frame with a more sturdy one which sits on one slab (preferably the one on which the rest of the MC sits).

Till we improve this mounting, I suggest that we avoid that area as much as possible.

 Thinks to do before the NEXT realignment:

B,  tie 4 ancher bolts on table legs to the floor

C,  tie 4 dog clamps between table and chamber

D,  check the locked position of the 4 x 4 positioning screws

E,  check bellow protecting 4 tubes are not shorting

A,  here is the concrete  slab cut

It reminds me to check the  IFO vacuum envelope dog clamps on the chambers to floor  with torque wrench.

I found that the sum of the ITMX oplev signals had gone down to zero yesterday.

I checked the ITMX table and found two iris on the He-Ne laser path were blocking the beam on their apertures.

I guess this is because we were working around there for installation of POP/POX and may have touched some of the oplev optics.

Then I fully opened the apertures of those two iris and the sum went back to nominal of 600 counts.

I finally got around to wrapping up the SUS input matrix diagonalizer. The files I have added to ...scripts/SUS/peakFit are:

• kickAll: Restores all SUS angle biases using /cvs/cds/rtcds/caltech/c1/medm/c1ifo/cmd/C1IFO_OPTICrestore.cmd XXXX &, then runs 'freeswing all'. Finally, writes an elog entry with the time when the optics were kicked and saves the gps time to 'kickAll.time'.
• diagAllSUS.m: An M-file that calls all the other individual M-files needed for the matrix generation. What it does:
  • Reads kick time from kickAll.time
  • Runs getSensors.m to get time series from all optics' sensors
  • Runs makeSUSSpectra.m to generate spectra from time series
  • Runs findPeaks.m to fit spectra for peak frequencies
  • Loops through all optics and runs, in sequence:
    • inmat = findMatrix(XXXX) to generate the matrix for each optic
    • writeSUSinmat(XXXX,inmat) to write that matrix to the frontend
• diagAllSUS: A wrapper for diagAllSUS.m. It also writes an elog entry and attaches the pre and post spectra demonstrating the diagonalization. The entry following this one is an example.

The following lines should eventually be added to the controls@nodus crontab:

0 8 * * 0 /cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit/kickAll

0 18 * * 0 /cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit/diagAllSUS

(i.e., do the kick at 8am on Sundays and run the diagonalization 10 hrs later at 6pm). They can be done on a different machine but then my elog commands will need to be modified.

Before we add them, we should check that they do, in fact, work. We can do this sometime while I'm at the 40m this week.
 

I actually tried to do this last night, but I was getting a terminal error from nodus. Jamie told me to just manually redefine the TERM variable and it would work. So, I have added kickAll and diagAllSUS to controls@nodus's crontab:

nodus:~>crontab -l

0 5 * * * /opt/rtcds/caltech/c1/scripts/backup/rsync.backup

0 7 * * * /opt/rtcds/caltech/c1/scripts/backup/check_backup.sh 

0 17 * * 0 /cvs/cds/caltech/users/jenne/LIGOX/LIGOX_Pizza_Reminders.sh

0 8 * * 0 /cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit/kickAll

0 18 * * 0 /cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit/diagAllSUS

Shade 12 black glass-green lines: 1/8" thick,  6" wide  x 7 " high  mounted on angle bracket below.  Aperture 1" diameter. Let me know what do you think.

It turns out that nodus doesn't know how to NDS2, so I can't run diagAllSUS as a cron job on nodus. To further complicate things, no other machines can run the elog utility, so I am going to have to do something shifty...

 Actually in the last 40m meeting we discussed the SUS diagnosis and decided not to post the results on the elog.

Alternatively we will have a summary web page which will contain all the information (sensitivity, UGF monitors, etc.,) and will be updated everyday like GEO.

This will be a place where we should post the SUS results every week.

So we don't need to worry about the cron-elog job, and for running the scripts you can simply use one of the lab machines as a cron host.

Once you get the scripts running on a machine as a cron job, it will be the point where we should quit developing the SUS diagonalizer and pass it to the web summary people.

EDIT: Ran Jamie's NDS2 reset method and it's still not serving up MC data.

I am debugging the SUS input matrix diagonalization code (which I have affectionately termed "Dr. SUS") offline by running everything except the actual writing of the matrices (%writeSUSinmat()) in a sandbox directory. Some notes:

• nodus doesn't know NDS2, but this isn't a problem given Kiwamu's reply, since we don't want to push results to the elog. I will still have the script push "optics kicked---don't touch" alerts to the elog, but I realized this can be done easily from a remote computer via in-script SSH commands. My next choice was pianosa, but even though she knows NDS2, she spits out MEX errors when I try to get data. Third stop was allegra, and she plays nice 
• NDS2 is unable to read the MC mirror sensor data. Data from all other optics are retrieved without a problem. I verified that raw data was being written by plotting playback data in dataviewer, and I tried various times and the problem persisted. I suspect the problem is on the NDS2 server side:

The most recent entry in .../users/jzweizig/nds2-mafalda/channel_history shows nothing for MC1 2 or 3

Now that NDS2 is working, I was able to run all parts of diagAllSUS (except for the matrix writing) without errors. Barring issues with standard commands (freeswing, turnOnWatchDogs.csh, caput, etc.), Dr. SUS should be ready to go.

As requested, I have suppressed the generation of an elog post containing the results of the diagonalization.

Dr. SUS will run on allegra as a cron job at 8:00am on Sundays:

allegra:peakFit>crontab -l

PATH=/bin/:/usr/bin/:/cvs/cds/caltech/apps/linux64/matlab/bin/:/cvs/cds/caltech/apps/linux/gds/bin/:/cvs/cds/caltech/apps/linux/tds/bin/

PWD=/cvs/cds/caltech/apps/linux/gds/lib/

LD_LIBRARY_PATH=/cvs/cds/caltech/apps/linux/gds/lib

01 05 * * 0 cat /cvs/cds/caltech/users/kiwamu/work/20091026_OMC-LSC-diag/src/OMC_LSC_timing.m | matlab -nosplash -nodesktop > /dev/null

0 8 * * 0 /cvs/cds/rtcds/caltech/c1/scripts/SUS/peakFit/runDrSUS

Also: should these variable definitions really be here?

 I locked the PMC and MC resonated in TM00 right on. Than I started checking anchoring screws with the torque wrench.

B,  3/8 foor bolts were tight, They were set  to 50 ft/lbs

C,  dog clamp bolts were tight at 80 ft/lbs

D,  horizontal position locks were lose. They were locked by finger.

E,  below covers were reset not to short

The MC is locking in TM01 mode now 

Cron starts with only a very minimal PATH.  However, this shouldn't be an issue if you specify the full path to the commands.

The rest of the env vars should probably not be there.

Also, Zach:  using /cvs/cds is now forbidden.  We need to put this script in the right place.

• I have replaced all instances of /cvs/cds/rtcds with /opt/rtcds in the Dr. SUS code. 

• We discussed placing a human in charge of whether or not the input matrices get written to the frontend. I am not sure if we reached a definitive conclusion. Currently, the code is set to write the matrices automatically. What to do?
  
  • If we decide that oversight is necessary, I suggest that we leave the publishing of the results to the elog intact. This way, it will be someone's job to read the weekly Dr. SUS diagnosis on Sunday night (or Monday morning), and run a simple script that writes the matrices. This seems like the most reasonable solution.

I await responses...

My inclination is to not do the writing of the matrices automatically nor to do the weekly kicks. Its nice to have long locks of the MC, etc.

I suggest just making the kick on Sundays when someone intentionally asks for it (e.g. by pressing a button on Friday). The free-swinging ringdown ought to be a opt-in kind of feature, not opt-out.

Ok, here's the deal:

• For the time being, I have written a "doirun" bit into runDrSUS (i.e. it runs if doirun is 1 and doesn't if it's 0). This is a bad way of doing this, so in the end I think we should put a switch on the IFO MEDM and have the script read the value when the cron job is run. If you want it to be an opt-in rather than a toggle, we can have the script write it back to 0 every time. I don't know how to do this yet because I am an MEDM n00b, but I will do it soon.
• Since we have decided to keep a human in the loop on the writing to the frontend, I have kept the elog results push.
• I have also edited diagAllSUS.m so that it archives all computed matrices (hierarchy: .../scripts/SUS/peakFit/inMats/(gps_time_of_kick)/inMat(optic_name).mat). There is a 'writematrices' bit in the M-file, currently set to 0.
• I have written the script 'writeAllInMats' and the accompanying M-file 'writeAllInMats.m'. This allows the user to write whichever set of input matrices he or she desires (syntax: writeAllInMats (gps_time_of_kick)). If no argument is given, then it reads the most recent kick time from 'kickAll.time' and writes the corresponding matrices.

So, here is an example of how this works:

1. Someone decides to do a diagonalization on a particular weekend, (eventually) clicking a switch in MEDM
2. Cron runs runDrSUS at 8am that Sunday. This:
   1. Kicks all the optics, lets them swing for 5 hours, then reengages the watchdogs. The kick time is saved in kickAll.time, and an alert is posted to the elog
   2. Runs diagAllSUS, which computes and saves all matrix data. A report of the results is posted to the elog.
3. On Monday morning---or whenever---someone looks at the entry and decides whether or not to write the files
   1. If the results are good, he or she runs writeAllInMats and the latest matrices are written
   2. If the results are bad, he or she does nothing. The matrices are still archived and can be written at any time in the future.

The code is set to run tomorrow morning. Everything but the writing will be done.

 Dr. SUS failed while trying to get the sensor data. Specifically, it couldn't get ETMY data. This is odd, because in my tribulations last week I ended up having to add the ETMY_SENSOR channels manually into the NDS2 channel files. After doing this, I was able to get ETMY data just fine (though I admitted that we would have problems again as soon as we wanted to update the channel files). I even ran the diagAllSUS code in a sandbox and it pulled data---and generated input matrices---just fine.

The error persists even if I try to get the data manually:

>> d = NDS2_GetData({'C1:SUS-ETMY_SENSOR_UL'},t0,10,'mafalda.martian:31200');

Connecting.... authenticate done

Warning: daq_recv_next failed

??? Error using ==> NDS2_GetData

Fatal Error getting channel data.

I think Jamie is still waiting for J.Z.'s help with this, but it is probably pointless to keep trying to run this code before NDS2 is working again. Another option is to just use NDS, but I think certain people are opposed to this. 

The ETMX oplev returning beam is well centered on the qpd. We lost this signal 2 days ago. I will check on the qpd.

ETMX sus damping restored.       c1iscex computer is down

 As reported a couple days ago, the ETMX IO chassis has no timing signal.  This is why we there is no QPD signal and why we turned off the ETMX watchdog.  In fact, I believe that there is probably nothing coming out of the ETMX suspension controller at all.

I'm working on getting the timing board replaced.  Hopefully today.

Mirko and  Den are measuring MC_F, which they will report about later, but I noticed that MC2 is totally crazy right now.  It shouldn't matter that they are doing things (like unplugging the feedback to the PSL's PZT), because we actuate on the laser, not on the MC.  I disabled the MC autolocker before they started working. 

Anyhow, somehow MC2 got kicked up (whatever, that happens), but it won't re-damp.  I think it's the WFS.  The yaw output from the WFS is truely crazy. 

I have disabled the WFS output / ASC input on the MC SUS screens, and MC2 was then able to damp.  My disabling only the MC2 WFS input at first kicked up MC1 and 3, so I disabled all of the WFS stuff, and all 3 MC mirrors are again happy. 

SURESH: FIX ME!  (signed, The WFS)

The following will be a stream-of-consciousness, approximately chronological story of my last hour or so of looking at screens....

In the old OAF days, we used to bypass the analog dewhitening in the coil driver path, using the XYCOMS.  See, ex. elog 2548.

I began to wonder if we needed to do the same thing now.  I checked several optics, to see how the switching works. 

For the whitening on the OSEM sensor input, FM1 is linked to the Contec binary I/O.  FM1 is the inverse whitening filter.  Turn it on, and the analog whitening is on (bit in the binary I/O screen turns red).  Turn it off, and the analog whitening is bypassed (bit in the binary I/O screen turns gray).  Good.  Makes sense.  Either way, the net transfer function is flat.

The dewhitening is not so simple.  In FM9 of the Coil Output filter bank, we have "SimDW", and in FM10, we have "InvDW".  Clicking SimDW on makes the bit in the binary I/O screen gray (off?), while clicking it off makes it red (on?).  Clicking InvDW does nothing to the I/O bits.  So.  I think that for dewhitening, the InvDW is always supposed to be on, and you either have Simulated DW, or analog DW enabled, so that either way your transfer function is flat.  Fine.  I don't know why we don't just tie the analog to the InvDW filter module, and delete the SimDW, but I'm sure there's a reason.

All optics have this setup, except MC1 and MC3.  They don't have the SimDW or InvDW filter modules.  Instead, in FM9 (which on all the other suspensions is SimDW, and controls the binary I/O) there is a 28Hz Elliptic Low Pass filter.  The only thing I can find about these is elog 1405 where Rana talks about implementing ELP28's in MC2.  But right now there is no ELP in the MC2 coil output filters.  So, if Rana's old elog is to be believed, we need to fix up the ELP28 situation.  But that elog was from a long time ago, so maybe things are different now?  If MC1 and MC3 need the SimDW and InvDW (why wouldn't they?) then the ELP28 needs to move to another filter module.  Because right now, when I click the ELP28's on and off, it changes bits in the binary I/O.  Which I don't think it should.  Maybe.  I don't really know.

Okay. So. Now we know where everything is, and which buttons do what.  Maybe not why, but at least what.

In the old world, Rob had lots and lots of trouble (elog 2027) with locking when the analog dewhitening was bypassed.  But right now, I think that all of the analog dewhitening filters are bypassed, for every single optic we have.  So.  Which way do we want things?  What's the new game plan.  What's going on??

\end{stream-of-consciousness}

The input/whitening filters used to be in a similarly confused state as the output filters, but they have been corrected.  There might have been a reason for this setup in the past, but it's not what we should be doing now.  The output filters all need to be fixed.  We just haven't gotten to it yet.

As with the inputs, all output filters should be set up so that the full output transfer function is always flat, no matter what state it's in.  The digital anti-dewhitening ("InvDW") and analog dewhitening should always be engaged and disengaged simultaneously.   The "SimDW" should just be removed.

This is on my list of things to do.

The Problem

Turning off the WFS servo loops usually should be done using the 'mcwfsoff' script.  The script takes care of switching off the integrators and Clears the History.  

'mcdown' and 'mcup' scripts run the 'mcwfsoff' and 'mcwfson' scripts so when the MC unlocks the WFS servos are shutdown and restarted properly.  However if the MC autolocker script is suspended by pressing the Enable/Disable switch in the LOCKMC screen and then the MC unlocks, it results in the WFS servo integrators accumulating large values.  If these values are passed through the ASC filter banks the optic will get a pretty huge kick.

The Solution

I have added some indicators which will let us know if the WFS Servo Filter outputs are larger than +/-1000.  When engaging the WFS loops the user has to take care to Clear History in the servo filter banks if these indicators are steadily Red.   before engaging the WFS Servo loops ensure that the servo filter outputs are zeroes. 

Koji and I discussed whether it would be useful to run the 'mcwfsoff' script when the Disable button is pressed in the autolocker.  His recommendation is that we should keep  the autolocker script simple and that user has to be cautious when switching on the WFS servos and when directing the ASC outputs to the suspensions.

The  PRM lost damping about a day ago. It was restored.

Our existing 300 series SS plungers from McMastercar #8476A43 are silver plated as Atm2 shows.

Problems:  1, they become magnetized after years being close to the magnets

                     2, they oxidize by time so it is hard to turn them

I looked around to replace them.

Titanium body, nose and beryllium copper spring. None magnetic for UHV enviorment.

Can be made in 7 weeks at an UNREASONABLE $169.00 ea at quantity of 50

I found that the slow machine c1auxey, which controls and monitoring the ETMY suspension things, were not responding.

The machine responded to ping but I wasn't able to telnet to it.

I went down there and power-cycled it by keying the power of the VME rack, and then it came back and seems working properly.

I have no idea why it ran into such condition.

[Rana / Mirko / Kiwamu]

 The watchdogs on the MC suspensions are not working.

Switching off the watchdogs doesn't stop feeding signals to the suspensions.

For tonight, we will leave the controller of the MC suspensions switched off so that the computer won't smash the optics accidentally.

We are now trying to understand why the coherence between SUS-X_SUSPOS_IN1 and SUS-X_SUSPOS_OUT is lost below 1 Hz for X = MC1, MC2, MC3, BS, ITMX, ITMY, ETMX, ETMY, SRM. It is OKEY only for PRM but the different filteres are used there. For PRM - 30:0.0 and Bounce Roll, for all others - 30:0.0 and Cheby. The transfer functions between these two signals plotted in foton and fft tools are also not the same.

If we switch off all the filters between these 2 signals, than the coherence is one. If one of the filters is switched on, everything is also fine. But if there are several present, than they filter the signal in unexpected way.

Moreover it seems that the coherence is dependent on the input signal. The coherence is better with local dumping than without.

 We have done the following on the c1sus and c1lsc computers : provided excitation, let the signal pass through filters 30:0.0 and Cheby and plotted the coherence between in and out signals.

c1sus computer - coherence is corrupted

c1lsc computer - coherence is not corrupted

It seems as if someone was looking into this on Wednesday, but as there's no elog, I think probably not.

Tonight we noticed that, in fact, the watchdogs don't work for any of the corner optics (I confirmed that they do work for the ETMs).

Whether the switch for the coil drivers is enabled or disabled, the voltage monitor on the coil drivers responds to the digital signals.

I tried restarting the c1susaux process, hitting reset on the crate, and also keying the crate. The +5V light on the front of the crate is flickering somewhat, but I'm not sure if this is new or not since the power outage. The next step is to track the Xycom signal from the card over to the coil driver to find where the signal is failing to happen.

Since its not working for any of the corner optics, I suspect the crate and not the cards. If that's the issue this will be a painful fix. We are sort of assuming that this is due to the power outage, but in fact, I cannot tell when the last time was that someone rigorously verified the working of these switches. [

I][B]We had better get ready for upgrading our SLOW controls, Jamie.[/B][/I]

[Rana, Mirko]

I tried out the virtual pendulum idea today. The idea is to compensate the physical pendulum via the control system, and then add a virtual pendulum formed in the control system. We know the actuator TF from p. 5900 and apply its inverse to the C1:SUS-MC?_SUSPOS filters. Also we add an pendulum Q=3 with a resonance frequency of 0.1Hz to the POS control loops.

The result is pretty awesome!

1. Black: Standard config. Wfs on. New Cheby filter in place ( p. 6031 )
2. Red: With virtual pendulum. Extra eliptic LP filter @ 2.5Hz

Filter shape:

This is stable for 5-10minutes, at which point it falls out of lock, swinging in yaw.

Our approach to making the F2A or F2P filters for the MC is to use the measured resonant frequencies and then calculating the appropriate mechanical dimensions of each suspension. This is basically because we don't have optical levers with normal incidence on these optics, but the method should be fine.

To find the formulas, I asked Gaby for her old cheat sheet: Its now in the DCC. Its only for Large optics, but you should be able to reconstruct the right ones for SOS by just changing the parameters.

In the above entry MC_f  signal is improved off of resonance by the implementation of the pendulum compensation. It should be checked if this is actually due to the implementation of the virtual pendulum at 0.1Hz. A way to check that might be to implement a simple double LP at 0.1Hz instead and look at the resulting MC_f signal. A projection of the OSEM FB noises with the compensation active might be interesting.
The physical resonance at 1Hz is clearly not compensated correctly, which is probably the reason for the lock losses after a few minutes. It might be a good start to measure the residual resonance with the compensation in place. The filters in bank 7 of C1:SUS-MC?_SUSPOS have both the compensation of the 1Hz resonance( really the inverse actuator TF ) and the virtual pendulum in them. The ‘pure’ compensation can be found in the InvTF module in the C1:OAF-ADAPT_MCL_CORR filter.
The fact that the beam swings in yaw at lock loss indicates that the separation of the DOFs might not be perfect. One should have a look into the yaw and pitch DOFs with the compensation active.

The watchdogs' issue has been solved and they are now working fine.

It was just because one of the Sorensens had been off.

[Koji Kiwamu]

We wonder if the flakiness of MC2 comes from the suspension or not.

To test it, we are shaking all of the suspension biases +/-1.0 with a script.

The script is here:
/users/koji/111216/SUS_hysteresis.sh

For this test, we closed the mechanical shutter before the MC.

Also some amount of misalignment is anticipated.
Don't be surprised if you see nothing is working when you come to the lab in the weekend.

The hysteresis test has been aborted.

All of the suspensions have accumulated unexpectedly big DC biases of about 5 from their nominal points.

Do you guys have timestamps for when you started/ended the test? I have been trying to take some long-term RAM data but keep getting foiled by stuff (this test, RTS upgrade, switching of RAMmon channels, etc.)

The test was from: 2011-12-17 09:48 to 11:49 (UTC).
This corresponds to the period from 2011-12-17 01:48 to 3:49 (PST).

ZK: Thanks

Koji has modified the script for the hysteresis measurement.

A new test started from 16:05 PT, Dec 18th and takes a couple of hours to finish the measurement.

Do not touch the suspensions until further notice.

The measurement finished at ~ 21:50 PT.

I have recentered the oplev beams, including BS, ITMs and ETMs.

Another hysteresis test has begun at 1:50 PT, Dec/19.

It will finish after 3 or 4 hours. During the measurement the PSL mechanical shutter will be kept closed.

While discussing the suspension hysteresis measurements, Koji, Kiwamu, and I realized that the suspension wire standoff is aluminum, whereas the standoff for the LIGO LOS are using quartz.

Using a soft aluminum standoff is bad. The movement of the suspension will slowly wear the groove and produce opportunities for mechanical upconversion and hysteresis.

In fact, the wire standoff as well as the clamping block on the top should be made of sapphire or ruby to prevent any such wearing issues. Steve is hot on the case.

ETMX sus damping restored

No we can't do that because the c1scx model is not working properly.

If you look into the real time controller screen you will find what I mean.