I installed Virtual Box on rossa. Then I put Windows 7 in there and am now installing Altium.

You can run Windows on rossa by just clicking Applications -> System Tools -> Virtual Box.

MC and PMC vis:

MC REFL Unlocked    = 4.4

MC REFL Locked      = 0.67

 1 - Locked/Unlocked = 85%

PMC REFL Unlocked   = 0.270

PMC REFL Locked     = 0.013

 1 - Locked/Unlocked = 95%

I checked (by looking through recent trends) that the zero level is zero on both channels. I tried to do a proper mode scan, but we have lost the PSL fast channels during the upgrade sadly. Also, the DC signal for the PMC REFL needs some gain. Unlocked level should be more like 3-5 V.

Also used the instructions from this page to add Google's sources to rosalba's apt-get list and then installed Chrome.

[Steve, Koji, JC]

Steve Vass came by today and was loaded with information. Here are a couple things we went over:
- Removal of the large optical table.
- Disconnecting the roughing pumps to prevent oil contamination. 
- Plexiglass End Enclosures
- TP1 maintenance Information

Removal of the large optical table. 

    Steve mention to us that there will be 2 machines that have to raise and rotate the optical. The optical table is 2 ft wide when rotated and this seems to easily give us clearance to get this guy out if we set it on piano dollies. The only issue we encountered is that we may have to disconnect the small power supply rack by 1X2 to make room for one of the machines that will rotate the table. If we manage to lift the table, slide it over to a more open area, lift again, and remove the dollies, then this may give us enough space for this machine and we won’t have to disconnect the power supply rack by 1x2. Overall, turning the table over on its side and maneuvering it into the aisle to go down the aisle along Xarm is the trickiest part. From there, we just have to find out where we are going to take this monster of an optical table.

Disconnecting the roughing pumps to prevent oil contamination
    

Disconnecting the roughing pumps to prevent oil contamination. 

    Steve let us know that he would disconnect to oil vacuum pumps after getting to your nominal vacuum pressure. This prevents these pump from possibly feeding oil into the system. Each of these pumps have an intentional leak vale that keeps the pressure above 300 mtorr. If the pressure get any lower, the pumps will begin to back pressure oil into the system.

Plexiglass Enclosures

  

   I asked Steve about some information on the end enclosures and PSL Enclosure. For the end enclosures, these were custom made. The film on the enclosure was added by people who do car window tinting. This film is not as strong as the plexiglass for the PSL Table. I have to look through Steve’s physical documents to find a receipt, or information of where he purchase the custom enclosure. As for the PSL enclosure, the film is actually imbedded/ combined with the plexiglass. This can withstand much more power than the end enclosures.

TP1 Maintenance
    
    Steve has sent TP1 and the controller for maintenance before. It seems like we need to do this roughly ever 4-6 years and the last time TP1 was shipped out for maintenance was in 2018. During this, Steve said he thinks he was shipped a temporary pump to use in the mean time while TP1 was being repaired. When there is an error with the pump, it will pop up on the controller as show in [elog 14189](https://nodus.ligo.caltech.edu:8081/40m/14189)Anyways, it seems like TP1 will have some issues coming up.

I've attempted to visualize the various components of the cost function in the way I've defined it for the current iteration of the Oplev optimal control loop design code. For each term in the cost function, the way the cost is computed depends on the ratio of the abscissa value to some threshold value (set by hand for now) - if this ratio is >1, the cost is the logarithm of the ratio, whereas if the ratio is <1, the cost is the square of the ratio. Continuity is enforced at the point at which this transition happens. I've plotted the cost function for some of the terms entering the code right now - indicated in dashed red lines are the approximate value of each of these costs for our current Oplev loop - the weights were chosen so that each of the costs were O(10) for the current controller, and the idea was that the optimizer could drive these down to hopefully O(1), but I've not yet gotten that to happen.

Based on the meeting yesterday, some possible ideas:

1. For minimizing the control noise injection - we know the transfer function from the Oplev control signal coupling to MICH from measurements, and we also have a model for the seismic noise. So one term could be a weighted integral of (coupling - seismic) - the weight can give more importance to f>30Hz, and even more importance to f>100Hz. Right now, I don't have any suc frequency dependant requirement on the control signal.
2. Try a simpler problem first - pendulum local damping. The position damping controller for example has fewer roots in the complex plane. Although it too has some B/R notches, which account for 16 complex roots, and hence, 32 parameters, so maybe this isn't really a simpler problem?
3. How do we pick the number of excess poles compared to zeros in the overall transfer function? The OL loop low-pass filters are elliptic filters, which achieve the fastest transition between the passband and stopband, but for the Oplev loop roll-off, perhaps its better to have a just have some poles to roll off the HF response?

Viton flouroelastomers have somewhat variable material properties. The following parameters are being used for eigenfrequency analysis.

Young's Modulus: 72,500-87,000 psi (cite: http://www.row-inc.com/pfa.html) *Accurate for PFAs

Poisson's Ratio: 0.48-0.50 (cite: http://www.engineeringtoolbox.com/poissons-ratio-d_1224.html) *Accurate for rubber

Density: 1800 kg/m^3 (cite: http://physics.nist.gov/cgi-bin/Star/compos.pl?matno=275)

 The Young's Modulus for PFA turned out to be orders of magnitude greater than for Viton. The revised values produced much more accurate results:

Young's Modulus for Viton-75: 1950 psi or 6.6 MPa

(Courtesy Row, Inc. and Steve Vass)

This website contains other details: http://www.row-inc.com/viton.html

The IBM laptop at EX was running Ubuntu 14, so I allowed it to start upgrading itself to Ubuntu16 as it desired. After it is done, I will upgrade it to 18.04 LTS. We should have them all run LTS.

I tested the voltage dividers and was getting up to about 3V. I retested the mic w/o the voltage divider in place, and, lo and behold, I was able to generate about 70-75V (previously, I maxed out at 45V). I'm not 100% sure why this was, but it occurs to me that, before, the sounds I was generating were short in duration (loud claps, short yelps). This time, I tried yelling continuously into the microphone. So, probably, I simply wasn't seeing the real peak before on the scope because it was too short to pick up. I have corrected the voltage dividers (by replacing the 25.5 kOhm resistors, which were in parallel with the ADC, with 10 kOhm resistors, taking the voltage ratio to ~60:1) and tested them. I haven't been able to generate more than 1500 mV, so I think they are safe. (It's possible we would have been fine with the old setup, since I think it would be hard to get any noises as loud as I was making, but better safe than sorry, right?)

I'm attaching a diagram of the new-and-improved voltage dividers.

I've measured the voltage noise of the SR560's lead acid battery outputs; they're not so bad.

Steve ordered us some replacement lead-acid batteries for our battery powered pre-amps (SR560). In the unit he replaced, I measured the noise using the following setup:

SR560                              Busby Box

(+12V/GND) -------------AC Input      Out  ----------------   SR785

The SR785 was DC coupled and auto-ranged. The input noise of the SR785 was measured via 50 Ohm term to be at least 10x less than the SR560's noise at all frequencies.

Its clear that this measurement was spoiled by the low frequency noise of the Busby box below 10 Hz. Needs a better pre-amp.

Be careful with this.  If Matlab starts re-saving models in a new file format that is unreadable by the RCG, then we won't be able to rebuild models until we do an svn revert.  Or the bigger danger, that the RCG *thinks* it reads the file and generates code that does something unexpected.

Of course this all may be an attempt to drive home the point that we need an RCG test suite.

After moving the c1lsc computer to 1X4, then connecting c1lsc to it's IO chassis in 1Y3 by a fiber PCIe extension cable, everything is back up and running and the status screen is all green.  c1lsc is now directly connected to c1sus via a short copper Dolphin cable.

After lunch we will do some more extensive testing of the system to make sure everything is working as expected.

We centerd the input of WFS QPD.

Yesterday I locked the MC and left at 8 pm. Analyzing the data I saw that MC was locked all time from 8 pm to 12.30 am when it lost lock. Moreover there was no light on transmition and reflected screens at all. I went to the PSL and saw that no red light comes to the MC from PSL, only green. I took infrared sensos to track the laser light. Then I came back to control room to study the medm diagram of the PSL. Then I came back and saw that the laser beam goes to the MC! I returned to control room and saw light on the MC screens. Does someone do something parallel with me through ssh?

I enabled the auto locker and saw the MC locked for a couple of seconds. After that the WFS were turned on automatically and I saw that the signal of the OSEM local sensors of the MC mirrors began to increase. So the WFS master provides not good feedback signal. I thought that it is due to my recompilation of c1mcs with a fixed if-statement line. And may be if c1mcs workes without digital noise and c1ioo with it then there might occur some mismatches and the signal is corrupted. For this assumption I've recompiled c1mcs back to 1e-20 in the if-statement and so added the digital noise back that I saw in the dtt tools.

However, the problem was still present - WFS feedback signal crashed the MC lock. I open the WFS master window and disabled the output to MC. I can see that the C1:IOO-WFS1_PIT_INMON and other input channels have reasonable values 8 - 20 but the output continues to increase up to 1000000. The output was off so the MC stayed at lock. As for now I turned off WFS so no feedback is applied to MC mirros.

With the help of Suresh we have adjusted optics near PMC and input to the MC on the PSL and in the black box where WFS are. Surprisingly, some optics near WFS was not attached to the table. But these mirrors are not used. One screw was near the hole but not screwed in. This mirror is used. Suresh could also rotate other screws. I thought that they must be attached to the table more rigidly.

Then we found that WFS output matrix is wrong and Suresh recalculated it. After that we've locked the MC using WFS. C1:IOO-MC_RFPD_DCMON is 0.7-0.8. 

We also recompiled and reinstalled C1MCS and C1IOO with fixed if-statement and again saw how MC_F curve moves down. WFS error signals are also improved. But still some more work on output matrix is needed.

[koji, gautam]

Attachment #1 shows the relevant parts of the schematic of the WFS demod board (not whitening board). 

• The basic problem was that the switchable gain channels were not accounted for in the Acromag channel list 😒.
• What this meant was that the DC gain was set to the default x100 (since the two DG211s that provide the switchable x10 and x1 gain options had their control logic pins pulled up to +5V because these pins weren't connected to any sinking BIO channel).
• Rather than set up new connections to Acromags inside the chassis (though we have plenty of spares), Koji and I decided to make these fixed to x1 gain.
• The actual fix was implemented as shown in the annotated schematic. There are some pictures 📷 of the PCB in the DCC entry linked above.
• Amusingly, this board will now require a sourcing BIO unit if we want to still have the capability of switching gains.

Before removing the boards from the eurocrate: 

• I dialled down the Kepco HV supplies
• disconnected all the cabling to these boards after noting down cable numbers etc.

After Koji effected the fix, the boards were re-installed, HV supplies were dialled back up to nominal voltage/currents, and the PMC/IMC were re-locked. The WFS DC channels now no longer saturate even when the IMC is unlocked 👏 👏 . I leave it to Yehonathan / Jon to calibrate these EPICS channels into physical units of mW of power. We should also fix the MEDM screen and remove the un-necessary EPICS channels.

Later in the evening, I took advantage of the non-saturated readbacks to center the beams better on the WFS heads. Then, with the WFS servos disabled, I manually aligned the IMC mirrors till REFLDC was minimized. Then I centered the beam on the MC2 transmission QPD (looking at individual quadrants), and set the WFS1/2 RF offsets and MC2 Trans QPD offsets in this condition.

I was checking on the slow machine channels and found something I could not understand.

On the IOO WFS HEAD screen, there are two sets of 4 switches (magenta rectangles in Attachment 1) labeled 2/4/8/16dB.
But as far as I could confirm with the WFS demod (D980233) and WFS head (D980012) drawings, they are the gain (attenuation) switches for the individual segments.
Their epics variable names are "C1:IOO-WFS1_SEG1_ATTEN", "C1:IOO-WFS1_SEG2_ATTEN", etc...

I confirmed the switches are alive (effective), and they are not all ON or OFF. I wonder what is the real situation there...

The unfortunate discovery today was that the attenuator switches on the IMC WFS heads are actually assigned to individual segments, and they are active. That means that we have been running the WFS with an uneven gain setting. The attached PDFs show that the signals with the attenuators on and off all at the same time, while the WFS servo output was frozen. A more annoying feature is that when some of the attenuators are on, this does not lower the gain completely. I mean that the attenuated channels show some reduction of the gain, but that is not the level of reduction we see when all attenuators are turned on. This RF could come from some internal RF coupling or some similar effect.

Moreover, the demodulation phases are quite off for most of the segments.

So far, the WFS is running with this uneven attenuation. We take time to characterize the gain and retune the demod phases and input matrices.

its painful, but you and I should probably take these out, bypass the switches and use them with fixed gain; the 'Reed Relay' attenuators are not a good part for this app.

The historical problem is that they tend to self oscillate with full gain because they had 2 MAX4106 in series which couple to each other in the bad way --- need to remove one of them and set the gain of the other one to 10.

The WFS servo loop will come on 5 seconds after the MC is locked. 

I have uncommented the lines in the mcup script which turn on the WFS servos.  But I shifted their location to the part after the MC is locked.

WFS Whitening and Demod boards were scavenged. ' iLIGO WFS cards are in big plastic boxes placed on the north wall around Section Y5 or Y6 (not under the arm). The WFS head PCBs, empty WFS housing, WFS components, I/Q demod components are at SectionY10 under the arm tube. ' - Koji . I have taken pictures of the boards and will upload them to the DCC once I find a way to add a Serial Number . (I will upload this to the eLog as a HowTo) The next step is to search for at least 2 extender boards to troubleshoot these board and find if there are any issues. We may have replace some components and retune the boards. Attachment #1 is an example of the WFS Whitening Filter and Attachment #2 is and example of the the WFS Demod Board.

If you use the Camera DO NOT remove the strap. I have also purchased lens caps for the camera, so after usage PUT THE LENS CAP BACK ON. 

The WFS error signals were recorded in the order

WFS1_PIT

WFS1_YAW

WFS2_PIT

WFS2_YAW

these measurements are made in the linear region, that is the MC is nearly perfectly aligned.

This is  the average and std. dev.of 5 measurements taken of the same signals over 10 secs each. The std. dev are under 10%. And hence, I will be using 10 secs for measurements for the WFS signals after perturbations to the mirrors.

avg =

829.4408
-517.1884
297.4168
-944.7892


std_dev =

9.0506
22.9317
15.4580
8.9827

I perturbed the Pitch and Yaw of the Three mirrors (in order MC1,2,3), using ezcastep and calculated the coefficients that relate these perturbations to the WFS error signals.

The perturbation made is of -0.01 in each dof , and after measuring the WFS error for it, the system is reverted back to the previous point before making the other perturbation.

I was able to calculate the coefficients since I have assumed a linear relationship..

Following are the coefficients calculated using 10 secs measurements

coef_mat =

   1.0e+05 *

                            MC1_P   MC1_Y  MC2_P   MC2_Y    MC3_P   MC3_Y  constant
WFS1_PIT        -0.1262    0.3677   -0.4539   -0.6297   -0.1889   -0.1356   0.013664
WFS1_YAW     -0.0112   -0.7415   -0.1844    2.4509   -0.0023   -0.3531  -0.016199
WFS2_PIT         0.1251    0.4824   -0.2028   -0.6188    0.0099   -0.1490   0.006890
WFS2_YAW      0.0120   -0.7957   -0.1793    0.9962   -0.0493    0.2672 -0.013695

Also, I measured the same thing for 100s, and to my surprize, even the signs of coeficients are different.

coef_mat =

   1.0e+05 *

                           MC1_P   MC1_Y  MC2_P   MC2_Y    MC3_P   MC3_Y   constant
WFS1_PIT       -0.1981    0.3065   -0.6084   -0.9349   -0.4002   -0.3538   0.009796
WFS1_YAW     0.0607   -0.6977    0.0592    2.8753    0.3507    0.0373   -0.008194
WFS2_PIT        0.0690    0.4769   -0.2859   -0.7821   -0.1115   -0.2953  0.004150
WFS2_YAW     0.0580   -0.8153   -0.0937    1.1424    0.0650    0.4203  -0.010629

The reason I can understand is that the measurements were not made at the same time, and hence conditions might have changed.

A thing to note in all these coefficients is that they relate the error signals to the 'perturbation' around a certain point (given below). That point is assumed to lie in the linear region.

MC1_PIT      2.6129
MC1_YAW   -5.1781
MC2_PIT       3.6383
MC2_YAW    -1.2872
MC3_PIT      -1.9393
MC3_YAW    -7.518

I just found the singular values and the condition number of the 4*4 matrix relating the WFS error signals and the MC1 and MC2 movements.

the condition number is ~12.5. I think its small enough to continue with the scheme. (if the measurements and all are reliable).

Since the MC spots are good, I put the beam back on WFS 1 and WFS 2.

Also, I changed the indicators on the LockMC screen to reflect the change in elog 7289, where we added another on/off switch for the WFS so that the ASS could be on, but the WFS off.  For the last month, the WFS could be disabled, but the MC screen's indicators would suggest that we were pushing very significantly on all 3 MC mirrors.  Now the MC screen reflects reality a little better.

I also uncommented the WFS lines in the mcup script.  Den had commented them out, but didn't elog about it!  C'mon Den, please elog stuff!!!!  (He confessed out loud the other day, but it still wasn't in the elog).

I'm leaving the WFS loops disabled (even though the MC autolocker tries to turn them on, I have them manually disabled using the extra on/off switch) since they're unstable.  I'm in the process of figuring out what's wrong.  So far, the WFS improve the MC alignment for a minute or two, and then they totally misalign the MC.  This is a work in progress.

We found that the MC REFL image was no longer round and that the MCWFS DC quadrant spots were mostly
in one quadrant. So we re-centered the MCWFS beams in the following way:

1) We unlocked the MZ and adjusted the PZT voltage to keep the beam on the WFS from saturating.
2) Re-aligned the black hole beam dump to center its beam in its aperture.
3) centered the beam on the MCWFS optics and MCWFS QPD displays.
4) Relocked MC.

Below is the image of the IOO Strip tool. You can see that the MC REFL DC is now more flat. The
MC pointing has also been changed (see the MC TRANS HOR & VERT channels). The MC transmitted
light is also now more stable and higher.

We tried to center the QPD, and we found that there were a few hundred mV of dark offset for each 
quadrant of QPD. We adjusted them with this scripts:
/cvs/cds/caltech/scripts/MC/WFS/McWFS_dc_offsets

We found the MC reflection was distorted . And WFC beam went to upward of QPD

We recentered WFC beam and these problems were fixed

in the lab, checkin on the WFS

Sun Jul 5 18:25:50 2020

I redid Gautam's measurements to get a baseline before changing the head, and my results are very different: To me it looks like the WFS2 quadrants are all OK.

Measurement Details:

1. The whole AG4395 + breadboard Jenne laser is wheeled over next to the SW side of the AP table.
2. The output of the 1611 goes into channel R of the 4395
3. I disconnected all the LEMO cables from the head and then plugged a LEMO-BNC cable into the plugs one at a time. The existing LEMO connectors, which take the signals back to the demode board, were all a little loose, so I adjusted them with some pliers (see video).
4. The Atten = 0 dB for all AG4395 channels
5. Source drive = 0 dBm. Checked with a -10 dBm drive that there was no change in the observed TFs, so I guess a 0 dBm drive doesn't make things nonlinear.
6. When I first turned the setup on, the Yellow 'limit' light was ON on the ILX laser current driver, so maybe the modulation wasn't getting to the laser diode as we wish.
7. did not change any WFS MEDM settings for these measurements. Not sure if any of those buttons work anyway.

I've left the setup as is in case either me or Gautam want to double check. If we're agreed on this response, I'll remove the notches and disable the RF attenuators.

Sun Jul 5 21:42:45 2020

I set up to do the WFS head modifications today, but I was shot down in flames due to a missing AC/DC adapter.

The Prologix GPIB-ethernet dongle needs +8-13 V to run. Some riff raff has removed the adapter and I was thunderstruck to see that it had not been returned.

I did the usual hunt around the lab looking for something with the right specs and connector. I found one that could do +9V and had the right connector, but it didn't light up the adapter so I put it back in black SP table.

I'll order a couple of these (5 ordered for delivery on Wednesday) in case there's a hot demand for the jack / plug combo that this one has. The setup is in the walkway, but I returned the AS table to the usual state and made sure the IMC is locking well.

Clearly this "riff raff" is referring to me. It won't help today I guess but there is one each on the carts holding the SR785 (currently both in the office/electronics bench area), and the only other unit available in the lab is connected to a Prologix box on the Marconi inside the PSL enclosure. 

After some hunting, I found this old SURF report with the WFS head measurements. The y-axes don't make much sense to me, and I can't find the actual data anywhere (her wiki page doesn't actually exist). So I think it's still unknown if these heads ever had the advertised transimpedance gain, or if the measured transimpedance of ~1kohm was what it always was.

Trying to figure out what's wrong with the MC WFS:

1) The symptom seems to be that the control signals become very large in the pitch and then the loop breaks when they saturate. Usually this is due to a degenerate matrix or improper inversion. Most likely some of the BURT restore is bad or the analog gain for one of the WFS has been switched when Jamie was doing the "Guardian" debugging.

2) In checking this out, I found that several buttons on the WFS  screens were not working (and apparently have never been working). Please try to test things in the future...The filter bank buttons in C1IOO_MC_TRANS_QPD were using relative path names; fixed these to use abs path names. The buttons in the WFS_MASTER for the IOO_PIT banks were using IOO_PITCH instead...

2.5) Recentered beams on WFS heads with MC alignment good and MC unlocked.

3) Main problem in the WFS still not found - disabling this in the autolocker.

Tried a bunch of stuff, but eventually just turned off the TRANS_QPD loops and loops are stable. Needs more debugging.

1. Modified the on/off scripts so that the Integrators are no longer toggled. No reason to turn them off since we are clearing the filter bank histories.
2. With QPD feedback OFF, I have lowered the overall gain by 15x so that its just drift control.
3. Deleted unused / bad filters from the main filter banks.
4. Gautam is going to debug the QPD with a red laser pointer and then elog.
5. Jamie is checking out the MC Coil dewhitening logic to see if that's in a funny state.

The low UGFs of the MC WFS servos made the MC insane thesedays:
The servos are too slow and we kept having significant misalignment left uncompensated.

I increased the total gain of the MC WFS from 0.01 to 0.4 (x40) to make the UGFs of the
WFS paths to ~2Hz. This was too much gain for the QPD path so the gains for the QPD paths
were reduced by a factor of 4 (x10 in total).

The script mcwfsup was also modified accordingly.

 Back around June 18, Jamie was debugging some Guardian code here to replace our MC autolocker. Afterwards our MC WFS stopped working. We never figured out what went wrong, but at the time we turned off the feedback from the MC trans QPD and it stabilized the response at DC.

Today, I noticed that the trans QPD feedback is on.  Did anyone do this on purpose?

Its problem causing behavior is slow, but you can catch it if you wait. With the nominal WFS gain of 0.4 the control signal ramps up monotonically at a rate of ~100 counts/minute. Depending upon the static alignment of the MC, this could let it take 10 minutes or a few hours before it rails the MC SUS actuators and breaks the lock. Very sneaky. Don't turn this loop back on without making sure its working and not breaking. I would trend it for you, but the SLOW channels associated with the TRANS QPD servo are not trended --- does anyone know how to get them in the channel list?

 I wanted to make sure that the QPD map on the C1IOO_MC_TRANS_QPD.adl screen corresponded to the actual physical quadrants on the photodiode at the MC2 table. We turned MC_WFS_OUT  OFF before fiddling around with a red laser pointer to try and map the quadrants.

I initially verified the correspondence between the various quadrants and the text-fields displaying the outputs using PV_Info. I found that there was good agreement in this respect. So for instance, field adjacent to the quadrant marked "1" on the C1IOO_MC_TRANS_QPD.adl screen had the following input channel: IOO_MC_TRANS_SEG1_INMON. The filter banks were empty and there was just an overall gain on -1 on all four channels. The channels leading to the filter-banks were the 'right' ones: quadrant 1 for the top bank, then quadrants 2,3 and 4 down.

Next, a red laser pointer was used to map the quadrants. Here, there was some disagreement between the physical quadrants and the map on the C1IOO_MC_TRANS_QPD.adl screen, which is summarised in the attached image-the whole thing is sort of rotated 180degrees about the centre. 

The interpretation of the figure is as follows:

quadrant 1 on screen QPD=bottom right quadrant on QPD

quadrant 2 on screen QPD=top right quadrant on QPD

quadrant 3 on screen QPD=top leftt quadrant on QPD

quadrant 4 on screen QPD=bottom left quadrant on QPD

MC_WFS_OUT was turned back ON.

I've completed the modifications on two WFS demod boards. This required replacing all 8 mixer ICs on each board. I also tuned each channel to get less than 2 mV offset on all of them.

I was able to complete testing the board SNo. 111B today. The results are attached. The test was done by feeding the board 22 MHz LO generated by frequency doubling. A signal at 11 MHz was generated using Moku:Lab at 1mVpp and then further attenuated by 10 dB to make a fair comparison with the previous testing of the IMC WFS board at 29.5 MHz. This board has the same response as the IMC WFS board at 29.5 MHz. I tested all four channels in the second plot.

I'll complete the testing of the second board SNo 112 B and then move on to setting up the optical path for AS WFS.

The other modified board 112B has been fixed and tested now. See the results attached. The issue was in some malfunctioning OP284 which have been replaced by AD8672.

I played with the PLL bit more today to understand the issue. From what I understand, the following is the summary:

Moku as VCO in WFS demod board PLL:

• Moku input in VCO mode is actually limited to ~ +/-21 V contrary to what it says on the app (10 Vpp)
• Whenever the VCO tuning signal goes beyond this range, Moku just ignores the input and sends a pure sine wave at the carrier frequency.
• I think because of this rail point behavior, the PLL goes off to a bad mode where the VCO tuning signal from demod board rails to +15 or -15V, and thus Moku does nothing to correct for it.
• I found a deterministic way of catching lock with Moku VCO:
  • With whatever carrier frequency, set the VCO slope to at least 1 MHz/V (10 MHz/V is better).
  • The VCO tuning signal most probably would rail to +15V or -15V.
  • Reduce +/- 15V supply, this moves the railing voltage with it.
  • When the voltage rails reach +/2 V, the PLL catches the lock.
  • Now slowly ramp back the power supply back to +/- 15V.
• This way I was able to repeatedly catch the lock (see attachment 1), but of course, this can't be done when our board is mounted in the Eurocrat.
• So I thought if I attenuate the VCO tuning signal by 20 dB and pass it through an SR560, I can control the VCO tuning signal amplitude. This approach however did not work. It was always required to reduce the +/- 15V supply to the board to catch the lock.
• This makes me think that the phase detector chip AD9901 needs to be turned off initially or supplied with low voltage rails. I'm not sure why.
• With this, I think we should scrap this idea of using Moku as VCO, it will be just too unreliable.
• So we need to move to the possibility of feeding 22 MHz signals to the WFS demod board where VCO output goes.

Basically, we make our own PLL outside the board to generate 2 times LO frequency or we create 2 times LO frequency by second harmonic generation.

Moku:Pro as a frequency multiplier

This white paper from liquid instruments describes how Moku:Pro can be used as a frequency multiplier in the phasemeter app now. This functionality however has not been extended to Moku:Lab, so in 40m, we can not do this right now. If we get access to Moku:Pro, following will be required:

• Send 11 MHz LO signal to Moku:Pro input 1 with phasemeter app on.
• Select frequency multiplier option of 2 at the output 1. Set voltage to 2 Vpp and feed this signal to VCO RF out port on the modified demod board.
• Leave VCO tuning port unconnected.
• This way we would replace the internal PLL with Moku digital PLL. Moku's PLL can be run upto 10 kHz bandwidth and would be very robust for such use.

Second harmonic generation using mixer and bandpass filter

• Split the 14 dBm 11 Mhz output from frequency generation box (I simulated this with benchtop function generator) using a splitter.
• Send both outputs to ZP-3+ mixer (level 7).
• Filter the output with SBP-21.4 band pass filter. Koji has measured this filter in 2013. See elog 40m/9010.
• Amplify the output twice, first with ZFL-500HLN+ (20dB amplification), then with ZFL-1HADX (11 dB).
• This setup provides enought output amplitude for the comparator chip AD96687 to generate clean ECL signal at 22 MHz without slipping. With only 20dB amplification, I could see the phase slip by 180 degrees enough times that the oscilloscope shows both outputs overlapped.
• Attachment 2 shows the ICLK and QCLK signals generated by the board with this setup.

Next steps:

• I'll modify one more board for sending in LO like this.
• I'll test the demodulation performance of the board with LO input from the second harmonic generation.
• Setup the optical path for AS WFS.

That's great - I think this solution will be best. Having the PLLs actually gives us some problems - the square wave action in these demod boards because of the ECL drives pollutes the air with all the harmonics.

In the future, it would be best to get rid of these boards and just use the new aLIGO boards with a direct LO feed.

Based on the previous two elog posts, Koji and I decided that we should use 11 MHz signal for arm cavity ASC and modify a spare WFS demod board to work at 11 MHz. This board LIGO-D980233, uses a PLL to lock the to LO input and generate I and Q ECL clock signals from it. For this purpose, it uses POS-XX minicircuits VCO. For IMC WFS boards the model number is POS-75 and with the board design, it can work for 18.75 MHz to 37.5 MHz modulation frequencies.

To make it work for 11 MHz, we have to swap this with POS-25 but that is not available for purchase anywhere. So Koji and I decided to use Moku:GO as a VCO and make connections to the pin holes on the board. Today, I modified a spare WFS board to make this possible. I added a right angle SMA connector to take in VCO output signal and a BNC connector to send out tuning signal. See attached photos for the details of this hack.

Then I went to 1X2 and tried on this modified board on a Euro crate empty slot. I used Moku:GO in a multi-instrument mode in which first instrument was a Waveform generator set to modulate from external input 1 at 6 MHz/V. The output RF level was checked on an oscilloscope and increased until I got about 9.5 dBm power at the output. The second instrument was just an spectrum analyzer to see if the test output from ICLK looks ok. I fed LO from a spare output port on RF distribution box for 11 MHz signal. I made sure to attenuate this signal to get 2 dBm LO signal which is the case for the WFS demod board LO input as well. 

This test however failed. I could not see any signal from ICLK or QCLK output. I then tried to use the same slot as the demod board for WFS1 is used and I still did not see any output on ICLK or QCLK. I split the VCO tuning signal coming from the board to see it on an oscilloscope and it was mostly noise of ~1 mV level. I then tried to check ICLK and QCLK on oscilloscope and saw that they had a huge offset of -1.7 V. I suspect some ground mismatch issue between Moku:GO and the demod board.

I decided to call it a day here.

I reset everything back to how it was on the rack and turned on IMC WFS again. It is working as usual keeping lock steady for atleast last 20 min that I have seen it.

[Koji, Anchal]

short version: We checked signals at different points in the circuit to make sense of why it was not working. We found out that teh comparator chip AD96687BR was not working as expected and was not converting the analog signal from our VCO or LO inputs to ECL. We tested 2 other spare board with same behavior. We decided to try replacing the comparator chip with a new one, and indeed that was the issue. The new chip was working as expected and we are able to get PLL lock on the board with Moku:Lab as the VCO. However, there are some issues that need to be ironed out. The PLL does not catch lock right away and we could not figure our a systematic way of reaching to a locked state. That smells fishy to me as in my experience, when PLLs work, they work very robustly. More analysis with data and figures will follow. For now, we have some hope that this can work.

There is always the option of not closing PLL loop and injected twice the demodulation frequency at the VCO port that we have access two. For this, I'll need to create a SHG unit for 11 MHz with 21.4 MHz BLP. I'll look into this solution as well.

I just sat down in the control room, and discovered the PMC (and everything else) unlocked.  I relocked the PMC, but the MC wasn't coming back.  After a moment of looking around, I discovered that the WFS were on, and railing.  I ran the "turn WFS off" script, and the MC came back right away, and the WFS came on as they should.

We need to relook at the WFS script, or the MC down script, to make sure that any time the MC is unlocked, no matter why it unlocked, the WFS output is off and the filter histories are cleared.

    The only script that can currently take this action is the MC autolocker.  If that is disabled first and the PMC unlocks later, the WFS will not be turned off.  During the last round of discussions we had about the autolocker script, sometime last Nov, we decided that too much automation is not desirable and that the autolocker should be kept as simple as possible.

I found that the MC WFS had large offset control signals going to the MC SUS. Even though the input switch was off, the integrators were holding the offset.

I have disabled the ASCPIT outputs in the MC SUS. Suresh is going to fix the MC autolocker script to gracefully handle the OFF and ON and then test the script before resuming the WFS testing.

MCL data for OAF may be suspect from this morning.

[aaron, rana]

While going to take some transfer functions of the MC WFS loop, LSC was down. When we tried to restart the FE using 'rtcds restart --all', c1lsc crashed and froze. We manually reset c1lsc, then laboriously determined the correct order of machines to reboot. Here's what works best:

on c1lsc:

rtcds start c1x04 c1lsc c1ass c1oaf c1cal c1daf

Starting c1dnn crashes the other FE

on c1ioo

rtcds restart --all

on c1sus

rtcds restart c1rfm c1sus c1mcs

restarting c1pem crashes the other FE on c1sus

We're seeing a lot of red IPC indicators--perhaps it's an issue with the order we're restarting?

via Polish chat, GV tells us to RTFE

As suggested, I ran the script cds/rebootC1LSC.sh

I got a timeout error when the script tried closing the PSL shutter ('C1:AUX-PSL_ShutterRqst' not found), but Rana and I closed the shutter before leaving last night. c1sus is down, so the script found no route to host c1sus; I'm thinking I need to reset c1sus for the script to run completely. Nonetheless, c1lsc was rebooted, which crashed c1ioo and left the c1lsc FE all red (probably because c1sus wasn't restarted).

Rebooting

I reset c1lsc, c1sus, and c1ioo.

I noticed that the script gives the command 'ssh c1XXX', but we have been getting no route to host using this command. Instead, the machines are currently only reachable as c1XXX.martian. I'm not sure why this is, so I just appended .martian in rebootC1LSC.sh

This time, the script does run. I did get 'no route to host' on c1ioo, so I think I need to reset that machine again. After reset, the script failed to login to c1ioo and c1lsc.

Fri Sep 6 13:09:05 2019

After lunch, I reset the computers again, and try the script again. There is again no route to host for c1ioo. I'm going inside to shutoff the power to c1ioo, since the reset buttom seems to not be working. I still can't login from nodus, so I'm bringing a keyboard and monitor over to plug in directly.

On reset, c1ioo repeatedly reaches the screen in attachment 1, before going black. Holding down shift or ctrl+alt+f1 doesn't get me a command prompt. After waiting/searching the elog for >>3 min, we decided to follow these instructions to cycle the power of c1ioo. The same problem recurred following power up. I found online some instructions that the SunSystems 4600 can hang during reboot if it has become too hot ("reboot during a thermal shutdown"); I did notice that the temperature light was on earlier in this procedure, so perhaps that is the problem. I followed the wiki instructions to shut down the computer again (pressed power button, unplugged 4 power supplies from back of machine), and left it unplugged for 10-30 min (Fri Sep 6 14:46:18 2019 ).

Fri Sep 6 15:03:31 2019

Rana plugged in the power supplies and reset the machine again.

Fri Sep 6 16:30:37 2019

c1ioo is still unreachable! I pressed reset once, and the reset button flashes white. The yellow warning light is still on.

Fri Sep 6 16:54:21 2019

The reset light has stopped flashing, but I still can't access c1ioo. I reset once more, this time watching c1ioo on a monitor directly. I'm still seeing the same boot screen repeatedly. I do see that CPU0 is not clocking, which seems weird.

Troubleshooting CPU module

Following gautam's elog here, I found the Sun Fire X4600 manual for locating faulty CPUs. After the white reset light stopped flashing, I held down the power button to turn off the system. Before shutdown, all of the CPU displayed amber lights; after shutdown, only the leftmost CPU (as viewed from the back, presumably CPU0) displays an amber light. The manual says this is evidence that the CPU or DIMM is faulty. Following the manual, I remove the standby power, then checked out these Instructions for replacing the CPU to remove the CPU; Gautam also has done this before.

Fri Sep 6 20:09:01 2019 Fri Sep 6 20:09:02 2019

I pulled the leftmost CPU module out, following the instructions above. The CPU module matches the physical layout and part number of the Sun Fire X4600 M2 8-DIMM CPU module; pressing the fault reminder light gives amber indicators at the DIMM ejectors, indicating faulty DIMMs (see). The other indicator LEDs did not illuminate.

I located several spare DIMMs in the digital cabinet along Y arm (and a couple with misc computer components in the control room), but didn't find the correct one for this CPU module. The DIMM is Sun PN 371-1764-01; I found it online and ordered eight. Please let me know if this is incorrect.

To protect the CPU module, I've put it in an ESD safe bag with some bubble wrap and a note. It's on the E shop bench.

Conclusion: Need new DIMM, didn't find the correct part but ordered it.

Assuming you are at pianosa, /etc/resolv.conf is like

# Generated by NetworkManager
nameserver 192.168.113.104
nameserver 8.8.8.8

But this should be like

nameserver 192.168.113.104
nameserver 131.215.125.1
nameserver 8.8.8.8

search martian

as indicated in https://nodus.ligo.caltech.edu:8081/40m/14767

I did this change for now. But this might get overridden by Network Manager.

The MC unlocked ~20 min ago, correlated with 2 consecutive earthquakes in Mexico.  The MC came back fine after a few minutes, but the WFS never engaged.  I turned them on by hand.  I think that Yuta mentioned once that he also had to turn the WFS on by hand.  There may be a problem in the unlock/relock catching that needs to be looked at, to make sure the WFS come back on automatically.

Also, someone (Masha and I) should look at the seismic BLRMS.  I have suspected for a few days that they're not telling us everything that we want to know.  Usually, if there's an earthquake close enough / big enough that it pops the MC out of lock, it is clear from the BLRMS that that's what happened, but right now it doesn't look like much of anything....just kind of flat for hours.

The arrangement of filters in the WFS loop filter banks have been altered, Rana will update with details of the motivation behind these changes. Here is how the screen looks now:

I have updated the C1IOO SDF table, and also the mcwfson script to reflect these changes. The latter has been svn committed.