Reboots for c1susaux, c1iscaux today.

MC autolocker and FSS loops were stuck because c1psl was unresponsive. I rebooted it and did a burtrestore to enable PSL locking. Then the IMC locked fine.

c1susaux and c1iscaux were also unresponsive so I keyed those crates as well, after taking the usual steps to avoid ITMX getting stuck - but it still got stuck when the Sat. Box. connectors were reconnected after the reboot, so I had to shake it loose with bias slider jiggling. This is annoying and also not very robust. I am afraid we are going to knock the ITMX magnets off at some point. Is this problem indicative of the fact that the ITMX magnets were somehow glued on in a skewed way? Or can we make the situation better by just tweaking the OSEM-holding fixtures on the cage?

In any case, I've started listing stuff down here for things we may want to do when we vent next.

MC autolocker was not working - PCdrive was railed at its upper rail for ~2 hours judging by the wall StripTool trace. I tried restarting the init processes on megatron, but that didn't fix the problem. The reason seems to have been related to c1iool0 failing - after keying the crate, autolocker came back fine and MC caught lock almost immediately.

Additionally, c1susaux, c1auxex,c1auxey and c1iscaux are also down. I'm not planning on using the IFO tonight so I am not going to reboot these now.

Eurocrate key turning reboots for c1susaux, c1auxex,c1auxey, c1iscaux, c1iscaux2 and c1aux. Usual precautions were taken for ITMX. Did burtrestore for c1iscaux andc1iscaux2  in order to restore the LSC PD whitening gains.

Un-related to this work: input pointing into PMC was tweaked as the PMC_REFL spot was pretty bright.

MC Autolocker was umnhappy because c1iool0 was unresponsive and hence it couldn't write to the "C1:IOO-MC_AUTOLOCK_BEAT" channel. I keyed the crate and IMC locked almost immediately. I'm moving this channel into the RTCDS model as we did for the IFO_STATE EPICS channel so that the autolocker isn't dependant on c1iool0 (which was the whole point of migrating the IFO-STATE variable anyways). I also commented out all of these channels in /cvs/cds/caltech/target/c1iool0/autolocker.db so that there aren't duplicate channels.

Steve reported problems getting the X arm locked. Alignment sliders were inaccessible. Eurocrate key turning reboots for c1susaux, c1auxex,c1auxey, c1iscaux and c1aux. Usual precautions were taken for ITMX.

This is becoming a once-a-week thing .

[ Gautam , Steve ]

c1susaux & c1iscaux were rebooted manually.

Eurocrate key turning reboots today morning for and c1susaux, c1auxey and c1iscaux. These were responding to ping but not telnet-able. Usual precautions were taken to minimize risk of ITMX getting stuck.

MC autolocker got stuck (judging by wall StripTool traces, it has been this way for ~7 hours) because c1psl was unresponsive so I power cycled it. Now MC is locked.

c1psl, c1susaux, and c1auxey today

[johannes, gautam]

It's been a while - but today, all slow machines (with the exception of c1auxex) were un-telnetable. c1psl, c1iool0, c1susaux, c1iscaux1, c1iscaux2, c1aux and c1auxey were rebooted. Usual satellite box unplugging was done to avoid ITMX getting stuck.

All slow machines (except c1auxex) were dead today, so I had to key them all. While I was at it, I also decided to update MC autolocker screen. Kira pointed out that I needed to change the EPCIS input type (in the RTCDS model) to a "binary input", as opposed to an "analog input", which I did. Model recompilation and restart went smooth. I had to go into the epics record manually to change the two choices to "ENABLE" and "DISABLE" as opposed to the default "ON" and "OFF". Anyways, long story short, MC autolocker controls are a bit more intuitive now I think.

Reboot for c1susaux and c1iscaux today. ITMX precautions were followed. Reboots went smoothly.

IMC is shuttered while Jon does PLL characterization...

Now relocked.

FSS slow wasn't running so PSL PZT voltage was swinging around a lot. Reason was that was c1psl unresponsive. I keyed the crate, now it's okay. Now ITMX is stuck - Johannes just told be about an un-elogged c1susaux reboot. Seems that ITMX got stuck at ~4:30pm yesterday PT. After some shaking, the optic was loosened. Please follow the procedure in future and if you do a reboot, please elog it and verify that the optic didn't get stuck.

Eurocrate key turning reboots today morning for and c1susaux, c1auxex and c1auxey. Usual precautions were taken to minimize risk of ITMX getting stuck.

The IFO hasn't been aligned in ~1week, so I recovered arm and PRM alignment by locking individual arms and also PRMI on carrier. I will try recovering DRMI locking in the evening.

As far as MC1 glitching is concerned, there hasn't been any major one (i.e. one in which MC1 is kicked by such a large amount that the autolocker can't lock the IMC) for the past 2 months - but the MC WFS offsets are an indication of when smaller glitches have taken place, and there were large DC offsets on the MC WFS servo outputs, which I offloaded to the DC MC suspension sliders using the MC WFS relief script.

I'd like for the save-restore routine that runs when the slow machines reboot to set the watchdog state default to OFF (currently, after a key-turning reboot, the watchdogs are enabled by default), but I'm not really sure how this whole system works. The relevant files seem to be in the directory /cvs/cds/caltech/target/c1susaux. There is a script in there called startup.cmd, which seems to be the initialization script that runs when the slow machine is rebooted. But looking at this file, it is not clear to me where the default values are loaded from? There are a few "saverestore" files in this directory as well:

• saverestore.sav
• saverestore.savB
• saverestore.sav.bu
• saverestore.req

Are the "default" channel values loaded from one of these?

Eurocrate key turning reboots today morning for c1psl and c1aux.c1auxex and c1auxey are also down but I didn't bother keying them for now. PSL FSS slow loop is now active again (its inactivity was what prompted me to check status of the slow machines).

Note that the EPCIS channels for PSL shutter are hosted on c1aux.But looks like the slow machine became unresponsive at some point during the weekend, so plotting the trend data for the PSL shutter channel would have you believe that the PSL shutter was open all the time. But the MC_REFL DC channel tells a different story - it suggests that the PSL shutter was closed at ~4AM on Sunday, presumably by the vacuum interlock system. I wonder:

1. How does the vacuum interlock close the PSL shutter? Is there a non-EPICS channel path? Because if the slow machine happens to be unresponsive when the interlock wants to close the PSL shutter via EPICS commands, it will be unable to. The fact that the PSL shutter did close suggests that there is indeed another path.
2. We should add some feature to the vacuum interlock (if it doesn't already exist) such that the PSL shutter isn't accidentally re-opened until any vacuum related issues are resolved. Steve was immediately able to identify that the problem was vacuum related, but I think I would have just re-opened the PSL shutter thinking that the issue was slow computer related.

The folding crane was fixed and tested this morning by the NNN rigging company. Pictures will be posted by Steve in the morning.

Afterwards, the ITM-east door was installed, jam-nuts checked. No high voltage was on for the in-vac PZTs.

The annulus spaces were roughed down to 350mTorr by Roughing Pump RP1. For this operation, we removed the low flow valve from the RP1 line.

After the spaces came down to ~400 mTorr, we closed their individual valves.

Warning: The VABSSC1 and VABSSC0 valves are incorrect and misleadingly drawn on the Vacuum overview screen.

We then:

1. Closed V6 (valve between RP1 and the annulus line).
2. Turned off RP1 from the MEDM screen.
3. Installed the soft -starting butterfly valve.
4. Turned on RP1.
5. Opened V3.
6. Closed VV1 (at the last minute - this is a vent valve and must be checked before each pumpdown)
7. and pumpdown was started with a 3/4 turn opening of manual valve RV1.

Our idea is to have a much slower pumpdown this time than the last time when we had a hurricane kick up the dust. Looks like it worked, but next time we should do only 1/2 turn.

The pumpdown started at 4 PM (2300 UTC). At 10 PM, we (Jenne, Jan, and I) opened up the RV1 valve to full open. That's the second inflection point in the plot.

As per Steve's instructions, at 12:43 AM, I used the following steps to stop the pumpdown until the morning:

1. Close RV1 using the 'steering-wheel' wrench.
2. Close V3.
3. Turn OFF RP1.
4. Disconnect RP1 hose at the plastic disconnect attached to the slow-start throttle valve.

We have reached 200 Torr at 12 hours of slow pumping speed. Kiwamu stopped the pumping for 11 hrs last night .and I restarted it this morning.

Now RV1 is fully open with butterfly valve in place  and the second roughing  pump RP3 was just turned on.

How to stop pumping:

1,  close RV1 manual valve with torque wheel

2, close V3

3, turn off roughing pumps RP1 & RP3

4, disconnect metal hose connection to butterfly valve

The pump down continued this morning by the removal of the butterfly valve. Two roughing pumps were used to reach 500 mTorr

The Maglev monitoring MEDM screen "Rotating" indicator is not working. It is on all times. Please look at Maglev controller monitor for real information.

Pump down is completed.           

Configuration: vacuum normal after 86 days at atm               CC1 = 1e-5 Torr                  

                      IFO is hungry for light (and maybe some goulash with a little paprikash too)

 Atm 2 is showing the butterfly valve that closes down down the orifice at higher pressure to slow down the pumping speed.

 See elog entry #2573

Bob and Steve closed BS chamber with the help of the manual Genie lift and the pump down started. The PSL shutter was closed and manual block was placed in the beam path. High voltage power supplies were checked to be off.

Pumping speed ~ 1 Torr/min was achieved at  1/8 of a turn opened roughing valve RV1

I have installed a slow start throttle valve in front of V3  This spring loaded valve will cut down on the flow at high pressures. There will be no more sand storme

and static built-up during pump down.

The PSL shutter was closed. The beam path  blocked two places. High voltage power supplies to IOO and OMC PZT were checked to be off. Oplevs are off. 

The south arm green cavity was misaligned yesterday

We would like to keep the vent speed at 1 torr / min. I'm venting with N2 now up to 25 PSI. We have 3 cylinder  of instrument grade air inside the lab. Additional supply will arrive later. It can be as late as 1pm

Blocked PSL output beam  into IFO

Checked: HV at IOO & OMC are off, jam nuts in position,

Closed V1 and VM3, opened VV1 to N2 regulator

We are venting at 1 Torr/min rate

For those of you who want to see plots from slower scan.

Small earth quakes and suspensions. Which one is the most free and most sensitive: ITMX

If I catch anyone putting small booties into the large bootie bin, I will make said person eat small booties.

Gautam and Steve,

The medm monitor & vac control screens were totally blank since ~ May 24, 2017    Experienced vacuum knowledge is required for this job.

IDENTIFY valve configuration:

                        How to confirm valve configuration when all vac mons are blank?  Each valve has a manual-mechanical position indicator. Look at pressure readings and turbo pump controllers. VAC NORMAL configuration was confirmed based on these information.

Preparation: disconnect valves ( disconnect meaning: valve closes and stays paralized ) in this sequence VC2, VC1 power, VA6, V5, V4 & V1 power,      at ifo pressure 7.3E-6 Torr-it  ( it  = InstruTech cold cathode gauge )

                            This gauge is independent from all other rack  mounted   instrumentation and it is still not logged.

                            Switching to this valve configuration with disconnected valves will insure NOT  venting of the vacuum envelope by accidental glitching voltage drop or computer malfunction.

RESET  v1Vac1 .........in 2-3 minutes........ ( v1Vac1 - 2 )  the vac control screen started reading pressures & position

                    Connected cables to valves (meaning: valve will open if it was open before it was disconnected and it will be control able from computer ) in the following order: V4, V1 power, V5, VA6, VC2 & VC1 power,      at ifo 2E-5 Torr-it.....

                     ....vac configuration is reading VAC NORMAL,

                     ifo 7.4E-6 Torr-it

We have to hook up the it-cold cathode gauge to be monitored - logged !  this should be the substitute for the out of order CC1 pressure gauge.

We corrected the MICH locking snap file C1configure_MI.req and saved an updated C1configure_MI.snap. Now the 'Restore MICH' script in IFO_CONFIGURE>!MICH>Restore MICH works. The corrections included adding the correct rows of PD_DOF matrices to be at the right settings (use AS55 as error signal). The MICH_A_GAIN and MICH_B_GAIN needed to be saved as well.

We also were able to get to PRMI SB resonance. PRM was misalgined earlier from optimal position and after some manual aligning, we were able to get it to lock just by hitting IFO_CONFIGURE>!PRMI>Restore PRMI SB (3f).

Yesterday, I had the great pleasure to solder a tiny 4 x 4 mm op amp with 16 legs (AD8336).

I figured out that the best and fastest way to do it is

1. to put solder with the soldering iron on every contact of the electronic board (top side)
2. heat the bottom side of the electronic board with a heat gun
3. use a needle to test if the solder is melted
4. if it is melted place the op amp on the electronic board
5. apply some vertical force on the op amp for proper contact and heat for 1 to 2 more minutes
6. done

This morning the pencil soldering iron of our Weller WD2000M Soldering Station suddenly stopped working and got cold after I turned the station on. The unit's display is showing a message that says "TIP". i checked out the manual, but it doesn't say anything about that. I don't know what it means. Perhaps burned tip?

Before asking Steve to buy a new one, I emailed Weller about the problem.

 There should be a supply of extra tips in the Blue Spinny Cabiney (I can never remember it's French name....)  The drawer is something like the top row of one of the bottom sets of drawers.  You can pick the shape of tip you want, and stick it in.

Albeto and Koji

We took the tip replacement from the blue tower.

I am looking at http://www.cooperhandtools.com/brands/weller/ for ordering the tips.

The burnt one seems to be "0054460699: RT6 Round Sloped Tip Cartridge for WMRP Pencil" We will buy one.

The replaced one is "0054460299: RT2 Fine Point Cartridge for WMRP Pencil" We will buy two.

I like to try this: "0054460999: RT9 Chisel Tip Cartridge for WMRP Pencil" We will buy one.

Q and Ignacio were taking a second look at the Pentek interface board which we're using to acquire the POP QPD, ALS trans, and MCF/MCL channels. It has a differential intput, two jumper able whitening stages inside and some low pass filtering.

I noticed that each channel has a 1.5 kHz pole associate with each 150:15 whitening stage. It also has 2 2nd order Butterworth low pass at 800 Hz. Also there's a RF filter on the front end. We don't need all that low passing, so I started modifying the filters. Tonight I moved the 800 Hz poles to 8000 Hz. Tomorrow we'll move the others if Steve can find us enough (> 16) 1 nF SMD caps (1206 NPO).

After this those signals ought to have less phase lag and more signal above 1 kHz. Since the ADC is running at 64 kHz, we don't need any analog filtering below 8 kHz.

Solid works 2010 was installed to m3, an windows machine in the control room.

Have fun !

Each bottle has matched seals. They are not interchangeable.

It is critical that the solvent do not reach the rubber bulb. Practice with the pipet.

In case of solvent touching the suction bulb: do not let the solvent go back into the bottle! Remove bulb, let it dry out and rinse pipet.

It is essential that the solvent bottle must be rinsed and refilled if it's content met with the rubber bulb.

Use glass syringe with SS needle in critical application: Hamilton ~0.1 ml

I ran the ASS/ADS for the arms because the X-arm was way out. There was also some problem with its locking due to bad ramps in FM2. I copied over the filters from YARM and then adjusted some of the ramps and thresh trigs in the filter file until the transients in POX got smaller. Basically, you should not really be ramping on Integrators. Secondly, we should do some testing when adjusting the filter parameters.

I hooked up the 4395 to the MC servo board OUT2 so that we can monitor the error point when the PCDRIVE goes nuts.

Somehow some DAQ channels for C1SUS have disappeared from the DAQ channel list.

Indeed there are only a few DAQ channels listed in the C1SUS.ini file.

I ran the activateDQ.py and restarted daqd.

Everything looks okay.  C1SUS and C1PEM were restarted because they became frozen.

I found again the ini files had been refreshed.

I ran the activateDQ.py script (link to the script wiki page) and restarted the daqd process on fb.

The activateDQ.py script should be included into the recompile or rebuild scripts so that we don't have to run the script everytime by hands.

I am going to add this topic on the CDS todo list (wiki page).

Found some LSC scripts didn't run on pianosa. Particularly all the scripts on the C1:IFO_CONFIGURE screen don't run.

They need to be fixed.

When I came to the 40m, I found most of the FB signals are dead.

The suspensions were not dumped but not too much excited. Use watchdog switches to cut off the coil actuators.

Restarted mxstream from the CDS_FE_STATUS screen. The c1lsc processes got fine. But the FB indicators for c1sus, c1ioo, c1iscex/y are still red.

Sshed into c1sus/ioo, run rtcds restart all . This made them came back under control.

Same treatment for c2iscex and c1iscey. This made c1sus stall again. Also c1iscey did not come back.

At this point I decided to kill all of the rt processes on c1sus/c1ioo/c1iscex/c1iscey to avoid interference between them.
And started to restart from the end machines.

c1iscex did not come back by rtcds restart all.
Run lsmod on c1iscey and found c1x05 persisted stay on the kernel. rmmod did not remove the c1x05 module.
Run software reboot of c1iscey. => c1iscey came back online.

c1iscey did not come back by rtcds restart all.
Run software reboot of c1iscex. => c1iscex came back online.

c1ioo just came back by rtcds restart all.

c1sus did not come back by rtcds restart all.
Run software reboot of c1sus => c1sus came back online.

This series of restarting made the fb connections of some of the c1lsc processes screwed up.
Run the following restarting commands => all of the process are running with FB connection.
rtcds restart c1sup
rtcds restart c1ass
rtcds restart c1lsc

Enable damping loops by reverting the watchdog switches.

All of the FE status are green except for the c1rfm bit 2 (GE FANUC RFM CARD 0).

First, things that were done:

• I was troubled by the odd-looking noise in the EOM temperature signal, so I investigated the circuit with a probe and found that there was quite a bit of line pickup, which I traced to the wires going to and from the RTD (if I placed a dummy resistor directly on the board, it went down markedly).
  • I put a 3-Hz RC LPF between the AD620 and the driver input buffer, which reduced the line noise significantly
  • The error signal looks much cleaner and there are no longer strong peaks in the error spectrum at ~1+ Hz and harmonics
• I had tried earlier to increase the gain of the servo at the driver input stage. It seemed to stay stable. Since I knew the error signal  with the loop closed was at the level of the ADC noise, I decided to push my luck with increasing the servo gain and juice up the AD620 gain from 100 to 990.
  • The servo stayed stable and the error signal level is now manageable.

Things that I noticed:

• With the latest increase in gain, I measured that the error signal was suppressed with the loop closed (the suppression is below ~0.1 Hz, and the reason that the high-frequency level is different is because it has been amplified above the ADC noise by the time of the second trace).

• Despite the above, the Stochmon signals remained unchanged no matter what I did. I noticed that the Stochmon signals, too, were fluctuating basically at bit-level. I terminated the 11-MHz signal and compared it to the normal level---it is not exactly the same, but only a factor of 2-3 lower, which is not great. Of course, the RMS detector is logarithmic, but I think we still want the dark noise to be at least an order of magnitude lower here.
  • I tried to amplify the signal with an SR560, but since the DC level is supposed to be ~1-2 V, I could only get about 2x gain---not enough.

Conclusion

I think there are two things that could be happening here, given the above information:

1. We are limited by the noise of the temperature sensing circuit, which would explain why the in-loop error signal is suppressed while the RAM levels appear not to be. This should be easy enough to test (though there's not enough time right now) with an out-of-loop sensor.
2. The RAM is not dominated by temperature noise here. With the loop open, one would expect to see coherence between the RAM signal and the temperature sensor, if indeed one was the cause of the other. Instead, we see that---while the 11- and 55-MHz signals ARE pretty coherent with each other---there is no appreciable coherence between the temperature and the Stochmon signal.

About the analog CARM control with ALS:

We're looking at using a Sigg designed remotely switchable delay line box on the currently undelayed side of the ALS DFD beat. For a beat frequency of 50MHz, one cycle is 20ns, this thing has 24ns total delay capability, so we should be able to get pretty close to a zero crossing of the analog I or Q outputs of the demod board. This can be used as IN2 for the common mode board. 

Gautam is testing the functionality of the delay and switching, and should post a link to the DCC page of the schematic. Rana and Koji have been discussing the implementation of the remote switching (RCG vs. VME). 

I spent some time this afternoon trying to lock the X arm in this way, but instead of at IR resonance, just wherever the I output of the DFD had a zero crossing. However, I didn't give enough thought to the loop shapes; Koji helped me think it through. Tomorrow, I'll make a little pomona box to go before the CM IN2 that will give the ALS loop shape a pole where we expect the CARM coupled cavity pole to be (~120Hz), so that the REFL11 and ALS signals have a similar shape when we're trying to transition. 

The common mode board does have a filter for this kind of thing for single arm tests, but puts in a zero as well, as it expects the single arm pole, which isn't present in the ALS sensing, so maybe I'll whip up something appropriate for this, too. 

Something odd is happening with the CM board. Measuring from either input to OUT1 (the "slow output") shows a nice flat response up until many 10s of kHz. 

However, when I connect my idependently confirmed 120Hz LPF to either input, the pole frequency gets moved up to ~360Hz and the DC gain falls some 10dB. This happens regardless if the input is used or not, I saw this shape at a tee on the output of the LPF when the other leg of the tee was connected to a CM board input. 

This has sabotaged my high bandwidth ALS efforts. I will investigate the board's input situation tomorrow.

1. Unresponsive c1psl, c1iool0, c1auxey and c1iscaux VME crates were keyed.
2. c1psl channels were burt-restored, did a burtrestore, and re-locked the PMC. Tweaked the pointing into the PMC on the PSL table to increase the PMC transmission from ~0.69 to ~0.71.
3. Re-locked IMC. Ran WFS offset script to relieve the ~100 DAC counts (~10 urad) DC offset from the WFS servos to the IMC suspensions (a serious calibration of this into physical units should be made part of the planned 40m WFS activity). Now that I think about it, since we change the IMC alignment to match the input beam alignment, some post-IMC clipping could modulate the power incident on the ITMs, which is a source of error for the arm cavity loss determination using DC reflection. We need a better normalizing data stream than the IMC transmission.
4. The IFO_OVEREVIEW medm screen was modified such that the threshold for the PMC transmitted beam to be visible was lowered from 0.7 to 0.6, so that now there is a continuous beam line from the NPRO to the PRM when the IMC is locked even when the PMC transmission degrades by 5% due to thermally driven pointing drifts on the PSL table.
5. The wmctrl utility on pianosa wasn't working so well, I wasn't able to use my usual locking MEDM autoconfig scripts. Turned out to be due to a zombie MEDM window which I killed with xkill, now it is working okay again.
6. The misaligned XARM was re-aligned and the loss measuring PDA520 at the AS port was removed from the beam path (mainly to avoid ADC saturations the fringing Michelson will cause).
7. I noticed that the ETMX Oplev HeNe SUM level has degraded to ~50% of its power level from 200 days ago [Attachment #1], may need a new HeNe here soon. @Chub, do we have spare HeNes in stock?

I want to collect some data with the arms locked to investigate the possibility/usefullness of having seismic feedforward implemented for the arms (it is already known to help the IMC length and PRC angular stability at low frequencies). To facilitate diagnostics I modified the file /users/Templates/Seismic/Seismic_vs_TRXTRYandMC.xml to have the correct channel names in light of Lydia's channel name changes in 2016. Looking at the coherence data, the alignment of the cartesian coordinate system of the Seismometers at the ends and the global interferometer coordinate system can be improved.

I don't know if for the MISO filter design if there is any difference in using TRX/TRY as the target, or the arm length control signal.

Data collection started at 1249018179. I've setup a script running in a tmux shell to turn off the LSC enable in 2 hours.