[Jenne, Koji]

This disturbance in the MC ASC channels were fixed.

This craziness happened ~10pm last night. Was there any action at the time? >> Sunday-night workers? (RXA: No, Nakano-kun and I left before 9:30 PM)

We found that the signals came from c1ioo. However, restarting, recompiling c1ioo and c1mcs didn't help
to clean up this issue. Just in case we cleaned up the corresponding entries in the ipc file /opt/rtcds/caltech/c1/chans/ipc/C1.ipc
and recomplied c1ioo and c1mcs because these are the channels we touched last week to mitigate the timing out issue of c1rfm.

Incidentally, we fell into a strange mode of the RCG: IOPs could not restart. We ended up running "sudo shutdown -r now"
on each machine (except for c1lsc which was not affected by this issue). This solved the issue.

Even now c1oaf could not be running properly. This is not affecting the IFO operation right now, but we need to look into this issue again
in order to utilize OAF.

- An Aluminum mirror instead of 2" unknown mirror for the pick-off for the rejected beam from the green faraday isolator (Steve)
=> Replaced. To be reviewed

- Faraday mount replacement. Check what we have for the replacement. (Steve)

- The green REFL PD should be closer to the pick-off mirror. (Steve)
=> Moved. To be reviewed

- A beam dump should be placed for the green REFL PD

- Move the green shutter to the place where the spot is small (Steve)
=> Moved. To be reviewed.

- The pole of the PZT mounting should be replaced with a reasonable one. (Steve with Manasa's supervision)

- Tidying up doubling oven cable. Make a hole on the wall. (Steve)
=> Done. To be reviewed.

- Tidying up the PZT cabling (Steve)

- The optics are dirty. To be drag wiped. (Manasa, Masayuki)

As I always tell everyone: Don't use a 10% reflector which produce ghost beams. Use a 90% reflector.

[Jenne Koji]

It seems that the PRM violin mode freqs shifted from 625-ish to 640Hz.
The peaks rang up because of the servo.

Once the notch freq was shifted to 640Hz, the violin mode started to decay.

ellip("BandStop",4,1,90,636,644) gain(1.12202)

Don't go for a hacky solution. We want to climb a staircase step by step.
Prepare an independent 110MHz demod ports.

Friday night locking

Much more stable DRMI lock was achieved, partly thanks to the Friday-night quiet seismic,
and partly because of the improved servo gain and LF boosts

55MHz thru-put

I wanted to confirm the enhancement of the 110MHz signal at the AS port.

As the AS110 PD is placed in the CCD path, there is nothing visible with PRMI.
The Thorlabs PD was moved to the main AS path. Now the AS110 PD is receiving 50% of the power.
 

With PRMI 110MHz peak was -30dBm (As it was fluctuating, anything more precise number did not make sense)
When the DRMI was locked, the peak was enhanced to 0dBm.

The 2f signal comes from the beat between the sidebands.
Thus the amplitude of the intensity is proportional to the power of the sidebands (assuming the +1 and -1 order sidebands have the same amplitude)
-30dBm -> 0dBm means 31.6 times amplitude of the intensity. Therefore the amplitude transmission of the sidebands is 5.6 times more. (Is this true?)

According to the wiki, the AS port thru-put (i.e. power transmission) for the 55MHz sideband is 0.0026 and 0.43 for PRMI and DRMI respectively.
This corresponds to the amplitude difference of ~13. So we still have only half of the sidebands leaking out from the IFO. This could be attributed
to both the smaller PR gain and SR gain.

Locking setup

Same as the one Jenne used the other day. Later I engaged several additional triggers.
The following is the trigger setting I used

MICH: Delay 2 sec, FM1/FM2/FM3/FM6/FM7

PRCL: Delay 0.5 sec, FM2/FM3/FM6

SRCL: Delay 5 sec, FM1/FM2/FM3/FM6

SRCL FM1 was modified from +3dB to +6dB

Lock stability

Once lock is acquired, it lasts tens of minutes. (see the attached striptool chart.)
Even the lock is lost, it reacquires quickly.

The videos to show the lock acquisition and the in-lock stability are attached below.
The AS port beam is very round. It is not so shaky, but some yaw motion is visible.
The mode at the AS port is defined by the SRM, putting a QPD at the AS port would help to
stabilize the spot.

IFO state upon leaving

I left the 40m with the arms aligned, PRM and SRM slightly misaligned, and LSC setting is for the DRMI locking.

TO DO

- AS110I/Q for triggering

- PRCL/MICH/SRCL normalization

- We should resurrect the IFO config scripts.

- Remove BS->SRCL actuation coupling

- Handing off to 3f signals (preparation for the full lock)

- Improve ALS stability

- SRM ASC: AS QPD for SRM control

Lock Acquisition Video
UL (REFL) / UR (POP)
LL (AS) / LR (PRM Face)

 
In-lock video
UL (REFL) / UR (POP)
LL (AS) / LR (PRM Face)

What do you mean???

What is the effect of the anti-whitening filter?

- Do we have an appropriate amplifier?

- True challenge could be to find a feedthrough for the new port. (or to find a space for the amplifier in the box)

- PDXXX channels is on the DC whitening filter module. There could be some modification on this module (like diabling the whitening gain selector).

- We don't have AS11 and AS165, and so far it is unlikely to use AS11. i.e. The feedthrough, the slot on the crate, the whitening, and the channels can be trasnsition from 11 to 110.

The reason we had the PCIe/RFM system was to test this mixed configuration in prior to the actual implementation at the sites.
Has this configuration been intesively tested at the site with practical configuration?

It's hard to believe that c1lsc -> c1sus only has 4 channels. We actuate ITMX/Y/BS/PRM/SRM for the length control.
In addition to these, we control the angles of ITMX/Y/BS/PRM (and SRM in future) via c1ass model on c1lsc.
So there should be at least 12 connections (and more as I ignored MCL).

I personally prefers to give the PCIe card to c1ioo and move the RFM card to c1lsc.
But in either cases, we want to quantitatively compare what the current configuration is (not omitting the bridging by c1rfm),
and what the future configuration will be including the addtional channels we want add in close future,
because RFM connections are really costly and moving the RFM card to c1lsc may newly cause the timeout of c1lsc
just instead of c1sus.

Summary

Stable DRMI lock was recovered. The AS110 phase was adjusted. PRCL and MICH were locked with REFL33I and REFL165Q.
Still SRCL is controlled with REFL55Q.

PRMI sensing matrix

Thursday night, Jenne and I found DRMI can not be locked at all. Also the PRMI lock with REFL55 showed change in the optical gain.

In order to investigate what is happening, the PRMI sensing matrix was measured and compared with the previous one taken in the night of 8/26.

VS

It shows that some signals are unchanged, some are partial change, and some are completely different.
My intuition saids something is wierd with the sensing matrix measurement.
Right now I can't trust these plots.
- Jenne and I have adjusted REFL55 demod angle so that REFL55Q has no PRCL. And I have confirmed with DTT that this is still true.
  However, the radar chart shows that REFL55Q is almost correct phase for PRCL instead of MICH.

- REFL11 shows the same amplitude and angle as before. But POX11/POY11 shows different MICH angle.

- I have rotated REFL55 demod phase and remearsured the sensing matrix. Evrything else looked same but REFL55.
  Since REFL55I&Q were not used for the control for this measurement, what we expect is to see no change of the sensing matrix and
  only see the angle of "I"&"Q" rotates. But the result was different from the expectation.

DRMI locking

Since no real info was obtained from the sensing matrix, I had to make a fight without any weapon.
After sevral hours of work, stable DRMI lock was recovered.

Basically I gave larger gains to REFL55 signals: REFL55I for SRCL was 100 instead of 1, and REFL55Q for MICH was 2 instead of 0.1.
This was enough to get a second locking. Using this short sections, I have optimized the FM triggers and the gain boosts (i.e. FM1)
as well as the mirror alignment.

Then, PRM ASS was left running during the lock. This actually stabilized the lock a lot.
This made thee lock indefinite.

The demod phase of AS110I was adjusted so that AS110Q fluctuates around zero.
In this condition, the nominal AS110I was 7300 with the whitening gain of 30dB.

Note that the AS110I&Q were also measured with PRMI. With the same phase and gains, AS110I and Q were -35,  -170, respectively.
Do we expect to have this phase shift? If I believe these numbers, the aplitude of 110MHz at the optimal phase is 173,
The ratio of AS110 between DRMI and PRMI is 7300/173 = 42. This corresponds to the ratio of the 110MHz sideband power at the AS port.
According to the wiki, this ratio shoud be ~160.

AS110I was in fact glitchy as you can see in the StripTool chart. I wonder this signal is suitable for the normalization or not.

=== SENSING ===

REFL11 -67deg / whitening gain 0dB
REFL33 -20deg / whitening gain 30dB
REFL55 45deg / whitening gain 6dB
REFL165 96deg / whitening gain 45dB

POP110 69deg whitening on / 15dB
POP22 102.2deg whitening on / 21dB
AS110 145deg whitening off / 30dB (seems to be related to AS11 whitening setting)

=== INPUT MATRIX ===

REFL11I x -0.125 => PRCL (REFL33I x 2.5 was also OK)
REFL55I x 100 => SRCL
REFL55Q x 2 => MICH (REFL165Q x 0.1 was also OK)

=== NORMALIZATION / TRIGGER ===

No normalization

Trigger settings
MICH POP22I UP:50 DOWN:10
PRCL POP22I UP:50 DOWN:10
SRCL POP22I UP:50 DOWN:25

=== SERVO FILTERS ===

MICH x -0.8 FM4/5 ON, no limitter
FM Trigger: delay 2sec, FM1 (modified from 6dB to 20dB), FM2, FM3

PRCL x +0.035 FM4/5 ON, no limitter
FM Trigger: delay 0.5sec, FM2/3/6

SRCL x -0.1 FM4/5 ON, no limitter
FM Trigger: delay 5sec, FM1, FM2

=== OUTPUT FILTERS ===

MICH => PRM -0.267 / BS +0.5

PRCL => PRM +1.0

SRCL => SRM +1.0

=== VIOLIN FILTER TRIGGER ===

delay 1sec: FM1/FM2/FM3/FM6

=== ASC/ASS ===

PRM ASC UP:50 DOWN:25
PITCH&YAW: FM1/9 (ALWAYS ON) + FM2/3 (turned on by the up-script)

PRM ASS left turned on for slow tracking

What was the beat freq for each arm?
The HF noise level depends on the frequency of the beat note.
As the BBPD has the freq dependent noise level. (See this entry)

I vote on PRMI+1arm -> PRFPMI

While Manasa, Jenne, and Masayuki are working on the preparing the interferometer, I write the elog for them.

- 6PM-ish: X and Y arms were was locked. They were aligned with ASS.

- PRMI was locked. The PRM was aligned with ASS.

- Jenne went into the lab and aligned the PRM ASC QPD.

- Jenne also aligned all of the oplev spots except for the SRM.

- 6:40PM Then, Manasa and Masayuki checked the out-of-loop stability of the arms.
The X and Y arms have the rms of 2.2kHz and 600Hz, respectively.
The X arm is significantly worse than the Y arm.
Masayuki saved the plot somewhere in his directory.

- 7:20PM X beat: 41.2MHz, Y beat: 14.8MHz

- 7:22PM PRMI locked POP110 115-120 

- 7:30PM Lost lock of everything. Start over. Taking the arm alignment.

- 7:45PM start the 2nd trial. PRMI+one arm ready.

- 8:00PM explosion! Lost lock.

- 8:30PM The Xarm ALS is not stable anymore. It loses the control in ~10sec.
We are investigating the out-of-loop stability of the Yarm ALS.
(i.e. Look at the beat note error signal while locking the Yarm with the IR PDH)

PMC aligned. Trans 0.78 -> 0.83

> all of complications of handing off

That involves:

- ALS error signals transfered to the LSC input matrix.

- Handing off from the ALS to the 1/sqrt(TRX)+offset signal

- Handing off to the RF signal

- And, of course, CM servo.

 Really? What cavity length did you use in the calculation?

Yes, the resonance of the 2nd-order sidebands to the IFO screws up the 3f scheme.

2f (~22MHz) and 10f (~110MHz) are at x 5.6 and x 27.9 FSR from the carrier, so that's not the case.

Could we also see how much gain fluctuation of the 3f signals we would experience when the arm comes into the resonance?

 You have the data. Why don't you just calculate 1/SQRT(TRX)?

...yeah, you can calculate it but of course you don't have no any reference for the true displacement...

General Remarks on the BBPD

- To form the LC network: Use fixed SMD inductors from Coilcraft. SMD tunable capacitors are found in the shelf right next to Steve's desk.
  If the tuning is too coarse, combine an appropriate fixed ceramic SMC C and the tunable C (in parallel, of course)

- L1/C1a/C1b pads are specifically designed for an additional notch

- Another notch at the diode stage can be formed between the middle PD pin (just left of the marking "C3b") to the large GND pad (between C1a/C1b to C3a).
  You have to scratch off the green resin with a small flat screw driver (or anything similar)

- A notch at the amplifier stage can be formed between the output of MAR-6SM ("+" marking)  and one of the GND pads (left side of the "U1" marking)

- The original design of the PD is broadband. So additional notches on the diode stage provides notches and resonances.
  Check if the resonances do not hit the signal frequencies.

- One would think the PD can have resonant feature to reduce the coupling of the undesired signals.
  In some sense it is possible but it will be different from the usual resonant tank circuit in the following two points.

  * Just adding a parallel L between the cathode and ground does not work. As this DC current should be directed to the DC path,
    L&C combo should be added. In fact this actually give a notch-resonance pair. This C should be big enough so that you can ignore it
    at the target resonant frequency. Supply complimentary small C if necessary to keep low impedance of the Cs at the target frequency.
    (i.e. Check SRF - self-resonant frequency of the big C)

  * Since the input impedance of MAR-6SM is 50Ohm, the top of the resonant curve will be cut at 50Ohm. So the resultant shape looks
    like a bandpass rather than a resonance.

- So in total, simulation of the circuit is very important to shape the transimpedance. And, consider the circuit can not be formed as simulated
  because of many practical imperfections like stray Ls and Cs.

Unfortunately this does not work. These WFSs are not the detectors which we can move freely.
In order to move the WFS detectors, we need the precise design of the Gouy phase for each WFS heads.
Without the design, we can't move the detectors.

 Don't forget to run burtrestore for the target.

Can't we somehow hook up this camera to the MUX with the movie mode?
I think both the MUX and the sensoray are compatible with the color video signal.
Only the old CRT is B/W.

I forgot how we could turn on the PRM oplev servo and the PRM ASC servo at the same time without conflict.
It seems that this new oplev servo covers 0.04 to 8Hz. It's pretty broadband. Do we inject the ASC signal to the oplev error?

This morning I came in the 40m then found
1) c1auxex was throwing out the same errors as recently seen.
2) c1iscex processes had errors which persisted even after the mx stream reset.

1) c1auxex - fixed

Tried telnet c1auxex => rejected by the host

Went down to the south end. Power cycled the target. Came back to the control room.
=> Confirmed the epics read/write is back.
Burtrestored the epics vars for the target to the snapshot on 31th Oct at 5:07.

2) c1iscex - still not fixed

ssh c1iscex
rtcds restart all => c1x01 is still in red. 
Followed the procedure on the elog entry 9007. => Still the same error.

At least c1x01 is stalled. Here is the status.

Sync Source is TDS.
C1:DAQ-DC0_C1X01_STATUS is 0x2bad.
C1:DAQ-DC0_C1X01_CRC_SUM stays 0.
The screen shot is attached.

dmesg related to c1x01

controls@c1iscex ~ 0$ dmesg |grep c1x01
[   32.152010] c1x01: startup time is 1069440223
[   32.152012] c1x01: cpu clock 3000325
[   32.152014] c1x01: Epics shmem set at 0xffffc9001489c000
[   32.152208] c1x01: IPC at 0xffffc90018947000
[   32.152209] c1x01: Allocated daq shmem; set at 0xffffc9000480c000
[   32.152210] c1x01: configured to use 4 cards
[   32.152211] c1x01: Initializing PCI Modules
[   32.152226] c1x01: ADC card on bus b; device 4 prim b
[   32.152227] c1x01: adc card on bus b; device 4 prim b
[   32.154801] c1x01: pci0 = 0xdc300400
[   32.154837] c1x01: pci2 = 0xdc300000
[   32.154842] c1x01: ADC I/O address=0xdc300000  0xffffc90003f62000
[   32.154845] c1x01: BCR = 0x84060
[   32.154858] c1x01: RAG = 0x117d8
[   32.154861] c1x01: BCR = 0x84260
[   32.583220] c1x01: SSC = 0x16
[   32.583223] c1x01: IDBC = 0x1f
[   32.583236] c1x01: DAC card on bus 14; device 4 prim 14
[   32.583237] c1x01: dac card on bus 14; device 4
[   32.584527] c1x01: pci0 = 0xdc400400
[   32.584546] c1x01: dac pci2 = 0xdc400000
[   32.584551] c1x01: DAC I/O address=0xdc400000  0xffffc90003f6a000
[   32.584555] c1x01: DAC BCR = 0x810
[   32.584678] c1x01: DAC BCR after init = 0x30080
[   32.584681] c1x01: DAC CSR = 0xffff
[   32.584687] c1x01: DAC BOR = 0x3415
[   32.584693] c1x01: set_8111_prefetch: subsys=0x8114; vendor=0x10e3
[   32.584722] c1x01: Contec 1616 DIO card on bus 23; device 0
[   32.593429] c1x01: contec 1616 dio pci2 = 0x4001
[   32.593430] c1x01: contec 1616 diospace = 0x4000
[   32.593434] c1x01: contec dio pci2 card number= 0x0
[   32.593439] c1x01: Contec BO card on bus 18; device 0
[   32.593447] c1x01: contec dio pci2 = 0x3001
[   32.593448] c1x01: contec32L diospace = 0x3000
[   32.593451] c1x01: contec dio pci2 card number= 0x0
[   32.593456] c1x01: 5565 RFM card on bus 7; device 4
[   32.597218] Modules linked in: c1x01(+) open_mx mbuf
[   32.599939]  [<ffffffffa002e430>] mapRfm+0x71/0x392 [c1x01]
[   32.600199]  [<ffffffffa002ec91>] mapPciModules+0x540/0x8cf [c1x01]
[   32.600458]  [<ffffffffa002f2c1>] init_module+0x2a1/0x9d6 [c1x01]
[   32.600717]  [<ffffffffa002f020>] ? init_module+0x0/0x9d6 [c1x01]
[   32.616194] c1x01: RFM address is 0xd8000000
[   32.616196] c1x01: CSR address is 0xdc000000
[   32.616206] c1x01: Board id = 0x65
[   32.616209] c1x01: DMA address is 0xdc000400
[   32.616213] c1x01: 5565DMA at 0xffffc90003f72400
[   32.616215] c1x01: 5565 INTCR = 0xf010100
[   32.616217] c1x01: 5565 INTCR = 0xf000000
[   32.616218] c1x01: 5565 MODE = 0x43
[   32.616220] c1x01: 5565 DESC = 0x0
[   32.616232] c1x01: 5 PCI cards found
[   32.616233] c1x01: ***************************************************************************
[   32.616234] c1x01: 1 ADC cards found
[   32.616235] c1x01:     ADC 0 is a GSC_16AI64SSA module
[   32.616236] c1x01:         Channels = 64
[   32.616236] c1x01:         Firmware Rev = 34
[   32.616238] c1x01: ***************************************************************************
[   32.616239] c1x01: 1 DAC cards found
[   32.616239] c1x01:     DAC 0 is a GSC_16AO16 module
[   32.616240] c1x01:         Channels = 16
[   32.616241] c1x01:         Filters = None
[   32.616242] c1x01:         Output Type = Differential
[   32.616242] c1x01:         Firmware Rev = 6
[   32.616244] c1x01: MASTER DAC SLOT 0 1
[   32.616244] c1x01: ***************************************************************************
[   32.616246] c1x01: 0 DIO cards found
[   32.616246] c1x01: ***************************************************************************
[   32.616248] c1x01: 0 IIRO-8 Isolated DIO cards found
[   32.616248] c1x01: ***************************************************************************
[   32.616250] c1x01: 0 IIRO-16 Isolated DIO cards found
[   32.616250] c1x01: ***************************************************************************
[   32.616252] c1x01: 1 Contec 32ch PCIe DO cards found
[   32.616252] c1x01: 1 Contec PCIe DIO1616 cards found
[   32.616253] c1x01: 0 Contec PCIe DIO6464 cards found
[   32.616254] c1x01: 2 DO cards found
[   32.616255] c1x01: TDS controller 0 is at 0
[   32.616256] c1x01: Total of 4 I/O modules found and mapped
[   32.616257] c1x01: ***************************************************************************
[   32.616259] c1x01: 1 RFM cards found
[   32.616260] c1x01:     RFM 0 is a VMIC_5565 module with Node ID 41
[   32.616261] c1x01: address is 0x18d80000
[   32.616261] c1x01: ***************************************************************************
[   32.616262] c1x01: Initializing space for daqLib buffers
[   32.616263] c1x01: Initializing Network
[   32.616264] c1x01: Found 1 frameBuilders on network
[   32.616265] c1x01: Epics burt restore is 0
[   33.616012] c1x01: Epics burt restore is 0
[   34.617018] c1x01: Epics burt restore is 0
[   35.618017] c1x01: Epics burt restore is 0
[   36.619011] c1x01: Epics burt restore is 0
[   37.621007] c1x01: Epics burt restore is 0
[   38.622008] c1x01: Epics burt restore is 0
[   39.733257] c1x01: Sync source = 4
[   39.733257] c1x01: Waiting for EPICS BURT Restore = 1
[   39.793001] c1x01: Waiting for EPICS BURT 0
[   39.793001] c1x01: BURT Restore Complete
[   39.793001] c1x01: Found a BQF filter 0
[   39.793001] c1x01: Found a BQF filter 1
[   39.793001] c1x01: Initialized servo control parameters.
[   39.794002] c1x01: DAQ Ex Min/Max = 1 3
[   39.794002] c1x01: DAQ XEx Min/Max = 3 53
[   39.794002] c1x01: DAQ Tp Min/Max = 10001 10007
[   39.794002] c1x01: DAQ XTp Min/Max = 10007 10507
[   39.794002] c1x01: DIRECT MEMORY MODE of size 64
[   39.794002] c1x01: daqLib DCU_ID = 19
[   39.794002] c1x01: Calling feCode() to initialize
[   39.794002] c1x01: entering the loop
[   39.794002] c1x01: ADC setup complete
[   39.794002] c1x01: DAC setup complete
[   39.794002] c1x01: writing BIO 0
[   39.814002] c1x01: writing DAC 0
[   39.814002] c1x01: Triggered the ADC
[   40.874003] c1x01: timeout 0 1000000
[   40.874003] c1x01: exiting from fe_code()

The Phase tracker outputs (= ALS X/Y error signals) are now conveyed to the LSC model.

Their entry points at the LSC model are C1:LSC-ALSX_IN1 and C1:LSC-ALSY_IN1.
They are connected to the signal matrix (28th and 29th signals) via signal conditioning filters (C1:LSC-ALSX and C1:LSC-ALSY).

The main LSC screen has not been updated. The conventional ALS servos are still remains as they were.

This renovation required the recompilation of c1als, c1rfm, and c1lsc. Two PCIe-RFM bridge paths were added resulting in
increase of the c1rfm timing budget from 38 to 44.

It seems that the front fan unit was running at the full speed. The fan itself seems still OK.

I talked with Jamie and make a power cycling (i.e. shutdown gracefully, unplug the power supply cables (x4), plug them in again, and pushed the power button)

The warning signal went off and the fan is quiet. FOR NOW.

Now, daqd and ndsd is down.

FB cannot mount /opt/rtcds and /cvs/cds during its boot.

After mounting these manually, I tried to run /opt/rtcds/caltech/c1/target/fb/start_daqd.inittab and /opt/rtcds/caltech/c1/target/fb/start_nds.inittab
but they don't keep running.

I'll be back to this issue tomorrow with Jamie's help.

Now FB is fixed: daqd and nds are running

When I rebooted FB, I noticed that any of the nfs file systems were not mounted.
I started tracking down the issues from here.

I googled the common issues of the nfs mounting during the boot sequence.
- It is good to give "_netdev" option to fstab to mount the system after the network connection is established.

- "auto" option specifies that the file system is mounted when mount -a is run

Resulting /etc/fstab is this:

But this didn't help mounting the nfs file systems at boot yet. I dug into google again and found a command "/sbin/rc-update".
"/sbin/rc-update show" shows what services are activated at boot. It did not include "nfsmount". So the following command
was executed

> sudo /sbin/rc-update add nfsmount boot

> /sbin/rc-update show

After rebooting, I confirmed that the nfs file systems are correctly mounted
and daqd and nds are automatically started.

This means that FB had never been configured to run correctly at boot. Shame on you!

c1scy had frequent time-over. This caused the glitches of the OSEM damping servos.

Today Eric Q was annoyed by the glitches while he worked on the green PDH inspection at the Y-end.

In order to mitigate this issue, low priority RFM channels are moved from c1scy to c1tst.
The moved channels (see Attachment 1) are supposed to be less susceptible to the additional delay.

This modification required the following models to be modified, recompiled, reinstalled, and restarted
in the listed order: c1als, c1sus, c1rfn, c1tst, c1scy

Now the models are are running. CDS status is all green.
The time consumption of c1scy is now ~30us (porevious ~60us) (see Attachment 2)

I am looking at the cavity lock of TEM00 and I have witnessed no glitch any more.
In fact, the OSEM signals have no glitch. (see Attachment 3)

We still have c1mcs having regularly time-over. Can I remove the WFS->OAF connections temporarily?

I worked on the CDS related stuffs for LSC yesterday and today.

1. Slow machines:

I checked the database files for c1iscaux and c1iscaux2 (slow machines). They are mainly
used for the control of LSC whitening filters. The channel names were totally random as we
reconfigured the RF PDs while the channel names had been unchanged.

- Now the database was modified so that the PD name and the channels are related.
- saverestore.req and autoBurt.req were also changed accordingly.

- PD interface channels are completely random. Don't use them.
- I found the whitening of DCPDs are not effective.

- We need to clean up /cvs/cds/caltech/target directory. The autoBurt requests in the old targets
are making unnecessary burt files.

2. LSC screens

- The channel names on the LSC OVERVIEW screen was modified. (Attachment 1)
- A new LSC Whitening screen was made. (Attachment 2)

3. LSC screen generator

To touch the main LSC screen is very tough. The screen was split in to several sub screens
and combined with a command.

/opt/rtcds/caltech/c1/medm/c1lsc/master/generateLSCscreen/generateLSCscreen.py

This command combines the multiple adl files into a single file with x&y offsets.
This way, you can work with the each section of the screen.
Also, moving the blocks are just easy.

4. LSC Code Bug?

During the screen making, I found that a couple of the whitening switches are not
working properly. e.g. When AS165 (either I or Q) FM1 is activated throught the whitening trigger,
the MSB bit (bit15) of the binary I/O (C1:LSC-BIO_0_0) does not .

SImilarly ASDC FM1 does not toggle bit15 of C1:LSC-BIO_0_1.

The other channels seems OK.

At first, I thought this is a bug of "Bit2Word" block. But an individual test of the block showed that
the block is not guilty. So why is only Bit15 malfunctioning???

In order to unify the theme for MEDM screens, I'll have to make many combinations of rectangles, polygons, and invisible related-screen buttons.
Everytime I change the size of the block, I have to modify the size of this combination. It is impossile for me.

So, I made a script to replace a certain type of rectangles with a combination of the objects with the same (or related) size.

The script is here (so far)

/opt/rtcds/caltech/c1/medm/c1lsc/master/generateLSCscreen/rect_replace.py

Usage:

cat C1LSC_OVERVIEW_ADC.adl | ./rect_replace.py > tmp.adl

Description:

The script takes stdin and spits the result to stdout. It parses a given ADL file. When a "rectangle" object
with Channel A with a string "REPLACE_XXXX", it replaces it with the objects predefined as "XXXX".

So far, there is only "TYPE1" for the predefinition. It actually takes another argument to specify the
path of the related screen to open when the block is clicked. The path should be filled in "Channel B"
slot of the original rectangle. (See Attachment 1)

The "TYPE1" style has the function calls as indicated below. place_rect is to place a rectangle object. You can
specify the filling method and color. place_rel_disp is to place an invisible button with the link to the specified
path by strOpt. place_polygon places a polygon. The cordinate array for the polygon is described with
the relative positions from the specified position.

        place_rect(rect_x-4,         rect_y-4,        rect_w+7, rect_h+7, "outline",  0) # outline white box
        place_rel_disp(rect_x, rect_y, rect_w, rect_h, strOpt, 0, 14)                    # invisible button
        place_rect(rect_x,           rect_y,          rect_w,   rect_h,   "fill",     3) # main gray box
        place_rect(rect_x+3,         rect_y,          rect_w-6, 3,        "fill",     0) # top white rim
        place_rect(rect_x,           rect_y,          3,        rect_h-3, "fill",     0) # left white rim
        place_rect(rect_x+rect_w-3 , rect_y,          3,        rect_h,   "fill",    10) # right gray rim
        place_rect(rect_x,           rect_y+rect_h-3, rect_w-3, 3,        "fill",    10) # bottom gray rim
        place_polygon(rect_x+rect_w-3,rect_y,3,3, "fill",  0, [[0,0],[2,0],[0,2],[0,0]]) # top-right white triangle
        place_polygon(rect_x,rect_y+rect_h-3,3,3, "fill",  0, [[0,0],[2,0],[0,2],[0,0]]) # bottom-left white triangle


Yesterday evening I inspected at IMC servo as a preparation of the CM servo recommissioning.

More details to come.

What a such long pain we suffered.

After more than three years from Kiwamu's discovery, the PDH box 50kHz oscillation issue was finally solved.

This "weird peak at 50kHz" was caused by the oscillation of the voltage regulator (ON's MC7912).
As it imposed common noise almost everywhere, it screwed up transfer function measurements
like EricQ saw recently.

The negative voltage regulator (79XX) tends to get unstable than the positive counter parts (78XX).

The oscillation was removed by adding 22uF electrolytic capacitor between the output pin (pin3) and the ground pin (pin1) of MC7912.
This is indeed more than 20 times of the specification you can find in the data sheet.

In order to accomplish CARM control with the PSL laser frequency, we use two actuators.

One is the longitudinal direction of one of the MC mirrors. The londitudinal motion of the MC induces
the laser frequency control via the MC servo. As we move the mirror, the range is sort of big,
but the BW is limited by the mechanical response.

The other is the additive offset path. We inject a signal to the additional input port of the MC.
The MC servo supresses this injection by giving the same amount but oppsite sign offset to
the error signal (before the addtion of the inputs). The bandwidth of this AO path is limited
by the bandwidth of the MC servo. Basically the BW of the AO path is about 1/10 of that of the MC servo.

In order to confirm the capability of the AO path as a frequency actuator, 1) OLTF of the MC servo
2) TF of the AO input to the servo error was measured.

Attachment 1 shows the openloop TF of the MC servo. The UGF seems just little bit higher than
100kHz. The OLTF is empirically modelled by LISO as seen in the figure.

Attachment 2 shows the TF from the additive input (In2) to the error monitor (MC Servo module Q error mon).
The gain setting of the MC servo box was: In1 +18dB, In2 0dB. The measured TF has arbitorary gain 
due to the gain setting, the measuemrent data was multiplied by 4 to mach the DC value to the unity.
This is to compare the measurement with the prediction from the OLTF.

The AO path TF is expected to show the character of -G/(1+G) where G is the OLTF. In my case,
G = 0.75*OLTF showed the best maching. There might have been some misalignment of the MC
upon the AO path measurement as I found after the measurement.

From the plot , we can see that the response is flat up to 20kHz. Above that it rapidly raises.
This should be dealt with the CM servo filter as the bump may hit the unity gain. Since we have to use
strong roll off to avoid the bump, this will eat the phase margin at low frequency.

In the case that we don't like this bump:
This bump is caused by low phase mergin of the OLTF at 30~40kHz. If the total gain
is increased, the bump is reduced. Or, we can decrease the PZT loop gain in order to
reduce the dip at the crossover ferquency between the PZT and PC loops. In both cases,
the PC path suffers more actuation. We may need to think about the HV actuation option
for the PC (Apex PA85).

Well, let's see how the CM servo can handle this.
The key point here is that we have enough data to start the design of the CM servo.

OK, the next question will be "why the loop bandwidth is so miserable?"
In other words, what is preventing us to have the bandwidth of 20~30kHz?
Your breaking down will give us the answer.

According to the measurement by Eric, the X-arm green PDH UGF is too low. We still have some room to increase the gain.

The out of loop stability of the ALS for each arm should be measured everyday.
Otherwise we can't tell whether the arm is prepared for advanced locking activities or not.

We expect to see the arm stablity of ~50pm_rms for the Y arm and ~150pm_rms for the X arm.

rect_replace.py script was updated.
This sounds crazy but it was actually quite easy as I could use a free font data.

/opt/rtcds/caltech/c1/medm/c1lsc/master/generateLSCscreen/rect_replace.py

Usage:

cat C1LSC_OVERVIEW_ADC.adl | ./rect_replace.py > tmp.adl

Description:

The script takes stdin and spits the result to stdout. It parses a given ADL file. When a "rectangle" object
with Channel A with a string "REPLACE_XXXX", it replaces it with the objects predefined as "XXXX".

Now new type "CHAR" (i.e. REPLACE_CHAR) was added. This replaces the string in Channel B slot
into 5x7 dot matrix representation of the string with the specified color. The dot size is derived from the
height of the original rectangular object.

Looks good.

Once the control cable (bakplane cable) is identified, we can install the module to the LSC analog rack.

We should be able to test the CM servo with either POX or POY and only one correspoding arm without modifying the servo TF.
Just for this test, we don't need to use MCL.

Now the module is inserted at the 2nd crate from the top of 1Y2 (LSC analog rack). It is next to the DCPD whitening module.

I found the backplane cable for the Common Mode servo module.
I traced a cable form XY220 at the right most module on the crate where iscaux2 is living.
This cable was connected to the upper backplane connector.

Switching of the module is tested. All the switches and gain control are doing their job.

It was found that the offset and slow readback are not responding.
I checked the schematic of the CM servo module (D040180).
It seems that there is another cable for the offset and read back voltages.

I found another backplane cable for the CM servo module. It is plugged to the module now.

I can see that C1:LSC-CM_SLOW_MON is responding to C1:LSC-CM_REFL_OFFSET.
But C1:LSC-CM_SUM_MON and C1:LSC-CM_FAST_MON are not replated to the given offset.
I probably need to check the cross connects.

I worked on the mitigation of c1mcs time-over issue this afternoon.

The timing for the c1mcs is successfully reduced from >60us to 45us.

The previous models are svned in redoubt as follows:

MCS rev. 6696
RFM rev. 6697
IOO rev. 6698

What I changed was:

- Remove connection from ALS (on c1ioo) to MCS (on c1sus). This should be all done in LSC. (# of RFM IPC in MCS -1)

- MC2 trans QPD filters are moved from IOO to MCS to reduce the RFM channels in MCS.
  Previously the signals for the 4 segments are sent. Now the processed siganls (pit/yaw/sum) are sent. (# of RFM IPC in IOO -1, MCS -1)

- WFS MC3 feedback channels are moved from MCS to RFM to distribute the RFM channels (# of RFM IPC in MCS -2, in RFM +2)

model    prev. timing[us] current timing[us]  diff in time[us]  diff in ch#
c1mcs         >60                45                -15              -4
c1rfm         47                 53                + 6              +2       
c1ioo         47                 36                -11              -1

Revisions of the new models:
MCS rev. 6702
RFM rev. 6701
IOO rev. 6700

Now netgpibdata is working again.

Usage:

cd /cvs/cds/rtcds/caltech/c1/scripts/general/netgpibdata   
./netgpibdata -i 192.168.113.108 -d AG4395A -a 10 -f meas01
./netgpibdata -i 192.168.113.105 -d SR785 -a 6 -f meas01   

Jenne witnessed and certified that they were working fine.
Now no one can say "it does not work"!

These weeks I was annoyed by the fact that netgpibdata was messed up by dummies.
A terrible situation was found:

1. Someone pushed the factory reset of one of the wifi bridges (LINKSYS WET54G).
2. Someone gave wrong IPs to the bridges (192.168.1.* instead of 192.168.113.*)
3. Someone left a default IP to the bridges. This means the routers had the same IPs.

-------

I gave the IPs to the bridges. According lines of /etc/hosts in linux1 were updated.

192.168.113.230 WET54G1
192.168.113.231 WET54G2

All of the network settings are taped on the bridge now.
The reset switch of each bridge was covered by a tape so that dummies can't randomly push the button.

The command was tested with each device.

A while ago, I noticed that the most significant bits of the LSC whitening DOs are not toggling.
I track this issue down and found what is happening. I need experts' help.

To illuminate the issue, terminators are connected to Bit15 of the Bit2Word blocks in the LSC model (attached screen shots).

The corresponding source file is found in c1lsc.c at the following location.
The last channels of the Bit2Word are connected to lsc_cm_slow (the filter module).
This is the source of the issue. This wrong assignment of the connections
can't be changed by connecting Go-From tags to the chennels.

/opt/rtcds/caltech/c1/rtbuild/src/fe/c1lsc/c1lsc.c

3881// Bit2Word:  LSC_cdsBit2Word1                                                                                                              
3882{
3883double ins[16] = {
3884        lsc_as110_logicaloperator4,
3885        lsc_as110_logicaloperator1,
3886        lsc_refl11_logicaloperator4,
3887        lsc_refl11_logicaloperator1,
3888        lsc_pox11_logicaloperator4,
3889        lsc_pox11_logicaloperator1,
3890        lsc_poy11_logicaloperator4,
3891        lsc_poy11_logicaloperator1,
3892        lsc_refl33_logicaloperator4,
3893        lsc_refl33_logicaloperator1,
3894        lsc_pop22_logicaloperator4,
3895        lsc_pop22_logicaloperator1,
3896        lsc_pop110_logicaloperator4,
3897        lsc_pop110_logicaloperator1,
3898        lsc_as165_logicaloperator4,
3899        lsc_cm_slow
3900};
3901lsc_cdsbit2word1 = 0;
3902for (ii = 0; ii < 16; ii++)
3903{
3904if (ins[ii]) {
3905lsc_cdsbit2word1 += powers_of_2[ii];
3906}
3907}
3908}

3946// Bit2Word:  LSC_cdsBit2Word2                                                                                                              
3947{                                                                                                                                           
3948double ins[16] = {                                                                                                                          
3949        lsc_as55_logicaloperator4,                                                                                                          
3950        lsc_as55_logicaloperator1,                                                                                                          
3951        lsc_refl55_logicaloperator4,                                                                                                        
3952        lsc_refl55_logicaloperator1,                                                                                                        
3953        lsc_pop55_logicaloperator4,                                                                                                         
3954        lsc_pop55_logicaloperator1,                                                                                                         
3955        lsc_refl165_logicaloperator4,                                                                                                       
3956        lsc_refl165_logicaloperator1,                                                                                                       
3957        lsc_logicaloperator_cm_ctrl,                                                                                                        
3958        ground,                                                                                                                             
3959        ground,                                                                                                                             
3960        lsc_logicaloperator_popdc,                                                                                                          
3961        lsc_logicaloperator_poydc,                                                                                                          
3962        lsc_logicaloperator_poxdc,                                                                                                          
3963        lsc_logicaloperator_refldc,                                                                                                         
3964        lsc_cm_slow                                                                                                                         
3965};                                                                                                                                          
3966lsc_cdsbit2word2 = 0;                                                                                                                       
3967for (ii = 0; ii < 16; ii++)                                                                                                                 
3968{                                                                                                                                           
3969if (ins[ii]) {                                                                                                                              
3970lsc_cdsbit2word2 += powers_of_2[ii];
3971}
3972}
3973}

up/down scripts are to be made

(Offset Edit on Dec 20 10:38PM)

Configuration:
POY11QMon -> CM Servo In1 -> CM Servo -->Out1 -> ADC -> CM Slow FM -> LSC MC Servo FM -> ETMY(x1.0) -> DAC -> ETMY
                                       |
                                       -->Servo Out -> SR560 (DC, 1st order 30kHz LPF) -> MC In2

Lock acquisition path 1


Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +8dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC MC servo gain +0.1, FM7/FM10 (Trigger FM2 with 3sec delay)
• Turn on MC

Transition:

• Enable AO path (CM servo In1 SW:ON, MC servo In2 SW:ON)
• LSC MC gain +0.1 -> +0.2
• AO path gain 8dB->14dB
• LSC MC gain +0.2 -> +0.35
• AO path gain 14dB->18dB
• CM servo offset -1.88 -2.7 -> 0.8 0.0 (gradually)
• Enable CM servo Boost

Lock acquisition path 2

Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +20dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC Yarm G=+0.25 FM4/5 (Trigger FM3/6/7/8)

Transition:

• Enable MC servo In2 (SW:ON)
• Set LSC MC gain +0.2 FM7/10
• Enable LSC MC (On)
• Enable CM servo In1 (SW:ON)
• Disable LSC Yarm (OFF)
• Change CM servo offset -1.88 -> +0.700 -2.70 -> 0.0
• Enable CM servo Boost
• Turn on LSC CM FM2 (optional)

Transition to ETMY LSC to MCL

• After all of the transition: LSC output matrix ETMY (+1.00)
• LSC output matrix MC2 (-1.00)
• LSC output matrix ETMY (0.00)

This too huge offset difference with/without "BOOST" switch should be checked.

The bug is still there but the incorrect bits are now overridden.

I checked the offset situation in the CM servo boost circuit. 

- The offset voltage on the CM servo screen is a raw DAC output. This number is diluted by the voltage divider before the amplifier.
  So, the actual offset of the boost circuit was much smaller. (~20mV)

- There is a offset trimmer on the board. This was adjusted so that the boost does not generate an output offset.

- So the default offset is 0V.

- When the arm was locked with (digital) POY11, the CM servo offset is necessary to be -2.7 (now).
  This means that analog POY11Q and digital POY11 has different offset for the best resonance transmission.
  That is believable if POY11I is contributing to the digital POY11 signal.

I'm leaving the 40m now. IFO is aligned. Everything look good.

- The main volume P1=5e-4, CC1=1.4e-5 is still pumped by TP1 and TP2

- RGA P4<0e-4, CC4 2.1e-7, is pumped by TP3

- The annuluses are isolated.

- RP1/2/3 are off.

The previous LSC whitening filters for the DCPDs were in an unknown state (although the transfer functions were actually measured and fit a while ago)
They had no DC gain control and some of the channels had modifications.

To make the setup clear, the filter module was replaced with the spare module without any modification.

The channels are now respoding to the whitening gain switches. So far there is no screen for the new  whitening gains yet.

Also I found that POX11 DC cable has not been connected. Now it is connected.

Dec 22 between 6AM and 7AM, physical or logical failure has occure on the 4th disk in the RAID array on linux1.
This caused the RAID disk fell into the readonly mode. All of the hosts dependent on linux1 via NFS were affected by the incident.

Today the system has been recovered. The failed filesystem was restored by copying all of the files (1.3TB total) on the RAID to a 2TB SATA disk.
The depending hosts were restarted and we recovered elog/wiki access as well as the interferometer control system.

Recovery process

o Recover the access to linux1

- Connect an LCD display on the host. The keyboard is already connected and on the machine.
- One can login to linux1 from one of the virtual consoles, which can be switched by Alt+1/2/3 ...etc
- The device file of the RAID is /dev/sda1
- The boot didn't go straightforward as mounting of the disks accoding to /dev/fstab doesn't go well.
- The 40m root password was used to login with the filesystem recovery mode.
- Use the following command to make the editing of /etc/fstab available

# mount -o rw, remount /

- In order to make the normal reboot successfull, the line for the RAID in /etc/fstab needed to be commented out.

o Connect the external disk on linux1

- Brought a spare 2TB SATA disk from rossa.
- Connect the disk via an USB-SATA enclosure (dev/sdd1)
- Mount the 2TB disk on /tmpdisk
- Run the following command for the duplication

# rsync -aHuv --progress /home/ /tmpdisk/ >/rsync_KA_20131229_0230.log

- Because of the slow SCSI I/F, the copy rate was limited to ~6MB/s. The copy started on 27th and finished 31st.

o Restart linux1

- It was found that linux1 couldn't boot if the USB drive is connected.
- The machine has two SATA ports. These two are used for another RAID array that is not actually used. (/oldhome)
- linux1 was pulled out from the shelf in order to remove the two SATA disks.
- The 2TB disk was installed on the SATA port0.
- Restart linux1 but didn't start as the new disk is recognized as the boot disk.
- The BIOS setting was changed so that the 80GB PATA disk is recognized as the boot disk.
- The boot process fell into the filesystem recovery mode again. /etc/fstab was modified as follows.

- Another reboot make the operating system launched as usual.

o What's happen to the RAID?

- Hot removal of the disk #4.
- Hot plug of the disk #4.
- Disk #4 started to get rebuilt -> ~3hours rebuilding done
- This made the system marked as "clean". Now the raid (/dev/sdb1) can be mounted as usual.

o Nodus

- Root password of nodus is not known.
- Connect an LCD monitor and a Sun keyboard on nodus.
- Type Stop-A. This leads the nodus transition to the monitor mode.
- Type sync.
- This leads the system rebooted.

IFO restart after the recovery of linux1

Machine recovery in the following order
- Start fb
- Start the following machines: mafalda, megatron, op340m
- Start c1ioo, c1lsc, c1sus, c1iscex, c1iscey

CDS recovery / burtrestore

- Confirm all of the RT systems are running in "green". If not, restart corresponding model.
- c1iscaux, ciscaux2 didn't have response (white boxes). Went to the LCS digital rack and power cycled these targets
- burtrestore: The snapshots at Dec 19 05:07 were used. For c1iscaux and c1iscaux2 the snapshots at Dec 22 05:07 were used.

fast machines
c1alsepics.snap
c1assepics.snap
c1asxepics.snap
c1calepics.snap
c1iooepics.snap
c1lscepics.snap
c1lspepics.snap
c1mcsepics.snap
c1oafepics.snap
c1pemepics.snap
c1rfmepics.snap
c1scxepics.snap
c1scyepics.snap
c1spxepics.snap
c1supepics.snap
c1susepics.snap
c1tstepics.snap

slow machines
c1auxex.snap
c1auxey.snap
c1aux.snap
c1iool0.snap
c1iscaux2.snap
c1iscaux.snap
c1psl.snap
c1susaux.snap

IFO recovery

- Reload watchdogs => restore sus damping
- MC misaligned but TEM00 was locked
- Gave a small touch on MC2 yaw => IMC almost aligned
- Autolocker wasn't running => Manually launched rather than wait for an hour for cron to launch it
- PMC was largely misaligned. => Aligned on the PSL table (PSLTRANS 0.640->0.753)
- MC WFS ON
- IFO X/Y arm locked and aligned with ASS.
- PRMI mode: manually aligned PRM. The PRMIsb momentally locked.