I guess we forgot to close V5, so we were indeed pumping on the ITMY and ETMY annuli, but the other three were isolated suggest a leak rate of ~200-300 mtorr/day, see Attachment #1 (consistent with my earlier post).

As for the main volume - according to CC1, the pressure saturates at ~250 uTorr and is stable, while the Pirani P1a reports ~100x that pressure. I guess the cold-cathode gauge is supposed to be more accurate at low pressures, but how well do we believe the calibration on either gauge? Either ways, based on last night's test (see Attachment #2), we can set an upper limit of 12 mtorr/day. This is 2-3x the number Steve said is normal, but perhaps this is down to the fact that the outgassing from the main volume is higher immediately after a vent and in-chamber work. It is also 5x lower rate of pressure increase than what was observed on Feb 2.

I am resuming the pumping down with the turbo-pumps, let's see how long we take to get down to the nominal operating pressure of 8e-6 torr, it ususally takes ~ 1 week. V1, VASV, VASE and VABS were opened at 1030am PST. Per Chub's request (see #14435), I ran RP1 and RP3 for ~30 seconds, he will check if the oil level has changed.

I was lucky to notice that the nitrogen supply line to the vacuum valves was leaking. Closed ALL valves. Open supply line to atm. Fixed leak. 

This was done fast so the pumps did not have to be shut down. Pressurized supply line and open valves to

"Vac Normal" condition in the right sequence.

The first real rain of this year finds only one leak at the 40m

Johannes found dripping water at the vac rack. It is safe. It is not catching anything. Actual precipitation was only 0.62"

Dan sealed the leak today.

We are leaving the PLL as it is locked in order to see the long term stability. And we will check the results in early morning of tomorrow.

DO NOT disturb our PLL !!

(what we did)

After Mott left, Matt and I started to put feedback signals to the temperature control of NPRO.

During doing some trials Matt found that NPRO temperature control input has an input resistance of 10kOhm.

Then we put a flat filter ( just a voltage divider made by a resistor of ~300kOhm and the input impedance ) with a gain of 0.03 for the temperature control to inject a relatively small signal, and we could get the lock with the pzt feedback and it.

In addition, to obtain more stable lock we then also tried to put an integration filter which can have more gain below 0.5Hz.

After some iterations we finally made a right filter which is shown in the attached picture and succeeded in obtaining stable lock.

Matt checked it in this morning and he found it's been locked during the night.

The MICH and PRCL motions have been measured in some different configurations.

According to the measurements :

      + PRCL is always noisier than MICH.

      + MICH motion becomes noisier when the configuration is Power-Recycled Michelson (PRMI).

The next actions are :

      + check the ASPD

      + check the demodulation phases

      + try different RFPDs to lock MICH

Tip-Tilts has almost no isolation up to 3Hz, and isolation of about 0.5 up to 10Hz.
They have vertical resonances at around 20Hz.

See Nicole's entry

For a comparison, the length fluctuation of Signal-Recycled ITMX (SRX) and ITMY (SRY) have been measured.

Roughly speaking the length motion of SRX and SRY are as loud as that of PRCL.

Some details about the measurement and data analysis can be found in the past elog entry (#5582).

In the process of converting the raw spectra to the calibrated displacements the SRM actuator was assumed to have a resonance at 1Hz with Q = 5.

(Notes on SRX/Y locking)

 How does it make sense that the motion at 0.1 Hz of PRC is 10x larger than MICH?

EDIT by KI:

After last night's challenge (or inspiration), we levitated our magnet this morning.  Since the nice Olympus camera is not currently in the 40m, we had to use my less stellar camera, but despite the poor video quality you can still see the magnet returning to its stable equilibrium position.  Once we recover the better camera, we will post new videos.  Also, we haven't yet figured out how to put videos in line in the elog entry, so here are the youtube links:

levitation 1

levitation 2

We adjusted the gain on coil 1 so that the resistance from the pots was 57.1k (maximum gain of 101.2,).

currents from power supply, pre-levitation: 0.08 A and 0.34 A

post levitation: 0.08 A and 0.11 A

note: we're not sure why changing the gain on coil 3 changes the current through the power supply, so we'd like to investigate that next.

Suresh, Kiwamu and Steve

Heavy chamber doors replaced by light ones at  ITMX-west and ITMY-north locations.

I made a test installation of ligo_viewer in /users/volodya/ligo_viewer-0.5.0c . It runs on pianosa (the Ubuntu machine) and needs Tcl/Tk 8.5.

To try it out run the following command on pianosa:

cd /users/volodya/ligo_viewer-0.5.0c/

./ligo_viewer.no_install

Press "CONNECT" to connect to the NDS server and explore. There are slides describing ligo_viewer at http://volodya-project.sourceforge.net/Ligo_viewer.pdf

Installation notes:

Use /users/volodya/ligo_viewer-0.5.0c.tgz or later version - it has been updated to work with 64 bit machines.

Make sure Tcl and Tk development packages are installed. You can find required packages by running

apt-file search tclConfig.sh

apt-file search tkConfig.sh

If apt-file returns empty output run apt-file update

Unpack ligo_viewer-0.5.0c.tgz, change into the created directory.

Run the following command to configure:

export CFLAGS=-I/usr/include/tcl8.5
./configure --with-tcl=/usr/lib/tcl8.5/ --with-tk=/usr/lib/tk8.5/

This works on Ubuntu machines. --with-tcl and --with-tk should point to the directories containing tclConfig.sh and tkConfig.sh correspondingly.

Run "make".

You can test the compilation with ./ligo_viewer.no_install

If everything works install with make install

If Tcl/Tk 8.5 is unavailable it should work with Tcl/Tk 8.3 or 8.4

(Suresh, Yuta)

Summary:
  We need MC to be locked and aligned well to align other in-vac optics.
  We continued to align the incident beam so that the beam passes the actuation nodes of MC1 and MC3.
  From the previous measurement, we found that beam height at IM1 has to be increased by ~3cm.
  Today, we increased it by ~1cm and achieved about 1/3 of the required correction.
  But we cannot proceed doing this because the beam is hitting IM1 at the edge already.

What is the goal of this alignment?:
  If the beam doesn't hit MC optics in the center, we see angle to length coupling, which is not good for the whole interferometer.
 
  Also, if the beam is tilted so much, transmitted beam though MC3 cannot go into FI at right after MC3.
  Say, FI has an aparture of 3mm and MC3-FT distance is 300mm. The beam tilt should be smaller than 3/300 rad. MC1-MC3 distance is 200mm, so the displacement at each mirror should be smaller than ~1mm.
  1mm is about 7% (see Koji's elog #2863) TO_COIL gain imbalance in A2L measurement.
 
  We are currently assuming that each coils are identical. If they have 5% variance, it is meaningless to try to reduce the beam displacement less than ~5%.

  So, we set the goal to 7%.

What we did:
  1. Leveled the MC table.

  2. Measured the table height using DISTO D3 laser gauge.
    PSL table 0.83m (+-0.01m)
    OMC table 0.82m
    MC table  0.81m

  3. Using the last steering mirror(SM@PSL) and IM1, tilted the beam vertically

Result:


  At t=0 (this morning), the beam tilt was ~40%/(MC1-MC3 distance). Now, it is ~30%/(MC1-MC3 distance).
  30%/(MC1-MC3 distance) is ~5/200 rad.

Plan:
 We have to somehow come up with the next story. Too much vertical tilt. What is wrong? Table leveling seems OK.
 - measure in-vac beam height
 - maybe OSEMs are badly aligned. we have to check that.

It didn't make sense in several points.

1. Is the Faraday aperture really 3mm? The beam has the gaussian radius of ~1.5mm. How can it be possible to go through the 3mm aperture?

2. Why the MC3-FT distance is the matter? We have the steering mirror after MC3. So we can hit the center of the Faraday.
But if we have VERTICAL TILT of the beam, we can not hit the center of the Faraday entrance and exit at the same time.
That would yield the requirement.

3. If each coil has 5% variance in the response, variance of the nodal point (measured in % of the coil imbalance) by those four coils will be somewhat better than 5%, isn't it?

1. We didn't measure the aperture size last night. We have to check that.

2. We have to measure the length of FI. Or find a document on this FI.

3. Yes, 5%/sqrt(4). But I didn't think the factor of 2 is important for this kind of estimation.

1. Look at the Faraday.

2. Look at the wiki. There is the optical layout in PNG and PDF.

3. 5% (0.8mm) and 2.5%(0.4mm) sounds a big difference for the difficulty, but if you say so, it is not so different.

Actualy, if you can get to the 5% level, it is easy to get to the 1-2% level as I did last time.
The problem is we are at the 15-20% level and can not improve it.

Connecting to NDS Server fb (TCP port 8088)
Connecting.... done
No data found

read(); errno=9
read(); errno=9
T0=12-04-04-02-14-29; Length=3600 (s)
No data output.

It looks like this is actually just a limit of how long we're saving the second trends, which is just not that long.  I'll look into extending the second trend look-back.

Dec 22 between 6AM and 7AM, physical or logical failure has occure on the 4th disk in the RAID array on linux1.
This caused the RAID disk fell into the readonly mode. All of the hosts dependent on linux1 via NFS were affected by the incident.

Today the system has been recovered. The failed filesystem was restored by copying all of the files (1.3TB total) on the RAID to a 2TB SATA disk.
The depending hosts were restarted and we recovered elog/wiki access as well as the interferometer control system.

Recovery process

o Recover the access to linux1

- Connect an LCD display on the host. The keyboard is already connected and on the machine.
- One can login to linux1 from one of the virtual consoles, which can be switched by Alt+1/2/3 ...etc
- The device file of the RAID is /dev/sda1
- The boot didn't go straightforward as mounting of the disks accoding to /dev/fstab doesn't go well.
- The 40m root password was used to login with the filesystem recovery mode.
- Use the following command to make the editing of /etc/fstab available

# mount -o rw, remount /

- In order to make the normal reboot successfull, the line for the RAID in /etc/fstab needed to be commented out.

o Connect the external disk on linux1

- Brought a spare 2TB SATA disk from rossa.
- Connect the disk via an USB-SATA enclosure (dev/sdd1)
- Mount the 2TB disk on /tmpdisk
- Run the following command for the duplication

# rsync -aHuv --progress /home/ /tmpdisk/ >/rsync_KA_20131229_0230.log

- Because of the slow SCSI I/F, the copy rate was limited to ~6MB/s. The copy started on 27th and finished 31st.

o Restart linux1

- It was found that linux1 couldn't boot if the USB drive is connected.
- The machine has two SATA ports. These two are used for another RAID array that is not actually used. (/oldhome)
- linux1 was pulled out from the shelf in order to remove the two SATA disks.
- The 2TB disk was installed on the SATA port0.
- Restart linux1 but didn't start as the new disk is recognized as the boot disk.
- The BIOS setting was changed so that the 80GB PATA disk is recognized as the boot disk.
- The boot process fell into the filesystem recovery mode again. /etc/fstab was modified as follows.

- Another reboot make the operating system launched as usual.

o What's happen to the RAID?

- Hot removal of the disk #4.
- Hot plug of the disk #4.
- Disk #4 started to get rebuilt -> ~3hours rebuilding done
- This made the system marked as "clean". Now the raid (/dev/sdb1) can be mounted as usual.

o Nodus

- Root password of nodus is not known.
- Connect an LCD monitor and a Sun keyboard on nodus.
- Type Stop-A. This leads the nodus transition to the monitor mode.
- Type sync.
- This leads the system rebooted.

Well done Koji!  I'm very impressed with the sysadmin skillz.

Since this configuration change, the daily backup was speeded up by factor of more than two.
It was really limited by the bandwidth of the RAID array.

/cvs/cds/rtcds/caltech/c1/scripts/backup/rsync.backup.cumlog:

...
rsync.backup start: 2013-12-20-05:00:00, end: 2013-12-20-07:04:28, errcode 0
...
rsync.backup start: 2014-01-05-05:00:00, end: 2014-01-05-05:55:04, errcode 0

(The daily backup starts from 5:00)

At around 2:30pm today something brought down most of the martian network.  All control room workstations, nodus, etc. were unresponsive.  After poking around for a bit I finally figured it had to be linux1, which serves the NFS filesystem for all the important CDS stuff.  linux1 was indeed completely unresponsive.

Looking closer I noticed that the Fibrenetix FX-606-U4 SCSI hardware RAID device connected to linux1 (see #1901), which holds cds network filesystem, was showing "IDE Channel #4 Error Reading" on it's little LCD display.  I assumed this was the cause of the linux1 crash.

I hard shutdown linux1, and powered off the Fibrenetix device.  I pulled the disk from slot 4 and replaced it with one of the spares we had in the control room cabinets.  I powered the device back up and it beeped for a while.  Unfortunately the device was requiring a password to access it from the front panel, and I could find no manual for the device in the lab, nor does the manufacturer offer the manual on it's web site.

Eventually I was able to get linux1 fully rebooted (after some fscks) and it seemed to mount the hardware RAID (as /dev/sdc1) fine.  The brought the NFS back.  I had to reboot nodus to get it recovered, but all the control room and front-end linux machines seemed to recover on their own (although the front-ends did need an mxstream restart).

The remaining problem is that the linux1 hardware RAID device is still currently unaccessible, and it's not clear to me that it's actually synced the new disk that I put in it.  In other words I have very little confidence that we actually have an operational RAID for /opt/rtcds.  I've contacted the LDAS guys (ie. Dan Kozak) who are managing the 40m backup to confirm that the backup is legit.  In the mean time I'm going to spec out some replacement disks onto which to copy /opt/rtcds, and also so that we can get rid of this old SCSI RAID thing.

The liquid nitrogen container has a pressure releif valve set to 35 PSI  This valve will open periodically when contains LN2

The exiting  very cold gas can cause burning so it should not hit directly your eyes or skin.  Set the pointing of this valve into the corner.

Leave entry door open so nitrogen concentration can not build up.

We could use similar load cells   to make the actual weight measurement on the Stacis legs. This seems practical in our case.

I have had bad experience with pneumatic Barry isolators.

Our approximate max compression loads are 1500 lbs on 2 feet and 2500 lbs on the 3rd one.

1500 and 2000 lbs load cells arrived from MIT to measure the vertical loads on each leg.

 Atm2, Chilean 8.2M eq arrived to the 40m 16:57 in 10 minutes. Our OSEMs did not see them.  Hanford got it 2 minutes later.

M3.4 Colton shake did not trip sus.

All suspension tripped. Their damping restored. The MC is locked.

ITMX-UL & side magnets are stuck.

I freed ITMX and coarsely realigned the IFO using the OPLEVs. All the alignments were a bit off from overnight.

The IFO is still only able to lock in MICH mode currently, which was the situation before the earthquake. This morning I additionally tried restoring the burt state of the four machines that had been rebooted in the last week (c1iscaux, c1aux, c1psl, c1lsc) but that did not solve it.

Local EQ 3.1 mag at Jun 2, 2016 11:06:16 PM UTC, Muscoy CA........no damage

Our STS should seen this shake.

Local earth quake 3.1 magnitude in Valencia, Ca did not trip our suspensions.

Local EQ 3.5 mag  at 2:28 UTC May 24, 2016 Rancho Cucamonga, Ca.....no damage

I tried to figure out what can add noise below 0.5 Hz to the MC_F. I compared MC1, MC2, MC3 suspos, suspit, susyaw and susside positions with damping (black curves) and without (red curves). Local damping is fine.

Then I compared MC1, MC2, MC3 suspos, suspit, susyaw and susside positions with WFS on (black curve) and off (red curve). WFS add noise to MC1 and MC3 measured by osems (MC2 is fine though). WFS should change osem readings but is it a correct way to do this below 0.5 Hz (?) It looks like just a flat noise. Need to think about the conclusion.

    WFS servo is moving the MC mirror angles to minimise TEM01 and TEM10 modes within the MC cavity.    This means it will compensate not only for angular noise in the mirrors but also for the PSL beam pointing fluctuations.  So the extra "noise" we see when WFS loops are on is because they are active below the WFS UGF of about 2 Hz.  Also if the HEPA airflow is above 20% (of its max), the PSL beam jitter (caused by the airflow) will add broadband noise into the WFS servo loops and this will show up in the OSEM signals.  See elog 5943 for details.

So I tracked down where the currently-used filter function code is defined (the following is all relative to /opt/rtcds/caltech/c1/core/release):

Looking at one of the generated front-end C source codes (src/fe/c1lsc/c1lsc.c) it looks like the relevant filter function is:

filterModuleD()

which is defined in:

src/include/drv/fm10Gen.c

and an associated header file is:

src/include/fm10Gen.h

We are interested in the following question : Can the structures defined in fm10Gen.h (or some other *.c *.h files with defined as FLOAT variables) create single precision instead of double in the filter calculations?

typedef struct FM_OP_IN{

  UINT32 opSwitchE;     /* Epics Switch Control Register; 28/32 bits used*/

  UINT32 opSwitchP;     /* PIII Switch Control Register; 28/32 bits used*/

  UINT32 rset;          /* reset switches */

  float offset;         /* signal offset */

  float outgain;        /* module gain */

  float limiter;        /* used to limit the filter output to +/- limit val */

  int rmpcmp[FILTERS];  /* ramp counts: ramps on a filter for type 2 output*/

                        /* comparison limit: compare limit for type 3 output*/

                        /* not used for type 1 output filter */

  int timeout[FILTERS]; /* used to timeout wait in type 3 output filter */

  int cnt[FILTERS];     /* used to keep track of up and down cnt of rmpcmp */

                        /* should be initialized to zero */

  float gain_ramp_time; /* gain change ramping time in seconds */

} FM_OP_IN;  

up/down scripts are to be made

(Offset Edit on Dec 20 10:38PM)

Configuration:
POY11QMon -> CM Servo In1 -> CM Servo -->Out1 -> ADC -> CM Slow FM -> LSC MC Servo FM -> ETMY(x1.0) -> DAC -> ETMY
                                       |
                                       -->Servo Out -> SR560 (DC, 1st order 30kHz LPF) -> MC In2

Lock acquisition path 1


Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +8dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC MC servo gain +0.1, FM7/FM10 (Trigger FM2 with 3sec delay)
• Turn on MC

Transition:

• Enable AO path (CM servo In1 SW:ON, MC servo In2 SW:ON)
• LSC MC gain +0.1 -> +0.2
• AO path gain 8dB->14dB
• LSC MC gain +0.2 -> +0.35
• AO path gain 14dB->18dB
• CM servo offset -1.88 -2.7 -> 0.8 0.0 (gradually)
• Enable CM servo Boost

Lock acquisition path 2

Initial condition:

CM Slow FM:

• Gain 2.6

CM Servo setting:

• In1 Gain 31dB, SW:OFF, Offset -1.880, Boost Off, Super Boost Off, AO +20dB

MC Servo setting:

• In2 10dB, SW:OFF

Acquisition:

• Engage LSC
• LSC Yarm G=+0.25 FM4/5 (Trigger FM3/6/7/8)

Transition:

• Enable MC servo In2 (SW:ON)
• Set LSC MC gain +0.2 FM7/10
• Enable LSC MC (On)
• Enable CM servo In1 (SW:ON)
• Disable LSC Yarm (OFF)
• Change CM servo offset -1.88 -> +0.700 -2.70 -> 0.0
• Enable CM servo Boost
• Turn on LSC CM FM2 (optional)

Transition to ETMY LSC to MCL

• After all of the transition: LSC output matrix ETMY (+1.00)
• LSC output matrix MC2 (-1.00)
• LSC output matrix ETMY (0.00)

In an effort to familiarize myself with the analog CM servo, I've begun to replicate Koji and Den's work from the ELOG post that this is a reply to.

I hooked POX11Q into the IN1 of the CM board. (POX is rotated by ~86 degrees in the CDS, meaning analog Q is almost perfect.)

While there, I took out the too-long delay cables Jenne introduced for REFLs 11 and 55. (Also note: when we do cable-based analog phasing in the future, we should do it on the LO side, instead of the PD input side.) I also heard a dangerously crinkly sound from the short SMA cable for REFL11, so I replaced it with a beefy looking new one I found on the SP table.

I messed with the gain and offset in the CM_SLOW input filter to get it to look just like POX11_I_ERR, and was able to lock the arm on it without an issue. I then put the SR560 between the CM and MC (30k pole, but also AC coupled, because I figure the digital loop should be doing the work down there, and don't want to kick the AO with an offset), and was able to turn on the AO path with a gain of 8dB on the CM board and 10dB on the MC board, as detailed in Koji's procedures.

I wasn't able to increase the AO gain to 9dB without breaking lock, but maybe this is ok, because by judging by the LSC filter gains, POX11 might be about 3 times bigger than POY, so maybe 8dB AO gain on POX ~ =18dB AO gain on POY? I was able to put the CM servo offset at 0, but turning on boosts promptly kicked the MC out of lock.

I'm stopping for the night; but tomorrow I'll bust out a spectrum analyzer to see if I actually have won some bandwidth with the CM servo, and check out the situation with the offsets and boosts.

locks last for about an hour.  this was true last night as well (see "arm power curve" entries).   the second lock shown here evolves differently for unknown reasons.  the jumps in the arm powers of the first lock are due to turning on DC readout.  length-to-angle needs tuning.