[root@c1omc controls]# /opt/gds/awgtpman -2 &
[1] 16618
[root@c1omc controls]# mmapped address is 0x55577000
32 kHz system
Spawn testpoint manager
no test point service registered
Test point manager startup failed; -1

[1]+  Exit 1                  /opt/gds/awgtpman -2

This turned out to be fallout from the /cvs/cds transition.  Remounting and restarting fixed it.

LockAcq is back on track, with the full script working well.  Measurements in progress.

The RGA region is beeing monitored during the vent. This will tell us how clean the RGA itself is.

The RGA back ground at day 29 of this vent.

I want to try to do the measurement with the network analyzer used as local oscillator, instead of the Marconis that I'm using now. Tha could give me better noise rejection. It would also give me information about the phase.

Also I wouldn't dislike abandoning the GPIB interfaces to acquire data.

I spent this afternoon trying to debug fb1, with very little to show for it.  We're back to running from fb.

The first thing I did was to recompile EPICS from source, so that all the libraries needed by daqd were compiled for the system at hand.  I compiled epics-3.14-12-2_long from source, and installed it at /opt/rtapps/epics on local disk, not on the /opt/rtapps network mount.  I then recompiled daqd against that, and the framecpp, gds, etc from the LSCSoft packages.  So everything has been compiled for this version of the OS.  The compilation goes smoothly.

There are two things that I see while running this new daqd on fb1:

instability with mx_streams

The mx stream connection between the front ends and the daqd is flaky.  Everything will run fine for a while, the spontaneously one or all of the mx_stream processes on the front ends will die.  It appears more likely that all mx_stream processes will die at the same time.  It's unclear if this is some sort of chain reaction thing, or if something in daqd or in the network itself is causing them all to die at the same time.  It is independent of whether or not we're using multiple mx "end points" (i.e. a different one for each front end and separate receiver threads in the daqd) or just a single one (all front ends connecting to a single mx receiver thread in daqd).

Frequently daqd will recover from this.  The monit processes on the front ends restart the mx_stream processes and all will be recovered.  However occaissionally, possibly if the mx_streams do not recover fast enough (which seems to be related to how frequently the receiver threads in daqd can clear themselves), daqd will start to choke and will start spitting out the "empty blocks" messages that are harbirnger of doom:

Aborted 2 send requests due to remote peer 00:30:48:be:11:5d (c1iscex:0) disconnected
00:30:48:d6:11:17 (c1iscey:0) disconnected
mx_wait failed in rcvr eid=005, reqn=182; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 005
mx_wait failed in rcvr eid=001, reqn=24; wait did not complete; status code is Remote endpoint is closed
disconnected from the sender on endpoint 001
[Wed Dec  9 18:40:14 2015] main profiler warning: 1 empty blocks in the buffer
[Wed Dec  9 18:40:15 2015] main profiler warning: 0 empty blocks in the buffer
[Wed Dec  9 18:40:16 2015] main profiler warning: 0 empty blocks in the buffer

My suspicion is that this time of failure is tied to the mx stream failures, so we should be looking at the mx connections and network to solve this problem.

frame writing troubles

There's possibly a separate issue associated with writing the second or minute trend files to disk.  With fair regularity daqd will die soon after it starts to write out the trend frames, producing the similar "empty blocks" messages.

I made a simple script to backup crontab (/opt/rtcds/caltech/c1/scripts/crontab/backupCrontab).

#!/bin/bash

crontab -l > /opt/rtcds/caltech/c1/scripts/crontab/crontab_$(hostname).$(date '+%Y%m%d%H%M%S')

I put this script into op340m crontab.

00 8 * * * /opt/rtcds/caltech/c1/scripts/crontab/backupCrontab

It took me 30 minutes to write and check this one line script. I hate shell scripts.

The backup of /cvs/cds (which runs as a cron job on fb40m; see /cvs/cds/caltech/scripts/backup/000README.txt) 
has been down since fb40m was rebooted on March 3.
I was unable to start it because of conflicting ssh keys in /home/controls/.ssh .
With help from Dan Kozak, we got it to work with both sets of keys
( id_rsa, which allows one to ssh between computers in our 113 network without typing a password,
 and backup2PB which allows the cron job to push the backup files to the archive in Powell-Booth).

It still goes down every time one reboots fb40m, and I don't have a solution.
A simple solution is for the script to send an email whenever it can't connect via ssh keys
(requiring a restart of ssh-agent with a passphrase), but email doesn't seem to work on fb40m.
I'll see if I can get help on how to have sendmail run on fb40m.

Ever since July 22, the backup script that runs on fb40m has failed to ssh to ldas-cit.ligo.caltech.edu to back up our trend frames and /cvs/cds.

This was a new failure mode which the scripts didn't catch, so I only noticed it when fb40m was rebooted a couple of days ago.

Alex fixed the problem (RAID array was configured with the wrong IP address, conflicting with the outside world), and I modified the script ( /cvs/cds/caltech/scripts/backup/rsync.backup ) to handle the new directory structure Alex made.

Now the backup is current and the automated script should keep it so, at least until the next time fb40m is rebooted...

Lock acquisition has gone bad tonight. 

The initial stage works fine, up through handing off control of CARM to MCL.  However, when increasing the AO path (analog gain), there are large DC shifts in the C1:IOO-MC_F signal.  Eventually this causes the pockels cell in the FSS loop to saturate, and lock is lost. 

 This problem has disappeared.  I don't know what it was. 

The first plot shows one of the symptoms.  The second plot is a similar section taken from a more normal acquisition sequence the night before.

All is not perfect, however, as now the handoff to RF CARM is not working.

The problem has returned.  I still don't know what it is, but it's making me angry. 

The Guralps were swapped.

What I did:   turned DC power off at 1X1,  hand carried them,  centered  each leveling bubbles, gently locked the jack screws and turned power back on.

ETMY at the east end now has CMG-T40-0008, sn T4157 as channel C1:PEM-SEIS_STS_1_Y_OUT_DQ.........

ETMX at south end now has CMG-T40-0053, sn T4Q17 as channel C1:PEM-SEIS_STS_2_Y_OUT_DQ.........

Conclusion:  Guralps are fine.  X  cable is bad. It was bad 6 months ago when it was made.

We can still swap the 3ft short cables at the granite base if Rana has not done it.

The fireworks of yesterday showing up in the lab. Pasadena out side air 6.6 million  cfm for 0.3 micron particles  and 1.5 million cfm for 0.5 micron size.

(Koji, Yuta)

Summary:
 The damping servos of the MC suspensions has not been working since yesterday. They had worked on Friday.
 We found that some of the filter coefficients somehow changed from the correct value during dealing with the sampling frequency shift (from 2048Hz to 16384Hz). (see elog #3659)

What we did:
We thought that the problem occurred from the CDS update on Monday. So, we restored them using svn.
 1. Went to /opt/rtcds/caltech/c1/core/advLigoRTS and ran svn update using the revision number 2125.
   svn update -r 2125
 2. Rebuilt all the front ends.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/core/advLigoRTS
   make clean-SYSNAME
   make SYSNAME
   make install-SYSNAME    (where SYSNAME=c1x02,c1sus,c1mcs,c1rms)
 3. Restarted the front end models.
   ssh -X c1sus
   cd /opt/rtcds/caltech/c1/scripts
   sudo startc1SYS    (where SYS=sus,mcs,rms)
 4. Restarted mx_streams.
   sudo /etc/restart_streams
See this wiki page for the restart procedures.

At this point we judged that the realtime system and "dataviewer" worked fine (by poking some suspensions / by measuring PSDs/TFs of the signals).

However, just restoring the RT system didn't fix the damping servos.

After that, we measured the transfer functions of the servo filters using DTT(diaggui on rosalba).
 5. The filter at MC1_SUSPOS, "3:0.0" should have a cut-off frequency at 3Hz, but it was around 0.3Hz.
 6. We used foton in order to look at the filter bank files (in /cvs/cds/rtcds/caltech/c1/chans). Then we found that the filter design was somehow changed to
   zpk([0],[0.375],2.66666,"n")
  instead of the correct one
   zpk([0],[3],0.333333,"n")

Why the filter designs changed?:
 We suspect that the filter designs were changed because of the replacement of sampling frequency in filter bank files(see elog #3659). We only replaced the lines like
  # SAMPLING SUS_MC3_SUSPOS 2048
 to
  # SAMPLING SUS_MC3_SUSPOS 16384
 and didn't changed the coefficients.
 This may caused a problem when opening the files with foton, and foton changed 3Hz to 0.375Hz (2048/16384) and other coefficients.

Note:
 The damping servo for SIDE has been working for every MCs. POS, PIT, YAW are the ones not working.

Next work:
 - Fix filters for every suspensions.
   There are backup files in /cvs/cds/rtcds/caltech/c1/chans/filter_archive directory, so this should help.
   But I don't think just fixing filters would fix the damping servo because the filter design of MC suspensions were changed this morning and the damping had worked until Friday.

Rana found out that a connection was bad in the shown place, due to which the MEDM screen was showing bad offset for length control.

Basically, the offset slider value would not go into the system because of that bad connection, and was locking the mode cleaner at the wrong location.

It turned out the oplev controls on ETMY were just bad.

It looks like the whitening filters have been OFF and because of that the resultant open-loop was not crossing the unity gain.

I will check the whitening filters.

(open-loop transfer function)

The blue dots are the measured data points and the green curve is the fit.

Apparently the open-loop doesn't go above the unity gain, so the oplev had been doing nothing.

If we try to increase the overall gain it will oscillate because of the phase delay of more than 180 deg around 3 Hz.

The red curve is the expected one with the whitening filters (WFs) properly engaged.

Note that WF are supposed to have two zeros at 1 Hz and two poles at 10 Hz.

In this morning I found daqawg didn't work.

After looking for the cause, I found one of the vme racks mounted on 1Y6 doesn't work correctly.

It looks like the vme rack mounting c0daqawg could not supply any power to the frontends.

Now Steve and I are trying to look for a spare for it.

Notes on May 25th

 Don't do the following things !! This causes bad cross-talking of CPUs mounted on the crate.

I moved c0daqawg and c1pem1 from 1Y6 vme crate to 1Y7 crate due to the bad power supply.

Another problem: c0dcu1 doesn't come back to the network. 

After moving them, I tried to get back them into the RFM network. However  c0dcu1 never came back, it still indicates red in C0DAQ_DETAIL.adl screen.

Alberto and I did even "nuclear option" (as instructed), but no luck.

 I got a VME crate from Peter's lab. It is already installed in 1Y6 instead of the old broken one.

I checked its power supply, and it looked fine. It successfully supplies +5, +12 and -12 V. And then I put c0daqawg and c1pem1 back from 1Y7.

Now I am trying to reboot all the front end computers with Peter's VME crate. A picture of the VME crate will be updated later.

With a curvature radius of about 57m for the ETMs, flat ITMs at the beam waist, and using 39m for the arm lengths, one finds that the beam radius at the ETMs is about 5.3mm. The clipping power loss of a 5.3mm beam through a 20mm radius baffle hole would be less than a ppm of a ppm if the beam was perfectly centered. If the baffle hole had 15mm radius, the clipping loss would be 0.01ppm. If the baffle hole had 10mm radius, the loss would be 810ppm. The loss values are calculated using the formula of the  "Gaussian beam" Wikipedia article, "Power through an aperture" section. So I did not check if that one is ok.

 This is how it was envisioned. The video camera was in nobodies mind to look through the 40 mm  diameter hole than.

 Rana is proposing 50 mm hole in the baffle plate that is attached to the tower.  Atm1

Atm2 is showing the back side where the solid line is 40 mm

Last night I aligned the incident beam axis and the Yarm by touching the PZT mirrors and the suspensions.

I didn't estimate how good they were aligned, but I guess the Y arm is now ready for the Y green light.

 Next : Y green alignment and the MC spots measurement / alignment.

 ++ Motivation ++

Prior to the coming vent we want to have the Y arm, incident beam axis and Y green light aligned so that we can align some necessary optics in the chamber.

Also alignment of the incident beam will allow us to re-position the incident beam alignment monitor (i.e. IPPOS and IPANG).

Our plan was to first align the Y arm using the ASS system and then align the Y green light to the Y arm.

++ what I failed ++

First I was trying to measure the spot positions on the MC mirrors to make me sure the beam axis has/hasn't changed.

Also I was going to align the MC suspensions to have nice spot position on each suspension using the MCASS system

because this will help us checking the beam clearance in the Faraday and perhaps re-positioning of the Faraday during the coming vent.

But essentially I failed and eventually gave up because MCASS didn't work. It seems that MCASS needs some modifications in the scripts.

Then, to make me feel better I moved on to the Y arm and beam axis alignment.

++ what I did ++

I tried using C1ASS to align the incident beam and suspensions on the Y arm, but it didn't work.

However the drive signals from ASS and its demodulated signals looked fine. Only the feedback did not work correctly.

Every time I enabled the feedback paths, the arm just lost the lock. Something is wrong in the feedback paths.

Then I started to align the cavity by my hands while looking at the demodulated signal from each LOCKIN module.

I aligned the things until each demodulated signal fluctuates around zero.

At the end the beam spots on the ETMY and ITMY camera looked well-aligned and the transmitted light became larger by a factor of 2ish.

You don't need the fourth glass piece on the diamond beam dump.

 IOO pointing monitoring qpds received razor beam dumps on their refs.

The Pos QPD was rotated and recentered.

The Ang QPD was left untouched.

TREND plot of 23 days is attached.

Anodized aluminum dumps replaced by 6 razor beam dumps.

Two more razor beam dumps added this afternoon.   The picture will updated tomorrow.

No wonder they could not find the beam dumps. Last night was Haloween. They should of just said: Trick or treat! where are the beam dumps?

 I studied more carefully beam path inside DRMI using PRM face camera and found that beam is clipping on PR3 edge.

Step 1: PRCL LOCK, MICH LOCK, power build up 30.

Note: left is right and vice versa on the PRM camera

 Step 2: PRLC - UNLOCK, MICH - LOCK, PRM is still aligned. Right photo is AS port. I've slightly misaligned ITMs such that disturbance of AS beam is clearly seen.

Step 3: PRCL - UNLOCK, MICH - LOCK, PRM misalined in yaw such such that the beam LASER -> PRM -> PR2 -> PR3 -> BS -> ITMX -> BS -> PR3 -> PR2 -> PRM -> PR2 -> PR3 is completely clipped on the TT edge. AS beam is now not clipped.

So the conclusion is that when PRC is not locked and beam is thin, it can avoid clipping. When PRC locked, beam size grows and it starts to clip. I think we need to move the mount next to PR3 because of it we to not have enough space to align the TT.

Step 4: PSL shutter is closed.

 Some explanation of how you define power buildup please. Also some plots showing the evidence.

 I think about power buildup as a ratio of the power in the cavity when it is locked and unlocked = (POYDC_LOCKED - POYDC_OFFSET) / (POYDC_UNLOCKED - POYDC_OFFSET). I do not multiply this number by PRM transmission.

POYDC_OFFSET = -0.006

POYDC_UNLOCK = 0.063

For example, on the plot below power buildup is 15.

That's OK, but its best to use standard notation. The power recycling gain is defined as the power incident on the BS divided by the power incident on the PRM from the laser side. You should also compare it with the PRC gain that you expect from mirror transmissions.

 I've made snapshots of PR2, PRM, ITMY and ITMX mirrors. Power buildup recycling gain (POWER BS / POWER PRM) was equal to 3-4.

 We've looked at PR2 face camera when PRM, BS and one of the ITMs were aligned. We saw an extra beam at PR2 when ITMX was aligned (right plot). This spot stays on the PR2 when prcl is locked.

Then we looked at PR3 transmission mirror and saw that the main beam is not on the edge of the mirror. Secondary beam is clipping on the mirror mount of PR3 that we see on BS_PRM camera.

Measured beam spot positions:

"+" for pitch means that the beam is too high, "-" too low

"+" for yaw means that the beam is left if you look from the back, "-" is right

Beam spots were measured using x, y arm and prcl locking to the carrier.

[Koji, Osamu and Kiwamu]

 We aligned the beam axis pointing down to both X and Y arm.

Now the beams are hitting the centers of both ETMX and ETMY.

Amazingly Osamu made X arm flashing by aligning the cavity.

(what we did)

- opened almost all the chambers except for the MC2 chamber.

- locked and aligned the MC.

   We set Marconi to the right frequency, which had been set to the default values, probably due to the power outage in the last weekend.

   Also we found a DAC cable disconnected from the IO chassis of c1sus. So we connected it in order to damp the MC suspensions.

- aligned MMT2 and PZT2 in order to let the beam go through the center of PRM.

- checked the beam centering at the two TTs (PR2, PR3).

- rotated PR3 to make the beam go through the centers of both ITMY and BS at the same time.

- tried finding the beam spot at the ETMY chamber, and successfully found it.

    To see such faint beam spot, we used an IR viewer.

    In addition to that, we put a large piece of aluminum foil as a screen in the chamber.

- aligned the beam to the center of ETMY by tweaking the PZT mirror (SM2).

- aligned the BS so that the reflected beam at the BS goes through the center of ITMX.

- tried finding the beam spot at the X end, and successfully found it hitting the wall in the chamber.

- aligned the BS in order to let the beam hit the center of ETMX.

- tried aligning ETMX and ITMX to the beam.

Eventually we made the X arm flashing.

However the flash was a bit too weak to completely align the cavity.

(plan for tomorrow)

- reinstall some steering mirrors into the BS chamber

- check and neutralize PZT1

- alignment of IP_ANG

Good news. I feel multi-chromatic-locking success is just around the corner.

By the way, there's a new presentation on the DCC from the ANU group where they've locked a short single cavity with both colors - G1000735:

https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?docid=14040

The beam profile of the LWE (LightWave Electronics) NPRO was measured.

Mode matching telescopes will be designed and setup soon based on the result of the measurements.

Here is a plot of the measured beam profile.

 (some notes)

The measurement was done by using Kevin's power attenuation technique (#3030).

An window was put just after the NPRO and the reflected beam was sampled for the measurement to avoid the beam scan saturated.

On Friday, Valera and I calculated the modematching for reference cavity from AOM.

We scan the beam profile where the spot should be.

The first beam waist in the AOM is 103 um, the lens (f= 183 mm, I'm not sure if I have the focal length right) is 280 mm away.

The data is attached. The first column is marking on the rail in inches,

the second column is distance from the lens, the third and fourth column are

vertical and horizontal spot radius in micron. Note that the beam is very elliptic because of the AOM.

I started to create a Finesse model of the PRC cavity. We have the phase maps for the PRC and the two ITMs. I could not find anything for PR2,3 and BS. All files can be found in my SVN folder /janosch/PRC40m. I used the AutoCAD model to determine angles of incidence and distances. These numbers are largely inconsistent with numbers that you can find elsewhere on the 40m wiki, but this certainly depends on what accuracy is required for interferometer alignment and I don't understand anything about alignment.

The phase maps come in a format that needs to be modified before they can be used in Finesse. I have started with this work, but maybe someone else can take over. The phase maps show tilts and the PRC also has the curvature. These have to be subtracted out before the maps can be loaded into Finesse. I asked GariLynn for the code that they use. The Finesse model (MichPRC_40m.kat) does not load the phasemaps yet, and I just wrote some random parameter values for the TEM00 input beam to the PRC. So these Gauss parameters need to be corrected.

I will only go on with this work if Rana tells me that I should do so, otherwise it is on hold until we have a volunteer.

I checked the spot positions on MC1 and MC3 by running Yuta's A2L script.

The amounts of the off-centering were good except for YAW of MC1.

 So we have to adjust the YAW alignment of the beam axis by steering the mirrors at the PSL table.

- - - (measured off-centering) - - - 

MC1_PIT  =  -0.711 mm

MC1_YAW =  1.62 mm

MC3_PIT  =   -0.0797 mm

MC3_YAW  =  - 0.223 mm