Cranes are  checked and they are ready for lifting. At the east end will use the manual Genei-lift to put door on.

Keven and Steve,

The 3 cranes tested and  wiped off as preparation for upcoming vent.

 Atm1, service report: load test were performed at max horizontal reach with 1998 lbs ( American Ton is 2000 lbs)

          Vertical drives and brakes worked well. The 5 minutes sagging test showed less than 1 mm movement .

          The wiring is correct. Earlier hypothesis regarding the wiring ignored the mechanical brake action.

Our cranes are certified now. Operator training and SOP is in the work.

Vertex Folding I -beam will get latch-lock and the south end I-beam will be leveled.

Atm2, south end

Atm3, east end

Atm4, folding crane at ITMX at 14 ft horizontal reach

Fred Goodbar of Konecrane has completed the annual certified crane inspection and maintenance of our cranes as required in safety document.

They are in good working condition and safe to use.

I gave up tonight's locking activity because the MC can't stay locked.

It seems that somehow the seismic noise became louder from about 1:00 AM.

I walked around the outside of the 40-m building to see what's going on, but no one was jumping or partying.

I am leaving the MC autolocker disabled so that the laser won't be driven crazy and the WFS won't kick the MC suspensions.

The attachment is a 3-hour trend of the seismometer outputs and the MC trans.

 Something has started shaking last night.  Everybody is claiming to be innocent next door.

I turned off the 40m AC at 11:06

I set a cron job on allegra.martian to run the diagnostic script every weekend.

I think this routine can be helpful to know how the trend of timing-shift goes

The cron runs the script on every Sunday 5:01AM and diagnostics will take about 5 min.

! Important:

During the running of the script, OMC and DARM can not be locked.

If you want to lock OMC and DARM in the early morning of weekend, just log in allegra and then comment out the command by using 'crontab -e'

As I wrote on Oct.27th, the cron job works every Sunday.

I found it worked well on the last Sunday (Nov.1st).

And I can not find any timing jitter in the data, its delay still stay 3*Ts.

In the weekly meeting, Jordan pointed out that we didn't receive the alert for the low N2 pressure.

To check the situation, I went around the machines and summarized the cronjob situation.
[40m wiki: cronjob summary]
Note that this list does not include the vacuum watchdog and mailer as it is not on cronjob.

Now, I found that there are two N2 scripts running:

1. /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh on megatron and is running every minute (!)
2. /opt/rtcds/caltech/c1/scripts/Admin/N2check/pyN2check.sh on c1vac and is running every 3 hours.

Then, the N2 log file was checked: /opt/rtcds/caltech/c1/scripts/Admin/n2Check.log

Wed Sep 1 12:38:01 PDT 2021 : N2 Pressure: 76.3621
Wed Sep 1 12:38:01 PDT 2021 : T1 Pressure: 112.4
Wed Sep 1 12:38:01 PDT 2021 : T2 Pressure: 349.2
Wed Sep 1 12:39:02 PDT 2021 : N2 Pressure: 76.0241
Wed Sep 1 12:39:02 PDT 2021 : N2 pressure has fallen to 76.0241 PSI !

Tank pressures are 94.6 and 98.6 PSI!

This email was sent from Nodus.  The script is at /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh

Wed Sep 1 12:40:02 PDT 2021 : N2 Pressure: 75.5322
Wed Sep 1 12:40:02 PDT 2021 : N2 pressure has fallen to 75.5322 PSI !

Tank pressures are 93.6 and 97.6 PSI!

This email was sent from Nodus.  The script is at /opt/rtcds/caltech/c1/scripts/Admin/n2Check.sh

...

The error started at 11:39 and lasted until 13:01 every minute. So this was coming from the script on megatron. We were supposed to have ~20 alerting emails (but did none).
So what's happened to the mails? I tested the script with my mail address and the test mail came to me. Then I sent the test mail to 40m mailing list. It did not reach.
-> Decided to put the mail address (specified in /etc/mailname , I believe) to the whitelist so that the mailing list can accept it.
I did run the test again and it was successful. So I suppose the system can now send us the alert again.
And alerting every minute is excessive. I changed the check frequency to every ten minutes.

What's happened to the python version running on c1vac?
1) The script is running, spitting out some error in the cron report (email on c1vac). But it seems working.
2) This script checks the pressures of the bottles rather than the N2 pressure downstream. So it's complementary.
3) During the incident on Sept 1, the checker did not trip as the pressure drop happened between the cronjob runs and the script didn't notice it.
4) On top of them, the alert was set to send the mails only to an our former grad student. I changed it to deliver to the 40m mailing list. As the "From" address is set to be some ligox...@gmail.com, which is a member of the mailing list (why?), we are supposed to receive the alert. (And we do for other vacuum alert from this address).

I wrapped rampdown.py in rampdown.sh, which is just these lines:

#!/bin/bash
source /ligo/cdscfg/workstationrc.sh
/opt/rtcds/caltech/c1/scripts/SUS/rampdown.py > /dev/null 2>&1

This is now what megatron's cron runs. It appears to be working.

I also added the workstationrc line to the n2 and chiara HDD checking scripts that run on nodus, which should resolve the issue from ELOG 11249

I accidentally overwrote crontab in op340m with an empty file.
By checking /var/cron in op340m, I think I restored it.
But somehow, autolockMCmain40m does not work in cron job, so it is currently running by nohup.

What I did:
  1. I ssh-ed op340m to edit crontab to change MC autolocker to usual power mode. I used "crontab -e", but it did not show anything. I exited emacs and op340m.
  2. Rana found that the file size of crontab went 0 when I did "crontab -e".
  3. I found my elog #6899 and added one line to crontab

55 * * * *  /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1

  4. It didn't run correctly, so Rana used his hidden power "nohup" to run autolockMCmain40m in background.
  5. Koji's hidden magic "/var/cron/log" gave me inspiration about what was in crontab. So, I made a new crontab in op340m like this;

34 * * * *  /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1
55 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/RCthermalPID.pl >/cvs/cds/caltech/logs/scripts/RCthermalPID.cronlog 2>&1
07 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo >/cvs/cds/caltech/logs/scripts/FSSslow.cronlog 2>&1
00 * * * * /opt/rtcds/caltech/c1/burt/autoburt/burt.cron >> /opt/rtcds/caltech/c1/burt/burtcron.log
13 * * * * /cvs/cds/caltech/conlog/bin/check_conlogger_and_restart_if_dead
14,44 * * * * /opt/rtcds/caltech/c1/scripts/SUS/rampdown.pl > /dev/null 2>&1

  6. It looks like some of them started running, but I haven't checked if they are working or not. We need to look into them.

Moral of the story:
  crontab needs backup.

I found some daily cron jobs for op340m I missed last night. Also, I edited timings of hourly jobs to maintain consistency with the past. Some of them looks old, but I will leave as it is for now.
At least, burt, FSSSlowServo and autolockMCmain40m seems like they are working now.
If you notice something is missing, please add it to crontab.

07 * * * * /opt/rtcds/caltech/c1/burt/autoburt/burt.cron >> /opt/rtcds/caltech/c1/burt/burtcron.log
13 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo >/cvs/cds/caltech/logs/scripts/FSSslow.cronlog 2>&1
14,44 * * * * /cvs/cds/caltech/conlog/bin/check_conlogger_and_restart_if_dead
15,45 * * * * /opt/rtcds/caltech/c1/scripts/SUS/rampdown.pl > /dev/null 2>&1
55 * * * *  /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/MC/autolockMCmain40m >/cvs/cds/caltech/logs/scripts/mclock.cronlog 2>&1
59 * * * * /opt/rtcds/caltech/c1/scripts/general/scripto_cron /opt/rtcds/caltech/c1/scripts/PSL/FSS/RCthermalPID.pl >/cvs/cds/caltech/logs/scripts/RCthermalPID.cronlog 2>&1

00 0 * * * /var/scripts/ntp.sh > /dev/null 2>&1
00 4 * * * /opt/rtcds/caltech/c1/scripts/RGA/RGAlogger.cron >> /cvs/cds/caltech/users/rward/RGA/RGAcron.out 2>&1
00 6 * * * /cvs/cds/scripts/backupScripts.pl
00 7 * * * /opt/rtcds/caltech/c1/scripts/AutoUpdate/update_conlog.cron

According to Google, you can add a line in the crontab to backup the crontab by having the cronback.py script be in the scripts/ directory. It needs to save multiple copies, or else when someone makes the file size zero it will just write a zero size file onto the old backup.

We need a plan how to minimize cross coupling in the OSEMs now

[ Koji / Kiwamu ]

The frequent unlock of the MC are most likely unrelated to ground motion.

Although the reason why MC became unstable is still unclear.

There are two facts which suggest that the ground motion and the MC unlock are unrelated :

(1) It turned out that the seismometer signals (C1:PEM-SEIS-STS_AAA ) have a big cross talk with the MC locking signals.

    For example, when we intentionally unlocked the MC, the seismometer simultaneously showed a step-shaped signals, which looked quite similar to what we have observed.

    I guess there could be some kind of electrical cross talk happening between some MC locking signals and the seismometer channels.

    So we should not trust the signals from the STS seismometers. This needs a further investigation.

(2) We looked at the OSEM and oplev signals of some other suspended optics, and didn't find any corresponding fluctuations.

    The suspensions we checked are ETMX, ETMY, ITMX and MC1.

     None of them showed an obvious sign of the active ground motions in the past 24 hours or so.

attached plot shows MC_IN1/MC_IN2.  needs work.

This is supposed to be a measurement of the relative gain of the MCL and AO paths in the CM servo. We expect there to

be a more steep slope (ideally 1/f). Somehow the magnitude is very shallow and so the crossover is not stable. Possible

causes? Saturations in the measurement, broken whitening filters, extremely bad delay in the digital system? needs work.

I tested the cryopump interlock today. It is touchy. I do not have full confidence in it.

I'm proposing that VC1 gate valve should be kept closed while nobody is working in the 40m lab.

How to open gate valve:

1, confirm temp of 12K on the gauge at the  bottom of the cryopump

2, if medm screen cryo reads OFF( meaning warm) hit reset will result reading ON (meaning cold 12K )

3, open VC1 gate valve if P1 is not higher than 20 mTorr

VC1 was closed at 18:25,

IFO condition: not pumped,

expected leak plus  out gassing should be less than 5 mTorr/day

The RGA is in bg-mode, annuloses are closed off

 The Cryo pump is running reliably since April 22 hence there is no need to close VC1 repeatedly.

The photo switch interlock was put back onto the H2 vapor pressure gauge and it is working.

Congratulation Yoichi and Peter for full rf locking at night. Let's remember that the cryopump was shaking the hole vac envelope and ifo during this full lock.

I found some neat signal analysis software for my mac (http://www.faberacoustical.com/products/), and took a spectrum of the ambient noise coming from the cryopump.  The two main noise peaks from that bad boy were nowhere near 3.7 kHz.

To make the mcwfson/off scripts work from rossa (and not just Jamie's pet machine) I swapped the sh-bang line at the top of the script to use 'env bash' instead of 'env csh' in the case of mcwfsoff and 'env tcsh' in the case of mcwfson.

The script was failing to work due to $OSTYPE being defined for pianosa csh/tcsh, but not on rossa.

During debugging I also bypassed the ezcawrapper for ezcaswitch so that now when ezcaswitch is called, it directly runs the binary and not the script which calls the binary with numerous retries. In the future, all new scripts will be rewritten to use cdsutils, but until then beware of ezcaswitch failures.

WFS scripts checked into the SVN.

This was all in an effort to get Koji to allow me to upgrade pianosa to ubuntu 12 so that I can have ipython notebook on there.

Objections to upgrading pianosa? (chiara and megatron are already running ubuntu 12)

[Jenne,Yuta]

Both arms are aligned starting from Y green.
We have all beams unclipped except for IPANG. I think we should ignore IPANG and go on to PRMI locking and FPMI locking using ALS.
IPANG/IPPOS and oplev steering mirrors are kept un-touched after pumping until now.

Current alignment situation:
 - Yarm aligned to green (Y green transmission ~240 uW)
 - TT1/TT2 aligned to Yarm (TRY ~0.86)
 - BS and Xarm alined to each other (TRX ~ with MI fringe in AS)
 - X green is not aligned yet
 - PRMI aligned

Current output beam situation:
 IPPOS - Coming out clear but off in yaw. Not on QPD.
 IPANG - Coming out but too high in pitch and clipped half of the beam. Not on QPD.
 TRY   - On PD/camera.
 POY   - On PD.
 TRX   - On PD/camera.
 POX   - On PD.
 REFL  - Coming out clear, on camera (centered without touching steering mirrors).
 AS    - Coming out clear, on camera (centered without touching steering mirrors).
 POP   - Coming out clear, on camera (upper left on camera).

Oplev values:
Optic    Pre-pump(pit/yaw)    PRFPMI aligned(pit/yaw)
ITMX    -0.26 /  0.60         0.25 /  0.95
ITMY    -0.12 /  0.08         0.50 /  0.39
ETMX    -0.03 / -0.02        -0.47 /  0.19
ETMY     0.37 / -0.62        -0.08 /  0.80
BS      -0.01 / -0.18        -1    /  1 (almost off)
PRM     -0.34 /  0.03        -1    /  1 (almost off)

 All values +/- ~0.01. So, oplevs are not useful for alignment reference.

OSEM values:
Optic    Pre-pump(pit/yaw)    PRFPMI aligned(pit/yaw)
ITMX    -1660 / -1680        -1650 / -1680
ITMY    -1110 /   490        -1070 /   440
ETMX     -330 / -5380         -380 / -5420
ETMY    -1890 /   490        -1850 /   430
BS        370 /   840          360 /   800
PRM      -220 /  -110         -310 /  -110

  All values +/- ~10.
  We checked that if there's ~1200 difference, we still see flash in Watec TR camera. So, OSEM values are quite good reference for optic alignment.

IPANG drift:
  On Saturday, when Rana, Manasa, and I are trying to get Y arm flash, we noticed IPANG was drifting quite a lot in pitch. No calibration is done yet, but it went off the IPANG QPD within ~1 hour (attached).
  When I was aligning Yarm and Xarm at the same time, TRY drifted within ~1 hour. I had to tweak TT1/TT2 mainly in yaw to keep TRY. I also had to keep Yarm alignment to Y green. I'm not sure what is drifting so much. Suspects are TT2, PR2/PR3, Y arm and Y green.

  I made a simple script(/opt/rtcds/caltech/c1/scripts/Alignment/ipkeeper) for keeping input pointing by centering the beam on IPPOS/IPANG using TT1/TT2. I used this for keeping input pointing while scanning Y arm alignment to search for Y arm flash this weekend (/opt/rtcds/caltech/c1/scripts/Alignment/scanArmAlignment.py). But now we have clipped IPANG.


So, what's useful for alignment after pumping?:
  Optic alignment can be close by restoring OSEM values. For input pointing, IPPOS/IPANG are not so useful. Centering the beam on REFL/AS (POP) camera is a good start. But green works better.

Background:
 The output range of the OSEM is 0-2V.
 So, the OSEM output should fluctuate around 1V.
 If not, we have to modify the position of it.

What I did:
 Measured current outputs of the 5 OSEMs for each 8 suspensions by reading sensor outputs(C1:SUS-XXX_YYPDMON) on medm screens.

Result:

 White: OK (0.8~1.2)
 Yellow: needs to be fixed
 Red: BAD. definitely need fix

Below is the overview of all the core IFO suspension input diagonalizatidons.

Summary: ITMY, PRM, BS are really bad (in that order) and are our top priorities.

UPDATE:

I had originally put the condition number of the calculated input matrix (M) in the last column.  However, after some discussion we decided that this is not in fact what we want to look at.  The condition number of a matrix is unity if the matrix is completely diagonal.  However, even our ideal input matrix is not diagonal, so the "best" condition number for the input matrix is unclear.

What instead we do know is that the matrix, B, that describes the difference between the calculated input matrix, M, and the ideal input matrix, M₀: should be diagonal (identity, in fact):

M = M₀ B

B should be diagonal (identity, in fact), and it's condition number should ideally be 1.  So now we calculate B^-1, since it can be calculated from the pre-inverted input matrix:

B^-1 = M^-1 * M₀

From that we calculate cond(B) == cond(B^-1).

cond(B) is our new measure of the "badness" of the OSEMS.

new summary: ITMY, PRM, BS are really bad (in that order) and are our top priorities.

TM		INPUT MATRIX (M)	cond(M)	cond(B)
PRM		pit     yaw     pos     side    butt UL   -2.000  -2.000  -2.000  -0.345   2.097  UR   -0.375  -0.227  -0.312  -0.060   0.247  LR    1.060   1.075   0.971   0.143  -0.984  LL   -0.565  -0.698  -0.717  -0.141   0.672  SD    1.513   1.485   1.498   1.000  -1.590	75.569	106.756
SRM		pit     yaw     pos     side    butt UL    0.791   1.060   1.114  -0.133   1.026  UR    1.022  -0.940   1.052  -0.061  -1.027  LR   -0.978  -0.987   0.886  -0.031   0.903  LL   -1.209   1.013   0.948  -0.103  -1.043  SD    0.286   0.105   1.249   1.000   0.030	2.6501	3.90776
BS		pit     yaw     pos     side    butt UL    1.420   0.818  -0.069   0.352   1.038  UR    0.276  -1.182   1.931  -0.217  -0.905  LR   -1.724  -0.274   1.940  -0.254   0.862  LL   -0.580   1.726  -0.060   0.315  -1.194  SD    0.560   0.171  -3.535   1.000   0.075	9.8152	7.28516
ITMX		pit     yaw     pos     side    butt UL    0.437   1.015   1.050  -0.065   0.714  UR    0.827  -0.985   1.129  -0.221  -0.957  LR   -1.173  -1.205   0.950  -0.281   1.245  LL   -1.563   0.795   0.871  -0.125  -1.084  SD   -0.581  -0.851   2.573   1.000  -0.171	4.08172	4.69811
ITMY		pit     yaw     pos     side    butt UL    0.905  -0.884  -0.873   0.197   0.891  UR   -1.095   1.088   1.127  -0.252  -1.115  LR   -0.012  -0.028   0.002   0.001   0.030  LL    1.988  -2.000  -1.998   0.451   1.964  SD    4.542  -4.608  -4.621   1.000   4.517	801.453	774.901
ETMX		pit     yaw     pos     side    butt UL    0.344   0.475   1.601   0.314   1.043  UR    0.283  -1.525   1.786  -0.071  -1.181  LR   -1.717  -1.569   0.399  -0.102   0.938  LL   -1.656   0.431   0.214   0.283  -0.837  SD    0.995  -2.632  -0.999   1.000  -0.110	4.26181	4.33518
ETMY		pit     yaw     pos     side    butt UL   -0.212   1.272   1.401  -0.127   0.941  UR    0.835  -0.728   1.534  -0.101  -1.054  LR   -0.953  -1.183   0.599  -0.066   0.827  LL   -2.000   0.817   0.466  -0.092  -1.177  SD   -0.172   0.438   2.238   1.000  -0.008	4.04847	4.33725

I moved amplifiers for beat PD at PSL table to 1X2 rack. Current beat setup from PD to ADC is shown below. Setup for X beat and Y beat are almost the same except for minor difference like cable kind for the delay line.

Currently, DC power for amplifiers ZHL-1000LN+ is supplied by Aligent E3620A.
I tried to use power supply from the side of 1X1 rack, but fuse plug(Phoenix Contact ST-SI-UK-4) showed red LED, so I couldn't use it.
Measured amplification was +25 dB for 10-100 MHz.

Measured gain from RF input to monitor output of the beat box was ~ -1 db for 10-100 MHz.

Summary:
   I set Q-values for each ringdown of PRM, BS, ITMX, ITMY, MC1, MC2, MC3 to ~5 using QAdjuster.py.
   Here are the results;


  Red ringdowns indicate the second try after gain setting.

Note:
  - ITMX and ITMY are referred according to MEDM screens in this entry.
  - ITMX(south) OSEM positions are currently so bad(LL and SD are all the way in/out).
        I have to change IFO_ALIGN slider values to check the damping servo. For SIDE, I couldn't do that. I reverted the slider change after the damping checking.
  - ITMY(west) somehow has opposite coil gain sign.
       Usually for the other optics, UL,UR,LR,LL is 1,-1,1,-1. But for ITMY to damp, they are -1,1,-1,1.
  - PRM damps, but ringdown doesn't look nice. There must be something funny going on.
  - SRM doesn't have OSEMs put in now.

Jamie pointed out that the compile and install instructions are different for c1dnn:

See also: https://nodus.ligo.caltech.edu:8081/40m/13383.

I think these build instructions have to be run on the c1lsc frontend - in the past, I have been able to compile and install models on any computer with the shared drive mounted (including the control room workstations), but I'm guessing that something has changed since the RCG upgrade. Jamie can correct me on this if I'm wrong.

This is the current procedure to start the c1dnn model:

Then to shutdown:

The daqd already knows about this model, so nothing should need to be done to the daqd to make the dnn channels available.

Current status of the new CDS test (summaries can be found on the wiki)

- Cables 

  All the necessary cables are in our hands, such as SCSIs, 37 pin D-sub and 3 pin power cables.

  With a big help from Jenne and Koji, we made 37 pin F/M cables and 3 pin power cables on the last Tuesday.

  Now all the cables are connected to the machines except for the 3 pin power cables.

  To install the power cables I will switch off Sorensens at new 1X5  for a minute.

- Suspension model file 

  The model file named C1SUS was made and I succeeded in compiling and building it. So it is ready for the test.

  But some medm screens look like not correctly complied.

  For example the channel names which are listed on C1SUS_MONITOR_ADC0.adl aren't correctly assigned.

- ADCs 

 All the ADC channels are working. Also the channel assignments are correct.

 I checked all the ADC channels if it was working while I ran the front end realtime code C1SUS.

 By injecting a signal from a function generator directly to the SCSIs, I could clearly see the numbers hopping on medm screens.

- DACs 

  None of them are working although the computer can recognize that all three DAC cards are mounted.

  I think something in the IOP file and the model file may be wrong because this symptom looks similar to that of C1ISCEX we tested two weeks before ( see this entry ) 

  I have to fix it anyhow because the DACs are very necessary part for this damping test.

- Binary Outputs

  They didn't work at all. Even the computer didn't recognize the binary output cards.

  I think I should put some magical components on the model files.

- Conversion of the filter coefficients 

  The conversion of the filter coefficients has been doen yesterday.

  It looks running well because I can load and unload these filters on medm screens.

  I manually copied the filter coefficients from the current suspension filter file to that of new filter file.

 The current file can be found under /cvs/cds/caltech/chans/,  and the new one is under /cvs/cds/rtcds/caltech/c1/chans

Suspended works

   - Binary Outputs

   - simlink realtime model with the new CDS PARTS ( see the notice from Joe )

 

To be done

   - let DACs work

   - damping tests

First, awesome progress.

Second, the Binary Output parts do in fact need to be added to the c1sus model.  They don't need to appear in the IOP though.  (They are somehow automatically taken care of by the IOP code). 

It looks like in the 2004 revision in SVN, the cdsCDO32 part (located in the CDS_PARTS.mdl/IO_PARTS) was broken.  I fixed it and updated the svn with it, so a svn update should pull the corrected version.

I'm not sure whats wrong with the DAQs.  I'll try to take a closer look at the model file tonight and see if I can make suggestions.  When you write outputs on DAC_0 or DAC_1 is the C1SUS GDS TP screen showing anything?

[Jamie / Suresh / Kiwamu] 

The in-vac work is ongoing.

Before we run out our energy we are posting this entry to briefly report the current status.

- (done) BS earthquake stop adjustment.

- (done) PRM earthquake stop adjustment

- (done) MC spot position check => They are almost the same within 10 %.

- (done) Injection and alignment of the ABSL laser to make the beam brighter in the vertex region.

- (done) POY => We repositioned an in-vac steering mirror to get the POY beam hitting the center of the steering mirror.

                           It's now coming out from the chamber.

-  (done)  IPANG => realigned two mirrors on the ETMY table to get the IPANG out from the chamber. Now it's reaching the ETMY optical table.

                             It needs a final touch before we pump down.  We revisited it later in the night after realigning the IFO and it is well aligned now.

- (done)  POP => We have aligned the ABSL laser injected from the AS port to reach the REFL camera.  We turned it up to max power of 300mW and used it as a substitute for the PRC beam.

                         Even this was not enough to see anything in the POP beam path after the PR2 (tip-tilt).  So we used a green beam from the Y-arm as a guide of the POP beam path because the ABSL (POP) beam was too dim to work with.

                         We placed a lens and a CCD camera to detect the green and then blocked the Y-green.  It was then possible to see the ABSL-POP beam in the CCD camera.  The lens and the CCD are markers for this beam. 

                         Do not remove these markers unless absolutely necessary.

-(done) POX => We located the ABSL (mimicking POX) beam on the POXM1 mirror and adjusted the mirror to ensure that the beam exits at the right height and a convenient location on the POX table. 

- (0%) OSEM mid-range adjustment

- (0%) IPPOS

- (0%) oplev re-alignment

We removed the curved mirror behind the AOM (ROC=0.3m) on PSL table. The mirror is now in PSL lab. See PSL:905 for more detail.

The four 4x25DSUB and single 8x25DSUB feedthrough flanges have arrived and will be picked up from the dock and brought to the 40M lab.

 The CIT Chemistry Glass Shop cuts turned out to be sloppy using diamond disc blade  cutter.

East coast Precision Glass & Optics  offered scribed cut and polished side. Their quote price was high and time consuming.

The GLASS HOUSE  shop in Pasadena 626 / 796-9151 on Walnut did a good and cheap job. Oscar the cutter will finish the rest of the cutting by next Friday, April 6

He used scribed cutting technic and his 1" x  0.5"  pieces are good. Bob will pick up them up.

After replacing filters, MC suspensions damped  last night.

Further measurement next time.

damped sus at atm1 and freeswingging sus at atm2

Background:
 Even if we can't use the Dataviewer to get the time series of each 4 DOF displacements, we can still use StripTool to monitor the ringdowns and see if the damping servo is working.

What I did:
1. Excited vibration by kicking the mirror randomly (by putting some offsets randomly and turing the filters on and off randomly).

2. Turned all the servo off by clicking "ShutDown" button.

3. Turned all the servo on by clicking "Normal" button.

3. Monitored each 4 DOF displacements with StripTool and see if there are any considerably low-Q ringdown after turning on the servo.
 The values I monitored are as follows;
  C1:SUS_MCX_SUSPOS_INMON
  C1:SUS_MCX_SUSPIT_INMON
  C1:SUS_MCX_SUSYAW_INMON
  C1:SUS_MCX_SDSEN_INMON  (X=1,2,3)

All the settings I used for this observation are automatically saved here;
  /cvs/cds/caltech/burt/autoburt/snapshots/2010/Oct/11/21:07/c1mcs.epics

Result:
 Attached is the screenshots of StripTool Graph window for each 3 MCs.
 As you can see, the dampings for each DOF, each MCs are basically working.

Note:
 Do NOT turn off all the damping servo by clicking "Damp" buttons or setting the SUSXXX_GAIN to 0. It crashes c1mcs.

Next work:
 - check and relate the signal sign with the actual moving direction of the optics
 - fix data aquisition system
 - measure Q-values when the servo is on and off using DAQ and Dataviewer

ETMY-south sus damping was restored

As a suspension test I am leaving all of the suspensions restored and damped with OSEMS but without oplevs

I just realized that the "damprestore" script that can be called from the watchdog screen did not have the new oplev names.  I have updated it, and added it to the svn.

GUR1 XYZ, GUR2 XYZ, MC_F channels are now recorded at 256 Hz.

EDIT by JCD:  What Den means to say here is that (a) he modified some .ini files, and (b) he restarted the fb.

I tried to add a test point to C1MCS model and spent next two hours rebooting front-ends, restarting models and realigning MC.

dmesg told me that DAQ channels can not be allocated as they already exist. Last time we met this problem Jamie emailed Alex about it. Jamie, what is the output? Restarting iop model does not help this time.

Den tried adding a SINGLE acquire channel to the c1ioo, which for some reason hung c1ioo and took down the entire DAQ network (at least all communication between the front ends and the fb).  We recovered by restarting c1ioo and restarting mx_stream on all the rest of the front-ends

After "recovering", though, c1ioo is failing to load models, or at least it's IOP. Here is the tail of dmesg when trying to start the IOP:

[ 1751.140283] c1x03: Initializing space for daqLib buffers
[ 1751.140284] c1x03: Initializing Network
[ 1751.140285] c1x03: Found 1 frameBuilders on network
[ 1751.250658] CPU 2 is now offline
[ 1751.250657] c1x03: Sync source = 4
[ 1751.250657] c1x03: Waiting for EPICS BURT Restore = 1
[ 1751.310008] c1x03: Waiting for EPICS BURT 0
[ 1751.310008] c1x03: BURT Restore Complete
[ 1751.310008] c1x03: Initialized servo control parameters.
[ 1751.311699] c1x03: DAQ Ex Min/Max = 1 3
[ 1751.311699] c1x03: DAQ XEx Min/Max = 3 53
[ 1751.311733] c1x03: DAQ Tp Min/Max = 10001 10007
[ 1751.311733] c1x03: DAQ XTp Min/Max = 10007 10507
[ 1751.311737] c1x03: DIRECT MEMORY MODE of size 64
[ 1751.311737] c1x03: daqLib DCU_ID = 33
[ 1751.311737] c1x03: Invalid num daq chans = 0
[ 1751.311737] c1x03: DAQ init failed -- exiting

The chan file for this model (/opt/rtcds/caltech/c1/chans/daq/C1X03.ini) looks totally fine, has two un-acquired channels uncommented, and has otherwise not been touched. The C1:FEC-33_MSGDAQ is also reading: "ERROR reading DAQ file!"

I'm at a loss for what is going on. I've tried restarting every CDS process on the machine, restarting the model multiple times, restarting fb, and even restarting the entire c1ioo machine, all to no affect.

For some reason this afternoon we've been experiencing a lot of problems with the framebuilder, and with the CDS network in general.  The framebuilder has been very unresponsive, although the daqd logs seem to indicate that things are ok.  All models will loose contact with fb for very long stretches.  Attempts to kill/restart daqd don't seem to fix the problem.

These problems seem to be associated with the general CDS network issues as well.  The network seems to become very slow, and the workstations all become very slow.  The later I assume is because of the network and that so much of the work we do is on network mounted filesystems (/opt/rtcds, /ligo, etc.).

My current speculation is that daqd on fb is doing something stupid, like trying to read or write a bunch of stuff from /frames, which is also network mounted, and that clogs up the entire network.  Some serious network debugging is going to be needed to figure out what's going on, though.

I'm afraid daqd is caught in some bad state now, though.  It's not responding to anything, and every attempt to kill it seems to bring it back into the bad state.  Hopefully I can get Alex to help me figure out what's going on tomorrow.   Maybe it will clear up on it's own tonight...

It looks like daqd is indeed caught in some bad state.  It seems to die at some point after making GPS corrections to minute trender:

...
[Mon Aug  6 19:45:13 2012] Minute trender made GPS time correction; gps=1028342727; gps%60=27
tail: `fb/logs/daqd.log' has been replaced;  following end of new file
263596
MX endpoint opened
startup file interpreter thread tid=140334118615312
calling yyparse(5, 6)
[Mon Aug  6 19:50:08 2012] ->5: #set avoid_reconnect
[Mon Aug  6 19:50:08 2012] ->5: set thread_stack_size=102400
[Mon Aug  6 19:50:08 2012] new threads will be created with the stack of size 102400K
[Mon Aug  6 19:50:08 2012] ->5: set allow_tpman_connect_fail
[Mon Aug  6 19:50:08 2012] ->5: #set dcu_status_check=5
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=-1
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=31535998
[Mon Aug  6 19:50:08 2012] ->5: ##set symm_gps_offset=347155213
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691215
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=378691212
[Mon Aug  6 19:50:08 2012] ->5: #set symm_gps_offset=315964799
[Mon Aug  6 19:50:08 2012] ->5: set symm_gps_offset=315964801
[Mon Aug  6 19:50:08 2012] ->5: set debug=0
[Mon Aug  6 19:50:08 2012] ->5: set log=2
[Mon Aug  6 19:50:08 2012] ->5: set zero_bad_data=0
[Mon Aug  6 19:50:08 2012] ->5: set dcu_status_check=9
[Mon Aug  6 19:50:08 2012] ->5: set controller_dcu=33
[Mon Aug  6 19:50:08 2012] ->5: set master_config="/opt/rtcds/caltech/c1/target/fb/master"
[Mon Aug  6 19:50:10 2012] finished configuring data channels
[Mon Aug  6 19:50:10 2012] ->5: configure channels begin end
Unable to find GDS node 90 system c1x00 in INI files
Unable to find GDS node 92 system c1tst2 in INI files
Unable to find GDS node 95 system c1x10 in INI files
[Mon Aug  6 19:50:10 2012] ->5: tpconfig "/opt/rtcds/caltech/c1/target/gds/param/testpoint.par"
[Mon Aug  6 19:50:10 2012] ->5: set gps_leaps = 820108813
[Mon Aug  6 19:50:10 2012] ->5: set detector_name="CIT"
[Mon Aug  6 19:50:10 2012] ->5: set detector_prefix="C1"
[Mon Aug  6 19:50:10 2012] ->5: set detector_longitude=-90.7742403889
[Mon Aug  6 19:50:10 2012] ->5: set detector_latitude=30.5628943337
[Mon Aug  6 19:50:10 2012] ->5: set detector_elevation=.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_azimuths=1.1,4.7123889804
[Mon Aug  6 19:50:10 2012] ->5: set detector_altitudes=1.0,2.0
[Mon Aug  6 19:50:10 2012] ->5: set detector_midpoints=2000.0, 2000.0
[Mon Aug  6 19:50:10 2012] ->5: set num_dirs = 10
[Mon Aug  6 19:50:10 2012] ->5: set frames_per_dir=225
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_per_file=1
[Mon Aug  6 19:50:10 2012] ->5: set full_frames_blocks_per_frame=16
[Mon Aug  6 19:50:10 2012] ->5: set frame_dir="/frames/full", "C-R-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set trend_num_dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set trend_frames_per_dir=1440
[Mon Aug  6 19:50:10 2012] ->5: set trend_frame_dir= "/frames/trend/second", "C-T-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: set raw-minute-trend-dir="/frames/trend/minute_raw"
[Mon Aug  6 19:50:10 2012] ->5: set nds-jobs-dir="/opt/rtcds/caltech/c1/target/fb"
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-num-dirs=10
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frames-per-dir=24
[Mon Aug  6 19:50:10 2012] ->5: set minute-trend-frame-dir="/frames/trend/minute", "C-M-", ".gwf"
[Mon Aug  6 19:50:10 2012] ->5: start main 10
[Mon Aug  6 19:50:12 2012] main started
[Mon Aug  6 19:50:12 2012] ->5: start profiler
[Mon Aug  6 19:50:12 2012] ->5: # comment out this block to stop saving data
[Mon Aug  6 19:50:12 2012] frame saver started
[Mon Aug  6 19:50:12 2012] ->5: start frame-saver
[Mon Aug  6 19:50:13 2012] ->5: sync frame-saver
[Mon Aug  6 19:50:13 2012] ->5: start trender
[Mon Aug  6 19:50:13 2012] trender started
[Mon Aug  6 19:50:13 2012] trend frame saver started
[Mon Aug  6 19:50:13 2012] ->5: start trend-frame-saver
[Mon Aug  6 19:50:14 2012] ->5: sync trend-frame-saver
[Mon Aug  6 19:50:14 2012] minute trend frame saver started
[Mon Aug  6 19:50:14 2012] ->5: start minute-trend-frame-saver
[Mon Aug  6 19:50:14 2012] Done creating ADC structures
[Mon Aug  6 19:50:15 2012] ->5: sync minute-trend-frame-saver
[Mon Aug  6 19:50:15 2012] raw minute trend frame saver started
[Mon Aug  6 19:50:15 2012] ->5: start raw_minute_trend_saver
[Mon Aug  6 19:50:15 2012] ->5: #frame-writer "225.225.225.1" broadcast="131.215.113.0" all
[Mon Aug  6 19:50:15 2012] ->5: #sleep 5
[Mon Aug  6 19:50:15 2012] producer started
[Mon Aug  6 19:50:15 2012] ->5: start producer
[Mon Aug  6 19:50:15 2012] ->5: start epics dcu
[Mon Aug  6 19:50:15 2012] MX receiver thread started
[Mon Aug  6 19:50:15 2012] edcu started
[Mon Aug  6 19:50:15 2012] ->5: start epics server "C0:DAQ-DC0_" "C1:DAQ-DC0_"
[Mon Aug  6 19:50:15 2012] epics server started
[Mon Aug  6 19:50:15 2012] ->5: start listener 8087
[Mon Aug  6 19:50:15 2012] ->5: start listener 8088 1
[Mon Aug  6 19:50:15 2012] ->5: sleep 60
[Mon Aug  6 19:50:15 2012] Epics server started
[Mon Aug  6 19:50:15 2012] EDCU has 2553 channels configured; first=0

[Mon Aug  6 19:50:18 2012] Minute trender made GPS time correction; gps=1028343032; gps%60=32
...

The "tail:..." line indicates that the log was moved and replaced, which indicates a daqd restart.  As far as I know this was not manually triggered.

After the restart the same thing happens again.  About once every five minutes.

When daqd is caught in this state it can not be killed.  It's in "uninterruptable sleep" ('D' state in the top output below).  This usually indicates that it's waiting for the kernel, usually due to some missing or hung IO.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
28038 controls  20   0 4430m 2.0g  19m D    0 27.1   0:15.00 daqd                                                                                         

The memory footprint also seems to be getting big.  It's clearly trying to do something stupid that it can't handle.

Background:
  Before Joe left(~ 1 hour ago), fb was working for a while. But after he left, daqd core dumped.
  This is maybe because we started c1sus and c1rms again for a delay measurement, just before he left.

What I did:
  I restarted IOP(c1x02) and FE models.
  Now it seems OK (we can use dataviewer and diaggui), but daqd reports bunch of errors like;

CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:SUS-ITMX_TO_COIL_0_3_INMON", Connecting to: 192.168.113.85:42367, Ignored: c1susdaq:42367"
    Source File: ../cac.cpp line 1208
    Current Time: Fri Oct 29 2010 18:07:39.132686519
..................................................................

tp node xx invalid  (xx is 38 to 36)

Current CDS status:

   Please add other stuff you need.

Below is an example of how the color code works:

dadq is logging the following error messages to it's log related to the fact that it can't connect to c1vac1 and c1vac2:

CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac2.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.355609430
..................................................................
CAC: Unable to connect because "Connection timed out"
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "c1vac1.martian:5064"
    Source File: ../cac.cpp line 1127
    Current Time: Tue Jan 07 2014 23:50:53.356568469
..................................................................

 Not sure if this is related to the full /frames issue that we've been seeing.

For some reason, the daqd_fw service was dead on FB. This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 . Simply restarting the fw service does not work, it crashes again after ~20 seconds. The problem may have to do with the indeterminate state of the c1lsc expansion chassis. However, this is not something that can immediately be fixed, as Chub is still working on the wiring there. So in summary, no frame data will be available until we fix this problem (it is still unclear what exactly the problem is). Team WFS can still work by getting online data.

Why were the CDS overview DC indicators not red???

Unrelated to this work: I had to key the c1psl crate to get the IMC autolocker functioning again. However, I found that the key 🔑 turns continuously - as opposed to having two well defined states, ON and OFF. Be careful while handling this.