40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 317 of 344  Not logged in ELOG logo
ID Date Author Type Category Subject
  1386   Wed Mar 11 14:51:01 2009 Kakeru, Joe, RobUpdateIOOMC alignment

This morning, MC alignment was gone and MC wasn't lock.

We checked old value of pitch, yaw, and position offset of each MC mirror, and found they were jumped.

We don't know the reason of this jump, but we restore each offset value and MC backed to lock.

  1385   Wed Mar 11 11:30:15 2009 josephbConfigurationCameras 

I modified the Video.db file used by c1aux located in  /cvs/cds/caltech/target/c1aux.

I added the following channels to the db file, intended for either read in or read out by the digital camera scripts.

C1:VID-ETMY_X_COM
C1:VID-ETMY_Y_COM

C1:VID-ETMY_X_STDEV

C1:VID-ETMY_Y_STDEV

C1:VID-ETMY_XY_COVAR

C1:VID-ETMY_EXPOSURE

C1:VID-ETMY_GAIN

C1:VID-ETMY_X_UL

C1:VID-ETMY_Y_UL

C1:VID_ETMY_X_SIZE

C1:VID_ETMY_Y_SIZE

A better naming scheme can probably be devised, but these will do for now.

  1384   Wed Mar 11 04:33:57 2009 ranaConfigurationComputer Scripts / Programswild ndsproxy tclshexe

The ndsproxy tcl task on nodus was eating up all the CPU and making the elog slow. I killed it and restarted it.

It looks like it hasn't been making a log file since January. Someone who has some skill in decoding the cryptic csh stdout redirection

syntax should look into this (its in target/ndsproxy/)

  1383   Wed Mar 11 01:16:40 2009 ranaSummaryIOOrogue trianglewave in the MC Servo offset slider

On Monday evening, I ran this command: trianglewave C1:IOO-MC_REFL_OFFSET 0 4 120 600;ezcawrite C1:IOO-MC_REFL_OFFSET 1.76

which I thought (from the syntax help) would move that offset slider with a period of 120 seconds for 600 seconds. In actuality, the last argument is the

run time in number of periods. So the offset slider has been changing by 8 Vpp for most of the last day. Oops. The attached image shows what effect

this had in the MC transmitted power (not negligible). This would also make the locking pretty difficult.

 

In the second plot you can see the zoom in view for ~30 minutes. During the first part, the MCWFS are on and there are large fluctuations

in the transmitted power as the WFS offset changes. This implies that the large TEM00 carrier offset we induce with the slider couples into

the WFS signals because of imbalances in the quadrant gains - we need someone to balance the RF gains in the WFS quadrants by injecting

an AM laser signal and adjusting the digital gains.

 

Since there is still a modulation of the MC RFPD DC with the WFS on, we can use this to optimize the REFL OFFSET slider. The third plot

shows a 8 minute second trend of this. Looks like the slider offset of zero would be pretty good.

  1382   Tue Mar 10 04:55:41 2009 YoichiUpdateLockingLocking: 3.7kHz large oscillation
Yoichi, Jenne, Alberto,

As I reported on the last Thursday, there is a large oscillation in CARM and DARM error signals (attm1).
I put notch filters (3.75kHz, Q=10, 30dB) in the CARM and DARM loops. This let us go up to the arm power of more than 20 and stay there for a while.
The dashed curves in the attm1 are the spectra when the notches are off, and the solid curves are when the notches are used.
We could somewhat suppress the DARM peaks but not CARM.
Of course this is clearly not a good solution. We should find the cause of the oscillation and kill it.

Attm2 is the spectrum of the PO_DC signal flowing in the CM board measured by the SR785. More specifically, CH1 is TP1A and CH2 is TP2A of the CM board.
This was taken right after the AO path was engaged. At this stage, the AO path gain is very low. But you can already see a seed of the oscillation in the spectrum.

Attm3 shows the same spectra taken after the arm power is increased to 4 but before the PO_DC hand off. You can see large peaks around 3.75kHz.
After this, the peaks grow as the power goes up.

Attm4 is the loop gain of the AO path after the PO_DC hand off (arm power = 4).
Attm5 is the zoom of the same TF around 3.7kHz. Clearly there is something wrong at this frequency. We should check the CM board and the MC board as well as the SPOB PD.

One time I was able to go up to arm power = 27 or so. At this power level, the DARM loop started to oscillate, probably, around the UGF.
However because of the 3.7kHz problem, we can't stay at this power level long enough to make diagnostic measurements (like open loop TF).
We should tackle the 3.7kHz issue first.
  1381   Mon Mar 9 23:55:38 2009 OsamuDAQComputersbscteststand and kami1 outside martian

This morning there was a confliction of tpman running on fb40m and kami1. Alex fixed it temporary but Rana suggested it was better to move both PCs outside martian. We moved both PCs physically to the control room and connected to general network with a local router. I believe it won't conflict anymore but if you guess these PC might have trouble please feel free to shutdown.

 

Today's work summary:

 *connected expansion chassis to bscteststand

 *obtained signals on dataviewer, dtt for both realtime and past data on bscteststand with 64kHz timing signal

 

Questions:

Excitation channels are not shown, only "other" is shown.

qts.mdl should run with 16kHz but 16kHz timing causes a slow speed on dataviewer and failing data aquisition on dtt. We are using 64kHz timing but is it really correct?

  1380   Mon Mar 9 23:13:22 2009 YoichiUpdateComputer Scripts / Programstdsdata doesn't work

Quote:

We confirmed that new tds(/cvs/cds/caltech/apps/linux/tds_090304/) works well on linux 64, and replaced it to /cvs/cds/caltech/apps/linux/tds/

The old /cvs/cds/caltech/apps/linux/tds is put in /cvs/cds/caltech/apps/linux/tds.bak

 The tdscntr.pl in the new tds was probably the one from LLO, which is actually the version I sent to Tobin. It had paths and channel names defined for the LLO. So I copied back my original 40m version.

  1379   Mon Mar 9 19:33:10 2009 ranaUpdateDMFseisBLRMS in temp condition

The seisBLRMS has been running on megatron via an open terminal ssh'd into there from allegra with matlab running. This

is because I couldn't get the compiled matlab functionality to work.

Even so, this running script has been dying lately because of some bogus 'NDS' error. So for today I

have set the NDS server for mDV on megatron to be fb40m:8088 instead of nodus.ligo.caltech.edu. If this seems to fix the problem

I will make this permanent by putting in a case statement to check whether or not the mDV'ing machine is a 40m-martian or not.

  1378   Mon Mar 9 19:27:16 2009 ranaConfigurationComputersMove of the CLIO Digital Controls test setup

Because of the network interference we've had from the CLIO system for the past 3-4 days, I asked the guys to remove

the test stand from the 40m lab area. It is now in the 40m control room. Since it needed an ethernet connection to get out

for some reason we've let them hook into GC. Also, instead of using a real timing signal slaved to the GPS, Jay suggested

just skipping it and having the Timing Slave talk to itself by looping back the fiber with the timing signal. Osamu will enter

more details, but this is just to give a status update.

  1377   Mon Mar 9 17:11:38 2009 AlbertoConfigurationoplevsoptical levers centering

Yoichi, Alberto

this afternoon we centered the optical levers for all the optics.

To do that we first ran the alignment scripts for all the cavities.

  1376   Mon Mar 9 16:54:52 2009 Kakeru, RanaUpdateComputer Scripts / Programstdsdata doesn't work

We confirmed that new tds(/cvs/cds/caltech/apps/linux/tds_090304/) works well on linux 64, and replaced it to /cvs/cds/caltech/apps/linux/tds/

The old /cvs/cds/caltech/apps/linux/tds is put in /cvs/cds/caltech/apps/linux/tds.bak

  1375   Mon Mar 9 14:57:30 2009 KakeruUpdateComputer Scripts / Programstdsdata doesn't work

I tested new tdsdata and found it was working well.

I excited C1:SUS-ITMY_SUSPIT_EXC with tdssine, and get data from C1:LSC-TRY_OUT (testpoint) and C1:SUS- ITMY_OPLEV_PERROR (recorded point) with new and old tdsdata.

With old tdsdata (/cvs/cds/caltech/apps/linux/tds/bin/tdsdata), I found some jumps of datapoint, which is a same problem with before (Attachment 1).

With new tdsdata (/cvs/cds/caltech/apps/linux/tds_090304/bin/tdsdata), there looks to be no jumps (Attachment 2; taken about 10 minutes after Attachment 1).

The problem of old tdsdata looks to be remaining even for recordedpoints.

You should use /cvs/cds/caltech/apps/linux/tds_090304/bin/tdsdata.

Quote:

Matt logged in and rebuilt the TDS stuff for us on Mafalda in /cvs/cds/caltech/apps/linux/tds_090304.

He says that he can't build his stuff on 64-bit because there's not a sanctioned 64-bit build of GDS yet.

This should have all the latest fixes in it. I tried using both the old and new code from allegra and they both are fine:

./tdsdata 16384 2 C1:IOO-MC_F > /users/rana/test.txt

I loaded the data I got with the above command and there were no data dropouts. Possibly the dropout problem is only

associated with testpoints and so we have to wait for the TP fix.

 

  1374   Mon Mar 9 12:04:18 2009 YoichiUpdateComputersTPs and AWG are back
I had to do one more reboot of tpman and daqd to get the TPs working.
I confirmed the alignment scripts run fine.

Now the oplevs of some optics are largely mis-centered. Alberto and I will center them after lunch.
  1373   Mon Mar 9 11:09:33 2009 AlbertoUpdateComputersRe: Not even data retrieval working

Quote:
Although getting the regular DAQ data works, we can't get any testpoints.

I tried restarting tpman several times; there's no inittab on fb40m for this so we should get Alex to set one up when he comes.
I also tried various power cycles and reboots: daqawg, daqctrl, etc. I also notice that Osamu's setup of new stuff is connected to
the same rack and power strips as all of our sensitive DAQ machines. We should find out if there was any hardware installed in the
last couple days; it would be easy to accidentally unplug or damage on of our fibers.

I moved the old tpman.log over to tpman.log.090308. It starts out with a header and then just lists when each TP is requested.

When restarting tpman it puts the following into the terminal:
fb:controls>./tpman &
[1] 1037
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0
which is OK?; its the same startup outputs that are in the old log file. It would be nice if there was not and error message about the RFM.
Requesting new testpoints via tdsdata, dtt, or the diag command line doesn't seem to work. tpman doesn't spit anything out although 'tp show 0'
does show that the TP is selected.

Once Alex fixes the 'tpman' issue, we should make sure to put an inittab or startup script in there so that tpman writes a log
file and also archives its old log files upon a restart.


Alex fixed the problem. It was caused by the awgtpman running on kami1.martian which conflicted with the tpman in fb0.

Killing awgtpman on kami1 allowed for the tpman on tp0 to work properly again.

If more test points are needed, Alex suggested to tune the GDS settings accordingly.
What this actually means, I still have to understand it.
  1372   Mon Mar 9 10:59:05 2009 AlanOmnistructureComputersssh agent on fb40m restarted for backup

After the boot-fest, the nightly backup to Powell-Booth failed, and an automatic email got sent to me. I restarted the ssh agent, following the instructions in /cvs/cds/caltech/scripts/backup/000README.txt .

  1371   Sun Mar 8 23:14:52 2009 ranaUpdateComputer Scripts / Programstdsdata doesn't work

Matt logged in and rebuilt the TDS stuff for us on Mafalda in /cvs/cds/caltech/apps/linux/tds_090304.

He says that he can't build his stuff on 64-bit because there's not a sanctioned 64-bit build of GDS yet.

This should have all the latest fixes in it. I tried using both the old and new code from allegra and they both are fine:

./tdsdata 16384 2 C1:IOO-MC_F > /users/rana/test.txt

I loaded the data I got with the above command and there were no data dropouts. Possibly the dropout problem is only

associated with testpoints and so we have to wait for the TP fix.

  1370   Sun Mar 8 23:09:26 2009 ranaUpdateComputersNot even data retrieval working
Although getting the regular DAQ data works, we can't get any testpoints.

I tried restarting tpman several times; there's no inittab on fb40m for this so we should get Alex to set one up when he comes.
I also tried various power cycles and reboots: daqawg, daqctrl, etc. I also notice that Osamu's setup of new stuff is connected to
the same rack and power strips as all of our sensitive DAQ machines. We should find out if there was any hardware installed in the
last couple days; it would be easy to accidentally unplug or damage on of our fibers.

I moved the old tpman.log over to tpman.log.090308. It starts out with a header and then just lists when each TP is requested.

When restarting tpman it puts the following into the terminal:
fb:controls>./tpman &
[1] 1037
fb:controls>VMIC RFM 5565 (0) found, mapped at 0x2868c90
VMIC RFM 5579 (1) found, mapped at 0x2868c90
Could not open 5565 reflective memory in /dev/daqd-rfm1
16 kHz system
Spawn testpoint manager
Channel list length for node 0 is 4168
Test point manager (31001001 / 1): node 0
which is OK?; its the same startup outputs that are in the old log file. It would be nice if there was not and error message about the RFM.
Requesting new testpoints via tdsdata, dtt, or the diag command line doesn't seem to work. tpman doesn't spit anything out although 'tp show 0'
does show that the TP is selected.

Once Alex fixes the 'tpman' issue, we should make sure to put an inittab or startup script in there so that tpman writes a log
file and also archives its old log files upon a restart.
  1369   Sat Mar 7 16:50:25 2009 YoichiUpdateComputersNot even data retrieval working
Now our digital system is really in trouble.
We can't even get data from tp channels.

I did another round of computer reboots, this time including the RFM bypass switch, c0daqctrl, c0dcu1 and fb40m itself.
But the problem still persists.

I guess there is nothing I can do until Alex comes in.
  1368   Fri Mar 6 18:26:37 2009 YoichiConfigurationComputersezca tools and tds tools work around
I updated the wrapper scripts so that they do not retry more than 6 times.
Otherwise, the wrapper scripts loop over infinitely when you give wrong arguments.



Quote:
Some of ezca commands and tds commands sporadically fail with a segmentation fault on linux machines.
As far as I know, ezcawrite, ezcastep, ezcaswitch, and tdswrite have this problem.
These are commands to write values into epics channels. So usually people do not check the exit status of those commands in their scripts.
This could cause incomplete execution of, for example, down scripts.
Ideally, this problem should be fixed in the source codes of the problematic commands.
However, I don't have a patience to wait it to happen, and I needed to fix these problems immediately for the lock acquisition.
So I resorted to a hacky solution.
I renamed those commands to *.bin, e.g. ezcawrite -> ezcawrite.bin.
Then wrote wrapper scripts to repeatedly call those commands until it succeeds.
For example, ezcawrite now looks like,
#!/bin/csh -f
setenv POSIXLY_CORRECT
while (! { ezcawrite.bin $* })
      echo "Retry $0 $*"
end
So, when ezcawrite.bin fails, the command retries it and show a message "Retry ....".

If you need to call the original commands, you can always do so by adding ".bin" at the end of the command name.
Currently the following commands are wrapped.
ezcawrite, ezcaservo, ezcastep, ezcaswitch, tdswrite, tdssine.

Please let me know if you have any trouble with this.
  1367   Fri Mar 6 18:22:42 2009 YoichiSummaryComputersScripts to restart the FE computers
While doing locking, the FE computers are overloaded sometimes and I have to reboot them.
Being sick of logging into the FE computers one by one to start front end codes, I wrote scripts to do this automatically.
The scripts are in /cvs/cds/caltech/scripts/FE/.
For example, you can restart c1lsc by typing
restartFE c1lsc
You can give multiple computer names to the restartFE command like,
restartFE c1lsc c1asc c1susvme1

To restart all the FE computers, type
restartFE all

For the scripts to work properly, the computers have to accept login, i.e. you either have to power cycle the computers or push "Reset" buttons on the RFMNETWORK medm screen prior to running the scripts.
  1366   Fri Mar 6 18:14:58 2009 YoichiUpdateComputersawg not working
Starting from this afternoon, the awg is not working.
I rebooted FE computers, c0daqawg as well as tpman and daqd processes on fb40m several times.
But the problem is still there.
I sent an email to Alex.
  1365   Fri Mar 6 15:23:39 2009 YoichiUpdateLockingLocking distracted by the QPD whitenning problem again
By looking at the time series of DARM signal at the time of a lock loss, the oscillation frequency was about 3.5kHz (see the attm1 and its zoomed version attm2).
I will measure the DARM loop gain around this frequency next.


Quote:
Tonight, I was able to ramp up the arm power to around 20. Then the DARM loop started to oscillate and the IFO lost lock in a few seconds.
I repeated this several times, then realized that the transmission QPDs were not working properly again due to the well known sticky slider problem.
I should have run slider_twiddle script. Since the DARM RF signal is normalized by the sqrt(TRX+TRY), it is reasonable that the DARM loop got unstable.

The fact that I was able to go up to arm power = 20 means there is nothing saturating below this power level.
  1364   Fri Mar 6 05:44:14 2009 YoichiUpdateLockingLocking distracted by the QPD whitenning problem again
Tonight, I was able to ramp up the arm power to around 20. Then the DARM loop started to oscillate and the IFO lost lock in a few seconds.
I repeated this several times, then realized that the transmission QPDs were not working properly again due to the well known sticky slider problem.
I should have run slider_twiddle script. Since the DARM RF signal is normalized by the sqrt(TRX+TRY), it is reasonable that the DARM loop got unstable.

The fact that I was able to go up to arm power = 20 means there is nothing saturating below this power level.
  1363   Fri Mar 6 01:04:49 2009 Kiwamu IZUMIConfigurationIOO!! lock-in amp disconnected !!

The power supply of a lock-in amp, which is on the Y-arm side of PSL clean room, was pulled out by my mistake.

Then I reconnected it, but I don't know whether it is re-adjusted properly.

I'm sorry about this. If you are using that amp, it should be checked.

  1362   Thu Mar 5 23:18:38 2009 KakeruConfigurationComputerstdsdata doesn't work
I found that tdsdata doesn't work.

When I star tdsdata, he takes a few ~ 10 seconds of data, and he dies with a message "Segmentation fault".
I tried to get data for some times and some channels, and this problem was observed everytime.
I also tried tdsdata on allegra, op440m and mafalda, and it didn't work on all of them.

Yesterday, I got a new version of tdsdata (which modified the problem of Message ID: 1328) and tried to build 
thme on my directory (/cvs/cds/caltech/users/kakeru.....)
This may have some relation to this problem.
  1361   Thu Mar 5 05:07:09 2009 YoichiUpdateLockingWednesday night locking
Tonight, I was having a problem with the PO_DC hand-off.
It fails most of the time.
I increased the averaging time for the PD1_DC offset measurement.
I also wrote a script to match the gain of the transmission DC and the PO_DC signals.
This script (/cvs/cds/caltech/scripts/CM/matchPODCGain ) measures the gains of the old (TRX+TRY) and new (PO_DC) signals at 150Hz and returns the optimal value to be put into the input matrix.
cm_step script calls matchPODCGain to determine the matrix element value for the PO_DC signal.

Even with this script, the hand-off was still unreliable.
I checked the AO path loop gain just before the hand off. It looked normal.
Then I realized that the oscilloscope I hooked up to the PO_DC signal using a T-BNC may be introducing some noise into the channel.
So I removed it. Then the PO_DC hand off went well at least once.
The IFO still loses lock at around arm power 10.

I attached time series of the latest lock loss. The second attachment is a zoom of the first one.
This time, there is a glitch in the ETM feedback signals, which is also present in the DARM and CARM and error signals.
I saw this kind of glitches several times today.
  1360   Thu Mar 5 02:24:19 2009 ranaConfigurationComputersyum.repos.d

I added the following repos which I found on allegra to megatron and then did a 'yum install sshfs' on both machines:

allegra:yum.repos.d>l
total 28
-rw-r--r-- 1 root root  428 Feb 12 16:47 rpmforge.repo
-rw-r--r-- 1 root root  684 Feb 12 16:47 mirrors-rpmforge
-rw-r--r-- 1 root root 1054 Feb 12 16:47 epel-testing.repo
-rw-r--r-- 1 root root  954 Feb 12 16:47 epel.repo
-rw-r--r-- 1 root root  626 Feb 12 16:47 CentOS-Media.repo
-rw-r--r-- 1 root root 1869 Feb 12 16:47 CentOS-Base.repo
-rw-r--r-- 1 root root  179 Feb 12 16:47 adobe-linux-i386.repo

This also required me to import the rpmforge GPG key:

sudo rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt

  1359   Thu Mar 5 01:09:29 2009 rana, albertoUpdateDMFstill not working
We tried to run DMF on mafalda, but it didn't work. I tried to compile it using Rob's elog instructions.
On mafalda, I started matlab and ran the mdv_config.m to set up mDV. I tested that the seisBLRMS.m
script ran and correctly produced changes in the seisBLRMS strip tool. but when I tried to compile it I got:
>> mcc -v -m -R -nojvm seisBLRMS.m
Warning: Duplicate directory name: /cvs/cds/caltech/apps/linux/matlab/toolbox/local.
Compiler version: 4.6 (R2007a)
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/matlab/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/signal/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/control/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/filterdesign/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/controllib/mcc.enc
Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/ident/mcc.enc
Warning: an error occurred while parsing class FilterDesignDialog.AbstractEditor:
Undefined function or variable 'DAStudio.Object'.

> In /cvs/cds/caltech/apps/linux/matlab/toolbox/shared/filterdesignlib/@FilterDesignDialog/@CoeffEditor/schema.p>schema at 9
Warning: an error occurred while parsing class FilterDesignDialog.CoeffEditor:
Invalid superclass handle.

Processing /cvs/cds/caltech/apps/linux/matlab/toolbox/fixedpoint/mcc.enc
terminate called after throwing an instance of 'ApplicationRedefinedException*'
Abort (core dumped)
"/cvs/cds/caltech/apps/linux/matlab/bin/mcc" -E "/tmp/fileRnU5Qj_31324": Aborted
??? Error executing mcc, return status = 134.

In the meantime, I've started up a green terminal on allegra which is ssh'd into megatron.
On megatron there is a regular matlab session which is running seisBLRMS.m as a matlab script
and the seis DMF channels are getting updated.
  1358   Thu Mar 5 00:06:32 2009 Kakeru, RanaUpdateIOOWFS centering
We found that the MC REFL image was no longer round and that the MCWFS DC quadrant spots were mostly
in one quadrant. So we re-centered the MCWFS beams in the following way:

1) We unlocked the MZ and adjusted the PZT voltage to keep the beam on the WFS from saturating.
2) Re-aligned the black hole beam dump to center its beam in its aperture.
3) centered the beam on the MCWFS optics and MCWFS QPD displays.
4) Relocked MC.

Below is the image of the IOO Strip tool. You can see that the MC REFL DC is now more flat. The
MC pointing has also been changed (see the MC TRANS HOR & VERT channels). The MC transmitted
light is also now more stable and higher.

We tried to center the QPD, and we found that there were a few hundred mV of dark offset for each 
quadrant of QPD. We adjusted them with this scripts:
/cvs/cds/caltech/scripts/MC/WFS/McWFS_dc_offsets
  1357   Wed Mar 4 23:42:45 2009 ranaConfigurationPSLpsl db change

I made the following change to correct the sign of the 126MON channel:

allegra:c1aux>ezcawrite C1:PSL-126MOPA_126MON.EGUF -410
C1:PSL-126MOPA_126MON.EGUF = -410
allegra:c1aux>ezcawrite C1:PSL-126MOPA_126MON.EGUL 410
C1:PSL-126MOPA_126MON.EGUL = 410
allegra:c1aux>

  1356   Wed Mar 4 17:59:14 2009 YoichiConfigurationComputersezca tools and tds tools work around
Some of ezca commands and tds commands sporadically fail with a segmentation fault on linux machines.
As far as I know, ezcawrite, ezcastep, ezcaswitch, and tdswrite have this problem.
These are commands to write values into epics channels. So usually people do not check the exit status of those commands in their scripts.
This could cause incomplete execution of, for example, down scripts.
Ideally, this problem should be fixed in the source codes of the problematic commands.
However, I don't have a patience to wait it to happen, and I needed to fix these problems immediately for the lock acquisition.
So I resorted to a hacky solution.
I renamed those commands to *.bin, e.g. ezcawrite -> ezcawrite.bin.
Then wrote wrapper scripts to repeatedly call those commands until it succeeds.
For example, ezcawrite now looks like,
#!/bin/csh -f
setenv POSIXLY_CORRECT
while (! { ezcawrite.bin $* })
      echo "Retry $0 $*"
end
So, when ezcawrite.bin fails, the command retries it and show a message "Retry ....".

If you need to call the original commands, you can always do so by adding ".bin" at the end of the command name.
Currently the following commands are wrapped.
ezcawrite, ezcaservo, ezcastep, ezcaswitch, tdswrite, tdssine.

Please let me know if you have any trouble with this.
  1355   Wed Mar 4 17:20:04 2009 josephbUpdateCamerasCamera code upgrades

I've updated the digital camera python code as well as changed the network topology.

At the moment, both cameras are connected to a small gigabit switch which only talks to Ottavia.  This means all camera servers must be run on Ottavia, allow camera output is still UDP multicast so any machine capable of running gstreamer can pick up the images.

The server and client programs now have the ability to read a configuration file for the setup of the cameras.  They default to pcameraSettings.ini, but this can default can be changed with a -c or --config option

For example, "serverV3.py --config pcam1.ini" will run the server using the pcam1.ini settings file.  Similarly, "client.py --config pcam1.ini" will also take the IP settings from the config file so that it knows at which port and IP to listen.

These programs and .ini files have been placed in /cvs/cds/caltech/apps/linux64/python/pcamera/

I've updated the cshrc.40m aliases so that it uses the new configuration file options, so now pcam1 calls "client.py -c pcam1.ini" in the above directory.

So to start a client use pcam1 or pcam2 (for the 32223 camera in PSL looking at MC trans or 44026 looking at an analog moniter in the control room respectively).  These can be run on Allegra, Rosalba or Ottavia at the moment.

To start a server, use pserv1 or pserv2.  These *must* be run on Ottavia.

I've also added a -n or --no-gui option at Yoichi's request, one which just starts up and plays, with no graphical gui.

Lastly, I've made some changes to the base pcamerasrc.py file, which should make display more robust.  After a failed transmission of an image from the camera to Ottavia, it should re-attempt up to 10 times before giving up. I'm hoping this will make it more robust against packet loss.  The change in network topology has also helped this, allowing 640x480 to be transmitted on both cameras before tens of minutes before a packet loss causes a stop.

  1354   Wed Mar 4 12:38:07 2009 AlbertoUpdateComputer Scripts / Programs3f locking simulations
I simulated the REFL signals demodulated at the differential frequencies of the sidebands (f2-f1), at their summed frequencies (f2+f1). I also simulated their combination as in the Double Demodulation.
 
I repeated the simulation for:
- Old (current) 40m
- 40m Upgrade
- AdvLIGO
 
I'm attaching the results to this elog entry.
 
The plots show how the signal varies exploring the two-dimensional space of the demodulation frequencies (differential and sum).

 Both the Upgrade and the Old40m's signals look anomalous since the zero-crossing point does not change with the demodulation phases.

I suspect there's is a problem with the optickle model of the 40m.

  1353   Wed Mar 4 03:50:17 2009 YoichiUpdateLockingStill won't go above arm power 10
Yoichi, Kentaro

The IFO still loses lock at around arm power 10.
I attached time series of various error signals when losing lock.
In all of the three cases, both of the arm powers go up rapidly just before losing lock.
(The first attachment was taken before I fixed the QPDX switching, so you can see saturation in TRX.)
But PD1_DC (the DC power in the PRC) did not go up in the third case.

I should also check correlations with laser power, CARM length (OSEM signals), seismic noise etc.
  1352   Wed Mar 4 03:37:51 2009 YoichiUpdateLockingLow-gain High-gain PD switching may not be working well

Quote:

This sounds like the QPD whitening gain sliders may be stuck. The slider twiddling script should be run, or the sliders should be twiddled by hand.


Yes, it was indeed the whitening gain slider problem.
I moved them and the QPDX gain went up suddenly. I reinstalled the BS in front of the QPDX and adjusted the offsets, gains accordingly.
Now the high-gain to low-gain switching is fine.
  1351   Tue Mar 3 23:59:26 2009 YoichiUpdateLockingLow-gain High-gain PD switching may not be working well

Quote:
I checked the switching of the QPDX from high gain to low gain.
Switching happens as expected, but the low gain QPDX output was very low compared to QPDY.
Also the digital gain for the high gain TRX was not matched with the low gain one. So when the switching happens, there is a large jump in the TRX.

I also found that the offset values for the low gain QPDX were totally wrong. I adjusted it.
Then I removed a beam splitter in front of the QPDX to increase the power falling on it.
But still the low gain QPDX output is four times lower than the low gain QPDY.

I'm still working on it. So don't expect the switching to work correctly at this moment.
I'm planning to be back after the dinner.



This sounds like the QPD whitening gain sliders may be stuck. The slider twiddling script should be run, or the sliders should be twiddled by hand.
  1350   Tue Mar 3 19:26:44 2009 YoichiUpdateLockingLow-gain High-gain PD switching may not be working well
I checked the switching of the QPDX from high gain to low gain.
Switching happens as expected, but the low gain QPDX output was very low compared to QPDY.
Also the digital gain for the high gain TRX was not matched with the low gain one. So when the switching happens, there is a large jump in the TRX.

I also found that the offset values for the low gain QPDX were totally wrong. I adjusted it.
Then I removed a beam splitter in front of the QPDX to increase the power falling on it.
But still the low gain QPDX output is four times lower than the low gain QPDY.

I'm still working on it. So don't expect the switching to work correctly at this moment.
I'm planning to be back after the dinner.


Quote:
Osamu, Yoichi

This afternoon, I run the locking script while doing calculations for the upgrade.
The IFO lost lock even at lower arm powers (around 6) if it was operated for a while (~ 5min).
It seemed as if there were some intermittent glitches (seismic? laser?) causing the lock losses.
We also saw once the TRX and TRY signals saturated at around arm power = 11 when there was a large fluctuation in the arm power.
Osamu suggested that it looked like the high-gain to low-gain PD switching was not working.

I won't come tonight as I may have caught a cold, but if someone comes tonight, it is worth checking the PD switching.
  1349   Tue Mar 3 11:39:50 2009 OsamuDAQComputers2 PCs in Martian

 Kiwamu and I brought 2 SUPER MICRO PCs from Willson house into 40m.

Both PCs are hooked up into Martian network. One is named as bscteststand for BSC which has been set up by Cds people and another one is named kami1 for temporary use for CLIO which is a bland new, no operating installed PC. This bland new PC will be returned Cds or 40m once another new PC which we will order within several days arrives.

IP address for each machine is 131.215.113.83 and 131.215.113.84 respectively.

We have installed CentOS5.2 into the new PC.

  1348   Tue Mar 3 10:39:07 2009 AlbertoUpdateLSCc1lsc discontinued functioning

The c1lsc has been unstable since last night. Its status on the DAQ screen was oscillating from green to red every minute.

Yesterday, I power recycled it. That brought it back but the MC got unclocked and the autolocker could not get engaged. I think it's because the power recycling also turned c1iscaux2 off which occupies the same rack crate.

Killing the autolocker on op340 e restarting didn't work. So I rebooted also c1dcuepis and burt-restored almost all snapshot files. To do that, as usual, I had to edit the snapshot files of c1dcuepics to move the quotes from the last line.

After that I restarted the autolocker that time it worked.

This morning c1lsc was again in the same unstable status as yesterday. This time I just reset it (no power recycling) and then I restarted it. It worked and now everything seems to be fine.

  1347   Tue Mar 3 08:44:31 2009 steveUpdatePEMair cond. maintenance today
IFO room 104 air conditions will be shut down for maintenance today.
This should be finished by noon.
The temperature and particle count variation can be more than usual.
  1346   Mon Mar 2 21:16:32 2009 YoichiUpdateLockingLow-gain High-gain PD switching may not be working well
Osamu, Yoichi

This afternoon, I run the locking script while doing calculations for the upgrade.
The IFO lost lock even at lower arm powers (around 6) if it was operated for a while (~ 5min).
It seemed as if there were some intermittent glitches (seismic? laser?) causing the lock losses.
We also saw once the TRX and TRY signals saturated at around arm power = 11 when there was a large fluctuation in the arm power.
Osamu suggested that it looked like the high-gain to low-gain PD switching was not working.

I won't come tonight as I may have caught a cold, but if someone comes tonight, it is worth checking the PD switching.
  1345   Mon Mar 2 16:27:40 2009 AlbertoConfigurationElectronicsMC Drum Mode SR560 Preamplifier Replaced

Today I checked out the SR560 around the lab. I confirmed that the one with the label "channel A noisy" is effectively mulfuctioning.

It was coonected to the lock-in amplifier set up for the drum mode peak readout.

I repleaced that with a working one.

  1344   Mon Mar 2 03:57:44 2009 YoichiUpdateLockingSunday night locking
Tonight's locking started with a boot fest of the FE computers which were all red when I came in.
It also took me sometime to realize that C1:IOO-MC_F was returning always zero to tdsavg, causing the offloadMCF script to do nothing.
I fixed this by rebooting c1iovme and c1iool0.

Like Rob on the thursday night, I was only able to reach arm power around 10.
This time, I turned down the MC WFS gain to 0.02 (from 0.3).
I also checked gains of most of the loops (MICH, PRC, SRC, DARM, CARM-MCL, CARM-AO).
All the loops looked fine until the lock was lost suddenly. Also the spectrum of MC_F did not change as the arm power was ramped up.
Actually, I was able to reach arm power=10 only once because I spent a long time checking the loop gains and spectrum at fine steps of the arm power.
So it is quite possible that this loss of lock was just caused by a seismic kick.
  1343   Fri Feb 27 13:49:19 2009 robUpdateLockingthurs night

Could not get past arm power of ~11 or so.  I was suspicious of the transmon high-gain/low-gain PD handover, so I ran the matchTransMon scripts, but that did not help.  I also removed the line in the cm_step script that increased the CM gain by 18dB at an arm power of 4.  The gain of the CM servo will increase naturally as the power in the IFO builds up, so it may not be good to crank it right away.  I tried several other CM gains, and watched the DARM loop, but still could not get past an arm power of ~10-11.  I'm not sure what's wrong, but it may be that mysterious CM-servo/McWFS conspiracy, so we can try turning down the McWFS gain next time.

  1342   Thu Feb 26 20:09:32 2009 YoichiHowToComputersSR785 python scripts now produce plots
I updated the python scripts to remotely perform measurements with an SR785.
Now these scripts can plot the results immediately using python's matplotlib capability. The sample plots can be seen in my previous elog entry.
In addition to the transfer function (TFSR785.py) and spectrum measurement (SPSR785.py) scripts, I also wrote a script for time series measurements (TSSR785.py).
This is useful when you want to check the signal level flowing in the channels before determining the excitation amplitude.
TSSR785.py will measure and show the time series and histogram of the signal measured by the SR785.
More detailed usage is explained in this wiki page:
http://lhocds.ligo-wa.caltech.edu:8000/40m/netgpib_package
  1341   Thu Feb 26 19:59:23 2009 YoichiUpdateLockingDaytime locking
Osamu, Yoichi

We tried locking today from about 2PM.
It took about 1000sec on average to acquire the initial lock.
After the initial lock is achieved, the hand-off/ramp-up steps were reasonably robust, although the AS beam sometimes fluctuates a lot (not good for mental health).

Like last night, the IFO loses lock at around arm-power=8.
We measured the CARM AO path loop gain at arm-power=4. We used the SR785 connected to the A-excitation channel of the common mode board through my TFSR785.py script.

The first attachment is the transfer function measured right after the arm power was ramped up to 4.
The overall bandwidth of the CM servo is only 400Hz. Note that since this is the loop gain of only the AO path, the low frequency gain is eaten by the MCL path.
The second attachment is the same transfer function measured after the AO path gain was increased by 6dB.
It is evident that the AO path is working.
We increased both the AO path and MCL gain by 18dB. The third attachment is the AO path TF in this state.
We then increased the arm power but lost lock at arm-power=6. We should have checked the DARM loop too.
BTW, these plots are automatically generated when you use TFSR785.py for transfer function measurements.


I added -notickle option to c1_watch_dr_bang, since tickling seems to be not necessary during the daytime (actually the initial lock was easier with no tickling).

As the construction work in the next building is now calmed down, I think it is ok to do locking during the day time, though I still plan to come at night.
The improvement of my brain efficiency during the day time may compensate for the longer wait time for initial lock.
  1340   Thu Feb 26 19:20:08 2009 YoichiConfigurationGeneralETMX camera centered. POX, POY PDs centered
Alberto, Yoichi

We centered the ETMX camera.
Alberto centered the POX and POY PDs.
We also updated the offset values of PD3 and PD4.
  1339   Thu Feb 26 01:24:44 2009 YoichiUpdateComputersMartian wireless is back
Today, a new wireless router arrived.
I configured and installed it. Now the martian wireless network is back.
I updated the wiki page about the wireless network.
http://lhocds.ligo-wa.caltech.edu:8000/40m/Network
  1338   Thu Feb 26 00:36:53 2009 YoichiSummaryComputersC1:LSC-TRX_OUT broken (and fixed later).
Today, Kakeru tried to convert C1:LSC-TRX_OUT and C1:LSC-TRY_OUT to DAQ channels.
He edited C1LSC.ini in the chans/daq directory to add the channel but it did not work.
Then he reverted the file back to the original one.
But after we still could not access these channels from dataviewer nor tds tools.
We restarted daqd and tpman on fb40m, but the problem persisted. Even rebooting the whole fb40m did not help.
After inspecting the log file of daqd, it was clear that tpman was failing to create test points for those channels.
I rebooted c1daqawg and then restarted tpman and daqd on fb40m again.
This time, the problem went away.
  1337   Wed Feb 25 11:48:02 2009 JenneUpdatePEMWiener filtering update - work on filtering some S5 DARM_CTRL data

Quick update on my wiener filtering status:

Joe has been helping me get on the GRID, so I now have  a grid certificate, and accounts on most/all of the clusters.

Joe also helped me get menkar to get S5 data so that I can do wiener filtering to the back-data. 

 

I've been running the wiener filtering algorithm, and right now, it doesn't do anything to improve the DARM_CTRL data.  I am confident that this is because something is funky in the wiener filtering algorithm somewhere.  The indicator of this is that the wiener filtering calculation takes the same amount of time (~95 seconds) to calculate a filter for 64 seconds of data as for 1 hour of data (both for N = 2000 taps). 

 

For reference, attached are my plots for the wiener filtering result for (1) 64 seconds of S5 data, and for (2) 3600 seconds of S5 data.

These plots were made using H1:DARM_CTRL as the signal to minimize, with 4 seismometers as the witness channels (EX_SEISX, EY_SEISY, LVEA_SEISX, LVEA_SEISY)

 

I'm working on figuring out what's going on with the filtering algorithm, and why it does work for C1:MC_L minimization, but does not work for H1:DARM_CTRL minimization.

 

 

 

ELOG V3.1.3-