40m QIL Cryo_Lab CTN SUS_Lab CAML OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 96 of 357  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  4298   Tue Feb 15 11:43:53 2011 JenneUpdateComputersOccasional error with NDS2

Just in case anyone has encountered this / knows how to fix it....

I'm running NDS2 on Rossa, trying to get a bunch of raw data from S5.  I get 10min of data at a time, and it goes through ~200 iterations successfully, and then throws the following error:

Getting new data
Connecting.... authenticate ... done
daq_recv_id: Wrong length read (0)
Error reading writerID in daq_recv_block
Warning: daq_request_data failed
 
??? Error using ==> NDS2_GetData
Fatal Error getting channel data.

Error in ==> getDARMdataTS at 37
oot = NDS2_GetData({...

Error in ==> SaveRawData_H1_DARM at 40
    oot = getDARMdataTS(t0s(ii), strideDuration, srate);

  4541   Mon Apr 18 21:09:45 2011 JamieConfigurationComputersnew control room machine: pianosa

I've just installed the new control room machine: "pianosa".   It is a replacement for the old sun machine "op440m" [0].

Hardware:

  • dual dual-core Intel Core i7-2600 CPU @ 3.40GH, hyperthreaded to provide 8 effective cores
  • 16G memory (4x 4G dimms)
  • nVidia GF108 GeForce GT 430

It's now running Ubuntu 10.04 LTS 64bit.  Unfortunately, the default 10.04 kernel is 2.6.32, which does not support pianosa's apparently very new network adapter, which is (from lspci):

00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)

To get around this I temporarily added a PCI nic so that I could get on the network.  I then added the Ubuntu kernel team PPA archive and installed linux-image-2.6.38-2, which is new enough to have the needed network driver, but not completely bleeding edge:

sudo add-apt-repository ppa:kernel-ppa/ppa
sudo apt-get update
sudo apt-get install linux-image-2.6.38-2-generic linux-headers-2.6.38-2-generic

Once the built in nic was working I removed the temporary one.  Everything seems to be working fine now.

I have not yet done any configuration to integrate pianosa into the CDS network.  I'll do that tomorrow.

[0] op44m has been moved into the control room rack next to linux1, in headless mode.  If there is still a need to run scripts that only run on solaris, op440m can still be accessed via ssh as normal.  Hopefully we can fully decommission this machine soon.

[1] https://launchpad.net/~kernel-ppa/+archive/ppa

  4542   Mon Apr 18 21:14:53 2011 JamieConfigurationComputersnew control room machine: pianosa
Also, op440m's Sun monitor did not work well with pianosa, so I'm lending pianosa my HP monitor until we can get a suitable replacement.
  4549   Wed Apr 20 23:20:49 2011 jamieSummaryComputersinstallation of CDS tools on pianosa

This is an overview of how I got (almost) all the CDS tools running on pianosa, the new Ubuntu 10.04 control room work station.

This is machine is experiment in minimizing the amount of custom configuration and source code compiling. I am attempting to install as many tools as possible from existing packages in

available packages

I was able to install a number of packages directly from the ubuntu archives, including fftw, grace, and ROOT:

apt-get install \
libfftw3-dev \
grace \
root-system

LSCSOFT

I installed all needed LSCSOFT packages (framecpp, libframe, metaio) from the well-maintained UWM LSCSOFT repository.

$ cat /etc/apt/sources.list.d/lscsoft.list
deb http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze
deb-src http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze contrib
sudo apt-get install lscsoft-archive-keyring
sudo apt-get update
sudo apt-get install ldas-tools-framecpp-dev libframe-dev libmetaio-dev lscsoft-user-en

You then need to source /opt/lscsoft/lscsoft-user-env.sh to use these packages.

EPICS

There actually appear to be a couple of projects that are trying to provide debs of EPICS. I was able to actually get epics working from one of them, but it didn't include some of the other needed packages (such as MEDM and BURT) so I fell back to using Keith's pre-build binary tarball.

Prereqs:

apt-get install \
libmotif-dev \
libxt-dev \
libxmu-dev \
libxprintutil-dev \
libxpm-dev \
libz-dev \
libxaw7-dev \
libpng-dev \
libgd2-xpm-dev \
libbz2-dev \
libssl-dev \
liblapack-dev \
gfortran

Pulled Keith's prebuild binary:

cd /ligo/apps
wget https://llocds.ligo-la.caltech.edu/daq/software/binary/apps/ubuntu/epics-3.14.10-ubuntu.tar.gz
tar zxf epics-3.14.10-ubuntu.tar.gz

GDS

I built GDS from svn, after I fixed some broken stuff [0]:

cd ~controls/src/gds
svn co https://redoubt.ligo-wa.caltech.edu/svn/gds/trunk
cd trunk
#fixed broken stuff [0]
source /opt/lscsoft/lscsoft-user-env.sh
./bootstrap
export GDSBUILD=online
export ROOTSYS=/usr
./configure --prefix=/ligo/apps/gds --enable-only-dtt --with-epics=/ligo/apps/epics-3.14.10
make
make install

dataviewer

I installed dataviewer from source:

cd ~controls/src/advLigoRTS
svn co https://redoubt.ligo-wa.caltech.edu/svn/advLigoRTS/trunk
cd trunk/src/dv
#fix stupid makefile /opt/rtapps --> /ligo/apps
make
make install

I found that the actual dataviewer wrapper script was also broken, so I made a new one:

$ cat /ligo/apps/dv/dataviewer
#!/bin/bash
export DVPATH=/ligo/apps/dv
ID=$$
DCDIR=/tmp/${ID}DC
mkdir $DCDIR
trap "rm -rf $DCDIR" EXIT
$DVPATH/dc3 -s ${NDSSERVER} -a $ID -b $DVPATH "$@"

environment

Finally, I made a environment definer file:

$ cat /ligo/apps/cds-user-env.sh
# source the lscsoft environment
. /opt/lscsoft/lscsoft-user-env.sh

# source the gds environment
. /ligo/apps/gds/etc/gds-user-env.sh

# special local epics setup
EPICS=/ligo/apps/epics
export LD_LIBRARY_PATH=${EPICS}/base/lib/linux-x86_64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${EPICS}/extensions/lib/linux-x86_64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=${EPICS}/modules/seq/lib/linux-x86_64:$LD_LIBRARY_PATH
export PATH=${EPICS}/base/bin/linux-x86_64:$PATH
export PATH=${EPICS}/extensions/bin/linux-x86_64:$PATH
export PATH=${EPICS}/modules/seq/bin/linux-x86_64:$PATH

# dataviewer path
export PATH=/ligo/apps/dv:${PATH}

# specify the NDS server
export NDSSERVER=fb

[0] GDS was not compiling, because of what looked like bugs. I'm not sure why I'm the first person to catch these things. Stricter compiler?

To fix the following compile error:

TLGExport.cc:1337: error: ‘atoi’ was not declared in this scope

I made the following patch:

Index: /home/controls/src/gds/trunk/GUI/dttview/TLGExport.cc
===================================================================
--- /home/controls/src/gds/trunk/GUI/dttview/TLGExport.cc (revision 6423)
+++ /home/controls/src/gds/trunk/GUI/dttview/TLGExport.cc (working copy)
@@ -31,6 +31,7 @@
#include <iomanip>

#include <string.h>

#include <strings.h>

+#include <stdlib.h>


namespace ligogui {
using namespace std;

To fix the following compile error:

TLGPrint.cc:264: error: call of overloaded ‘abs(Int_t&)’ is ambiguous

I made the following patch:

Index: /home/controls/src/gds/trunk/GUI/dttview/TLGPrint.cc
===================================================================
--- /home/controls/src/gds/trunk/GUI/dttview/TLGPrint.cc (revision 6423)
+++ /home/controls/src/gds/trunk/GUI/dttview/TLGPrint.cc (working copy)
@@ -22,6 +22,7 @@
#include <fstream>

#include <map>
#include <cmath>

+#include <cstdlib>


namespace ligogui {
using namespace std;

  4552   Thu Apr 21 15:03:29 2011 steveUpdateComputersjuction board finds home

The anti aliasing box was opened up at the back to accommodate the junction board and the SCSI cable towards the ADC. Aluminum plate was attached to the bottom to hold the strain relief clamp.

Three more hanging junction cards will be replaced in this manner.

Attachment 1: P1070570.JPG
P1070570.JPG
Attachment 2: P1070568.JPG
P1070568.JPG
  4563   Mon Apr 25 11:23:41 2011 AlastairBureaucracyComputerswiki?

40m wiki seems to have been down for quite a while now but I can't see any info in the elog about it.  Is there some ongoing problem?

119427.strip.gif

  4564   Mon Apr 25 11:58:37 2011 JamieBureaucracyComputerswiki?

Quote:

40m wiki seems to have been down for quite a while now but I can't see any info in the elog about it.  Is there some ongoing problem?

 There was an email from Dave Barker about this.  They had to reorganize the DNS at LHO.  The URL that should be used is: http://blue.ligo-wa.caltech.edu:8000/40m

  4565   Mon Apr 25 12:55:19 2011 JenneBureaucracyComputerswiki?

Quote:

Quote:

40m wiki seems to have been down for quite a while now but I can't see any info in the elog about it.  Is there some ongoing problem?

 There was an email from Dave Barker about this.  They had to reorganize the DNS at LHO.  The URL that should be used is: http://blue.ligo-wa.caltech.edu:8000/40m

 Nope.  I know I had the right address, and it was down for me too all weekend.  It's better now though.  blue.ligo-wa.caltech.edu was up though.

  4566   Mon Apr 25 12:55:35 2011 AlastairBureaucracyComputerswiki?

Quote:

Quote:

40m wiki seems to have been down for quite a while now but I can't see any info in the elog about it.  Is there some ongoing problem?

 There was an email from Dave Barker about this.  They had to reorganize the DNS at LHO.  The URL that should be used is: http://blue.ligo-wa.caltech.edu:8000/40m

 Thanks Jamie, I've updated the links from the ATF wiki to reflect this.

  4603   Tue May 3 00:44:02 2011 KojiConfigurationComputersMartian WIreless Bridge

The Martian wireless bridge has the ethernet cable inserted in the wrong connector.

It should be inserted to one of the four port. Not in the "INTERNET" connector.

Once the connector has been changed, the martian net as well as the internet became accessible from the laptops.

  4642   Thu May 5 15:26:52 2011 Leo SingerConfigurationComputers'glue' installed on some control room computers

I installed 'glue' on Rossa, Allegra, and Rosalba.  This is a Python module that includes a facility for LIGO_LW XML files.  Oddly, I couldn't find the glue package on Pianosa.

  4712   Fri May 13 14:54:20 2011 Leo SingerUpdateComputersDiaggui fixed on pianosa

I fixed diaggui on pianosa.  Previously, it was not able to start because it depended on libreadline5, whereas Ubuntu distributed libreadline6.  Now pianosa has both libreadline5 and libreadline6, so diaggui works.

  4784   Fri Jun 3 18:12:03 2011 steveUpdateComputersno PEM channels

AA filter box was removed and modified at 1Y7 today. The -5V power supply was current limited when I plugged it back in. It was removed for medical attention.

NO PEM channels available! because of this.

 

  4789   Mon Jun 6 17:28:31 2011 steveUpdateComputersstill, no PEM channels

Quote:

AA filter box was removed and modified at 1Y7 today. The -5V power supply was current limited when I plugged it back in. It was removed for medical attention.

NO PEM channels available! because of this.

 

 D68L8EX-850Hz filter chips were removed and bypassed-shorted as Rana's entry on March 17, 2006 in old elog.

This unit is still has a short somewhere.

Attachment 1: P1070846.JPG
P1070846.JPG
  4866   Thu Jun 23 10:35:12 2011 steveUpdateComputersold computers leaving the lab

Rod Luna picked up these computers for Larry Wallace yesterday: Dell Inspiron 530, Dell Dimension 4600 and SunBlade 1000

Attachment 1: P1070905.JPG
P1070905.JPG
Attachment 2: P1070904.JPG
P1070904.JPG
Attachment 3: blade1000.JPG
blade1000.JPG
  4897   Tue Jun 28 11:25:56 2011 AlastairBureaucracyComputersrestarted elog

The manual instructions on the 40m wiki for restarting wouldn't work.  I killed the process okay, but then I got an error saying it "couldn't bind to port 8080, please try again using -p to select port".  The automated script worked though.

  4982   Mon Jul 18 14:39:53 2011 JenneUpdateComputersLSC PD chans acquired

There's too much tromping around, so I'm not going to actually measure PRC length right now, but I did set some channels to be acquired (POPDC, POXDC, POYDC) in addition to ASDC which was already acquired, so that I can look at the resonance fringes when I sweep the ABSL laser (hopefully later tonight....)

  4991   Tue Jul 19 20:36:08 2011 ranaUpdateComputersVirtualBox + Windows 7 on rossa

I installed Virtual Box on rossa. Then I put Windows 7 in there and am now installing Altium.

You can run Windows on rossa by just clicking Applications -> System Tools -> Virtual Box.

  5002   Wed Jul 20 17:43:33 2011 sureshUpdateComputersrestarted the frame builder

I restarted the frame builder in the last 15mins. 

I was making a change to a DAC channel in the C1IOO model.

  5058   Thu Jul 28 21:52:40 2011 ranaUpdateComputersanother attempt to use pianosa
  1. Pianosa doesn't cache the SVN pwds so you need to re-enter at each SVN up or commit. This is different from the rest of our workstations. We need to determine what behavior we want.
  2. Tried to use the netGPIB scripts:

pianosa:gpib 0> ./readSR785.csh rb2
rb2
netgpibdata.py: Command not found.

  5184   Thu Aug 11 08:29:28 2011 steveUpdateComputersdataviewer at Rosalba

I'm having this problem with DTV every morning at Rosalba only. It wants to start with a negative GPS time and it can not connect to the frame builder.

Normally after a few time of starting it, it will work.

Attachment 1: gpsfmb.png
gpsfmb.png
  5379   Sat Sep 10 18:52:45 2011 ranaUpdateComputersconlog getting filled up

One of the reasons that conlog seems so slow lately is that its been writing 100's of MB of .log files every day since early summer. It looks like the people who have been working on the mdl builds have not been properly adjusting the conlog channel lists. When this is not done conlog just gets filled up with non-control channels like OUT, OUTPUT, OUTMON, etc.

Peter Shawhan has supplied us with many scripts in the conlog directory to clean up these bloated files and fix the channel list.

  5548   Mon Sep 26 17:49:21 2011 JenneUpdateComputersWe now have BURT restore for slow channels

Koji and Suresh found that there have not been any autoburt snapshots taken of slow channels since ~December 13th 2010.  Not good!

We have found an elog from Joe talking about autoburt changes from that day:  elog 4046

Joe pointed all of the autoburt stuff to the new directory system, so it now decides to take a snapshot of every system in the *new* target directory.  This means, since all of the aux things were left in the *old* target directory that none of them were getting snapshots taken.  I have added the old target path back to the autoburt cron file so that every hour it will search through both old and new target directories and take snapshots of everything in both. 

So, the systems which will now once again have autoburt snapshots taken are the following:

c1aux

c1auxex

c1auxey

c1dcuepics

c1iool0

c1iscaux

c1iscaux2

c1iscepics

c1losepics

c1omcepics

c1psl

c1susaux

c1vac1

c1vac2

 

I moved some old stuff (and especially things which would conflict with the new stuff) to the old target directory/oldfe/ :  c1ass, c1assepics, c1susvme1, c1susvme2, c1sosvme, c1iovme.

The following systems don't have an autoburt.req file, so don't get snapshots:  c0daqawg, c1daqctrl, c1dcu1, c1iscex, c1iscey.  If any of these need autoburts, we should create them.

All the new systems in the new target directory still have their autoburts working.

The first test of this will be in a few minutes, at 18:07:00 Pacific during the regular cron job.  Hopefully nothing crashes....

  5552   Mon Sep 26 22:40:41 2011 JenneUpdateComputersWe now have BURT restore for slow channels
[Jenne, Koji]

After much Perl-learning and a few iterations, we have fixed the burt restore script, so that it actually does the slow channels. We have so far had one successful run, at 22:25, and the regular cron job should start doing the slow channels as of 23:07.
  5558   Tue Sep 27 15:33:03 2011 JenneUpdateComputersMaking models, wreaking havoc

[Jenne, Mirko, Den]

We have entered into an adventure in model compiling.  What follows is a stream-of-consciousness report of what the hell we're doing, so Jamie can figure it out and fix it if everything goes to hell.

Note that for the first part of things, we have used a new version of the Adaptive XFCODE, which Mirko and Den modified last night to be able to handle multiple control signal inputs.  

On c1lsc, make uninstall-daq-c1oaf, make clean-c1oaf, make c1oaf.

***ERROR: The following IPCx RECEIVER module(s) not found in the file /opt/rtcds/caltech/c1/chans/ipc/C1.ipc:

                C1:RFM_OAF_MCL

On c1sus, make uninstall-daq-c1sus, make clean-c1sus, make c1sus.  (This was an accident.  I should have been making c1rfm.  Oops.)  Then make install-c1sus.  It looks like this automatically did make install-daq-c1sus and make install-screens-c1sus, so I'm not doing those. 

On c1sus, make uninstall-daq-c1rfm, make clean-c1rfm, make c1rfm.

***ERROR: The following IPCx RECEIVER module(s) not found in the file /opt/rtcds/caltech/c1/chans/ipc/C1.ipc:

                C1:IOO-RFM_MCL


On c1ioo, make uninstall-daq-c1ioo, make clean-c1ioo, make c1ioo.  No errors.

On c1lsc, make c1oaf.  Here's some of the ouptut, with some of the error stuff:

: warning: ISO C90 forbids mixed declarations and code
/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf/../controller.c:2954: warning: ISO C90 forbids mixed declarations and code
make[3]: *** [/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf/c1oaffe.o] Error 1
make[2]: *** [_module_/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf] Error 2
make[2]: Leaving directory `/usr/src/linux-2.6.34.1-cs'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf'
make: *** [c1oaf] Error 2

Again on c1lsc, make clean-c1oaf, make c1oaf.  Here are some things:

Warning:  variable "sysnum" is used but not declared.

In file included from build/c1oafepics/c1oaf.i:38:
src/include/fmReadCoeff.h:4:1: warning: "NO_FM10GEN_C_CODE" redefined
<command-line>: warning: this is the location of the previous definition

build/c1oafepics/c1oaf.i:5156: warning: passing argument 2 of 'strcpy' discards qualifiers from pointer target type
/usr/include/string.h:127: note: expected 'const char * __restrict__' but argument is of type 'volatile char *'

/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf/../../include/drv/inputFilterModule1.h:5: note: expected 'double *' but argument is of type 'long unsigned int *'

/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf/../controller.c:2780: warning: ISO C90 forbids mixed declarations and code

make[3]: *** [/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf/c1oaffe.o] Error 1
make[2]: *** [_module_/opt/rtcds/caltech/c1/core/branches/branch-2.1/src/fe/c1oaf] Error 2
make[1]: *** [default] Error 2
make: *** [c1oaf] Error 2

Again, on c1lsc, make clean-c1oaf, make c1oaf.  More errors, pretty similar.  Then we changed the name of the adaptive filtering code, so maybe it will work now?  We had called the block "TOP_XFCODE", but that was the name of the old .c code.  The block used to be called "XFCODE", in a subsystem "TOP".  So now we named the .c code "ADAPT_XFCODE" since the block is "XFCODE", and the subsystem is "TOP".

Again, on c1lsc, make clean-c1oaf, make c1oaf.  Errors, they look the same.

Mirko is now modifying c1oaf.mdl to look more like the old version, with only one control signal input, so that we can use the old XFCODE that has been around for years.

First, we completely took out the .c code entirely.  Now the c1oaf.mdl is just signals and matricies, no c-code is called.  We did make clean-c1oaf, make c1oaf, and pretty much all of the same errors are present.
 

We took out the buses, and did make clean-c1oaf, make c1oaf, and we got a whole lot of warnings, but no "Error 2"'s.  This seems good.  We're going to try replacing those buses with Muxes, and see how that goes.

Now we're going to try to install the c1oaf, because maybe all the errors and shit we're seeing is just useless crap, and there aren't actually problems...here we go!

That seemed to work, and the c1oaf model on the GDS status screen seems happy.  Numbers are moving around, which is my only current diagnostic.

Okay, now Mirko is going back to the full, new c1oaf, but replacing the Buses with Muxes. 

Did a make clean-c1oaf, make c1oaf, got errors again.

Once again removed the .c code.  Just put in a matrix instead. Did make clean-c1oaf, make c1oaf.  No errors. 

Den did a reorganization of the .c code, and we put it back in to the simulink model.  Trying again the making stuff.  Fail..Basically the same errors as before.

Next up:  Putting in .c-code, but something which basically does nothing.  Just defines all the outs as zeros.  Make stuff.  Still had problems, same errors.  Grrrr, argh. 

Found the RCG manual:  T080135-v4.  In it, when talking about including c-code, it had an example of totally simple code.  We tried out their version of simple code, and it worked.  No errors!  Now to figure out what is same and different between our simple code and theirs.

 PUT THE RIGHT STUFF in the Block Properties for the c-code, including name of the .c file, and path to the .c file.  This is critical!!  Now we can make some of our simple versions work, but not all.  We're slowly increasing complexity of our c-code...

 At some point in the last hour, I tried a make install-c1oaf, and then checked the screens, and they all had bad white boxes.  So even though the model seemed to compile (at one point), the channels and screens aren't happy yet.  But that's really a project for after the code compiles happily.

Okay, some progress was made to get the c-code running, and compiling, but it's not all there yet.  We're putting in a simple, useless version of the c-code so we can try compiling everything else.

Everything is compiled, installed, there are no red lights on the GDS_STATUS screen.  All seems okay for locking for tonight.

 

  5565   Wed Sep 28 14:15:40 2011 JenneUpdateComputersEdits to c1pem, c1oaf

[Mirko, Jenne]

Mirko edited c1pem to have some new BLRMS channels.

I added a master Enable switch to the c1oaf.

Both were compiled, and restarted.  fb rebooted.  All looks okay (hopefully)

  5568   Wed Sep 28 21:23:23 2011 MirkoUpdateComputersISCY FE network card / cable not ok

[Mirko,Jenne]

We discovered that the left network cable is not rigidly connected to the back of the ISCY FE computer. You can easily pull it out a mm disconnecting it. It should click rigidly in place. Not clear if it's the cable or the network card.

  5569   Wed Sep 28 21:28:34 2011 MirkoUpdateComputersTorturing control computers. Fine again now

[Mirko, Jenne]

We tried to run an extended version of Matt's LMS adaptive filter c-code. We got the extension to compile separately in gcc first. Then after some tweaking of the code we could make-install c1oaf with the c-code.

This killed c1lsc (the FE computer running c1oaf). Not responding to ssh or even pings. We replaced the bad c-code with harmless code, then reset c1lsc via the hardware button. While looking for c1lsc we discovered the problem with c1iscy network card (see previous entry).

After c1lsc reboot, restart of the FB, and a BURT restore not ok yet

  5571   Wed Sep 28 22:25:25 2011 JenneUpdateComputersTorturing control computers. Fine again now

Quote:

[Mirko, Jenne]

We tried to run an extended version of Matt's LMS adaptive filter c-code. We got the extension to compile separately in gcc first. Then after some tweaking of the code we could make-install c1oaf with the c-code.

This killed c1lsc (the FE computer running c1oaf). Not responding to ssh or even pings. We replaced the bad c-code with harmless code, then reset c1lsc via the hardware button. While looking for c1lsc we discovered the problem with c1iscy network card (see previous entry).

After c1lsc reboot, restart of the FB, and a BURT restore not ok yet

 We had lots of trouble damping the Vertex Suspensions after this craziness.  A symptom was that even if all of the damping servos on an optic were OFF, and I turned the watchdog on (LSC is disabled, so no OAF siganls, no LSC signals), there were signals going to the coils. 

We did a reboot of the c1sus computer, did another BURT restore, and the optics started damping happily.   Burt restore, at least for c1susepics and c1mcsepics, seems to not be happening automatically.  I thought it was supposed to happen when the model was restarted?

Things now appear to be normal again.

  5573   Thu Sep 29 00:16:35 2011 DenUpdateComputersSegmentation fault fixed.

The OAF c-code crashed because of the segmentation fault. We've created arrays of static variables

    static int pst[nDOF];
    static int isFirst[nDOF];
    static adaptive_filter_state state[nDOF];

an tried to give in the to the ITERATE - function their current values

        datOut[i] = ITERATE(iterateDatIn, iterateNIn, pst[i], isFirst[i], state[i]);

ITERATE function was declared as

double ITERATE(double *datIn, int nIn, int pst, int isFirst, adaptive_filter_state state) {}

Here the segmentation fault comes out. Static variables are meant to be created only once but here in the function ITERATE we try to create them once again in a local form, because we give the variables by their values.

Instead, we must give the variables by their pointer, then the variables won't be created again during the function call and will be changed in the function.

        datOut[i] = ITERATE(iterateDatIn, iterateNIn, &pst[i], &isFirst[i], &state[i]);

       double ITERATE(double *datIn, int nIn, int *pst_s, int *isFirst_s, adaptive_filter_state *state_s)

In order not to change significantly Matt's code and use his notations we can add in the ITERATE function

    int pst = *pst_s;
    int isFirst = *isFirst_s;
    static adaptive_filter_state state;
    state = *state_s;

..................................Matt's code.........................................

    *pst_s = pst;
    *isFirst_s = isFirst;
    *state_s = state;

I've tested the program, now it does not give any segmentation faults and conserves memory that it uses.

  5577   Thu Sep 29 20:37:12 2011 MirkoUpdateComputersNew c1oaf c-code: Breaking in new way

[Mirko, Jenne]

Programmed a new implementation of the LMS in C. Compiles fine in gcc. The full code still kills c1lsc computer. Tried to go through the code uncommenting more and more. Not perfect in reproducability. The attached version should compile and keep c1oaf running, but not actually produce an adaptive filter. At some point the code just keeps c1oaf from starting up. Leaves the c1lsc computer in working order. At some point I got error messages like ..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_BS_Name08", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.219208306
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_BS_Name09", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.225999915
..................................................................
CA.Client.Exception...............................................
    Warning: "Identical process variable names on multiple servers"
    Context: "Channel: "C1:OAF-ADAPT_CORR_ETMX_SW1R", Connecting to: 192.168.113.62:50970, Ignored: c1lsc:34533"
    Source File: ../cac.cpp line 1209
    Current Time: Thu Sep 29 2011 19:18:17.243037042
..................................................................

This usually indicates that there are multiple carepeater running. Didn't find where that would be. After rebooting c1lsc and c1sus a couple of times everything seems fine.

Attachment 1: ADAPT_XFCODE11-09-29---20-12.c
// LMS algorithm adaptive filtering for the Caltech 40m -- Sept. 2011 by M. Prijatelj

#define nDOF 8
#define nWIT 21
#define nTAPS 1000

void ADAPT_XFCODE(double *in, int inSize, double *out, int outSize) {

//First the DOFs and their switches
//int nDOF=8;
... 136 more lines ...
  5733   Tue Oct 25 01:19:17 2011 SureshUpdateComputersfb restarted and c1ioo model committed to svn

When I installed the new model I restarted the fb between 1 and 1:30 AM PDT Oct 25, 2011

  5757   Fri Oct 28 15:33:06 2011 JenneUpdateComputersNifty screen generator

Suresh showed me a cool script that Mirko made, but didn't elog about.

You tell the script what filter banks you want, and it creates a screen for each with a bunch of different filter module display formats.  Then you can copy the format you like into the actual screen you're modifying. 

Currently PEM, LSC and IOO (and maybe others?) have "fmX" folders inside their medm/c1.../master folders.  For each subsystem, we need to copy this folder, and modify the generic .adl file so that it puts in the correct subsystem letters.  Once this is done, you can just run the generateFMscreens.py after putting in your filter bank names.

  5758   Fri Oct 28 15:45:52 2011 MirkoUpdateComputersNifty screen generator

Quote:

Suresh showed me a cool script that Mirko made, but didn't elog about.

You tell the script what filter banks you want, and it creates a screen for each with a bunch of different filter module display formats.  Then you can copy the format you like into the actual screen you're modifying. 

Currently PEM, LSC and IOO (and maybe others?) have "fmX" folders inside their medm/c1.../master folders.  For each subsystem, we need to copy this folder, and modify the generic .adl file so that it puts in the correct subsystem letters.  Once this is done, you can just run the generateFMscreens.py after putting in your filter bank names.

 Wasn"t me.

  5778   Tue Nov 1 18:45:48 2011 JenneUpdateComputersAllegra's screens

I was trying to give Allegra a second head, and it didn't quite work.  It's still in progress.  Steve, you might not like how I've 'mounted' the second monitor, so we can talk about something that might work tomorrow.

  5780   Tue Nov 1 23:13:28 2011 ZachUpdateComputersNDS2 channel files

I did some messing around with the NDS2 config and channel files and things seem to be working as expected... for now. SENSOR channel data can be acquired for all sensors on all hanging optics.

What I did:

  • NDS2 gets its channel lists from .../users/jzweizig/nds2-mafalda/nds2.conf, which is called in the start-nds2 script. Within this, there are channel.file lines that specify which channels are available for raw and trend data. The four files that were listed were:
    • C-R-raw-channel_list.txt
    • C-M-ChanList.txt
    • C-T-ChanLIst.txt
    • C-R-online-channel_list.txt (this one was listed after a hashed line, which was suspicous---see below)
  • I noticed that a grep for SENSOR only returned lines for non-MC mirrors in both "R" files
  • I also noticed that calling NDS2_GetChannels('mafalda.martian:31200') did not return any non-suffixed (i.e., raw) channel names for MCx_SENSOR channels, while non-MC SENSOR channels each had two non-suffixed listings. I thought this was strange.
  • I manually added the line "C1:SUS-MC1_SENSOR_UL 0 real_4 2048 C-R" to one of the "R" channel files, then restarted the NDS2 server, and that channel was still not served. I figured that the second "R" channel file might have been left in the config file as a mistake, so I commented it out, restarted the NDS2 server, and was able to get MC1_SENSOR_UL data. I have left the comment-out there, with a signed EDIT.
  • Wary of (and too lazy to) manually add lines for all 5 sensors for each MC mirror, I decided to try generating a channel file using the most recent .gwf file in the frames, as indicated in Joe and John's elog post. To do this, while in .../nds2-mafalda/, I ran:
    • /cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList /frames/full/10042/C-R-1004246528-16.gwf > C-R-raw-channel_list.txt
  • A grep for SENSOR in the new C-R-raw-channel_list.txt now returned lines for all MC mirror sensors... BUT NOT FOR ETMY(?!). I tried some slightly older .gwf files (all from today), but the ETMY files never showed up. I had no choice but to enter them manually. Another odd thing is that the channel file generated this way seems to be fairly jumbled up, in the sense that there is no clear top-town order (e.g. SUS-BS_blah then SUS-ETMX_blah). Instead, some SUS-BS channels are here, some are after SUS-ETMX or SUS-PRM channels, etc. Look at the file to know what I mean.
  • The original raw channel file is there as C-R-raw-channel_list.txt.bak.1004247481.

In any case, as I said, everything appears to be working now, but as soon as we try to generate a new channel file using the prescribed means, there will inevitably be channels omitted. Someone who knows more than me should get to the bottom of this and wiki a strict, detailed procedure for how this is to be done.

  5830   Mon Nov 7 14:50:19 2011 JenneUpdateComputersISCEX is having a bad day

I clicked on the FE status screen, just to check on things, and everything on the c1iscex section was red (the IOP and c1scx).  Upon deciding that was probably a bad thing, I did a soft reboot from the control room.  Now the IOP says "NO SYNC", and the c1scx status thing is totally frozen. 

I have sent Jamie a whiny email. He promises to be here soon to fix it.

  5836   Mon Nov 7 17:27:28 2011 jamieUpdateComputersISCEX IO chassis timing slave dead

It appears that the timing slave in the c1iscex IO chassis is dead.  It's front "link" lights are dark, although there appears to be power to the board (other on-board leds are lit).  These front lights should either be on and blinking steadily if the board is talking to the timing system, or blinking fast if there is no connection to the timing distribution box.  This likely indicates that the board has had some sort of internal failure.

Unfortunately Downs has no spare timing slave boards lying around at the moment; they're all stuffed in IO chassis awaiting shipping.  I'm going to email Rolf about stealing one, and if he agrees we'll work with Todd Etzel to pull one out for a transplant

  5843   Tue Nov 8 19:08:21 2011 MirkoHowToComputersNew DV

Quote:
To use the new ligoDV (previously GEO DV) to look at 40m data, open up a matlab, set up for mDV as usual,
and then from the /cvs/cds/caltech/apps/ligoDV/ directory, type 'ligoDV'.

Then select which NDS server you want to look at and then start clicking to get some plots.

To start ligodv go in matlab to /cvs/cds/caltech/apps/ligoDV/ and call ligodv. Ligodv will start up when you are in another directory, but will give strange errors. Only seems to work with NDS2 server mafalda port 31200. This doesn't have all channels. When pointing it to fb port 8088 it freezes when you try to adjust the start/stop time. Make sure to ask for the correct UTC time, not the local time.
  5847   Wed Nov 9 13:44:04 2011 MirkoUpdateComputersNDS1 missing channels in matlab

The Matlab NDS1 access seems to only work for some channels. With other channels it just hangs 'Busy' and does nothing.
Below you can see some channels that make matlab hang. Everyting happened on allegra. I tried compiling the NDS1 sources (from https://www.gravity.phy.syr.edu/dokuwiki/doku.php?id=ligodv:nds1_ligodv_install ) into mex files myself. Same result. I
 

a=NDS_GetChannels('fb:8088'); %/cvs/cds/caltech/apps/linux64/share/matlab/NDS_GetChannels.m
%data=NDS_GetData({'C1:IOO-MC_F_DQ'},1004826500,100,'fb:8088',a)     %Works
%data=NDS_GetData({'C1:IOO-WFS1_PIT_IN1_DQ'},1004826500,100,'fb:8088',a)     %Works
data=NDS_GetData({'C1:LSC-AS11_I_OUT'},1004826500,100,'fb:8088',a)         %Doesn't work, hangs
%%%which NDS_GetData.m: /cvs/cds/caltech/apps/linux64/share/matlab/NDS_GetData.m

  5853   Wed Nov 9 17:41:17 2011 JenneUpdateComputersETMX restored

Jamie did computer magic.  I burt restored scxepics, and restored ETMX damping.

  5863   Thu Nov 10 16:26:46 2011 MirkoUpdateComputersFirefox kills elog

Had to restart the elog many times. For some reason firefox 8 on Win 7 kills the elog pretty consistently when trying to make a new entry. IE9 works fine ....

  5880   Sat Nov 12 02:27:00 2011 SureshUpdateComputersC1IOO front end suddenly froze. Was restarted remotely

[Koji Suresh]

No one was messing with the c1ioo or any other machine.   The medm screens for WFS and MC alignment froze while I was working on Rossa.

There were number of red lights pertaining to c1ioo machine on the CDS_FR_STATUS screen.  So we logged into c1ioo   from Rossa and restarted it with 'sudo shutdown -r now'.  It came back up but the C1IOO_MC_TRANS_SUM, P and Y signals were not available on the C1IOO LOCKMC screen.

I saw several messages similar to the one here


Sat Nov 12 02:09:14 PST 2011

  medmCAExceptionHandlerCb: Channel Access Exception:
  Channel Name: Unavailable
  Native Type: Unavailable
  Native Count: 0
  Access: Unavailable
  IOC: Unavailable
  Message: Virtual circuit disconnect
  Context: c1ioo.martian:43553
  Requested Type: TYPENOTCONN
  Requested Count: 0
  Source File: ../cac.cpp
  Line number: 1126

 

The MC autolocker script wasnt running.  The heartbeat bit was not blinking on the MC_LOCKMC screen.  So we manually restarted the script.  Hopefully it will return to normal operation.

I restarted the fb at Sat Nov 12 02:12:19 PST 2011  in an attempt to see this resolves the problem.

It didnt.

 

  5881   Sat Nov 12 02:44:18 2011 SureshUpdateComputersC1IOO front end suddenly froze. Was restarted remotely

Quote:

[Koji Suresh]

No one was messing with the c1ioo or any other machine.   The medm screens for WFS and MC alignment froze while I was working on Rossa.

There were number of red lights pertaining to c1ioo machine on the CDS_FR_STATUS screen.  So we logged into c1ioo   from Rossa and restarted it with 'sudo shutdown -r now'.  It came back up but the C1IOO_MC_TRANS_SUM, P and Y signals were not available on the C1IOO LOCKMC screen.

I saw several messages similar to the one here


Sat Nov 12 02:09:14 PST 2011

  medmCAExceptionHandlerCb: Channel Access Exception:
  Channel Name: Unavailable
  Native Type: Unavailable
  Native Count: 0
  Access: Unavailable
  IOC: Unavailable
  Message: Virtual circuit disconnect
  Context: c1ioo.martian:43553
  Requested Type: TYPENOTCONN
  Requested Count: 0
  Source File: ../cac.cpp
  Line number: 1126

 

The MC autolocker script wasnt running.  The heartbeat bit was not blinking on the MC_LOCKMC screen.  So we manually restarted the script.  Hopefully it will return to normal operation.

I restarted the fb at Sat Nov 12 02:12:19 PST 2011  in an attempt to see this resolves the problem.

It didnt.

 

 The problem was resolved after I burtrestored (c1mcs c1ioo and c1rfm) epics snapshots.

 

  5886   Mon Nov 14 12:16:41 2011 JenneUpdateComputersOAF model died for unknown reason

I am meditating on the OAF, and had it running and calculating things.  I had the outputs disabled so I could take reference traces in DTT, but the Adapt block was calculating for MCL.  At some point, all the numbers froze, and the CPU meter had gone up to ~256ms.  Usually it's around ~70 or so for the configuration I had (2 witness sensors and one degree of freedom enabled....no c-code calculations on any other signals).  The "alive" heartbeat was also frozen.

I ssh'ed into c1lsc, ran ./startc1oaf in the scripts directory, and it came back just fine.

Anyhow, I don't know why it got funny, but I wanted to record the event for posterity.  I'm back to OAFing now.

  5889   Mon Nov 14 21:22:48 2011 ranaConfigurationComputersprimetime RSYNC slowing down NODUS

nodus:elog>w; who ; date
  9:20pm  up 44 day(s),  5:14,  5 users,  load average: 0.29, 1.04, 1.35
User     tty           login@  idle   JCPU   PCPU  what
controls pts/1         9:18pm            5         -tcsh
controls pts/2         2:37pm  6:39  25:02  25:02  /opt/rsync/bin/rsync -avW /cvs/c
controls pts/3         9:14pm                      w
controls pts/4         4:20pm  1:56   5:02   5:02  ssh -X rosalba
controls pts/8         8:23pm    47   4:03         -tcsh
controls   pts/1        Nov 14 21:18    (pianosa.martian)
controls   pts/2        Nov 14 14:37    (ldas-cit.ligo.caltech.edu)
controls   pts/3        Nov 14 21:14    (rosalba)
controls   pts/4        Nov 14 16:20    (192.168.113.128)
controls   pts/8        Nov 14 20:23    (gwave-103.ligo.caltech.edu)
Mon Nov 14 21:20:48 PST 2011

we will ask the man to stop running backups at this time of night...

  6016   Sat Nov 26 07:22:20 2011 sureshUpdateComputers 

c1sus has been shutdown so that the optics dont bang around.  This is because the watch dogs are not working.

  6108   Mon Dec 12 16:30:17 2011 JenneUpdateComputersDid someone just do something to fb??

Dataviewer couldn't connect to the framebuilder, so I checked the CDS status screen, and all the fb-related things on each model went white, then red, then computer-by-computer they came back green.  Now dataviewer works again.  Is someone secretly doing shit while not in the lab???  Not cool man!

  6112   Tue Dec 13 11:51:33 2011 JamieUpdateComputersDid someone just do something to fb??

Quote:

Dataviewer couldn't connect to the framebuilder, so I checked the CDS status screen, and all the fb-related things on each model went white, then red, then computer-by-computer they came back green.  Now dataviewer works again.  Is someone secretly doing shit while not in the lab???  Not cool man!

This happens on occasion, and I have reported it to the CDS guys.  Something apparently causes the framebuilder to crash, but I haven't figured out what it is yet.  I doubt this particular instance had anything to do with remote futzing.

  6117   Wed Dec 14 12:22:00 2011 VladimirHowToComputersligo_viewer installed on pianosa

I made a test installation of ligo_viewer in /users/volodya/ligo_viewer-0.5.0c . It runs on pianosa (the Ubuntu machine) and needs Tcl/Tk 8.5.

 

To try it out run the following command on pianosa:

cd /users/volodya/ligo_viewer-0.5.0c/

./ligo_viewer.no_install

 

Press "CONNECT" to connect to the NDS server and explore. There are slides describing ligo_viewer at http://volodya-project.sourceforge.net/Ligo_viewer.pdf

 

Installation notes:

Use /users/volodya/ligo_viewer-0.5.0c.tgz or later version - it has been updated to work with 64 bit machines.

Make sure Tcl and Tk development packages are installed. You can find required packages by running

apt-file search tclConfig.sh

apt-file search tkConfig.sh

If apt-file returns empty output run apt-file update

Unpack ligo_viewer-0.5.0c.tgz, change into the created directory.

Run the following command to configure:

export CFLAGS=-I/usr/include/tcl8.5
./configure --with-tcl=/usr/lib/tcl8.5/ --with-tk=/usr/lib/tk8.5/

This works on Ubuntu machines. --with-tcl and --with-tk should point to the directories containing tclConfig.sh and tkConfig.sh correspondingly.

Run "make".

You can test the compilation with ./ligo_viewer.no_install

If everything works install with make install

If Tcl/Tk 8.5 is unavailable it should work with Tcl/Tk 8.3 or 8.4

 

 

Attachment 1: ligo_viewer_40m2.png
ligo_viewer_40m2.png
ELOG V3.1.3-