40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 92 of 344  Not logged in ELOG logo
ID Date Author Type Categoryup Subject
  6517   Tue Apr 10 23:56:44 2012 ranaUpdateComputersSensoray

Now that Mike has got the Sensoray working, Jenne/Suresh should grab some new images of the ETM cage as Keiko did so that we can analyze them for another mode matching diagnostic.

  6518   Wed Apr 11 12:25:11 2012 RyanUpdateComputersUpdating aLIGO Conlog

Over the next few days, I will be working on upgrading the aLIGO Conlog install to include new bugfixes distributed by Patrick T.  The currently running conlog *should* not be affected, but please let me know if it is (ryan.fisher@ligo.org).

  6530   Thu Apr 12 22:04:17 2012 Mike J.UpdateComputersNew Hysteresis Model & Plots

The new hysteresis model uses a triangle wave with offset zero points as the position function and a sinusoidal force function, creating a loop similar to this. Model is at /users/mjenson/matlab/ferro_hyst.mdl.

ferro_hyst.pnghyst_combo.png

  6539   Tue Apr 17 10:55:50 2012 RyanUpdateComputersUpdating aLIGO Conlog

Quote:

Over the next few days, I will be working on upgrading the aLIGO Conlog install to include new bugfixes distributed by Patrick T.  The currently running conlog *should* not be affected, but please let me know if it is (ryan.fisher@ligo.org).

 The upgrade to the aLIGO Conlog is completed.  The conlog is once again running on megatron in a screen session. (see http://nodus.ligo.caltech.edu:8080/40m/6396)

  6585   Mon Apr 30 18:46:34 2012 ranaUpdateComputersmegatron

Last week I found that megatron was still off after the power outage. Apparently, the power recovery checklist had not been followed during the recovery.

Please remember to use the checklist and post the checklist results after each power outage. Megatron is now on and functioning.

  6592   Tue May 1 17:42:15 2012 Mike J.UpdateComputersSensoray

I've upgraded the Sensoray GUI so it can now switch the video channel it receives, thanks to the videoswitch script.

V4L2_Capture_Demo_r01.png

  6645   Tue May 15 23:40:46 2012 Mike J.UpdateComputersImage Subtraction

I acquired 2 raw frames of MC2 using "/users/mjenson/sensoray/sdk_2253_1.2.2_linux/capture -n -s 720x480 -f 1", one while the laser was off the mode cleaner and another while it was on:

mc2_1.bmp mc2_2.bmp

I then used "/users/mjenson/sensoray/sdk_2253_1.2.2_linux/imsub/display-image.py" to generate bitmaps of the raw images, which I then subtracted using the Python Imaging Library to generate a new image:

mc2_1-mc2_2.bmp

It doesn't look all that different, but the first image didn't have that much lit up in it to begin with. I should be able to write a script that does all of this without needing to generate new files in between acquisition and subtraction.

  6646   Wed May 16 11:53:45 2012 JenneUpdateComputersImage Subtraction

Quote:

It doesn't look all that different, but the first image didn't have that much lit up in it to begin with.

 This is totally cool!  You can see that the OSEM lights are almost entirely gone in the subtracted image.

Can you switch to trying with one of the *TM*F cameras?  (ITMXF, ITMYF, ETMYF, ETMXF)  They tend to have more background, so there should be a more dramatic subtraction.  Den or Suresh should be able to lock one of the arms for you.

  6662   Tue May 22 20:24:06 2012 JamieUpdateComputersrossa is now running Ubuntu 10.04

Now same as pianosa and rosalba.  I'll upgrade allegra on Friday.

  6677   Thu May 24 16:13:05 2012 yutaUpdateComputersASS scripts on new ubuntu machines

[Den, Yuta]

Background:
 ASS and many other scripts don't work on new ubuntu machines.

What we did:
1. Installed C-shell on rossa and rosalba(Ubuntu machine).
  sudo apt-get insall csh

2. Find out that
  /opt/rtcds/caltech/c1/scripts/AutoDither/alignY

runs, but
  /opt/rtcds/caltech/c1/scripts/medmrun /opt/rtcds/caltech/c1/scripts/AutoDither/alignY

doesn't run. It gives us the following error messages.

ezcawrite: error while loading shared libraries: libca.so: cannot open shared object file: No such file or directory
ezcaswitch: error while loading shared libraries: libca.so: cannot open shared object file: No such file or directory

Result:
 ASS scripts run on rossa and rosalba, but not with medmrun.
 At least ASS scripts run on pianosa(ubuntu machine) with medmrun. So we decided to wait for JAMIE to fix it.

  6683   Fri May 25 16:58:54 2012 JamieConfigurationComputers.bashrc for workstations

I have setup a shared .bashrc for all the workstations that is symlinked to the normal location on all machines:

controls@rossa:~ 0$ ls -al /home/controls/.bashrc 
lrwxrwxrwx 1 controls controls 23 2012-05-25 15:37 /home/controls/.bashrc -> /users/controls/.bashrc
controls@rossa:~ 0$ 

This should help simplify maintenance considerably.  Editing that file on one machine will edit it for all.  Just edit this one file!  Don't try to get fancy and add extra files!

I also added a bunch of aliases that had previously been missing.  This should help with some of the problems that people had been having.

NOTE: PLEASE DO NOT CHANGE THE DEFAULT SHELL!  We are using bash, because that's what the sites are now using and we want to be as compatible as possible.

You can of course still write scripts in csh/tcsh or use tcsh in a shell if you wish.   Just don't change the default shell for the controls user.

  6684   Fri May 25 17:50:38 2012 JamieUpdateComputersASS scripts on new ubuntu machines

Quote:

[Den, Yuta]

Background:
 ASS and many other scripts don't work on new ubuntu machines.

What we did:
1. Installed C-shell on rossa and rosalba(Ubuntu machine).
  sudo apt-get insall csh

2. Find out that
  /opt/rtcds/caltech/c1/scripts/AutoDither/alignY

runs, but
  /opt/rtcds/caltech/c1/scripts/medmrun /opt/rtcds/caltech/c1/scripts/AutoDither/alignY

doesn't run. It gives us the following error messages.

ezcawrite: error while loading shared libraries: libca.so: cannot open shared object file: No such file or directory
ezcaswitch: error while loading shared libraries: libca.so: cannot open shared object file: No such file or directory

Result:
 ASS scripts run on rossa and rosalba, but not with medmrun.
 At least ASS scripts run on pianosa(ubuntu machine) with medmrun. So we decided to wait for JAMIE to fix it.

Apparently the environment was not being properly inherited by the scripts launched from medmrun.  We modified the medmrum script so that it executes things with an interactive shell ("bash -i -c ...") and this fixed the problem (by assuring that it sources all the interactive environment configs (i.e. ~/.bashrc)).  I'm still not sure why we were seeing different behavior on pianosa, but at least the solution we have now should be robust.

As a reminder, all scripts launched from MEDM should use medmrun:

/opt/rtcds/caltech/c1/scripts/medmrun
  6685   Fri May 25 17:52:08 2012 JamieUpdateComputersallegra now running Ubuntu 10.04

The last of the control room machines is now upgraded.

  6703   Tue May 29 15:29:16 2012 JamieUpdateComputerslatest pynds installed on all new control room machines

The DASWG lscsoft package repositories have a lot of useful analysis software.  It is all maintained for Debian "sqeeze", but it's mostly installable without modification on Ubuntu 10.04 "lucid" (which is based on Debian squeeze).  Basically the only thing that needs to access the lscsoft repositories is to add the following repository file:

controls@rossa:~ 0$ cat /etc/apt/sources.list.d/lscsoft.list 
deb http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze contrib
deb-src http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze contrib

deb http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze-proposed contrib
deb-src http://www.lsc-group.phys.uwm.edu/daswg/download/software/debian/ squeeze-proposed contrib
controls@rossa:~ 0$ 

A simple "apt-get update" then makes all the lscsoft packages available.

lscsoft includes the nds2 client packages (nds2-client-lib) and pynds (python-pynds).  Unfortunately the python-pynds debian squeeze package currently depends on libboost-python1.42, which is not available in Ubuntu lucid.  Fortunately, pynds itself does not require the latest version and can use what's in lucid.  I therefore rebuilt the pynds package on one of the control room machines:

$ apt-get install dpkg-dev devscripts debhelper            # these are packages needed to build a debian/ubuntu package
$ apt-get source python-pynds                              # this downloads the source of the package, and prepares it for a package build
$ cd python-pynds-0.7
$ debuild -uc -us                                          # this actually builds the package
$ ls -al ../python-pynds_0.7-lscsoft1+squeeze1_amd64.deb
-rw-r--r-- 1 controls controls 69210 2012-05-29 11:57 python-pynds_0.7-lscsoft1+squeeze1_amd64.deb

I then copied the package into a common place:

/ligo/apps/debs/python-pynds_0.7-lscsoft1+squeeze1_amd64.deb

I then installed it on all the control room machines as such:

$ sudo apt-get install libboost-python1.40.0 nds2-client-lib python-numpy   # these are the dependencies of python-pynds
$ sudo dpkg -i /ligo/apps/debs/python-pynds_0.7-lscsoft1+squeeze1_amd64.deb

I did this on all the control room machines.

It looks like the next version of pynds won't require us to jump through these extra hoops and should "just work".

  6722   Thu May 31 00:56:13 2012 JamieMetaphysicsComputersPlease remember to check in code changes

I know it's really hard to remember, but our future selves will thank us dearly if we remember to commit all of our code changes to the svn with nice log messages.  At the moment there's a LOT of modified stuff in the userapps working directory that needs to be committed:

controls@pianosa:/opt/rtcds/userapps/release 0$ svn status | grep '^M'
M       cds/c1/models/c1rfm.mdl
M       sus/c1/medm/templates/SUS_SINGLE.adl
M       sus/c1/models/c1mcs.mdl
M       sus/c1/models/c1sus.mdl
M       sus/c1/models/c1scx.mdl
M       sus/c1/models/c1scy.mdl
M       isc/c1/models/c1pem.mdl
M       isc/c1/models/c1ioo.mdl
M       isc/c1/models/ADAPT_XFCODE_MCL.c
M       isc/c1/models/c1oaf.mdl
M       isc/c1/models/c1gcv.mdl
M       isc/common/medm/OAF_OVERVIEW.adl
M       isc/common/medm/OAF_DOF_BLRMS.adl
M       isc/common/medm/OAF_OVERVIEW_BAK.adl
M       isc/common/medm/OAF_ADAPTATION_MICH.adl
controls@pianosa:/opt/rtcds/userapps/release 0$ 

This doesn't even include things that haven't even been added yet.  It doesn't take much time.  Just copy and paste what you elog about the changes.

  6737   Fri Jun 1 02:33:40 2012 JenneUpdateComputersc1sus and c1iscex - bad fb connections

Something bad happened to c1sus and c1iscex ~20 min ago.  They both have "0x2bad" 's.  I restarted the daqd on the framebuilder, and then rebooted c1sus, and nothing changed.  The SUS screens are all zeros (the gains seem to be set correctly, but all of the signals are 0's).

If it's not fixed when I get in tomorrow, I'll keep poking at it to make it better.

  6738   Fri Jun 1 08:01:46 2012 steveUpdateComputersc1sus and c1iscex are down

Quote:

Something bad happened to c1sus and c1iscex ~20 min ago.  They both have "0x2bad" 's.  I restarted the daqd on the framebuilder, and then rebooted c1sus, and nothing changed.  The SUS screens are all zeros (the gains seem to be set correctly, but all of the signals are 0's).

If it's not fixed when I get in tomorrow, I'll keep poking at it to make it better.

 

 

  6740   Fri Jun 1 09:50:50 2012 JamieUpdateComputersc1sus and c1iscex - bad fb connections

Quote:

Something bad happened to c1sus and c1iscex ~20 min ago.  They both have "0x2bad" 's.  I restarted the daqd on the framebuilder, and then rebooted c1sus, and nothing changed.  The SUS screens are all zeros (the gains seem to be set correctly, but all of the signals are 0's).

If it's not fixed when I get in tomorrow, I'll keep poking at it to make it better.

 This is at least partially related to the mx_stream issue I reported previously.  I restarted mx_stream on c1iscex and that cleared up the models on that machine.

Something else is happening with c1sus.  Restarting mx_stream on c1sus didn't help.  I'll try to fix it when I get over there later.

  6742   Fri Jun 1 14:40:24 2012 JamieUpdateComputersc1sus and c1iscex - bad fb connections

Quote:

This is at least partially related to the mx_stream issue I reported previously.  I restarted mx_stream on c1iscex and that cleared up the models on that machine.

Something else is happening with c1sus.  Restarting mx_stream on c1sus didn't help.  I'll try to fix it when I get over there later.

I managed to recover c1sus.  It required stopping all the models, and the restarting them one-by-one:

$ rtcds stop all     # <-- this does the right to stop all the models with the IOP stopped last, so they will all unload properly.

$ rtcds start iop

$ rtcds start c1sus c1mcs c1rfm

I have no idea why the c1sus models got wedged, or why restarting them in this way fixed the issue.

  6761   Wed Jun 6 00:37:11 2012 JenneUpdateComputersmx_stream restarted on iscey, sus

Just as Yuta and I were sitting down to look at our beatnote (really the output of the freq discriminator) on Dataviewer, all the FB net boxes on the CDS status screen were white.  I restarted daqd, and most of the computers came back fine.  c1iscey and c1sus still had some red bad boxes.  So I restarted mx_stream on both, and they are now fine.

Somehow, this also fixed whatever I had done to the lsc model yesterday (although I think TRY is still not recorded at this time - not messing with it yet).

  6765   Wed Jun 6 14:17:56 2012 steveUpdateComputersiMac ordered

Rana, Steve,

 


from Apple Store on line

  • $2,139.00

27-inch iMac

  • Part number: Z0M6

Configuration

  • 2.7GHz Quad-Core Intel Core i5
  • 16GB 1333MHz DDR3 SDRAM - 4x4GB
  • 1TB Serial ATA Drive
  • AMD Radeon HD 6770M 512MB GDDR5
  • Apple Mouse
  • Apple Keyboard with Numeric Keypad (English) & User's Guide
  • Accessory Kit
  6772   Wed Jun 6 22:04:19 2012 JamieUpdateComputersFailed attempt to get pianosa to support dual monitors

I've spent an inordinate amount of time trying to figure out how to get pianosa to support dual monitors.  It apparently has some special nvidia graphics that are not well supported in Ubuntu (10.04 at least).  I've tried installing a newer kernel (3.0.0) and installing the special nvidia compile-from-source Ubuntu packages, but nothing is working.  And now unfortunately he's not giving me any video at all.  I'm sick of this today, so I'll try again tomorrow.

In the future we should avoid these bullshit nvidia cards like the plague.

  6773   Wed Jun 6 22:23:53 2012 JamieUpdateComputersFailed attempt to get pianosa to support dual monitors

I managed to get pianosa working again with just a single monitor.  I'm done trying to configure it for dual, though.

  6774   Wed Jun 6 22:25:05 2012 JamieUpdateComputerszita now running ubuntu 10.04

I'll finish the rest of the config to turn it into a full-fledged workstation tomorrow.

  6783   Thu Jun 7 10:39:21 2012 JamieUpdateComputerspianosa dual monitor working

I decided to remove what I thought was the problematic extra nVidia video card, since there are already two DVI outputs build-in.  The card turned out to not even be nVidia, so I don't know what was going on there.

I futzed with the BIOS to configure the primary video card, which is some new Intel card.  The lucid (10.04) support for it is lacking, but it was easy enough to pull in new drivers from the appropriate Ubuntu PPA repository:

controls@pianosa:~ 0$ sudo apt-add-repository ppa:f-hackenberger/x220-intel-mesa
controls@pianosa:~ 0$ sudo apt-add-repository ppa:glasen/intel-driver
controls@pianosa:~ 0$ sudo apt-get update
...
controls@pianosa:~ 0$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  libdrm-intel1 libdrm-nouveau1 libdrm-radeon1 libdrm2 libgl1-mesa-dri libgl1-mesa-glx libglu1-mesa xserver-xorg-video-intel
8 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,212kB of archives.
After this operation, 25.2MB disk space will be freed.
Do you want to continue [Y/n]?
...
controls@pianosa:~ 0$

After a reboot, both monitors came up fine.

http://www.subcritical.org/running_ubuntu_lts_on_sandy_bridge/

  6784   Thu Jun 7 12:49:50 2012 JamieUpdateComputerszita network configured

Added zita to the martian host table, and configured his network accordingly:

zita            A       192.168.113.217

https://wiki-40m.ligo.caltech.edu/Martian_Host_Table

  6795   Mon Jun 11 22:51:11 2012 JenneUpdateComputerstdsavg not working

tdsavg isn't working:

controls@rossa:/opt/rtcds/caltech/c1/scripts/LSC 6$ tdsavg 10 C1:LSC-ASDC_IN1
ERROR: LDAQ - Unable to find NDS host "fb0"
ERROR: LDAQ - Unable to find NDS host "fb1"
ERROR: LDAQ - Unable to open socket to NDS.

 

When this command is executed inside a script, it doesn't return anything.  eg:

set offset = `tdsavg 10 C1:LSC_ASDC_IN1`
echo $offset

returns a blank line. 

Past elog research said lots of things about test points.  I didn't suspect that, since there aren't many test points occupied (according to the CDS status screens), but I cleared the test points anyway (elog 6319).  Didn't change anything, still broken.

LSCoffsets script, and any others depending on tdsavg will not work until this is fixed.

  6796   Tue Jun 12 00:15:51 2012 JenneUpdateComputersrtcds: command not found

We can't compile any changes to the LSC or the GCV models since Jamie's new script / program isn't found.  I don't know where it is (I can't find it either), so I can't do the compiling by hand, or point explicitly to the script.  The old way of compiling models in the wiki is obsolete, and didn't work :(

This means we can't (a) record TRY or (b) add the Q quadrature of the beat PD to the real time system tonight.

We're going to try just using Yuta's pynds script to capture data in real time, so we can keep working for tonight.

  6797   Tue Jun 12 01:03:18 2012 JenneUpdateComputersmini boot fest

As usual, we noticed the frame builder wasn't connecting happily with the rest of the computers just as we were about to lock some stuff (we never notice it being bad when we're not trying to use the frame builder....)

All the big rectangles by each computer were white.  I restarted daqd, and that brought most things back.  c1lsc and c1sus needed their mx_streams restarted manually to get everything green again.

stock-vector-wine-icon-on-computer-keyboard-original-illustration-58731781.jpg

I've had about enough whine with my computers for tonight.

  6800   Tue Jun 12 02:09:43 2012 JenneUpdateComputersmini boot fest

Quote:

stock-vector-wine-icon-on-computer-keyboard-original-illustration-58731781.jpg

 I'm starting to feel like a wine-o here.  Yuta wanted to glance at the PRM oplev dataviewer, and lo and behold, the fb lost connection just as he decided to do that.  We had checked the front end status screen not 1 minute beforehand, and everything was green.  Lame.

  6801   Tue Jun 12 11:25:05 2012 JamieUpdateComputersrtcds: command not found

Quote:

We can't compile any changes to the LSC or the GCV models since Jamie's new script / program isn't found.  I don't know where it is (I can't find it either), so I can't do the compiling by hand, or point explicitly to the script.  The old way of compiling models in the wiki is obsolete, and didn't work :(

Sorry about that.  I had modified the path environment that pointed to the rtcds util.  The rtcds util is now in /opt/rtcds/caltech/c1/scripts/rtcds, which is in the path.  Starting a new shell should make it available again.

  6807   Tue Jun 12 17:46:09 2012 JenneUpdateComputersrtcds: command found

Quote:

Quote:

We can't compile any changes to the LSC or the GCV models since Jamie's new script / program isn't found.  I don't know where it is (I can't find it either), so I can't do the compiling by hand, or point explicitly to the script.  The old way of compiling models in the wiki is obsolete, and didn't work :(

Sorry about that.  I had modified the path environment that pointed to the rtcds util.  The rtcds util is now in /opt/rtcds/caltech/c1/scripts/rtcds, which is in the path.  Starting a new shell should make it available again.

 Added TRX and TRY and POY11_I_ERR and POX11_I_ERR to the c1lsc.mdl using a new-style DAQ Channels block, recompiled, installed, started the model, all good.  Restarted the daqd on the framebuilder, and everything is green.  I can go back and get recorded data using dataviewer (for the last few minutes since I started fb), so it all looks good.

Note on the new DAQ Channels block:  Put the text block (from CDS_PARTS) at the same level as the channel you want to save, and name it exactly as it is in the model.  The code-generator will add the _DQ for you.  i.e. if you define a channel "TRY_OUT_DQ" in the lsc model, you'll end up with a channel "C1:LSC-TRY_OUT_DQ_DQ".

  6830   Mon Jun 18 17:28:03 2012 yutaSummaryComputersbugs in CDS_PARTS/simLinkParts/Fcn

Fcn module in CDS_PARTS is used to include a user defined function in a model.
We should be able to use this by entering desired function, but I found some bugs.

BUG1: Fcn doen't work without ";"

If you put ";" after the function, we can compile.

 sin(u[1]);

But if you put without ";", like

 sin(u[1])

you get the following error message when compiling.

controls@c1ioo
~ 0$ rtcds make c1gcv
### building c1gcv...
Cleaning c1gcv...
Done
Parsing the model c1gcv...
Done
Building EPICS sequencers...
Done
Building front-end Linux kernel module c1gcv...
echo >> target/c1gcvepics/README.making_changes
echo 'Built on date' `date` >> target/c1gcvepics/README.making_changes
make[1]: Leaving directory `/opt/rtcds/caltech/c1/rtbuild'

make[1]: Entering directory `/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv'
make -C /lib/modules/2.6.34.1/build SUBDIRS=/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv modules
make[2]: Entering directory `/usr/src/linux-2.6.34.1-cs'
  CC [M]  /opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv/c1gcv.o
make[2]: Leaving directory `/usr/src/linux-2.6.34.1-cs'
make[1]: Leaving directory `/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv'
/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv/c1gcv.c: In function 'feCode':
/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv/c1gcv.c:615: error: expected expression before ';' token
make[3]: *** [/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv/c1gcv.o] Error 1
make[2]: *** [_module_/opt/rtcds/caltech/c1/rtbuild/src/fe/c1gcv] Error 2
make[1]: *** [default] Error 2
make: *** [c1gcv] Error 1


BUG2: sindeg doesn't work properly

sindeg should work as cosine with input in degrees.
I made a simple model to test this(below).
model_sindegbug.png


Output of the filter module C1:ALS-BEATY_FINE_PHASE goes to "PHASE_in"
sindeg of this goes to C1:ALS-BEATY_FINE_I_ERR
cosdeg of this goes to C1:ALS-BEATY_FINE_Q_ERR

If you sweep the phase input, you should get sin and cos, but you get the following.
cosdeg (C1:ALS-BEATY_FINE_Q_ERR) looks OK, but sindeg (C1:ALS-BEATY_FINE_I_ERR) looks funny. It looks like ~20000 is its period.

dv_sindegbug.png

  6854   Fri Jun 22 13:37:17 2012 JenneUpdateComputersfb lost connection

...Perhaps related to the fact that Jamie is copying a lot of stuff over the network to back up Ottavia before converting her to Ubuntu, perhaps totally independent. 

After restarting the daqd, c1lsc was the only computer whose mx_stream came up on its own.  I restarted c1sus. c1ioo, c1iscey, c1iscex by hand.

  6856   Fri Jun 22 19:52:47 2012 JamieUpdateComputersottavia reconfigured as CDS workstation

ottavia has been reinstalled with Ubuntu 10.04 LTS, and has been configured as a CDS workstation.

I have been maintaining a script that takes a stock 10.04 install and configures it as a workstation.  I've attached it here, but it lives at:

/users/controls/workstation-setup.sh

The script is designed to be idempotent, i.e. it can be run on a machine that has already been configured and it will either have no affect or update.

  6861   Sat Jun 23 19:57:22 2012 yutaSummaryComputersc1ioo is down

I tried to restart c1ioo becuase I can't live without him.

I couldn't ssh or ping c1ioo, so I did hardware reboot.
c1ioo came back, but now ADC/DAC stats are all red.

c1ioo was OK until 3am when I left the control room last night. I don't know what happened, but StripTool from zita tells me that MC lock went off at around 4pm.

  6865   Mon Jun 25 10:35:59 2012 JenneSummaryComputersc1ioo is down

Quote:

I tried to restart c1ioo becuase I can't live without him.

I couldn't ssh or ping c1ioo, so I did hardware reboot.
c1ioo came back, but now ADC/DAC stats are all red.

c1ioo was OK until 3am when I left the control room last night. I don't know what happened, but StripTool from zita tells me that MC lock went off at around 4pm.

 c1ioo was still all red on the CDS status screen, so I tried a couple of things.

mxstreamrestart (which aliases on the front ends to sudo /etc/init.d/mx_stream restart) didn't help

sudo shutdown -r now didn't change anything either....c1ioo came back with red everywhere and 0x2bad on the IOP

eventually doing as Jamie did for c1sus in elog 6742, rtcds stop all, then rtcds start all fixed everything.  Interestingly, when I tried rtcds start iop, I got the error
Cannot start/stop model 'iop' on host c1ioo, so I just tried rtcds start all, and that worked fine....started with c1x03, then c1ioo, then c1gcv.

  6866   Mon Jun 25 11:23:14 2012 JenneUpdateComputerstdsavg not working

Quote:

LSCoffsets script, and any others depending on tdsavg will not work until this is fixed.

 LSCoffsets is working again. 

tdsavg (now, but didn't used to) needs "LIGONDSIP=fb" to be specified.  Jamie just put this in the global environment, so tdsavg should just work like normal again.

Also, the rest of the LSCoffsets script (really the subcommand offset2) was tsch syntax, so I created offset3 which is bash syntax.

Now we can use LSCoffsets again.

  6867   Mon Jun 25 11:27:13 2012 JamieSummaryComputersc1ioo is down

Quote:

Quote:

I tried to restart c1ioo becuase I can't live without him.

I couldn't ssh or ping c1ioo, so I did hardware reboot.
c1ioo came back, but now ADC/DAC stats are all red.

c1ioo was OK until 3am when I left the control room last night. I don't know what happened, but StripTool from zita tells me that MC lock went off at around 4pm.

 c1ioo was still all red on the CDS status screen, so I tried a couple of things.

mxstreamrestart (which aliases on the front ends to sudo /etc/init.d/mx_stream restart) didn't help

sudo shutdown -r now didn't change anything either....c1ioo came back with red everywhere and 0x2bad on the IOP

eventually doing as Jamie did for c1sus in elog 6742, rtcds stop all, then rtcds start all fixed everything.  Interestingly, when I tried rtcds start iop, I got the error
Cannot start/stop model 'iop' on host c1ioo, so I just tried rtcds start all, and that worked fine....started with c1x03, then c1ioo, then c1gcv.

There seems to be a problem with how models are coming up on boot since the upgrade.  I think the IOP isn't coming up correctly for some reason, which is then preventing the rest of the models from starting since they depend on the IOP.

The simple way to fix this is to run the following:

ssh c1ioo
rtcds restart all

The "restart" command does the smart thing, by stopping all the models in the correct order (IOP last) and then restarting them also in the correct order (IOP first).

  6912   Wed Jul 4 18:25:44 2012 ZachUpdateComputersNDS2 client now working on Ubuntu machines

After plenty of work, NDS2 can now be used to get site data within MATLAB using the following machines:

  • allegra
  • megatron
  • ottavia
  • pianosa
  • rosalba
  • rossa

What I did

NDS2 was not working on any of the machines, so the first thing I did was simply to install the newest version. I downloaded the latest tarball (0.9.1) from the LDAS Wiki, unzipped and installed it

/users/zach $ tar -xvf nds2-client-0.9.1.tar

/users/zach $ cd nds2-client-0.9.1

/users/zach $ sudo ./configure --prefix=/cvs/cds/caltech/apps/linux64 --with-matlab=/cvs/cds/caltech/apps/linux64/matlab/bin/matlab

/users/zach $ sudo make

/users/zach $ sudo make install

 

Even with the new version, it still didn't work.

Solution: The main problem was that the cyrus-sasl-gssapi authentication protocol was not installed on these machines, so that even with a kerberos ticket the datalink could not be established. Using information from the LDAS Wiki, I used aptitude to install it as:

$ sudo aptitude install lscsoft-auth

This group installs both the SASL protocol and the package python-kerberos

 

I also needed to update the kerberos config file for each machine, which is located at /etc/krb5.conf. I found that ottavia had a nice one with many realms, so I copied that one over to the other machines. In any case where there was an old config file overwritten, it is now /etc/krb5.conf.old.

Finally, the matlab path for NDS2 was still set to the old 2010a directory (/cvs/cds/caltech/apps/linux64/lib/matlab2010a) that was created by the NDS2 install when Rana originally did it. The new install I made above created the appropriate 2010b mexa64 files, so I changed the matlab path within matlab to this one:

>> rmpath /cvs/cds/caltech/apps/linux64/lib/matlab2010a

>> addpath /cvs/cds/caltech/apps/linux64/lib/matlab2010b

>> savepath

 

Now everything works fine on all these machines. As in Rana's original post, you get data in the following way:

$ kinit albert.einstein %then enter password

$ matlab -nosplash -nodesktop

>> d = NDS2_GetData({'H1:LSC-NPTRX_OUT16.mean'},963968415,6000,'nds.ligo.caltech.edu:31200')

 

d = 


            name: 'H1:LSC-NPTRX_OUT16.mean' 

            chan_type: 'm-trend'             

            rate: 0.0167       

            data_type: 'real_8'     

            signal_gain: 1   

            signal_offset: 0     

            signal_slope: 1     

            signal_units: ''   

            start_gps_sec: 963968415     

            duration_sec: 6000             

            data: [100x1 double]           

            exists: 1

 

>> quit % since you've seen that the data is really here

$ kdestroy % so that no one uses your credentials

 

Some thoughts

  • I would like to extend this to the 32-bit machines, but I have to figure out the best way to install the proper NDS2 client without interfering with the 64-bit version. I think it is just a matter of specifying the matlabroot in the .../linux/ instead of .../linux64/
  • It would be nice to find a way that the nice tool gps('MM/DD/YYYY XX:XX:XX UTC'), which calls the ligotool executable tconvert, can be automatically usable when calling NDS2 functions. Right now, there seems to be an issue preventing that: even though tconvert can be run in the terminal, gps() returns an error and even directly running unix('tconvert now') or !tconvert returns the same error. I have emailed Peter Shawhan to see if he has any advice.

 

 

  6919   Thu Jul 5 12:06:35 2012 JamieUpdateComputersNDS2 client now working on Ubuntu machines

What I did

NDS2 was not working on any of the machines, so the first thing I did was simply to install the newest version. I downloaded the latest tarball (0.9.1) from the LDAS Wiki, unzipped and installed it

/users/zach $ tar -xvf nds2-client-0.9.1.tar

/users/zach $ cd nds2-client-0.9.1

/users/zach $ sudo ./configure --prefix=/cvs/cds/caltech/apps/linux64 --with-matlab=/cvs/cds/caltech/apps/linux64/matlab/bin/matlab

/users/zach $ sudo make

/users/zach $ sudo make install

No no, this is totally unnecessary.  NDS2 was already installed on every machine from the official packaged releases (apt-get install nds2-client), and it's known to work fine. We use it with pynds all the time. If the matlab component is not working we should figure out the right way to fix it with the existing packages.

In general, please only manually install software as a very last resort.  Manually installed software doesn't get maintained, where as the officially packaged stuff is being actively maintained by the collaboration. If there is a problem with the distributed packaging we should report it and get it fixed (and hint I was the one who built the original Debian packaging for nds2, so I know how to fix all the issues).  I'm trying to bring the 40m out of the dark days of complete chaos, where random software was installed in random locations.

Even with the new version, it still didn't work. 

That's because this wasn't the problem!

Solution: The main problem was that the cyrus-sasl-gssapi authentication protocol was not installed on these machines, so that even with a kerberos ticket the datalink could not be established. Using information from the LDAS Wiki, I used aptitude to install it as:

$ sudo aptitude install lscsoft-auth

This group installs both the SASL protocol and the package python-kerberos

I also needed to update the kerberos config file for each machine, which is located at /etc/krb5.conf. I found that ottavia had a nice one with many realms, so I copied that one over to the other machines. In any case where there was an old config file overwritten, it is now /etc/krb5.conf.old.

Finally, the matlab path for NDS2 was still set to the old 2010a directory (/cvs/cds/caltech/apps/linux64/lib/matlab2010a) that was created by the NDS2 install when Rana originally did it. The new install I made above created the appropriate 2010b mexa64 files, so I changed the matlab path within matlab to this one:

>> rmpath /cvs/cds/caltech/apps/linux64/lib/matlab2010a

>> addpath /cvs/cds/caltech/apps/linux64/lib/matlab2010b

>> savepath

This sounds like it's more likely the issues. You did the right thing by going to apt to fix the authentication packages.  It's curious to me that you did that here, whereas you went totally out of band for the nds2 client stuff.  Why?

The matlab mex files are the other problem.  But there is also a nds2-client-matlab Debian/Ubuntu package for that as well.  The problem is that the package just distributes the source, and it needs to be compiled.  I'll help figure out a good way to do that.

  • I would like to extend this to the 32-bit machines, but I have to figure out the best way to install the proper NDS2 client without interfering with the 64-bit version. I think it is just a matter of specifying the matlabroot in the .../linux/ instead of .../linux64/

Again, this is handled by the packaging!  Just use apt and the right architecture is installed automatically.

But what 32 bit machines are you referring to?  I think basically everything is 64 bit nowadays.

  • It would be nice to find a way that the nice tool gps('MM/DD/YYYY XX:XX:XX UTC'), which calls the ligotool executable tconvert, can be automatically usable when calling NDS2 functions. Right now, there seems to be an issue preventing that: even though tconvert can be run in the terminal, gps() returns an error and even directly running unix('tconvert now') or !tconvert returns the same error. I have emailed Peter Shawhan to see if he has any advice. 

We are now using lalapps_tconvert for tconvert.  We're not using that ligotools crap anymore.  I've aliased that to tconvert on the command line, but maybe matlab isn't getting the message.  I'll try to think of a more robust solution (e.g. make a wrapper script).

  6921   Thu Jul 5 13:12:12 2012 ZachUpdateComputersNDS2 client now working on Ubuntu machines

From my conversations with JZ and Leo, it seemed there was no package that generated the appropriate mex files. It was clear that the right ones weren't there from the absence of a /cvs/cds/caltech/apps/linux64/lib/matlab2010b directory. I'm sorry if I screwed anything up with pynds, but I have repeatedly asked for help with NDS2+matlab and no one has done anything.

It would be nice to do it via apt if there indeed is a versioned package that can make the mexs. Sorry again if I jumped the gun, but I didn't think anyone was going to do anything.

I guess the only 32-bit machine I can think of is mafalda.

About tconvert, I think the best solution is to make a new wrapper M-file. gps was just a convenient remnant of mDV, but all that we need is some matlab function that can output a GPS time given a date/time string. We can use whatever command-line utility you want.

  6923   Thu Jul 5 16:49:35 2012 JenneUpdateComputersc1sus is funny

I was trying to use a new BLRMs c-code block that the seismic people developed, instead of Mirko's more clunky version, but putting this in crashed c1sus.

I reverted to a known good c1pem.mdl, and Jamie and I did a reboot, but c1sus is still funny - none of the models are actually running. 

rtcds restart all - all the models are happy again, c1sus is fine.

But, we still need to figure out what was wrong with the c-code block.

Also, the BLRMS channels are listed in a Daq Channels block inside of the (new) library part, so they're all saved with the new CDS system which became effective as of the upgrade.  (I made the Mirko copy-paste BLRMS into a library part, including a DAQ channels block before trying the c-code.  This is the known-working version to which I reverted, and we are currently running.)

  6924   Fri Jul 6 01:12:02 2012 JenneUpdateComputersc1sus is fine

Quote:

I was trying to use a new BLRMs c-code block that the seismic people developed, instead of Mirko's more clunky version, but putting this in crashed c1sus.

I reverted to a known good c1pem.mdl, and Jamie and I did a reboot, but c1sus is still funny - none of the models are actually running. 

rtcds restart all - all the models are happy again, c1sus is fine.

But, we still need to figure out what was wrong with the c-code block.

Also, the BLRMS channels are listed in a Daq Channels block inside of the (new) library part, so they're all saved with the new CDS system which became effective as of the upgrade.  (I made the Mirko copy-paste BLRMS into a library part, including a DAQ channels block before trying the c-code.  This is the known-working version to which I reverted, and we are currently running.)

 The reason I started looking at BLRMS and c1sus today was that the BLRMS striptool was totally wacky.  I finally figured out that the pemepics hadn't been burt restored, so none of the channels were being filtered.  It's all better now, and will be even better soon when Masha finishes updating the filters (she'll make her own elog later)

  6927   Fri Jul 6 08:38:04 2012 steveUpdateComputerstime out

Why do we have these timing blanks?

  6928   Fri Jul 6 09:00:34 2012 not ZachUpdateComputersNDS2 client now working on Ubuntu machines

Quote:

From my conversations with JZ and Leo, it seemed there was no package that generated the appropriate mex files. It was clear that the right ones weren't there from the absence of a /cvs/cds/caltech/apps/linux64/lib/matlab2010b directory. I'm sorry if I screwed anything up with pynds, but I have repeatedly asked for help with NDS2+matlab and no one has done anything.

It would be nice to do it via apt if there indeed is a versioned package that can make the mexs. Sorry again if I jumped the gun, but I didn't think anyone was going to do anything.

There is a package that provides the mex source, but it doesn't actually provide the mex binaries.  The problem is that the binary depends on the matlab version, so you can't possibly provide binaries for every version.

The solution is to just build the binaries from the source package.  We should put together a nice script that builds the binaries from the source, and installs them in the directory of your choosing.  If we get something nice working, we can probably get them to include it with the package, to make it easier in the future.

Here's what's included in the source package:

controls@pianosa:~ 0$ sudo apt-get install nds2-client-matlab
...
controls@pianosa:~ 0$ dpkg -L nds2-client-matlab | sort
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nds2-client-matlab
/usr/share/doc/nds2-client-matlab/changelog.Debian.gz
/usr/share/doc/nds2-client-matlab/changelog.gz
/usr/share/doc/nds2-client-matlab/copyright
/usr/share/matlab
/usr/share/matlab/NDS2_GetChannels.m
/usr/share/matlab/NDS2_GetData.m
/usr/share/matlab/NDS_GetChannels.m
/usr/share/matlab/NDS_GetData.m
/usr/share/matlab/NDS_GetMinuteTrend.m
/usr/share/matlab/NDS_GetSecondTrend.m
/usr/share/matlab/src
/usr/share/matlab/src/NDS2_GetChannels.c
/usr/share/matlab/src/NDS2_GetData.c
/usr/share/matlab/src/NDS_GetChannels.c
/usr/share/matlab/src/NDS_GetData.c
/usr/share/matlab/src/nds_mex_utils.c
/usr/share/matlab/src/nds_mex_utils.h
controls@pianosa:~ 0$ 
  6944   Mon Jul 9 11:27:27 2012 JenneUpdateComputersc1oaf has been down for several days - BURT restore wasn't done correctly on startup

The c1oaf model hasn't been running for a few days (since the leap second problems we were having last week).  I had looked into it, but finally figured it out (with Jamie's help) today. 

The BURT restore has to be given to the model during startup, but for whatever reason it wasn't BURT restoring until *after* the model had already failed to start.  The symptoms were:  no 'heartbeat' for the oaf model, no connection to the fb, NO SYNC on the GDS screen, 0x4000.  the BURT restore button was green, which threw me off the scent, but that's just because it did, in fact, get set, just way too late.

I ended up looking in the dmesg of the lsc computer, and the last set of stuff was several lines of "[3354303.626446] c1oaf: Epics burt restore is 0".  Nothing else was written after that.  Jamie pointed out that this meant the BURT restore wasn't getting sent before the model unloaded itself and decided not to run.

The solution:  restart the model, and manually click the BURT restore button as soon as you're able (after everything comes back from being white).  We used to have to do this, but then there was a "fix", which apparently isn't super robust and failed for the oaf (even though it used to work just fine).  Bugzilla report submitted.

  6993   Thu Jul 19 15:55:21 2012 ranaUpdateComputersused 'aptitude' to install software (TexLive) on rosalba

I used APTITUDE to install texlive-full on rosalba so that I could work on a paper. pdflatex was not found on pianosa, rosalba, megatron, etc. I used this command:

sudo aptitude install texlive-full

While this was installing, I read a bunch of forums which claim that its better to bypass the apt-get and use the 
TexLive installer instead so that we can use its own package updater 'tlmgr'. Otherwise, the standard Ubuntu distributions
are years out of date (e.g. doesn't have RevTex 4.1).



  7013   Mon Jul 23 20:34:38 2012 JamieOmnistructureComputersCHECK IN YOUR CHANGES TO THE SVN

I'm seeing LOTS OF STUFF NOT CHECKED INTO THE SVN!!!  both modified things that haven't been updated, and things that looked like they haven't been checked in at all.

Please check in your stuff to the SVN!  We need the record!

Look through EVERYTHING that you think you might have touched, or even care about, and make sure it's checked in.

  7138   Fri Aug 10 09:47:19 2012 MashaUpdateComputersFE Status

The c1lsc and c1sus screens are red in the front-end status. I restarted the frame builder, and hit the "global diag reset" button, but to no avail. Yesterday, the only thing Den and I did to c1sus was install a new c1pem model. I got rid of the changes and switched to the old one (I ran rtcds build, install, restart), but the status is still the same.

ELOG V3.1.3-