40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 147 of 339  Not logged in ELOG logo
ID Date Author Type Category Subjectup
  11678   Fri Oct 9 11:41:09 2015 ericqUpdateVACN2 Pressure fell

At 10:02AM, the N2 Pressure fell below 60 PSI. The watch script saw this happen, but I did not recieve the email it is supposed to send frown

C1:Vac-P1_pressure reads 7e-4, which is the same as it has for the past ~2 days, so the V1 interlock worked fine. 

I've put some fresh N2 into the system, and Bob will pop in over the weekend to check it. I'll stay on top of it until Steve gets back. 

After consulting ELOGs and the 40m wiki, I reasoned it was ok to open the V1 to reconnect the turbo pump to the main IFO volume and VM1 to reconnect the RGA, and have now done so. 

  15354   Tue May 26 10:04:54 2020 JordanUpdateGeneralN2 Replacement

Replaced empty N2 tank, left tank at ~2000 psi, right tank ~2600 psi.

  15403   Tue Jun 16 16:05:26 2020 JordanUpdateGeneralN2 Replacement

I replaced an empty N2 cylinder, there are now two empty tanks in the outside rack.

  12293   Tue Jul 12 18:13:19 2016 KojiUpdateVACN2 bottle replaced

Gautam, Koji

We replaced the right N2 bottle as it was empty.

  15314   Thu Apr 30 07:29:01 2020 ChubUpdateVACN2 delivered.

Hi All,

The new nitrogen cylinders were delivered to the rack at the west entrance.  We only get one Airgas delivery per week during the stay-at-home order, but so far they've not let us down.

  14331   Tue Dec 4 18:24:05 2018 gautamOmnistructureGeneralN2 line disconnected

[jon, gautam]

In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. When we went into the drill-press area, we heard a hiss, turns out that one of the cylinders is leaking (to be fair, this was labelled, but i thought it isn't great to have a higher N2 concentration in an enclosed space). Since we don't need any actuation ability, I valved off the leaky cylinder, and disconnected the other properly functioning one. Attachment #1 shows the current state.

Attachment 1: IMG_7195.JPG
IMG_7195.JPG
  14333   Thu Dec 6 17:33:33 2018 JonOmnistructureGeneralN2 line disconnected

I believe I finally have the N2 gauge working correctly. The wiring is unchanged from its original state and the controller has been recalibrated.

After letting the line pressure drop to 0 PSI as indicated by the analog gauge in the drill-press room, I recorded the number of counts read by the Omega controller. Then I pressurized the line to 80 PSI, again indicated by the analog gauge, and recorded the Omega counts again. I entered these two reference points into the controller (automatically determines the gain and offset from these), then confirmed the readings to agree with the anaog gauge as I varied the line pressure.

The two reference points are:

0 PSI  :  13 counts
80 PSI : 972 counts

 

Quote:

[jon, gautam]

In the latest installment in this puzzler: turns out that maybe the trend of the "N2 pressure" channel increasing over the ~3 day timescale it takes a cylinder of N2 to run out is real, and is a feature of the way our two N2 cylinder lines/regulators are setup (for the automatic switching between cylinders when one runs out). In order to test this hypothesis, we'd like to have the line pressure be 0 initially, and then just have 1 cylinder hooked up. 

 

  1212   Thu Jan 1 01:15:45 2009 YoichiSummaryVACN2 line leak ?
I've been replacing the N2 bottles recently.
I noticed that the consumption is too high. I had to replace them every two days.
Normally the interval is three or more days.
I suspect there is some leak in the line.

Strangely, it is always the left hand bottle which goes empty. The right hand bottle has been
there for more than a week at about 1000 psi.

We should check it when Steve is back.
  14377   Fri Dec 21 11:13:13 2018 gautamOmnistructureVACN2 line valved off

Per the discussion yesterday, I valved off the N2 line in the drill press room at 11 am PST today morning so as to avoid any accidental software induced gate-valve actuation during the holidays. The line pressure is steadily dropping...

Attachment #1 shows that while the main volume pressure was stable overnight, the the pumpspool pressure has been steadily rising. I think this is to be expected as the turbo pumps aren't running and the valves can't preserve the <1mtorr pressure over long timescales?

Attachment #2 shows the current VacOverview MEDM screen status.

Attachment 1: VacGauges.png
VacGauges.png
Attachment 2: Screenshot_from_2018-12-21_13-02-06.png
Screenshot_from_2018-12-21_13-02-06.png
  14379   Fri Dec 21 12:57:10 2018 KojiOmnistructureVACN2 line valved off

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

  14383   Fri Jan 4 10:25:19 2019 JonOmnistructureVACN2 line valved off

Yes, for TP2 and TP3. They both have a small vent valve that opens automatically on shutdown.

Quote:

Independent question: Are all the turbo forelines vented automatically? We manually did it for the main roughing line.

 

 

  11260   Mon Apr 27 14:37:55 2015 SteveUpdateVACN2 pneumatic pressure watch

 

Quote:

Based on Jenne's chiara disk usage monitoring script, I made a script that checks the N2 pressure, which will send an email to myself, Jenne, Rana, Koji, and Steve, should the pressure fall below 60psi. I also updated the chiara disk checking script to work on the new Nodus setup. I tested the two, only emailing myself, and they appear to work as expected. 

The scripts are committed to the svn. Nodus' crontab now includes these two scripts, as well as the crontab backup script. (It occurs to me that the crontab backup script could be a little smarter, only backing it up if a change is made, but the archive is only a few MB, so it's probably not so important...)

This watch script gives little time to replace N2 cylinder. When the regulated supply drops below 60 psi the cylinder pressure is 60 psi too.

It is more of a statement that V1 is closed and act accordingly. It's only practical if you are in the lab.

Rana pointed it out correctly that we need this message 24 hrs before it happens. This requires monitoring  of total supplies , not the regulated one.

So we need pressure transducers on each nitrogen cylinder, before the regulator. The sum of the two N2 cylinder when they are full 4000 - 4500 psi

The first email should be send out at 1000 psi as sum of the two cylinders. This means that you have  1 day to replace nitrogen cylinder.

 Most of the time the daily consumption is 750 +-50 psi   

However sometimes  this variation goes up   ~750 +-150 psi

  11264   Thu Apr 30 16:30:25 2015 SteveUpdateVACN2 pneumatic pressure watch set up

We have 2 transduser PX303-3KG5V   http://www.omega.com/pressure/pdf/PX303.pdf They  will be installed on the out put of the N2 cylinders to read the supply pressure.

I will order one DC power supply  http://www.omega.com/pptst/PSU93_FPW15.html      PSU-93

One full cylinder pressure is ~ 2400 PSI max so two of them will give us ~9Vdc

The email reminder should be send at 1000 PSI  =  1.8 V

 

 

 

 

  14325   Fri Nov 30 15:53:52 2018 JonOmnistructureGeneralN2 pressure gauge fix

I've made a repair to the N2 pressure monitor. I don't believe the polarity of the analog signal into the controller actually was reversed. I found the data sheet (attached) for the transducer model we have installed. Its voltage should read ~0 at 0 PSI and 100mV at 100 PSI. As wired, the input voltage reads +80 mV as it should.

The controller calibrates the sensor voltage to PSI (i.e., applies a scale and offset) based on two settable reference points which appeared to be incorrect. I changed them to:

  1. 0 mV = 0 PSI (neglecting the small dark bias)
  2. 100 mV = 100 PSI

After the change, the pressure reads 80 PSI. Let's see if the time history now shows a sensible trend. 

Quote:

[koji, gautam, jon, steve]

  • We suspect analog voltage from N2 pressure gauge is connected to interfacing Omega controller with the 'wrong' polarity (i.e pressure is rising over ~4 days and then rapidly falling instead of the other way around). This should be fixed.

 

Attachment 1: N2_transducer_datasheet.pdf
N2_transducer_datasheet.pdf
  12195   Fri Jun 17 15:22:31 2016 gautamUpdateVACN2 supply line restored after retiling
Quote:

The drill room floor will be retiled Thursday, June 16. Temporary nitrogen line set up will allow emptying the hole area.

 

Ifo room entry will be through control room.

 

The retiling work has finished, Steve and I restored the N2 supply configuration to its normal state. The sequence of steps followed was:

  1. Went to the X end and closed the following valves, roughly in this order: VAEE, VAEV, VABS, VABSCI, VASV, VASE, V4, V1.
  2. Checked the RPM on the various turbo pump controllers to make sure they were in their nominal states
  3. Disconnect the electrical connections to V1, V4, V5 and VA6 - just to make sure some spurious signal doesn't unintentionally open any of these valves while we are mucking around with the N2 supply
  4. Close the valves on the N2 cylinders in the drill room. Disconnect the temporary nitrogen line (at this point, the N2 pressure to the IFO valves goes down from ~7-PSI to 0), reconnect the old supply chain, taking care that we aren't unintentionally loosening any of the Swagelock connections while unscrewing stuff
  5. Replaced one of the N2 cylinders that was running low.
  6. Reopen the cylinders, restore N2 pressure to IFO valves to ~70PSI.
  7. Do steps 1-3 in reverse: i.e. reconnect power to all valves, open them in the reverse order we closed them while monitoring the state of the various turbo pumps. 
  8. Acknowledged the error message on the C0VAC medm screen

Note: the valve isolating the RGA automatically shutoff during this work, possibly because it detected a pressure above its threshold - after checking the appropriate pressure gauges, we reopened this valve as well. 

The attached screenshot suggests that everything went as planned and that the vacuum system is back to normal...

 

 

Attachment 1: c0vac_06172016.png
c0vac_06172016.png
  1226   Wed Jan 14 09:22:43 2009 steveHowToVACN2 supply pressure lowered for vac valves

Quote:
I've been replacing the N2 bottles recently.
I noticed that the consumption is too high. I had to replace them every two days.
Normally the interval is three or more days.
I suspect there is some leak in the line.

Strangely, it is always the left hand bottle which goes empty. The right hand bottle has been
there for more than a week at about 1000 psi.

We should check it when Steve is back.



All vac valves operated by N2
The two cylinders are located in entry room 103
There is an auto switch over valve between them to insure continuous supply.
The pressure regulator should be set 70 PSI on the gauge
This pressure we keep constant.
All vac valves will close in case of running out of N2 or losing ac power so
it is essential that one replaces empty N2 cylinder.

Simply disconnect large CGA580 fitting with a crescent wrench.
Swap in full cylinder from outside, reconnect fitting tightly and open cylinder valve.
Now you should be reading the full cylinder pressure.
Write each cylinder pressure and date-time on the board so one can see if there is a leak
Attachment 1: n280d.jpg
n280d.jpg
  10245   Mon Jul 21 10:51:06 2014 SteveUpdateVACN2 supply run out

Interlock closed valve V1, V4, V5 and VM1 when the nitrogen supply run out. The IFO pressure rose to P1 1 mTorr

In order to recover Vacuum Normal valve configuration I did the following:

Replaced both nitrogen cylinders. Confirmed pneumatic nitrogen pressure 70 PSI.   Opened valves V4 and V5

At P2 < 1 mTorr, Maglev rotation 560 Hz , V1 was opened.

VM1 was opened when CC1 pressure dropped below < 1e-5 torr

 

Please  take a look at the N2 cylinders pressure on Friday to insure that there is enough for the week end.

The daily consumption is 600-700 PSI

Attachment 1: outofN2.png
outofN2.png
  542   Wed Jun 18 18:32:09 2008 Max JonesUpdateComputer Scripts / ProgramsNB Update
I am reconfiguring the the noisebudget code currently in use at the sights. To that end, I have done the following things (in addition to the elog I posted earlier)

In get_dtt_dataset.m - I added C1 specific cases for DARM_CTRL, SEIS, and ITMTRX changing the specific channels to make those in use at caltech

In LocalizeSite.m - I changed the NDS_PATH to match caltechs. I left NDS_HOST untouched.

Since I am trying to get SEIS and DARM to work initially I added C1 specific cases to both of these.

Better documentation may be found in /users/mjones/DailyProgressReport/06_18_08. Suggestions are appreciated. Max.
  12929   Wed Apr 5 16:05:47 2017 gautamUpdateGeneralNB code checkout

[evan, gautam]

We spent some time trying to get the noise-budgeting code running today. I guess eventually we want this to be usable on the workstations so we cloned the git repo into /ligo/svncommon. The main objective was to see if we had all the dependencies for getting this code running already installed. The way Evan has set the code up is with a bunch of dictionaries for each of the noise curves we are interested in - so we just commented out everything that required real IFO data. We also commented out all the gwpy stuff, since (if I remember right) we want to be using nds2 to get the data. 

Running the code with just the gwinc curves produces the plots it is supposed to, so it looks like we have all the dependencies required. It now remains to integrate actual IFO data, I will try and set up the infrastructure for this using the archived frame data from the 2016 DRFPMI locks..

  13097   Wed Jul 5 19:10:36 2017 gautamUpdateGeneralNB code checkout - updated

I've been making NBs on my laptop, thought I would get the copy under version control up-to-date since I've been negligent in doing so.

The code resides in /ligo/svncommon/NoiseBudget, which as a whole is a git directory. For neatness, most of Evan's original code has been put into the sub-directory  /ligo/svncommon/NoiseBudget/H1NB/, while my 40m NB specific adaptations of them are in the sub-directory /ligo/svncommon/NoiseBudget/NB40. So to make a 40m noise budget, you would have to clone and edit the parameter file accordingly, and run python C1NB.py C1NB_2017_04_30.py for example. I've tested that it works in its current form. I had to install a font package in order to make the code run (with sudo apt-get install tex-gyre ), and also had to comment out calls to GwPy (it kept throwing up an error related to the package "lal", I opted against trying to debug this problem as I am using nds2 instead of GwPy to get the time series data anyways).

There are a few things I'd like to implement in the NB like sub-budgets, I will make a tagged commit once it is in a slightly neater state. But the existing infrastructure should allow making of NBs from the control room workstations now.

Quote:

[evan, gautam]

We spent some time trying to get the noise-budgeting code running today. I guess eventually we want this to be usable on the workstations so we cloned the git repo into /ligo/svncommon. The main objective was to see if we had all the dependencies for getting this code running already installed. The way Evan has set the code up is with a bunch of dictionaries for each of the noise curves we are interested in - so we just commented out everything that required real IFO data. We also commented out all the gwpy stuff, since (if I remember right) we want to be using nds2 to get the data. 

Running the code with just the gwinc curves produces the plots it is supposed to, so it looks like we have all the dependencies required. It now remains to integrate actual IFO data, I will try and set up the infrastructure for this using the archived frame data from the 2016 DRFPMI locks..

 

  2976   Mon May 24 16:34:22 2010 KevinUpdatePSLND Filters for 2W Beam Profile

I tried to measure the beam profile of the 2W laser today but ran into problems with the ND filters. With the measurements I made a few weeks ago, I used a reflective ND 4.0 filter on the PD. The PD started to saturate and Koji and I noticed that a lot of the metallic coating on the filter had been burnt off. Koji told me to use an absorptive ND 4.0 filter in front of a reflective ND 0.6 filter. I tried this today but noticed that a few holes were being burned into the absorptive filter and that the coating on the reflective filter behind it was also being burned off in spots. I don't think we wanted to use a polarizing beam splitter to reduce the power before the PD but I didn't want to ruin any more filters.

  14352   Thu Dec 13 18:12:47 2018 gautamUpdateIOOND filter on AS camera changed

In order to see the AS beam a bit more clearly in our low-power config, I swapped out the ND=1.0 filter on the AS camera for ND=0.5.

  6360   Mon Mar 5 23:47:15 2012 keikoConfigurationLSCND2 at REFL OSA

ND filter ND3 (which is at the REFL port to the REFL OSA) is removed. Don't forget to put it back when you restore PRM!!!

  3716   Thu Oct 14 12:45:47 2010 josephbUpdateComputersNDS 2 server installation on mafalda

[Joe, John Zweizig]

John stopped by around noon today to install the NDS 2 server.  He installed it /cvs/cds/caltech/users/jzweizig/nds2-server/.

Once John is done, I will be moving this to a more sensible install location that is not his user directory, but its there for the moment.

We had to install a couple more packages including bzip2, bzip2-devel, gcc-c++, openssl, and openssl-devel. 

We mounted the /frames directory from the fb machine to mafalda by modifying the /etc/fstab file with the line:

fb:/frames              /frames                 nfs     bg,ro           0 0

 

If we change channels recorded by the frame builder, we need to update a channel list file for the NDS 2 server.  There's an excutable located at:

/cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList

This builds the file list if given a .gwf file.  These are written by the frame builder, and can be found in /frames/full/####, where #### are the first 4 gps digits of the gravity files contained in that directory.

Upon questing about when we get to GPS time 1000000000, he said there's some updates he needs to do so it rather throws away the last 5 digits, rather than keeping the first 4.

An example command run on the fb or mafalda machine is:

/cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList  /frames/full/9711/C-R-971119728-16.gwf > nds2-mafalda/C-R-ChanList.txt

For a seconds trends file (located in /frames/trends/second/ instead of /frames/full)

/cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList  /frames/trends/second/9711/ C-T-971106780-60.gwf > nds2-mafalda/C-T-ChanList.txt

For a minute trends file (located in /frames/trends/minute)

/cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList  /frames/trends/minute/9711/C-T-971106780-60.gwf  > nds2-mafalda/C-M-ChanList.txt

In these cases, John was putting the lists in the /cvs/cds/caltech/users/jzweizig/nds2-mafalda/ directory.

 

Both the  C-raw-cache.txt file and the nds2.conf files need to be configured to point at the correct files in the nds2-mafalda directory.

 

  3720   Thu Oct 14 14:09:30 2010 josephbUpdateCDSNDS 2 server status

[Joe, John]

The nds 2 server is about 50% of the way there.  You can connect to it and get channel lists, but its having issues actually serving the data.  The errors we're getting basically say it can't find the source data for the channel

John had to go get lunch at 2pm, but he said he'd log in remotely later and try to configure it properly.

  14104   Wed Jul 25 22:46:15 2018 gautamConfigurationComputersNDS access from outside

After this work, I've been having some trouble getting data with Python NDS. Eventually, I figured out that the nds connection request has to be pointed at '131.215.115.200' (the address of the NAT router which faces the outside world), port 31200 (it used to work with 'nds40.ligo.caltech.edu' or '131.215.115.189'). So the following snippet in python allows a connection to be opened. Offline access of frame data via NDS2 now seems possible.

import nds2
conn = nds2.connection('131.215.115.200',31200)
Quote:
 

So far, ssh (22), web services (30889), and elog (8081, 8080) were tested. We also need to test megatron NDS port forwarding and rsync for nodus, too.Finally I turned off the firewall rules of shorewall on nodus as it is no longer necessary.

  1485   Wed Apr 15 03:52:27 2009 ranaUpdateDMFNDS client32 updated for DMF
 Since our seisBLRMS.m complains about 'can't find hostname' after a few hours, even though matlab is able to ping fb40m, 
I have recompiled the NDS mex client for 32-bit linux on mafalda and stuck it into the nds_mexs/ directory. This time I 
 

 compiled using the 'gcc' compiler instead of the 'ANSI C' compiler that is recommended in the README (which, I notice,

 

 is now missing from Ben Johnsons web page!). Let's see how long this runs.

  5847   Wed Nov 9 13:44:04 2011 MirkoUpdateComputersNDS1 missing channels in matlab

The Matlab NDS1 access seems to only work for some channels. With other channels it just hangs 'Busy' and does nothing.
Below you can see some channels that make matlab hang. Everyting happened on allegra. I tried compiling the NDS1 sources (from https://www.gravity.phy.syr.edu/dokuwiki/doku.php?id=ligodv:nds1_ligodv_install ) into mex files myself. Same result. I
 

a=NDS_GetChannels('fb:8088'); %/cvs/cds/caltech/apps/linux64/share/matlab/NDS_GetChannels.m
%data=NDS_GetData({'C1:IOO-MC_F_DQ'},1004826500,100,'fb:8088',a)     %Works
%data=NDS_GetData({'C1:IOO-WFS1_PIT_IN1_DQ'},1004826500,100,'fb:8088',a)     %Works
data=NDS_GetData({'C1:LSC-AS11_I_OUT'},1004826500,100,'fb:8088',a)         %Doesn't work, hangs
%%%which NDS_GetData.m: /cvs/cds/caltech/apps/linux64/share/matlab/NDS_GetData.m

  3657   Wed Oct 6 00:32:01 2010 ranaSummaryDAQNDS2

This is the link to the NDS2 webpage:

https://www.lsc-group.phys.uwm.edu/daswg/wiki/NetworkDataServer2

We should install this so that we can use this modern interface to get 40m data from outside and inside of the 40m.

  3702   Tue Oct 12 23:45:55 2010 ranaConfigurationDAQNDS2

I installed the NDS2 Client onto the workstations today using the instructions that Zach put onto the Wiki with a couple of modifications.

1) Instead of the adding path stuff in Matlab, I added the LD_LIBRARY_PATH and MATLABPATH variables into the .cshrc as instructed by JZ's NDS2 Wiki.

2) I installed the stuff into the shared /cvs/cds/caltech/apps/linux64/ partition so that it works now on all the 64-bit CentOS 5.5 workstations.

To run it you do:

> kinit albert.einstein

> matlab -nodesktop -nosplash

> help NDS2_GetData

(set the server to the NDS2 server that you like - the example in the help is fine)

> result = NDS2_GetData({'L1:LSC-DARM_ERR'}, 957313530, 10, server);

> plot(result.data)

Now you can get any of the S6 data super fast.

(** Remember to run kdestroy as soon as you are finished so that no one else in the control room can use your personal credentials. **)

Attachment 1: cerberus.jpg
cerberus.jpg
  6381   Wed Mar 7 21:13:30 2012 ranaUpdateDAQNDS2

 I noticed that NDS2 was not running on mafalda as it should be. Instead, there were a couple of zombie MEDMs using up 99% of the CPU. I killed the zombies and have run the 'build channel list' script. When it finished, I tried to restart the nds server, but got the following error in the log file. Email has been dispatched to JZ.

mafalda:logs>less nds2-mafalda-201203072111.log

Configuring from file: nds2.conf
Allow list: ALL
terminate called after throwing an instance of 'std::runtime_error'
  what():  Insufficient arguments
  8750   Tue Jun 25 23:57:30 2013 ranaUpdateSUSNDS2 Status

I've restarted the NDS2 process on Megatron so that we can use it for getting past data and eventually from outside the 40m.

1) from /home/controls/nds2 (which is not a good place for programs to run) I ran nds2-megatron/start-nds2

2) this is just a script that runs the binary from /usr/bin/ and then leaves a log file in ~/nds2/log/

3) I tested with DTT that I could access megatron:31200 and get data that way.

There is a script in usr/bin called nds2_nightly which seems to be the thing we should run by cron to get the channel list to get updated, but I' m not sure. Let's see if we can get an ELOG entry about how this works.

Then we want Jamie to allow some kind of tunneling so that the 40m data can be accessed from outside, etc.

  8757   Wed Jun 26 15:11:08 2013 John ZweizigUpdateSUSNDS2 Status

Quote:

I've restarted the NDS2 process on Megatron so that we can use it for getting past data and eventually from outside the 40m.

1) from /home/controls/nds2 (which is not a good place for programs to run) I ran nds2-megatron/start-nds2

2) this is just a script that runs the binary from /usr/bin/ and then leaves a log file in ~/nds2/log/

3) I tested with DTT that I could access megatron:31200 and get data that way.

There is a script in usr/bin called nds2_nightly which seems to be the thing we should run by cron to get the channel list to get updated, but I' m not sure. Let's see if we can get an ELOG entry about how this works.

Then we want Jamie to allow some kind of tunneling so that the 40m data can be accessed from outside, etc.

 I have done the following:

  * installed the nds2-client in /ligo/apps/nds2-client

  * moved the nds2 configuration directories to /ligo/apps/nds2/nds2-megatron

  * set up a cron job to update the channel list every morning at 5 am. The cron line is:

     15 5 * * * /usr/bin/nds2_nightly /ligo/apps/nds2/channel-tracker /ligo/apps/nds2/nds2-megatron

    cron will send an email each time the channel list changes, at which point you will have to restart the server with:

     cd /ligo/apps/nds2/nds2-megatron
     pkill nds2
     ./start-nds2

  * restarted nds2 with updated channel lists.

  8787   Fri Jun 28 17:33:33 2013 John ZweizigUpdateSUSNDS2 Status

Quote:

Quote:

I've restarted the NDS2 process on Megatron so that we can use it for getting past data and eventually from outside the 40m.

1) from /home/controls/nds2 (which is not a good place for programs to run) I ran nds2-megatron/start-nds2

2) this is just a script that runs the binary from /usr/bin/ and then leaves a log file in ~/nds2/log/

3) I tested with DTT that I could access megatron:31200 and get data that way.

There is a script in usr/bin called nds2_nightly which seems to be the thing we should run by cron to get the channel list to get updated, but I' m not sure. Let's see if we can get an ELOG entry about how this works.

Then we want Jamie to allow some kind of tunneling so that the 40m data can be accessed from outside, etc.

 I have done the following:

  * installed the nds2-client in /ligo/apps/nds2-client

  * moved the nds2 configuration directories to /ligo/apps/nds2/nds2-megatron

  * set up a cron job to update the channel list every morning at 5 am. The cron line is:

     15 5 * * * /usr/bin/nds2_nightly /ligo/apps/nds2/channel-tracker /ligo/apps/nds2/nds2-megatron

    cron will send an email each time the channel list changes, at which point you will have to restart the server with:

     cd /ligo/apps/nds2/nds2-megatron
     pkill nds2
     ./start-nds2

  * restarted nds2 with updated channel lists.

 I have set the cron job up to restart the nds2 server automatically if the channel list changes. The only change is that the cron command was changes to /ligo/apps/nds2/nds2-megatron/test-restart.

 

  8861   Tue Jul 16 19:16:12 2013 ranaUpdateDAQNDS2 Status

I have modified the settings on the router that connects our Martian network to the outside world so that one can access the NDS2 server running on megatron:31200.

To get at the data you point your data getting client (Matlab, ligoDV, DTT, etc.) at our router and the megatron port will be forwarded to you:

131.215.115.189:31200

is what you should point to. Now, it should be possible to run DetChar jobs (e.g. our 40m Summary pages) from the outside on some remote server. You can also grab 40m data on your laptop directly by using matlab or python NDS software.

  5780   Tue Nov 1 23:13:28 2011 ZachUpdateComputersNDS2 channel files

I did some messing around with the NDS2 config and channel files and things seem to be working as expected... for now. SENSOR channel data can be acquired for all sensors on all hanging optics.

What I did:

  • NDS2 gets its channel lists from .../users/jzweizig/nds2-mafalda/nds2.conf, which is called in the start-nds2 script. Within this, there are channel.file lines that specify which channels are available for raw and trend data. The four files that were listed were:
    • C-R-raw-channel_list.txt
    • C-M-ChanList.txt
    • C-T-ChanLIst.txt
    • C-R-online-channel_list.txt (this one was listed after a hashed line, which was suspicous---see below)
  • I noticed that a grep for SENSOR only returned lines for non-MC mirrors in both "R" files
  • I also noticed that calling NDS2_GetChannels('mafalda.martian:31200') did not return any non-suffixed (i.e., raw) channel names for MCx_SENSOR channels, while non-MC SENSOR channels each had two non-suffixed listings. I thought this was strange.
  • I manually added the line "C1:SUS-MC1_SENSOR_UL 0 real_4 2048 C-R" to one of the "R" channel files, then restarted the NDS2 server, and that channel was still not served. I figured that the second "R" channel file might have been left in the config file as a mistake, so I commented it out, restarted the NDS2 server, and was able to get MC1_SENSOR_UL data. I have left the comment-out there, with a signed EDIT.
  • Wary of (and too lazy to) manually add lines for all 5 sensors for each MC mirror, I decided to try generating a channel file using the most recent .gwf file in the frames, as indicated in Joe and John's elog post. To do this, while in .../nds2-mafalda/, I ran:
    • /cvs/cds/caltech/users/jzweizig/nds2-server/bin/buildChannelList /frames/full/10042/C-R-1004246528-16.gwf > C-R-raw-channel_list.txt
  • A grep for SENSOR in the new C-R-raw-channel_list.txt now returned lines for all MC mirror sensors... BUT NOT FOR ETMY(?!). I tried some slightly older .gwf files (all from today), but the ETMY files never showed up. I had no choice but to enter them manually. Another odd thing is that the channel file generated this way seems to be fairly jumbled up, in the sense that there is no clear top-town order (e.g. SUS-BS_blah then SUS-ETMX_blah). Instead, some SUS-BS channels are here, some are after SUS-ETMX or SUS-PRM channels, etc. Look at the file to know what I mean.
  • The original raw channel file is there as C-R-raw-channel_list.txt.bak.1004247481.

In any case, as I said, everything appears to be working now, but as soon as we try to generate a new channel file using the prescribed means, there will inevitably be channels omitted. Someone who knows more than me should get to the bottom of this and wiki a strict, detailed procedure for how this is to be done.

  6912   Wed Jul 4 18:25:44 2012 ZachUpdateComputersNDS2 client now working on Ubuntu machines

After plenty of work, NDS2 can now be used to get site data within MATLAB using the following machines:

  • allegra
  • megatron
  • ottavia
  • pianosa
  • rosalba
  • rossa

What I did

NDS2 was not working on any of the machines, so the first thing I did was simply to install the newest version. I downloaded the latest tarball (0.9.1) from the LDAS Wiki, unzipped and installed it

/users/zach $ tar -xvf nds2-client-0.9.1.tar

/users/zach $ cd nds2-client-0.9.1

/users/zach $ sudo ./configure --prefix=/cvs/cds/caltech/apps/linux64 --with-matlab=/cvs/cds/caltech/apps/linux64/matlab/bin/matlab

/users/zach $ sudo make

/users/zach $ sudo make install

 

Even with the new version, it still didn't work.

Solution: The main problem was that the cyrus-sasl-gssapi authentication protocol was not installed on these machines, so that even with a kerberos ticket the datalink could not be established. Using information from the LDAS Wiki, I used aptitude to install it as:

$ sudo aptitude install lscsoft-auth

This group installs both the SASL protocol and the package python-kerberos

 

I also needed to update the kerberos config file for each machine, which is located at /etc/krb5.conf. I found that ottavia had a nice one with many realms, so I copied that one over to the other machines. In any case where there was an old config file overwritten, it is now /etc/krb5.conf.old.

Finally, the matlab path for NDS2 was still set to the old 2010a directory (/cvs/cds/caltech/apps/linux64/lib/matlab2010a) that was created by the NDS2 install when Rana originally did it. The new install I made above created the appropriate 2010b mexa64 files, so I changed the matlab path within matlab to this one:

>> rmpath /cvs/cds/caltech/apps/linux64/lib/matlab2010a

>> addpath /cvs/cds/caltech/apps/linux64/lib/matlab2010b

>> savepath

 

Now everything works fine on all these machines. As in Rana's original post, you get data in the following way:

$ kinit albert.einstein %then enter password

$ matlab -nosplash -nodesktop

>> d = NDS2_GetData({'H1:LSC-NPTRX_OUT16.mean'},963968415,6000,'nds.ligo.caltech.edu:31200')

 

d = 


            name: 'H1:LSC-NPTRX_OUT16.mean' 

            chan_type: 'm-trend'             

            rate: 0.0167       

            data_type: 'real_8'     

            signal_gain: 1   

            signal_offset: 0     

            signal_slope: 1     

            signal_units: ''   

            start_gps_sec: 963968415     

            duration_sec: 6000             

            data: [100x1 double]           

            exists: 1

 

>> quit % since you've seen that the data is really here

$ kdestroy % so that no one uses your credentials

 

Some thoughts

  • I would like to extend this to the 32-bit machines, but I have to figure out the best way to install the proper NDS2 client without interfering with the 64-bit version. I think it is just a matter of specifying the matlabroot in the .../linux/ instead of .../linux64/
  • It would be nice to find a way that the nice tool gps('MM/DD/YYYY XX:XX:XX UTC'), which calls the ligotool executable tconvert, can be automatically usable when calling NDS2 functions. Right now, there seems to be an issue preventing that: even though tconvert can be run in the terminal, gps() returns an error and even directly running unix('tconvert now') or !tconvert returns the same error. I have emailed Peter Shawhan to see if he has any advice.

 

 

  6919   Thu Jul 5 12:06:35 2012 JamieUpdateComputersNDS2 client now working on Ubuntu machines

What I did

NDS2 was not working on any of the machines, so the first thing I did was simply to install the newest version. I downloaded the latest tarball (0.9.1) from the LDAS Wiki, unzipped and installed it

/users/zach $ tar -xvf nds2-client-0.9.1.tar

/users/zach $ cd nds2-client-0.9.1

/users/zach $ sudo ./configure --prefix=/cvs/cds/caltech/apps/linux64 --with-matlab=/cvs/cds/caltech/apps/linux64/matlab/bin/matlab

/users/zach $ sudo make

/users/zach $ sudo make install

No no, this is totally unnecessary.  NDS2 was already installed on every machine from the official packaged releases (apt-get install nds2-client), and it's known to work fine. We use it with pynds all the time. If the matlab component is not working we should figure out the right way to fix it with the existing packages.

In general, please only manually install software as a very last resort.  Manually installed software doesn't get maintained, where as the officially packaged stuff is being actively maintained by the collaboration. If there is a problem with the distributed packaging we should report it and get it fixed (and hint I was the one who built the original Debian packaging for nds2, so I know how to fix all the issues).  I'm trying to bring the 40m out of the dark days of complete chaos, where random software was installed in random locations.

Even with the new version, it still didn't work. 

That's because this wasn't the problem!

Solution: The main problem was that the cyrus-sasl-gssapi authentication protocol was not installed on these machines, so that even with a kerberos ticket the datalink could not be established. Using information from the LDAS Wiki, I used aptitude to install it as:

$ sudo aptitude install lscsoft-auth

This group installs both the SASL protocol and the package python-kerberos

I also needed to update the kerberos config file for each machine, which is located at /etc/krb5.conf. I found that ottavia had a nice one with many realms, so I copied that one over to the other machines. In any case where there was an old config file overwritten, it is now /etc/krb5.conf.old.

Finally, the matlab path for NDS2 was still set to the old 2010a directory (/cvs/cds/caltech/apps/linux64/lib/matlab2010a) that was created by the NDS2 install when Rana originally did it. The new install I made above created the appropriate 2010b mexa64 files, so I changed the matlab path within matlab to this one:

>> rmpath /cvs/cds/caltech/apps/linux64/lib/matlab2010a

>> addpath /cvs/cds/caltech/apps/linux64/lib/matlab2010b

>> savepath

This sounds like it's more likely the issues. You did the right thing by going to apt to fix the authentication packages.  It's curious to me that you did that here, whereas you went totally out of band for the nds2 client stuff.  Why?

The matlab mex files are the other problem.  But there is also a nds2-client-matlab Debian/Ubuntu package for that as well.  The problem is that the package just distributes the source, and it needs to be compiled.  I'll help figure out a good way to do that.

  • I would like to extend this to the 32-bit machines, but I have to figure out the best way to install the proper NDS2 client without interfering with the 64-bit version. I think it is just a matter of specifying the matlabroot in the .../linux/ instead of .../linux64/

Again, this is handled by the packaging!  Just use apt and the right architecture is installed automatically.

But what 32 bit machines are you referring to?  I think basically everything is 64 bit nowadays.

  • It would be nice to find a way that the nice tool gps('MM/DD/YYYY XX:XX:XX UTC'), which calls the ligotool executable tconvert, can be automatically usable when calling NDS2 functions. Right now, there seems to be an issue preventing that: even though tconvert can be run in the terminal, gps() returns an error and even directly running unix('tconvert now') or !tconvert returns the same error. I have emailed Peter Shawhan to see if he has any advice. 

We are now using lalapps_tconvert for tconvert.  We're not using that ligotools crap anymore.  I've aliased that to tconvert on the command line, but maybe matlab isn't getting the message.  I'll try to think of a more robust solution (e.g. make a wrapper script).

  6921   Thu Jul 5 13:12:12 2012 ZachUpdateComputersNDS2 client now working on Ubuntu machines

From my conversations with JZ and Leo, it seemed there was no package that generated the appropriate mex files. It was clear that the right ones weren't there from the absence of a /cvs/cds/caltech/apps/linux64/lib/matlab2010b directory. I'm sorry if I screwed anything up with pynds, but I have repeatedly asked for help with NDS2+matlab and no one has done anything.

It would be nice to do it via apt if there indeed is a versioned package that can make the mexs. Sorry again if I jumped the gun, but I didn't think anyone was going to do anything.

I guess the only 32-bit machine I can think of is mafalda.

About tconvert, I think the best solution is to make a new wrapper M-file. gps was just a convenient remnant of mDV, but all that we need is some matlab function that can output a GPS time given a date/time string. We can use whatever command-line utility you want.

  6928   Fri Jul 6 09:00:34 2012 not ZachUpdateComputersNDS2 client now working on Ubuntu machines

Quote:

From my conversations with JZ and Leo, it seemed there was no package that generated the appropriate mex files. It was clear that the right ones weren't there from the absence of a /cvs/cds/caltech/apps/linux64/lib/matlab2010b directory. I'm sorry if I screwed anything up with pynds, but I have repeatedly asked for help with NDS2+matlab and no one has done anything.

It would be nice to do it via apt if there indeed is a versioned package that can make the mexs. Sorry again if I jumped the gun, but I didn't think anyone was going to do anything.

There is a package that provides the mex source, but it doesn't actually provide the mex binaries.  The problem is that the binary depends on the matlab version, so you can't possibly provide binaries for every version.

The solution is to just build the binaries from the source package.  We should put together a nice script that builds the binaries from the source, and installs them in the directory of your choosing.  If we get something nice working, we can probably get them to include it with the package, to make it easier in the future.

Here's what's included in the source package:

controls@pianosa:~ 0$ sudo apt-get install nds2-client-matlab
...
controls@pianosa:~ 0$ dpkg -L nds2-client-matlab | sort
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/nds2-client-matlab
/usr/share/doc/nds2-client-matlab/changelog.Debian.gz
/usr/share/doc/nds2-client-matlab/changelog.gz
/usr/share/doc/nds2-client-matlab/copyright
/usr/share/matlab
/usr/share/matlab/NDS2_GetChannels.m
/usr/share/matlab/NDS2_GetData.m
/usr/share/matlab/NDS_GetChannels.m
/usr/share/matlab/NDS_GetData.m
/usr/share/matlab/NDS_GetMinuteTrend.m
/usr/share/matlab/NDS_GetSecondTrend.m
/usr/share/matlab/src
/usr/share/matlab/src/NDS2_GetChannels.c
/usr/share/matlab/src/NDS2_GetData.c
/usr/share/matlab/src/NDS_GetChannels.c
/usr/share/matlab/src/NDS_GetData.c
/usr/share/matlab/src/nds_mex_utils.c
/usr/share/matlab/src/nds_mex_utils.h
controls@pianosa:~ 0$ 
  4407   Sun Mar 13 00:00:58 2011 jzweizig, ranaConfigurationDAQNDS2 code change and restart

 John has changed the NDS2 code and restarted it on Mafalda. The issue is that it goes off the rails everytime the DAQD is restarted on FB because of filename convention war between GDS and CDS.

Until this is resolved, please make sure to restart the NDS2 process on Mafalda everytime you restart DAQD by doing this:

pkill -KILL nds2

/users/jzweizig/nds2-mafalda/start_nds2

  4926   Thu Jun 30 21:55:16 2011 ranaConfigurationDAQNDS2 conf change

As I recently had trouble getting all of the SUS SENSOR channels at once from NDS2, I asked J.Z. for help. He found that the number of buffers on mafalda was set to only allow a small amount of data to be requested at one time.

He's going to have to figure out a more permanent fix, but for now he's increased the data buffer size to allow somewhat larger chunks to be gotten. I have made a work around in matlab, which gets smaller chunks and then cats them together.

Its in SUS/peakFit/.

Attachment 1: Untitled.png
Untitled.png
  6392   Fri Mar 9 11:59:38 2012 Zweizig the ELOG MavenSummaryCDSNDS2 restart

 Hi Rana,

It looks like the channe list file has a few blank lines that the channel list reader is choking on. I removed the lines and it is working now.. I have made the error message a bit more obvious (gave the file name  and line number) and allowed it to ignore empty lines so this won't cause problems with future versions (when installed). The bottom line is nds2 is now running on mafalda.
Best regards,

John   

  15341   Wed May 20 20:10:34 2020 rana, John ZUpdateComputer Scripts / ProgramsNDS2 server / conf updated - seems OK now

We noticed about a week ago that the NDS2 channel lists were not getting updated on megatron. JZ and I investigated; he was able to fix it all up this afternoon by logging in and snooping around Megatron.

Please try it out and tell me about any problems in getting fresh data.


  1. The NDS2 server is what we connect to through our python NDS2 client software to download some data.
  2. It has been working for years, but it looks like there was a file corruption of the channel lists that it makes back in 2017.
  3. Since the NDS2 server code tries to make incremental changes, it was failing to make a new channel list. Was failing to parse the corrupted file.
  4. there was a controls crontab entry to restart the server every morning, but the file name in that tab had a typo, so that wasn't working. I commented it out, since it shouldn't be necessary (lets see how it goes...)
  5. the nds2mgr account also has a crontab, but that was failing since it didn't have sudo permission. JZ added nds2mgr to the sudoers list so that should work now.
  6. I was able to get new channels as of 4 PM today, so it seems to be working.

* we should remember to rebuild the NDS2 server code for Ubuntu. The thing running on there is for CentOS / SL7, but we moved to Ubuntu recently since the SL7 support is going away.

** the nds2 code & conf files are not backed up anywhere since its not on /cvs/cds. It has 52 GB(!!) of txt channel lists & archives which we don't need to backup

  5094   Tue Aug 2 16:43:23 2011 jamieUpdateCDSNDS2 server on mafalda restarted for access to new channels

In order to get access to new DQ channels from the NDS2 server, the NDS2 server needs to be told about the new channels and restarted.  The procedure is as follows:

ssh mafalda
cd /users/jzweizig/nds2-mafalda
./build_channel_history
./install_channel_list
pkill nds2
# wait a few seconds for the process to quit and release the server port
./start_nds2

This procedure needs to be run every time new _DQ channels are added.

We need to set this up as a proper service, so the restart procedure is more elegant.

An additional comment from John Z.:

    The --end-gps parameter in ./build_channel_history seems to be causeing
    some trouble. It should work without this parameter, but there is a
    directory with a gps time of 1297900000 (evidently a test for GPS1G)
    that might screw up the channel list generation. So, it appears that
    the end time requires a time for which data already exists. this
    wouldn't seem to be a big deal, but it means that it has to be modified
    by hand before running. I haven't fixed this yet, but I think that I
    can probably pick out the most recent frame and use that as an end-time
    point. I'll see if I can make that work...

  10279   Sat Jul 26 15:30:15 2014 Joseph AreedaUpdateComputer Scripts / ProgramsNDS2 server propem on megatron

The NDS2 server on megatron was unresponsive for what i think was the last couple of days.

The NDS the log file (~nds2mgr/logs/nds2-201407151045.log) started reporting "Stage: parser output queue is full." at 2014.7.24 14:47:54 also there are 16 connections still not closed with LindmeierLaptop.cacr.caltech.edu (131.215.146.102) with 15 of them in CLOSE_WAIT. 

To identify these zombie sockets we use "netstat -an | grep 31200"

The server was in a condition that /etc/init.d/nds2 stop didn't work and the process had to be manually kill -9'ed and then about 3 or 4 minutes later the zombie sockets were gone at /etc/init.d/nds2 start was used to restart the server.

The LindemejerLaptop was using pynds to get a bunch of channels at once to test drive a streaming visualization code for glitches.  It's unclear whether this bumped into a server limitation.  We have seen similar states in ldvw that seem to be the result of errors which result in client-server connections not being closed properly, leaving data in an output buffer causing Linux to wait for the other side to empty the buffer.

  13293   Tue Sep 5 14:41:58 2017 gautamUpdateCDSNDS2 server restarted on megatron

I was unable to download data using nds2. Gabriele had reported similar problems a week ago but I hadn't followed up on this.

I repeated steps 5-7 from elog 13161, and now it seems that I can get data from the nds2 servers again. Unclear why the nds2 server had to be restarted. I wonder if this is somehow related to the mysterious acromag EPICS server tmux session dropout.

  13331   Tue Sep 26 13:40:45 2017 gautamUpdateCDSNDS2 server restarted on megatron

Gabriele reported problems with the nds2 server again. I restarted it again.

update: had to do it again at 1730 today - unclear why nds2 is so flaky. Log files don't suggest anything obvious to me...

Quote:

I was unable to download data using nds2. Gabriele had reported similar problems a week ago but I hadn't followed up on this.

I repeated steps 5-7 from elog 13161, and now it seems that I can get data from the nds2 servers again. Unclear why the nds2 server had to be restarted. I wonder if this is somehow related to the mysterious acromag EPICS server tmux session dropout.

 

  13161   Thu Aug 3 00:59:33 2017 gautamUpdateCDSNDS2 server restarted, /frames mounted on megatron

[Koji, Nikhil, Gautam]

We couldn't get data using python nds2. There seems to have been many problems.

  1. /frames wasn't mounted on megatron, which was the nds2 server. Solution: added /frames 192.168.113.209(sync,ro,no_root_squash,no_all_squash,no_subtree_check) to /etc/exportfs on fb1, followed by sudo exportfs -ra. Using showmount -e, we confirmed that /frames was being exported.
  2. Edited /etc/fstab on megatron to be fb1:/frames/ /frames nfs ro,bg,soft 0 0. Tried to run mount -a, but console stalled.
  3. Used nfsstat -m on megatron. Found out that megatron was trying to mount /frames from old FB (192.168.113.202). Used sudo umount -f /frames to force unmount /frames/ (force was required).
  4. Re-ran mount -a on megatron.
  5. Killed nds2 using /etc/init.d/nds2 stop - didn't work, so we manually kill -9'ed it.
  6. Restarted nds2 server using /etc/init.d/nds2 start.
  7. Waited for ~10mins before everything started working again. Now usual nds2 data getting methods work.

I have yet to check about getting trend data via nds2, can't find the syntax. EDIT: As Jamie mentioned in his elog, the second trend data is being written but is inaccessible over nds (either with dataviewer, which uses fb as the ndsserver, or with python NDS, which uses megatron as the ndsserver). So as of now, we cannot read any kind of trends directly, although the full data can be downloaded from the past either with dataviewer or python nds2. On the control room workstations, this can also be done with cds.getdata.

  13162   Thu Aug 3 10:51:32 2017 ranaUpdateCDSNDS2 server restarted, /frames mounted on megatron

same issue on NODUS; I edited the /etc/fstab and tried mount -a, but it gives this error:

controls@nodus|~ 1> sudo mount -a
mount.nfs: access denied by server while mounting fb1:/frames

needs more debugging - this is the machine that allows us to have backed up frames in LDAS. Permissions issues from fb1 ?

ELOG V3.1.3-