40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 118 of 344  Not logged in ELOG logo
ID Date Author Type Category Subject
  11397   Wed Jul 8 21:02:02 2015 JamieSummaryCDSCDS upgrade: another step forward, so we're back to where we started (plus a bit?)

Koji did a bit of googling to determine that 'Wrong Network' status message could be explained by the fb myrinet  operating in the wrong mode:
(This was the useful link to track down the issue (KA))

    Network:    Myrinet 10G

I didn't notice it before, but we should in fact be operating in "Ethernet" mode, since that's the fabric we're using for the DC network.  Digging a bit deeper we found that the new version of mx (1.2.16) had indeed been configured with a different compile option than the 1.2.15 version had:

controls@fb ~ 0$ grep '$ ./configure' /opt/src/mx-1.2.15/config.log          
  $ ./configure --enable-ether-mode --prefix=/opt/mx
controls@fb ~ 0$ grep '$ ./configure' /opt/src/mx-1.2.16/config.log
  $ ./configure --enable-mx-wire --prefix=/opt/mx-1.2.16
controls@fb ~ 0$

So that would entirely explain the problem.  I re-linked mx to the older version (1.2.15), reloaded the mx drivers, and everything showed up correctly:

controls@fb ~ 0$ /opt/mx/bin/mx_info
MX Version: 1.2.12
MX Build: root@fb:/root/mx-1.2.12 Mon Nov  1 13:34:38 PDT 2010
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
Instance #0:  299.8 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:        Running, P0: Link Up
    Network:    Ethernet 10G

    MAC Address:    00:60:dd:46:ea:ec
    Product code:    10G-PCIE-8AL-S
    Part number:    09-03916
    Serial number:    352143
    Mapper:        00:60:dd:46:ea:ec, version = 0x00000000, configured
    Mapped hosts:    6

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:46:ea:ec fb:0                              1,0
   1) 00:25:90:0d:75:bb c1sus:0                           1,0
   2) 00:30:48:be:11:5d c1iscex:0                         1,0
   3) 00:30:48:d6:11:17 c1iscey:0                         1,0
   4) 00:30:48:bf:69:4f c1lsc:0                           1,0
   5) 00:14:4f:40:64:25 c1ioo:0                           1,0
controls@fb ~ 0$

The front end hosts are also showing good omx info (even though they had been previously as well):

controls@c1lsc ~ 0$ /opt/open-mx/bin/omx_info
Open-MX version 1.5.2
 build: controls@fb:/opt/src/open-mx-1.5.2 Tue May 21 11:03:54 PDT 2013

Found 1 boards (32 max) supporting 32 endpoints each:
 c1lsc:0 (board #0 name eth1 addr 00:30:48:bf:69:4f)
   managed by driver 'igb'

Peer table is ready, mapper is 00:30:48:d6:11:17
  0) 00:30:48:bf:69:4f c1lsc:0
  1) 00:60:dd:46:ea:ec fb:0
  2) 00:25:90:0d:75:bb c1sus:0
  3) 00:30:48:be:11:5d c1iscex:0
  4) 00:30:48:d6:11:17 c1iscey:0
  5) 00:14:4f:40:64:25 c1ioo:0
controls@c1lsc ~ 0$

This got all the mx_stream connections back up and running.

Unfortunately, daqd is back to being a bit flaky.  With all frame writing enabled we saw daqd crash again.  I then shut off all trend frame writing and we're back to a marginally stable state: we have data flowing from all front ends, and full frames are being written, but not trends.

I'll pick up on this again tomorrow, and maybe try to rebuild the new version of mx with the proper flags.

  11396   Wed Jul 8 20:37:02 2015 JamieSummaryCDSCDS upgrade: one step forward, two steps back

After determining yesterday that all the daqd issues were coming from the frame writing, I started to dig into it more today.  I also spoke to Keith Thorne, and got some good suggestions from Gerrit Kuhn at GEO.

I  realized that it probably wasn't the trend writing per se, but that turning on more writing to disk was causing increased load on daqd, and consequently on the system itself.  With more frame writing turned on the memory consuption increased to the point of maxing out the physical RAM.  The system the probably starting swaping, which certainly would have choked daqd.

I noticed that fb only had 4G of RAM, which Keith suggested was just not enough.  Even if the memory consumption of daqd has increased significantly, it still seems like 4G would not be enough.  I opened up fb only to find that fb actually had 8G of RAM installed!  Not sure what happend to the other 4G, but somehow they were not visible to the system.  Koji and I eventually determined, via some frankenstein operations with megatron, that the RAM was just dead.  We then pulled 4G of RAM from megatron and replaced the bad RAM in fb, so that fb now has a full 8G of RAM cool.

Unfortunately, when we got fb fully back up and running we found that fb is not able to see any of the other hosts on the data concentrator network sad.  mx_info, which displays the card and network status for the myricom myrinet fiber card, shows:

MX Version: 1.2.16
MX Build: controls@fb:/opt/src/mx-1.2.16 Tue May 21 10:58:40 PDT 2013
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
Instance #0:  299.8 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:        Running, P0: Wrong Network
    Network:    Myrinet 10G

    MAC Address:    00:60:dd:46:ea:ec
    Product code:    10G-PCIE-8AL-S
    Part number:    09-03916
    Serial number:    352143
    Mapper:        00:60:dd:46:ea:ec, version = 0x63e745ee, configured
    Mapped hosts:    1

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:46:ea:ec fb:0                            D 0,0

Note that all front end machines should be listed in the table at the bottom, and they're not.   Also note the "Wrong Network" note in the Status line above.  It appears that the card has maybe been initialized in a bad state?  Or Koji and I somehow disturbed the network when we were cleaning up things in the rack.  "sudo /etc/init.d/mx restart" on fb doesn't solve the problem.  We even rebooted fb and it didn't seem to help.

In any event, we're back to no data flow.  I'll pick up again tomorrow.

  11395   Wed Jul 8 17:46:20 2015 JessicaSummaryGeneralUpdated Time Delay Plots

I re-measured the transfer function for Cable B, because the residuals in my previous post for cable B indicated a bad fit. 

I also realized I had made a mistake in calculating the time delay, and calculated more reasonable time delays today. 

Cable A had a delay of 202.43 +- 0.01 ns.

Cable B had a delay of 202.44 +- 0.01 ns. 

Attachment 1: resid_CableA.png
Attachment 2: resid_CableB.png
  11394   Tue Jul 7 23:26:19 2015 KojiUpdateCDSAttempt to list CDS issues

As Jamie succeded to realize somewhat workable condition of the 40m CDS, I tried to list the obvious CDS issues so that we can attack them one by one.

  1. c1cal is constantly time-outing now (t>60usec). c1sus is close to it (t=56~57us)
  2. We should check the trends of the CPU meters ("C1:FEC-**_CPU_METER"). In fact this should be listed in the summary pages in a new CDS tab.
  3. Probably this is related to 1): c1lsc is constantly showing IPC error (bit0 = shmem). C1LSC_IPC_STATUS.adl is telling that this is coming from the IPC error between c1lsc and c1cal. ("C1:CAL-LSC_SENSMAT_OSC_****"). This information is found by opning C1LSC_GDS_TP.adl screen and click RT NET STAT button next to the IPC error status.
  4. We wonder how the RFM access is accelerated or decelerated by this upgrade.
  5. We need tests to see if the time delays of the models/IPCs are still reasonable.
  6. LSC Overviw screen has a small digest of the CDS status. Now there are many white boxes that correspond to the channels "C1:FEC-**_DIAG1".
  7. All realtime systems have default (0 or 1) epics channel values (i.e. gains, FM switches, matrices, etc). Need burtrestores.
  8. I tried to burtrestore the models but burtgooey indicated there are some errors.
  9. Detailed check of the snapshot files comparing snapshot files in /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jul/7/19:07 and /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jun/1/19:07 :
    • c1alsepics shows bunch of volatile channels to be snapshot. It seems that all of the static epics channels are missing in the snapshot file. Is this related to the current omission of the slow data acquisition? => No actually this must be the modification of the ALS model to accommodate the ALS in the LSC model for the new ALS setup.
    • c1lscepics was checked indeed slow channels were properly snapshot. So what was the problem in burting???
    • I made a simple csh script to restore the snapshots one by one while collecting the error messages.
      This script is located as /users/koji/150707/burtrevert.sh
    • #!/bin/csh
      echo 'This script restores all of the snapshot files found in' $argv[1] '.'
      echo 'Are you sure? y/n'

      set ans = $<

      set ANS = `echo $ans | tr "[:upper:]" "[:lower:]" `
      if ($ANS == y) then
          foreach fname ($argv[1]/*epics.snap)
          echo ''
          echo '#################################'
          echo $fname
          echo '#################################'

              burtwb -f $fname
          echo "exiting..."
    •  Now I ran the command
      ./burtrevert.sh /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jun/1/19:07 &>burt.log
      This lists up the missing channels. The zipped log is attached to this entry.
    • Burting old snapshot always crashes the RT process "c1sus" (not the c1sus host). If I use the newly generated snapshot today,
      the process does not crash. The process halts at the cycle time of 74us (>60us). I left the process crashed so that we can take a new snapshot with the matrix numbers filled. Once we have the correct snapshot, we don't need to worry about this crash. Let's see.
    • c1sus still crashes with the new burt file. Theremust be a trigeer that makes the model frozen. We need to split the burtfile into pieces
      to figure out which line causes the halt.
Attachment 1: burt.log.zip
  11393   Tue Jul 7 18:27:54 2015 JamieSummaryCDSCDS upgrade: progress!

After a couple of days of struggle, I made some progress on the CDS upgrade today:

Front end status:

  • RTS upgraded to 2.9.4, and linked in as "release":

/opt/rtcds/rtscore/release -> tags/advLigoRTS-2.9.4

  • mbuf kernel module built installed
  • All front ends have been rebooted with the latest patched kernel (from 2.6 upgrade)
  • All models have been rebuilt, installed, restarted.  Only minor model issues had to be corrected (unterminated unused inputs mostly).
  • awgtpman rebuilt, and installed/running on all front-ends
  • open-mx upgraded to 1.5.2:

/opt/open-mx -> open-mx-1.5.2

  • All front ends running latest version of mx_stream, built against 2.9.4 and open-mx-1.5.2.

We have new GDS overview screens for the front end models:

It's possible that our current lack of IRIG-B GPS distribution means that the 'TIM' status bit will always be red on the IOP models.  Will consult with Rolf.

There are other new features in the front ends that I can get into later.

DAQ (fb) status:

  • daqd and nds rebuilt against 2.9.4, both now running on fb

40m daqd compile flags:

cd src/daqd
./configure --enable-debug --disable-broadcast --without-myrinet --with-mx --enable-local-timing --with-epics=/opt/rtapps/epics/base --with-framecpp=/opt/rtapps/framecpp
make clean
install daqd /opt/rtcds/caltech/c1/target/fb/

However, daqd has unfortunately been very unstable, and I've been trying to figure out why.  I originally thought it was some sort of timing issue, but now I'm not so sure.

I had to make the following changes to the daqdrc:

set gps_leaps = 820108813 914803214 1119744016;

That enumerates some list of leap seconds since some time.  Not sure if that actually does anything, but I added the latest leap seconds anyway:

set symm_gps_offset=315964803;

This updates the silly, arbitrary GPS offset, that is required to be correct when not using external GPS reference.

Finally, the last thing I did that finally got it running stably was to turn off all trend frame writing:

# start trender;
# start trend-frame-saver;
# sync trend-frame-saver;
# start minute-trend-frame-saver;
# sync minute-trend-frame-saver;
# start raw_minute_trend_saver;

For whatever reason, it's the trend frame writing that that was causing things daqd to fall over after a short amount of time.  I'll continue investigating tomorrow.


We still have a lot of cleanup burt restores, testing, etc. to do, but we're getting there.

  11392   Tue Jul 7 17:22:16 2015 JessicaSummary Time Delay in ALS Cables

I measured the transfer functions in the delay line cables, and then calculated the time delay from that.

The first cable had a time delay of 1272 ns and the second had a time delay of 1264 ns. 

Below are the plots I created to calculate this. There does seem to be a pattern in the residual plots however, which was not expected. 

The R-Square parameter was very close to 1 for both fits, indicating that the fit was good. 

Attachment 1: cableA_fit.jpg
Attachment 2: cableA_resid.jpg
Attachment 3: cableB_fit.jpg
Attachment 4: cableB_resid.jpg
  11391   Sun Jul 5 18:14:13 2015 IgnacioUpdatePEMWilcoxon Accelerometer Huddle Test

Updated: On Thursday/Friday (sorry for late elog) I was messing with Eric's Wilcoxon 731A accelerometer huddle test data that was taken without the box and cables being adjusted properly. Anyways, I performed the three cornered hat analysis as he had done but I also performed a six cornered hat method as well instead of permuting around in pairs of three accelerometers. The following plots of the ASD's show the results,

It is interesting to note the improvement at low frequencies when six accelerometers are used instead of six while at higher frequencies we can clearly see how the results are worst than the three hat results.

I decided to take a mean of all six accelerometers measured ground signal as well as that for their computed selfnoises, this is plotted below,


Notice the obvious improvement along the entire frequency band of the measurements when all accelerometers are used in the data analysis.

I also performed some Wiener filtering of this data. There was an obvious improvement in the results,

The mean of the signals is also plotted below, just as I did with the cornered hat methods,


I also compared the mean self noise of the accelerometers against the manufacturers calculated selfnoise that Rana put up on Github. Both methods are compared against what the manufacturer claims,

As expected the measured noise curves of the Wilcoxon is not as good as what the manufactures stated. This should improve once we redo the huddle test with a better experimental setup. We have some pretty interesting results with the six cornered hat method at around 5 Hz, it is surprisingly accurate given how rough the calculations seemed to be.

I have attached my code for reference: code_accel.zipselfnoise_allsix.png

SEE attachments for better plots of the six accelerometers...

Attachment 5: code_accel.zip
Attachment 6: selfnoise_allsix_miso.png
Attachment 8: selfnoise_allsix.png
  11390   Wed Jul 1 19:16:21 2015 JamieSummaryCDSCDS upgrade in progress

The CDS upgrade is now underway

Here's what's happened so far:

  • Installed and linked in all the RTS supporting software packages in /opt/rtapps (only on front end machines and fb):
    controls@c1lsc ~ 2$ find /opt/rtapps/ -mindepth 1 -maxdepth 1 -type l -ls
    12582916    0 lrwxrwxrwx   1 controls 1001           12 Jul  1 13:16 /opt/rtapps/gds -> gds-
    12603452    0 lrwxrwxrwx   1 controls 1001           10 Jul  1 13:17 /opt/rtapps/fftw -> fftw-3.3.2
    12603451    0 lrwxrwxrwx   1 controls 1001           15 Jul  1 13:16 /opt/rtapps/libframe -> libframe-8.17.2
    12603450    0 lrwxrwxrwx   1 controls 1001           13 Jul  1 13:16 /opt/rtapps/libmetaio -> libmetaio-8.2
    12582915    0 lrwxrwxrwx   1 controls 1001           34 Jul  1 15:24 /opt/rtapps/framecpp -> ldas-tools-1.19.32-p1/linux-x86_64
    12582914    0 lrwxrwxrwx   1 controls 1001           20 Jul  1 13:15 /opt/rtapps/epics -> epics-
  • Checked out the RTS source for the version we'll be using: 2.9.4


  • built and installed all of the RTS components:
    • mbuf
    • mx_stream
    • daqd
    • nds
    • awgtpman
  • mx_stream is not working. Unknown why. It won't start on the front end machines (only tested on c1lsc so far) with the following error:
    controls@c1lsc ~ 1$ /opt/rtcds/caltech/c1/target/fb/mx_stream -s c1x04 c1lsc c1ass c1oaf c1cal -d fb:0
    mmapped address is 0x7ff7b71a0000
    send len = 263596
    mx_connect failed Remote Endpoint is Closed
    controls@c1lsc ~ 1$
    Have contact Keith T. and Rolf B. for backup.  This is a blocker, since this is what ferries the data from the front ends.
  • Rebuilt almost all models.  This was good.  Initially nothing would compile because of IPC creation errors, so I moved the old chans/ipc/C1.ipc file out of the way and generated a new one and then everything compiled (of course senders have to be compiled before receivers).
    I only had to fix a couple of things in the models themselves:
    • c1ioo - unterminated FiltCtrl inputs
    • C1_SUS_SINGLE_CONTROL - unterminated FiltCtrl inputs
    • c1oaf - bad part named "STATIC". There is some hacky namespace stuff going on in the RCG. I was able to just explode that part and it now works.
    • c1lsc - unterminated FiltCtrl inputs
    Haven't installed or tried to run anything yet, but the fact they compile is good.
    Some models are not compiling because they have C code in src blocks that are throwing errors:
    • c1lsc
    • c1cal
    It shouldn't be too hard to fix whatever is causing those compile errors.

That's it for today.  Will pick up again first thing tomorrow

  11389   Wed Jul 1 16:16:46 2015 IgnacioUpdateGeneralAccelerometers reinstalled for future huddle test

Today, I installed the Wilcoxon accelerometers in the table near the end of the mode cleaner. I only set three of them up instead of all six. They were set up just as Rana suggeted we should have them properly set up, i.e. cables being tighten up, and a box on top to prevent any airflow introducing any disturbances. We are planning on running the huddle test on these guys once the upgrade? to the interferometer is done.

The cables were tightly clamped to the table as shown below, I used a thick piece of shock absorbing rubber to do this.

 A small piece of thin rubber was used to hold each of the accelerometers tightly to the table in order not to damage them.

We had to borrow Megan's and Kate's piece of black foam in order to seal one of the sides properly, as the cable had to come out through somewhere. We didn't want to mess with drilling any holes into the box! 

There was a small crack even after using the foam. I sealed it up with duck tape.

The box isn't perfect, so there were multiple cracks along the bottom and top of it that could potentially allow for air to flow to the inside. Eric suggested that we should be super careful this time and do it right, so every crack was sealed up with ducktape.


Finally, we needed something heavy to be placed on top of the box to hold everything well. We used Rana's baby to accomplish this goal.

Just kidding! Rana's baby is too delicate for our purposes. A layman box of heavy cables was used instead. 


  11388   Wed Jul 1 11:45:48 2015 SteveUpdatePEMcleaning around ETMX chamber

Keven is our regular janitor is out for a few weeks.

The sub is careful- gentel Mario. We wiped arouind the vertex cambers north side on the floor and east arm racks.


Mario wiped the floor around the ETMX chamber today.

  11387   Wed Jul 1 10:01:25 2015 SteveUpdateVACRGA scan pd78 day 275



Attachment 1: rga275d.png
  11386   Wed Jul 1 09:33:31 2015 KojiUpdateGeneralShutters closed, watch dogs disabled for the RCG upgrade

I closed the PSL/GREEN shutters and shut off the LSC feedback/SUS watch dogs at 9AM PDT, to allow Jamie to start his disruptive work.

  11385   Tue Jun 30 20:26:24 2015 Eve UpdateGeneralMinor Summary Page Changes

I made several small, nit-picky changes to the summary pages.



I'm still working on getting used to editing the summary pages. I also wanted to change some of the easy-to-alter cosmetics of the pages.


What I did:

I changed axis ranges, axis labels, and typos throughout the summary pages. Read below for an excrutiating list of the minor details of my alterations, if you wish:

  • Changed axes on LSC control signals plots on the Summary tab (but will probably change these back to their original state)
  • Moved an OpLev plot from the Sandbox tab to "Eve" tab
  • Increased the y axis range on IOO MC2 Trans QPD and IMC REFLY RFPD DC plots (which may change when I better incorporate triggers into these plots)
  • Fixed title on IOO Whitened Spectrogram and Rayleigh Spectrogram
  • Fixed degree sign on Weather: Temperature and PSL Table Temperature
  • Fixed percent sign on Weather: Humidity


So far, everything looks good. I'll continue to make more changes later this week and hope to soon get on to more substatial changes.

  11384   Tue Jun 30 11:33:00 2015 JamieSummaryCDSprepping for CDS upgrade

This is going to be a big one.  We're at version 2.5 and we're going to go to 2.9.3.

RCG components that need to be updated:

  • mbuf kernel module
  • mx_stream driver
  • iniChk.pl script
  • daqd
  • nds

Supporting software:

  • ldas-tools (framecpp) 1.19.32-p1
  • libframe 8.17.2
  • gds
  • fftw 3.3.2

Things to watch out for:

  • RTS 2.6:
    • raw minute trend frame location has changed (CRC-based subdirectory)
    • new kernel patch
  • RTS 2.7:
    • supports "commissioning frames", which we will probably not utilize.  need to make sure that we're not writing extra frames somewhere
  • RTS 2.8:
    • "slow" (EPICS) data from the front-end processes is acquired via DAQ network, and not through EPICS.  This will increase traffic on the DAQ lan.  Hopefully this will not be an issue, and the existing network infrastructure can handle it, but it should be monitored.
  11383   Tue Jun 30 05:47:38 2015 ranaUpdateLSCUse BALUNs

The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.


  11382   Mon Jun 29 17:40:56 2015 Max IsiUpdateGeneralSummary pages "Code status" page fixed

It was brought to my attention that the "Code status" page (https://nodus.ligo.caltech.edu:30889/detcharsummary/status.html) had been stuck showing "Unknown status" for a while.
This was due to a sync error with LDAS and has now been fixed. Let me know if the issue returns.

  11381   Mon Jun 29 12:28:45 2015 ericqUpdateLSCALS reconstruction in progress

Turns out the reason that the BEATY signal wasn't working is that one of the two RF amplifiers (both of which are model ZHL-32A), isn't amplifying. Voltage at the pins is fine, so maybe its just broken. When the ZHL-3As that Rana ordered arrive, I'll install those. 

Switching the working amplifier between the two channels, and using a Marconi driving -20dBm (the Y green beatnote amplitude), the phase tracker output RMSs are 70Hz and 150Hz for X and Y, respectively, which isn't too exciting. There is enough whitening gain and filtering that I don't think ADC noise is an issue (The magnitude of the phase tracker Q is ~10kcounts after +6dB whitening gain). 

The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.

Also, for now, so I don't have to lug the marconi around everywhere, I'm currently driving both channels of the demod board with a spare 55MHz LO output of the LSC LO distribution box, which ends up being a factor of 5 smaller phase tracker error signal, but the noise level is about the same as with the marconi. 

  11380   Fri Jun 26 23:18:52 2015 Eve ChaseUpdateCDSSummary Page Updates


My SURF project largely focuses on updating and improving the 40m summary pages. I began to explore and experiment with the already existing summary page code to better understand the required code and eventually lead to tangible improvements of the summary pages.


What I did:

I practiced moving from one tab of the summary pages to another. Specifically, I copied the ETM Optical Levers plot from SUS: OpLev to Sandbox, without removing it from its original location. Additionally, I created my own tab to further test the summary pages, titled “Eve”.

KA ed: The configuration files are located at /cvs/cds/caltech/chans/GWsummaries. It is under svn control.


All changed appeared on the summary pages without much hassle and without any errors.

  11379   Fri Jun 26 03:24:18 2015 ericqUpdateLSCALS reconstruction in progress

Too sleepy to make full ELOG. Stay tuned. 

Two 25dB amplifiers (with fins!) are living in the top shelf on the PSL table, inputs currently grounded. I broke out the fused 24V power from the AOM driver to power the two amps and the AOM driver. I used the POP55 and AS165 heliax cables to get their outputs to the LSC rack, through delay lines, into demod board. 

Driving with -20dBm at 55MHz, the BEATX signal chain has about 60Hz RMS noise, which is about what I measured for driving the old beatbox with a marconi. High frequency noise is a much nicer shape, though. The BEATY signal didn't seem to be getting through, will double check soon. 

Still old delay cables, not nicely shielded or isolated or anything. We'll have to pipe the monitor signal from the LSC rack over to the control room analyzer now. 


  11378   Thu Jun 25 18:20:15 2015 ericqUpdateLSCALS reconstruction in progress

I've been working on getting a working ALS up and running. Things are in a bit of a transient state right now; I'm off to softball and dinner, and will resume work tonight. There will be a more detailed ELOG then, but here are some quick notes:

  • c1als has been gutted, phase trackers are working successfully in c1lsc frontend. All channel names remain the same. 
  • BEATX is on the ADC channels where AS165 used to live, BEATY at POP55. 
  • Used a marconi to drive the aLIGO LSC demod board in the LSC rack, was able to lock digital phase tracker on two channels
  • Noise looks pretty cruddy. Lots of 60Hz harmonics on both channels, maybe from marconi drive? Pickup in the delay line?
  • BEATX whitening filter maybe has something fishy going on; excess noise at 2kHz
  • Unclear if BEATY whitening filter is actually doing anything. 
  • Whitening gain switching works fine for both, though. Haven't revisited the switching code, so its controlled in the old RFPD place for now.
  • Whitening triggering is not set up, will require some thought and model work that isn't neccesary yet. 
  • Agilent analyzer, marconi, and old delay lines are currently stashed behind the LSC rack; I will resume work with them tonight. 

The main thing left to do is to install the RF amplifiers at the PSL table and route the green beat signals over to the LSC rack. I fear that some investigation into the whitening filters will be neccesary to make the performance adequate, however. 

  11377   Thu Jun 25 15:07:50 2015 SteveUpdatesafetySurfs Safety 2015

Jessica Pena, Megan Kelly, Eve Chase and Ignacio Magana received 40m specific basic safety traning today.

Attachment 1: surfs2015.jpg
  11376   Thu Jun 25 14:18:46 2015 Max IsiUpdateGeneralSummary page status

The pages are live again. Please allow some time for the system to catch up and process missed days. If there are any further issues, please let me know.
URL reminder: https://nodus.ligo.caltech.edu:30889/detcharsummary/


The summary pages have been down due to incompatibilities with a software update and problems with the LDAS cluster. I'm working at the moment to fix the former and the LDAS admins are looking into the latter. Overall, we can expect the pages will be fully functional again by Monday.


  11375   Thu Jun 25 12:03:42 2015 Max IsiUpdateGeneralSummary page status

The summary pages have been down due to incompatibilities with a software update and problems with the LDAS cluster. I'm working at the moment to fix the former and the LDAS admins are looking into the latter. Overall, we can expect the pages will be fully functional again by Monday.

  11374   Wed Jun 24 17:30:45 2015 ericqUpdateLSCDFD Delay length

This afternoon, I had a fruitful conversation with Rich Abbott about various kinds of cables.

I've sent an email to Steve to ask him to buy 2 x 50m LMR-195 cables, with male SMA connectors. Rich highly recommends these for their polyethylene insulation, which makes them less microphonic and less susceptible to thermal expansion, low loss, and multi-ply bonded foil shielding.

50m means that the peak to peak mixer output swing corresponds to about a MHz. 1nV of mixer output noise looks like ~6mHz frequency noise, for a Level 3 mixer appropriately driven. As a comparison, the lowest our in-loop green PDH error signals get is 0.1Hz/rtHz. 

The cable attenuation should be around 4.2dB at 50MHz, and 7.3dB at 150MHz, according to the data sheet. Thus, we should not be in the regieme where we are losing sensitivity to the attenuation. 

By my rough geometric estimation, these two should fit in the 2U box I got ahold of today fine. Jessica is designing the front panel. 

We currently have ~30m of RG-408 and RG-142 as our delay lines. 

  11373   Wed Jun 24 08:05:43 2015 SteveUpdatePSLlaser turned on

The laser went off around 11am yesterday. It was turned on

Attachment 1: laserWasOff.png
  11372   Tue Jun 23 11:18:20 2015 SteveUpdatePEMcleaning around chambers

Keven is our regular janitor is out for a few weeks.

The sub is careful- gentel Mario. We wiped arouind the vertex cambers north side on the floor and east arm racks.


  11371   Mon Jun 22 20:59:19 2015 ericqUpdateLSCDFD Delay length

I've been thinking a bit about what the ideal cable length / delay time for the upgraded ALS beatbox should be. Here are some thoughts, but no conclusions yet. 

If you're not running your beatbox mixer in compression, there are two competing effects when you change the cable length. At first, more delay gives better sensitivity, but this does not go on to infinity, because cable attenuation eventually kills your signal. It turns out that the ideal length can be derived to be whatever length gives you 20/ln(10) = -8.7dB of attenuation. Frank found this out in PSL ELOG 825, and I found an HP document that derives this (and other useful DFD math) to the wiki, here.

In PSL ELOG 826, Frank calculated this ideal length for a 160MHz carrier in various kinds of cables. 

However, this is not the end of the story. In the case of the DFD, we actually benefit from operating the mixer in compression, as makes our sensitivity less sensitive to flucuations in the beat amplitude. In this situation, the HP doc states "For maximum sensitivity, more delay can be added until the signal level out of the delay line is 8.7dB below the phase detector (mixer) compression point." I'm not sure I really understand the logic behind this statement, though. 

Lastly, Koji mentioned the fact that the splitter in the demod board does not split at exactly 90 degrees, making the trajectory in the IQ plane an ellipse. This means that if the beat signal is moving around the ellipse a lot, or even wrapping around it, we can suffer from some nonlinear signal conversion. Also, if the raw DFD sensitivity is very high, the free swinging mirrors will cause the signal to swing around faster than the phase tracker can keep up. This should be easy to avoid, however; I doubt we will use so much cable that the beat would move by so much. 

I intend to take all of this into account when picking a cable length! Jessica is going to help us make a nice box for them, too. 

  11370   Mon Jun 22 14:53:37 2015 ranaSummaryLSCX/Y green beat mode overlap measurement redone
  • Why is there a factor of 20 power difference? Some of it is the IR laser power difference, but I thought that was just a factor of 4 in green.
  • Why is the mode overlap only 50% and not more like 75%?
  • IF we have enough PSL green power, we could do the Y-beat with a 80/20 instead of a 50/50 and get better SNR.
  • The FFD-100 response is more like 0.33 A/W at 532 nm, not 0.25 A/W.

In any case, this signal difference is not big, so we should not need a different amplifier chain for the two signals. The 20 dB of amplification in the BeatBox was a fine way, but not great in circuit layout.

The BBPD has an input referred current noise of 10 pA/rHz and a transimpedance of 2 kOhm, so an output voltage noise of 20 nV/rHz (into 50 Ohms). This would be matched by an Amp with NF = 26 dB, which is way worse than anything we could bur from mini-circuits, so we should definitely NOT use anything like the low-noise, low output power amps used currently (e.g. ZFL-1000LN....never, ever use these for anything). We should use a single ZHL-3A-S (G = 25 dB, NF < 6 dB, Max Out = 30 dBm) for each channel (and nothing else) before driving the cables over to the LSC rack into the aLIGO demod board. I just ordered two of these now.

  11369   Mon Jun 22 14:21:42 2015 SteveMetaphysicsTreasureJenne and Den graduated

Last supper before departing at  "Grazie" El Portal. All the best on your journey!

Attachment 1: ls.jpg
  11368   Mon Jun 22 12:57:09 2015 ericqSummaryLSCX/Y green beat mode overlap measurement redone

I took measurements at the green beat setup on the PSL table, and found that our power / mode overlap situation is still consistent with what Koji and Manasa measured last September [ELOG 10492]. I also measured the powers at the BBPDs with the Ophir power meter.

Both mode overlaps are around 50%, which is fine. 

The beatnote amplitudes at the BBPD outputs at a frequency of about 50MHz are -20.0 and -27.5 dBm for the X and Y beats, respectively. This is consistent with the measured optical power levels and a PD response of ~0.25 A/W at 532nm. The main reason for the disparity is that there is much more X green light than Y green light on the table (factor of ~20), and the greater amount of green PSL light on the Y BBPD (factor of ~3) does not quite make up for it. 

One way to punch up the Y beat a little might be to adjust the pickoff optics. Of 25uW of Y arm transmitted green light incident on the polarizing beamsplitter that seperates the X and Y beams, only 13uW makes it to the Y BBPD, but this would only win us a couple dBms at most. 

In any case, with the beat setup as it exists, it looks like we should design the next beatbox iteration to accept RF inputs of around -20 to -30 dBm. 

In the style of the referenced ELOG, here are today's numbers. 

            XARM   YARM 
o BBPD DC output (mV)
 V_DARK:   +  1.0  + 2.2
 V_PSL:    +  7.1  +21.3
 V_ARM:    +165.0  + 8.2

o BBPD DC photocurrent (uA)
I_DC = V_DC / R_DC ... R_DC: DC transimpedance (2kOhm)

 I_PSL:       3.6   10.7
 I_ARM:      82.5    4.1

o Expected beat note amplitude
I_beat_full = I1 + I2 + 2 sqrt(e I1 I2) cos(w t) ... e: mode overwrap (in power)

I_beat_RF = 2 sqrt(e I1 I2)

V_RF = 2 R sqrt(e I1 I2) ... R: RF transimpedance (2kOhm)

P_RF = V_RF^2/2/50 [Watt]
     = 10 log10(V_RF^2/2/50*1000) [dBm]

     = 10 log10(e I1 I2) + 82.0412 [dBm]
     = 10 log10(e) +10 log10(I1 I2) + 82.0412 [dBm]

for e=1, the expected RF power at the PDs [dBm]
 P_RF:      -13.2  -21.5

o Measured beat note power (no alignment done)     
 P_RF:      -20.0  -27.5  [dBm] (53.0MHz and 46.5MHz) 
    e:       45.7   50.1  [%]                         


  11367   Sun Jun 21 13:21:03 2015 ranaUpdatePEMWilcoxon Accelerometer Huddle Test

To improve our sensor noise estimate, the ACC cables should be clamped using a rubber pad and a big dog clamp on the SP tabletop before exiting the table. Also the sensors themselves should be covered with a foam box or a double cardboard box. The high-frequency acoustic noise may limit the 10 Hz performance since piezos are not very linear.


I've set up the Wilcoxen accelerometers on the SP table for a huddle test, to estimate their noise levels. 

They're clamped down to the table along the same axis, and their housings are in good contact, in hopes to make them as correlated as possible. 

Wilcoxon. Not Wilconox or Wilco or Wilcoxen. Have some pity on future keyword searchers.

  11366   Fri Jun 19 16:54:20 2015 JenneUpdateComputer Scripts / ProgramsWiener scripts in scripts directory

I have put the Wiener filter scripts into  /opt/rtcds/caltech/c1/scripts/Wiener/  .  They are under version control. 

The idea is that you should copy ParameterFile_Example.m into your own directory, and modify parameters at the top of the file, and then when you run that script, it will output fitted filters ready to go into Foton.  (Obviously you must check before actually implementing them that you're happy with the efficacy and fits of the filters). 

Things to be edited in the ParameterFile include:

  • Channel names for the witness sensors (which should each have a corresponding .txt file with the raw data)
  • Channel name for the target
  • Folder where this raw data is saved
  • Folder to save results
  • 1 or 0 to determine if need to load and downsample the raw data, or if can use pre-downsampled data
    • This should probably be changed to just look to see if the pre-downsampled data already exists, and if not, do the downsampling
  • 1 or 0 to determine if should use actuator pre-weighting
  • Data folder for measured actuator TFs (only if using actuator pre-weighting)
    • Actuator TFs can be many different exported text files from DTT, and they will be stitched together to make one set of measurements, where all points have coherence above some quantity (that you set in the ParameterFile)
  • Coherence threshold for actuator data (only use data points with coherence above this amount)
  • Fit order for actuator transfer function's vectfit
  • 1 or 0 to decide if should use preweighting filter
  • zeros and poles for preweighting filters
  • 1 or 0 to decide if should use lowpass after Wiener filters (will be provided corresponding SOS coefficients for this filter, if you say yes)
  • Lowpass filter parameters: cuttoff freq, order and ripple for the Cheby filter
  • New sample rate for the data
  • Number of Wiener filter taps
  • Decide if use brute force matrix inversion or Levinson method
  • Calibrations for witnesses and target
  • Fit order for each of the Wiener filters

I think that's everything that is required.

  11365   Fri Jun 19 03:00:56 2015 ericqUpdateCDSLibrary Model Parts examined

All simulink diagrams being used at the 40m are now under version control. I have compiled, installed, and restarted all current models to make sure that the files are all in a working state, which they seem to be. I have checked the latest version of the userapps svn repository to /opt/rtcds/userapps2.9, to compare the files therein with our current state. 

Surprisingly, only two files in the userapps svn have been changed since they were checked out here, and only one of these is a real change of any kind. 

LSC_TRIGGER.mdl was edited at some point simply to align some drawn lines; no functionality was changed. 

SCHMITTTRIGGER.mdl was edited to change the "INVERT" epics channel from an arbitrary EPICS input, to binary (true/false) input. This does not change the connectivity diagram, and in fact, I don't think we use this option in any of our scripts, nor is it exposed on our medm screens. 

Thus, I think that the only place for block changes can bite us is changes in the fundamental blocks in CDS_PARTS that are used in our custom 40m library parts. 

For posterity, these are the files used in compiling all of our running models. (Path base: /opt/rtcds/userapps/release)

  11364   Fri Jun 19 01:55:35 2015 ericqUpdateGreen LockingBeatBox Assay: not looking good

But, it has no whitening. Can we do the whitening part externally? Perhaps we can run the RF signals from the output of the beat RF Amps over to the LSC rack and then put the outputs into the LSC Whitening board and acquire the signals in the LSC ?

I like this idea; it gives us more control over the whitening, and saves the IPC delay. We could use the currently vacant AS165 and POP55 channels. 

We'd only have to move the phase trackers to c1lsc, which means 12 more FMs total. This is really the only part of the c1als model our current system uses, the rest is from before the ALS->LSC integration. 

  11363   Fri Jun 19 01:24:26 2015 rana, kojiUpdateGreen LockingBeatBox Assay: not looking good

We had decided a few days ago, to bypass the IF part of the BeatBox board and put some of the Haixing Maglev generic filter boards in there so that we could get more whitening and also have it be low noise.

Tonight we wondered if we can ditch the whole BeatBox and just use the quad aLIGO demod box (D0902745) that Rich gave us a few years ago. Seems like it can.

But, it has no whitening. Can we do the whitening part externally? Perhaps we can run the RF signals from the output of the beat RF Amps over to the LSC rack and then put the outputs into the LSC Whitening board and acquire the signals in the LSC ?

  11362   Wed Jun 17 15:31:50 2015 ericqUpdatePEMAccelerometers fully installed

MC1 accelerometer has been plugged in. The modecleaner locking has been intermittent today, but I looked at a 15 minute lock in DTT, looking at the STS1 seismometer and both accelerometer triplets. Plot and DTT xml attached.

For the sake of not cluttering up everything with legends, the coherence plots are organized by direction (x, y, z), and include the coherence of each of the three sensors (sts, acc1, acc2) with the IMC control signal and the IMC transmitted RIN. 

Some remarks:

  • The 1 Hz pendulum motion is about equal amounts of X and Y, which makes sense, as MC1 and 3 are at an angle
  • The ~3 Hz stack motion seems to be entirely in the X direction. Why?
  • The bounce/roll bands are strongly coherent with Z motion at MC2. 
  • The STS does not appear to have any low frequency advantage over the accelerometers, in terms of coherence, contrary to what I would expect even without a thermal enclosure. 
  • The control signal and RIN RMSs are clearly dominated by noise in the 1-3Hz band, where we have reasonable coherence. Good prospects for noise subtraction...

Attachment 1: IMCcoherence_Jun172015.xml.zip
Attachment 2: IMCcoherence.png
  11361   Mon Jun 15 22:36:40 2015 rana, kojiUpdateGreen LockingBeatBox Assay: not looking good

Because the ALS beatbox schematic is out-of-date and misleading, we pulled the box to photograph the current implementation and figure out how to proceed. The box is out on the EE bench right now. Schematic Doc added to 40m Document tree: https://dcc.ligo.org/LIGO-D1102241. Some notes:

  1. The soldering on this board is pretty messy and there are a lot of flying wire and flying component hacks. I wouldn't trust all of the connections.
  2. The GV-81 RF amps in the front end are both stuffed. The 1 dB compression point is 19 dBm, so we want to use them below 10 dBm output. They have a gain of +10.5 dB, so that means they should not be used with and input to the beatbox of more than -10 dBm. Otherwise there will be nonlinear noise generation.
  3. Not stuffed: U1-Comparator, A1-attenuator, U2-splitter.
  4. Why is the filter after the mixer only 2nd order?? That's not a valid filter choice in any RF world. How much do we want to cut off the 2f mixer output before sending into our low noise, audio frequency (and prone to downconversion) amplifier? The Mini-Circuits amplifiers would have given us >60 dB attenuation in the stop band. This one is only going to give us 20-30 dB when the beat frequency is low. Get rid of diplexer. The schematic claims that its just one pole?? Seems like a 2nd order LP filter to me.
  5. The modified schematic (see Koji elog 8855) shows that an OP27 is used for the whitening stage. The current noise of the OP27 with the 3k resistor makes the OP27 current noise dominate below 1 Hz. And what is going on with that filter capacitor choice? We never want to use these tiny things for sensitive filter applications. (cf. Sigg doc on resistor and capacitor choice, the noise reduction book by Ott, H&H, etc.). That's why we have the larger metal-poly, paper, mylar, etc. caps sitting around.

Probably we ought to install a little daughter board to avoid having to keep hacking this dead horse. Koji has some of Haixing'g maglev filter boards. Meanwhile Koji is going to make us a new beatbox circuit in Altium and we can start fresh later this summer.

Interesting link on new SMD cap technology.

Photos of circuit as found

  11360   Mon Jun 15 20:36:48 2015 ranaUpdateIOOIOO QPDs centred

after re-aligning the beam into the PMC, I touched up the steering mirros into the IOO QPDs so that the beams are now centered again. Please don't adjust these references without prior authorization and training.

This plot shows the 10-minute trends for these QPDs over the last 400 days.

Attachment 1: Untitled.png
  11359   Mon Jun 15 16:55:39 2015 ericqUpdatePEMAccelerometers installed

The accelerometers have been installed at MC1 and MC2. MC2 data is live, I haven't yet run the cables from the MC1 set to the preamp yet, though. 

Attachment 1: MC2.jpg
Attachment 2: MC1.jpg
Attachment 3: mc2accspectra.png
  11358   Mon Jun 15 11:54:44 2015 ericqUpdateCDSParts not in SVN

I ran the following command to find which models/parts are not under version control, or have modifications not commited:

grep "mdl" $(cat models.txt) | awk '{print $NF}' | sort | uniq | xargs svn status

models.txt includes lines like "/opt/rtcds/caltech/c1/rtbuild/c1ass.log" for each running model. These are the build logs that detail every file being sourced. 

The resultant list is:

?       /opt/rtcds/userapps/release/cds/common/models/BLRMS_HIGHFREQ.mdl
C       /opt/rtcds/userapps/release/cds/common/models/rtdemod.mdl
M       /opt/rtcds/userapps/release/cds/common/models/SCHMITTTRIGGER.mdl
?       /opt/rtcds/userapps/release/isc/c1/models/blrms.mdl
?       /opt/rtcds/userapps/release/isc/c1/models/IQLOCK.mdl
?       /opt/rtcds/userapps/release/isc/c1/models/PHASEROT.mdl
?       /opt/rtcds/userapps/release/sus/c1/models/QPD_WHITE_CTRL_MUX.mdl

I will commit the uncommited c1 parts, and think about what to do about the rtdemod and SCHMITTRIGGER parts. 

Once I check out the latest revision of the userapps repo (in a seperate location), I will do something similar to look for models that have changed since the svn revision that is checked out in our running system. 

  11357   Sat Jun 13 23:52:14 2015 ericqUpdateLSCBeatbox needs whitening gain

Nice find!

We ought to use our noise model of the ALS signal chain to determine what the right gains are, rather than hunt and peck. More likely we'll start from the right gains.​

Once the gains and/or whitening filters make sense, maybe we'll see some effect from fixing the green PDH loops.

  11356   Sat Jun 13 18:37:46 2015 ericqUpdateLSCBeatbox needs whitening gain

Here are the promised details!

I was worried about the lack of whitening gain, as I saw that the DC phase tracker Q output (which is the magnitude of the signal in the beatbox's I-Q plane) was no more than 200 ADC counts for X (~120mV), and 800 for Y (~500mV). I.e. this is the largest DC value that either the I or Q ADC channels can see, and the RMS fluctuations are on the order of mV, meaning we're wasting our entire ADC rangeno

However, I also had doubts about this since, even in the nominal state, the ASD of the ADC signals before dewhitening was higher than the expected ADC noise level. However, because of the non-linearity of the conversion of the BEAT_I and Q signals into the phase tracker output, evaluating the contribution of the beatbox output and ADC input voltage noises takes a few more steps. 

So, I hooked up a Marconi as the signal source for the beatbox's X channel , with no modulation (presumably the phase noise of the Marconi output is significantly lower than the sensitivity of the beatbox). For all of these measurements, the beat frequency was kept around 50MHz, with an amplitude of -30dBm on the control room analyzer, which is a typical X ALS operating point. 

At this point, the beatbox output noise was below the ADC noise (as measured by an SR785). Nevertheless, I found that the beat spectrum driven by the Marconi lined up to be very close to the ALS beat spectrum across a wide band, explaining much of the noise. 

At this point, I inserted SR560s in between the beatbox I and Q outputs, and the AA chassis leading to the ADC. A gain of 10 reduced the resultant phase tracker noise by that same factor at nearly all frequencies. A further increase in gain did not lead to the noise changing appreciably, probably because the real beatbox noise was now contributing, as is suggested by some common peaks in the direct beatbox output phase tracker spectra.

Going back to the real green beat signal with the SR560s still at G=10, I obtained the result shown in ELOG 11355. I will soon repeat this process with the Y ALS. 

As I mentioned in the previous ELOG, we may be further helped by more whitening gain than can be provided by the SR560s (and we obviously need a robust long term circuit for this gain). If it then turns out we are limited by beatbox noise to a degree we are not happy with, we could perhaps look into reintroducing some RF gain into the X beat. As Koji mentions in ELOG 8855, the beatbox operates best at an RF input of around -4dBm.

Attachment 1: ADCdiagnosis.png
  11355   Fri Jun 12 17:09:58 2015 ericqUpdateLSCBeatbox needs whitening gain

Short entry just to preview a new development; more detail about this investigation will soon follow. 

The beatbox I and Q signals are too small at the ADC! I was able to reduce the RMS out of loop ALS sensitivity (arm locked on POX) by 300Hz with G=10 SR560s between the beatbox output and the ADC whitening chassis input. Increasing the beat note amplitude via RF amplifier had no positive effect. 

There is still a reasonable gap between this and the beatbox's noise levels, as measured with a marconi. There may be additional headroom for whitening gain; the SR560 maximum output range is smaller than the ADC input range. 

The high frequency noise has >0.5 coherence with the PDH error signal above a kHz or so, but not much below that. 

We should probably either modify the output whitening of the beatbox, or introduce some (variable?) whitening gain in a seperate circuit. 

Attachment 1: beatGain.png
  11354   Fri Jun 12 08:40:17 2015 SteveUpdateVACVacuum comp. rebooted

Koji and Steve,

One computer expert and one vacuum expert required.


The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

Atm 1,  problem condition: gauges are not reading for a week, error message "NO COMM" and all computer LEDs are green

Atm 2, prepare to safe reboot:

            a, close V1, disconnect it's power cable and turn off Maglev, wait till rotation stops

            b, close PSL shutter ( take adrenaline if needed )

            c, close V4, V5, VA6 valves and disconnect their cables. "Moving" error message indicating this condition.

               V1 is not showing "Moving" because its power cable disconnected only! It will show it if its position indicator cable is disconnected too. There is no need for that.

               These valves closed and disabled will not allow accidental venting of main volume.

            d, push reset, reseting c1vac2 will reset c1vac1 also, wait ~ 6 minutes

"Vacuum Normal" valve configuration was restored after succesful reboot as follows:

             a, reconnect cable and open V4 and V5 at P2 & P3 <1e-1 Torr

             b, observe that P2  < 1e-3 Torr and retsart Maglev

             c, wait till Maglev reaches full speed of 560 Hz and reconnect-open V1

             d, reconnect-open VA6 at P3 <1e-3 Torr

NOTE: VM1 valve was locked in open position and it was not responding before and after reboot

          Error message on Atm2 is indicating this locked condition: "opening VM1 will vent IFO"

          This is a fauls message. The valve is frozen in open position. We need a softwear expert help.



Attachment 1: vacMonNoGauges.png
Attachment 2: prepReboot.png
  11353   Thu Jun 11 19:40:59 2015 KojiUpdateVACVacuum comp. rebooted

The serial connections to the vacuum gauges were recovered by rebooting c1vac1 and c1vac2.

Steve claimed that the vacuum screen had showed "NO COMM" at the vacuum pressure values.
The epics connection to c1vac was fine. We could logged in to c1vac1 with telnet too although c1vac2 had no response.

After some inspection, we decided to reboot the slow machines. Steve manually XXXed YYY valves (to be described)
to prepare for any possible unwanted switching. Initially Koji thought only c1vac2 can be rebooted. But it was wrong.
If the reset button is pushed, all of the modules on the same crate is reset. So everything was reset. After ~3min we still
don't have the connection to c1vac1 restored. We decided to another reboot. This time I pushed c1vac1 reset button.
After waiting about two minutes, the ADCs started to show green lights and the switch box started scanning.
We recovered the telnet connection to c1vac1 and epics functions. c1vac2 is still note responding to telnet, and
the values associated with c1vac2 are still blank.

Steve restored the valves and everything was back to normal.

  11352   Wed Jun 10 15:54:14 2015 SteveUpdateVACVacuum comp. rebooted

Koji and Steve succeded rebooting C1vac1, C1vac2 and pressure reading is working now

More tomorrow .........


Attachment 1: afterReb061015.png
  11351   Wed Jun 10 03:19:58 2015 ericqUpdateLSCAUX PDH error measured in CDS

Looking over the old noise budget in the green locking paper, it seems the main technical noise sources were the AUX PDH error and DFD noise. I'm working on quantifying the current state of these noises. 

Rather than lugging out the analyzers every time I wanted to make a measurement of the AUX PDH error signals, I set out to make the existing digitized channels (ALS-[X/Y]_ERR_MON) usable for easier, and continuous, monitoring. Sadly, up until now the signals were poorly conditioned, and drowning in ADC noise. (When locked, the Y error signal was only +-10 counts!)

Of course, given the bandwidth of the green servos (10kHz), this won't tell us the full story of the what the green PDH lock is doing, but does indicate how much residual frequency noise exists in the ALS control band. 

I'm currently using SR560s at G=20 at each end to amplify the ERR MON outputs of the uPDH boards before sending them to the ADCs; now that I've found a gain that works, I'll modify the error point monitor buffer opamps inside the uPDH boards themselves during the daytime. 

The AUX Y error signal was going into an AA board with some funky filtering going on that I didn't want to mess with. Instead, I've moved the signal to the pentek generic board whose first four channels are used for the oplev segments, and the second quartet are unused, save for the TST channel I hooked up yesterday. 

On the pentek board, I changed the 4th order 800Hz lowpass to a 4th order 8kHz lowpass on the last three channels through some resistor swapping. (At first it was just the last two, but I found I was getting weird signals in the 32nd channel; and if I recall correctly from my cymac work, the 32 ADC channel is used for some timing signal or something...). I also turned off the 1:10 whitening filters on the last four channels via PCB jumper. 

I then unhooked the PZT drive and let the PDH error signals swing around, to calibrate the ADC counts into HZ. Now, the ALS-[X/Y]_ERR_MON_OUT_DQ are calibrated in green Hz! Here are the spectra.

As we've seen in the past (ELOG 10464), the X green is limited by the dark noise of the PD from 10-100Hz. This isn't so great. The RMS noise from 300Hz downwards (which is maybe the band where the ALS control would inject noise into the mirror motion) is about Y:10Hz X:40Hz.

During this time, the test masses were not under any longnitudinal control, so I'm not sure why there is such a difference in the height of the suspension resonance peaks, unless there's some differene in the low frequency PDH TFs that I've forgotten about. 

Now, with these references, we can easily check if the PDH loops change qualitatively over longer time periods.

I'll be including the effect of these noises in the upcoming revised ALS noise budget.

Attachment 1: AUXspectra.png
  11350   Wed Jun 10 02:50:39 2015 ericqUpdatePEMWilcoxon Accelerometer Huddle Test

Here are some results for the 3-corner hat subtraction for the six accelerometers, from ~1 hour of data that didn't look to have any sharp features/glitches from human activity in the lab. 

I used the python uncertainties package to try and get an estimate of the uncertainty in the subtracted noise floor, by taking into account every possible possible combination of 3 sensors and the fluctuations in the spectrograms of the subtracted signals. I've attached the python code if anyone is interested in the implementation. 

I pulled out the accelerometer data sheets to read their slightly varying V/g calibration (which differs on the order of a few percent from unit to unit). This improved the subtraction factor from ~20 to over 100 at some frequencies. I've edited the filter modules such that the OUT_DQ channels are already calibrated into m/s^2.

Attachment 1: hats2Acc.png
Attachment 2: 3hatcode.zip
  11349   Tue Jun 9 10:57:12 2015 SteveUpdateSUS BS oplev laser tuned

Lenses removed from oplev beam path at elog entry 11246


Attachment 1: 100dBStrend0425.png
Attachment 2: 18dBSdrift.png
  11347   Mon Jun 8 15:51:31 2015 ericqUpdateCDSRCG Diagnostics

I've started making some model changes for RCG diagnostic tests. 

I put some blocks down in C1TST and C1RFM to test the delays of all-digital loops and one loop with a direct DAC->ADC (which currently uses a janky 1-pin lemo -> BNC -> 2-pin lemo situation (which will be improved)). 

Here's what C1TST looks like now. 

I've taken TFs of all three loops. The all digital loops are flat on the order of microdBs. 

The delay in loop A (single loop, one model) is consistent with one 16k cycle, plus or minus 0.25 nsec. 

The delay in loop C (single loop, two models connected via RFM) is consistent with two 16k cycles, plus or minus 0.5 nsec. 

I haven't yet grabbed the whitening and AA/AI shapes for loopB, to calibrate the real delay. 

All of these files currently live in /users/ericq/2015-06-CDSdiag, but I'll make somewhere outside of the user directory to collect all of these tests soon. 

Attachment 1: newTST.png
ELOG V3.1.3-