40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 110 of 336  Not logged in ELOG logo
ID Date Author Type Category Subject
  11419   Thu Jul 16 03:01:57 2015 ericqUpdateLSCOld beatbox hooked back up

I was having issues trying to get reasonable noise performance out of the aLIGO demod board as an ALS DFD. Terminating the inputs to the LSC whitening inputs did not show much 60Hz noise, and an RMS in the single Hz range. 

A 60Hz line of hundreds of uV was visible in the power spectrum of the single ended BNC and double-ended DB25 outputs of the board no matter how I drove or terminated.

So, I tried out hooking up the ALS beatbox. It turns out to work better for the time being; not only is the 60Hz line in the analog outputs about ten times smaller, the broadband noise floor in the resultant beat spectrum when driven by a 55MHz LO on the LSC rack is a fair bit lower too. I wonder if this is due to not driving the aLIGO board LO at the +10dBm it expects. With the amplifiers and beat note amplitudes we have, we'd only be able to supply around 0 dBm anyways. 

Here's a comparison of the aLIGO board (black) and ALS beatbox (dark green) driven with the 55MHz LO, both going through the LSC whitening filters for a resultant magnitude of 3kCounts in the I-Q plane. The RMS sensing noise is about 30 times lower for the beatbox. (Note, this is with the old delay cables. When we switch to the 50m cables, we'll win further frequency noise sensitivity through the better degrees->Hz calibration.) I'm very interested to see what the green beat spectrum looks like with this setup. 

Not only is the 60Hz line smaller, there is simply less junk in the beatbox signal. I did not expect this to be the case. 

There were some indications of funky status of the aLIGO board: channels 3 and 4 are totally nonfunctioning, so who knows what's going on in there. I've pulled it out, to take a gander if I can figure out how to make it suitiable for our purposes. 

Attachment 1: beat_comparison.png
beat_comparison.png
Attachment 2: aLIGO_vs_beatbox.xml.zip
  11418   Thu Jul 16 01:04:21 2015 ericqUpdateGeneralStarting IFO recovery, DAC troubles

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  crying

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 


As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

  11417   Wed Jul 15 18:19:12 2015 JamieSummaryCDSCDS upgrade: tentative stabilty?

Keith Thorne provided his eyes on the situation today and had some suggestions that might have helped things

Reorder ini file list in master file.  Apparently the EDCU.ini file (C0EDCU.ini in our case), which describes EPICS subscriptions to be recorded by the daq, now has to be specified *after* all other front end ini files.  It's unclear why, but it has something to do with RTS 2.8 which changed all slow channels to be transported over the mx network.  This alone did not fix the problem, though.

Increase second trend frame size.  Interestingly, this might have been the key.  The second trend frame size was increased to 600 seconds:

start trender 600 60;

The two numbers are the lengths in seconds for the second and minute trends respectively.  They had been set to "60 60", but Keith suggested that longer second trend frames are better, for whatever reason.  It seems he may be right, given that daqd has been running and writing full and trend frames for 1.5 hours now without issue. 


As I'm writing this, though, the daqd just crashed again.  I note, though, that it's right after the hour, and immediately following writing out a one hour minute trend file.  We've been seeing these hour, on the hour, crashes of daqd for quite a while now.  So maybe this is nothing new.  I've actually been wondering if the hourly daqd crashes were associated with writing out the minute trend frames, and I think we might have more evidence to point to that.

If increasing the size of the second trend frames from 60 seconds (35M) to 600 seconds (70M) made a difference in stability, could there be an issue since writing out files that are smaller than some value?  The full frames are 60M, and the minute trends are 35M.

  11416   Wed Jul 15 17:05:06 2015 JessicaUpdateGeneralBandpass Pre-Filter created

I applied a bandpass filter to the accelerometer huddle data as a pre-filter. The passband was from 5 Hz to 20 Hz. I found that applying this pre-filter did very little when comparing the PSD after pre-filtering to the PSD with no pre-filtering. There was some improvement though, just not a significant amount. For some reason, it also seemed as though the second accelerometer improved the most from pre-filtering the data, while the first and third remained closer to the unfiltered noise. Also, I have not yet figured out a consistent method for choosing passband ripple and stopband attentuation, both of which determine how good the filter is. 

My next step in pre-filtering will be determining a good method for choosing passband ripple and stopband attenuation, along with implementing other pre-filtering methods to combine with the bandpass filter. 

Attachment 1: acc1.png
acc1.png
Attachment 2: acc2.png
acc2.png
Attachment 3: acc3.png
acc3.png
  11415   Wed Jul 15 13:19:14 2015 JamieSummaryCDSCDS upgrade: reducing mx end-points as last ditch effort

I tried one last thing, suggested by Keith and Gerrit.  I tried reducing the number of mx end-points on fb to zero, which should reduce the total number of fb threads, in the hope that the extra threads were causing the chokes.

On Tue, Jul 14 2015, Keith Thorne <kthorne@ligo-la.caltech.edu> wrote:
> Assumptions
>  1) Before the upgrade (from RCG 2.6?), the DAQ had been working, reading out front-ends, writing frames trends
>  2) In upgrading to RCG 2.9, the mx start-up on the frame builder was modified to use multiple end-points
> (i.e. /etc/init.d/mx has a line like
> # 1 10G card - X2
> MX_MODULE_PARAMS="mx_max_instance=1 mx_max_endpoints=16 $MX_MODULE_PARAMS"
>  (This can be confirmed by the daqd log file with lines at the top like
> 263596
> MX has 16 maximum end-points configured
> 2 MX NICs available
> [Fri Jul 10 16:12:50 2015] ->4: set thread_stack_size=10240
> [Fri Jul 10 16:12:50 2015] new threads will be created with the stack of size 10
> 240K
>
> If this is the case, the problem may be that the additional thread on the frame-builder (one per end-point) take up so many slots on the 8-core
> frame-builder that they interrupt the frame-writing thread, thus preventing the main buffer from being emptied.  
>
> One could go back to a single end-point. This only helps keep restart of front-end A from hiccuping DAQ for front-end B.
>
> You would have to remove code on front-ends (/etc/init.d/mx_stream) that chooses endpoints. i.e.
> # find line number in rtsystab. Use that to mx_stream slot on card (0-15)
> line_num=`grep -v ^# /etc/rtsystab | grep --perl-regexp -n "^${hostname}\s" | se
> d 's/^\([0-9]*\):.*/\1/g'`
> line_off=$(expr $line_num - 1)
> epnum=$(expr $line_off % 2)
> cnum=$(expr $line_off / 2)
>
>     start-stop-daemon --start --quiet -b -m --pidfile /var/log/mx_stream0.pid --exec /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream -- -e 0 -r "$epnum" -W 0 -w 0 -s "$sys" -d x2daqdc0:$cnum -l /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream_logs/$hostname.log

As per Keith's suggestion, I modified the mx startup script to only initialize a single endpoint, and I modified the mx_stream startup to point them all to endpoint 0.  I verified that indeed daqd was a single MX end-point:

MX has 1 maximum end-points configured

It didn't help.  After 5-10 minutes daqd crashes with the same "0 empty blocks" messages.

I should also mention that I'm pretty sure the start of these messages does not seem coincident with any frame writing to disk; further evidence that it's not a disk IO issue.

Keith is looking at the system now, so we if he can see anything obvious.  If not, I will start reverting to 2.5.

  11414   Tue Jul 14 17:14:23 2015 EveSummarySummary PagesFuture summary pages improvements

Here is a list of suggested improvements to the summary pages. Let me know if there's something you'd like for me to add to this list!

  • A lot of plots are missing axis labels and titles, and I often don't know what to call these labels. I could use some help with this.
  • Check the weather and vacuum tabs to make sure that we're getting the expected output. Set the axis labels accordingly. 
  • Investigate past periods of missing data on DataViewer to see if the problem was with the data requisition process, the summary page production process, or something else.
  • Based on trends in data over the past three months, set axis ranges accordingly to encapsulate the full data range.
  • Create a CDS tab to store statistics of our digital systems. We will use the CDS signals to determine when the digital system is running and when the minute trend is missing. This will allow us to exclude irrelevant parts of the data.
  • Provide duty ratio statistics for the IMC.
  • Set triggers for certain plots. For example, for channels C1:LSC-XARM OUT DQ and page 4 LIGO-T1500123–v1 C1:LSC-YARM OUT DQ to be plotted in the Arm LSC Control signals figures, C1:LSCTRX OUT DQ and C1:LSC-TRY OUT DQ must be higher than 0.5, thus acting as triggers.
  • Include some flag or other marking indicating when data is not being represented at a certain time for specific plots.
  • Maybe include some cool features like interactive plots.
  11413   Tue Jul 14 17:06:00 2015 jamieUpdateCDSrunning test on daqd, please leave undisturbed

I have reverted daqd to the previous configuration, so that it's writing frames to disk.  It's still showing instability.

  11412   Tue Jul 14 16:51:01 2015 JamieSummaryCDSCDS upgrade: problem is not disk access

I think I have now determined once and for all that the daqd problems are NOT due to disk IO contention.

I have mounted a tmpfs at /frames/tmp and have told daqd to write frames there.  The tmpfs exists entirely in RAM.  There is essentially zero IO wait for such a filesystem, so daqd should never have trouble writing out the frames.

But yet daqd continues to fail with the "0 empty blocks in the buffer" warnings.  I've been down a rabbit hole.

  11411   Tue Jul 14 16:47:18 2015 EveUpdateSummary PagesSummary page updates continue during upgrade

I've continued to make changes to the summary pages on my own environment, which I plan on implementing on the main summary pages when they are back online.

 

Motivation:

I created my own summary page environment and manipulated data from June 30 to make additional plots and change already existing plots. The main summary pages (https://nodus.ligo.caltech.edu:30889/detcharsummary/ or https://ldas-jobs.ligo.caltech.edu/~max.isi/summary/) are currently down due to the CDS upgrade, so my own summary page environment acts as a temporary playground to continue working on my SURF project. My summary pages can be found here (https://ldas-jobs.ligo.caltech.edu/~eve.chase/summary/day/20150630/); they contian identical plots to the main summary pages, except for the Summary tab. I'm open to suggestions, so I can make the summary pages as useful as possible.

 

What I did:

  • SUS OpLev: For every already existing optical lever timeseries, I created a corresponding spectrum, showing all channels present in the original timeseries. The spectra are now placed to the right of their corresponding timeseries. I'm still playing with the axes to make sure I set the best ranges.
  • SUSdrift: I added two new timeseries, DRMI SUS Pitch and DRMI SUS Yaw, to add to the four already-existing timeseries in this tab. These plots represent channels not previously displayed on the summary pages
  • Minor changes
    • Added axis labels on IOO plot 6
    • Changes axis ranges of IOO: MC2 Trans QPD and IOO: IMC REFL RFPD DC
    • Changes axis label on PSL plot 6

 

Results:

So far, all of these changes have been properly implemented into my personal summary page environment. I would like some feedback as to how I can improve the summary pages.

 

 

  11410   Tue Jul 14 13:55:28 2015 jamieUpdateCDSrunning test on daqd, please leave undisturbed

I'm running a test with daqd right now, so please do not disturb for the moment.

I'm temporarily writing frames into a tempfs, which is a filesystem that exists purely in memory.  There should be ZERO IO contention for this filesystem, so if the daqd failures are due to IO then all problems should disappear.  If they don't, then we're dealing with some other problem.

There will be no data saved during this period.

  11409   Tue Jul 14 11:57:27 2015 jamieSummaryCDSCDS upgrade: left running in semi-stable configuration
Quote:

There remains a pattern to some of the restarts, the following times are all reported as restart times. (There are others in between, however.)

daqd: Tue Jul 14 00:02:48 PDT 2015
daqd: Tue Jul 14 01:02:32 PDT 2015
daqd: Tue Jul 14 03:02:33 PDT 2015
daqd: Tue Jul 14 05:02:46 PDT 2015
daqd: Tue Jul 14 06:01:57 PDT 2015
daqd: Tue Jul 14 07:02:19 PDT 2015
daqd: Tue Jul 14 08:02:44 PDT 2015
daqd: Tue Jul 14 09:02:24 PDT 2015
daqd: Tue Jul 14 10:02:03 PDT 2015

Before the upgrade, we suffered from hourly crashes too:

daqd_start Sun Jun 21 00:01:06 PDT 2015
daqd_start Sun Jun 21 01:03:47 PDT 2015
daqd_start Sun Jun 21 02:04:04 PDT 2015
daqd_start Sun Jun 21 03:04:35 PDT 2015
daqd_start Sun Jun 21 04:04:04 PDT 2015
daqd_start Sun Jun 21 05:03:45 PDT 2015
daqd_start Sun Jun 21 06:02:43 PDT 2015
daqd_start Sun Jun 21 07:04:42 PDT 2015
daqd_start Sun Jun 21 08:04:34 PDT 2015
daqd_start Sun Jun 21 09:03:30 PDT 2015
daqd_start Sun Jun 21 10:04:11 PDT 2015

So, this isn't neccesarily new behavior, just something that remains unfixed. 

That's interesting, that we're still seeing those hourly crashes.

We're not writing out the full set of channels, though, and we're getting more failures than just those at the hour, so we're still suffering.

  11408   Tue Jul 14 10:28:02 2015 ericqSummaryCDSCDS upgrade: left running in semi-stable configuration

There remains a pattern to some of the restarts, the following times are all reported as restart times. (There are others in between, however.)

daqd: Tue Jul 14 00:02:48 PDT 2015
daqd: Tue Jul 14 01:02:32 PDT 2015
daqd: Tue Jul 14 03:02:33 PDT 2015
daqd: Tue Jul 14 05:02:46 PDT 2015
daqd: Tue Jul 14 06:01:57 PDT 2015
daqd: Tue Jul 14 07:02:19 PDT 2015
daqd: Tue Jul 14 08:02:44 PDT 2015
daqd: Tue Jul 14 09:02:24 PDT 2015
daqd: Tue Jul 14 10:02:03 PDT 2015

Before the upgrade, we suffered from hourly crashes too:

daqd_start Sun Jun 21 00:01:06 PDT 2015
daqd_start Sun Jun 21 01:03:47 PDT 2015
daqd_start Sun Jun 21 02:04:04 PDT 2015
daqd_start Sun Jun 21 03:04:35 PDT 2015
daqd_start Sun Jun 21 04:04:04 PDT 2015
daqd_start Sun Jun 21 05:03:45 PDT 2015
daqd_start Sun Jun 21 06:02:43 PDT 2015
daqd_start Sun Jun 21 07:04:42 PDT 2015
daqd_start Sun Jun 21 08:04:34 PDT 2015
daqd_start Sun Jun 21 09:03:30 PDT 2015
daqd_start Sun Jun 21 10:04:11 PDT 2015

So, this isn't neccesarily new behavior, just something that remains unfixed. 

  11407   Tue Jul 14 10:23:27 2015 IgnacioUpdateGeneralOptimal detector array placement thoughts

Over the past few days, I've been thinking about how to workout the details conerning Rana's request about a 'map' of the vicinity of the 40m interferometer. This map will take the positions of N randomly placed seismic sensors as well as the signals measured by each one of them and the calculated cross correlations between the sensors and between the sensors and the test mass of interest to give out a displacement vector with new sensor positions that are close to optimum for better seismic (and Newtonian) noise cancellation.

Now, I believe that much of the mathematical details have been already work out by Jenne in her thesis. She explains that the quantity of interest that we wish to minimize in order to find an optimal array is the following,

R = \sqrt{1-\frac{\vec{C}_{SN}^T C_{SS}^{-1}\vec{C}_{SN} }{C_{NN}}}

where  \vec{C}_{SN} is the cross-correlation vector between the seismic detectors and the seismic (or Newtonian) noise, C_{SS} is the cross-correlation matrix between the sensors and C_{NN} is the seismic (or Newtonian) noise variance. 

I looked at the paper that Jenne cited from which she obtained the above quantity and noted that it is a bit different as it contains an extra term inside the square root, it is given by

R' = \sqrt{1-\frac{\vec{C}_{SN}^T (C_{SS}^{-1}+C_{\Sigma\Sigma})\vec{C}_{SN} }{C_{NN}}}

where the new term, C_{\Sigma\Sigma} is the matrix describing the self noise of the sensors. I think Jenne set this term to zero since we can always perform a huddle test on our detectors and know the self noise, thus effectively subtracting it from the signals of interest that we use to calculate the other cross correlation quantities.

Anyways, the quantity R above is a function of the positions of the sensors. In order to apply it to our situation, I'm planning on:

     1) Performing the huddle tests on our sensors, redoing it for the accelerometers and then the seismometers (once the data aquisition system is working... )  

     2) Randomly (well not randomly, there are some assumptions we can make as to what might work best in terms of sensor placement) place the sensors around the interferometer. I'm planning on using all six Wilcoxon 731A accelerometers, the two Guralps and the STS seismometer (any more?).

     3) Measure the ground signals and use wiener filtering in order to cancel out their self noises.

     4) From the measured signals and their present positions we should be able to figure out where to move the sensors in order to optimize subtraction.

i have also been messing around with Jenne's code on seismic field simulations with the hopes of simulating a version of the seismic field around the 40m in order to understand the NN of the site a little better... maybe. While the data aquisition gets back to a working state, I'm planning on using my simulated NN curve as a way to play around with sensor optimization before its done experimentally.

i have as well been thinking and learning a little bit about source characterization through machine learning methods, specially using neural networks as Masha did back in her SURF project on 2012. I have also been looking at Support vector machines. The reasons why I have been looking at machine learning algorithms is because of the nature of the everchanging seismic field around the interferometer. Suppose we find a pretty good sensor array that we like. How do we make sure that this array is any good at some time t after it has been found? If the array mostly deals with the usual seismic background (quiet) of the site of interest, we could incorporate machine learning techniques in order to mitigate any of the more random disturbances that happen around the sites, like delivery trucks, earthquakes, etc.

  11406   Tue Jul 14 09:08:37 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configuration

Overnight daqd restarted itself only about twice an hour, which is an improvement:

controls@fb /opt/rtcds/caltech/c1/target/fb 0$ tail logs/restart.log
daqd: Tue Jul 14 03:13:50 PDT 2015
daqd: Tue Jul 14 04:01:39 PDT 2015
daqd: Tue Jul 14 04:09:57 PDT 2015
daqd: Tue Jul 14 05:02:46 PDT 2015
daqd: Tue Jul 14 06:01:57 PDT 2015
daqd: Tue Jul 14 06:43:18 PDT 2015
daqd: Tue Jul 14 07:02:19 PDT 2015
daqd: Tue Jul 14 07:58:16 PDT 2015
daqd: Tue Jul 14 08:02:44 PDT 2015
daqd: Tue Jul 14 09:02:24 PDT 2015

Un-exporting /frames might have helped a bit.  However, the problem is obviously still not fixed.

  11405   Mon Jul 13 18:27:27 2015 EveConfigurationGeneralHow to set up your own summary page environment on the LDG cluster

I'd like to build off of Koji's instructions with a few useful tips I discovered while setting up my own summary page environment.

To only make a specified selection of tabs for the summary pages, copy only the corresponding .ini files into /home/albert.einstein/summary/config and run the gw_daily_summary_custom following Koji's instructions below. When asked for nodus's password either hit "enter" three times without providing the password or comment out this section of the code to stop the summary page creation process from taking current data files from nodus. This is especially helpful when the 40m is down (like it is now).

After running the summary page code, the pages can be viewed at https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/YYYYMMDD/ and corresponding error logs can be found at ~/public_html/summary/logs/gw_summary_pipe_local-687496-0.err.

  11404   Mon Jul 13 18:12:50 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configuration

I have been watching daqd all day and I don't feel particularly closer to understanding what the issues are.  However, things are

Interestingly, though, the stability appears highly variable at the moment.  This morning, daqd was very unstable and was crashing within a couple of minutes of starting.  However this afternoon, things seemed much more stable.  As of this moment, daqd has been running for for 25 minutes now, writing full frames as well as minute and second trends (no minute_raw), without any issues.  What has changed?

To reiterate, I have been closing watching disk IO to /frames.  I see no indication that there is any disk contention while daqd is failing.  It's still possible, though, that there are disk IO issues affecting daqd at a level that is not readily visible.  From dstat, the frame writes are visible, but nothing else.

I have made one change that could be positively affecting things right now: I un-exported /frames from NFS.  This eliminates anything external from reading /frames over the network.  In particular, it also shuts off the transfer of frames to LDAS.  Since I've done this, daqd has appeared to be more stable.  It's NOT totally stable, though, as the instance that I described above did eventually just die after 43 minutes, as I was writing this.

In any event, as things are currently as stable as I've seen them, I'm leaving it running in this configuration for the moment, with the following relevant daqdrc parameters:

start main 16;
start frame-saver;
sync frame-saver;
start trender 60 60;
start trend-frame-saver;
sync trend-frame-saver;
start minute-trend-frame-saver;
sync minute-trend-frame-saver;
start profiler;
start trend profiler;
  11403   Mon Jul 13 14:08:10 2015 ericqUpdateElectronicsNew RF amps, housed

I made a little box for the new RF amplifiers we'll be using for the green beatnotes, to keep things tidy on the PSL table. They are both Minicircuits model ZHL-3A-S.

I took TFs of their response with the agilient analyzer (calibrating out the cables, splitters, etc.) Powered at +24V, we get a solid ~27dB of gain up to around 200MHz, which is fine for our needs. The phase profile is mostly a 6-7 nsec delay, which is negligible for ALS. Data files are attached. 

Koji looked at me like I was crazy for using a BNC connector for the DC power. I haven't yet been able to find panel mount banana connectors, but when I do, I'll replace it. 


Banana'd:

Attachment 1: ampBox.jpg
ampBox.jpg
Attachment 2: ampTFs.png
ampTFs.png
Attachment 3: ampTFs.zip
Attachment 4: ampBox2.jpg
ampBox2.jpg
  11402   Mon Jul 13 01:11:14 2015 JamieSummaryCDSCDS upgrade: current assessment

daqd is still behaving unstably.  It's still unclear what the issue is.

The current failures look like disk IO contention.  However, it's hard to see any evidince of daqd is suffering from large IO wait while it's failing.

The frame size itself is currently smaller than it was before the upgrade:

controls@fb /frames/full 0$ ls -alth 11190 | head
total 369G
drwxr-xr-x 321 controls controls  36K Jul 12 22:20 ..
drwxr-xr-x   2 controls controls 268K Jun 23 06:06 .
-rw-r--r--   1 controls controls  67M Jun 23 06:06 C-R-1119099984-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:06 C-R-1119099968-16.gwf
-rw-r--r--   1 controls controls  69M Jun 23 06:05 C-R-1119099952-16.gwf
-rw-r--r--   1 controls controls  69M Jun 23 06:05 C-R-1119099936-16.gwf
-rw-r--r--   1 controls controls  67M Jun 23 06:05 C-R-1119099920-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:05 C-R-1119099904-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:04 C-R-1119099888-16.gwf
controls@fb /frames/full 0$ ls -alth 11208 | head
total 17G
drwxr-xr-x   2 controls controls  20K Jul 13 01:00 .
-rw-r--r--   1 controls controls  45M Jul 13 01:00 C-R-1120809632-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 01:00 C-R-1120809408-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 00:56 C-R-1120809392-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 00:56 C-R-1120809376-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 00:56 C-R-1120809360-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 00:55 C-R-1120809344-16.gwf
-rw-r--r--   1 controls controls  50M Jul 13 00:55 C-R-1120809328-16.gwf
controls@fb /frames/full 0$

This would seem to indicate that it's not an increase in frame size that's to blame.

Because slow data is now transported to daqd over the MX data concentrator network rather than via EPICS (RTS 2.8), there is more network on the MX network.   I note also that the channel lists have increased in size:

controls@fb /opt/rtcds/caltech/c1/chans/daq 0$ ls -alt archive/C1LSC* | head -20
-rw-r--r-- 1 4294967294 4294967294 262554 Jul  6 18:21 archive/C1LSC_150706_182146.ini
-rw-r--r-- 1 4294967294 4294967294 262554 Jul  6 18:16 archive/C1LSC_150706_181603.ini
-rw-r--r-- 1 4294967294 4294967294 262554 Jul  6 16:09 archive/C1LSC_150706_160946.ini
-rw-r--r-- 1 4294967294 4294967294  43366 Jul  1 16:05 archive/C1LSC_150701_160519.ini
-rw-r--r-- 1 4294967294 4294967294  43366 Jun 25 15:47 archive/C1LSC_150625_154739.ini
...

I would have thought, though, that data transmission errors would show up in the daqd status bits.

  11401   Fri Jul 10 17:57:38 2015 Max IsiUpdateGeneralSummary pages down

The summary pages are currently unstable due to priority issues on the cluster*. The plots had been empty ever since the CDS updated started anyway. This issue will (presubmably) disappear once the jobs are moved to the new 40m shared LDAS account by the end of next week.

*namely, the jobs are put on hold (rather, status "idle") because we have low priority in the processing queue, making the usual 30min latency impossible.

  11400   Thu Jul 9 16:50:13 2015 JamieSummaryCDSCDS upgrade: if all else fails try throwing metal at the problem

I roped Rolf into coming over and adding his eyes to the problem.  After much discussion we couldn't come up with any reasonable explanation for the problems we've been seeing other than daqd just needing a lot more resources that it did before.  He said he had some old Sun SunFire X4600s from which we could pilfer memory.  I went over to Downs and ripped all the CPU/memory cards out of one of his machines and stuffed them into fb:

fb now has 8 CPU and 16G of RAM

Unfortunately, this is still not enough.  Or at least it didn't solve the problem; daqd is showing the same instabilities, falling over a couple of minutes after I turn on trend frame writing.  As always, before daqd fails it starts spitting out the following to the logs:

[Thu Jul  9 16:37:09 2015] main profiler warning: 0 empty blocks in the buffer

followed by lines like:

[Thu Jul  9 16:37:27 2015] GPS MISS dcu 44 (ASX); dcu_gps=1120520264 gps=1120519812

right before it dies.

I'm no longer convinced that this is a resource issue, though, judging by the resource usage right before the crash:

top - 16:47:32 up 48 min,  5 users,  load average: 0.91, 0.62, 0.61
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.9%us,  0.9%sy,  0.0%ni, 89.1%id,  0.9%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  15952104k total, 13063468k used,  2888636k free,   138648k buffers
Swap:  1023996k total,        0k used,  1023996k free,  7672292k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12016 controls  20   0 8098m 4.4g 104m S  106 29.1   6:45.79 daqd
 4953 controls  20   0 53580 6092 5096 S    0  0.0   0:00.04 nds

Load average less than 1 per CPU, plenty of free memory (~3G free, 0 swap), no waiting for IO (0.9%wa), etc.  daqd is utilizing lots of  threads, which should be spread across many cpus, so even the >100%CPU should be ok.   I'm at a loss...

  11399   Thu Jul 9 16:39:03 2015 KojiConfigurationGeneralHow to set up your own summary page environment on the LDG cluster

Here is the summary of my investigation how to set up my own "summary page" environment on the LDG (LIGO Data Grid) cluster.

Here all albert.einstein must be replaced with your own LIGO.ORG user name.


1. Obtain LDAS cluster account

Run the following from any of the terminal and use your LIGO.ORG credential
ssh albert.einstein@ssh.ligo.org
You will be suggested to visit a particular web site. Fill the form on the web site and wait for the approval e-mails.

2. Use LDG SSH login portal

Once you received the approval of the account, you should be able to log in to the system. Type the following command again from your local terminal
ssh albert.einstein@ssh.ligo.org
You are asked to select the site and machines. Select 2- CIT and b. ldas-pcdev1, c. ldas-pcdev2, or d. ldas-pcdev3.

3. Setup bash environment

Setting up the python library path is very important for the proper processing.

Here is my setup for .bash_profile

# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

if [ -f ~/.profile ]; then
        . ~/.profile
fi

# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH

# So that ssh will work, take care with X logins - see .xsession
[[ -z $SSH_AGENT_PID && -z $DISPLAY ]] &&
  exec -l ssh-agent $SHELL -c "bash --login"

and .bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Set Python environment (based on gpwy-env script)

# clean path environment variable of duplicate entries
cleanpath() {
    if [[ -z "$1" ]]; then
       $1=PATH
    fi
    # map to local variable
    local badpath=$(eval echo \$$1)
    badpath=${badpath%%:}
    # remove duplicates
    badpath="$(echo "${badpath}" | awk -v RS=':' -v ORS=":" '!a[$1]++')"
    # remove trailing colon
    badpath=${badpath%%:}
    # reset variable and export
    eval $1=${badpath}
    eval export $1
}

# set PATH
cleanpath PATH
cleanpath PYTHONPATH

PATH=${HOME}/.local/bin:${PATH}
PYTHONPATH=${HOME}/.local/lib/python2.6/site-packages:${PYTHONPATH}

The order of cleanpath and PYTHONPATH= is important as we want to use the local library installation before anything kicks in.

4. Install required Python libraries

Run the following lines with this order so that they are installed in your "~/local"

# PIP installation
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user
# numpy, scipy, distribute, matplotlib, astropy, importlib installation
pip install numpy --upgrade --user
pip install scipy --upgrade --user
pip install distribute --upgrade --user
pip install matplotlib --user --upgrade
pip install astropy --upgrade --user
pip install importlib --user --upgrade

# We need to use dev branch of gwpy to run gwsumm propery
cp -r /home/detchar/opt/gwpysoft/lib/python2.6/site-packages/gwpy* ~/.local/lib/python2.6/site-packages/

# gwsumm installation
pip install --user git+https://github.com/gwpy/gwsumm

5. Setup summary pages for the 40m

Copy summary page setting from Max's directory.

cp -r ~max.isi/summary ~/

And make temporary directory for the summary pages.

mkdir /usr1/albert.einstein/summary

6. Modify typos in gw_summary_custon

Use your own editor to fix typos

emacs ~/summary/bin/gw_daily_summary_custom

replace max.isi to albert.einstein
change summary40m -> summary


Now the installation is done. From here, the description is for the routine procedure.

7. Run your summary page code

Run Kerberos authentication
kinit albert.einstein@LIGO.ORG

Run a summary page code for a specific date (e.g. for Jul 1st, 2015)
bash ${HOME}/summary/bin/gw_daily_summary_custom --day 20150701

The result can be checked under

https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/
https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/20150701/

Rerun a code for a specific page. This requires the page structure already exists.
The red texts should be modified depending on what ini file you want to run for what day.

/home/albert.einstein/.local/bin/gw_summary day --on-segdb-error warn --verbose  --output-dir . --multi-process 20 --no-html  --ifo C1 --archive C1EVE 20150630  --config-file /mnt/qfs2/albert.einstein/public_html/summary/etc/defaults.ini,/mnt/qfs2/albert.einstein/public_html/summary/etc/c1eve.ini

This command can actually be found in
https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/gw_summary_pipe.sh

8. Some useful command

To check which python library is used
python -c 'import gwpy; print gwpy.__file__'

To list installed python libraries and versions
pip list

This should return the list like the following.

...
astropy (1.0.3)
...
gwpy (0.1b1.dev121)
gwsumm (0.0.0.dev854)
...
matplotlib (1.4.3)
...
numpy (1.9.2)
...
scipy (0.15.1)
...

 

 

  11398   Thu Jul 9 13:26:47 2015 JamieSummaryCDSCDS upgrade: new mx 1.2.16 installed

I rebuilt/installed mx 1.2.16 to use "ether-mode", instead of the default MX-10G:

controls@fb /opt/src/mx-1.2.16 0$ ./configure --enable-ether-mode --prefix=/opt/mx-1.2.16
...
controls@fb /opt/src/mx-1.2.16 0$ make
..
controls@fb /opt/src/mx-1.2.16 0$ make install
...

I then rebuilt/installed daqd so that it properly linked against the updated mx install:

controls@fb /opt/rtcds/rtscore/release/src/daqd 0$ ./configure --enable-debug --disable-broadcast --without-myrinet --with-mx --with epics=/opt/rtapps/epics/base --with-framecpp=/opt/rtapps/framecpp --enable-local-timing
...
controls@fb /opt/rtcds/rtscore/release/src/daqd 0$ make
...
controls@fb /opt/rtcds/rtscore/release/src/daqd 0$ install daqd /opt/rtcds/caltech/c1/target/fb/

It's now back to running and receiving data from the front ends (still not stable yet, though).

  11397   Wed Jul 8 21:02:02 2015 JamieSummaryCDSCDS upgrade: another step forward, so we're back to where we started (plus a bit?)

Koji did a bit of googling to determine that 'Wrong Network' status message could be explained by the fb myrinet  operating in the wrong mode:
(This was the useful link to track down the issue (KA))
 

    Network:    Myrinet 10G

I didn't notice it before, but we should in fact be operating in "Ethernet" mode, since that's the fabric we're using for the DC network.  Digging a bit deeper we found that the new version of mx (1.2.16) had indeed been configured with a different compile option than the 1.2.15 version had:

controls@fb ~ 0$ grep '$ ./configure' /opt/src/mx-1.2.15/config.log          
  $ ./configure --enable-ether-mode --prefix=/opt/mx
controls@fb ~ 0$ grep '$ ./configure' /opt/src/mx-1.2.16/config.log
  $ ./configure --enable-mx-wire --prefix=/opt/mx-1.2.16
controls@fb ~ 0$

So that would entirely explain the problem.  I re-linked mx to the older version (1.2.15), reloaded the mx drivers, and everything showed up correctly:

controls@fb ~ 0$ /opt/mx/bin/mx_info
MX Version: 1.2.12
MX Build: root@fb:/root/mx-1.2.12 Mon Nov  1 13:34:38 PDT 2010
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
===================================================================
Instance #0:  299.8 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:        Running, P0: Link Up
    Network:    Ethernet 10G

    MAC Address:    00:60:dd:46:ea:ec
    Product code:    10G-PCIE-8AL-S
    Part number:    09-03916
    Serial number:    352143
    Mapper:        00:60:dd:46:ea:ec, version = 0x00000000, configured
    Mapped hosts:    6

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:46:ea:ec fb:0                              1,0
   1) 00:25:90:0d:75:bb c1sus:0                           1,0
   2) 00:30:48:be:11:5d c1iscex:0                         1,0
   3) 00:30:48:d6:11:17 c1iscey:0                         1,0
   4) 00:30:48:bf:69:4f c1lsc:0                           1,0
   5) 00:14:4f:40:64:25 c1ioo:0                           1,0
controls@fb ~ 0$

The front end hosts are also showing good omx info (even though they had been previously as well):

controls@c1lsc ~ 0$ /opt/open-mx/bin/omx_info
Open-MX version 1.5.2
 build: controls@fb:/opt/src/open-mx-1.5.2 Tue May 21 11:03:54 PDT 2013

Found 1 boards (32 max) supporting 32 endpoints each:
 c1lsc:0 (board #0 name eth1 addr 00:30:48:bf:69:4f)
   managed by driver 'igb'

Peer table is ready, mapper is 00:30:48:d6:11:17
================================================
  0) 00:30:48:bf:69:4f c1lsc:0
  1) 00:60:dd:46:ea:ec fb:0
  2) 00:25:90:0d:75:bb c1sus:0
  3) 00:30:48:be:11:5d c1iscex:0
  4) 00:30:48:d6:11:17 c1iscey:0
  5) 00:14:4f:40:64:25 c1ioo:0
controls@c1lsc ~ 0$

This got all the mx_stream connections back up and running.

Unfortunately, daqd is back to being a bit flaky.  With all frame writing enabled we saw daqd crash again.  I then shut off all trend frame writing and we're back to a marginally stable state: we have data flowing from all front ends, and full frames are being written, but not trends.

I'll pick up on this again tomorrow, and maybe try to rebuild the new version of mx with the proper flags.

  11396   Wed Jul 8 20:37:02 2015 JamieSummaryCDSCDS upgrade: one step forward, two steps back

After determining yesterday that all the daqd issues were coming from the frame writing, I started to dig into it more today.  I also spoke to Keith Thorne, and got some good suggestions from Gerrit Kuhn at GEO.

I  realized that it probably wasn't the trend writing per se, but that turning on more writing to disk was causing increased load on daqd, and consequently on the system itself.  With more frame writing turned on the memory consuption increased to the point of maxing out the physical RAM.  The system the probably starting swaping, which certainly would have choked daqd.

I noticed that fb only had 4G of RAM, which Keith suggested was just not enough.  Even if the memory consumption of daqd has increased significantly, it still seems like 4G would not be enough.  I opened up fb only to find that fb actually had 8G of RAM installed!  Not sure what happend to the other 4G, but somehow they were not visible to the system.  Koji and I eventually determined, via some frankenstein operations with megatron, that the RAM was just dead.  We then pulled 4G of RAM from megatron and replaced the bad RAM in fb, so that fb now has a full 8G of RAM cool.

Unfortunately, when we got fb fully back up and running we found that fb is not able to see any of the other hosts on the data concentrator network sad.  mx_info, which displays the card and network status for the myricom myrinet fiber card, shows:

MX Version: 1.2.16
MX Build: controls@fb:/opt/src/mx-1.2.16 Tue May 21 10:58:40 PDT 2013
1 Myrinet board installed.
The MX driver is configured to support a maximum of:
    8 endpoints per NIC, 1024 NICs on the network, 32 NICs per host
===================================================================
Instance #0:  299.8 MHz LANai, PCI-E x8, 2 MB SRAM, on NUMA node 0
    Status:        Running, P0: Wrong Network
    Network:    Myrinet 10G

    MAC Address:    00:60:dd:46:ea:ec
    Product code:    10G-PCIE-8AL-S
    Part number:    09-03916
    Serial number:    352143
    Mapper:        00:60:dd:46:ea:ec, version = 0x63e745ee, configured
    Mapped hosts:    1

                                                        ROUTE COUNT
INDEX    MAC ADDRESS     HOST NAME                        P0
-----    -----------     ---------                        ---
   0) 00:60:dd:46:ea:ec fb:0                            D 0,0

Note that all front end machines should be listed in the table at the bottom, and they're not.   Also note the "Wrong Network" note in the Status line above.  It appears that the card has maybe been initialized in a bad state?  Or Koji and I somehow disturbed the network when we were cleaning up things in the rack.  "sudo /etc/init.d/mx restart" on fb doesn't solve the problem.  We even rebooted fb and it didn't seem to help.

In any event, we're back to no data flow.  I'll pick up again tomorrow.

  11395   Wed Jul 8 17:46:20 2015 JessicaSummaryGeneralUpdated Time Delay Plots

I re-measured the transfer function for Cable B, because the residuals in my previous post for cable B indicated a bad fit. 

I also realized I had made a mistake in calculating the time delay, and calculated more reasonable time delays today. 

Cable A had a delay of 202.43 +- 0.01 ns.

Cable B had a delay of 202.44 +- 0.01 ns. 

Attachment 1: resid_CableA.png
resid_CableA.png
Attachment 2: resid_CableB.png
resid_CableB.png
  11394   Tue Jul 7 23:26:19 2015 KojiUpdateCDSAttempt to list CDS issues

As Jamie succeded to realize somewhat workable condition of the 40m CDS, I tried to list the obvious CDS issues so that we can attack them one by one.

  1. c1cal is constantly time-outing now (t>60usec). c1sus is close to it (t=56~57us)
  2. We should check the trends of the CPU meters ("C1:FEC-**_CPU_METER"). In fact this should be listed in the summary pages in a new CDS tab.
  3. Probably this is related to 1): c1lsc is constantly showing IPC error (bit0 = shmem). C1LSC_IPC_STATUS.adl is telling that this is coming from the IPC error between c1lsc and c1cal. ("C1:CAL-LSC_SENSMAT_OSC_****"). This information is found by opning C1LSC_GDS_TP.adl screen and click RT NET STAT button next to the IPC error status.
  4. We wonder how the RFM access is accelerated or decelerated by this upgrade.
  5. We need tests to see if the time delays of the models/IPCs are still reasonable.
  6. LSC Overviw screen has a small digest of the CDS status. Now there are many white boxes that correspond to the channels "C1:FEC-**_DIAG1".
  7. All realtime systems have default (0 or 1) epics channel values (i.e. gains, FM switches, matrices, etc). Need burtrestores.
  8. I tried to burtrestore the models but burtgooey indicated there are some errors.
  9. Detailed check of the snapshot files comparing snapshot files in /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jul/7/19:07 and /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jun/1/19:07 :
    • c1alsepics shows bunch of volatile channels to be snapshot. It seems that all of the static epics channels are missing in the snapshot file. Is this related to the current omission of the slow data acquisition? => No actually this must be the modification of the ALS model to accommodate the ALS in the LSC model for the new ALS setup.
    • c1lscepics was checked indeed slow channels were properly snapshot. So what was the problem in burting???
    • I made a simple csh script to restore the snapshots one by one while collecting the error messages.
      This script is located as /users/koji/150707/burtrevert.sh
    • #!/bin/csh
      echo 'This script restores all of the snapshot files found in' $argv[1] '.'
      echo 'Are you sure? y/n'

      set ans = $<

      set ANS = `echo $ans | tr "[:upper:]" "[:lower:]" `
      if ($ANS == y) then
          foreach fname ($argv[1]/*epics.snap)
          echo ''
          echo '#################################'
          echo $fname
          echo '#################################'

              burtwb -f $fname
          end
      else
          echo "exiting..."
      endif
       
    •  Now I ran the command
      ./burtrevert.sh /opt/rtcds/caltech/c1/burt/autoburt/snapshots/2015/Jun/1/19:07 &>burt.log
      This lists up the missing channels. The zipped log is attached to this entry.
    • Burting old snapshot always crashes the RT process "c1sus" (not the c1sus host). If I use the newly generated snapshot today,
      the process does not crash. The process halts at the cycle time of 74us (>60us). I left the process crashed so that we can take a new snapshot with the matrix numbers filled. Once we have the correct snapshot, we don't need to worry about this crash. Let's see.
    • c1sus still crashes with the new burt file. Theremust be a trigeer that makes the model frozen. We need to split the burtfile into pieces
      to figure out which line causes the halt.
Attachment 1: burt.log.zip
  11393   Tue Jul 7 18:27:54 2015 JamieSummaryCDSCDS upgrade: progress!

After a couple of days of struggle, I made some progress on the CDS upgrade today:

Front end status:

  • RTS upgraded to 2.9.4, and linked in as "release":

/opt/rtcds/rtscore/release -> tags/advLigoRTS-2.9.4

  • mbuf kernel module built installed
  • All front ends have been rebooted with the latest patched kernel (from 2.6 upgrade)
  • All models have been rebuilt, installed, restarted.  Only minor model issues had to be corrected (unterminated unused inputs mostly).
  • awgtpman rebuilt, and installed/running on all front-ends
  • open-mx upgraded to 1.5.2:

/opt/open-mx -> open-mx-1.5.2

  • All front ends running latest version of mx_stream, built against 2.9.4 and open-mx-1.5.2.

We have new GDS overview screens for the front end models:

It's possible that our current lack of IRIG-B GPS distribution means that the 'TIM' status bit will always be red on the IOP models.  Will consult with Rolf.

There are other new features in the front ends that I can get into later.

DAQ (fb) status:

  • daqd and nds rebuilt against 2.9.4, both now running on fb

40m daqd compile flags:

cd src/daqd
./configure --enable-debug --disable-broadcast --without-myrinet --with-mx --enable-local-timing --with-epics=/opt/rtapps/epics/base --with-framecpp=/opt/rtapps/framecpp
make
make clean
install daqd /opt/rtcds/caltech/c1/target/fb/

However, daqd has unfortunately been very unstable, and I've been trying to figure out why.  I originally thought it was some sort of timing issue, but now I'm not so sure.

I had to make the following changes to the daqdrc:

set gps_leaps = 820108813 914803214 1119744016;

That enumerates some list of leap seconds since some time.  Not sure if that actually does anything, but I added the latest leap seconds anyway:

set symm_gps_offset=315964803;

This updates the silly, arbitrary GPS offset, that is required to be correct when not using external GPS reference.

Finally, the last thing I did that finally got it running stably was to turn off all trend frame writing:

# start trender;
# start trend-frame-saver;
# sync trend-frame-saver;
# start minute-trend-frame-saver;
# sync minute-trend-frame-saver;
# start raw_minute_trend_saver;

For whatever reason, it's the trend frame writing that that was causing things daqd to fall over after a short amount of time.  I'll continue investigating tomorrow.

 

We still have a lot of cleanup burt restores, testing, etc. to do, but we're getting there.

  11392   Tue Jul 7 17:22:16 2015 JessicaSummary Time Delay in ALS Cables

I measured the transfer functions in the delay line cables, and then calculated the time delay from that.

The first cable had a time delay of 1272 ns and the second had a time delay of 1264 ns. 

Below are the plots I created to calculate this. There does seem to be a pattern in the residual plots however, which was not expected. 

The R-Square parameter was very close to 1 for both fits, indicating that the fit was good. 

Attachment 1: cableA_fit.jpg
cableA_fit.jpg
Attachment 2: cableA_resid.jpg
cableA_resid.jpg
Attachment 3: cableB_fit.jpg
cableB_fit.jpg
Attachment 4: cableB_resid.jpg
cableB_resid.jpg
  11391   Sun Jul 5 18:14:13 2015 IgnacioUpdatePEMWilcoxon Accelerometer Huddle Test

Updated: On Thursday/Friday (sorry for late elog) I was messing with Eric's Wilcoxon 731A accelerometer huddle test data that was taken without the box and cables being adjusted properly. Anyways, I performed the three cornered hat analysis as he had done but I also performed a six cornered hat method as well instead of permuting around in pairs of three accelerometers. The following plots of the ASD's show the results,

It is interesting to note the improvement at low frequencies when six accelerometers are used instead of six while at higher frequencies we can clearly see how the results are worst than the three hat results.

I decided to take a mean of all six accelerometers measured ground signal as well as that for their computed selfnoises, this is plotted below,

 

Notice the obvious improvement along the entire frequency band of the measurements when all accelerometers are used in the data analysis.

I also performed some Wiener filtering of this data. There was an obvious improvement in the results,

The mean of the signals is also plotted below, just as I did with the cornered hat methods,

 

I also compared the mean self noise of the accelerometers against the manufacturers calculated selfnoise that Rana put up on Github. Both methods are compared against what the manufacturer claims,

As expected the measured noise curves of the Wilcoxon is not as good as what the manufactures stated. This should improve once we redo the huddle test with a better experimental setup. We have some pretty interesting results with the six cornered hat method at around 5 Hz, it is surprisingly accurate given how rough the calculations seemed to be.

I have attached my code for reference: code_accel.zipselfnoise_allsix.png

SEE attachments for better plots of the six accelerometers...

Attachment 5: code_accel.zip
Attachment 6: selfnoise_allsix_miso.png
selfnoise_allsix_miso.png
Attachment 8: selfnoise_allsix.png
selfnoise_allsix.png
  11390   Wed Jul 1 19:16:21 2015 JamieSummaryCDSCDS upgrade in progress

The CDS upgrade is now underway

Here's what's happened so far:

  • Installed and linked in all the RTS supporting software packages in /opt/rtapps (only on front end machines and fb):
    controls@c1lsc ~ 2$ find /opt/rtapps/ -mindepth 1 -maxdepth 1 -type l -ls
    12582916    0 lrwxrwxrwx   1 controls 1001           12 Jul  1 13:16 /opt/rtapps/gds -> gds-2.16.3.2
    12603452    0 lrwxrwxrwx   1 controls 1001           10 Jul  1 13:17 /opt/rtapps/fftw -> fftw-3.3.2
    12603451    0 lrwxrwxrwx   1 controls 1001           15 Jul  1 13:16 /opt/rtapps/libframe -> libframe-8.17.2
    12603450    0 lrwxrwxrwx   1 controls 1001           13 Jul  1 13:16 /opt/rtapps/libmetaio -> libmetaio-8.2
    12582915    0 lrwxrwxrwx   1 controls 1001           34 Jul  1 15:24 /opt/rtapps/framecpp -> ldas-tools-1.19.32-p1/linux-x86_64
    12582914    0 lrwxrwxrwx   1 controls 1001           20 Jul  1 13:15 /opt/rtapps/epics -> epics-3.14.12.2_long
  • Checked out the RTS source for the version we'll be using: 2.9.4

/opt/rtcds/rtscore/tags/advLigoRTS-2.9.4

  • built and installed all of the RTS components:
    • mbuf
    • mx_stream
    • daqd
    • nds
    • awgtpman
       
  • mx_stream is not working. Unknown why. It won't start on the front end machines (only tested on c1lsc so far) with the following error:
    controls@c1lsc ~ 1$ /opt/rtcds/caltech/c1/target/fb/mx_stream -s c1x04 c1lsc c1ass c1oaf c1cal -d fb:0
    mmapped address is 0x7ff7b71a0000
    send len = 263596
    mx_connect failed Remote Endpoint is Closed
    controls@c1lsc ~ 1$
    
    Have contact Keith T. and Rolf B. for backup.  This is a blocker, since this is what ferries the data from the front ends.
     
  • Rebuilt almost all models.  This was good.  Initially nothing would compile because of IPC creation errors, so I moved the old chans/ipc/C1.ipc file out of the way and generated a new one and then everything compiled (of course senders have to be compiled before receivers).
    I only had to fix a couple of things in the models themselves:
    • c1ioo - unterminated FiltCtrl inputs
    • C1_SUS_SINGLE_CONTROL - unterminated FiltCtrl inputs
    • c1oaf - bad part named "STATIC". There is some hacky namespace stuff going on in the RCG. I was able to just explode that part and it now works.
    • c1lsc - unterminated FiltCtrl inputs
    Haven't installed or tried to run anything yet, but the fact they compile is good.
    Some models are not compiling because they have C code in src blocks that are throwing errors:
    • c1lsc
    • c1cal
    It shouldn't be too hard to fix whatever is causing those compile errors.

That's it for today.  Will pick up again first thing tomorrow

  11389   Wed Jul 1 16:16:46 2015 IgnacioUpdateGeneralAccelerometers reinstalled for future huddle test

Today, I installed the Wilcoxon accelerometers in the table near the end of the mode cleaner. I only set three of them up instead of all six. They were set up just as Rana suggeted we should have them properly set up, i.e. cables being tighten up, and a box on top to prevent any airflow introducing any disturbances. We are planning on running the huddle test on these guys once the upgrade? to the interferometer is done.

The cables were tightly clamped to the table as shown below, I used a thick piece of shock absorbing rubber to do this.

 A small piece of thin rubber was used to hold each of the accelerometers tightly to the table in order not to damage them.

We had to borrow Megan's and Kate's piece of black foam in order to seal one of the sides properly, as the cable had to come out through somewhere. We didn't want to mess with drilling any holes into the box! 

There was a small crack even after using the foam. I sealed it up with duck tape.

The box isn't perfect, so there were multiple cracks along the bottom and top of it that could potentially allow for air to flow to the inside. Eric suggested that we should be super careful this time and do it right, so every crack was sealed up with ducktape.

 

Finally, we needed something heavy to be placed on top of the box to hold everything well. We used Rana's baby to accomplish this goal.

Just kidding! Rana's baby is too delicate for our purposes. A layman box of heavy cables was used instead. 

 

  11388   Wed Jul 1 11:45:48 2015 SteveUpdatePEMcleaning around ETMX chamber
Quote:

Keven is our regular janitor is out for a few weeks.

The sub is careful- gentel Mario. We wiped arouind the vertex cambers north side on the floor and east arm racks.

 

Mario wiped the floor around the ETMX chamber today.

  11387   Wed Jul 1 10:01:25 2015 SteveUpdateVACRGA scan pd78 day 275

 

 

Attachment 1: rga275d.png
rga275d.png
  11386   Wed Jul 1 09:33:31 2015 KojiUpdateGeneralShutters closed, watch dogs disabled for the RCG upgrade

I closed the PSL/GREEN shutters and shut off the LSC feedback/SUS watch dogs at 9AM PDT, to allow Jamie to start his disruptive work.
 

  11385   Tue Jun 30 20:26:24 2015 Eve UpdateGeneralMinor Summary Page Changes

I made several small, nit-picky changes to the summary pages.

 

Motivation:

I'm still working on getting used to editing the summary pages. I also wanted to change some of the easy-to-alter cosmetics of the pages.

 

What I did:

I changed axis ranges, axis labels, and typos throughout the summary pages. Read below for an excrutiating list of the minor details of my alterations, if you wish:

  • Changed axes on LSC control signals plots on the Summary tab (but will probably change these back to their original state)
  • Moved an OpLev plot from the Sandbox tab to "Eve" tab
  • Increased the y axis range on IOO MC2 Trans QPD and IMC REFLY RFPD DC plots (which may change when I better incorporate triggers into these plots)
  • Fixed title on IOO Whitened Spectrogram and Rayleigh Spectrogram
  • Fixed degree sign on Weather: Temperature and PSL Table Temperature
  • Fixed percent sign on Weather: Humidity
  •  

Results:

So far, everything looks good. I'll continue to make more changes later this week and hope to soon get on to more substatial changes.

  11384   Tue Jun 30 11:33:00 2015 JamieSummaryCDSprepping for CDS upgrade

This is going to be a big one.  We're at version 2.5 and we're going to go to 2.9.3.

RCG components that need to be updated:

  • mbuf kernel module
  • mx_stream driver
  • iniChk.pl script
  • daqd
  • nds

Supporting software:

  • EPICS 3.14.12.2_long
  • ldas-tools (framecpp) 1.19.32-p1
  • libframe 8.17.2
  • gds 2.16.3.2
  • fftw 3.3.2

Things to watch out for:

  • RTS 2.6:
    • raw minute trend frame location has changed (CRC-based subdirectory)
    • new kernel patch
  • RTS 2.7:
    • supports "commissioning frames", which we will probably not utilize.  need to make sure that we're not writing extra frames somewhere
  • RTS 2.8:
    • "slow" (EPICS) data from the front-end processes is acquired via DAQ network, and not through EPICS.  This will increase traffic on the DAQ lan.  Hopefully this will not be an issue, and the existing network infrastructure can handle it, but it should be monitored.
  11383   Tue Jun 30 05:47:38 2015 ranaUpdateLSCUse BALUNs
Quote:

The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.

BALUNs

  11382   Mon Jun 29 17:40:56 2015 Max IsiUpdateGeneralSummary pages "Code status" page fixed

It was brought to my attention that the "Code status" page (https://nodus.ligo.caltech.edu:30889/detcharsummary/status.html) had been stuck showing "Unknown status" for a while.
This was due to a sync error with LDAS and has now been fixed. Let me know if the issue returns.

  11381   Mon Jun 29 12:28:45 2015 ericqUpdateLSCALS reconstruction in progress

Turns out the reason that the BEATY signal wasn't working is that one of the two RF amplifiers (both of which are model ZHL-32A), isn't amplifying. Voltage at the pins is fine, so maybe its just broken. When the ZHL-3As that Rana ordered arrive, I'll install those. 

Switching the working amplifier between the two channels, and using a Marconi driving -20dBm (the Y green beatnote amplitude), the phase tracker output RMSs are 70Hz and 150Hz for X and Y, respectively, which isn't too exciting. There is enough whitening gain and filtering that I don't think ADC noise is an issue (The magnitude of the phase tracker Q is ~10kcounts after +6dB whitening gain). 

The RMS in both channels mostly comes from a whole mess of 60Hz harmonics. I'll see what I can do by taking better care of the delay line cables, but it is kind of weird that this would be worse now, given that there was little care given to them before either.

Also, for now, so I don't have to lug the marconi around everywhere, I'm currently driving both channels of the demod board with a spare 55MHz LO output of the LSC LO distribution box, which ends up being a factor of 5 smaller phase tracker error signal, but the noise level is about the same as with the marconi. 

  11380   Fri Jun 26 23:18:52 2015 Eve ChaseUpdateCDSSummary Page Updates

Motivation:

My SURF project largely focuses on updating and improving the 40m summary pages. I began to explore and experiment with the already existing summary page code to better understand the required code and eventually lead to tangible improvements of the summary pages.

 

What I did:

I practiced moving from one tab of the summary pages to another. Specifically, I copied the ETM Optical Levers plot from SUS: OpLev to Sandbox, without removing it from its original location. Additionally, I created my own tab to further test the summary pages, titled “Eve”.

KA ed: The configuration files are located at /cvs/cds/caltech/chans/GWsummaries. It is under svn control.

Result:

All changed appeared on the summary pages without much hassle and without any errors.

  11379   Fri Jun 26 03:24:18 2015 ericqUpdateLSCALS reconstruction in progress

Too sleepy to make full ELOG. Stay tuned. 

Two 25dB amplifiers (with fins!) are living in the top shelf on the PSL table, inputs currently grounded. I broke out the fused 24V power from the AOM driver to power the two amps and the AOM driver. I used the POP55 and AS165 heliax cables to get their outputs to the LSC rack, through delay lines, into demod board. 

Driving with -20dBm at 55MHz, the BEATX signal chain has about 60Hz RMS noise, which is about what I measured for driving the old beatbox with a marconi. High frequency noise is a much nicer shape, though. The BEATY signal didn't seem to be getting through, will double check soon. 

Still old delay cables, not nicely shielded or isolated or anything. We'll have to pipe the monitor signal from the LSC rack over to the control room analyzer now. 

 

  11378   Thu Jun 25 18:20:15 2015 ericqUpdateLSCALS reconstruction in progress

I've been working on getting a working ALS up and running. Things are in a bit of a transient state right now; I'm off to softball and dinner, and will resume work tonight. There will be a more detailed ELOG then, but here are some quick notes:

  • c1als has been gutted, phase trackers are working successfully in c1lsc frontend. All channel names remain the same. 
  • BEATX is on the ADC channels where AS165 used to live, BEATY at POP55. 
  • Used a marconi to drive the aLIGO LSC demod board in the LSC rack, was able to lock digital phase tracker on two channels
  • Noise looks pretty cruddy. Lots of 60Hz harmonics on both channels, maybe from marconi drive? Pickup in the delay line?
  • BEATX whitening filter maybe has something fishy going on; excess noise at 2kHz
  • Unclear if BEATY whitening filter is actually doing anything. 
  • Whitening gain switching works fine for both, though. Haven't revisited the switching code, so its controlled in the old RFPD place for now.
  • Whitening triggering is not set up, will require some thought and model work that isn't neccesary yet. 
  • Agilent analyzer, marconi, and old delay lines are currently stashed behind the LSC rack; I will resume work with them tonight. 

The main thing left to do is to install the RF amplifiers at the PSL table and route the green beat signals over to the LSC rack. I fear that some investigation into the whitening filters will be neccesary to make the performance adequate, however. 

  11377   Thu Jun 25 15:07:50 2015 SteveUpdatesafetySurfs Safety 2015

Jessica Pena, Megan Kelly, Eve Chase and Ignacio Magana received 40m specific basic safety traning today.

Attachment 1: surfs2015.jpg
surfs2015.jpg
  11376   Thu Jun 25 14:18:46 2015 Max IsiUpdateGeneralSummary page status

The pages are live again. Please allow some time for the system to catch up and process missed days. If there are any further issues, please let me know.
URL reminder: https://nodus.ligo.caltech.edu:30889/detcharsummary/

Quote:

The summary pages have been down due to incompatibilities with a software update and problems with the LDAS cluster. I'm working at the moment to fix the former and the LDAS admins are looking into the latter. Overall, we can expect the pages will be fully functional again by Monday.

 

  11375   Thu Jun 25 12:03:42 2015 Max IsiUpdateGeneralSummary page status

The summary pages have been down due to incompatibilities with a software update and problems with the LDAS cluster. I'm working at the moment to fix the former and the LDAS admins are looking into the latter. Overall, we can expect the pages will be fully functional again by Monday.

  11374   Wed Jun 24 17:30:45 2015 ericqUpdateLSCDFD Delay length

This afternoon, I had a fruitful conversation with Rich Abbott about various kinds of cables.

I've sent an email to Steve to ask him to buy 2 x 50m LMR-195 cables, with male SMA connectors. Rich highly recommends these for their polyethylene insulation, which makes them less microphonic and less susceptible to thermal expansion, low loss, and multi-ply bonded foil shielding.

50m means that the peak to peak mixer output swing corresponds to about a MHz. 1nV of mixer output noise looks like ~6mHz frequency noise, for a Level 3 mixer appropriately driven. As a comparison, the lowest our in-loop green PDH error signals get is 0.1Hz/rtHz. 

The cable attenuation should be around 4.2dB at 50MHz, and 7.3dB at 150MHz, according to the data sheet. Thus, we should not be in the regieme where we are losing sensitivity to the attenuation. 

By my rough geometric estimation, these two should fit in the 2U box I got ahold of today fine. Jessica is designing the front panel. 

We currently have ~30m of RG-408 and RG-142 as our delay lines. 

  11373   Wed Jun 24 08:05:43 2015 SteveUpdatePSLlaser turned on

The laser went off around 11am yesterday. It was turned on

Attachment 1: laserWasOff.png
laserWasOff.png
  11372   Tue Jun 23 11:18:20 2015 SteveUpdatePEMcleaning around chambers

Keven is our regular janitor is out for a few weeks.

The sub is careful- gentel Mario. We wiped arouind the vertex cambers north side on the floor and east arm racks.

 

  11371   Mon Jun 22 20:59:19 2015 ericqUpdateLSCDFD Delay length

I've been thinking a bit about what the ideal cable length / delay time for the upgraded ALS beatbox should be. Here are some thoughts, but no conclusions yet. 

If you're not running your beatbox mixer in compression, there are two competing effects when you change the cable length. At first, more delay gives better sensitivity, but this does not go on to infinity, because cable attenuation eventually kills your signal. It turns out that the ideal length can be derived to be whatever length gives you 20/ln(10) = -8.7dB of attenuation. Frank found this out in PSL ELOG 825, and I found an HP document that derives this (and other useful DFD math) to the wiki, here.

In PSL ELOG 826, Frank calculated this ideal length for a 160MHz carrier in various kinds of cables. 

However, this is not the end of the story. In the case of the DFD, we actually benefit from operating the mixer in compression, as makes our sensitivity less sensitive to flucuations in the beat amplitude. In this situation, the HP doc states "For maximum sensitivity, more delay can be added until the signal level out of the delay line is 8.7dB below the phase detector (mixer) compression point." I'm not sure I really understand the logic behind this statement, though. 

Lastly, Koji mentioned the fact that the splitter in the demod board does not split at exactly 90 degrees, making the trajectory in the IQ plane an ellipse. This means that if the beat signal is moving around the ellipse a lot, or even wrapping around it, we can suffer from some nonlinear signal conversion. Also, if the raw DFD sensitivity is very high, the free swinging mirrors will cause the signal to swing around faster than the phase tracker can keep up. This should be easy to avoid, however; I doubt we will use so much cable that the beat would move by so much. 

I intend to take all of this into account when picking a cable length! Jessica is going to help us make a nice box for them, too. 

  11370   Mon Jun 22 14:53:37 2015 ranaSummaryLSCX/Y green beat mode overlap measurement redone
  • Why is there a factor of 20 power difference? Some of it is the IR laser power difference, but I thought that was just a factor of 4 in green.
  • Why is the mode overlap only 50% and not more like 75%?
  • IF we have enough PSL green power, we could do the Y-beat with a 80/20 instead of a 50/50 and get better SNR.
  • The FFD-100 response is more like 0.33 A/W at 532 nm, not 0.25 A/W.

In any case, this signal difference is not big, so we should not need a different amplifier chain for the two signals. The 20 dB of amplification in the BeatBox was a fine way, but not great in circuit layout.

The BBPD has an input referred current noise of 10 pA/rHz and a transimpedance of 2 kOhm, so an output voltage noise of 20 nV/rHz (into 50 Ohms). This would be matched by an Amp with NF = 26 dB, which is way worse than anything we could bur from mini-circuits, so we should definitely NOT use anything like the low-noise, low output power amps used currently (e.g. ZFL-1000LN....never, ever use these for anything). We should use a single ZHL-3A-S (G = 25 dB, NF < 6 dB, Max Out = 30 dBm) for each channel (and nothing else) before driving the cables over to the LSC rack into the aLIGO demod board. I just ordered two of these now.

ELOG V3.1.3-