40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
 40m Log, Page 117 of 344 Not logged in
ID Date Author Type Category Subject
11447   Mon Jul 27 16:47:53 2015 ericqUpdateIOOMC2 -> MCL Actuator TF

Our noise cancellation SURFS will be doing online subtraction on the mode cleaner length, among other things.

I made a measurement of the MC2 actuator transfer function by injecting noise from 1-100Hz into LSC_MC2_EXC for about 15 minutes, then estimating the TF from MC2_OUT to IOO_MC_L with CSD/PSD. The inverse of this TF will be applied to their Wiener target data to give us the direct subtration filter we want.

I figured I would post the results here for posterity. The last time this seems to have been done is in ELOG 5900. There are some differences found here, the effective Q of the 1Hz pendulum resonance seems lower, and the behavior above 20Hz has definitely changed.

IIR fits will be done by one of the SURFs to be used in their Wiener filter calculations.

Data attached!

Attachment 1: mc2_2_mcl.png
Attachment 2: MC2_2_MCL_TF.txt.zip
11446   Fri Jul 24 23:08:53 2015 IgnacioUpdatePEMMC1 accelerators moved for future huddle test

I have moved the MC1 accelerators and cable to the huddle test set up, in order to see how a six witness huddle test with the improved set up will do.

Here is a picture of the accelerometer set up,

Our motivation for doing this is to see if more witness signals used in the Wiener filter really does indeed improve subtraction, as it was seen from previous huddle results, specially in the region above 10 Hz.

11445   Fri Jul 24 20:32:15 2015 ericqUpdateElectronicsLSC LO distribution box power button replaced

As happened with the RF distribution box in the IOO rack a while back, the shiny blue power button in the LSC LO distribution box failed today. I replaced it with a simple switch, but since the original was a double throw, the replacement was way too big to fit without major panel surgery. So, instead, I installed it in the grille on the roof of the chassis. It a tight press/snap fit, though; I don't think it is at risk of easily coming loose.

After reinstalling the box, I confirmed that POX POY and AS55 could all lock arms, so I deem the operation a success.

Before:

After:

Attachment 1: 2015-07-24_15.43.56.jpg
Attachment 2: 2015-07-24_16.57.19.jpg
11444   Fri Jul 24 18:12:52 2015 Max IsiUpdateGeneralData missing

For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?

Thursday morning I found the control room emergency exit not locked.

Please check the doors when leaving the lab , specially when you are the last one out.

11442   Thu Jul 23 21:12:14 2015 IgnacioUpdateGeneralNewer accelerometer huddle test data + detrending

In the last post concerning the self noise of the accelerometers, I mentioned the differences between the two data sets I was playing with. In order to give a more concrete analysis of the accelerometers self noise, we came to the conclusion that data taking time should be the same.

The plots below show the analysis for the following two datasets:

Old Data:  6 accelerometers, no cables clamped, no box, 55 mins worth of data.

Newer data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 57.66 mins worth of data, (we had 20 mins of data in the previous data set).

Filter parameters were kept the same in all calculations, the only change that was added to the analysis was the detrending of the signals using the detrend function on Matlab, this improved the results as the plots show. I also plotted the error bars for the Wiener filtered result for reference as well as the manufactures claimed self noise.

After detrending the data and taking a longer dataset we can see the improvement brought upon by the foam box in the low frequency band of 0.5 - 10 Hz, as shown in the bottom left plot. There is a lot of noise that needs to be cancelled out from 10 Hz and on, which brings to our attention the plot on the bottom right corner. This plot uses the old data but uses all six accelerometers as witnesses, it also improved overall after having detrended the data, but what is peculiar about this plot is the fact that it manages to mitigate the higher frequency 10 - ~100 Hz band noise.

Attachment 1: old_data_1.png
Attachment 2: old_data_2.png
Attachment 3: new_data.png
Attachment 4: six_old_data.png
11441   Thu Jul 23 20:57:15 2015 JessicaSummaryGeneralApplying Pre-filter to data before IIR Wiener Filtering

I updated my bandpass filter and have included the bode plot below in Figure 1. It is a fourth order elliptic bandpass filter with a passband ripple of 1dB and a stopband attenuation of 30 dB. It emphasizes the area between 3 and 40 Hz.

Below, I applied this filter to the huddle test data. The results from this were only slightly better in the targeted region than when no pre-filter was applied.

When I pre-filtered the mode cleaner data and then used an IIR wiener filter, I found that the results did not differ much from the data that was not pre-filtered. I'm not sure yet if I'm targeting the right region of this data with my bandpass filter, and will be looking more into choosing a better region. Also, I am only using certain regions of ff when calculating the transfer function, and need to optimize that region also. I uploaded the code I used to make these plots to github.

Attachment 1: BodePlot.png
Attachment 2: acc_bandpass.png
Attachment 3: mcl_seis.png
11440   Thu Jul 23 20:54:42 2015 JessicaUpdateGeneralALS Delay Line Box

The front panels for the ALS delay line box came in last week. Some of the holes for the screws were slightly misaligned, so I filed those and everything is now put together. I just need to test both front panels to determine if the SMAs should be isolated or not.

Koji had also suggested making the holes in the front and back panel conical recesses so that flat head screws could be used and would counteract the anodization of the panel and avoid the SMAs being isolated. I think if we did that then conductivity would be ensured throughout the panel and also through the rest of the box. I also think one way we could test this before drilling conical recesses would be to test both front panels now, as one has isolated SMAs and one has conductive SMAs. If the anodization of the panel isolated the SMA regardless, we could potentially figure this out by testing both panels. But, would it also be that it is possible that the isolation of the SMA itself does not matter and so this test would tell us nothing? Is there a better way to test if the SMAs are being isolated or not? Or would this be more time consuming than just drilling conical recesses as a preventative measure?

11439   Thu Jul 23 15:41:41 2015 SteveUpdatePEMfitting ants with TERRO

Pasadena got 0.2" of  rain on Saturday. Temperatures were high with high humidity since than.The ants were back in the Control room east side benches.

We have started using TERRO Liquid Ant Baits in January 2015   This worked very well to this point.

Tree new packages were opened  yesterday and the ants are gone.

We can conclude that these baits must be replaced after 6 mounts.

The liquid baits contains BORAX  and it is safe.

11438   Thu Jul 23 03:09:05 2015 ericqUpdateLSCaLIGO demod board lives!

I'm a little mystified. Peeking inside the aLIGO demod board, I saw that the reason that two of the channels weren't working was that their power connectors weren't plugged in, so no real mystery there.

I hooked up the board at the electronics bench, and found the noise to be completely well behaved, in contrast to the measurements I made when it was in the LSC rack. I've taken it back out to the LSC rack, and given it the X beatnote, and it seems to be performing pretty well.

I switched between the aLIGO demod board and beatbox during the same lock / beat. The LSC board performs margnially better from 3-100 Hz. The high frequency noise comes from the green PDH loop (coherence is near one above a few hundred Hz), so we don't expect any difference there.

To me, the beatbox noise looks like there is a broad feature that is roughly the same level as the real cavity motion in the 10-100 Hz range. So, I think we should use the aLIGO board afterall, presuming the noise doesn't shoot back up when I remount it in the rack...

The ALS noise is getting low enough where our normal approach of measuring ALS sensing noise by simply taking the PSD of the signal when the arm is PDH locked is not quite valid anymore, as it is sensing the real cavity fluctuations. Doing a frequency domain coherent subtration of the PSDs suggests a sensing noise RMS of ~150Hz for ALSX.

When the X arm is locked on ALS, POX sees about 250Hz RMS out of loop noise, which isn't the greatest; however, I used to be happy with 500Hz. By eye, sweeping through IR resonance is smoother. The real test is to get the Y arm ALS running, and swing it through PRFPMI resonance...

Fair warning, the LSC rack area is not so tidy right now, the demod board is resting on a stool (but not in the way of walking down the arm). I'll clean this up tomorrow.

Attachment 1: beatbox_vs_aLIGO.png
11437   Wed Jul 22 22:06:42 2015 EveSummarySummary PagesFuture summary pages improvements

- CDS Tab

We want to monitor the status of the digital control system.

1st plot
Title: EPICS DAQ Status
I wonder we can plot the binary numbers as statuses of the data acquisition for the realtime codes.
We want to use the status indicators. Like this:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20150722/plots/H1-MULTI_A8CE50_SEGMENTS-1121558417-86400.png

channels:
C1:DAQ-DC0_C1X04_STATUS
C1:DAQ-DC0_C1LSC_STATUS
C1:DAQ-DC0_C1ASS_STATUS
C1:DAQ-DC0_C1OAF_STATUS
C1:DAQ-DC0_C1CAL_STATUS

C1:DAQ-DC0_C1X02_STATUS
C1:DAQ-DC0_C1SUS_STATUS
C1:DAQ-DC0_C1MCS_STATUS
C1:DAQ-DC0_C1RFM_STATUS
C1:DAQ-DC0_C1PEM_STATUS

C1:DAQ-DC0_C1X03_STATUS
C1:DAQ-DC0_C1IOO_STATUS
C1:DAQ-DC0_C1ALS_STATUS

C1:DAQ-DC0_C1X01_STATUS
C1:DAQ-DC0_C1SCX_STATUS
C1:DAQ-DC0_C1ASX_STATUS

C1:DAQ-DC0_C1X05_STATUS
C1:DAQ-DC0_C1SCY_STATUS
C1:DAQ-DC0_C1TST_STATUS

1st plot
Title: IOP Fast Channel DAQ Status
These have two bits each. How can we handle it?
If we need to shrink it to a single bit take "AND" of them.
C1:FEC-40_FB_NET_STATUS (legend: c1x04, if a legend placable)
C1:FEC-20_FB_NET_STATUS (legend: c1x02)
C1:FEC-33_FB_NET_STATUS (legend: c1x03)
C1:FEC-19_FB_NET_STATUS (legend: c1x01)
C1:FEC-46_FB_NET_STATUS (legend: c1x05)

3rd plot
Title C1LSC CPU Meters
channels:
C1:FEC-40_CPU_METER (legend: c1x04)
C1:FEC-42_CPU_METER (legend: c1lsc)
C1:FEC-48_CPU_METER (legend: c1ass)
C1:FEC-22_CPU_METER (legend: c1oaf)
C1:FEC-50_CPU_METER (legend: c1cal)
The range is from 0 to 75 except for c1oaf that could go to 500.
Can we plot c1oaf with the value being devided by 8? (Then the legend should be c1oaf /8)

4th plot
Title C1SUS CPU Meters
channels:
C1:FEC-20_CPU_METER (legend: c1x02)
C1:FEC-21_CPU_METER (legend: c1sus)
C1:FEC-36_CPU_METER (legend: c1mcs)
C1:FEC-38_CPU_METER (legend: c1rfm)
C1:FEC-39_CPU_METER (legend: c1pem)
The range is be from 0 to 75 except for c1pem that could go to 500.
Can we plot c1pem with the value being devided by 8? (Then the legend should be c1pem /8)

5th plot
Title C1IOO CPU Meters
channels:
C1:FEC-33_CPU_METER (legend: c1x03)
C1:FEC-34_CPU_METER (legend: c1ioo)
C1:FEC-28_CPU_METER (legend: c1als)
The range is be from 0 to 75.

6th plot
Title C1ISCEX CPU Meters
channels:
C1:FEC-19_CPU_METER (legend: c1x01)
C1:FEC-45_CPU_METER (legend: c1scx)
C1:FEC-44_CPU_METER (legend: c1asx)
The range is be from 0 to 75.

7th plot
Title C1ISCEY CPU Meters
channels:
C1:FEC-46_CPU_METER (legend: c1x05)
C1:FEC-47_CPU_METER (legend: c1scy)
C1:FEC-91_CPU_METER (legend: c1tst)
The range is be from 0 to 75.

=====================

IOO

We want a duty ratio plot for the IMC. C1:IOO-MC_TRANS_SUM >1e4 is the good period.

Duty ratio plot looks like the right plot of the following link
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20150722/lock/segments/

=====================

SUS: OPLEV

OL_PIT_INMON and OL_YAW_INMON are good for the slow drift monitor.
But their sampling rate is too slow for the PSDs.
Can you use
C1:SUS-ETM_OPLEV_PERROR
C1:SUS-ETM_OPLEV_YERROR
etc...
For the PSDs? They are 2kHz sampling DQ channels. You would be able to plot
it up to ~1kHz. In fact, we want to monitor the PSD from 100mHz to 1kHz.
How can you set up the resolution (=FFT length)?

=====================

LSC / ASC / ALS tabs

Let's make new tabs LSC, ASC, and ALS

LSC:

We should have a plot for
C1:LSC-TRX_OUT_DQ
C1:LSC-TRY_OUT_DQ
C1:LSC-POPDC_OUT_DQ
It's OK to use the minute trend for now.
You can check the range using dataviewer.

ASC:

Let's use
C1:SUS_MC1_ASCPIT_OUT16 (legend: IMC WFS)
C1:ASS-XARM_ITM_YAW_OSC_CLKGAIN (legend: XARM ASS)
C1:ASS-YARM_ITM_YAW_OSC_CLKGAIN (legend: YARM ASS)
C1:ASX-XARM_M1_PIT_OSC_CLKGAIN (legend: XARM Green ASS)
as the status indicators. There is no YARM Green ASS yet.

ALS:

Title: ALS Green transmission
We want a time series of
ALS-TRX_OUT16
ALS-TRY_OUT16

Title: ALS Green beatnote
Another time series
ALS-BEATX_FINE_Q_MON
ALS-BEATY_FINE_Q_MON

Title: Frequency monitor
We have frequency counter outputs, but I have to talk to Eric to know the channel names

11436   Wed Jul 22 15:09:40 2015 ericqUpdateGeneralIMC work

The other night, I spent some time with the mode cleaner.

First, I made some model changes to the MC_TRANS part of c1mcs.mdl. Specifically, I brought in the userapps QPD part that we are using for the transmon and oplev QPDs. My main motivation for doing so was to have FMs for the pitch and yaw values, to be able to set offsets. Up until now, we have used a QPD centering servo in conjunction with the WFS, but the center of the QPD is not perfectly aligned to represent the center of MC2. Using offsets at the servo isn't really practical, since there are integrators.

I then spent some time manually aligning the mode cleaner mirrors with WFS off, followed by centering the in-lock MC REFL beam on the WFS QPDs, and setting the WFS and MC_TRANS offsets. (I updated the WFS offset script, and made one for MC_TRANS in scripts/MC/WFS. They now use averaging instead of servoing to zero, a la LSC PD offset script).

The resultant intracavity power and RIN was an improvement over the older offsets. (RMS RIN went down by half, I think.)

Since Monday, the autolocker seems to be having some trouble. At first, I suspected the changes I made weren't so hot after all, but I've now noticed a pattern. Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh.

11435   Wed Jul 22 14:10:04 2015 ericqUpdateCDSRCG Diagnostics

Now that we've seemed to landed on a working configuration, I've re-ran the tests first described in ELOG 11347. I've also compared the real filtering of filter modules to their designs.

TL;DR: No adverse, or even observable, differences have been witnessed.

As a reminder: In c1tst, there are three loops, called LOOPA, LOOPB, and LOOPC.

• LOOPA is a filter module feeding back onto its own input, with a unit time delay block
• LOOPB is a FM whose output goes to the DAC. In meatspace, the AI output is hooked up directly to an AA chassis input, and back to the FM in CDS
• LOOPC includes RFM connections to c1rfm and back again.

Here are the loop delay results, which measure the slope of the phase response of the OLTF. For the purely digital loops (A, C), we know the number of cycles we expect to compare the delay to.

At this time, I haven't done the adding up of cycles, zero-order-holds, etc. to get the delay we expect from the analog loop (B), so I've just looked at whether it changed at all.

Anyways, I've attached the code that analyzes data from DTT-exported text files containing the continuous phase data from the loop measurements.

Before:

Single Model loop cycles: 1.0000000+/-0.0000006, disparity: -0.00+/-0.25 nsec 2 Model RFM loop cycles: 1.9999999+/-0.0000013, disparity: 0.0+/-0.5 nsec ADC->DAC loop time: 338.2+/-0.4 usec

After:

Single Model loop cycles: 0.9999999+/-0.0000008, disparity: 0.02+/-0.29 nsec 2 Model RFM loop cycles: 2.0000001+/-0.0000011, disparity: -0.0+/-0.4 nsec ADC->DAC loop time: 338.18+/-0.35 usec

So, the digital loops take the number of cycles we expect, and there are no real differences after the upgrade.

Additionally, for all three loops, I created a simple 100:10 filter in foton, and injected broadband noise with awggui, to measure the real TF applied by the FM code. I want to turn this whole process into a single script that will switch the filter on and off, read the foton file, and compare the measured TF to the ideal shape.

In our system, before and after the upgrade, all three loops showed no appreciable difference from the designed filter shape, other than some tiny uptick in phase when approaching the nyquist frequency. This may be due to the fact that I'm comparing to the ideal analog filter, rather than what a 16kHz digital filter looks like.

What I've plotted below is the devitation from the ideal zpk(100Hz, 10Hz, 0.1) frequency response, i.e. Hmeasured / Hideal. The code to do this analysis is also attached, it estimates the TF by dividing the CSD of the filter input and output by the PSD of the input. The single worst coherence in any bin of all the measurements is 0.997, so I didn't really bother to estimate the error of the TF estimate.

Attachment 1: filterShapes.png
Attachment 2: CDS_diag_codes.zip
11434   Tue Jul 21 21:33:22 2015 Max IsiUpdateGeneralSummary pages moved to 40m LDAS account

The summary pages are now generated from the new 40m LDAS account. The nodus URL (https://nodus.ligo.caltech.edu:30889/detcharsummary/) is the same and there are no changes to the way the configuration files work. However, the location on LDAS has changed to https://ldas-jobs.ligo.caltech.edu/~40m/summary/ and the config files are no longer version-controlled on the LDAS side (this was redundant, as they are under VCS in nodus).

I have posted a more detailed description of the summary page workflow, as well as instructions to run your own jobs and other technical minutiae, on the wiki: https://wiki-40m.ligo.caltech.edu/DailySummaryHelp

11433   Tue Jul 21 21:25:18 2015 Max IsiUpdateGeneral40m LDAS account

A shared LIGO Data Grid (LDG) account was created for use by the 40m lab. The purpose of this account is to provide access to the LSC computer cluster resources for 40m-specific projects that may benefit from increased computational power and are not linked to any user in particular (e.g. the summary pages).

For further information, please see https://wiki-40m.ligo.caltech.edu/40mLDASaccount

11432   Tue Jul 21 05:17:09 2015 IgnacioUpdateGeneralMore clear accelerometer huddle tests results

I generated the following plots from the two sets of huddle test data we have for the accelerometers.

Old data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.

New data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 20 mins worth of data.

I made sure to use the same Impuse response time (6 sec) and sampling frequency (256 Hz), as well as every other parameter for the calculations.

Top left: The resultant self noise curve using the new data, there is definitely and improvement in the 0.5-2 Hz band.

Top right: Resultant self noise using the old data, for the first set of three accelerometers.

Bottom left: Old data result for the remaining three accelerometers.

Bottom right: Old data result, using all six accelerometers as witnesses instead.

Attachment 1: new_data.png
Attachment 2: new_data.png
Attachment 3: old_data_1.png
Attachment 4: old_data_1.png
Attachment 5: old_data_2.png
11431   Mon Jul 20 16:45:15 2015 Max IsiConfigurationGeneralSummary page c1sus.ini error corrected

Bad syntax errors in the c1sus.ini config file were causing the summary pages to crash: a plot type had not been indicated for plots 5 and 6, so I've made these "timeseries."
In the future, please remember to always specify a plot type, e.g.:

5 = C1:SUS-ETMX_SUSPIT_INMON.mean,
C1:SUS-ITMY_SUSPIT_INMON.mean,

GOOD:

3 = C1:SUS-ETMX_SUSPIT_INMON.mean,
C1:SUS-ITMY_SUSPIT_INMON.mean timeseries

By the way, the pages will continue to be unavailable while I transfer them to the new shared account.

11430   Mon Jul 20 11:57:17 2015 ericqUpdateGeneralArm Locking recovered

The interferometer is warming up!

I had some issues locking the IMC at first. It turned out that the MC3 side OSEM signal wasn't getting to the ADC. A satellite box sqush fixed it.

I touched up the PMC alignment; the best I could do is 0.75V, probably due to the AOM being in place.

I haven't touched the WFS offsets, but the current ones seem to be doing ok. I'll touch them up tonight when the seismic activity has calmed.

I made some changes to the state of the PZT/PC crossover gain in the mcdown script, resulting in the IMC catching lock quicker.

Thankfully, the tip tilt pointing stayed good during the upgrade. I barely had to touch the ETM alignment to lock the arms. ETMX is showing some errant motion, though...

fb has been loading a 'symmetricom' kernel module, presumably because it was once being used to help with timing.  It's no longer needed, so I unloaded it and commented out the lines that loaded it in /etc/conf.d/local.start.

11428   Sat Jul 18 16:03:00 2015 jamieUpdateCDSEPICS freezes persist

I notice that the periodic EPICS freezes persist.  They last for 5-10 seconds.  MEDM completely freezes up, but then it comes back.

The sites have been noticing similar issues on a less dramatic scale.  Maybe we can learn from whatever they figure out.

11427   Sat Jul 18 15:37:19 2015 JamieSummaryCDSCDS upgrade: current status

So it appears we have found a semi-stable configuration for the DAQ system post upgrade:

Here are the issues:

## daqd

dadq is running mostly stably for the moment, although it still crashes at the top of every hour (see below).  Here are some relevant points of about the current configuration:

• recording data from only a subset of front-ends, to reduce the overall load:
• c1x01
• c1scx
• c1x02
• c1sus
• c1mcs
• c1pem
• c1x04
• c1lsc
• c1ass
• c1x05
• c1scy
• 16 second main buffer:
start main 16;
• trend lengths: second: 600, minute: 60
start trender 600 60;
• writing to frames:
• full
• second
• minute
• (NOT raw minute trends)
• frame compression ON

This elliminates most of the random daqd crashing.  However, daqd still crashes at the top of every hour after writing out the minute trend frame. Still unclear what the issue is, but Keith is investigating.  In some sense this is no worse that where we were before the upgrade, since daqd was also crashing hourly then.  It's still crappy, though, so hopefully we'll figure something out.

The inittab on fb automatically restarts daqd after it crashes, and monit on all of the front ends automatically restarts the mx_stream processes.

## front ends

The front end modules are mostly running fine.

One issue is that the execution times seem to have increased a bit, which is problematic for models that were already on the hairy edge.  For instance, the rough aversage for c1sus has some from ~48us to 50us.  This is most problematic for c1cal, which is now running at ~66us out of 60, which is obviously untenable.  We'll need to reduce the load in c1cal somehow.

All other front end models seem to be working fine, but a full test is still needed.

There was an issue with the DACs on c1sus, but I rebooted and everything came up fine, optics are now damped:

11426   Sat Jul 18 14:55:33 2015 jamieUpdateGeneralall front ends back up and running

After some surgery yesterday the front ends are all back up and running:

• Eric found that one of the DAC cards in the c1sus front end was not being properly initialized (with the new RCG code).  Turned out that it was an older version DAC, with a daughter board on top of a PCIe board.  We suspected that there was some compatibility issue with that version of the card, so Eric pulled an unused card from c1ioo to replace the one in c1sus.  That worked and now c1sus is running happily.
• Eric put the old DAC card into c1ioo, but it didn't like it and was having trouble booting.  I removed the card and c1ioo came up fine on it's own.
• After all front end were back up and running, all RFM connections were dead.  I tracked this down to the RFM switch being off, because the power cable was not fully seated.  This probably happened when Steve was cleaning up the 1X4/5 racks.  I re-powered the RFM switch and all the RFM connections came back on-line
• All receivers of Dolphin (DIS) "PCIE" IPC signals from c1ioo where throwing errors.  I tracked this down to the Dolphin cable going to c1ioo being plugged in to the wrong port on the c1ioo dolphin card.  I unplugged it and plugged it into the correct port, which of course caused all front end modules using dolphin to crash.  Once I restarted all those models, everything is back:

11425   Sat Jul 18 06:12:07 2015 IgnacioUpdateGeneralMCL Wiener filtering + FIR to IIR conversion using vectfit (Update)

(updateAfter Eric gave me feedback on my previous elog post, I went back and fixed some of the silly stuff I stated.

First of all, I have come to realized that it makes zero sense to plot the ASD's of the mode cleaner against the seismometer noise. These measurements are not only quite different, but elementary, they posess different units. I have focused my attention to the MCL being Wiener filtered with the three siesmometer signals.

One of the major improvements that I make in the following analysis is,

1) Prefiltering; a band pass filter from 1 to 5 Hz, in order to emphasize subtraction of the bump shown in the figure below.

2) I have used vectfit exclusively in the 1 to ~5 Hz range, in order to model the FIR filter properly, as in, the kind of subtraction that we care about. Limiting myself to the 1 - 5 Hz range has allowed me to play freeley with the number of poles, hence being able to fit the FIIR filter properly with an IIR rational transfer function properly,

The resulting ASD's are shown below, in blue we show the raw MCL output, in blac the Wiener filter (FIR) result, and finally in black, the resultant data being filtered with the calculated IIR Wiener filter.

Now, in the following plots I show the IIR Wiener filters for each of the three seismometers,

X Seismometer,

For the Y seismometer,

and for the Z seismometer,

The matlab code for this work is attached: code.zip

Attachment 1: Wiener_MCL_seismometers_iir.png
Attachment 2: seisx_mag.png
Attachment 3: seisx_mag.png
Attachment 4: seisx_mag.png
Attachment 5: seisx_ph.png
Attachment 6: seisy_mag.png
Attachment 7: seisy_mag.png
Attachment 8: seisy_mag.png
Attachment 9: seisy_ph.png
Attachment 10: seisz_ph.png
Attachment 11: seisz_ph.png
Attachment 12: code.zip
Attachment 13: seisz_mag.png
Attachment 14: seisz_mag.png
Attachment 15: seisz_ph.png
11424   Fri Jul 17 04:56:37 2015 IgnacioUpdateGeneralMCL Wiener filtering + FIR to IIR conversion using vectfit

We took data for the mode cleaner a while ago, June 30th I believe. This data contained signals from the six accelerometers and the three seismometers. In here I have only focused on the seimometer signals as witnesses in order to construct Wiener filters for each of the three seismometer signals (x,y,z) and for the combined seismometers signal. The following plot showing the ASD's shows the results,

Wiener filtering works beautifully for the seismometers. Note that subtraction is best when we use all three seismometers as the witnesses in the Wiener filter calculation, as can be clearly seen in the first plot above.

Now, I used vectfit to conver the Wiener FIR filters for each seismometer to their IIR versions. The following are the bode plots for the IIR filters,

For the x-direction seismometer,

For the y-direction seismometer

,

And for the z-direction seismometer,

The IIR filters were computed using 5 zeros and 5 poles using vectfit. That was the maximum number of poles that I could use wihtout running into trouble with matrices being almost singular in Matlab. I still need to figure out how to deal with this issue in more detail as fitting the y-seismometer was a bit problematic. I think having a greater number of poles will make the fitting a bit easier.

Attachment 1: Wiener_MCL_seismometers.png
Attachment 2: seisx_mag.png
Attachment 3: seisx_mag.png
Attachment 4: seisx_mag.png
Attachment 5: seisx_phase.png
Attachment 6: seisy_mag.png
Attachment 7: seisy_phase.png
Attachment 8: seisz_mag.png
Attachment 9: seisz_phase.png
11423   Fri Jul 17 02:46:07 2015 IgnacioUpdateGeneralNew huddle test data for Wilcoxon 731A results

On Thursday, new huddle test data for the Wilcoxon 731A was aquired by Eric.

The difference between this new data and the previous data, is:

1) We used three accelerometers instead of six this time around.

2) We used a foam box, and clamped cables on the experimental set up as shown in the previous elog, http://nodus.ligo.caltech.edu:8080/40m/11389

I have analyzed the new data. Here I present my results.

The following plot shows the ASD's for the three accelerometers raw outputs as well as their error signals computed using the three cornered hat method,

As before, I computed the mean for the output signals of the accelerometers above as well as their mean self noise to get the following plot

,

Now, below I compare the new results with the results that I got from the old data,

Did the enclosure and cable clamping do much? Not really, according to the computed three hat results. Also, notice how much better, even if its a small improvement, we get from using six accelerometers and calculating their self noise by the six cornered hat method.

Now, I moved on to analyzing the same data with Wiener Filtering.

Here are again, the raw outputs, and the self noises of each individual accelerometer calculated using Wiener filtering,

The accelerometer in the Y direction is show a kind of funky signal at low frequncies. Why? Anyways, I calculated the mean of the above signals as I did for the three cornered hat method above to get the following, I also show the means of the signals computed with the old data using wiener filtering,

Is the enclosure really doing much? The Wiener filter that I applied to the huddle test old data gave me a much better, by an order of magnitude better self noise curve. Keep in mind that this was using SIX accelerometers, not THREE as we did this time. I want to REDO the huddle test for the WIlcoxon accelerometers using SIX accelerometers with the improved experimental setup to see what I get.

Finally, I compare the computed self noises above with what the manufacturer gives,

,

As I expected, the self noise using six accelerometers and Wiener filtering is the best I could work out. The three cornered hat method works out pretty well from 1 to 10 Hz, but the noise is just too much anywhere higher than 10 Hz. The enclosed, clamped, 3 accelerometer wiener filter result is an order of magnitude worse than the six accelerometer wiener filtered result, and two orders of magnitude worse than the three cornered hat method in the 1 to 10 Hz frequency band.

As I stated, I think we must performed the huddle test with SIX accelerometers and see what kind of results we get.

Attachment 1: selfnoise_allthree_threehat_enclosed.png
Attachment 2: selfnoise_3hat_enclosed_averages.png
Attachment 3: selfnoise_3hat_6hat_enc.png
Attachment 4: miso_wiener_enclosedall.png
Attachment 5: selfnoise_wiener_enclosed.png
Attachment 6: compare_encl.png
11422   Thu Jul 16 16:46:18 2015 ericqUpdateGeneralStarting IFO recovery, DAC troubles

Jamie showed me how to use the SDF system. We created new safe.snap files for all of the running models based on the autoburts from the morning of July 1st, before the upgrade began, and then pruned them of invalid channels.

Now all of the models start up without having to race for the BURT button.

We saw that c1sus was timing out all over the place once the filter settings had been restored. I was thinking I would move one of the vertex optics into c1mcs, but instead I found it easier to remove the global damping parts. Now the c1sus model runs at ~50usec.

The c1sus frontend's DAC is still nonfunctional. Jamie is seeking advice.

11421   Thu Jul 16 16:33:56 2015 JessicaUpdateGeneralAdded Bode Plots of Bandpass Filter

I updated the bandpass filter I was using, finding that having different stopband attenuations before and after the passband better emphasized the area from 3 Hz to 20 Hz. I chose a low passband ripple but high stopband attenuation to do this. My passband ripple was 2 dB, the first stopband was 25 dB, and the second stopband attenuation was 40 dB. As can be seen in the filter Magnitude plot, this resulted in a fairly smooth passband and a fairly step dropoff to the stopband, which will better emphasize the region I am trying to isolate. My goal was to emphasize the 3-20 Hz region 10-30 times more than the outside regions. I think I accomplished this by looking at the Bode plot, but I may have chosen the second stopband attenuation to be slightly too high for this.

Attachment 1: acc1_update.png
Attachment 2: acc2_update.png
Attachment 3: acc3_update.png
Attachment 4: bp_BodeMag.png
Attachment 5: bp_BodePhase.png
11420   Thu Jul 16 11:18:37 2015 jamieUpdateGeneralStarting IFO recovery, DAC troubles
 Quote: I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock.  Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal.  A series of investigations revealed no signals coming out of c1sus's DAC.   The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625): "As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer." The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part: DAC #0 16-bit fifo_status=2 (OK) DAC #1 16-bit fifo_status=3 (empty) DAC #2 16-bit fifo_status=2 (OK) Squishing cables and restarting the frontend have not helped anything.  c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels.

We need to update the indicators on the CDS_FE_STATUS screen to expose the new indicators, so that we have better visibility for these issues.

I'm not sure why this DAC is failing. It may indicate an actual problem with the DAC itself.

 Quote: As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so.

This is related to changes to how the front ends load their safe.snaps. I think they're now explictly expecting the file:

targtet/<model>/<model>epics/burt/safe.snap

I'll come over this afternoon and we can get acquainted with the new SDF system that now handles management of the safe.snap files.

11419   Thu Jul 16 03:01:57 2015 ericqUpdateLSCOld beatbox hooked back up

I was having issues trying to get reasonable noise performance out of the aLIGO demod board as an ALS DFD. Terminating the inputs to the LSC whitening inputs did not show much 60Hz noise, and an RMS in the single Hz range.

A 60Hz line of hundreds of uV was visible in the power spectrum of the single ended BNC and double-ended DB25 outputs of the board no matter how I drove or terminated.

So, I tried out hooking up the ALS beatbox. It turns out to work better for the time being; not only is the 60Hz line in the analog outputs about ten times smaller, the broadband noise floor in the resultant beat spectrum when driven by a 55MHz LO on the LSC rack is a fair bit lower too. I wonder if this is due to not driving the aLIGO board LO at the +10dBm it expects. With the amplifiers and beat note amplitudes we have, we'd only be able to supply around 0 dBm anyways.

Here's a comparison of the aLIGO board (black) and ALS beatbox (dark green) driven with the 55MHz LO, both going through the LSC whitening filters for a resultant magnitude of 3kCounts in the I-Q plane. The RMS sensing noise is about 30 times lower for the beatbox. (Note, this is with the old delay cables. When we switch to the 50m cables, we'll win further frequency noise sensitivity through the better degrees->Hz calibration.) I'm very interested to see what the green beat spectrum looks like with this setup.

Not only is the 60Hz line smaller, there is simply less junk in the beatbox signal. I did not expect this to be the case.

There were some indications of funky status of the aLIGO board: channels 3 and 4 are totally nonfunctioning, so who knows what's going on in there. I've pulled it out, to take a gander if I can figure out how to make it suitiable for our purposes.

Attachment 1: beat_comparison.png
Attachment 2: aLIGO_vs_beatbox.xml.zip
11418   Thu Jul 16 01:04:21 2015 ericqUpdateGeneralStarting IFO recovery, DAC troubles

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock.

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal.

A series of investigations revealed no signals coming out of c1sus's DAC.

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK) DAC #1 16-bit fifo_status=3 (empty) DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything.

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels.

As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so.

11417   Wed Jul 15 18:19:12 2015 JamieSummaryCDSCDS upgrade: tentative stabilty?

Keith Thorne provided his eyes on the situation today and had some suggestions that might have helped things

Reorder ini file list in master file.  Apparently the EDCU.ini file (C0EDCU.ini in our case), which describes EPICS subscriptions to be recorded by the daq, now has to be specified *after* all other front end ini files.  It's unclear why, but it has something to do with RTS 2.8 which changed all slow channels to be transported over the mx network.  This alone did not fix the problem, though.

Increase second trend frame size.  Interestingly, this might have been the key.  The second trend frame size was increased to 600 seconds:

start trender 600 60;

The two numbers are the lengths in seconds for the second and minute trends respectively.  They had been set to "60 60", but Keith suggested that longer second trend frames are better, for whatever reason.  It seems he may be right, given that daqd has been running and writing full and trend frames for 1.5 hours now without issue.

As I'm writing this, though, the daqd just crashed again.  I note, though, that it's right after the hour, and immediately following writing out a one hour minute trend file.  We've been seeing these hour, on the hour, crashes of daqd for quite a while now.  So maybe this is nothing new.  I've actually been wondering if the hourly daqd crashes were associated with writing out the minute trend frames, and I think we might have more evidence to point to that.

If increasing the size of the second trend frames from 60 seconds (35M) to 600 seconds (70M) made a difference in stability, could there be an issue since writing out files that are smaller than some value?  The full frames are 60M, and the minute trends are 35M.

11416   Wed Jul 15 17:05:06 2015 JessicaUpdateGeneralBandpass Pre-Filter created

I applied a bandpass filter to the accelerometer huddle data as a pre-filter. The passband was from 5 Hz to 20 Hz. I found that applying this pre-filter did very little when comparing the PSD after pre-filtering to the PSD with no pre-filtering. There was some improvement though, just not a significant amount. For some reason, it also seemed as though the second accelerometer improved the most from pre-filtering the data, while the first and third remained closer to the unfiltered noise. Also, I have not yet figured out a consistent method for choosing passband ripple and stopband attentuation, both of which determine how good the filter is.

My next step in pre-filtering will be determining a good method for choosing passband ripple and stopband attenuation, along with implementing other pre-filtering methods to combine with the bandpass filter.

Attachment 1: acc1.png
Attachment 2: acc2.png
Attachment 3: acc3.png
11415   Wed Jul 15 13:19:14 2015 JamieSummaryCDSCDS upgrade: reducing mx end-points as last ditch effort

I tried one last thing, suggested by Keith and Gerrit.  I tried reducing the number of mx end-points on fb to zero, which should reduce the total number of fb threads, in the hope that the extra threads were causing the chokes.

On Tue, Jul 14 2015, Keith Thorne <kthorne@ligo-la.caltech.edu> wrote:
> Assumptions
>  1) Before the upgrade (from RCG 2.6?), the DAQ had been working, reading out front-ends, writing frames trends
>  2) In upgrading to RCG 2.9, the mx start-up on the frame builder was modified to use multiple end-points
> (i.e. /etc/init.d/mx has a line like
> # 1 10G card - X2
> MX_MODULE_PARAMS="mx_max_instance=1 mx_max_endpoints=16 $MX_MODULE_PARAMS" > (This can be confirmed by the daqd log file with lines at the top like > 263596 > MX has 16 maximum end-points configured > 2 MX NICs available > [Fri Jul 10 16:12:50 2015] ->4: set thread_stack_size=10240 > [Fri Jul 10 16:12:50 2015] new threads will be created with the stack of size 10 > 240K > > If this is the case, the problem may be that the additional thread on the frame-builder (one per end-point) take up so many slots on the 8-core > frame-builder that they interrupt the frame-writing thread, thus preventing the main buffer from being emptied. > > One could go back to a single end-point. This only helps keep restart of front-end A from hiccuping DAQ for front-end B. > > You would have to remove code on front-ends (/etc/init.d/mx_stream) that chooses endpoints. i.e. > # find line number in rtsystab. Use that to mx_stream slot on card (0-15) > line_num=grep -v ^# /etc/rtsystab | grep --perl-regexp -n "^${hostname}\s" | se
> d 's/^$$[0-9]*$$:.*/\1/g'
> line_off=$(expr$line_num - 1)
> epnum=$(expr$line_off % 2)
> cnum=$(expr$line_off / 2)
>
>     start-stop-daemon --start --quiet -b -m --pidfile /var/log/mx_stream0.pid --exec /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream -- -e 0 -r "$epnum" -W 0 -w 0 -s "$sys" -d x2daqdc0:$cnum -l /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream_logs/$hostname.log

As per Keith's suggestion, I modified the mx startup script to only initialize a single endpoint, and I modified the mx_stream startup to point them all to endpoint 0.  I verified that indeed daqd was a single MX end-point:

MX has 1 maximum end-points configured

It didn't help.  After 5-10 minutes daqd crashes with the same "0 empty blocks" messages.

I should also mention that I'm pretty sure the start of these messages does not seem coincident with any frame writing to disk; further evidence that it's not a disk IO issue.

Keith is looking at the system now, so we if he can see anything obvious.  If not, I will start reverting to 2.5.

11414   Tue Jul 14 17:14:23 2015 EveSummarySummary PagesFuture summary pages improvements

Here is a list of suggested improvements to the summary pages. Let me know if there's something you'd like for me to add to this list!

• A lot of plots are missing axis labels and titles, and I often don't know what to call these labels. I could use some help with this.
• Check the weather and vacuum tabs to make sure that we're getting the expected output. Set the axis labels accordingly.
• Investigate past periods of missing data on DataViewer to see if the problem was with the data requisition process, the summary page production process, or something else.
• Based on trends in data over the past three months, set axis ranges accordingly to encapsulate the full data range.
• Create a CDS tab to store statistics of our digital systems. We will use the CDS signals to determine when the digital system is running and when the minute trend is missing. This will allow us to exclude irrelevant parts of the data.
• Provide duty ratio statistics for the IMC.
• Set triggers for certain plots. For example, for channels C1:LSC-XARM OUT DQ and page 4 LIGO-T1500123–v1 C1:LSC-YARM OUT DQ to be plotted in the Arm LSC Control signals figures, C1:LSCTRX OUT DQ and C1:LSC-TRY OUT DQ must be higher than 0.5, thus acting as triggers.
• Include some flag or other marking indicating when data is not being represented at a certain time for specific plots.
• Maybe include some cool features like interactive plots.
11413   Tue Jul 14 17:06:00 2015 jamieUpdateCDSrunning test on daqd, please leave undisturbed

I have reverted daqd to the previous configuration, so that it's writing frames to disk.  It's still showing instability.

11412   Tue Jul 14 16:51:01 2015 JamieSummaryCDSCDS upgrade: problem is not disk access

I think I have now determined once and for all that the daqd problems are NOT due to disk IO contention.

I have mounted a tmpfs at /frames/tmp and have told daqd to write frames there.  The tmpfs exists entirely in RAM.  There is essentially zero IO wait for such a filesystem, so daqd should never have trouble writing out the frames.

But yet daqd continues to fail with the "0 empty blocks in the buffer" warnings.  I've been down a rabbit hole.

I've continued to make changes to the summary pages on my own environment, which I plan on implementing on the main summary pages when they are back online.

Motivation:

I created my own summary page environment and manipulated data from June 30 to make additional plots and change already existing plots. The main summary pages (https://nodus.ligo.caltech.edu:30889/detcharsummary/ or https://ldas-jobs.ligo.caltech.edu/~max.isi/summary/) are currently down due to the CDS upgrade, so my own summary page environment acts as a temporary playground to continue working on my SURF project. My summary pages can be found here (https://ldas-jobs.ligo.caltech.edu/~eve.chase/summary/day/20150630/); they contian identical plots to the main summary pages, except for the Summary tab. I'm open to suggestions, so I can make the summary pages as useful as possible.

What I did:

• SUS OpLev: For every already existing optical lever timeseries, I created a corresponding spectrum, showing all channels present in the original timeseries. The spectra are now placed to the right of their corresponding timeseries. I'm still playing with the axes to make sure I set the best ranges.
• SUSdrift: I added two new timeseries, DRMI SUS Pitch and DRMI SUS Yaw, to add to the four already-existing timeseries in this tab. These plots represent channels not previously displayed on the summary pages
• Minor changes
• Added axis labels on IOO plot 6
• Changes axis ranges of IOO: MC2 Trans QPD and IOO: IMC REFL RFPD DC
• Changes axis label on PSL plot 6

Results:

So far, all of these changes have been properly implemented into my personal summary page environment. I would like some feedback as to how I can improve the summary pages.

11410   Tue Jul 14 13:55:28 2015 jamieUpdateCDSrunning test on daqd, please leave undisturbed

## I'm running a test with daqd right now, so please do not disturb for the moment.

I'm temporarily writing frames into a tempfs, which is a filesystem that exists purely in memory.  There should be ZERO IO contention for this filesystem, so if the daqd failures are due to IO then all problems should disappear.  If they don't, then we're dealing with some other problem.

There will be no data saved during this period.

11409   Tue Jul 14 11:57:27 2015 jamieSummaryCDSCDS upgrade: left running in semi-stable configuration
 Quote: There remains a pattern to some of the restarts, the following times are all reported as restart times. (There are others in between, however.) daqd: Tue Jul 14 00:02:48 PDT 2015 daqd: Tue Jul 14 01:02:32 PDT 2015 daqd: Tue Jul 14 03:02:33 PDT 2015 daqd: Tue Jul 14 05:02:46 PDT 2015 daqd: Tue Jul 14 06:01:57 PDT 2015 daqd: Tue Jul 14 07:02:19 PDT 2015 daqd: Tue Jul 14 08:02:44 PDT 2015 daqd: Tue Jul 14 09:02:24 PDT 2015 daqd: Tue Jul 14 10:02:03 PDT 2015 Before the upgrade, we suffered from hourly crashes too: daqd_start Sun Jun 21 00:01:06 PDT 2015 daqd_start Sun Jun 21 01:03:47 PDT 2015 daqd_start Sun Jun 21 02:04:04 PDT 2015 daqd_start Sun Jun 21 03:04:35 PDT 2015 daqd_start Sun Jun 21 04:04:04 PDT 2015 daqd_start Sun Jun 21 05:03:45 PDT 2015 daqd_start Sun Jun 21 06:02:43 PDT 2015 daqd_start Sun Jun 21 07:04:42 PDT 2015 daqd_start Sun Jun 21 08:04:34 PDT 2015 daqd_start Sun Jun 21 09:03:30 PDT 2015 daqd_start Sun Jun 21 10:04:11 PDT 2015 So, this isn't neccesarily new behavior, just something that remains unfixed.

That's interesting, that we're still seeing those hourly crashes.

We're not writing out the full set of channels, though, and we're getting more failures than just those at the hour, so we're still suffering.

11408   Tue Jul 14 10:28:02 2015 ericqSummaryCDSCDS upgrade: left running in semi-stable configuration

There remains a pattern to some of the restarts, the following times are all reported as restart times. (There are others in between, however.)

daqd: Tue Jul 14 00:02:48 PDT 2015 daqd: Tue Jul 14 01:02:32 PDT 2015 daqd: Tue Jul 14 03:02:33 PDT 2015 daqd: Tue Jul 14 05:02:46 PDT 2015 daqd: Tue Jul 14 06:01:57 PDT 2015 daqd: Tue Jul 14 07:02:19 PDT 2015 daqd: Tue Jul 14 08:02:44 PDT 2015 daqd: Tue Jul 14 09:02:24 PDT 2015 daqd: Tue Jul 14 10:02:03 PDT 2015

Before the upgrade, we suffered from hourly crashes too:

daqd_start Sun Jun 21 00:01:06 PDT 2015 daqd_start Sun Jun 21 01:03:47 PDT 2015 daqd_start Sun Jun 21 02:04:04 PDT 2015 daqd_start Sun Jun 21 03:04:35 PDT 2015 daqd_start Sun Jun 21 04:04:04 PDT 2015 daqd_start Sun Jun 21 05:03:45 PDT 2015 daqd_start Sun Jun 21 06:02:43 PDT 2015 daqd_start Sun Jun 21 07:04:42 PDT 2015 daqd_start Sun Jun 21 08:04:34 PDT 2015 daqd_start Sun Jun 21 09:03:30 PDT 2015 daqd_start Sun Jun 21 10:04:11 PDT 2015

So, this isn't neccesarily new behavior, just something that remains unfixed.

11407   Tue Jul 14 10:23:27 2015 IgnacioUpdateGeneralOptimal detector array placement thoughts

Over the past few days, I've been thinking about how to workout the details conerning Rana's request about a 'map' of the vicinity of the 40m interferometer. This map will take the positions of N randomly placed seismic sensors as well as the signals measured by each one of them and the calculated cross correlations between the sensors and between the sensors and the test mass of interest to give out a displacement vector with new sensor positions that are close to optimum for better seismic (and Newtonian) noise cancellation.

Now, I believe that much of the mathematical details have been already work out by Jenne in her thesis. She explains that the quantity of interest that we wish to minimize in order to find an optimal array is the following,

$R = \sqrt{1-\frac{\vec{C}_{SN}^T C_{SS}^{-1}\vec{C}_{SN} }{C_{NN}}}$

where  $\vec{C}_{SN}$ is the cross-correlation vector between the seismic detectors and the seismic (or Newtonian) noise, $C_{SS}$ is the cross-correlation matrix between the sensors and $C_{NN}$ is the seismic (or Newtonian) noise variance.

I looked at the paper that Jenne cited from which she obtained the above quantity and noted that it is a bit different as it contains an extra term inside the square root, it is given by

$R' = \sqrt{1-\frac{\vec{C}_{SN}^T (C_{SS}^{-1}+C_{\Sigma\Sigma})\vec{C}_{SN} }{C_{NN}}}$

where the new term, $C_{\Sigma\Sigma}$ is the matrix describing the self noise of the sensors. I think Jenne set this term to zero since we can always perform a huddle test on our detectors and know the self noise, thus effectively subtracting it from the signals of interest that we use to calculate the other cross correlation quantities.

Anyways, the quantity $R$ above is a function of the positions of the sensors. In order to apply it to our situation, I'm planning on:

1) Performing the huddle tests on our sensors, redoing it for the accelerometers and then the seismometers (once the data aquisition system is working... )

2) Randomly (well not randomly, there are some assumptions we can make as to what might work best in terms of sensor placement) place the sensors around the interferometer. I'm planning on using all six Wilcoxon 731A accelerometers, the two Guralps and the STS seismometer (any more?).

3) Measure the ground signals and use wiener filtering in order to cancel out their self noises.

4) From the measured signals and their present positions we should be able to figure out where to move the sensors in order to optimize subtraction.

i have also been messing around with Jenne's code on seismic field simulations with the hopes of simulating a version of the seismic field around the 40m in order to understand the NN of the site a little better... maybe. While the data aquisition gets back to a working state, I'm planning on using my simulated NN curve as a way to play around with sensor optimization before its done experimentally.

i have as well been thinking and learning a little bit about source characterization through machine learning methods, specially using neural networks as Masha did back in her SURF project on 2012. I have also been looking at Support vector machines. The reasons why I have been looking at machine learning algorithms is because of the nature of the everchanging seismic field around the interferometer. Suppose we find a pretty good sensor array that we like. How do we make sure that this array is any good at some time t after it has been found? If the array mostly deals with the usual seismic background (quiet) of the site of interest, we could incorporate machine learning techniques in order to mitigate any of the more random disturbances that happen around the sites, like delivery trucks, earthquakes, etc.

11406   Tue Jul 14 09:08:37 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configuration

Overnight daqd restarted itself only about twice an hour, which is an improvement:

controls@fb /opt/rtcds/caltech/c1/target/fb 0$tail logs/restart.log daqd: Tue Jul 14 03:13:50 PDT 2015 daqd: Tue Jul 14 04:01:39 PDT 2015 daqd: Tue Jul 14 04:09:57 PDT 2015 daqd: Tue Jul 14 05:02:46 PDT 2015 daqd: Tue Jul 14 06:01:57 PDT 2015 daqd: Tue Jul 14 06:43:18 PDT 2015 daqd: Tue Jul 14 07:02:19 PDT 2015 daqd: Tue Jul 14 07:58:16 PDT 2015 daqd: Tue Jul 14 08:02:44 PDT 2015 daqd: Tue Jul 14 09:02:24 PDT 2015 Un-exporting /frames might have helped a bit. However, the problem is obviously still not fixed. 11405 Mon Jul 13 18:27:27 2015 EveConfigurationGeneralHow to set up your own summary page environment on the LDG cluster I'd like to build off of Koji's instructions with a few useful tips I discovered while setting up my own summary page environment. To only make a specified selection of tabs for the summary pages, copy only the corresponding .ini files into /home/albert.einstein/summary/config and run the gw_daily_summary_custom following Koji's instructions below. When asked for nodus's password either hit "enter" three times without providing the password or comment out this section of the code to stop the summary page creation process from taking current data files from nodus. This is especially helpful when the 40m is down (like it is now). After running the summary page code, the pages can be viewed at https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/YYYYMMDD/ and corresponding error logs can be found at ~/public_html/summary/logs/gw_summary_pipe_local-687496-0.err. 11404 Mon Jul 13 18:12:50 2015 JamieSummaryCDSCDS upgrade: left running in semi-stable configuration I have been watching daqd all day and I don't feel particularly closer to understanding what the issues are. However, things are Interestingly, though, the stability appears highly variable at the moment. This morning, daqd was very unstable and was crashing within a couple of minutes of starting. However this afternoon, things seemed much more stable. As of this moment, daqd has been running for for 25 minutes now, writing full frames as well as minute and second trends (no minute_raw), without any issues. What has changed? To reiterate, I have been closing watching disk IO to /frames. I see no indication that there is any disk contention while daqd is failing. It's still possible, though, that there are disk IO issues affecting daqd at a level that is not readily visible. From dstat, the frame writes are visible, but nothing else. I have made one change that could be positively affecting things right now: I un-exported /frames from NFS. This eliminates anything external from reading /frames over the network. In particular, it also shuts off the transfer of frames to LDAS. Since I've done this, daqd has appeared to be more stable. It's NOT totally stable, though, as the instance that I described above did eventually just die after 43 minutes, as I was writing this. In any event, as things are currently as stable as I've seen them, I'm leaving it running in this configuration for the moment, with the following relevant daqdrc parameters: start main 16; start frame-saver; sync frame-saver; start trender 60 60; start trend-frame-saver; sync trend-frame-saver; start minute-trend-frame-saver; sync minute-trend-frame-saver; start profiler; start trend profiler; 11403 Mon Jul 13 14:08:10 2015 ericqUpdateElectronicsNew RF amps, housed I made a little box for the new RF amplifiers we'll be using for the green beatnotes, to keep things tidy on the PSL table. They are both Minicircuits model ZHL-3A-S. I took TFs of their response with the agilient analyzer (calibrating out the cables, splitters, etc.) Powered at +24V, we get a solid ~27dB of gain up to around 200MHz, which is fine for our needs. The phase profile is mostly a 6-7 nsec delay, which is negligible for ALS. Data files are attached. Koji looked at me like I was crazy for using a BNC connector for the DC power. I haven't yet been able to find panel mount banana connectors, but when I do, I'll replace it. Banana'd: Attachment 1: ampBox.jpg Attachment 2: ampTFs.png Attachment 3: ampTFs.zip Attachment 4: ampBox2.jpg 11402 Mon Jul 13 01:11:14 2015 JamieSummaryCDSCDS upgrade: current assessment daqd is still behaving unstably. It's still unclear what the issue is. The current failures look like disk IO contention. However, it's hard to see any evidince of daqd is suffering from large IO wait while it's failing. The frame size itself is currently smaller than it was before the upgrade: controls@fb /frames/full 0$ ls -alth 11190 | head
total 369G
drwxr-xr-x 321 controls controls  36K Jul 12 22:20 ..
drwxr-xr-x   2 controls controls 268K Jun 23 06:06 .
-rw-r--r--   1 controls controls  67M Jun 23 06:06 C-R-1119099984-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:06 C-R-1119099968-16.gwf
-rw-r--r--   1 controls controls  69M Jun 23 06:05 C-R-1119099952-16.gwf
-rw-r--r--   1 controls controls  69M Jun 23 06:05 C-R-1119099936-16.gwf
-rw-r--r--   1 controls controls  67M Jun 23 06:05 C-R-1119099920-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:05 C-R-1119099904-16.gwf
-rw-r--r--   1 controls controls  68M Jun 23 06:04 C-R-1119099888-16.gwf
controls@fb /frames/full 0$ls -alth 11208 | head total 17G drwxr-xr-x 2 controls controls 20K Jul 13 01:00 . -rw-r--r-- 1 controls controls 45M Jul 13 01:00 C-R-1120809632-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 01:00 C-R-1120809408-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 00:56 C-R-1120809392-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 00:56 C-R-1120809376-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 00:56 C-R-1120809360-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 00:55 C-R-1120809344-16.gwf -rw-r--r-- 1 controls controls 50M Jul 13 00:55 C-R-1120809328-16.gwf controls@fb /frames/full 0$

This would seem to indicate that it's not an increase in frame size that's to blame.

Because slow data is now transported to daqd over the MX data concentrator network rather than via EPICS (RTS 2.8), there is more network on the MX network.   I note also that the channel lists have increased in size:

controls@fb /opt/rtcds/caltech/c1/chans/daq 0$ls -alt archive/C1LSC* | head -20 -rw-r--r-- 1 4294967294 4294967294 262554 Jul 6 18:21 archive/C1LSC_150706_182146.ini -rw-r--r-- 1 4294967294 4294967294 262554 Jul 6 18:16 archive/C1LSC_150706_181603.ini -rw-r--r-- 1 4294967294 4294967294 262554 Jul 6 16:09 archive/C1LSC_150706_160946.ini -rw-r--r-- 1 4294967294 4294967294 43366 Jul 1 16:05 archive/C1LSC_150701_160519.ini -rw-r--r-- 1 4294967294 4294967294 43366 Jun 25 15:47 archive/C1LSC_150625_154739.ini ... I would have thought, though, that data transmission errors would show up in the daqd status bits. 11401 Fri Jul 10 17:57:38 2015 Max IsiUpdateGeneralSummary pages down The summary pages are currently unstable due to priority issues on the cluster*. The plots had been empty ever since the CDS updated started anyway. This issue will (presubmably) disappear once the jobs are moved to the new 40m shared LDAS account by the end of next week. *namely, the jobs are put on hold (rather, status "idle") because we have low priority in the processing queue, making the usual 30min latency impossible. 11400 Thu Jul 9 16:50:13 2015 JamieSummaryCDSCDS upgrade: if all else fails try throwing metal at the problem I roped Rolf into coming over and adding his eyes to the problem. After much discussion we couldn't come up with any reasonable explanation for the problems we've been seeing other than daqd just needing a lot more resources that it did before. He said he had some old Sun SunFire X4600s from which we could pilfer memory. I went over to Downs and ripped all the CPU/memory cards out of one of his machines and stuffed them into fb: fb now has 8 CPU and 16G of RAM Unfortunately, this is still not enough. Or at least it didn't solve the problem; daqd is showing the same instabilities, falling over a couple of minutes after I turn on trend frame writing. As always, before daqd fails it starts spitting out the following to the logs: [Thu Jul 9 16:37:09 2015] main profiler warning: 0 empty blocks in the buffer followed by lines like: [Thu Jul 9 16:37:27 2015] GPS MISS dcu 44 (ASX); dcu_gps=1120520264 gps=1120519812 right before it dies. I'm no longer convinced that this is a resource issue, though, judging by the resource usage right before the crash: top - 16:47:32 up 48 min, 5 users, load average: 0.91, 0.62, 0.61 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie Cpu(s): 8.9%us, 0.9%sy, 0.0%ni, 89.1%id, 0.9%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 15952104k total, 13063468k used, 2888636k free, 138648k buffers Swap: 1023996k total, 0k used, 1023996k free, 7672292k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12016 controls 20 0 8098m 4.4g 104m S 106 29.1 6:45.79 daqd 4953 controls 20 0 53580 6092 5096 S 0 0.0 0:00.04 nds Load average less than 1 per CPU, plenty of free memory (~3G free, 0 swap), no waiting for IO (0.9%wa), etc. daqd is utilizing lots of threads, which should be spread across many cpus, so even the >100%CPU should be ok. I'm at a loss... 11399 Thu Jul 9 16:39:03 2015 KojiConfigurationGeneralHow to set up your own summary page environment on the LDG cluster Here is the summary of my investigation how to set up my own "summary page" environment on the LDG (LIGO Data Grid) cluster. Here all albert.einstein must be replaced with your own LIGO.ORG user name. 1. Obtain LDAS cluster account Run the following from any of the terminal and use your LIGO.ORG credential ssh albert.einstein@ssh.ligo.org You will be suggested to visit a particular web site. Fill the form on the web site and wait for the approval e-mails. 2. Use LDG SSH login portal Once you received the approval of the account, you should be able to log in to the system. Type the following command again from your local terminal ssh albert.einstein@ssh.ligo.org You are asked to select the site and machines. Select 2- CIT and b. ldas-pcdev1, c. ldas-pcdev2, or d. ldas-pcdev3. 3. Setup bash environment Setting up the python library path is very important for the proper processing. Here is my setup for .bash_profile  # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi if [ -f ~/.profile ]; then . ~/.profile fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH # So that ssh will work, take care with X logins - see .xsession [[ -z$SSH_AGENT_PID && -z $DISPLAY ]] && exec -l ssh-agent$SHELL -c "bash --login"

and .bashrc

 # .bashrc # Source global definitions if [ -f /etc/bashrc ]; then     . /etc/bashrc fi # Set Python environment (based on gpwy-env script) # clean path environment variable of duplicate entries cleanpath() {     if [[ -z "$1" ]]; then$1=PATH     fi     # map to local variable     local badpath=$(eval echo \$$1) badpath=${badpath%%:}     # remove duplicates     badpath="$(echo "${badpath}" | awk -v RS=':' -v ORS=":" '!a[$1]++')" # remove trailing colon badpath=${badpath%%:}     # reset variable and export     eval $1=${badpath}     eval export $1 } # set PATH cleanpath PATH cleanpath PYTHONPATH PATH=${HOME}/.local/bin:${PATH} PYTHONPATH=${HOME}/.local/lib/python2.6/site-packages:${PYTHONPATH} The order of cleanpath and PYTHONPATH= is important as we want to use the local library installation before anything kicks in. 4. Install required Python libraries Run the following lines with this order so that they are installed in your "~/local"  # PIP installation wget https://bootstrap.pypa.io/get-pip.py python get-pip.py --user # numpy, scipy, distribute, matplotlib, astropy, importlib installation pip install numpy --upgrade --user pip install scipy --upgrade --user pip install distribute --upgrade --user pip install matplotlib --user --upgrade pip install astropy --upgrade --user pip install importlib --user --upgrade # We need to use dev branch of gwpy to run gwsumm propery cp -r /home/detchar/opt/gwpysoft/lib/python2.6/site-packages/gwpy* ~/.local/lib/python2.6/site-packages/ # gwsumm installation pip install --user git+https://github.com/gwpy/gwsumm 5. Setup summary pages for the 40m Copy summary page setting from Max's directory. cp -r ~max.isi/summary ~/ And make temporary directory for the summary pages. mkdir /usr1/albert.einstein/summary 6. Modify typos in gw_summary_custon Use your own editor to fix typos emacs ~/summary/bin/gw_daily_summary_custom replace max.isi to albert.einstein change summary40m -> summary Now the installation is done. From here, the description is for the routine procedure. 7. Run your summary page code Run Kerberos authentication kinit albert.einstein@LIGO.ORG Run a summary page code for a specific date (e.g. for Jul 1st, 2015) bash${HOME}/summary/bin/gw_daily_summary_custom --day 20150701

The result can be checked under

https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/ https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/day/20150701/

Rerun a code for a specific page. This requires the page structure already exists.
The red texts should be modified depending on what ini file you want to run for what day.

/home/albert.einstein/.local/bin/gw_summary day --on-segdb-error warn --verbose  --output-dir . --multi-process 20 --no-html  --ifo C1 --archive C1EVE 20150630  --config-file /mnt/qfs2/albert.einstein/public_html/summary/etc/defaults.ini,/mnt/qfs2/albert.einstein/public_html/summary/etc/c1eve.ini

This command can actually be found in
https://ldas-jobs.ligo.caltech.edu/~albert.einstein/summary/gw_summary_pipe.sh

8. Some useful command

To check which python library is used
python -c 'import gwpy; print gwpy.__file__'

To list installed python libraries and versions
pip list

This should return the list like the following.

... astropy (1.0.3) ... gwpy (0.1b1.dev121) gwsumm (0.0.0.dev854) ... matplotlib (1.4.3) ... numpy (1.9.2) ... scipy (0.15.1) ...

11398   Thu Jul 9 13:26:47 2015 JamieSummaryCDSCDS upgrade: new mx 1.2.16 installed

I rebuilt/installed mx 1.2.16 to use "ether-mode", instead of the default MX-10G:

controls@fb /opt/src/mx-1.2.16 0$./configure --enable-ether-mode --prefix=/opt/mx-1.2.16 ... controls@fb /opt/src/mx-1.2.16 0$ make
..
controls@fb /opt/src/mx-1.2.16 0$make install ... I then rebuilt/installed daqd so that it properly linked against the updated mx install: controls@fb /opt/rtcds/rtscore/release/src/daqd 0$ ./configure --enable-debug --disable-broadcast --without-myrinet --with-mx --with epics=/opt/rtapps/epics/base --with-framecpp=/opt/rtapps/framecpp --enable-local-timing ... controls@fb /opt/rtcds/rtscore/release/src/daqd 0$make ... controls@fb /opt/rtcds/rtscore/release/src/daqd 0$ install daqd /opt/rtcds/caltech/c1/target/fb/

It's now back to running and receiving data from the front ends (still not stable yet, though).

ELOG V3.1.3-