40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 117 of 344  Not logged in ELOG logo
ID Date Author Type Category Subject
  11461   Wed Jul 29 21:40:39 2015 KojiSummaryCDSStatus of the frame data syncing

The trend data hasn't been synced with LDAS since Jul 27 5AM local.

40m:

controls@nodus|minute > pwd
/frames/trend/minute
controls@nodus|minute > ls -l 11222 | tail
total 590432
-rw-r--r-- 1 controls controls 35758781 Jul 29 11:59 C-M-1122228000-3600.gwf
-rw-r--r-- 1 controls controls 35501472 Jul 29 12:59 C-M-1122231600-3600.gwf
-rw-r--r-- 1 controls controls 35296271 Jul 29 13:59 C-M-1122235200-3600.gwf
-rw-r--r-- 1 controls controls 35459901 Jul 29 14:59 C-M-1122238800-3600.gwf
-rw-r--r-- 1 controls controls 35550346 Jul 29 15:59 C-M-1122242400-3600.gwf
-rw-r--r-- 1 controls controls 35699944 Jul 29 16:59 C-M-1122246000-3600.gwf
-rw-r--r-- 1 controls controls 35549480 Jul 29 17:59 C-M-1122249600-3600.gwf
-rw-r--r-- 1 controls controls 35481070 Jul 29 18:59 C-M-1122253200-3600.gwf
-rw-r--r-- 1 controls controls 35518238 Jul 29 19:59 C-M-1122256800-3600.gwf
-rw-r--r-- 1 controls controls 35514930 Jul 29 20:59 C-M-1122260400-3600.gwf

 

LDAS Minute trend:

[koji.arai@ldas-pcdev3 C-M-11]$ pwd
/archive/frames/trend/minute-trend/40m/C-M-11
[koji.arai@ldas-pcdev3 C-M-11]$ ls -l | tail
-rw-r--r-- 1 1001 1001 35488497 Jul 26 19:59 C-M-1121997600-3600.gwf
-rw-r--r-- 1 1001 1001 35477333 Jul 26 21:00 C-M-1122001200-3600.gwf
-rw-r--r-- 1 1001 1001 35498259 Jul 26 21:59 C-M-1122004800-3600.gwf
-rw-r--r-- 1 1001 1001 35509729 Jul 26 22:59 C-M-1122008400-3600.gwf
-rw-r--r-- 1 1001 1001 35472432 Jul 26 23:59 C-M-1122012000-3600.gwf
-rw-r--r-- 1 1001 1001 35472230 Jul 27 00:59 C-M-1122015600-3600.gwf
-rw-r--r-- 1 1001 1001 35468199 Jul 27 01:59 C-M-1122019200-3600.gwf
-rw-r--r-- 1 1001 1001 35461729 Jul 27 02:59 C-M-1122022800-3600.gwf
-rw-r--r-- 1 1001 1001 35486755 Jul 27 03:59 C-M-1122026400-3600.gwf
-rw-r--r-- 1 1001 1001 35467084 Jul 27 04:59 C-M-1122030000-3600.gwf

 

  11460   Wed Jul 29 17:51:56 2015 IgnacioUpdatePEMAccelerators moved back to MC1 and MC2

We are done taking accelerator huddle test data. So I moved back all six accelerometers and cables to MC1 and MC2. I also relabel each of the accelerometers properly since the labels on them were confusing.

QED

 

Attachment 1: FullSizeRender.jpg
FullSizeRender.jpg
Attachment 2: FullSizeRender_2.jpg
FullSizeRender_2.jpg
  11459   Wed Jul 29 14:32:01 2015 SteveUpdateGeneralHow not to solder

 

Quote:

 

Quote:

Koji and Steve,

The result: bad Guralp x-arm cable.

I will swap the short cables tomorrow at the base.

 

Short 46" long cables at the base plates were swapped. Their solderings looked horrible.

This cable actually worked at 5-5-2015

Bad cable at ETMY station now.  The new cable should be a little bit longer ~52"

Koji could pull out easily 11 of the wires  from their socket.

Attachment 1: coldSoldering.jpg
coldSoldering.jpg
  11458   Wed Jul 29 11:15:21 2015 JessicaSummaryLSCPSDs of Arms with seismometer subtraction

Ignacio and I downloaded data from the STS, GUR1, and GUR2 seismometers and from the mode cleaner and the x and y arms. The PSDs along the arms have the most noise at frequencies greater than 1 Hz, so we should focus on filtering in that area. The noise levels start dropping at around 30 Hz, but are still much higher than is seen at frequencies below 1 Hz. However, because the spectra is so low at frequencies below that, Wiener filtering alone injected a significant amount of noise into those frequencies and did not do much to reduce the noise at higher frequencies. Pre-filtering will be required, and I have started implementing a pre-filter, but with no improvements yet. So far, I have tried making a bandpass filter, but a highpass filter may be ideal in this case because so much of the noise is above 1 Hz. 

Attachment 1: Arms_PSD.png
Arms_PSD.png
Attachment 2: XArm_PSD.png
XArm_PSD.png
Attachment 3: YArm_PSD.png
YArm_PSD.png
  11457   Wed Jul 29 10:34:42 2015 IgnacioSummaryLSCCoherence of arms and seismometers

Jessica and I took 45 mins  (GPS times from 1122099200 to 1122101950) worth of data from the following channels:

C1:IOO-MC_L_DQ (mode cleaner)
C1:LSC-XARM_IN1_DQ (X arm length)
C1:LSC-YARM_IN1_DQ (Y arm length)

and for the STS, GUR1, and GUR2 seismometer signals.

The PSD for MCL and the arm length signals is shown below,

I looked at the coherence between the arm length and each of the three seismometers, plot overload incoming below,

For the coherence between STS and XARM and YARM,

  

For GUR1,

  

Finally for GUR2,

 

A few remarks:

1) From the coherence plots, we can see that the arm length signals are coherent with the seismometer signals the most from 0.5 - 50 Hz. This is most evident in the coherence with STS. I think subtraction will be most useful in this range. This agrees with what we see in the PSD of the arm length signals, the magnitude of the PSD starts increasing from 1 Hz and reaches a maximum at about 30 Hz. This is indicative of which frequencies most of the noise is present.

2) Eric did not remember which of  GUR1 and GUR2 corresponded to the ends of XARM and YARM. So, I went to the end of XARM, and jumped for a couple seconds to disturb whatever Gurald was in there. Using dataviewer I determined it was GUR1. Anyways, my point is, why is GUR1 less coherent with both arms and not just XARM?  Since it is at the end of XARM, I was expecting GUR1 to be more coherent with XARM. Is it because, though different arms, the PSD's of both arms are roughly the same? 

3) Similarly, GUR2 shows about the same levels of coherence for both arms, but it is more coherent. Is GUR2 noisier because of its location?

Code: ARMS_COH.m.zip

Attachment 1: PSD_ARMS_MCL.png
PSD_ARMS_MCL.png
Attachment 2: XARM_STS_COH.png
XARM_STS_COH.png
Attachment 3: YARM_STS_COH.png
YARM_STS_COH.png
Attachment 4: XARM_GUR1_COH.png
XARM_GUR1_COH.png
Attachment 5: YARM_GUR1_COH.png
YARM_GUR1_COH.png
Attachment 6: XARM_GUR2_COH.png
XARM_GUR2_COH.png
Attachment 7: YARM_GUR2_COH.png
YARM_GUR2_COH.png
Attachment 8: ARMS_COH.m.zip
  11456   Tue Jul 28 20:42:50 2015 JessicaSummaryGeneralNew Seismometer Data Coherence

I was looking at the new seismometer data and plotted the coherence between the different arms of C1:PEM_GUR1 and C1:PEM_GUR2. There was not much coherence in the X arms, Y arms, or Z arms of each seismometer, but there were within the x and y arms of the seismometer.

I think the area we should focus on with filtering is lower ranges, between 0.01 and 0.1, because that it where coherence is most clearly high. It is higher in high frequencies but also incredibly noisy, meaning it probably wouldn't be good to try to filter there. 

Attachment 1: Coherence1.png
Coherence1.png
Attachment 2: Coherence2.png
Coherence2.png
  11455   Tue Jul 28 17:07:45 2015 JamieUpdateGeneralData missing
Quote:

For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?

Possible explanations:

  • The data transfers to LDAS had been shut off while we were doing the DAQ debugging. I don't know if they have been turned back on.  Unlikely this is the problem since you would probably see no data at all if this were the case.
  • wiper script parameters might have been changed to store less of the trend data for some reason.
  • Frame size is different and therefore wiper script parameters need to be adjusted.
  • Steve deleted it all.
  • ...
  11454   Tue Jul 28 16:42:25 2015 SteveUpdateGeneralJamies entry was deleted

Sorry Jamie, I accidentally deleted your elog entry #11453

  11453   Tue Jul 28 15:06:27 2015 SteveUpdateIOOPSL HEPA turned on

crying

Attachment 1: noHepa.jpg
noHepa.jpg
  11452   Tue Jul 28 15:05:09 2015 SteveUpdatesafetyfire alarm test ? no

We just had fire alarm trigged avacuation of the 40m lab.

It turned out that the CES building second floor sensor caused this action.

Attachment 1: 6minGaps.png
6minGaps.png
  11451   Tue Jul 28 14:58:59 2015 SteveUpdateGeneraldo not place anything on optical table tops

angrySpecially heavy items: old analoge scope or hardware loaded boxes......etc

 The table cover section holding crossbars are not evenly spaced.no

You have to center each cover section on the cross bar so it is supported on both sides !

 

I will clean up on this table tomorrow

 

 

Attachment 1: tabletops!.jpg
tabletops!.jpg
Attachment 2: tabletops!!.jpg
tabletops!!.jpg
  11450   Tue Jul 28 09:31:35 2015 IgnacioUpdateGeneralNewest accelerometer huddle test

I downloaded new accelerometer huddle test data from last night and analyzed it. This new data set is ~55 mins and uses the same Wiener filter parameters as previous huddle test analysis, the main difference being six accelerometers used in the Wiener filter and the improved experimental set up.

After computing the ASD for the self noise for each of the six accelerometers, (being witnessed by the remaining five), we get,

Now computing the mean of the above signals and the corresponding error bars gives the following result,

Comparing to prevoius huddle tests, I can note two trends on the Wiener subtraction:

1) When using six accelerometers, the subtraction above ~8 Hz drastically improves.

2) When using three accelerometers, there is better cancellation in the 1-5 Hz region, see http://nodus.ligo.caltech.edu:8080/40m/11442. This might have to do with how much more closer the accelerometers are to each other? 

Attachment 1: selfnoise_allsix_miso_newest.png
selfnoise_allsix_miso_newest.png
  11449   Tue Jul 28 05:00:03 2015 IgnacioUpdateGeneralSeismometer cans

I've have been talking a little bit with Steve about the seismometer enclosures.

We want to improve on the current stainless steel cans that cover the two Guralps at the end of the arms. In order to do this, we want to cover the interior of the cans with copper foil to improve the thermal conductivity of the enclosure to better control the temperature inside it. Ideally, we would want to copper plate the cans, but cost and difficulty has been an issue.

I have done some rough calculations and it seems that we need a copper layer of thickness being about a third that of the stainless steel can. This happens to be around 0.5-0.6 mm since we have 16 gauge (~1.6 mm) stainless steel cans. 

After wrapping the cans interior with copper, we will insulate them with foam in order to improve its thermal inertia. We want to probably use the same foam that Megan has been using for her seismometer enclosure. I have yet to think about a heater, but something similar to Megans resistor thing would work only smaller. I would be placed inside the can, right on the center of its bottom in order to ditribute heat evenly.

 

  11448   Mon Jul 27 17:51:06 2015 EveUpdateSummary PagesNew summary page tabs and other improvements

The summary pages can still be found at https://nodus.ligo.caltech.edu:30889/detcharsummary/ (EDIT: in an older version of this post I listed an incorrect url). They are operational and include data from some channels for intermittent periods of time.

 

Motivation: to make the summary pages more informative and useful for all

 

What I did:

I have added tabs for ALS, ASC, and LSC subsystems. While there is currently no data on the plots, I plan on checking all channels with DataViewer to set appropriate axis ranges so that we can actually see the data.

I altered which channels are used to represent spectra for OpLev systems to more appropriately provide PSDs.

I've changed the check code status page to include "warning" labels. Previously, when the summary pages ran, resulting in a warning message, the check code status page would list this as an "error", implying that the summary pages were not properly produced.

 

Results:

All features were implemented, but I need to investigate some of these channels to understand why we aren't seeing many channels in the plots. I am working on some other changes to the summary pages, including providing a Locked status which will only show data in a timeseries for a selected period of time.

  11447   Mon Jul 27 16:47:53 2015 ericqUpdateIOOMC2 -> MCL Actuator TF

Our noise cancellation SURFS will be doing online subtraction on the mode cleaner length, among other things. 

I made a measurement of the MC2 actuator transfer function by injecting noise from 1-100Hz into LSC_MC2_EXC for about 15 minutes, then estimating the TF from MC2_OUT to IOO_MC_L with CSD/PSD. The inverse of this TF will be applied to their Wiener target data to give us the direct subtration filter we want. 

I figured I would post the results here for posterity. The last time this seems to have been done is in ELOG 5900. There are some differences found here, the effective Q of the 1Hz pendulum resonance seems lower, and the behavior above 20Hz has definitely changed. 

IIR fits will be done by one of the SURFs to be used in their Wiener filter calculations. 

Data attached!

Attachment 1: mc2_2_mcl.png
mc2_2_mcl.png
Attachment 2: MC2_2_MCL_TF.txt.zip
  11446   Fri Jul 24 23:08:53 2015 IgnacioUpdatePEMMC1 accelerators moved for future huddle test

I have moved the MC1 accelerators and cable to the huddle test set up, in order to see how a six witness huddle test with the improved set up will do. 

Here is a picture of the accelerometer set up,

Our motivation for doing this is to see if more witness signals used in the Wiener filter really does indeed improve subtraction, as it was seen from previous huddle results, specially in the region above 10 Hz.

  11445   Fri Jul 24 20:32:15 2015 ericqUpdateElectronicsLSC LO distribution box power button replaced

As happened with the RF distribution box in the IOO rack a while back, the shiny blue power button in the LSC LO distribution box failed today. I replaced it with a simple switch, but since the original was a double throw, the replacement was way too big to fit without major panel surgery. So, instead, I installed it in the grille on the roof of the chassis. It a tight press/snap fit, though; I don't think it is at risk of easily coming loose. 

After reinstalling the box, I confirmed that POX POY and AS55 could all lock arms, so I deem the operation a success.

Before:

After:

Attachment 1: 2015-07-24_15.43.56.jpg
2015-07-24_15.43.56.jpg
Attachment 2: 2015-07-24_16.57.19.jpg
2015-07-24_16.57.19.jpg
  11444   Fri Jul 24 18:12:52 2015 Max IsiUpdateGeneralData missing

For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?

  11443   Fri Jul 24 13:49:09 2015 SteveUpdatesafetyLock doors please !

Thursday morning I found the control room emergency exit not locked.

Please check the doors when leaving the lab , specially when you are the last one out.

  11442   Thu Jul 23 21:12:14 2015 IgnacioUpdateGeneralNewer accelerometer huddle test data + detrending

In the last post concerning the self noise of the accelerometers, I mentioned the differences between the two data sets I was playing with. In order to give a more concrete analysis of the accelerometers self noise, we came to the conclusion that data taking time should be the same.

The plots below show the analysis for the following two datasets:

Old Data:  6 accelerometers, no cables clamped, no box, 55 mins worth of data.

Newer data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 57.66 mins worth of data, (we had 20 mins of data in the previous data set).

Filter parameters were kept the same in all calculations, the only change that was added to the analysis was the detrending of the signals using the detrend function on Matlab, this improved the results as the plots show. I also plotted the error bars for the Wiener filtered result for reference as well as the manufactures claimed self noise.

   

After detrending the data and taking a longer dataset we can see the improvement brought upon by the foam box in the low frequency band of 0.5 - 10 Hz, as shown in the bottom left plot. There is a lot of noise that needs to be cancelled out from 10 Hz and on, which brings to our attention the plot on the bottom right corner. This plot uses the old data but uses all six accelerometers as witnesses, it also improved overall after having detrended the data, but what is peculiar about this plot is the fact that it manages to mitigate the higher frequency 10 - ~100 Hz band noise. 

 

Attachment 1: old_data_1.png
old_data_1.png
Attachment 2: old_data_2.png
old_data_2.png
Attachment 3: new_data.png
new_data.png
Attachment 4: six_old_data.png
six_old_data.png
  11441   Thu Jul 23 20:57:15 2015 JessicaSummaryGeneralApplying Pre-filter to data before IIR Wiener Filtering

I updated my bandpass filter and have included the bode plot below in Figure 1. It is a fourth order elliptic bandpass filter with a passband ripple of 1dB and a stopband attenuation of 30 dB. It emphasizes the area between 3 and 40 Hz.

Below, I applied this filter to the huddle test data. The results from this were only slightly better in the targeted region than when no pre-filter was applied. 

When I pre-filtered the mode cleaner data and then used an IIR wiener filter, I found that the results did not differ much from the data that was not pre-filtered. I'm not sure yet if I'm targeting the right region of this data with my bandpass filter, and will be looking more into choosing a better region. Also, I am only using certain regions of ff when calculating the transfer function, and need to optimize that region also. I uploaded the code I used to make these plots to github.

Attachment 1: BodePlot.png
BodePlot.png
Attachment 2: acc_bandpass.png
acc_bandpass.png
Attachment 3: mcl_seis.png
mcl_seis.png
  11440   Thu Jul 23 20:54:42 2015 JessicaUpdateGeneralALS Delay Line Box

The front panels for the ALS delay line box came in last week. Some of the holes for the screws were slightly misaligned, so I filed those and everything is now put together. I just need to test both front panels to determine if the SMAs should be isolated or not.

 

Koji had also suggested making the holes in the front and back panel conical recesses so that flat head screws could be used and would counteract the anodization of the panel and avoid the SMAs being isolated. I think if we did that then conductivity would be ensured throughout the panel and also through the rest of the box. I also think one way we could test this before drilling conical recesses would be to test both front panels now, as one has isolated SMAs and one has conductive SMAs. If the anodization of the panel isolated the SMA regardless, we could potentially figure this out by testing both panels. But, would it also be that it is possible that the isolation of the SMA itself does not matter and so this test would tell us nothing? Is there a better way to test if the SMAs are being isolated or not? Or would this be more time consuming than just drilling conical recesses as a preventative measure?

  11439   Thu Jul 23 15:41:41 2015 SteveUpdatePEMfitting ants with TERRO

Pasadena got 0.2" of  rain on Saturday. Temperatures were high with high humidity since than.The ants were back in the Control room east side benches.

We have started using TERRO Liquid Ant Baits in January 2015   This worked very well to this point.

Tree new packages were opened  yesterday and the ants are gone.

We can conclude that these baits must be replaced after 6 mounts.

The liquid baits contains BORAX  and it is safe.

 

  11438   Thu Jul 23 03:09:05 2015 ericqUpdateLSCaLIGO demod board lives!

I'm a little mystified. Peeking inside the aLIGO demod board, I saw that the reason that two of the channels weren't working was that their power connectors weren't plugged in, so no real mystery there. 

I hooked up the board at the electronics bench, and found the noise to be completely well behaved, in contrast to the measurements I made when it was in the LSC rack. I've taken it back out to the LSC rack, and given it the X beatnote, and it seems to be performing pretty well. 


I switched between the aLIGO demod board and beatbox during the same lock / beat. The LSC board performs margnially better from 3-100 Hz. The high frequency noise comes from the green PDH loop (coherence is near one above a few hundred Hz), so we don't expect any difference there. 

To me, the beatbox noise looks like there is a broad feature that is roughly the same level as the real cavity motion in the 10-100 Hz range. So, I think we should use the aLIGO board afterall, presuming the noise doesn't shoot back up when I remount it in the rack...


The ALS noise is getting low enough where our normal approach of measuring ALS sensing noise by simply taking the PSD of the signal when the arm is PDH locked is not quite valid anymore, as it is sensing the real cavity fluctuations. Doing a frequency domain coherent subtration of the PSDs suggests a sensing noise RMS of ~150Hz for ALSX. 

When the X arm is locked on ALS, POX sees about 250Hz RMS out of loop noise, which isn't the greatest; however, I used to be happy with 500Hz. By eye, sweeping through IR resonance is smoother. The real test is to get the Y arm ALS running, and swing it through PRFPMI resonance...


Fair warning, the LSC rack area is not so tidy right now, the demod board is resting on a stool (but not in the way of walking down the arm). I'll clean this up tomorrow. 

Attachment 1: beatbox_vs_aLIGO.png
beatbox_vs_aLIGO.png
  11437   Wed Jul 22 22:06:42 2015 EveSummarySummary PagesFuture summary pages improvements

- CDS Tab

We want to monitor the status of the digital control system.

1st plot
Title: EPICS DAQ Status
I wonder we can plot the binary numbers as statuses of the data acquisition for the realtime codes.
We want to use the status indicators. Like this:
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20150722/plots/H1-MULTI_A8CE50_SEGMENTS-1121558417-86400.png

channels:
C1:DAQ-DC0_C1X04_STATUS
C1:DAQ-DC0_C1LSC_STATUS
C1:DAQ-DC0_C1ASS_STATUS
C1:DAQ-DC0_C1OAF_STATUS
C1:DAQ-DC0_C1CAL_STATUS

C1:DAQ-DC0_C1X02_STATUS
C1:DAQ-DC0_C1SUS_STATUS
C1:DAQ-DC0_C1MCS_STATUS
C1:DAQ-DC0_C1RFM_STATUS
C1:DAQ-DC0_C1PEM_STATUS

C1:DAQ-DC0_C1X03_STATUS
C1:DAQ-DC0_C1IOO_STATUS
C1:DAQ-DC0_C1ALS_STATUS

C1:DAQ-DC0_C1X01_STATUS
C1:DAQ-DC0_C1SCX_STATUS
C1:DAQ-DC0_C1ASX_STATUS

C1:DAQ-DC0_C1X05_STATUS
C1:DAQ-DC0_C1SCY_STATUS
C1:DAQ-DC0_C1TST_STATUS

1st plot
Title: IOP Fast Channel DAQ Status
These have two bits each. How can we handle it?
If we need to shrink it to a single bit take "AND" of them.
C1:FEC-40_FB_NET_STATUS (legend: c1x04, if a legend placable)
C1:FEC-20_FB_NET_STATUS (legend: c1x02)
C1:FEC-33_FB_NET_STATUS (legend: c1x03)
C1:FEC-19_FB_NET_STATUS (legend: c1x01)
C1:FEC-46_FB_NET_STATUS (legend: c1x05)

3rd plot
Title C1LSC CPU Meters
channels:
C1:FEC-40_CPU_METER (legend: c1x04)
C1:FEC-42_CPU_METER (legend: c1lsc)
C1:FEC-48_CPU_METER (legend: c1ass)
C1:FEC-22_CPU_METER (legend: c1oaf)
C1:FEC-50_CPU_METER (legend: c1cal)
The range is from 0 to 75 except for c1oaf that could go to 500.
Can we plot c1oaf with the value being devided by 8? (Then the legend should be c1oaf /8)

4th plot
Title C1SUS CPU Meters
channels:
C1:FEC-20_CPU_METER (legend: c1x02)
C1:FEC-21_CPU_METER (legend: c1sus)
C1:FEC-36_CPU_METER (legend: c1mcs)
C1:FEC-38_CPU_METER (legend: c1rfm)
C1:FEC-39_CPU_METER (legend: c1pem)
The range is be from 0 to 75 except for c1pem that could go to 500.
Can we plot c1pem with the value being devided by 8? (Then the legend should be c1pem /8)

5th plot
Title C1IOO CPU Meters
channels:
C1:FEC-33_CPU_METER (legend: c1x03)
C1:FEC-34_CPU_METER (legend: c1ioo)
C1:FEC-28_CPU_METER (legend: c1als)
The range is be from 0 to 75.

6th plot
Title C1ISCEX CPU Meters
channels:
C1:FEC-19_CPU_METER (legend: c1x01)
C1:FEC-45_CPU_METER (legend: c1scx)
C1:FEC-44_CPU_METER (legend: c1asx)
The range is be from 0 to 75.

7th plot
Title C1ISCEY CPU Meters
channels:
C1:FEC-46_CPU_METER (legend: c1x05)
C1:FEC-47_CPU_METER (legend: c1scy)
C1:FEC-91_CPU_METER (legend: c1tst)
The range is be from 0 to 75.

=====================

IOO

We want a duty ratio plot for the IMC. C1:IOO-MC_TRANS_SUM >1e4 is the good period.

Duty ratio plot looks like the right plot of the following link
https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20150722/lock/segments/

=====================

SUS: OPLEV

OL_PIT_INMON and OL_YAW_INMON are good for the slow drift monitor.
But their sampling rate is too slow for the PSDs.
Can you use
C1:SUS-ETM_OPLEV_PERROR
C1:SUS-ETM_OPLEV_YERROR
etc...
For the PSDs? They are 2kHz sampling DQ channels. You would be able to plot
it up to ~1kHz. In fact, we want to monitor the PSD from 100mHz to 1kHz.
How can you set up the resolution (=FFT length)?

=====================

LSC / ASC / ALS tabs

Let's make new tabs LSC, ASC, and ALS

LSC:

We should have a plot for
C1:LSC-TRX_OUT_DQ
C1:LSC-TRY_OUT_DQ
C1:LSC-POPDC_OUT_DQ
It's OK to use the minute trend for now.
You can check the range using dataviewer.

ASC:

Let's use
C1:SUS_MC1_ASCPIT_OUT16 (legend: IMC WFS)
C1:ASS-XARM_ITM_YAW_OSC_CLKGAIN (legend: XARM ASS)
C1:ASS-YARM_ITM_YAW_OSC_CLKGAIN (legend: YARM ASS)
C1:ASX-XARM_M1_PIT_OSC_CLKGAIN (legend: XARM Green ASS)
as the status indicators. There is no YARM Green ASS yet.

ALS:

Title: ALS Green transmission
We want a time series of
ALS-TRX_OUT16
ALS-TRY_OUT16

Title: ALS Green beatnote
Another time series
ALS-BEATX_FINE_Q_MON
ALS-BEATY_FINE_Q_MON

Title: Frequency monitor
We have frequency counter outputs, but I have to talk to Eric to know the channel names

  11436   Wed Jul 22 15:09:40 2015 ericqUpdateGeneralIMC work

The other night, I spent some time with the mode cleaner. 

First, I made some model changes to the MC_TRANS part of c1mcs.mdl. Specifically, I brought in the userapps QPD part that we are using for the transmon and oplev QPDs. My main motivation for doing so was to have FMs for the pitch and yaw values, to be able to set offsets. Up until now, we have used a QPD centering servo in conjunction with the WFS, but the center of the QPD is not perfectly aligned to represent the center of MC2. Using offsets at the servo isn't really practical, since there are integrators. 

I then spent some time manually aligning the mode cleaner mirrors with WFS off, followed by centering the in-lock MC REFL beam on the WFS QPDs, and setting the WFS and MC_TRANS offsets. (I updated the WFS offset script, and made one for MC_TRANS in scripts/MC/WFS. They now use averaging instead of servoing to zero, a la LSC PD offset script). 

The resultant intracavity power and RIN was an improvement over the older offsets. (RMS RIN went down by half, I think.)

Since Monday, the autolocker seems to be having some trouble. At first, I suspected the changes I made weren't so hot after all, but I've now noticed a pattern. Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh. 

  11435   Wed Jul 22 14:10:04 2015 ericqUpdateCDSRCG Diagnostics

Now that we've seemed to landed on a working configuration, I've re-ran the tests first described in ELOG 11347. I've also compared the real filtering of filter modules to their designs. 

TL;DR: No adverse, or even observable, differences have been witnessed.

As a reminder: In c1tst, there are three loops, called LOOPA, LOOPB, and LOOPC.

  • LOOPA is a filter module feeding back onto its own input, with a unit time delay block
  • LOOPB is a FM whose output goes to the DAC. In meatspace, the AI output is hooked up directly to an AA chassis input, and back to the FM in CDS
  • LOOPC includes RFM connections to c1rfm and back again. 

Here are the loop delay results, which measure the slope of the phase response of the OLTF. For the purely digital loops (A, C), we know the number of cycles we expect to compare the delay to.

At this time, I haven't done the adding up of cycles, zero-order-holds, etc. to get the delay we expect from the analog loop (B), so I've just looked at whether it changed at all. 

Anyways, I've attached the code that analyzes data from DTT-exported text files containing the continuous phase data from the loop measurements. 

Before:

Single Model loop cycles: 1.0000000+/-0.0000006, disparity: -0.00+/-0.25 nsec
2 Model RFM loop cycles: 1.9999999+/-0.0000013, disparity: 0.0+/-0.5 nsec
ADC->DAC loop time: 338.2+/-0.4 usec

After:

Single Model loop cycles: 0.9999999+/-0.0000008, disparity: 0.02+/-0.29 nsec
2 Model RFM loop cycles: 2.0000001+/-0.0000011, disparity: -0.0+/-0.4 nsec
ADC->DAC loop time: 338.18+/-0.35 usec

So, the digital loops take the number of cycles we expect, and there are no real differences after the upgrade. 


Additionally, for all three loops, I created a simple 100:10 filter in foton, and injected broadband noise with awggui, to measure the real TF applied by the FM code. I want to turn this whole process into a single script that will switch the filter on and off, read the foton file, and compare the measured TF to the ideal shape. 

In our system, before and after the upgrade, all three loops showed no appreciable difference from the designed filter shape, other than some tiny uptick in phase when approaching the nyquist frequency. This may be due to the fact that I'm comparing to the ideal analog filter, rather than what a 16kHz digital filter looks like. 

What I've plotted below is the devitation from the ideal zpk(100Hz, 10Hz, 0.1) frequency response, i.e. Hmeasured / Hideal. The code to do this analysis is also attached, it estimates the TF by dividing the CSD of the filter input and output by the PSD of the input. The single worst coherence in any bin of all the measurements is 0.997, so I didn't really bother to estimate the error of the TF estimate. 

Attachment 1: filterShapes.png
filterShapes.png
Attachment 2: CDS_diag_codes.zip
  11434   Tue Jul 21 21:33:22 2015 Max IsiUpdateGeneralSummary pages moved to 40m LDAS account

The summary pages are now generated from the new 40m LDAS account. The nodus URL (https://nodus.ligo.caltech.edu:30889/detcharsummary/) is the same and there are no changes to the way the configuration files work. However, the location on LDAS has changed to https://ldas-jobs.ligo.caltech.edu/~40m/summary/ and the config files are no longer version-controlled on the LDAS side (this was redundant, as they are under VCS in nodus).

I have posted a more detailed description of the summary page workflow, as well as instructions to run your own jobs and other technical minutiae, on the wiki: https://wiki-40m.ligo.caltech.edu/DailySummaryHelp

  11433   Tue Jul 21 21:25:18 2015 Max IsiUpdateGeneral40m LDAS account

A shared LIGO Data Grid (LDG) account was created for use by the 40m lab. The purpose of this account is to provide access to the LSC computer cluster resources for 40m-specific projects that may benefit from increased computational power and are not linked to any user in particular (e.g. the summary pages).

For further information, please see https://wiki-40m.ligo.caltech.edu/40mLDASaccount

  11432   Tue Jul 21 05:17:09 2015 IgnacioUpdateGeneralMore clear accelerometer huddle tests results

I generated the following plots from the two sets of huddle test data we have for the accelerometers. 

Old data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.

New data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 20 mins worth of data.

I made sure to use the same Impuse response time (6 sec) and sampling frequency (256 Hz), as well as every other parameter for the calculations.

  

 

Top left: The resultant self noise curve using the new data, there is definitely and improvement in the 0.5-2 Hz band. 

Top right: Resultant self noise using the old data, for the first set of three accelerometers.

Bottom left: Old data result for the remaining three accelerometers.

Bottom right: Old data result, using all six accelerometers as witnesses instead.

Attachment 1: new_data.png
new_data.png
Attachment 2: new_data.png
new_data.png
Attachment 3: old_data_1.png
old_data_1.png
Attachment 4: old_data_1.png
old_data_1.png
Attachment 5: old_data_2.png
old_data_2.png
  11431   Mon Jul 20 16:45:15 2015 Max IsiConfigurationGeneralSummary page c1sus.ini error corrected

Bad syntax errors in the c1sus.ini config file were causing the summary pages to crash: a plot type had not been indicated for plots 5 and 6, so I've made these "timeseries."
In the future, please remember to always specify a plot type, e.g.:

BAD:

5 = C1:SUS-ETMX_SUSPIT_INMON.mean,
       C1:SUS-ITMY_SUSPIT_INMON.mean,

GOOD:

3 = C1:SUS-ETMX_SUSPIT_INMON.mean,
       C1:SUS-ITMY_SUSPIT_INMON.mean timeseries

By the way, the pages will continue to be unavailable while I transfer them to the new shared account.

  11430   Mon Jul 20 11:57:17 2015 ericqUpdateGeneralArm Locking recovered

The interferometer is warming up!

I had some issues locking the IMC at first. It turned out that the MC3 side OSEM signal wasn't getting to the ADC. A satellite box sqush fixed it. 

I touched up the PMC alignment; the best I could do is 0.75V, probably due to the AOM being in place. 

I haven't touched the WFS offsets, but the current ones seem to be doing ok. I'll touch them up tonight when the seismic activity has calmed. 

I made some changes to the state of the PZT/PC crossover gain in the mcdown script, resulting in the IMC catching lock quicker. 

Thankfully, the tip tilt pointing stayed good during the upgrade. I barely had to touch the ETM alignment to lock the arms. ETMX is showing some errant motion, though... 

  11429   Sat Jul 18 16:59:01 2015 jamieUpdateCDSunloaded, turned off loading of, symmetricom kernel module on fb

fb has been loading a 'symmetricom' kernel module, presumably because it was once being used to help with timing.  It's no longer needed, so I unloaded it and commented out the lines that loaded it in /etc/conf.d/local.start.

  11428   Sat Jul 18 16:03:00 2015 jamieUpdateCDSEPICS freezes persist

I notice that the periodic EPICS freezes persist.  They last for 5-10 seconds.  MEDM completely freezes up, but then it comes back.

The sites have been noticing similar issues on a less dramatic scale.  Maybe we can learn from whatever they figure out.

  11427   Sat Jul 18 15:37:19 2015 JamieSummaryCDSCDS upgrade: current status

So it appears we have found a semi-stable configuration for the DAQ system post upgrade:

Here are the issues:

daqd

dadq is running mostly stably for the moment, although it still crashes at the top of every hour (see below).  Here are some relevant points of about the current configuration:

  • recording data from only a subset of front-ends, to reduce the overall load:
    • c1x01
    • c1scx
    • c1x02
    • c1sus
    • c1mcs
    • c1pem
    • c1x04
    • c1lsc
    • c1ass
    • c1x05
    • c1scy
  • 16 second main buffer:
    start main 16;
  • trend lengths: second: 600, minute: 60
    start trender 600 60;
  • writing to frames:
    • full
    • second
    • minute
    • (NOT raw minute trends)
  • frame compression ON

This elliminates most of the random daqd crashing.  However, daqd still crashes at the top of every hour after writing out the minute trend frame. Still unclear what the issue is, but Keith is investigating.  In some sense this is no worse that where we were before the upgrade, since daqd was also crashing hourly then.  It's still crappy, though, so hopefully we'll figure something out.

The inittab on fb automatically restarts daqd after it crashes, and monit on all of the front ends automatically restarts the mx_stream processes.

front ends

The front end modules are mostly running fine.

One issue is that the execution times seem to have increased a bit, which is problematic for models that were already on the hairy edge.  For instance, the rough aversage for c1sus has some from ~48us to 50us.  This is most problematic for c1cal, which is now running at ~66us out of 60, which is obviously untenable.  We'll need to reduce the load in c1cal somehow.

All other front end models seem to be working fine, but a full test is still needed.

There was an issue with the DACs on c1sus, but I rebooted and everything came up fine, optics are now damped:

  11426   Sat Jul 18 14:55:33 2015 jamieUpdateGeneralall front ends back up and running

After some surgery yesterday the front ends are all back up and running:

  • Eric found that one of the DAC cards in the c1sus front end was not being properly initialized (with the new RCG code).  Turned out that it was an older version DAC, with a daughter board on top of a PCIe board.  We suspected that there was some compatibility issue with that version of the card, so Eric pulled an unused card from c1ioo to replace the one in c1sus.  That worked and now c1sus is running happily.
  • Eric put the old DAC card into c1ioo, but it didn't like it and was having trouble booting.  I removed the card and c1ioo came up fine on it's own.
  • After all front end were back up and running, all RFM connections were dead.  I tracked this down to the RFM switch being off, because the power cable was not fully seated.  This probably happened when Steve was cleaning up the 1X4/5 racks.  I re-powered the RFM switch and all the RFM connections came back on-line
  • All receivers of Dolphin (DIS) "PCIE" IPC signals from c1ioo where throwing errors.  I tracked this down to the Dolphin cable going to c1ioo being plugged in to the wrong port on the c1ioo dolphin card.  I unplugged it and plugged it into the correct port, which of course caused all front end modules using dolphin to crash.  Once I restarted all those models, everything is back:

  11425   Sat Jul 18 06:12:07 2015 IgnacioUpdateGeneralMCL Wiener filtering + FIR to IIR conversion using vectfit (Update)

(updateAfter Eric gave me feedback on my previous elog post, I went back and fixed some of the silly stuff I stated.

First of all, I have come to realized that it makes zero sense to plot the ASD's of the mode cleaner against the seismometer noise. These measurements are not only quite different, but elementary, they posess different units. I have focused my attention to the MCL being Wiener filtered with the three siesmometer signals. 

One of the major improvements that I make in the following analysis is,

1) Prefiltering; a band pass filter from 1 to 5 Hz, in order to emphasize subtraction of the bump shown in the figure below.

2) I have used vectfit exclusively in the 1 to ~5 Hz range, in order to model the FIR filter properly, as in, the kind of subtraction that we care about. Limiting myself to the 1 - 5 Hz range has allowed me to play freeley with the number of poles, hence being able to fit the FIIR filter properly with an IIR rational transfer function properly,

The resulting ASD's are shown below, in blue we show the raw MCL output, in blac the Wiener filter (FIR) result, and finally in black, the resultant data being filtered with the calculated IIR Wiener filter.

 

Now, in the following plots I show the IIR Wiener filters for each of the three seismometers, 

X Seismometer,

For the Y seismometer,

and for the Z seismometer,

 

The matlab code for this work is attached: code.zip

Attachment 1: Wiener_MCL_seismometers_iir.png
Wiener_MCL_seismometers_iir.png
Attachment 2: seisx_mag.png
seisx_mag.png
Attachment 3: seisx_mag.png
seisx_mag.png
Attachment 4: seisx_mag.png
seisx_mag.png
Attachment 5: seisx_ph.png
seisx_ph.png
Attachment 6: seisy_mag.png
seisy_mag.png
Attachment 7: seisy_mag.png
seisy_mag.png
Attachment 8: seisy_mag.png
seisy_mag.png
Attachment 9: seisy_ph.png
seisy_ph.png
Attachment 10: seisz_ph.png
seisz_ph.png
Attachment 11: seisz_ph.png
seisz_ph.png
Attachment 12: code.zip
Attachment 13: seisz_mag.png
seisz_mag.png
Attachment 14: seisz_mag.png
seisz_mag.png
Attachment 15: seisz_ph.png
seisz_ph.png
  11424   Fri Jul 17 04:56:37 2015 IgnacioUpdateGeneralMCL Wiener filtering + FIR to IIR conversion using vectfit

 

We took data for the mode cleaner a while ago, June 30th I believe. This data contained signals from the six accelerometers and the three seismometers. In here I have only focused on the seimometer signals as witnesses in order to construct Wiener filters for each of the three seismometer signals (x,y,z) and for the combined seismometers signal. The following plot showing the ASD's shows the results,

 Wiener filtering works beautifully for the seismometers. Note that subtraction is best when we use all three seismometers as the witnesses in the Wiener filter calculation, as can be clearly seen in the first plot above.

Now, I used vectfit to conver the Wiener FIR filters for each seismometer to their IIR versions. The following are the bode plots for the IIR filters,

For the x-direction seismometer,

 

For the y-direction seismometer

,

 

And for the z-direction seismometer,

 The IIR filters were computed using 5 zeros and 5 poles using vectfit. That was the maximum number of poles that I could use wihtout running into trouble with matrices being almost singular in Matlab. I still need to figure out how to deal with this issue in more detail as fitting the y-seismometer was a bit problematic. I think having a greater number of poles will make the fitting a bit easier.

Attachment 1: Wiener_MCL_seismometers.png
Wiener_MCL_seismometers.png
Attachment 2: seisx_mag.png
seisx_mag.png
Attachment 3: seisx_mag.png
seisx_mag.png
Attachment 4: seisx_mag.png
seisx_mag.png
Attachment 5: seisx_phase.png
seisx_phase.png
Attachment 6: seisy_mag.png
seisy_mag.png
Attachment 7: seisy_phase.png
seisy_phase.png
Attachment 8: seisz_mag.png
seisz_mag.png
Attachment 9: seisz_phase.png
seisz_phase.png
  11423   Fri Jul 17 02:46:07 2015 IgnacioUpdateGeneralNew huddle test data for Wilcoxon 731A results

On Thursday, new huddle test data for the Wilcoxon 731A was aquired by Eric. 

The difference between this new data and the previous data, is:

1) We used three accelerometers instead of six this time around.

2) We used a foam box, and clamped cables on the experimental set up as shown in the previous elog, http://nodus.ligo.caltech.edu:8080/40m/11389

I have analyzed the new data. Here I present my results.

The following plot shows the ASD's for the three accelerometers raw outputs as well as their error signals computed using the three cornered hat method,

As before, I computed the mean for the output signals of the accelerometers above as well as their mean self noise to get the following plot

,

Now, below I compare the new results with the results that I got from the old data, 

Did the enclosure and cable clamping do much? Not really, according to the computed three hat results. Also, notice how much better, even if its a small improvement, we get from using six accelerometers and calculating their self noise by the six cornered hat method.

 

Now, I moved on to analyzing the same data with Wiener Filtering.

Here are again, the raw outputs, and the self noises of each individual accelerometer calculated using Wiener filtering,

The accelerometer in the Y direction is show a kind of funky signal at low frequncies. Why? Anyways, I calculated the mean of the above signals as I did for the three cornered hat method above to get the following, I also show the means of the signals computed with the old data using wiener filtering,

Is the enclosure really doing much? The Wiener filter that I applied to the huddle test old data gave me a much better, by an order of magnitude better self noise curve. Keep in mind that this was using SIX accelerometers, not THREE as we did this time. I want to REDO the huddle test for the WIlcoxon accelerometers using SIX accelerometers with the improved experimental setup to see what I get.

Finally, I compare the computed self noises above with what the manufacturer gives,

,

As I expected, the self noise using six accelerometers and Wiener filtering is the best I could work out. The three cornered hat method works out pretty well from 1 to 10 Hz, but the noise is just too much anywhere higher than 10 Hz. The enclosed, clamped, 3 accelerometer wiener filter result is an order of magnitude worse than the six accelerometer wiener filtered result, and two orders of magnitude worse than the three cornered hat method in the 1 to 10 Hz frequency band. 

As I stated, I think we must performed the huddle test with SIX accelerometers and see what kind of results we get.

Attachment 1: selfnoise_allthree_threehat_enclosed.png
selfnoise_allthree_threehat_enclosed.png
Attachment 2: selfnoise_3hat_enclosed_averages.png
selfnoise_3hat_enclosed_averages.png
Attachment 3: selfnoise_3hat_6hat_enc.png
selfnoise_3hat_6hat_enc.png
Attachment 4: miso_wiener_enclosedall.png
miso_wiener_enclosedall.png
Attachment 5: selfnoise_wiener_enclosed.png
selfnoise_wiener_enclosed.png
Attachment 6: compare_encl.png
compare_encl.png
  11422   Thu Jul 16 16:46:18 2015 ericqUpdateGeneralStarting IFO recovery, DAC troubles

Jamie showed me how to use the SDF system. We created new safe.snap files for all of the running models based on the autoburts from the morning of July 1st, before the upgrade began, and then pruned them of invalid channels. 

Now all of the models start up without having to race for the BURT button. yes

We saw that c1sus was timing out all over the place once the filter settings had been restored. I was thinking I would move one of the vertex optics into c1mcs, but instead I found it easier to remove the global damping parts. Now the c1sus model runs at ~50usec.

The c1sus frontend's DAC is still nonfunctional. Jamie is seeking advice. 

  11421   Thu Jul 16 16:33:56 2015 JessicaUpdateGeneralAdded Bode Plots of Bandpass Filter

I updated the bandpass filter I was using, finding that having different stopband attenuations before and after the passband better emphasized the area from 3 Hz to 20 Hz. I chose a low passband ripple but high stopband attenuation to do this. My passband ripple was 2 dB, the first stopband was 25 dB, and the second stopband attenuation was 40 dB. As can be seen in the filter Magnitude plot, this resulted in a fairly smooth passband and a fairly step dropoff to the stopband, which will better emphasize the region I am trying to isolate. My goal was to emphasize the 3-20 Hz region 10-30 times more than the outside regions. I think I accomplished this by looking at the Bode plot, but I may have chosen the second stopband attenuation to be slightly too high for this. 

Attachment 1: acc1_update.png
acc1_update.png
Attachment 2: acc2_update.png
acc2_update.png
Attachment 3: acc3_update.png
acc3_update.png
Attachment 4: bp_BodeMag.png
bp_BodeMag.png
Attachment 5: bp_BodePhase.png
bp_BodePhase.png
  11420   Thu Jul 16 11:18:37 2015 jamieUpdateGeneralStarting IFO recovery, DAC troubles
Quote:

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  crying

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 

We need to update the indicators on the CDS_FE_STATUS screen to expose the new indicators, so that we have better visibility for these issues.

I'm not sure why this DAC is failing. It may indicate an actual problem with the DAC itself.

Quote:

As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

This is related to changes to how the front ends load their safe.snaps. I think they're now explictly expecting the file:

targtet/<model>/<model>epics/burt/safe.snap

I'll come over this afternoon and we can get acquainted with the new SDF system that now handles management of the safe.snap files.

  11419   Thu Jul 16 03:01:57 2015 ericqUpdateLSCOld beatbox hooked back up

I was having issues trying to get reasonable noise performance out of the aLIGO demod board as an ALS DFD. Terminating the inputs to the LSC whitening inputs did not show much 60Hz noise, and an RMS in the single Hz range. 

A 60Hz line of hundreds of uV was visible in the power spectrum of the single ended BNC and double-ended DB25 outputs of the board no matter how I drove or terminated.

So, I tried out hooking up the ALS beatbox. It turns out to work better for the time being; not only is the 60Hz line in the analog outputs about ten times smaller, the broadband noise floor in the resultant beat spectrum when driven by a 55MHz LO on the LSC rack is a fair bit lower too. I wonder if this is due to not driving the aLIGO board LO at the +10dBm it expects. With the amplifiers and beat note amplitudes we have, we'd only be able to supply around 0 dBm anyways. 

Here's a comparison of the aLIGO board (black) and ALS beatbox (dark green) driven with the 55MHz LO, both going through the LSC whitening filters for a resultant magnitude of 3kCounts in the I-Q plane. The RMS sensing noise is about 30 times lower for the beatbox. (Note, this is with the old delay cables. When we switch to the 50m cables, we'll win further frequency noise sensitivity through the better degrees->Hz calibration.) I'm very interested to see what the green beat spectrum looks like with this setup. 

Not only is the 60Hz line smaller, there is simply less junk in the beatbox signal. I did not expect this to be the case. 

There were some indications of funky status of the aLIGO board: channels 3 and 4 are totally nonfunctioning, so who knows what's going on in there. I've pulled it out, to take a gander if I can figure out how to make it suitiable for our purposes. 

Attachment 1: beat_comparison.png
beat_comparison.png
Attachment 2: aLIGO_vs_beatbox.xml.zip
  11418   Thu Jul 16 01:04:21 2015 ericqUpdateGeneralStarting IFO recovery, DAC troubles

I've been trying to start recovering IFO functionality, but quickly hit a frustrating roadblock. 

Upon opening the PSL shutter, and deactivating the MC mirror watchdogs, I saw the MC reflected beam moving way more than normal. 

A series of investigations revealed no signals coming out of c1sus's DAC.  crying

The IOP (c1x02) shows two of its DAC-related statewords (DAC and DK) in a fault mode, which means (quoting T1100625):

"As of RCG V2.7, if an error is detected in oneor more DAC modules, the IOP will continue to run but only write zero values to the DAC modules as a protective measure. This can only be cleared by restarting the IOP and all applications running on the affected computer."

The offending card may be DAC1, which has its fourth bit red even with only the IOP running, which corresponds to a "FIFO error". /proc/c1x02/status states, in part:

DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=3 (empty)
DAC #2 16-bit fifo_status=2 (OK)

Squishing cables and restarting the frontend have not helped anything. 

c1lsc, c1isce[x/y] are not suffering from this problem, and appear to be happily using their DACs. c1ioo does not use any DAC channels. 


As a further headache, any time I restart any of the models on the c1sus frontend, the BURT restore is totally bunk. Moreover, using burtgooey to restore a good snapshot to the c1sus model triggers a timing overflow and model crash, maybe not so surprising since the model seems to be averaging ~56usec or so. 

  11417   Wed Jul 15 18:19:12 2015 JamieSummaryCDSCDS upgrade: tentative stabilty?

Keith Thorne provided his eyes on the situation today and had some suggestions that might have helped things

Reorder ini file list in master file.  Apparently the EDCU.ini file (C0EDCU.ini in our case), which describes EPICS subscriptions to be recorded by the daq, now has to be specified *after* all other front end ini files.  It's unclear why, but it has something to do with RTS 2.8 which changed all slow channels to be transported over the mx network.  This alone did not fix the problem, though.

Increase second trend frame size.  Interestingly, this might have been the key.  The second trend frame size was increased to 600 seconds:

start trender 600 60;

The two numbers are the lengths in seconds for the second and minute trends respectively.  They had been set to "60 60", but Keith suggested that longer second trend frames are better, for whatever reason.  It seems he may be right, given that daqd has been running and writing full and trend frames for 1.5 hours now without issue. 


As I'm writing this, though, the daqd just crashed again.  I note, though, that it's right after the hour, and immediately following writing out a one hour minute trend file.  We've been seeing these hour, on the hour, crashes of daqd for quite a while now.  So maybe this is nothing new.  I've actually been wondering if the hourly daqd crashes were associated with writing out the minute trend frames, and I think we might have more evidence to point to that.

If increasing the size of the second trend frames from 60 seconds (35M) to 600 seconds (70M) made a difference in stability, could there be an issue since writing out files that are smaller than some value?  The full frames are 60M, and the minute trends are 35M.

  11416   Wed Jul 15 17:05:06 2015 JessicaUpdateGeneralBandpass Pre-Filter created

I applied a bandpass filter to the accelerometer huddle data as a pre-filter. The passband was from 5 Hz to 20 Hz. I found that applying this pre-filter did very little when comparing the PSD after pre-filtering to the PSD with no pre-filtering. There was some improvement though, just not a significant amount. For some reason, it also seemed as though the second accelerometer improved the most from pre-filtering the data, while the first and third remained closer to the unfiltered noise. Also, I have not yet figured out a consistent method for choosing passband ripple and stopband attentuation, both of which determine how good the filter is. 

My next step in pre-filtering will be determining a good method for choosing passband ripple and stopband attenuation, along with implementing other pre-filtering methods to combine with the bandpass filter. 

Attachment 1: acc1.png
acc1.png
Attachment 2: acc2.png
acc2.png
Attachment 3: acc3.png
acc3.png
  11415   Wed Jul 15 13:19:14 2015 JamieSummaryCDSCDS upgrade: reducing mx end-points as last ditch effort

I tried one last thing, suggested by Keith and Gerrit.  I tried reducing the number of mx end-points on fb to zero, which should reduce the total number of fb threads, in the hope that the extra threads were causing the chokes.

On Tue, Jul 14 2015, Keith Thorne <kthorne@ligo-la.caltech.edu> wrote:
> Assumptions
>  1) Before the upgrade (from RCG 2.6?), the DAQ had been working, reading out front-ends, writing frames trends
>  2) In upgrading to RCG 2.9, the mx start-up on the frame builder was modified to use multiple end-points
> (i.e. /etc/init.d/mx has a line like
> # 1 10G card - X2
> MX_MODULE_PARAMS="mx_max_instance=1 mx_max_endpoints=16 $MX_MODULE_PARAMS"
>  (This can be confirmed by the daqd log file with lines at the top like
> 263596
> MX has 16 maximum end-points configured
> 2 MX NICs available
> [Fri Jul 10 16:12:50 2015] ->4: set thread_stack_size=10240
> [Fri Jul 10 16:12:50 2015] new threads will be created with the stack of size 10
> 240K
>
> If this is the case, the problem may be that the additional thread on the frame-builder (one per end-point) take up so many slots on the 8-core
> frame-builder that they interrupt the frame-writing thread, thus preventing the main buffer from being emptied.  
>
> One could go back to a single end-point. This only helps keep restart of front-end A from hiccuping DAQ for front-end B.
>
> You would have to remove code on front-ends (/etc/init.d/mx_stream) that chooses endpoints. i.e.
> # find line number in rtsystab. Use that to mx_stream slot on card (0-15)
> line_num=`grep -v ^# /etc/rtsystab | grep --perl-regexp -n "^${hostname}\s" | se
> d 's/^\([0-9]*\):.*/\1/g'`
> line_off=$(expr $line_num - 1)
> epnum=$(expr $line_off % 2)
> cnum=$(expr $line_off / 2)
>
>     start-stop-daemon --start --quiet -b -m --pidfile /var/log/mx_stream0.pid --exec /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream -- -e 0 -r "$epnum" -W 0 -w 0 -s "$sys" -d x2daqdc0:$cnum -l /opt/rtcds/tst/x2/target/x2daqdc0/mx_stream_logs/$hostname.log

As per Keith's suggestion, I modified the mx startup script to only initialize a single endpoint, and I modified the mx_stream startup to point them all to endpoint 0.  I verified that indeed daqd was a single MX end-point:

MX has 1 maximum end-points configured

It didn't help.  After 5-10 minutes daqd crashes with the same "0 empty blocks" messages.

I should also mention that I'm pretty sure the start of these messages does not seem coincident with any frame writing to disk; further evidence that it's not a disk IO issue.

Keith is looking at the system now, so we if he can see anything obvious.  If not, I will start reverting to 2.5.

  11414   Tue Jul 14 17:14:23 2015 EveSummarySummary PagesFuture summary pages improvements

Here is a list of suggested improvements to the summary pages. Let me know if there's something you'd like for me to add to this list!

  • A lot of plots are missing axis labels and titles, and I often don't know what to call these labels. I could use some help with this.
  • Check the weather and vacuum tabs to make sure that we're getting the expected output. Set the axis labels accordingly. 
  • Investigate past periods of missing data on DataViewer to see if the problem was with the data requisition process, the summary page production process, or something else.
  • Based on trends in data over the past three months, set axis ranges accordingly to encapsulate the full data range.
  • Create a CDS tab to store statistics of our digital systems. We will use the CDS signals to determine when the digital system is running and when the minute trend is missing. This will allow us to exclude irrelevant parts of the data.
  • Provide duty ratio statistics for the IMC.
  • Set triggers for certain plots. For example, for channels C1:LSC-XARM OUT DQ and page 4 LIGO-T1500123–v1 C1:LSC-YARM OUT DQ to be plotted in the Arm LSC Control signals figures, C1:LSCTRX OUT DQ and C1:LSC-TRY OUT DQ must be higher than 0.5, thus acting as triggers.
  • Include some flag or other marking indicating when data is not being represented at a certain time for specific plots.
  • Maybe include some cool features like interactive plots.
  11413   Tue Jul 14 17:06:00 2015 jamieUpdateCDSrunning test on daqd, please leave undisturbed

I have reverted daqd to the previous configuration, so that it's writing frames to disk.  It's still showing instability.

  11412   Tue Jul 14 16:51:01 2015 JamieSummaryCDSCDS upgrade: problem is not disk access

I think I have now determined once and for all that the daqd problems are NOT due to disk IO contention.

I have mounted a tmpfs at /frames/tmp and have told daqd to write frames there.  The tmpfs exists entirely in RAM.  There is essentially zero IO wait for such a filesystem, so daqd should never have trouble writing out the frames.

But yet daqd continues to fail with the "0 empty blocks in the buffer" warnings.  I've been down a rabbit hole.

ELOG V3.1.3-