The interferometer is warming up!
I had some issues locking the IMC at first. It turned out that the MC3 side OSEM signal wasn't getting to the ADC. A satellite box sqush fixed it.
I touched up the PMC alignment; the best I could do is 0.75V, probably due to the AOM being in place.
I haven't touched the WFS offsets, but the current ones seem to be doing ok. I'll touch them up tonight when the seismic activity has calmed.
I made some changes to the state of the PZT/PC crossover gain in the mcdown script, resulting in the IMC catching lock quicker.
Thankfully, the tip tilt pointing stayed good during the upgrade. I barely had to touch the ETM alignment to lock the arms. ETMX is showing some errant motion, though...
I generated the following plots from the two sets of huddle test data we have for the accelerometers.
Old data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.
New data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 20 mins worth of data.
I made sure to use the same Impuse response time (6 sec) and sampling frequency (256 Hz), as well as every other parameter for the calculations.
Top left: The resultant self noise curve using the new data, there is definitely and improvement in the 0.5-2 Hz band.
Top right: Resultant self noise using the old data, for the first set of three accelerometers.
Bottom left: Old data result for the remaining three accelerometers.
Bottom right: Old data result, using all six accelerometers as witnesses instead.
A shared LIGO Data Grid (LDG) account was created for use by the 40m lab. The purpose of this account is to provide access to the LSC computer cluster resources for 40m-specific projects that may benefit from increased computational power and are not linked to any user in particular (e.g. the summary pages).
For further information, please see https://wiki-40m.ligo.caltech.edu/40mLDASaccount
The summary pages are now generated from the new 40m LDAS account. The nodus URL (https://nodus.ligo.caltech.edu:30889/detcharsummary/) is the same and there are no changes to the way the configuration files work. However, the location on LDAS has changed to https://ldas-jobs.ligo.caltech.edu/~40m/summary/ and the config files are no longer version-controlled on the LDAS side (this was redundant, as they are under VCS in nodus).
I have posted a more detailed description of the summary page workflow, as well as instructions to run your own jobs and other technical minutiae, on the wiki: https://wiki-40m.ligo.caltech.edu/DailySummaryHelp
Now that we've seemed to landed on a working configuration, I've re-ran the tests first described in ELOG 11347. I've also compared the real filtering of filter modules to their designs.
TL;DR: No adverse, or even observable, differences have been witnessed.
As a reminder: In c1tst, there are three loops, called LOOPA, LOOPB, and LOOPC.
Here are the loop delay results, which measure the slope of the phase response of the OLTF. For the purely digital loops (A, C), we know the number of cycles we expect to compare the delay to.
At this time, I haven't done the adding up of cycles, zero-order-holds, etc. to get the delay we expect from the analog loop (B), so I've just looked at whether it changed at all.
Anyways, I've attached the code that analyzes data from DTT-exported text files containing the continuous phase data from the loop measurements.
Single Model loop cycles: 1.0000000+/-0.0000006, disparity: -0.00+/-0.25 nsec
2 Model RFM loop cycles: 1.9999999+/-0.0000013, disparity: 0.0+/-0.5 nsec
ADC->DAC loop time: 338.2+/-0.4 usec
Single Model loop cycles: 0.9999999+/-0.0000008, disparity: 0.02+/-0.29 nsec
2 Model RFM loop cycles: 2.0000001+/-0.0000011, disparity: -0.0+/-0.4 nsec
ADC->DAC loop time: 338.18+/-0.35 usec
So, the digital loops take the number of cycles we expect, and there are no real differences after the upgrade.
Additionally, for all three loops, I created a simple 100:10 filter in foton, and injected broadband noise with awggui, to measure the real TF applied by the FM code. I want to turn this whole process into a single script that will switch the filter on and off, read the foton file, and compare the measured TF to the ideal shape.
In our system, before and after the upgrade, all three loops showed no appreciable difference from the designed filter shape, other than some tiny uptick in phase when approaching the nyquist frequency. This may be due to the fact that I'm comparing to the ideal analog filter, rather than what a 16kHz digital filter looks like.
What I've plotted below is the devitation from the ideal zpk(100Hz, 10Hz, 0.1) frequency response, i.e. Hmeasured / Hideal. The code to do this analysis is also attached, it estimates the TF by dividing the CSD of the filter input and output by the PSD of the input. The single worst coherence in any bin of all the measurements is 0.997, so I didn't really bother to estimate the error of the TF estimate.
The other night, I spent some time with the mode cleaner.
First, I made some model changes to the MC_TRANS part of c1mcs.mdl. Specifically, I brought in the userapps QPD part that we are using for the transmon and oplev QPDs. My main motivation for doing so was to have FMs for the pitch and yaw values, to be able to set offsets. Up until now, we have used a QPD centering servo in conjunction with the WFS, but the center of the QPD is not perfectly aligned to represent the center of MC2. Using offsets at the servo isn't really practical, since there are integrators.
I then spent some time manually aligning the mode cleaner mirrors with WFS off, followed by centering the in-lock MC REFL beam on the WFS QPDs, and setting the WFS and MC_TRANS offsets. (I updated the WFS offset script, and made one for MC_TRANS in scripts/MC/WFS. They now use averaging instead of servoing to zero, a la LSC PD offset script).
The resultant intracavity power and RIN was an improvement over the older offsets. (RMS RIN went down by half, I think.)
Since Monday, the autolocker seems to be having some trouble. At first, I suspected the changes I made weren't so hot after all, but I've now noticed a pattern. Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh.
I'm a little mystified. Peeking inside the aLIGO demod board, I saw that the reason that two of the channels weren't working was that their power connectors weren't plugged in, so no real mystery there.
I hooked up the board at the electronics bench, and found the noise to be completely well behaved, in contrast to the measurements I made when it was in the LSC rack. I've taken it back out to the LSC rack, and given it the X beatnote, and it seems to be performing pretty well.
I switched between the aLIGO demod board and beatbox during the same lock / beat. The LSC board performs margnially better from 3-100 Hz. The high frequency noise comes from the green PDH loop (coherence is near one above a few hundred Hz), so we don't expect any difference there.
To me, the beatbox noise looks like there is a broad feature that is roughly the same level as the real cavity motion in the 10-100 Hz range. So, I think we should use the aLIGO board afterall, presuming the noise doesn't shoot back up when I remount it in the rack...
The ALS noise is getting low enough where our normal approach of measuring ALS sensing noise by simply taking the PSD of the signal when the arm is PDH locked is not quite valid anymore, as it is sensing the real cavity fluctuations. Doing a frequency domain coherent subtration of the PSDs suggests a sensing noise RMS of ~150Hz for ALSX.
When the X arm is locked on ALS, POX sees about 250Hz RMS out of loop noise, which isn't the greatest; however, I used to be happy with 500Hz. By eye, sweeping through IR resonance is smoother. The real test is to get the Y arm ALS running, and swing it through PRFPMI resonance...
Fair warning, the LSC rack area is not so tidy right now, the demod board is resting on a stool (but not in the way of walking down the arm). I'll clean this up tomorrow.
Pasadena got 0.2" of rain on Saturday. Temperatures were high with high humidity since than.The ants were back in the Control room east side benches.
We have started using TERRO Liquid Ant Baits in January 2015 This worked very well to this point.
Tree new packages were opened yesterday and the ants are gone.
We can conclude that these baits must be replaced after 6 mounts.
The liquid baits contains BORAX and it is safe.
The front panels for the ALS delay line box came in last week. Some of the holes for the screws were slightly misaligned, so I filed those and everything is now put together. I just need to test both front panels to determine if the SMAs should be isolated or not.
Koji had also suggested making the holes in the front and back panel conical recesses so that flat head screws could be used and would counteract the anodization of the panel and avoid the SMAs being isolated. I think if we did that then conductivity would be ensured throughout the panel and also through the rest of the box. I also think one way we could test this before drilling conical recesses would be to test both front panels now, as one has isolated SMAs and one has conductive SMAs. If the anodization of the panel isolated the SMA regardless, we could potentially figure this out by testing both panels. But, would it also be that it is possible that the isolation of the SMA itself does not matter and so this test would tell us nothing? Is there a better way to test if the SMAs are being isolated or not? Or would this be more time consuming than just drilling conical recesses as a preventative measure?
In the last post concerning the self noise of the accelerometers, I mentioned the differences between the two data sets I was playing with. In order to give a more concrete analysis of the accelerometers self noise, we came to the conclusion that data taking time should be the same.
The plots below show the analysis for the following two datasets:
Old Data: 6 accelerometers, no cables clamped, no box, 55 mins worth of data.
Newer data: 3 accelerometers, cables clamped, foam box put on placed and completely sealed, 57.66 mins worth of data, (we had 20 mins of data in the previous data set).
Filter parameters were kept the same in all calculations, the only change that was added to the analysis was the detrending of the signals using the detrend function on Matlab, this improved the results as the plots show. I also plotted the error bars for the Wiener filtered result for reference as well as the manufactures claimed self noise.
After detrending the data and taking a longer dataset we can see the improvement brought upon by the foam box in the low frequency band of 0.5 - 10 Hz, as shown in the bottom left plot. There is a lot of noise that needs to be cancelled out from 10 Hz and on, which brings to our attention the plot on the bottom right corner. This plot uses the old data but uses all six accelerometers as witnesses, it also improved overall after having detrended the data, but what is peculiar about this plot is the fact that it manages to mitigate the higher frequency 10 - ~100 Hz band noise.
Thursday morning I found the control room emergency exit not locked.
Please check the doors when leaving the lab , specially when you are the last one out.
For the past couple of days, the summary pages have shown minute trend data disappear at 12:00 UTC (05:00 AM local time). This seems to be the case for all channels that we plot, see e.g. https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20150724/ioo/. Using Dataviewer, Koji has checked that indeed the frames seem to have disappeared from disk. The data come back at 24 UTC (5pm local). Any ideas why this might be?
As happened with the RF distribution box in the IOO rack a while back, the shiny blue power button in the LSC LO distribution box failed today. I replaced it with a simple switch, but since the original was a double throw, the replacement was way too big to fit without major panel surgery. So, instead, I installed it in the grille on the roof of the chassis. It a tight press/snap fit, though; I don't think it is at risk of easily coming loose.
After reinstalling the box, I confirmed that POX POY and AS55 could all lock arms, so I deem the operation a success.
I have moved the MC1 accelerators and cable to the huddle test set up, in order to see how a six witness huddle test with the improved set up will do.
Here is a picture of the accelerometer set up,
Our motivation for doing this is to see if more witness signals used in the Wiener filter really does indeed improve subtraction, as it was seen from previous huddle results, specially in the region above 10 Hz.
Our noise cancellation SURFS will be doing online subtraction on the mode cleaner length, among other things.
I made a measurement of the MC2 actuator transfer function by injecting noise from 1-100Hz into LSC_MC2_EXC for about 15 minutes, then estimating the TF from MC2_OUT to IOO_MC_L with CSD/PSD. The inverse of this TF will be applied to their Wiener target data to give us the direct subtration filter we want.
I figured I would post the results here for posterity. The last time this seems to have been done is in ELOG 5900. There are some differences found here, the effective Q of the 1Hz pendulum resonance seems lower, and the behavior above 20Hz has definitely changed.
IIR fits will be done by one of the SURFs to be used in their Wiener filter calculations.
The summary pages can still be found at https://nodus.ligo.caltech.edu:30889/detcharsummary/ (EDIT: in an older version of this post I listed an incorrect url). They are operational and include data from some channels for intermittent periods of time.
Motivation: to make the summary pages more informative and useful for all
What I did:
I have added tabs for ALS, ASC, and LSC subsystems. While there is currently no data on the plots, I plan on checking all channels with DataViewer to set appropriate axis ranges so that we can actually see the data.
I altered which channels are used to represent spectra for OpLev systems to more appropriately provide PSDs.
I've changed the check code status page to include "warning" labels. Previously, when the summary pages ran, resulting in a warning message, the check code status page would list this as an "error", implying that the summary pages were not properly produced.
All features were implemented, but I need to investigate some of these channels to understand why we aren't seeing many channels in the plots. I am working on some other changes to the summary pages, including providing a Locked status which will only show data in a timeseries for a selected period of time.
I've have been talking a little bit with Steve about the seismometer enclosures.
We want to improve on the current stainless steel cans that cover the two Guralps at the end of the arms. In order to do this, we want to cover the interior of the cans with copper foil to improve the thermal conductivity of the enclosure to better control the temperature inside it. Ideally, we would want to copper plate the cans, but cost and difficulty has been an issue.
I have done some rough calculations and it seems that we need a copper layer of thickness being about a third that of the stainless steel can. This happens to be around 0.5-0.6 mm since we have 16 gauge (~1.6 mm) stainless steel cans.
After wrapping the cans interior with copper, we will insulate them with foam in order to improve its thermal inertia. We want to probably use the same foam that Megan has been using for her seismometer enclosure. I have yet to think about a heater, but something similar to Megans resistor thing would work only smaller. I would be placed inside the can, right on the center of its bottom in order to ditribute heat evenly.
I downloaded new accelerometer huddle test data from last night and analyzed it. This new data set is ~55 mins and uses the same Wiener filter parameters as previous huddle test analysis, the main difference being six accelerometers used in the Wiener filter and the improved experimental set up.
After computing the ASD for the self noise for each of the six accelerometers, (being witnessed by the remaining five), we get,
Now computing the mean of the above signals and the corresponding error bars gives the following result,
Comparing to prevoius huddle tests, I can note two trends on the Wiener subtraction:
1) When using six accelerometers, the subtraction above ~8 Hz drastically improves.
2) When using three accelerometers, there is better cancellation in the 1-5 Hz region, see http://nodus.ligo.caltech.edu:8080/40m/11442. This might have to do with how much more closer the accelerometers are to each other?
Specially heavy items: old analoge scope or hardware loaded boxes......etc
The table cover section holding crossbars are not evenly spaced.
You have to center each cover section on the cross bar so it is supported on both sides !
I will clean up on this table tomorrow
We just had fire alarm trigged avacuation of the 40m lab.
It turned out that the CES building second floor sensor caused this action.
Sorry Jamie, I accidentally deleted your elog entry #11453
Koji and Steve,
The result: bad Guralp x-arm cable.
I will swap the short cables tomorrow at the base.
Short 46" long cables at the base plates were swapped. Their solderings looked horrible.
This cable actually worked at 5-5-2015
Bad cable at ETMY station now. The new cable should be a little bit longer ~52"
Koji could pull out easily 11 of the wires from their socket.
We are done taking accelerator huddle test data. So I moved back all six accelerometers and cables to MC1 and MC2. I also relabel each of the accelerometers properly since the labels on them were confusing.
Eric downloaded MC2 to MCL transfer function data (H) as well as its inverse, MCL to MC2 (Hinv). He also downloaded new MCL and MC2 data.
I used vectfit to fit the MC2 to MCL transfer function,
The ZPK parameters for this fit were,
Zeros 1278.36719876674 + 0.00000000000000i
-100.753249679343 + 0.00000000000000i
-18.6014192997845 + 13.0294910760217i
-18.6014192997845 - 13.0294910760217i
Poles -1.11035771175328 + 7.03549674098987i
-1.11035771175328 - 7.03549674098987i
-18.8655320274072 + 0.00000000000000i
-690.294337433234 + 0.00000000000000i
Using the above vectfit model, I filtered the raw MC2 signal to get 'MCL'. The PSD's of the raw MCL data and the filtered MC2 result is shown below,
The lack of accuracy of the transfer function at replicating MCL at frequencies lower than 0.7Hz is expected, the vectfit model I generated fails to follow accurately the raw transfer function data. My question: Does it matter? My guess: Probably not. In order to mitigate seismic noise from the mode cleaner we are mainly concerened with the 1-3 Hz region.
I also used vectfit to fit the transfer function for MCL to MC2,
This one was harder to fit accurately for some reason, I could do it with four pairs of zeros and poles but it took some preweighting.
The ZPK parameters for the above fit were,
Zeros 0.173068278283995 + 0.00000000000000i
0.995140531040529 + 0.0268079821980457i
0.995140531040529 - 0.0268079821980457i
0.894476816129099 + 0.00000000000000i
Poles -19.9566906920707 + 18.0649464375308i
-19.9566906920707 - 18.0649464375308i
-109.275971483008 + 0.00000000000000i
-1791.88947801703 + 0.00000000000000i
Similarly, using this ZPK model, I filtered the MCL signal to get 'MC2'. I plotted the PSD for the MC2 signal and the filtered MCL to get,
Again, the lack of accuracy of the filtered MC2 at replicating MCL below 0.7 Hz and above 12 Hz is due to the inverse transfer function failing to converge in these ranges.
The refreshed ALS didn't look so bad today (elog forthcoming), so I decided to give PRFPMI locking a shot tonight. I was able to hold the PRMI while swinging through resonsance, but transitions to RF signals failed. Demod angles / whitening gains/ etc. etc. all need to be rechecked
Some little things here and there that got cleaned up...
Koji soldered new 50" long cable for the Y station.
Please check the spectra. If something is wrong, please swap the cables between X and Y in order to see if the cable is still the issue. I believe the cable was nicely made as I carefully checked the connection twice or more during and after the soldering work.
I've switches the ALS, ASC, and LSC plots on the summary pages from plotting raw frames, to plotting minute trends, instead. Now, the plots contain information, instead of being completely blank, but data is not recorded on the plots after 12UTC.
Typically, I make changes to the summary pages on my own version of the pages, found at https://ldas-jobs.ligo.caltech.edu/~eve.chase/summary/day/, where I change the summary pages for June 30 and then import such changes into the main summary pages.
ALS is not currently limited by the demod board or whitening electronics.
The noise budget in the green locking paper shows the main noise sources to be these two, plus the residual fluctuations of the green PDH loop.
So, one next step is AUX PDH noise budget.
However, I wonder how much of the low frequency noise can be explained by instability of the beat alignement on the PSL table, and how this might be quantified.
Yesterday, I put together a few measurements to asses whether the new demod board has moved us in the right direction. Specifically I measured the output of the phase tracker in the following states, adjusting the phase tracker gain to maintain a ~2kHz UGH (but no boost on):
Results: The beat frequency spectrum is above the measured demod board and whitening chassis/ADC noise at all frequencies. It's a little close at 10Hz.
One nice feature is that the beat spectra are far more similar to each other than they used to be. RMS noise is in the 300-400Hz range, which isn't mindblowing, but not terrible. On the order of 50 pm for each arm. Most of this comes from below 10Hz.
Another thing to note is that, when we switch in the 50m cables, we should win a fair bit of Hz/V gain and push down these noises futher. (We're currently using 30m cables.)
By looking at some coherences, we can attribute some of the noise when IR locked to both colors of PDH loops.
Specifically, the coherence with the Green PDH error implicates the residual frequency noise of the AUX laser above a few hundred Hz, whereas the feature from 20-50Hz is probably real cavity motion, not ALS sensing noise. Some of the 1-3Hz noise is from real suspension/stack resonances too.
If it turns out that we do want to push the demod board noise down further, we could think about increasing the RF amplification. Driving the board harder translates directly to better noise performance. The 60Hz harmonics aren't so exciting, but not the end of the world.
Data files are attached, if you're in to that sort of thing.
Atm1, New short-50" long cable was installed at ETMY end ( Y-station ) between Guralp-B ( MIT ) and granite base.
Interface box input 2 was left connected to cable 1 and input 1 to cable 2. This plot shows no change.
Atm2, Than I swapped the two long cables at the interface box
Now the signal seems to be ok <2 Hz,
>2 Hz some problem exist.
50" short cable
I will look for more bad soldering tomorrow. How many cables did she make?
I followed my hunch, and the truth comes out.
I recalled that the aLIGO demod board has a handy DB9 output on the back panel for the detected power at the RF and LO inputs. I hooked this up into the BEATY ADC channels while checking the ALSX spectrum in IR lock.
This is assuredly the limiting factor in our ALS sensitivity.
Note: I'm calling the fluctuations of the beatnote amplitude "RF Amplitude RIN," to put things in reasonble units. I haven't looked up the board's conversion of dBm to V, but the LO should be around 0dBm in this measurement.
The coherence between the phase tracker output and the LO amplitude is significant over a broad range, mostly dipping where real cavity motion peeks up into the spectrum.
Also, the feature from 10-100Hz in the RIN spectrum is one I've often seen directly in ALS spectra when beatnote alignement is bad or the beatnote frequency is high, convincing me further that this is what's to blame.
So: what do we do? Is there anything we can do to make the green alignment more stable?
I tested both of the front panels (conductive and isolated SMAs) with the ALS Delay Line Box by driving extremely close frequencies through the cables. By doing this, we would expect that a spike would show up in the PSD if there was crosstalk between the cables.
In the plots below, for the conductive panel, the frequencies used were
X Arm: 22.329 MHz Y Arm: 22.3291 MHz
For the isolated panel, the frequencies were
X Arm: 22.294 MHz Y Arm: 22.2943 MHz
This gives a difference of 100 Hz for the conductive panel and 300 Hz for the isolated panel. Focusing on these areas of the PSD, it can be seen that in the Y Arm cable there is a very clear spike within 30 Hz of these differences when frequencies are being driven through both cables as opposed to the signal being in only the Y Arm. In the X Arms, the noise in general is higher when both cables are on, but there is no distinct spike at the expected frequencies. This indicates that some sort of crosstalk is probably happening due to the strong spikes in the Y Arm cables.
Rana pointed out that another way to mitigate seismic motion at in the mode cleaner would be to look at the YAW and PITCH output channels of the WFS sensors that control the angular alignment of the mode cleaner.
I downloaded 45 mins of data from the following two channels:
And did some quick offline Wiener filtering with no preweighting, the results are shown in the PSD's below,
I'm quite surprised at the Wiener subtraction obtained for the YAW signal, it required no preweighting and there is about an order of magnitude improvement in our region of interest, 1-3 Hz. The PIT channel didn't do so bad either.
We have to redo this cable also
I've added states to the summary pages to only show data for times at which one certain channel is above a specified threshold. So far, I've incorporated states for the IOO tab to show when the mode cleaner is locked.
You can see these changes implemented in the IOO tab of my personal summary pages for June 30: https://ldas-jobs.ligo.caltech.edu/~eve.chase/summary/day/20150630/ioo/.
I've written a description of how to add states to summary pages here: https://wiki-40m.ligo.caltech.edu/DailySummaryHelp#How_to_Define_and_Implement_States.
Previously, I had gotten the same results for the conductive and the isolated front panels. Today, I sanded off the anodized part on the back of the conductive front panel. I checked afterwards with a mulitmeter to ensure that it was indeed conductive through all the SMA connectors.
I drove a frequency of 29.359 Hz through the X Arm cable and 29.3592 Hz through the Y Arm cable, giving a difference of 200 Hz. Previously, there would only be a spike in the Y Arm at the difference, while the X Arm did not change if the Y arm was on or off. Now that the panel is fully conductive, a spike can also be seen in the X arm, indicating that crosstalk may possibly be happening with this panel, now that the spike corresponds to both the X arm and Y arm. These results are only after one set of data. Tomorrow I'll take two more sets of data with this panel and do a more in depth comparison of these results to what had been previously seen.
Notes from tonight's work:
I've explored the beatnote fluctuations a bit further.
First, I realized that we already had a channel than functions much like an RF level monitor: the phase tracker Q output. I verified that indeed, the Q signal agrees with the RF monitor signals from the demod board within the phase tracker bandwidth. This simplifies things a little.
I also found that the Y beat suffers a fair bit less from these effects; which isn't too surprising given our experience with the alignment stability.
One possible caveat to my earlier conclusions is that the beatnote amplitude could be fluctuating due to real RIN of the green light transmitted through the cavity. In fact, this effect is indeed present, but can't explain all of the coherence. If it did, we would expect the DC green PDs (ALS-TR[X/Y]) to show the same coherence profile as the RF monitors, which they don't.
The next thing I was interested was whether the noise level predicted via coherence was realistic.
To this end, I implemented a least-squares subtraction of the RF level signal from the phase tracker output. I included a quadratic term of the RF power, but this turned out to be insiginficant.
Indeed, using the right gain, it is possible to subtract some noise, reproducing nearly the same spectrum as the coherence based estimate. The discrepency at 1Hz is possible from 1Hz cavity RIN, as suggested by the presence of some coherence with TRX.
However, this is actually kind of weird. In reality, I would've expected the coupling of RF level fluctuations to be more like a bilinear coupling; changing the gain of the mixer, rather than directly introducing a linearly added noise component. Maybe I just discovered the linear part, and the bilinear coupling is the left over low frequency noise... I need to think this over a little more.
Last night around 1AM, many of the the frontend models crashed due to an ADC timeout. (But none of the IOPs, and all the c1lsc models were fine.)
[1502036.695639] c1rfm: ADC TIMEOUT 0 46281 9 46153
[1502036.945259] c1pem: ADC TIMEOUT 0 56631 55 56695
[1502036.965969] c1mcs: ADC TIMEOUT 1 56706 2 56770
[1502036.965971] c1sus: ADC TIMEOUT 1 56706 2 56770
Then, simultaneously on c1ioo, c1iscex, and c1iscey. (Wed Aug 5 01:10:53 PDT 2015)
[1509007.391124] c1ioo: ADC TIMEOUT 0 46329 57 46201
[1509007.702792] c1als: ADC TIMEOUT 1 63128 24 63192
[2448096.252002] c1scx: ADC TIMEOUT 0 46293 21 46165
[2448096.258001] c1asx: ADC TIMEOUT 0 46669 13 46541
[1674945.583003] c1scy: ADC TIMEOUT 0 46297 25 46169
[1674945.685002] c1tst: ADC TIMEOUT 0 52993 1 52865
I'm still working on getting things back up and running. Just restarting models wasn't working, so I'm trying some soft reboots...
UPDATE: A soft reboot of all frontends seems to have worked,
I've fixed the ASC tab on the summary pages to populate the graphs with data without causing an error.
Motivation: The ASC tab was showing no data. It resulted in a name error when generated.
A name error indicates a bad channel name in the plot definition. I identified two errors in the code:
The plots are not processing without error. However, no titles or axis labels are present on the plots- I'll work on adding these.
Since Chiara's onboard ethernet card has a reputation to be flaky in Linux, Koji suggested we could just buy a new ethernet card and throw it in there, since they're cheap.
I've installed a Intel EXPI9301CT ethernet card in Chiara, which detected it without problems. I changed over the network settings in /etc/networking/interfaces to use eth1 instead of eth0, restarted nfs and bind9, and everything looked fine.
Sadly, EPICS/network slowdowns are still happening. :(
Tonight, I've taken a bunch of data where the PRC is carrier locked and the ITM oplevs have the DC coupling FM turned on, as we use during locking. This is to inform new feedforward filters to stabilize the PRC angular motion, by using Wiener filtering with the POP QPD as the target, and local seismometers/accelerometers as witnesses. So far I've looked at the 1800 seconds leading up to GPS time 1122885600, but there has been plenty of locked time tonight if I need to retrieve more.
I've also measured the PRM ASC output torque -> POP QPD spot motion with high (>0.95) coherence from 0.1Hz to 10Hz.
Prefiltering so far consists of a 4th order elliptic LP at 5 Hz, with the target subtraction band being the 1-3Hz range.
With offline FIR filtering, the RMS pitch motion is reduced by a factor of 3 just with the STS1_X data. IIR fitting remains to be done.
The PRC yaw motion, which is marginally noisier, is a little more tangled up across X and Y.
Plots / filters forthcoming pending more analysis.
Koji had suggested that I sync up the two function generators to ensure that they have the same base frequency and so that crosstalk will actually appear at the expected frequency. After syncing up the two function generators, I drove the following frequencies through each cable:
X: 29.537 MHz Y: 29.5372 MHz
X: 29.545 MHz Y: 29.5452 MHz
Each time, the difference between the frequencies was 200 Hz, so if there was crosstalk, a spike should appear in the PSDs at 200 Hz when frequencies are being driven through both cables simulataneously, but not when just one is on. We very clearly see a spike at 200 Hz in both the X arm and the Y arm with the conductive SMAs, indicating crosstalk. For the front panel with isolated SMAs, we see a spike at 200 Hz when both frequencies are on, but it is much less pronounced than with the conductive SMAs. It seems as though there will be crosstalk using either panel, just less with the isolated SMAs.
Often when I come to manually lock the mode cleaner due to a long unlocked period, I find that the sliders are not in the state specified by the mcdown script. Furthermore, it's not the same channels every time; sometimes the servo gain is left high, sometimes the boosts are left on. I fear that some of the caput commands are failing to execute. Ugh.
This continues to happen. I believe the network latency boogeyman is to blame.
There was a long unlocked period because the enable switch for the MC servo fast path (FASTSW) was left off. Running the mcdown script fixed this, but included the error message:
Channel connect timed out: 'C1:IOO-MC_REFL_GAIN' not found.
CA Client Library: Ignored duplicate create channel response from CA server?
which means the IN1 gain didn't get touched. A second pass of the script produced no errors.
I'm thinking of adding some logic that if the autolocker has failed to lock for some period (5 minutes?), it should rerun mcdown.
The wasp nest will be removed tomorrow from from the out side of the east arm window.
The resonant frequency of the newly arrived gravity bee detector is not known.
I'm attaching a SISO IIR Wiener filter here for reference purposes that will go online either tonight or tomorrow evening. This is a first test to convince myself that I can get this to work, MISO IIR filters are close to being ready and will soon be employed.
This Wiener filter uses the STS-X channel as a witness and MCL as target. The bode plot for the filter is shown below,
The performance of the FIR and IIR Wiener filters and the ammount of subtraction achive for MCL is shown below,
Output from quack to be loaded with foton: filter.zip