Working in Xend with mask on has become unbearable. It is very hot there and I would really like if we fix this issue.
Today, the Xend Green laser was just unable to hold lock for longer than 10's of seconds. The longest I could see it hold lock was for about 2 minutes. I couldn't find anything obviously wrong with it. Attached are noise spectrums of error and control points. The control point spectrum shows good matching with typical free running laser noise.
Are the few peaks above 10 kHz in error point spectrum worrysome? I need to think more about it in a cooler place to make sure.
I wanted to take a high frequency spectrum of error point to make sure that higher harmonics of 250 kHz modulation frequency are not leaking into the PDH box after demodulation. However, the lock could not be maintained long enough to take this final measurement. I'll try again tomorrow morning. It is generally cooler in the mornings.
This post is just an update on what's happening. I need to work more to get some meaningful inferences about this loop.
Attachment 1 shows the case when excitation is sent at control point i.e. the PZT output. As before, free running laser noise in units of Hz/rtHz is added after the actuator and I've also shown shot noise being added just before the detector.
Again, we have a access to three output points for measurement. right at the output of mixer (the PDH error signal), the feedback signal to be applied by uPDH box (PZT Mon) and the output of the summing box SR560.
Doing loop algebra as before, we get:
So measurement of can be done by
This means at 100 Hz, with 10 integration cycles, we should have needed only 3 mV of excitation signal to get an SNR of 100. However, we have been unable to get good measurements with even 25 mV of excitation. We tried increasing the cycles, that did not work either.
This post is to summarize this analysis. We need more tests to get any conclusions.
We did the measurement of OLTF for Xend green laser PDH loop with excitation added at control point using a SR560 as shown in attachment 1 of 16202. We also measured coherence in our measurement, see attachment 1.
Here I describe efforts to cancel the AUX laser PZT mechanical resonances from ~200 kHz-400kHz. While these may not be the resonances we end up wanting to suppress, I chose this region as an exercise because it contains the most significant peaks.
The PZT transfer measurement was taken on 09/06 by myself and Anchal. The Moku:Go outputted a swept-sine (1kHz - 1MHz) I sent to the AUX laser PZT. The beat note between the AUX and frequency-doubled PSL was sent to the DFD, and the I and Q channels were routed back as input to the Moku:Go. We also took a calibration transfer function of the Moku:Go, sending output 1 to inputs 1 and 2.
Almost all of the signal was present in the I channel, so I proceeded to use the I data for fitting/next steps. After normalizing the measured frequency response by the calibration measurement (and adjusting for the calculated time delays in the loop - see ), I fit the resulting data using vectfit [Attachment 1]. I supplied the function with n_poles=16, which in reality fit for 16 complex pairs of poles. This complexity of fit was not necessary to capture the 3 prominent peaks, but would likely be needed to fit any of the more heavily-damped resonances.
I chose to invert all fitted poles between 200 kHz and 367 kHz and the corresponding fitted zeros. The result of this filter applied to the original frequency response data can be seen in Attachment 2, where the blue-shaded region contains the inverted poles/zeros. In total, 9 pairs of poles and 9 pairs of zeros were inverted.
This looks really good to me. Rather than fully invert the plant, what we would like to do is now design a filter which allow this loop to have a high UGF and a high gain below 1 kHz. Anchal and Paco probably have gain requirements for this loop in the ALS-CAL paper they are writing. The loop would have the cavity transfer function, as well as the demod electronics for the green PDH loop.
In addition to the gain requirements, we would also like to have a phase margin > 30 deg, and a gain margin of > 10 dB.
There was some uncertainty as to which channels were being input into the Adaptive Filtering screen, so I checked it out to confirm. As expected, the rows on the ASS_TOP_PEM screen directly correspond to the BNC inputs on the PEM_ADCU board in the 1Y6 (I think it's 6...) rack. So C1:ASS-TOP_PEM_1_INMON corresponds to the first BNC (#1) on the ADCU, etc.
After checking this out, I put text tags next to all the inputs on the ASS_TOP_PEM screen for all of the seismometers (which had not been there previously). Now it's nice and easy to select which witness channels you want to use for the adaptation.
When Sanjit and I were looking at the adaptive filtering system on Monday and Friday, we noticed that turning on the Accelerometers (which had been used in the past) seemed to do good things, but that turning on the seismometers (which I just put into the system last week) made the OAF output integrate up. Rana pointed out that this is an indication of a missing high pass filter. And indeed, when I put the seismometers in, I neglected to copy the high pass filter at low frequencies, and the low pass at 64Hz from the accelerometer path to the seismometer path. The accelerometers had a HP at 1Hz, which is okay since they don't really do useful things down to the mHz level. I gave all of the seismometers HP at 1mHz. These are now in the filter banks in the ASS_TOP_PEM screen. The accelerometers are on channels 15, 16, 17, 18, 19, 20 and the seismometers are on channels 2, 3, 4, 10, 11, 12, 24.
I now need to modify the upass script to turn these filters on before doing adaptive filtering.
It seems now that we are able to get the OAF system to do a pretty good job of approximating the MC_L signal, but we can't get it to actually do any subtracting. I think that we're not correctly setting the phase delay between the witness and the MC_L channels or something (I'm not sure though why we get a good filter match if the delay is set incorrectly, but we do get a good filter match for very different delay settings: 1, 5, 100, 1000 all seem to do equally well at adjusting the filter to match MC_L).
The Matt Evans document in elog 395 suggests measuring the phase at the Nyquist frequency, and calculating the appropriate delay from that. The sticking point with this is that we can't get test points for any channel which starts with C1:ASS. I've emailed Alex to see what he can do about this. Elog 1982 has a few words about how we're perhaps using a different awgtpman on the ass machine than we used to, which may be part of the problem.
The golden plan, which in my head will work perfectly, is as follows: Alex will fix the testpoint problem, then Sanjit and I will be able to measure the phase between our OAF signal and the incoming MC_L signal, we will be able to match them as prescribed in the Matt Evans document, and then suddenly the Adaptive Filtering system will do some actual subtracting!
The plot below shows the Reference MC_L without any OAF system (black), the output of the OAF (green), and the 'reduced' MC_L (red). As you can see, the green trace is doing a pretty good job of matching the black one, but the red trace isn't getting reduced at all.
The OAF system did something useful today! Attached is a plot. Black is the reference (13 averages) with the OAF off. Blue is the output of the OAF, and red is the reduced MC_L signal (13 averages). If you turn tau and mu both to 0, it "pauses" the filter, but keeps the feedforward system working, so that you can take a long average to get a better idea of how well things are working. If you ramp down the output of the CORR filter bank, that lets you take a long average with the OAF "off", but doesn't mess up your nicely adapted filter. The cyan and gold traces in the upper plot are 2 of the Guralp channels, so you can see the real seismic motion.
In the lower plot, you can see that the cyan and light green seismic channels have good coherence with IOO-MC_L (the names don't really mean anything right now...these 2 seismometer channels are the 2 Guralps' channels, one per end of the MC, which are aligned with the MC.) The dark blue trace is the coherence between IOO-MC_L and the output of the OAF.
500 taps, delay=5, 2 Guralp channels (the ones aligned with the MC), tau~0.00001 (probably), and mu~0.01 or 0.005
The up and down scripts accessible from the OAF (still C1:ASS-TOP) screen are now totally functional and awesome. They are under the blue ! button. The up script can either be for the Seismometers, or the Accelerometers at this time. The only difference between these 2 is which burt restore file they look at: the seismometer version puts all 7 seismometer channels in the PEM selecting matrix, while the accelerometer version puts the 6 accelerometer channels in that matrix. Both scripts also turn on HP_1mHz filters in the ERR_EMPH filter bank and all of the witness filter banks, and the AA32 and AI32 filters in ERR_EMPH, CORR and PEM filter banks. This makes all of the starting filters the same between the witness paths and the error path.
If you want to use a different combination of sensors, run one of the up scripts, then change the PEM matrix by hand.
The down script disables the output to the optics, and resets the adapted filter coefficients. DO NOT use this script if you're trying to "pause" the filter to take some nice long averages.
As per Matt's instructions in his OAF document (elog 395) in the Tuning section, Sanjit and I took a transfer function measurement from the output of the OAF system, to the input. i.e. we're trying to measure what happens out in the real world when we push on MC1, and how that is fed back to the input of our filter as MC_L. The game plan is to measure this transfer function, and read off the phase at the nyquist frequency, and use this value to calculate the appropriate sample-and-hold delay to be used in the OAF. The downsample rate for the OAF is 32, so that we're running at 64Hz instead of the 2048Hz of the front-end. Thus, our Nyquist frequency is 32Hz.
Phase@Nyquist * ------------------------
In the attached figure we do a swept sine from CORR_EXC to ERR_EMPH_OUT to determine the transfer function. Here, we turn off all of the filters in both the CORR and EXC banks, because those are already matched/taken into account in the PEM filter banks.
Using the cursor on DTT, we find that the phase at 29.85Hz is -228.8deg, and at 37.06Hz is -246.0deg. Extrapolating, this means that at 32Hz, we expect about -234deg phase. Using our handy-dandy formula, this means that we should try a delay of 41 or 42 (41.6 is between these two...)
We'll give this a shot!
Phase@Nyquist * ------------------------ = Delay
As Rana pointed out to me last night, I was using continuous phase, which is not good. When using regular phase, I find: (29.85Hz, 131.216deg), (37.06Hz, 113.963deg), so extrapolating gives (32Hz, 126.07deg). Plugging this in to our handy-dandy formula, we get a delay of 22.4, so we should try both 22 and 23.
Here's a plot of the spectra of the seismometers and MCL. The coherence shows which axes are aligned right now: MC1_X is coherent with GUR_NS which means that its mis-oriented.
I've now swapped the "MC1" cables: so the old "NS" now goes into EW and the old EW now goes into NS. VERT is unchanged.
Also fixed the channel names - the Guralp previously named MC1 is now GUR1 and the other one is GUR2. Also no more EW, NS, & VERT. Its all XYZ.
DAQD restarted with the new channel names.
I remeasured the OAF time delay using the OAF-TF template from the Templates/ directory.
Troublingly, I found the MC1 dewhitening switches set OFF - please make sure that the MC1 dewhitening is back ON after each OAF tuning so that the interferometer locking is not hosed.
The OAF-TF template had the excitation amplitude set ~20x too high. I reduced it and the coherence was still > 0.95. The phase at 32 Hz was still ~126 deg as Jenne had measured, but since the phase at DC is 180 deg, the overall phase lag is just 180-126 = 54 deg. So the delay should be 54/180 * 32 = 9.7 => 10. Luckily, Jenne is working on an instructional manual for OAF that will make all of this crystal clear.
I spiffed up the order of the cables / sensors plugged into the PEM ADCU. Now all of the seismometers are labeled as Rana left them, and the 2 Guralp's have their sets of 3 channels next to eachother in channel-number-land. None of the accelerometer names/cabling have changed recently. In the table, Cable-label refers to the physical tag tied to the end of the cables plugged into the ADCU...they are meant to be descriptive of what seismometer channels they are hooked up to, and then the names change to something useful for us when they come into the DAQ system. Also, the labels of input channels on the ASS_TOP_PEM screen have been updated accordingly.
There is some craziness going on with the delay in the PEM path for the OAF. We plot the difference between the C1:PEM-SEIS_GUR1_X and C1:ASS-TOP_PEM_10. These are physically the same channel, plugged into the PEM ADCU, and then the signal is used as a regular PEM channel, and is also sent to the ASS computer and used there for the OAF system. As you can see in the blue trace on the bottom plot, there is a huge amount of delay, and it's very noisy. We also plot the _GUR2_X / ASS-TOP_PEM_2 pair (red), and it has a similar amount of delay, but it is not nearly as fuzzy and noisy. For comparison, we plot the SUS-MC2_MCL (which is identical to IOO-MC_L) and ASS-TOP_ERR_MCL pair (green), and they don't have any big overall delay problems, so it's not totally a problem with the signals getting to the ASS computer.
This problem was present during/after all of the following attempts to fix it:
* The sample rate on the ASS computer is 2048. The PEM channels were being acquired the ADCU at 512. We changed the ADCU sampling rate to 2048 to match.
* We soft rebooted the ASS computer, in case it was a timing problem.
* Doing a "sudo shutdown -r now" while logged in as controls.
We might also try resetting/power cycling c0dcu in the morning. Alex has been emailed to help us try to figure this out.
In other news, the time delay that we measure from the plot gives us 180degrees in ~210Hz. This corresponds to a little more than 2msec of delay, with the C1:ASS version lagging behind the C1:PEM version. (2 samples at 840Hz) Converting to the 2048 sampling rate, we have a delay of 4.8, so 5 front-end cycles. Since Rana measured this morning that the delay indicated by the transfer function is 10 cycles, and this delay shows that the ASS lags the actual seismometer signal by 5 cycles, we should subtract this 5 from the 10 from the transfer function, giving us a final sample-and-hold delay of 5. Coincidentally(?), 5 is the delay that was found in the C1:ASS-TOP screen, after it's one year of dormancy. The point of the delay feature in the code is to help match the delay in the two signal paths: the PEM path and the output path of the filter. Since the output has a lag of 10, and the PEM path has a lag of 5, to make them match, we artificially put in a delay of 5.
The c1ass computer, which is now used for the OAF system, has many remnants from the days when it was actually used as an ASS. These PIT and YAW filter banks and other things were taking up a lot of unnecessary space, so I deleted them in the ass.mdl file. These files are all backed up, so we can always revert back to an older version when we want some Alignment Stabilization again someday. I then did a make ass, following the instructions on the 40m Wiki -> Computers and Scripts -> Simulink to Front-End Code page. Rana moved some things around, most notably all of the things (like the ASS screens) which were only in ...../users/alex/.... are now in ....../caltech/cds/advLigo/..... . This required a few restarts of the c1ass machine (after a couple different versions of the simulink diagram....one to make sure we knew how to do it, and then again actually deleting the unused portions).
The big lesson of the night was that there are 2 signal paths for the PEM channels. As is shown in Figure 3 in the mevans document, the PEM channels get the matching filters when they go to the adaptation algorithm, but when they go to the FIR filter, they do not get the matching filters. This is implemented by taking the output of the giant PEM matrix, and having a duplicate of each of the channels "selected for adaptation", one which gets filtered through the PEM_N_ADPT banks, and one which goes straight (in code-land) to the FIR filter. So, it seems like all the filters which we had been including in the input side of the matrix for matching purposes need to be put in the output side. One of the AA32 filters needs to stay in the input side, for actual anti imaging of the PEM channels, then we put the AA32 and AI32 which are for matching the ERR_EMPH and CORR filter banks up in the PEM_N_ADAPT banks. Rana and I made these filters, and they are now turned on appropriately with the OAF down script (so that all the filters are ready and waiting for the OAF to be turned on).
A little success with getting the 3Hz peak reduced, but not a lot beyond that. Tomorrow I'll put the accelerometers back where they used to be to see if they help out at all.
Alex came in a week ago Friday to help figure this timing problem out, and some progress was made, although there's more to be done.
Here are the (meager) notes that I took while he was working:
we can rename the tpchn_C1_new back to tpchn_C1, but the _new one works right now, so why change it?
need to find dcuDma.c source code...this is (?) what sends the PEM channels over to ASS. Found: source code is dcu.c, th
en the binary is dcuDma.o Trying to recompile/remake dcuDma to make everything (maybe) good again.
Possibility: maybe having so many channels written to the RFM takes too long? shouldn't be a problem, but maybe it is. I
n the startup.cmd (or similar?) change the number of ISC modules to 1, instead of 2, since we only have one physical board
to plug BNCs into, even though we have 2 isc boards. c0dcu1 rebooted fine with the one isc board. now can't get ass tes
tpoints to try the DTT timing measurement again. rebooting fb40m to see if that helps. fb40m is back, but we still don't
have ASS testpoints. Alex had to leave suddenly, so maybe more later.
Also, next possibility is that c0dcu and c1ass are not synched together properly....we should look at the timing of the AS
After these adventures, the noisy trace in the timing delay (in the plot in elog 2066) has become quiet, as shown below (The blue trace, which was noisy in 2066 is now hiding behind the red trace). However, the overall timing delay problem still exists, and we don't quite understand it. Alex and I are meeting tomorrow morning at the 40m to try and suss this out. Our first plan of attack is to look at the ASS code, to see if it puts any weird delays in.
Rana pointed out that the delay may be caused by the 110B DAQ, as it integrates over 2ms (5 clock cycles at 2048Hz on the fe computer), to make low noise measurement. However, the C0DCU knows about this delay and corrects it by fudging the time stamp, before sending it to the frame builder, so that the time stamps match the actual measurement time. But, the ASS computer is not aware of such an integration time, so it does not adjust the time. We verified that it is indeed the case. This is what we did (as suggested by Rana):
We split the signal from the MODE cleaner board "OUT" port using a T-splitter to the original PENTEK board (C1:SUS-MC2-MCL-IN) and the PEM ADCU channel #2. Then measured the mutual delays between the signals that are processed by C0DCU and the ASS computer for both the MC_L signal and the corresponding output through the PEM channel. We clearly see the same delay (compare red and brown in the bottom panel) between the signals that are going through 110B and the PENTEK DAQ. This delay is a bit noisy, possibly because the PENTEK is not as low noise as the 110B is.
There is some delay (pink curve in the bottom panel) between the PENTEK DAQ and the frame builder corrected 110B output, much smaller than 2ms, could be ~200-400 u sec. Which should correspond to the 1 or 1/2 cycle delay caused by the PENTEK DAQ.
So, once we have the planned advLIGO DAQ system, there should not be any long delay. Perhaps, to solve the problem and make OAF functional soon, we will upgrade the PEM DAQ asap, rather than waiting for the rest of the upgrades...
An email from Rolf about the delay in the 110Bs:
"...we do take the ~2msec pipeline delay into account when we send the data to DAQ. If I remember correctly, the delay is about 39 samples. On startup, the first 39 samples are 'thrown away', such that, from then on, data lines up with the correct time (just read 2msec later then Penteks)."
[Alex, Jenne, Sanjit]
Alex came to the 40m today, and did several awesome things in OAF-land.
We discovered that there is, in fact, an ADC board connected to the ASS machine. The tricky bit is that it only has a ribbon cable connector, so before we can use this ADC, we need to figure out how to make a breakout board/cable/something to connect the seismometer/accelerometer/microphone BNCs to this little board. This is the same little board that connects the timing slave to the ASS machine. For good or for ill, the timing slave is connected to this board via clip-doodles. Potentially we can connect an ADC tester board to this board, and go from seismometer BNCs to clipdoodles to the tester board, but I'm not in love with the idea of utilizing clipdoodles as a semi-permanent solution until the upgrade. I emailed Ben to see if he has a better idea, or (better yet) some spare hardware now that's the same as we'll use after the upgrade. If we can use this ADC, it may solve our timing problem which is caused by the 110B ADC used by the PEM computer. Alex showed Sanjit and I how to connect the ASS's ADC card to the simulink diagram, when we're ready for that.
We also poked around in the code, and it seems that we can now save and restore OAF coefficients at will. I added buttons to the OAF (ASS) screen, and Alex made it so the OAF coefficients are saved in RFM shared memory whenever you click the "save coeffs" button, and are restored when you click the "restore coeffs" button. The buttons are the same as the 'Reset' button which has been there for a long time, so they seem to maybe have a similar problem in that you have to hold the button for a while in order for the code to realize that the button has been depressed. We couldn't fix this easily, because it looks like our SimuLink cds stuff is a little out of date. Some day (before/when Joe and Peter make new screens for the new 40m), we need to update these things. Alex was concerned that it might take a while to do this, if the update broke some of the blocks that we're currently using. Also, Sanjit and I now need to check that the coefficient-saving is going as planned. When I have DTT open, and the OAF running, I see a certain shape to the signal which is sent to MC1 to correct for the seismic motion. This shape includes at least several peaks at resonant frequencies that exist in our stacks/suspensions. I can then save the coefficients, reset the active filter, and then restore the coefficients. When I do this while watching DTT, it seems as though the general shape of the filter is restored, but none of the detailed features are. The reason for this is still under investigation.
The code-modifications involved a few iterations of 'remaking the ass'.
Sanjit has been working today on trying to get the OAF coefficients to save properly. Alex got us most of the way, but right now it's looking like the filter that is being saved is totally constant (all the values are the same). We're poking around trying to figure out why this is.
Also, we're starting again (as we should have been for the last week or so since Alex came in to help us) to check in the TOP_XFCODE whenever we make changes to it, and when we recompile the front end code.
We are manually restarting assepics, but the terminal logs us out after sometime and ass may crash. I set autologout=0 in the terminal for the time being. Once the testing process is over, assepics will start automatically when the computer is turned on, so we wont have to worry about this.
(if ass crashes tonight, it is not unexpected!)
I made some changes in the code (all commented in the installed and SVN version) to print the filter coefficients. I got crazy output. Sometimes memory bugs lead to such crazy behavior. So far I could not find any bugs, but will have to spend more time on it.
I made some changes in the code (all commented in the installed and SVN version) to print the filter coefficients. I got crazy output. Sometimes memory bugs lead to such crazy behavior. So far I could not find any bugs, but will have to spend more time on it
Something strange was going on in the OAF code, printf would print a double precision number in %f format but not in %lf or %e format!
Since we know this problem now, we can move forward, but it will be important to know why printf was restricted and if there are other such constraints which we should remember while making changes in the codes.
Mostly a note to self: I have put terminators on the ADC inputs which are usually the PEM-SEIS-GUR2_(XYZ) channels. Since these 3 signals are currently going into the ASS ADC, these PEM ADC inputs are open, and have predefined channel names. I'll collect the data and put it as the ADC noise level in my nifty plot which will show the noise limits of all things which affect Wiener Filtering.
Sanjit and I are trying to put names to some signals which exist in SimuLink land, but which don't (yet) exist in EPICS land. The deelio is that for each of the chosen SEIS signals in the ASS_TOP_PEM screen, the signal is split. One part of the signal is used to decide how the adaptive filter should look, and the other part is actually used when doing the on-line subtraction. Previously only the part of the signal which is used to decide on the Adaptive Filter could be seen on the screens, and had names.
Before touching anything on the Simulink ASS.mdl, I did an svn check in, which put things at revision 36639.
To try to make the desired signals exist, I put cdsFilt boxes (to create filter modules for each of these signals), and gave each of them a name (kind of like the Neverending Story....once they have a name, they'll exist). My new names are C1:ASS-TOP_PEM_#_APPLY, which correspond to the previously-existing C1:ASS-TOP_PEM_#_ADPT (these are the ones that are along the top of the ASS_TOP_PEM matrix screen). This version of the simulink model was checked in, and the svn is now at revision 36640.
We then did some "make clean", "make ass" and "make install-ass" action, and burt restored c1assepics, but nothing seems to be happening. The screen doesn't have white boxes all over the place, and we didn't get any errors when we did the makes, and I'm sure we burt restored correctly (made sure the ASS GDS screen had a 1 in the lower left box etc), but all the values on the screen are still zero.
When we ran the ass front end in terminal on the c1ass machine, we did see an error: "Invalid chan num found 2 = 30624" "DAQ init failed -- exiting". I think this means that we need to have told some file somewhere that I was going to be adding 8 new channels. (maybe an .ini file?) Hopefully the Joe & Peter team can help us out with this, since they've been doing this kind of thing for the new system.
Moral of the story is, the new (non-working) simulink file has been svn checked in as revision 36640, and we're reverting to revision 36639, which was before I touched anything today.
We would like several channels from the OAF/ASS screen to be saved to frames, so that we can use the channels for our OAF model. In theory, this should involve uncommenting the desired channels in the .ini file (.../caltech/chans/daq/C1ASS.ini), and restart the frame builder. Since this .ini file was generated a long time ago, and things have been changed since then, the chnnums in the .ini file and the corresponding .par file don't match up. We need to go through the .par file (/cvs/cds/gds/param/tpchn_c3.par), and look up the chnnums for our channels, and copy those numbers into the .ini file. Figuring out what was going on involved many fb40m restarts, but on the last one of the night, I restarted the backup script, so it should (hopefully) run tonight, and get all of the frames that we've been missing.
Notes to self:
* When adding channels to other front ends, the end of the process is to click the blue button on the C0DAQ_DETAIL screen next to your computer. C1ASS isn't on that screen. Instead, in the C1ASS_GDS screen, click DAQ Reload.
* The channel names for the Test Points and the .ini files must be different. That's why there's a '_2048' suffix at the end of every channel in our .ini file.
* tpchn_C1 is all of the old-style system test points. tpchn_C2 is the C1OMC, and tpchn_C3 is for the C1ASS testpoints.
* When uncommenting channels in the C1ASS.ini file, make sure acquire is set to 1 for every channel we want saved. The default in this .ini file is set to acquire = 0.
[Sanjit, Jenne, Rob, Joe]
We added and tested the following channels from "/cvs/cds/gds/param/tpchn_C3.par" to "/cvs/cds/caltech/chans/daq/C1ASS.ini" appending a "_2048" extension to the channel name (as the name of a channel in .ini and .par files must be different):
These five-line entries for each channels in the .par file were manually copy pasted from the .ini file, should think about a smarter way...
The old .par file is kept as: /cvs/cds/caltech/chans/daq/C1ASS.ini.20Nov2009
The current one is also saved as: /cvs/cds/caltech/chans/daq/C1ASS.ini.24Nov2009
And, the current one is committed to the svn.
NOTE: In the first attempt, the channel names were mistakenly kept the same in both the .ini and .par files and this caused DAQ daemon to crash badly. It could only be recovered by hard reboot of the frame builder. Important info here: Jenne's elog 2316
Sometimes back we modified /cvs/cds/caltech/chans/daq/C1ASS.ini to save some of the channels. The file was reverted to default after the recent changes in ASS.
We again uncommented and made acquire=1 to save the following three channels using daqconfig:
The script automatically created a back up in /cvs/cds/caltech/chans/daq/archive
OAF is successfully canceling noise again, thanks to Matt!
Here is a plot showing more than a factor of 10 noise reduction around 3Hz (similar to what we saw in the simulations)
The changes that has made it work are:
Matt suggested playing with the emphasis (EMPH) filters to cancel noise in different frequency bands.
We turned on the OAF again to make sure it works. We got it to work well with the Ranger as well as the Guralp channels. The previous problem with the ACC is that Sanjit and Matt were using the "X" channels which are aligned the "Y" arm. Another casualty of our ridiculous and nonsensical coordinate system. Long live the Right Hand Rule!!
The changes that were made are:
Other parameters which were kept at usual setting:
Screenshots are attached.
Burt snapshot is kept as: /cvs/cds/caltech/scripts/OAF/snaps/ass_burt_100126_211330.snap
taken using the script: /cvs/cds/caltech/scripts/OAF/saveOAF
we should put this in ASS screen.
ERROR Detected in filter ASS_TOP_PEM_24 (RANGER): 1: was actually typed as a 1Hz high pass filter!
(Correcting this one seems to spoil the adaptation)
Possibly this makes sense, we may not want to block witness signals in the 0.1-20 Hz range.
11:40 PM: Leaving the lab with the OAF running on 5 PEM channels (Ranger + Guralp 1&2 Y & Z). There's a terminal open on op440m which will disable the OAF in ~2.8 hours. Feel free to disable sooner if you need the MC/IFO.
I tried downsampling value 32 (instead of 16), to see if it has any effect on OAF. Last week I encountered some stability issue - adaptation started to work, but the mode cleaner was suddenly unlocked, it could be due to some other effect too.
One point to note is that different downsampling did not have any effect on the CPU meter (I tried clicking the "RESET" button few times, but no change).
I tried some combination of PEM channels and filters to improve OAF performance at other frequencies, where we do not have any improvement so far. There is progress, but still no success.
Here are the main things I tried:
For the ACC channels replaced the 0.1 Hz high pass filters by 3Hz high pass and turned off the 1: filter.
Then I tried to incorporate the Z ACC/GUR channels, with some reasonable combination of the others.
The Z axis Guralp and Accelerometers were making OAF unstable, so I put a 0.1 gain in all four of those.
Following the PEM noise curves Rana has put up, we should probably use
In the end I tried this combination, it was stable after I reduced the GUR_Z gain, but looked very similar to what we got before, no improvement at 5Hz or 0.5Hz. But there was a stable hint of better performance at > 40Hz.
Possibly we need to increase the GUR_Z gain (but not 1) and try to use ACC_Z channels also. Since we can not handle many channels, possibly using one GUR_Z and one ACC_Z would be worth checking.
At 0.1-1.0Hz, there is some coherence between MC_L and RANGER_Y & GUR_Y, see the first figure. Also GUR_Z has low noise there. So I used all five of them, increased the gains of GUR_Z from 0.1 to 0.5. Some improvement near 0.5Hz. We possibly can not do any better with these PEM measurement, as the coherence of the adapted error signal and the PEM channels is almost zero, see the second figure. May be we need to think about placing the seismometers at different places/orientations.
However, there is lot more scope at higher frequencies, lot of coherence at 5-100Hz.
There is lot of coherence between the error signal and PEM channels at 5-100Hz. We had been applying a 1Hz low pass filter to all the GUR and RANGER channels for stability. I turned those off and OAF still works with mu=0.0025, this will give us some more freedom. Kind of annoying for testing though, it takes about 45min to adapt!
In any case, there is no significant improvement at high frequencies as compared to our usual OAF performance. Also, the low frequency improvement (see previous e-log) is lost in this set up. I think, we have to adjust the number of taps and channels to do better at high frequencies. Also, delay can be important at these frequencies, needs some testing.
Did some more test to get better performance at higher frequencies.
Increased # taps to 4000 and reduced downsampling to 4, without changing the AA32 filters, from CORR, EMPH and the matching ADPT channels. But for testing I turned off AA32 from the input PEM channels. So that high frequency still gets blocked at CORR, but the adaptive filters have access to higher frequencies. Once we fix some reasonable downsampling, we should create corresponding AA filters.
I used only two channels, RANGER/GUR2_Y and GUR1_Z, and basically they had only one filter 0.1:0
This set up gave little better performance (more reduction at more frequencies), at some point even the 16HZ peak was reduced by a factor of 3. The 24Hz peak was a bit unstable, but became stable after I removed the Notch24 filters from PEM channels, to ensure that OAF is aware of those lines. There was some improvement also at the 24Hz peak.
I started on the 16th with a very intense lab tour & was fed with a large amount of data (I can't guarantee that I remember everything....)
Then... did some (not much) reading on filters since I'm dealing with seismic noise cancellation this summer with Jenne at the 40m lab.
I'll be using the Streckeisen STS-2 seismometers & I need to use the anti aliasing filter board that has the 4 pin lemo connectors with the seismometers & its boxes that require BNC connectors. I spent most of the time trying to solder the wires properly into the connectors. I was very slow in this as this is the first time I'm soldering anything.... & till now I've soldered 59 wires in the BNC connectors....
[Jenne, Mirko, with supervision from Jamie]
We are starting to create the new OAF model, so that it works with the new CDS system.
I created (and did an "svn add" for) a new c1oaf.mdl, in the same place as the current c1lsc.mdl . Since the oaf will kind of be working with ISC things, I decided to put it in that folder. So far this new OAF model just has SHMEM/PCIE memory sharing things to get info from the LSC and PEM models. The OAF model has dcuid=22 (the same as in the old system), and lives on the LSC machine with specific_cpu=4. This is the CPU number that Yoichi was going to use, but he ended up putting his FF stuff directly into the LSC model for delay reasons.
I modified the c1rfm.mdl to take seismometer and accelerometer info from the PEM model, and give it to the "rfm" via shmem, and then using PCIE (dolphin) to get the channel to the OAF model.
I modified the c1lsc model to have shmem outputs that go from the degrees of freedom to the OAF, and shmem inputs from the OAF's output to sum into the DoFs, just like Yoichi's FF stuff. I also removed the old OAF_OUT, because it would only allow me to select one DoF at a time, and I will eventually want the ability to do multiple amounts of OAFing at the same time. Hopefully.
All of the above changes have been svn'ed with nice log messages into the cds svn.
I have not yet modified the PEM model to give the seis/acc information to the RFM model.
I will need to acquire the PSL's PZT input as a representation of the mode cleaner's length if I want to apply the OAF to the MC to recreate past work.
For the acquisition of the MC_F channel, I suggest taking the FAST_MON BNC output from the blue FSS interface card in the Eurocard crate in the PSL rack. This can then be piped into the 2-pin LEMO plug (Ch. 1) of the Generic Pentek DAQ card which used to acquire the MC_L signal from the MC Servo Board.
This modification of the LSC model made the rows of the LSC output matrix shifted. This caused the ASS scripts nonfunctional.
Kiwamu fixed the channel names in the ASS script.
Suresh tells us that he already has this channel physically plugged in. Probably as a result of Valera's MCASS work. Neat. We just have to make the channel. Right now the signal goes straight into some lockin stuff, so there is no actual "C1:IOO-MC_F" channel.
We don't want to make the new channel right now, since it is nighttime, and Kiwamu and Suresh are working on things. So. Tomorrow. In the morning:
We will add a fast test point to the C1IOO model, and call it "C1:IOO-MC_F". We will also route this signal via memory stuff over to the OAF model so that we can do adaptive filtering on the MC. Then we will compile all the things. Or at least all the things that we touched. This will go hand-in-hand with the compling of Mirko's sweet new OAF model, which we were planning on compiling in the morning anyway. Neat.
Things to compile tomorrow: c1ioo and c1rfm, because of channel routing. c1oaf because of all the new stuff. That should be all.
Is it okay to have two names for the same signal? We would have both MCS_MCL and MC_F referring to MC length signal. This signal is picked up from the MC-Servo (analog) and brought into the CDS through the adc_0_0 channel in C1IOO. Then this signal is sent from C1IOO to C1MCS model without going through the c1rfm model. This seems to break the current protocol that signals passed between machines have to go through the c1rfm model. It should be sufficient to send this signal to c1rfm once and from there redirect to MCS and OAF from there, with an appropriate name.
Suresh makes a fine point. I think the channel between c1ioo and c1mcs should always have had to go through the c1rfm model. I don't know why it wasn't. Anyhow, I have changed things so that there is one signal passing from c1ioo to c1rfm, and that signal is split in two, and goes to both c1oaf and c1mcs. The naming convention I used last night is the one I kept: C1:IOO-RFM_MCL goes from c1ioo to c1rfm, and then C1:RFM-OAF_MCL goes from c1rfm to c1oaf, and C1:RFM-MCS_MCL goes from c1rfm to c1mcs.
We can't compile until Mirko and I figure out what to do with the OAF model though. Mirko, Den and I were looking at the c1oaf model, to make sure it is ready to compile, and I'm not sure that it is. And we need everything with common channel names to be compiled at the same time, so I can't compile any of the models today, until we get this figured out.
The problem is thus: The old TOP_XFCODE that mevans wrote back in 2008 takes in a certain number of inputs, calculates the new filter coefficients, and spits out the filtered signals. Back in those days, we only ever gave the adaptive system one control (target) signal at a time. Now, we want to be able to do multiple, if we so desire. I don't know exactly how to do this yet. Either we need to modify the code to make it a super-code, or we can have one copy of the code for each control signal (MC_F, XARM, YARM, DARM, MICH, etc...). Do we want to have one code adapt everything at once, and have a giant MIMO system, or do we want to have many SISO-like systems in parallel (SISO-like, because each one would take in one control signal, and many seismometer signals, and output many filtered seis signals, but it wouldn't be combining control signals together)?
Either one of these options could be waaay to computationally tough for the computer. The old computer was basically railed when we had one adaptive block, with one control signal and 7 seismometers. 7 was the max number of auxiliary channels we could use. So, how much faster are the new computers?? Do we need to have one OAF per DoF that we want to filter?
I am leaving the OAF disabled, so there should be nothing that goes to the suspensions from the OAF.
Disabled for the OAF means all the outputs are multiplied by 0 just before the signals are sent back over to the LSC system to be summed in and sent to the suspensions. So 0 times anything should mean that the OAF isn't injecting signals.
In other news, while Mirko was trying to figure out the c-code, I began making a new OAF screen. It is still in progress, but it's in c1oaf/master/C1OAF_OVERVIEW.adl If you open it up, you can see for yourself that the OAF is disabled. (Eventually I'll put an enable/disable button on the LSC screen also, but that hasn't happened yet.)
Mirko edited the c-code to use Den's stuff that he put in the elog last night. We then tried to compile and install, but it crashed c1lsc again. We reverted to the simple, working c-code, pushed the physical reset button on c1lsc, and things started getting better. The suspensions had the same problem as last night...we had to do a soft shutdown of c1sus to get things better again.
I did a by-hand burt restore for all of the models on c1lsc and c1sus, since the auto burt restore isn't working.
[Mirko, Den, Jenne]
We're modifying the c1oaf model, and we got to talking about the "Fx" part of "FxLMS". In the past, we put in a guess for the filter. What if we use the static wiener filter as the F, and send the wiener filtered witness channels to the adaptation algorithm?
Are the wiener filter and the plant compensation filter ("Fx") the same? Will this work? Or is there some reason that they must be different?
Also: Notes to self: Put in filtered witness channels as well as regular into Adapt block. Make sure that the order/number of inputs is correct.
Mirko and I were talking about the tunability required for each DoF. For right now, we're going to give each DoF its own mu and tau (adaptation gain and adaptation decay), but we're leaving all of the other things (number of taps, number of witnesses, delay, downsample rate) the same for each DoF's adaptation. This may need to change later, but we'll get there when we get there.
The one I'm most concerned about is the number of witnesses... Mirko is suggesting that we give each adaptation algorithm all witnesses, and unused ones should converge to ~0. I agree in principle, but I'm not sure that it's actually going to work that way. I think we may need to be able to tell the algorithm which witnesses to look at for which DoF.
Also, the Delay.... we may need to adjust it for each DoF. In Matt's OAF "manual" in elog 395, he mentions the need to calculate the delay based on a plant TF. Presumably since (except for the MC) all the witnesses come in together, and all the DoFs come in together, the delay should be about the same for all? We'll have to see...
Toyed around some more with the adaptive filters.
nTaps Downsampling factor Execution time average / max in ca. 3 min [us], (480 us available)
1000 16 110 / 150
2000 16 280 / 340
3000 16 380 / 470
4000 16 Over limit
Now we are running with Downsampling 32, 4000 Taps => max 410us execution time.
I tried to desynchronize the downsampled operations of the filters of the different DOFs. That however increased execution time by about 10%. So I undid that.
We currently have the code running for all DOFs using all witness channels. By default nothing is applied. C-Code parameters can be changed via the respective EPICS variables. Sanity checks in the C-Code make sure the code doesn't crash when nothing / zeros are fed to the code. Let's look into applying FF to one DOF only as a starting point. We start with MCL.
Remember there are two possible signals to look into MC-F and MC-Servo. See page 5695 http://nodus.ligo.caltech.edu:8080/40m/5695
=> At least 20dB SNR. ADC noise should not be an issue. Of course more is always better.
STS1 is located at the vertex. x-axis along the x arm.
GUR1 is located at the IMC MC2 mirror. Same orientation.
=> 1. Only the x-direction has good coherence (to be expected)
2. Only good coherence at 1.5-4Hz (huh?)
So probably other noise sources are dominating. Let's look into noise projections. Remember IMC autoalignment is off.
A quick adaptive filter run with only the GUR1 and STS1 witnesses applied only to MCL didn't really do anything. Some more thought needs to be invested into the AA and shaping filters.