The last MC_F calibration was done by Ayaka : Elog 7823
And does anyone know what the MC_F calibration is?
I saw that entry, but it doesn't state what the calibration is in units of Hz/counts. It just gives the final calibrated spectrum.
It's railed. This is what halted locking progess on Monday night, as this channel is used for the offloadMCF script, which slowly feeds back a CARM signal to the ETMs to prevent the VCO from saturating.
Attached is a 5 day trend, which shows that the channel went dead a few days ago. All the channels shown are being collected from the same ICS110B (I think), but only some are dead. It looks like they went dead around the time of the "All computers down" from Sunday.
Attached are the channels being recorded from the ICS110B in 1Y2 (the IOO rack). Channels 12, 13, 16, 17, 22, 24, 25 appear to have gone dead after the computer problems on Sunday.
This has been fixed by one of the two most powerful & useful IFO debugging techniques: rebooting. I keyed the crate in 1Y2.
Sometimes I like to plot the spectrum of MC_F. Its a good diagnosis of whether something is wrong.
The red trace is noisier than the blue reference. What is the cause of this?
MC_F low frequency noise might be due to local damping electronics. I did not measure OSEM noise, but even without it electronics (AA -> ICS 110 -> ADC) provide sufficient amount of noise.
These 2 image show electronics noise and coherence between OSEM signal / seismic
From these 2 plots we might think that SNR > 10 and coherence OSEM / GUR is high at the frequency range 0.1 - 10 Hz and this is not a big problem.
However, at low frequencies the length of seismic waves becomes large enough and relative oscillations of MC2 and MC13 decrease.
For 1 wave ( u(MC2) - u(MC1) ) / u(MC2) = sin(2 * pi * L * f / c), L - distance between MC1 and MC2 where 2 seismometers are located. So MC123 move according to seismic motion and electronics noise is not seen unless we look at MC Length. Here this noise is seen, because mirrors move in a synchronistic manner.
To check this I measured seismic noise with 2 guralps at distance 12 meters - at MC1 and MC2. Then I've computed the difference between these signals. And indeed at low frequencies, relative motion is much less.
Green, blue - GUR1,2_X
Red - differential motion GUR1_X - GUR2_X
The following plot illustrates how electronics noise effects MC_F. Green is the signal to coils. Red - electronics noise. Blue, black, cyan - simulated contribution to MC_F for different seismic waves speed. Most probably seismic waves have waves in the range 50 - 800 m/s, others are deep. The plot shows that electronics noise is big enough to disturb coherence between MC_F and seismic noise.
Here is a rough calculation of the seismic waves speed. The following plot shows the ratio of psd of differential MC2-MC1 motion to MC2 motion.
If seismometers would be very far, ratio would be 1 if we neglect the difference in transfer function SEISMOMETER -> ADC for each channel. The drift of the ratio from 1 to 1.3 demonstrates this effect. Ratio starts to decrease at 15 Hz according to sin (2*pi*L*f/c) ~ 2*pi*L/c * f. So 2*pi*L/c * f_0 = pi/2 => c = 4 * L * f = 600 m / sec.
We looked at the different outputs of the MC servo board to make sure they make some kind of sense. As per my elog 6625, the names of the channels were wrong, but we wanted to confirm that we have something sensible.
Currently, OUT1 of the servo board is called "MC_F" and the SERVO out is called "MC_SERVO". We looked at the spectrum of each, and the transfer function between them.
You can see that in addition to a 2kHz pole, MC_L also seems to have a 10-100 zero-pole pair.
Also, while cleaning things up in the models, I fixed the names of these MCL/MCF channels. OUT1 is now called MC_L, and is connected to ADC0_0, and SERVO is called MC_F and is connected to ADC0_6. Both MC_L and MC_F go to the RFM, and thence on to the OAF. MC_L (which used to be mis-named MC_F) still goes both to the MCS model for actuation on MC2, and to the OAF for MC-OAF-ing. Right now MC_F is unused in the OAF model, but we can change that later if we want.
I grabbed the a plot of the iLIGO PSL frequency noise spectrum from the Rana manifesto:
Rana's contention is that this spectrum (red trace) is roughly the same as for our NPRO.
From the jenne/mevans/pepper/rana paper Active noise cancellation in a suspended interferometer I pulled a plot of the calibrated MC_L noise spectrum:
The green line on this plot is a rough estimate of where the above laser frequency noise would fall on this plot. The conversion is:
L / f = 10 m / 2.8e14 Hz = 3.5e-14 m/Hz
which at 10 Hz is roughly 1.5e-11 m. This puts the crossover somewhere between 1 and 10 Hz.
I drove MC2 in POS and used the resulting response in MC_F to calibrate the IOO-MC_L channel.
Yoichi did an excellent job of calibrating MC_F last year. I have used his calibration of MC_F (220 Hz/count @ DC) to get the MC_L calibration at DC as well as at high frequencies. The hardware dewhitening was OFF for all these measurements.
1. For the DC measurement I excited C1:SUS-MC2_MCL_EXC at 0.0731 Hz. At these frequencies, the MC_L path has much more gain than the MC_F path. So this excitation at the error point makes the length path to drive itself to cancel the digital excitation. Since the overall MC servo gain is huge, this causes the MC_F path to compensate the residual MC_L motion. One can simply take the ratio of MC_L/MC_F to get the calibration of MC_L in Hz.
2. For the AC measurement, I engaged FM9 of the MC2_MCL filter bank. This guy is an elliptic LP with a notch at 660.38 Hz. I then drove MC2_LSC at 660.38 Hz with a sine wave of 500 counts amplitude. The notch makes the gain of the MC_L feedback zero at that frequency. So MC_F has to do all the work. We can simply measure the ratio of MC2_LSC/MC_F to get the AC calibration of MC2_MCL_OUT (aka IOO-MC_L) and MC2_LSC_OUT (aka LSC-MC_L).
MCF/MCL @ 0.0731 Hz = 569.4. So the MC_L calibration at DC is 220 x 569.4 = 125 kHz/count or 6.02 nm/count.
From this we would expect the AC calibration to be (6 nm/count)*(660.38/f_pend)^2 = 13.0 x10^-15 m/count.
The AC measurement gave 1445 counts_peak** of MC_F for the 500 counts (peak) excitation in MC2_LSC. From Yoichi's entry we get that the high frequency calibration of MC_F should be 0.089 Hz/count. So the MC_L calibration at 660 Hz is 0.089*1445/500 = 0.25 Hz / count or 12.3 x 10^-15 m/count. So the AC/DC ratio is close to 1.
Splitting the difference, the new official MC_L calibration is 5.87 nm/counts @ DC with a complex pole pair at 0.972 Hz.
** note: To convert from the peak height observed in DTT with a 50% Overlap Hanning window you must use the following intuitive formula: counts_peak = (counts / rHz) * sqrt(2 * BW). In this case, BW is the number that DTT reports as BW on the screen, NOT the BW that you asked for on the measurement tab.
*** note: when measuring peak heights in a DTT FFT, make sure to unclick the box for 'Bin' under the config tab. Bin suppresses peaks in a plot with a lot of points and is ON by default.
**** note: I have updated the MCF reference in the Templates directory with the Yoichi calibration - spectrum attached. This is probably the most accurate MCF spectrum we have ever put in the elog in the history of the 40m. The implication is that the VCO phase noise is ~5 mHz/rHz. Not bad.
***** note: with the OAF off, I drove a 1.55 Hz sine wave into MC1 and measured the ratio of MC1_MCL_OUT/IOO-MC_L. This gives the DC calibration of MC1_MCL_OUT = 7.98 nm/count.
While Clara was working on her Wiener filtering and optimizing the locations of the accelerometers, she discovered that MC_L and MC_L_256 are totally flatlined. I looked at them, and it looks like they've been dead since ~9:30pm-ish on Sunday night. Bootfest-type activities shall commence shortly.
Under Alberto's tutalage, I rebooted the whole vme set (iovme, sosvme, susvme1, susvme2), and after that MC_L was all good again.
Jenne and I did adaptive filtering of MC_L and measured how X and Y ARM control signals change compared to non-filtered MC_L. We did the test during 1.5 Hz seismic noise activity and adaptive filter was able to subtract it. However, it adds noise at high frequencies, It is not seen in MC_L but it is present in the ARMs control signals.
I'll investigate this problem. May be we need to reduce adaptation gain. In this experiment it was 0.1 and adaptive filter convergence time was equal to 1-2 mins.
I've applied LQR approach to MC_L locking. Results show that LQR does not make MC_F signal smaller below 0.3 Hz in contrast with classical locking. This might indicate that in this frequency range we see sensing noise as LQR was provided with state-space model of MC only so it tries to reduce displacement noise. It is also possible that state-space model is not accurate enough.
I tried to plot a long trend MC Transmitted today. I could not get farther than 2017 Aug 4
The mode cleaner was misaligned probably due to the earthquake (the drop in the MC transmitted value slightly after utc 7:38:52 as seen in the second plot). The plots show PMC transmitted and MC sum signals from 10th june 07:10:08 UTC over a duration of 17 hrs. The PMC was realigned at about 4-4:15 pm today by rana. This can be seen in the first plot.
Without adding significant amounts of noise to other WFS loops I have engaged the MC2_TRANS_PIT and YAW loops.
After several attempts to measure the system response and computing the output matrix, none of which gave any useful results, I gave up on that and decided to find three orthogonal actuation vectors which enable us to close the loops. So using the last good output matrix (below left side) as a template, I rounded it off to the nearest set of orthogonal vectors and arrived at the following matrix (right side):
I also decided that WFS1 and 2 need not drive MC2. This is just to decouple the loops and minimise cross-talk. This (albeit heuristic) matrix seems to work pretty well and the real matrix is probably quite close to it.
I show below the suppressed error signals after tweaking the gains a bit. The blue line is with no WFS, the green one with only WFS1 and 2 loops on, while the red is with all loops turned on. The WFS1Yaw and MC2_Trans_pit loops might benefit from a more careful study to determine a better output matrix.
I'm quite sure that this is not good: since MC2 can produce a signal in WFS1 and WFS2, it cannot be removed in this way from the actuation without introducing a significant cross-coupling between the MC_TRANS and WFS loops.
Really need loop TFs measured and compared with the model.
The WFS noise model will also show that we need to have a much lower UGF in the MCT loop since that sensor is just a DC QPD: it can never have as good of a sensing noise as a good WFS. In the current case with no Whitening, this is even more so.
Jenne, Koji, Rana
After fixing up the Mode Cleaner a bit more (fiddling more with the MC_align sliders to get the alignment before locking, making sure that it is able to lock), we noticed that the MC Trans path could use some help. To align the MC, we put MC1 and MC3 back into the position where Rob left it on Thursday and then maximized the transmission with MC2. Then we went back and maximized with MC1/3 keeping in mind the Faraday. We got a good transmission and the X-arm had a transmission of 0.8 without us touching its alignment.
Upon looking at the AP table portion of the MC_trans path, we decided that it was all pretty bad. The light travels around the edge of the AP table, then out the corner of the table toward the PSL table. A periscope brings it down to the level of the PSL table, and then it travels through a few optics to the MC_trans QPD.
The light was clipping on the way through the periscope, and so the MC_trans QPD was totally unreliable as a method of fine-tuning the alignment of the Mode Cleaner. Ideally we'd like to be able to maximize MC_trans, and say that that's a good MC alignment, but that doesn't work when the beam is clipped.
1. The first turning mirror on the AP table after the beam comes out of the vacuum was changed from a 1" optic to a 2" optic, because the spot size is ~4-6mm. We were careful to avoid clipping the OMCT beam, by using a nifty U200 mount (C-shaped instead of ring-shaped).
2. We placed a lens with a RoC of 1m (focal length for 1064nm is ~2m), a 2" optic, between the first two mirrors, to help keep the beam small-ish when it gets to the periscope, to help avoid clipping.
3. Rana adjusted the angle of the upper periscope mirror, because even when the beam was centered on the steering mirror directly in front of the periscope and the spot was centered on the first periscope mirror, the beam wouldn't hit the bottom periscope mirror.
4. We noticed that the bottom periscope mirror was mounted much too low. It was mounted as if the optics after it were 3" high, which is true for all of the input optics on the PSL table. However, for the MC_trans stuff, all the optics are 4". We moved the periscope up one hole, which made it the correct height.
5. We removed the skinny beam tube which guided/protected the beam coming off the periscope after a steering mirror since it (a) wasn't necessary and (b) was clipping the beam. We cannot use such skinny tubes anymore Steve.
6. There was a lens just before the 2nd steering mirror on the PSL table portion, which we removed since we had placed the other lens earlier in the path. 2 lenses made the beam too skinny at the QPD.
7. After this 2nd steering mirror, there had been a pickoff, to send a bit of beam at a crazy angle over to the RFAM mon, which we removed. This results in a much brighter beam at the MC_trans QPD, and at the camera. The QPDs readouts are now a factor of ~3.5 higher than they used to be. These (especially the camera) could use some ND-filtering action.
8. The steering optic directly in front of the MC_trans QPD is a beamsplitter, and instead of dumping the light which doesn't go to the MC_trans QPD, we used this to go over to the RFAM mon (instead of the pickoff which we had removed).
9. Koji fixed up the optics directly in front of the RFAM mon, accomodating the new position of the input light (now at a much more reasonable angle, and about 15cm farther back from the PD). Note the beam dump which is preventing the cables from the FSS board from entering the beam path. This included removing an ND filter wheel, so the RFAM mon values will all be higher now. Koji also has the beam going to the PD going at a slight angle, so that the beam isn't directly reflected on itself, so that it can be dumped.
10. We aligned the beam onto the MC_trans QPD using the first steering mirror on the PSL table.
11. We also removed the giant wall of beam dumps separating the squeezing section of the table from the rest of the table.
Alberto will elog things about the RFAM mon, including different values of the PD output, etc.
Still on the to-do list:
A. Replace the second steering mirror on the AP table after the MC_trans light leaves the vacuum with a 2" optic, since the lens we placed isn't tight enough to make the spot small there yet. Us a U200A mount if possible, because they are really nice mounts.
B. Put an ND filter in front of the MC_trans camera, because the image is too bright.
C. Normalize the MC_trans QPD - the horz and vert are pretty much direct voltage readouts, with no normalization. They should be divided by the DC value. This lack of normalization results in higher sensitivity to input pointing.
D. Long term, next time someone wants to optimize the MC_trans path, move all the optics, including the MC_trans QPD and the camera closer to the periscope on the PSL table. There's no reason for the beam to be traveling nearly the full width of the PSL table when we're not manuvering around squeezing stuff.
E. Never, ever purchase these horrible U100 or U200 mounts with the full ring and the little plastic clips. They are the "AC28" version. Bad, bad, bad.
Image 1: The new setup of the AP table, Mc_trans portion.
Image 2: New setup of the MC_trans part of the PSL table.
The MC_trans QPD Pitch and Yaw readout on the Lock_MC screen are now normalized by the trans_sum. I used the method described in my entry elog 1488.
/caltech/target/c1iool0/ioo.db now includes:
field(SCAN, ".1 second")
field(SCAN, ".1 second")
The Lock_MC screen was changed to show these new P and Y channels.
The MCautolocker had stalled - there were no additional lines to the logfile after 12:17pm (~20mins ago). Normally, it suffices to ssh into megatron and run sudo initctl restart MCautolocker - but it seems that there was no running initctl instance of this, so I had to run sudo initctl start MCautolocker. The FSS Slow control initctl process also seemed to have been terminated, so I ran sudo initctl start FSSslowPy.
I rebooted megatron around 12:20 today. It had dozens of stalled medm process (some of them there since February!). I couldn't kill them without them coming back like zombies, so I did sudo reboot.
I've noticed this a couple of times today - when the autolocker runs the mcdown script, sometimes it doesn't seem to actually change the various gain sliders on the PSL FSS. There is no handshaking built in to the autolocker at the moment. So the autolocker thinks that the settings are correct for lock re-acquisition, but they are not. The PCdrive signal is often railing, as is the PZT signal. The autolocker just gets stuck waiting to re-acquire lock. This has happened today ~3 times, and each time, the Autolocker has tried to re-acquire lock unsuccessfully for ~1hour.
Perhaps I'll add a line or two to check that the signal levels are indicative of mcdown being successfully executed.
Looks like this problem presisted over the weekend - Attachment #1 is the wall StripTool trace for PSL diagnostics, seems like the control signal to the NPRO PZT and FSS EOM were all over the place, and saturated for the most part.
I traced down the problem to an unresponsive c1iool0. So looks like for the IMC autolocker to work properly (on the software end), we need c1psl, c1iool0 and megatron to all be running smoothly. c1psl controls the FSS box gains through EPICS channels, c1iool0 controls the MC servo board gains through EPICS channels, and megatron runs the various scripts to setup the gains for either lock acquisition or in lock states. In this specific case, the autolocker was being foiled because the mcdown script wasn't running properly - it was unable to set the EPICS channel C1:IOO-MC_VCO_GAIN to its lock acquisition value of -15dB, and was stuck at its in-lock value of +7dB. Curiously, the other EPICS channels on c1iool0 seemed readable and were reset by mcdown. Anyways, keying the c1iool0 crate seems to have fixed the probelm.
(keerthana, gautam, jon)
In the morning, Jon gave me an overview of the Auxiliary laser system which we are planning to setup. Based on the diagram he uploaded in the elog, I have made the MEDM diagram for controlling and displaying the parameters. Here the parameters which we will be controlling are temperature (in terms of voltage), oscilator frequency ( with the help of IFR 2023B), the frequency offset and the PID controls. The display includes the beat frequency, error signal voltage, control voltage and a switch to give feed back to the AUX laser. As the frequency counter is not connected at the moment, I haven't included its channel number in it. The screenshot of the diagram is attached with this. I am also considering to give a PID feedback to the slow control from the AUX feedback signal. The screen can be accessed from the PSL dropdown menu in sitemap.
I noticed at LLO (?) that the LSC screen there uses up ~25-30% of the CPU time on a single core for the control room iMac workstations - this seems excessive.
Here is an accounting of CPU usage percentages for some of our screens:
These were measured using the program 'glances' on rosalba. MEDM running with only the sitemap used up 0.9% of a CPU. With the screens running, the fluctuation from sample to sample could be ~ +/- 0.5%. While the LSC screen seems to be the biggest pig, it is only big in comparison to small pigs. Certainly this pig has gotten bigger after getting sent to Louisiana.
Looks like 2 different MEDM Snapshot functiions (at least) are broken.
The regular update of the screens here as well as the usual "Update Snapshot" and view "previous snapshot" button on all of the auto-generated screens.
Also, how do we add the snapshot button to the custom made screens?
A new MEDM tab has been added to the summary pages (https://nodus.ligo.caltech.edu:30889/detcharsummary/day/20160708/medm/), although some of the screens are not updated when /cvs/cds/projects/statScreen/cronjob.sh is run. In /cvs/cds/projects/statScreen/log.txt, the following error is given for those files: import: unable to read X window image `0x20011f': Resource temporarily unavailable @ error/xwindow.c/XImportImage/5027. If anyone has seen this error before or knows how to fix it, please let me know.
In the meantime, I'll be working on creating an archive of MEDM screens for every hour to be displayed on the summary pages.
Some of the screens are up-to-date, and some are not. Are the errors associated with the screens that failed to get updated?
Thanks! Yes, only the screens that are not updated when the script is executed show this error. I'll try to keep debugging over the weekend.
Successfully completed the rudimentary GUI for PDFR system. (users/nichin/PDFR)
Pressing any of the buttons above runs the script that does the following:
1) Change RF mux channel to the required one.
2) Frequency sweep on the network analyzer. The common sweep parameters are in a file named param_NWAG4395A.yml . PD specific parameters are in param_[PD name].yml in their respective folders
3) The transimpedance is calculated and the plot is saved as PDF in the respective folder for the PD. Each set of measurement data and plot is in a timestamped subfolder.
To take transimpedance readings for each PD and create a canonical set of data that can be used to compare with data obtained for every measurement run.
The original matrix naming conventions for the front ends was broken. It used _11, _12,...,_1e, _1f, _110, _111 and so forth. The code was changes to use _1_1, _1_2,...,_1_16,_1_17, and so on.
In addition the matrix of filter banks was modified to use the same naming convetion (instead of starting at zero, it now start with one).
I rebuilt all the models, and restarted them all.
I wrote a simple script to modify the burt restore files to have the correct names for all the stored matrix values.
I also modified all the suspension screens, by modifying the default screens in /opt/rtcds/caltech/c1/medm/master/
The C1SUS, C1SCX, C1SPX, C1SCY, C1SPY, and C1MCS models had their foton filter files modified to put filters into the newly changed named filters
Just a curiosity:
I just wonder how you have distingushed the difference between _111 and _111.
They are equivalent alone themselves. Have you looked at the contexts of the lines?
Or you just did not have the larger matrix than 16x16, did you?
They are equivalent alone themselves. Have you looked at the contexts of the lines?
Or you just did not have the larger matrix than 16x16, did you?
I created an MEDM screen for the PID control. In addition, I added a new EPICS channel for the setpoint so that it could be adjusted using the MEDM screen.
Edit: forgot to mention the channel name is C1:PEM-SEIS_EX_TEMP_SETPOINT
Edit #2: the path for the MEDM is /opt/rtcds/caltech/c1/medm/c1pem/C1PEM_SEIS_EX_TCTRL.adl
An update to the screen. I changed the min/max values for some of the parameters, as well as changing the script so that I could specify the integral gain in terms of 1e-5. I've also added this screen to the PEM tab in the sitemap.
Another update. I've changed the on/off button so that it's visible which state it's in. I did that by changing the type of C1:PEM-SEIS-EX_TEMP_SLOWLOOP from ai to bi (I checked the FSS script and copied the entry for the slowloop). Previously, MEDM was giving me an error that it wasn't an ENUM value when I wanted to use a choice button to indicate the value of slowloop, and this solved the issue. I've also added a StripTool button.
changed the setpoint of the EX Seismomter T ctrl servo from 35 to 39 C to see if this helps the stability by decreasing the cooldown time constant.
Since Nodus is a Solaris machine it can barely handle doing the ImageMagick commands (such as convert and import). I removed the auto MEDM snapshot routine from it
awhile ago and I think the rate of ELOGD crashes has decreased, although its not definitive.
The snapshots have now been re-actived to run on MAFALDA, after I fixed the absolute pathnames in the scripts and installed (via yum) the packages that mafalda
needed to run this (Xvfb, openmotif, compat, etc.). The snapshots web page is now refreshing by itself and the statScreen/cronjob.sh is running on mafalda 5x per hour.
We need more organized MEDM screens. Let's use macro.
What I did:
1. Edited /opt/rtcds/userapps/trunk/sus/c1/medm/templates/SUS_SINGLE.adl using replacements below;
sed -i s/#IFO#SUS_#PART_NAME#/'$(IFO)$(SYS)_$(OPTIC)'/g SUS_SINGLE.adl
sed -i s/#IFO#SUS#_#PART_NAME#/'$(IFO)$(SYS)_$(OPTIC)'/g SUS_SINGLE.adl
sed -i s/#IFO#:FEC-#DCU_ID#/'$(IFO):FEC-$(DCU_ID)'/g SUS_SINGLE.adl
sed -i s/#CHANNEL#/'$(IFO):$(SYS)-$(OPTIC)'/g SUS_SINGLE.adl
sed -i s/#PART_NAME#/'$(OPTIC)'/g SUS_SINGLE.adl
2. Edited sitemap.adl so that they open SUS_SINGLE.adl with arguments like
instead of opening ./c1mcs/C1SUS_MC1.adl.
3. I also fixed white blocks in the LOCKIN part.
Now you don't have to generate every suspension screens. Just edit SUS_SIGNLE.adl.
Things to do:
- fix every other MEDM screens which open suspension screens, so that they open SUS_SINGLE.adl
- make SUS_SINGLE.adl more cool
Very nice, Yuta! Don't forget to commit your changes to the SVN. I took the liberty of doing that for you. I also tweaked the file a bit, so we don't have to specify IFO and SYS, since those aren't going to ever change. So the arguments are now only: OPTIC=MC1,DCU_ID=36. I updated the sitemap accordingly.
Yuta, if you could go ahead and modify the calls to these screens in other places that would be great. The WATCHDOG, LSC_OVERVIEW, MC_ALIGN screens are ones that immediately come to mind.
And also feel free to make cool new ones. We could try to make simplified version of the suspension screens now being used at the sites, which are quite nice.
Actually, it looks like we're not quite done here. All the paths in the SUS_SINGLE screen need to be updated to reflect the move. We should probably make a macro that points to /opt/rtcds/caltech/c1/screens, and update all the paths accordingly.