Just a quick note, plots and data will come tomorrow:
I grabbed an unused uPDH board from the ATF (thanks Zach!), and re-stuffed almost the entire thing to match Jenne's latest schematic for the y end box. I also threw some 22uF caps on the regulators, as Koji did with the previous box, to eliminate some oscillations up in the high 10s of kHz. I replaced the tragedy of a box that I created on Wednesday with this new box. The arm locks pretty stably with the boost on, 30 degrees of phase margin with 10kHz UGF, and locks pretty darn reliably.
Now we should now have two nicely boosted PDH loops. I'll do a noise/loop breakdown again in the upcoming days.
I measured the noise spectra and loop TF of the green PDH with the newly stuffed board. Unfortunately, I never took the noise below 100Hz of the previous box, so we can't see what has happened to the overall RMS, or more specifically, the RMS due to the pendulum resonance. All of these plots are in the boosted state, as that is how we intend to use the box.
Here is the loop, which does not have quite as much margin as the y-arm, but 10dB of gain peaking is probably ok, since the RMS at 10s of kHz is not so important to ALS. (OL measured, CL inferred) We see the 1/f shape from 1k to 50k or so, and 1/f^2 under 1k, as desired.
Comparing in the in loop error signals, we see the effect from the increased gain from 100Hz to 10kHz. (Here is where I regret not looking at the low frequency spectrum two weeks ago)
Finally, here is the noise breakdown.
The error signal RMS is now dominated by the 1Hz peak. We have talked about using digital feedback for this, since we have the PDH error signal coming into an ADC, and can sum in a DAC signal into the servo output. This also lets us intelligently trigger a sub-10Hz boost once the PDH box locks itself. With a good boost, we maybe could bring the in-loop RMS of the error signal to under 1kHz.
Something odd that Rana brought to my attention, however, is that my measurement and calibration indicates an RMS of ~5kHz, but the cavity pole should be something like 18kHz. If this is true, how can we be seeing stable power? This maybe means that my calibration is too many Hz per Volt.
I performed the calibration by creating a MIST model of the arm, and generating the PDH error signal on a demodulated PD, I then find the slope of Hz per arbitrary error signal unit. Then, looking at a scope trace, I match up the horn-to-horn voltage to the horn-to-horn arbitrary error signal units, which lets me finally find Hz per error signal volt.
However, there is some qualitative difference in the shape between the simulated and observed error signals, namely, that the outer horns are larger than the inner horns in the real signal.
Does this matter? Is there something in my simulation that I can correct that would give a more accurate calibration?
Data, plots, code, attached.
I used a modulation depth of 0.3, which, if I recall correctly, is what we aimed for on the Y-arm when we adjusted the LO signal there. However, this is probably not the case for the X arm.
In any case, I found the bug in my RMS calculation. (I had forgotten to flip the x array in addition to the y array for the right-to-left integration, and had uneven bin spacing, so the integration bandwidths weren't correct...)
Here are the updated plots. The properly evaluated RMS is ~600Hz, which seems to mostly come in around 10k, so we may want to turn down the gain for less gain peaking in that region.
Q and Steve will follow elog 10028 entry to prepare the vacuum system for safe reboot
Here's the sequence of the morning so far:
The IFO is still down, as the PMC won't lock without the rack power, and we haven't pinned down the shorting mechanism. We don't want the replacement sorensen to immediately blow when plugged in.
We replaced the +15V sorensen at 1X1, and brought the power supplies back up symmetrically, and everything seems fine. I noted that a quarter turn counter-clockwise took the current limit down by one amp, so I set the knob to just letting 2.8A (the nominal current), and then added one half turn, shooting for ~4A current limit.
In doing so, we had to cut power to the c1psl VME box. It didn't come back happily. We had to do the chiara /etc/hosts things, like we did for c1auxexx, to get it back.
I checked chiara's tables, all seemed fine. I switched ethernet cables from the black one labelled "allegra," which seemed maybe fragile, for the teal one that may have been chiara's old ethernet cable. It's back on the network now; hopefully it lasts.
Some small things I did tonight which did little to nothing to help:
My main concern with tonights situation was the huge low frequency fluctuations of TRY while CARM/DARM locked on ALS. We saw this being very smooth very recently, but when one arm is fluctuating by multiple line widths, it isn't surprising that locks aren't stable. I want to know why the out of loop stability is so unpredictable.
I took a quick measurement of the ALS stability, using POX and POY as out of loop sensors, using a CARM calibration line to line POX and POY up to the calibrated PHASE_OUT channels at 503Hz.
Since DRMI didn't get fully commissioned, I tried my hand at PRFPMI locking with the newly improved ALS performance.
ALS seemed reliable, I think my main limiting factor was the PRMI locking. We should set up a restore script for PRFPMI that is a superset of the ALS CARM DARM, because the current restore script doesn't put all the vertex settings back, so I was trying to lock for a while without the FM boosts on PRCL and MICH, which really hurt my stability.
Transitioning to SqrtInv works fine; a couple of times I've gotten to arm power of ~10, and have been able to sit there for a while as I set up excitation line comparisons with the CM board's REFLDC, but the PRC would always lose it before I did anything interesting.
The PRMI locks with a reasonable MICH offset, I found that adding a offset of 20 to 40 makes the AS spot visibly dimmer, and ASDC falls to ~0.05 from .1-.2.
I looked into adding a boost to the CARM loop after transitioning to sqrtInv, but we only have 30 degrees of margin, and the error signal is already fairly white, so there isn't much to do, really.
The ALS locking script is sporadically hanging a fair while, as well, which is strange. Otherwise, not much to report...
REFL11 I, as seen in digital land, is connected to the slow output of the CM board. I tuned the demod angle of the REFL11 demodulator board by cable length back in ELOG 9850. It would be good to check that the phase is still good. If the CM board gains are at 0dB, we should be able to used the digital angle adjustment as normal.
We need to get an interferometric estimation of the SRC length error / SRC sideband splitting, because if the 7.5cm hand-measured error is true, it looks like it might be hard to control the DRMI on 3F.
I did some DRMI sensing simulations, to get an idea if sensing matrix elements might change as the CARM offset changes. Last night, I tried just going to zero CARM offset on ALS, and was having problems keeping the PRMI locked on REFL33, so I wanted to confirm that it should at least work in theory.
Thus, I simulated what happens to the sensing matrix element in the vertex DoFs as the CARM offset is reduced, in both the PR and SR cases. I normalized all of the elements to PRCL at zero carm offset, to get an idea of what the good relative gains should be for MICH and SRCL.
In the end, there don't seem to be significant DC gain changes, or demod angle fluctuations, in either the PRFPMI or DRFPMI case, as the CARM offset changes, which is good.
However, the SRC length as hand-measured, seems to mess up the MICH angle in the DRFPMI case, and really lowers the SRCL signal amplitude.
To be fair, past efforts of simulating demodulation angles haven't always been borne out on the IFO, so we should still forge ahead experimentally until it becomes apparent that there is a real problem.
Here are the simulations for the IFO as-is:
(A note on the plots. Though they kind of look like Bodes, they're just the sensing element represented as a complex number in the I-Q plane,I being phase=0 and Q = 90)
All three signals are along the I axis in the DRMI case, which seems like it would be tough to control, since we only have 2 3F diodes... We've been using REFL33Q when PRMIing, which is simulated at around 45 deg; it should be easy to verify this empirically.
Here are the same plots with the SRC length corrected. Now MICH shows up mostly in the Q phase as desired in the DRMI case. SRCL in REFL165 also wins 20dB of optical gain, as well.
To drive the point home, here's a simulated scan of AS110 and REFL55 Q to show the effect of the measured length error:
Our janitor turned off the laser accidentally.
The PMC wasn't locking very happily after this. I tweaked the pointing onto the PMC REFL diode, to make sure it was centered, and touched the alignment into the PMC. I also reset the FSS Slow output to zero. It took a little while for the laser to settle in, for some reason, but the transmission is up at 0.80 now.
Tweaked MC2 pointing to get the MC transmission high enough to let WFS kick in, which nicely got the rest of the MC alignment done. After that, I offloaded the WFS into the MC suspensions.
Lastly, I ran the command that Rana posted in ELOG 10391, to set the FSS input offset (From -0.18 to -0.06)
I have not had any success the past two days in getting an interferometric measurement of the SRC length.
So, the question posed at today's meeting was: "How precisely do we need to change the SRC length to be able to lock the DRMI on 3F"
The two ways I could think to quantify this are:
REFL33 should have its phase set to put PRCL along I, and REFL165 should have SRCL along I, so the simulation result that matters is the angle of MICH in these planes. The cross couplings are then given by the appropriate trigonometric projections. In the following plots, I used 10% as the acceptable cross coupling in either direction.
Code (finesse + pykat + ipython notebook) and plots are attached.
More AO efforts. No huge news.
Came at AO from each side. For each sign, I lost lock just a few dB from the AO portion of the loop crossing unity gain. Both attempts were about arm powers of 1, which should correspond to ~300pm CARM offset, which I have simulated the crossover as possible with my current loop models (including latest MC loop). The gain steps were usually 6dB in between measurements.
Positive polarity on CM board screen:
I made it to +5 dB of the last plot here, but the 6th broke it open. Gains on CM In2, CM AO, and MC In2 were -6, -4, -2 on that last, lock breaking, step.
Negative polarity on CM board screen:
Lost it just 2dB above the last trace. Gains were -6, +1, -2 (So, overall 5dB higher than the other polarization)
Many things happened in between these two lock stretches, but I'm not sure what may or may not have affected things. They include:
The location of the CARM resonance peak lines up with my simulation, which is good, but there appears to be less phase than expected... I tried making sure that we don't have any whitening uncompensated for, but it looked ok. All my AO path loop model contains is the CM board TF (measured and fitted), the IMC seen as an actuator(measured and fitted), and the REFLDC optical TF (simulated in MIST). Maybe the DC path of whatever diode this is coming from needs to be included...
Discontinuities / glitches could be seen in the CM board fast output when MC board gains were changed, which isn't so nice. Incidentally, I notice now that each lock loss corresponded to a step of AO gain on the CM board.
Rather than using a CAD drawing, I used Gabriele's code from ELOG 9590 to try and judge if we could shorten the SRC by the appropriate length, without clipping the SR3-SR2 beam.
Specifically, I used these lines:
% Move SRM 7.5 towards SR2, parallel to beam
dAS = BS2-AS; % Vector from SRM to SR2
dASmag = sqrt(dAS(1)^2+dAS(2)^2);
dMove = delta*dAS/dASmag; % delta times unit vector
CS = CS+dMove;
As a reminder, Gabriele's code used the following logic:
In my opinion, this is the best estimate of beam trajectory that we currently have.
Thus, from looking at the plot above, I claim we can correct the SRC length without clipping the beam by moving the SRM forward by the required 7.5cm.
Although the measured distance may be off on the order of a cm (since our PRC correction had a 0.5cm disagreement between interferometric and hand distance measurements), this will nevertheless markedly improve our 3F DRMI sensing, based on my previous ELOG.
Hence, given our discussions last week, Jenne and I will proceed to ready the interferometer for venting in the morning, by following the vent checklist.
Our sole objective for this vent is this move of the SRM.
Steve, please check the jam nuts, and begin the vent when you get in. Thanks!
Interferometer alignment is restored
ASS has been run on each arm, recycling mirrors were aligned by overlapping on AS camera.
ETMY was not getting its ASC pitch and yaw signals. C1SCY had a red RFM bit (although, it still does now...)
I took a look at the c1rfm simulink diagram and found that C1RFM had an RFM block called C1:RFM-TST_ETMY_[PIT/YAW] and C1SCY had one called C1:TST-SCY_ETMY_[PIT/YAW].
It seems that C1TST was illegally being used in a real signal chain, and Jenne's recent work with c1tst broke it. I renamed the channels in C1RFM and C1SCY to C1:RFM-SCY_ETMY_[PIT/YAW], saved, compiled, installed, restarted. All was well.
There are still some in SCY that have this TST stuff going on, however. They have to do with ALS, it seems, but are SHMEM blocks, not RFM. Namely:
Ottavia was having some severe interaction latency today. Xorg was taking up >90% of the CPU, just sitting around. The machine was logged in to a desktop session with lots of graphical effects turned on. I changed the system default session to "gnome-fallback" in /etc/lightdm/lightdm.conf, which was already set as the default for controls, but wouldn't get chosen for the autologin that happens on boot.
Hopefully this helps ottavia stay usable...
I think I found out why rossa was mad.
An apt-get update on the 18th downloaded kernel 2.6.32-65-generic, so 2.6.32-58-generic, which what was previously chosen as a working kernel, had moved down in the grub ordering.
It turns out the grub configuration accepts strings, so I changed it to GRUB_DEFAULT="Ubuntu, with Linux 2.6.32-58-generic", ran sudo update-grub, and Rossa now seems to boot happily.
GRUB_DEFAULT="Ubuntu, with Linux 2.6.32-58-generic",
[q, Jenne, Manasa]
I figured out that didn't change the initial guess for the fit routine in Gabriele's code. I also changed the fminsearch criteria to least squares fitting, instead of minimax. The consistency checks now look just as good as the previous time we did these kind of measurements, no disagreements bigger than 1.6mm.
Thus, the current estimate of the SRC length after yesterday's motions is 5402mm, where we desire 5399mm. So, we will try to move SRM 3mm closer to SR2, after confirming that we are not clipping the POY beam. After all that, we will level the table.
Today so far:
Here's my quick brain dump of things to do before we can pump down (anyone see anything missing?):
POY has >2 inches of clearance from the SRM cage.
Distance reconstruction indicates an SRC length of 5399mm, which was exactly our target.
We have completed the above points; the ITMY table is still level.
Despite what the wiki says, the SRM LR OSEM open voltage is ~1.97V instead of ~1.64, so we shot for half of that.
The in-air steering of the SRM oplev return beam needs adjustment. I'll estimate the beam path length when I'm taking pictures and closing up.
Left to do:
Related In-Air work:
Everything is aligned, AS and POY make it out of vacuum unclipped, OSEM readings look good.
I set up the SRM oplev, centered all oplevs.
Tomorrow, we just have to take pictures of the ITMY chamber before we put the heavy doors on.
I closed the PSL shutter as we didn't want to burn the mirror surface when we are not working.
Photos have been taken of the ITMY chamber, and uploaded to picasa. Here's a slideshow:
I've installed a new 2pin lemo cable going from the CM servo out to in2 of the MC servo board, and removed the temporary BNC. I used some electrical tape to give the cable some thickness where the lemo head screws on to try to strain relieve the solder joints; hopefully this cable is more robust than the last.
I put an excitation into the CM board, and saw it come out of MC_F, so I think we're set.
The IFO is ready for 3F DRMI comissioning
I haven't been able to lock the DRMI tonight, neither with 1F and no arms nor 3F and arms held off with ALS... I tried previous recipes, and new combinations informed by simulations I've run, to no avail.
I touched the alignment of the green beat PD on the PSL table, since the X beatnote was rather low, but wasn't able to improve it by much. I never took a spectrum, since it wasn't my main focus tonight, but the low frequency motion of both arms on ALS, as observed by RIN, was good as I've ever seen it.
In our WFS work earlier today, Koji and I reset the WFS offsets, and it actually seems to have helped a good deal, in terms of the "fuzz" of MC REFL on the wall striptool. I had previously presumed this to be due to excess angular motion, but perhaps it is more accurately described as an alignment offset that let the nominal angular motion couple into the RIN more.
Yesterday, Koji and I measured the transfer function of pitch and yaw excitations of each MC mirror, directly to each quadrant of each WFS QPD.
When I last touched the WFS settings, I only used MC2 excitations to set the individual quadrant demodulation phases, but Koji pointed out that this could be incomplete, since motion of the curved MC2 mirror is qualitatively different than motion of the flat 1&3.
We set up a DTT file with twenty TFs (the excitation to I & Q of each WFS quadrant, and the MC2 trans quadrants), and then used some perl find and replace magic to create an xml file for each excitation. These are the files called by the measurement script Koji wrote.
I then wrote a MATLAB script that uses the magical new dttData function Koji and Nic have created, to extract the TF data at the excitation frequency, and build up the sensing elements. I broke the measurements down by detector and excitation coordinate (pitch or yaw).
The amplitudes of the sensing elements in the following plots are normalized to the single largest response of any of the QPD's quadrants to an excitation in the given coordinate, the angles are unchanged. From this, we should be able to read off the proper digital demodulation angles for each segment, confirm the signs of their combinations for pitch and yaw, and construct the sensing matrix elements of the properly rotated signals.
The axes of each quadrant look consistent across mirrors, which is good, as it nails down the proper demod angle.
The xml files and matlab script used to generate these plots is attached. (It requires the dttData functions however, which are in the svn (and the dttData functions require a MATLAB newer than 2012b))
I brought back the PMC, MC and Arms.
The autolocker is now working, but I didn't change anything to make it so. I was just putting in some echo statements, to see where it was getting hung up, and it started working... This isn't the first time I've had this experience.
It turns out IOO had a bad BURT restore. I restored from 5AM this morning, the WFS are ok now.
We're back! It was entirely my fault.
Some months ago I wrote a script that chiara calls every night, that rsyncs its hard drive to an external drive. With the power outage yesterday, the external drive didn't automatically mount, and thus chiara tried to rsync its disk to the mount point, which was at the time just a local folder, which made it go splat.
I'm fixing the backup script to only run if the destination of the rsync job is not a local volume.
I've changed the LSC rack wiring a little bit, to give us some flexibility when it comes to REFL11.
Previous, the REFL11 demod I output was fed straight to the CM servo board, and the slow CM board output was hooked up to the REFL11I ADC channel. Thus, it wasn't really practical to ever even look at sensing angles in REFL11, since the I and Q inputs were subject to different signal paths/gains. (Also, doing LSC offsets would do wonky things to refl11 depending on the state of the switches on the CM board screen.)
Thus, I've hooked up the CM board slow output into the the previously existing, aptly named, CM_SLOW channel. The REFL11 demod board I output is split to IN1 of the CM board, and the REFL11 I ADC channel.
So, there is no longer hidden behavior in behind the REFL11 input filters, channels are what they claim to be, and the CM board output is just as easily accessible to the LSC filters as before.
We attempted some of the same old CARM offset reduction tonight, but from the other direction. (We have no direct knowledge of which is the spring and which is the anti-spring side)
We we able to get to, and sit at, arm powers on the order of 5. Really, we kind of wanted just to push things to try and inform our current ideas of what our limiting factor is, so as to appropriately expend our efforts.
We took many digital CARM OLTFs at different offsets; it never really looked like a burgeoning pole was about to make things unstable. The low frequency OLTF data had bad SNR, so it wasn't clear if we were losing gain there. We weren't at arm powers where we would expect the DC transmission curve to flatten out yet, from simulations (which is above a few tens).
My impression from at least our last lock loss was a DARM excursion. However, using the DRMI won't get rid of the second two points.
In my previous simulation results, I've always plotted W/m, which isn't exactly straightforward. We often think about the displacement that a given mirror actuator output will induce, but when we're locking the full IFO, radiation pressure effects modify the mechanical response depending on the current detuning, making the meaning of W/m transfer functions a little fuzzy.
So, I've redone my MIST simulations to report Watts of signal response due to actual actuator newtons, which is what we actually control with the digital system. Note, however, that these Watts are those that would be sensed by a detector directly at the given port, and doesn't take into account the power reduction from in-air beamsplitters, etc.
As an example, here are the SqrtInv and REFLDC CARM TFs for the anti-spring case:
The units of the SqrtInv plot are maybe a little weird, these TFs are the exact shape of the TRX W/N TFs with the DC value adjusted by the ratio of the DC sweep derivatives of TRX and SqrtInv.
All of the results live in /svn/trunk/modeling/PRFPMI_radpressure/
I touched up the PMC alignment.
While bringing back the MC, I realized IOO got a really old BURT restore again... Restored from midnight last night. WFS still working.
Now aligning IFO for tonight's work
I made some measurements to try and see if any difference could be seen with different CARM offset signs.
Specifically, at various offsets, I used a spare DAC channel to drive IN1 of the CM board, as an "AO Exciter." I used CM_SLOW to monitor the signal that was actually on the board. I used the CARM_IN1 error signal to see how the optical plant responded to the AO excitation. Rather than a swept sine, I used a noise injection kind of TF measurement.
Here are plots of CARM_IN1 / CM_SLOW at different CARM FM offsets; I chose to plot this in an attempt to divide out some of the common things like AA and delays and make the detuned CARM pole more evident). The offsets chosen correspond roughly to powers of 2, 2.5, and 3. I tried to go higher than that, but didn't remain locked for long enough to measure the TF.
By eye, I don't see much of a difference. We can zpk fit the data, and see what happens.
I went back into the DQ channels to look at the TF from AO injection to REFLDC (which is easy to do with this kind of noise injection TF).
I fear that REFL does not seem to have as much phase under the resonance as we have modeled, lacking about 10-20 degrees. This could result from the zero in the REFL DC response that we've modeled at ~200ish Hz is actually higher. I'll look into what affects the frequency of that feature.
It is, of course, possible, that this measurement doesn't properly cancel out the various digital effects, but the REFLDC phase curves do seem to settle to (+/-) 90 after the pole as expected.
DTT XML file is attached.
This CPU load may have been me deleting some old frame files, to see if that would allow daqd to come back to life.
Daqd was segfaulting, and behaving in a manner similar to what is described here: (stack exchange link). However, I couldn't kill or revive daqd, so I rebooted the FB.
Things seem ok for now...
When I came in, Rossa was booted to Ubuntu 10. I tried rebooting to select 12, but couldn't ever successfully boot again. Since Diego was setting up Allegra from scratch, I've wiped and done the same with Rossa.
I've done some preliminary modeling to see if there is a good candidate for an IR DARM control signal that is available before the AS55 sign flip. From a DC sweep point of view, ASDC/(TRX+TRY) may be a candidate for further exploration.
As a reminder, both Finesse and MIST predict a sign flip in the AS55 Q control signal for DARM in the PRFPMI configuration, at a CARM offset of around 118pm.
The CARM offset where this sign flip occurs isn't too far off of where we're currently losing lock, so we have not had the opportunity to switch DARM control off of ALS and over to the quieter IR RF signal of AS55.
Here are simulated DC DARM sweep plots of our current PRFPMI configuration, with a whole bunch of potential signals that struck me.
Although the units of most traces are arbitrary in each plot (to fit on the same scale), each plot uses the same arbitrary units (if that makes any sense) so slopes and ratios of values can be read off.
In the 300 and 120pm plot, you can see that the zero crossing of AS55 is at some considerable DARM offset, and normalizing by TRX doesn't change much about that. "Hold on a second," I hear you say. "Your first plots said that the sign flip happens at around 120pm, so why does the AS55 profile still look bad at 50pm?!" My guess is that, probably due to a combination of Schnupp and arm length asymmetry, CARM offsets move where the peak power is in the DARM coordinate. This picture makes what I mean more clear, perhaps:
Thus, once we're on the other side of the sign flip, I'm confident that we can use AS55 Q without much problem.
Now, back to thoughts about an interim signal:
ASDC by itself does not really have the kind of behavior we want; but the power out of AS as a fraction of the ARM power (i.e. ASDC/TRX in the plot) seems to have a rational shape, that is not too unlike what the REFLDC CARM profile looks like.
Why not use POPDC or REFLDC? Well, at the CARM offsets we're currently at, POPDC is dominated by the PRC resonating sidebands, and REFLDC has barely begun to decline, and at lower CARM offsets, they each flatten out before the peak of the little ASDC hill, and so don't do much to improve the shape. Meanwhile, ASDC/TRX has a smooth response to points within some fraction of the DARM line width in all of the plots.
Thus, as was discussed at today's meeting, I feel it may be possible to lock DARM on ASDC/(TRX+TRY) with some offset, until AS55 becomes feasible.
(In practice, I figure we would divide by the sum of the powers, to reduce the influence of the DARM component of just TRX; we don't want to have DARM/DARM in the error signal for DARM)
Two caveats are:
[Code and plots live in /svn/trunk/modeling/PRFPMI_radpressure]
I've been trying to figure out why daqd keeps crashing, but nothing is fixed yet.
I commented out the line in /etc/inittab that runs daqd automatically, so I could run it manually. Each time I run it ( with ./daqd -c ./daqdrc while in c1/target/fb), it churns along fine for a little while, but eventually spits out something like:
[Thu Oct 16 11:43:54 2014] main profiler warning: 1 empty blocks in the buffer
[Thu Oct 16 11:43:55 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:43:56 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:43:57 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:43:58 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:43:59 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:44:00 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:44:01 2014] main profiler warning: 0 empty blocks in the buffer
[Thu Oct 16 11:44:02 2014] main profiler warning: 0 empty blocks in the buffer
GPS time jumped from 1097520250 to 1097520257
FATAL: exception not rethrown
I looked for time disagreements between the FB and the frontends, but they all seem fine. Running ntpdate only corrected things by 5ms. However, looking through /var/log/messages on FB, I found that ntp claims to have corrected the FB's time by ~111600 seconds (~31 hours) when I rebooted it on Monday.
Maybe this has something to do with the timing that the FB is getting? The FE IOPs seem happy with their sync status, but I'm not personally currently aware of how the FB timing is set up.
On Monday, Jamie suggested checking out the situation with FB's RAID. Searching the elog for "empty blocks in the buffer" also brought up posts that mentioned problems with the RAID.
I went to the JetStor RAID web interface at http://192.168.113.119, and it reports everything as healthy; no major errors in the log. Looking at the SMART status of a few of the drives shows nothing out of the ordinary. The RAID is not mounted in read-only mode either, as was the problem mentioned in previous elogs.
I've added (TRX-TRY)/(TRX+TRY) to the DC DARM sweep plots, and it looks like an even better candidate. The slope is closer to linear, and it has a zero crossing within ~10pm of the true DARM zero across the different CARM offsets, so we might not even need to use an intentional DARM offset.
I've been able to repeatedly get off of ALS and onto (TRY-TRX)/(TRY+TRX). Nevertheless, lock is lost between arm powers of 10 and 20.
I do the transition at the same place as the CARM->SqrtInv transition, i.e. arm powers about 1.0 Jenne started a script for the transition, and I've modified it with settings that I found to work, and integrated it into the carm_cm_up script. I've also modified carm_cm_down to zero the DARM normalization elements.
I was thwarted repeatedly by the frequent crashing of daqd, so I was not able to take OLTFs of CARM or DARM, which would've been nice. As it was, I tuned the DARM gain by looking for gain peaking in the error signal spectrum. I also couldn't really get a good look at the lock loss events. Once the FB is behaving properly, we can learn more.
Turning over to difference in transmission as an error signal naturally squashes the difference in arm transmissions:
I was able to grab spectra of the error and control signals, though I did not take the time to calibrate them... We can see the high frequency sensing noise for the transmission derived signals fall as the arm power increases. The low frequency mirror motion stays about the same.
So, it seems that DARM was not the main culprit in breaking lock, but it is still gratifying to get off of ALS completely, given its out-of-loop-noise's strong dependence on PSL-alignment.
BUT, what we really need (instead of just the DC sweeps) is the DC sweep with the uncertainty/noise displayed as a shaded area on the plot, as Nic did for us in the pre-CESAR modelling.
I've taken a first stab at this. Through various means, I've made an estimation of the total noise RMS of each error signal, and plotted a shaded region that shows the range of values the error signal is likely to take, when the IFO is statically sitting at one CARM offset.
I have not included any effects that would change the RMS of these signals in a CARM-offset dependent way. Since this is just a rough first pass, I didn't want to get carried away just yet.
For the transmission PDs, I measured the RMS on single arm lock. I also measured the incident power on the QPDs and thorlabs PDs for an estimate of shot noise, but this was ridiculously smaller than the in-loop RIN. I had originally though of just plotting sensing noise for the traces (i.e. dark+shot), because the amount of seismic and frequency noise in the in-loop signal obviously depends on the loop, but this gives a very misleading, tiny value. In reality we have RIN from the PRC due to seismic noise, angular motion of the optics, etc., which I have not quantified at this time.
So: for this first, rough, pass, I am simply multiplying the single transmission noise RMSs by a factor of 10 for the coupled RMS. If nothing else, this makes the SqrtInv signal look plausible when we actually practically find it to be plausible.
For the REFL PDs, I misaligned the ITMs for a prompt PRM reflection for a worst-case shot noise situation, and took the RMS of the spectra. (Also wrote down the dark RMSs, which are about a factor of 2 lower). I then also multiplied these by ten, to be consistent with the transmission PDs. In reality, the shot noise component will go down as we approach zero CARM offset, but if other effects dominate, that won't matter.
Enough blathering, here's the plot:
Now, in addition to the region of linearity/validity of the different signals, we can hopefully see the amount of error relative to the desired CARM offset. (Or, at least, how that error qualitatively changes over the range of offsets)
This suggests that we MAY be able to hop over to a normalized RF signal; but this is a pretty big maybe. This signal has the response of the quotient of two nontrivial optical plants, which I have not yet given much thought to; it is probably the right time to do so...
After the second of the two recent power outages, the outlet powering Chiara's external drive for local backups didn't come back. The modification to the backup script I made correctly identified that the drive wasn't mounted, and happily logged its absence and didn't try to stuff the internal drive with a copy of itself. However, I hadn't checked the logs to see if the backups were proceeding until today... maybe I should set up an email alert for these, too.
I plugged the external drive into a live outlet, and mounted the 40mBackup drive with: sudo udisks --mount /dev/sdc1, which is a helpful command that puts the drive at /media/40mBackup as it should be, based on the drive label.
sudo udisks --mount /dev/sdc1
The /cvs/cds backup is now proceeding, to make up for lost time.
I spent some time trying to debug our inability to get MICH onto REFL165Q while the arms are held off with ALS, to no real success.
I set up our usual repeatable situation of PRMI on 33 I&Q, arms held off with ALS. I figured that it may help to first sideband lock on REFL55, since 165 is looking for the f2 sidebands and maybe there is some odd offset between the locking points for f1 and f2 or other weirdness.
REFL 55 settings:
Demod angle 98->126 (was previously set for PRY locking)
PRCL = 0.5 * REFL55 I (UGF of ~200 Hz) (FM gain unchanged from REFL33 situation of -0.02)
MICH = 0.125 * REFL55 Q (UGF of ~60Hz) (same FM gain as 33)
Some REFL55 offset adjusting had to be done in order to not disturb the 33-initiated lock when handing off.
I also adjusted POP110 phase to zero the Q when locked, and switched the triggering over to 110I
The PRMI can acquire lock like this with arms held off with ALS, no problem.
Here, I tried to hop over to 165. PRCL was no problem, needing a +1 on 165I. However, I had no success in handing off MICH. I twiddled many knobs, but none that provably helped.
I saw indications that the sensing angle in 165 is small (~20deg), which is not consistent with current knowledge of the cavity lengths. We last interferometrically measured the PRC length by letting the PRMI swing and looking at sideband splitting in POP110. At LLO, they did a length measurement by looking at demod angle differences in PRMI carrier vs. sideband locking. (alog8562) This might be worth checking out...
I took some spectra of the error signals and MC2 Trans RIN with the loops off (blue) and on (red) during the current conditions of daytime seismic noise.
Short report: Further frustrated by 165 tonight. The weird thing is, the procedure I'm trying with the arms held off on ALS (i.e. excitation line in MICH and PRCL, adjust relative gains to make the signs and magnitudes mach, ezcastep over) works flawlessly with the ETMs misaligned. One can even acquire SB PRMI lock on 165 I&Q, with 80-90 degrees of demod angle between MICH and PRCL. The only real difference in REFL55 settings for misaligned vs. ALS-offset arms is an extra factor of two in the FM gains to maintain the same UGF, so I hoped that the matrix elements for 165 with misaligned arms would hold for ALS-offset arms.
Alas, no such fortune. I still have no clear explanation for why we can't get MICH on 165Q with the arms held off on ALS.
I also gave a quick try to measuring the PRCL->REFL55 demod phase difference between carrier and sideband lock (with arms misaligned), and got something on the order of 55 degrees, which really just makes me think I wasn't well set up / aligned, rather than actually conveying information about the PRC length...