Nodus (solaris) is dead, long live Nodus (ubuntu).
Diego and I are smoothing out the Kinks as they appear, but the ELOG is running smoothly on our new machine.
SVN is working, but your checkouts may complain because they expect https, and we haven't turned SSL on yet...
However, the PRMI would not acquire lock with the arms held off resonance.
This is entirely my fault.
Last week, while doing some stuff with PRY, I put this filter in SUS_PRM_LSC, to stop some saturations from high frequency sensing noise
After the discussion at today's meeting, it struck me that I might have left it on. Turns out I did.
20 degree phase lag at 200Hz can explain the instability, some non-flat shape at few hundreds of Hz explains the non 1/f shape.
Sorry about all that...
I have completed all of the model modifications and medm screen updates to allow for feedback from the transmon QPD pitch and yaw signals to the ITMs. Now, we can design and test actual loops...
The signals come from c1sc[x/y] to c1rfm via RFM, and then go to c1ass via dolphin.
Out of curiosity about the RFM+dolphin delay, I took a TF of an excitation at the end SUS model (C1:SUS-ETM[X/Y]_QPD_[PIT/YAW]_EXC) to the input FM in the ASC model (C1:ASC-ETM[X/Y]_QPD_[PIT/YAW]_IN1). All four signals exhibit the same delay of 122usec. I saved the dtt file in Templates/ASC/transmonQPDdelay.xml
This is less than a degree under 20Hz, so we don't have to worry about it.
Some locking efforts tonight; many locklosses due to PRC angular motion. Furthest progress was arm powers of 15, and I've stared at the corresponding lockloss plot, with little insight into what went wrong. (BTW, lastlock.sh seems to catch the lock loss reliably in the window)
CARM and DARM loops were measured not long before this lock loss, and had nominal UGFs (~120Hz, ~20deg PM). However, there was a reasonably clear 01 mode shape at the AS camera, which I did nothing to correct. Here's a spectrum from *just* before the lockloss, recovered via nds. Nothing stands out to me, other than a possible loss of DARM optical gain. (I believe the references are the error signal spectra taken in ALS arms held away + PRMI on 3F configuration)
The shape in the DARM OLTF that we had previously observed and hypothesized as possible DARM optical spring was not ever observed tonight. I didn't induce a DARM offset to try and look for it either, though.
Looking into some of the times when I was measuring OLTFs, the AS55 signals do show coherence with the live DARM error signal at the excitation frequencies, but little to no coherence under 30Hz, which probably means we weren't close enough to swap DARM error signals yet. This arm power regime is where the AS55 sign flip has been modeled to be...
A fair amount of time was spent in pre-locking prep, including:
Since the Nodus switch, the offsite backup scripts (scripts/backup/rsync.backup) had not been running successfully. I tracked it down to the weird NFS file ownership issues we've been seeing since making Chiara the fileserver. Since the backup script uses rsync's "archive" mode, which preserves ownership, permissions, modification dates, etc, not seeing the proper ownership made everything wacky.
Despite 99% of the searches you do about this problem saying you just need to match your user's uid and gid on the NFS client and server, it turns out NFSv4 doesn't use this mechanism at all, opting instead for some ID mapping service (idmapd), which I have no inclination of figuring out at this time.
Thus, I've configured /etc/fstab on Nodus (and the control room machines) to use NFSv3 when mounting /cvs/cds. Now, all the file ownerships show up correctly, and the offsite backup of /cvs/cds is churning along happily.
I just stumbled upon this while poking around:
Since the great crash of June 2014, the scripts backup script has not been workingon op340m. For some reason, it's only grabbing the PRFPMI folder, and nothing else.
Megatron seems to be able to run it. I've moved the job to megatron's crontab for now.
I've set up nodus to start the ELOG on boot, through /etc/init/elog.conf. Also, thanks to this, we don't need to use the start-elog.csh script any more. We can now just do:
controls@nodus:~ $ sudo initctl restart elog
I also tweaked some of the ELOG settings, so that image thumbnails are produced at higher resolution and quality.
Given that op340m showed some undesired behavior, and that the FSS slow seems prone to railing lately, I've moved the FSS slow servo job over to megatron in the same way I did for the MC autolocker.
Namely, there is an upstart configuration (megatron:/etc/init/FSSslow.conf), that invokes the slow servo. Log file is in the same old place (/cvs/cds/caltech/logs/scripts), and the servo can be (re)started by running:
controls@megatron|~ > sudo initctl start FSSslow
Maybe this won't really change the behavior. We'll see
I ssh'd in, and was able to run each script manually successfully. I ran the initctl commands, and they started up fine too.
We've seen this kind of behavior before, generally after reboots; see ELOGS 10247 and 10572.
In order to fix ELOG search, I have started running ELOG v2.9.2 on Nodus.
Sadly, due to changes in the software, we can no longer use one global write password. Instead, we must now operate with registered users.
Based on recent elog users, I'll be creating user accounts with the following names, using the same old ELOG write password. (These will be valid across all logbooks)
All of these users will be "Admins" as well, meaning they can add new users and change settings, using the "Config" link.
Let me know if I neglected to add someone, and sorry for the inconvenience.
RXA: What Eric means to say, is that "upgrading" from Solaris to Linux broke the search and made us get a new elog software that;s worse than what we had.
Steve and I switched chiara over to the UPS we bought for it, after ensuring the vacuum system was in a safe state. Everything went without a hitch.
Also, Diego and I have been working on getting some of the new computers up and running. Zita (the striptool projecting machine) has been replaced. One think pad laptop is missing an HD and battery, but the other one is fine. Diego has been working on a dell laptop, too. I was having problems editing the MAC address rules on the martian wifi router, but the working thinkpad's MAC was already listed.
Turns out that, as the martian wifi router is quite old, it doesn't like Chrome; using Firefox worked like a charm and now also giada (the Dell laptop) is on 40MARS.
So, despite having registered users, it turns out that the "Author" field is still open for editing when making posts. I.e. we don't really need to make new accounts for everyone.
Thus, I've made a user named "elog" with the old write password that can write to all ELOGs.
(Also, I've added a user called "jamie")
The BS was showing some excess motion. I think I've fixed it. Order of operations:
I'm not sure how this might have gotten switched on...
I lost the connecting cable from the CM to the AO input (unlabeled).
This afternoon, I labelled both ends of this cable, and reconnected it to the MC servo board.
Two plots from tonight:
Lock loss. Based on the fact that it looked like the DARM servo was running away, Rana posited an effective sign flip in the DARM loop, perhaps due to a parasitic angular feedback mechanism.
While Jenne was probing the IFO at lower powers, we noticed a sudden jump in ASDC. Found the GPS time and fed it to the lockloss plotter. Seems fairly evident that some sudden ETMX motion was to blame. (~2urad kick in yaw)
As Jenne mentioned, I created a model of the DARM OLG to see why we have so little phase margin. However, it turns out I can explain the phase after all.
Chris sent me his work for the aLIGO DARM phase budget, which I adapted for our situation. Here's a stacked-area plot that shows the contributions of various filters and delays on our phase margin, and a real measurement from a few days ago .
This isn't so great! Informed by Chris's model, the digital delays look like: (Here I'm only listing pure delays, not phase lags from filters)
This adds up to about 570usec, 20.5 degrees at 100Hz, largely due to the sheer number of computer hops the transmission loops involve.
As a check, I divided the measured OLG by my model OLG, to see if there is any shape to the residual, that my model doesn't explain. It looks like it fits pretty well. Plot:
So, unless we undertake a bunch of computer work, we can only improve our transmission loops through our control filter design.
Everything I used to generate these plots is attached.
I added UGF servos for the DRMI DoFs, after creating a library block for the servos. I also deleted the FMs before the phase rotation, since we can just do it afterwards in other existing FMs. I've only added the MICH and PRCL buttons to the LSC screen because in the end, I feel like a dropdown is better, but I just wanted to get it running quickly tonight. The LSC model and the UGF block have been committed to the svn.
We were able to use the PRCL UGF servo successfully, as Jenne was exploring MICH offset space.
I've installed the very fresh ELOG 3.0, for nothing else than the new built in text editor which has a LATEX capable equation editor built right in.
Check out this sweet limerick:
The restored offset script used old tdsavg calls that our workstations can't do, and didn't include things like the transmon QPDs. I've written yet another offset script that uses cdsutils averaging to do the thing, and committed to the svn.
We want to have some angular control of the arms during lock acquistion.
In single arm lock, Diego and I shook the TMs and measured how the QPDs responded. (I would've liked to do a swept sine in DTT, but the user envelope function still isnt' working!)
For now, we can close simple loops with QPD sensor and ITM actuator, but, as Rana pointed out to Diego and me today, this will drive some amount of the angular cavity degree of freedom that the QPD doesn't sense. So, ideally, we want to come up with the right combination of ITM and ETM motion that lies entirely within the DoF that the QPD senses.
I created a rudimentary loop for Yarm yaw, was able to get ~20Hz for the upper UGF, a few mHz for the lower, but it was starting to leak into the length error signal. Further tweaking will be neccesary...
I've upgraded our cdsutils installation to v382; there have been some changes to pydv which will allow me to implement the auto y-scaling on our lockloss plots.
After some brief testing, things seem to still work...
Chiara threw another network hissy fit. Dmesg was spammed with a bunch of messages like eth0: link up appearing rapidly.
eth0: link up
Some googling indicated that this error message in conjuction with the very ethernet board and driver that Chiara had in use could be solved by updating with an appropriate driver from the manufacturer.
In essence, I followed steps 1-7 from here: http://ubuntuforums.org/showthread.php?t=1661489
So far, so good. We'll keep an eye out to see how it works...
I was looking into the status of IPC communications in our realtime network, as Chris suggested that there may be more phase missing that I thought. However, the recent continual red indicators on a few of the models made it hard to tell if the problems were real or not. Thus, I set out to fix what I could, and have achieved full green lights in the CDS screen.
The frontend models have been svn'd. The BLRMs block has not, since its in a common cds space, and am not sure what the status of its use at the sites is...
The UGF servos have been moved to the control point, are are once again totally linear!
I've measured the sensing for each of the arms, by using our calibrated oplevs, in terms of QPD counts per micron. It is:
ETMY: QPD PIT / OPLEV PIT = 22.0 count/urad
QPD YAW / OPLEV YAW = 17.1 count/urad
ITMY: QPD PIT / OPLEV PIT = -6.0 count/urad
QPD YAW / OPLEV YAW = 5.9 count/urad
ETMX: QPD PIT / OPLEV PIT = 16.6 count/urad
QPD YAW / OPLEV YAW = -9.3 count/urad
ITMX: QPD PIT / OPLEV PIT = 4.0 count/urad
QPD YAW / OPLEV YAW = -6.0 count/urad
In the absence of a lens, the QPD would be significantly more sensitive to cavity axis translation than tilt, and thus about equally sensitive to ITM and ETM angle. However, there are lenses on the end tables. I didn't go out and look at them, but found some elogs from Annalisa that mentioned 1m focal length lenses. Back-of-the-envelope calculations convince me that this can plausibly lead to the above sensitivity ratios.
I used these quantities to come up with an actuation matrix for the ASC loops, and measured the effective plant seen by the FM, fitted it to some poles( looks like zpk(,-2*pi*[1.47+3.67i,1.47-3.67i],160); ), and designed a control servo. Here is the designed loop:
The servo works on both arms, both DoFs. A DTT measurement agrees with the designed loop shape, up to a few degrees, which are probably due to the CDS delay. The RMS of the QPD error signals goes down by about 20dB, and are currently dominated by the bounce mode, so maybe we can try to sneak in some resonant gain...?
Once we confirm that they work when locking, we can write up and down lines into the locking scripts...
PMC realigned again... The transmission was down to 0.70, and the MC was having a hard time trying to autolock.
EDIT: Sleepy Eric doesn't understand loops. The conditions for this observation included active oplev loops. Thus, obviously, looking at the in-loop signal after the ASC signl joins the oplev signal will produce this kind of behavior.
After some talking with Rana, I set out on making an even better-er QPD loop. I made some progress on this, but a new mystery halted my progress.
I sought to have a more physical undertanding of the plant TF I had measured. Earlier, I had assumed that the 4Hz plant features I had measured for the QPD loops were coming from the oplev-modified pendulum response, but this isn't actually consistent with the loop algebra of the oplev servos. I had seen this feature in both the oplev and qpd error signals when pushing an excitation from the ASC-XARM_PIT (and so forth) FMs.
However, when exciting via the SUS-ETMX-OLPIT FMs (and so forth), this feature would not appear in either the QPD or oplev error signals. That's weird. The outputs of these two FMs should just be summed, right before the coil matrix.
I started looking at the TF from ASC-YARM_PIT_OUT to SUS-ETMY_TO_COIL_1_2, which should be a purely digital signal routing of unity, and saw it exhibit the phase shape at 4Hz that I had seen in earlier measurements. Here it is:
I am very puzzled by all of this. Needs more investigation.
As mentioned in a previous ELOG, in single arm lock, I measured the QPD response with respect to the calibrated oplev signals. They were:
ETMX: QPD PIT / OPLEV PIT = 16.6 count/urad
QPD YAW / OPLEV YAW = -9.3 count/urad
ITMX: QPD PIT / OPLEV PIT = 4.0 count/urad
QPD YAW / OPLEV YAW = -6.0 count/urad
For reference, one microradian of either ITM or ETM motion produces about 60um of ETM beam spot displacement, compared to the spot size of ~5mm.
However, given the lenses on the end tables that are used for green mode matching, that the IR transmitted beam also passes through, the QPDs are not directly imaging the ETM spot position; if they were, they would have equal sensitivity to ITM and ETM motion due to our flat/curved arm geometry.
From this data, I calculated the actuation coefficients for each DoF as , and similarly for the ITMs, where the d's come from the table above. However, it occurs to me that maybe this isn't the way to go... I'll mention this later.
Up until now, at every turn, I had not properly been thinking about how the oplev loop plays into all of this. I went to the foton filters, and grabbed the loop and plant models for the ETMY oplev, and constructed the closed loop gain, 1/1+G, and the modified plant, P/1+G, which is what the ASC loop sees as its plant.
Here, the purple trace explains all of the features I was confused about earlier.
With this in hand, I set up to design a loop to satisfy our motivations.
To do this, I inverted the oplev closed loop plant pole around 4Hz to smooth the whole thing out. Here's a comparison of the measured OLG with what I modelled.
There's a little bit of phase discrepency around 10Hz, but I think it looks about right overall.
So, here's the part that counts: How does this actually perform? I took spectra of the QPD error signals, the relevant OpLev signals as out of loop sensors, the PDH error signal and transmitted RIN while single arm locked, with this loop off, and on for all 4 DoFs simultaneously.
This is what makes me think I may need to revisit the actuation matrix. If I did it wrong, I am driving the "invisible" quadrant of the cavity angular DoFs, and this could be what is injecting noise into the oplevs.
In the end, I have a better understanding of what is going on, and I don't think we're quite there yet.
Although the QPD loops are less than ideal right now, I made changes to the ASC model to trigger the QPD loops on and off politely, depending on TRX and TRY. The settings are exposed on the ASC screen. However, I have not yet exposed the FM triggering that I also set up to make sure the integrator doesn't misbehave if the arm loses lock. We probably don't want to trigger them on at anything lower than arm powers of about 1.0.
I've tested the triggering by randomly turning LSC mode on and off, and making sure that the optics don't recieve much of a kick as the QPD loops engage a few seconds after the LSC boosts do, or when lock is lost. This works as long as there isn't much of a DC offset befire the loops are engaged. (Under 20 counts or so is fine)
As a side note, I was going to use the TRIG_SIG signals sent via the LSC model via SHMEM blocks for the ASC triggering, but oddly, the data streams that made it over were actually the MICH and SRCL TRIG_SIGs, instead of XARM and YARM as labelled. I double checked the simulink diagrams; everything seemed fine to me. In any case, ASS was already recieving TRX and TRY directly via RFM, so I just piped those over to the ASC block. This way is probably better anyways, because it directly references the arm powers, instead of the less obvious LSC triggering matrix.
Since Rana's overhaul of the IMC, the FSS input offset had been sitting at zero.
Over the last day or so, I had noticed the MC refl wall striptool trace looking noisier, and earlier this evening, we were suffering from a fair amount of downtime due to IMC unlocks, and failure to autolock for times on the order of ten minutes.
While we had used ezcaservo for this in the past, I set the FSS offset manually tonight. Namely, I popped open a dataviewer trace of MC_F, and scanned the FSS offset to make MC_F go to zero. It required a good amount of offset, 4.66 V according to the FSS screen. I did this while the FSS slow servo was on, which held the FSS Fast output at zero.
That was four hours ago; MC_F is still centered on zero, and we have not had a single IMC unlock since then. The MC refl trace is thinner too, though this may be from nighttime seismic.
Does netgpibdata/TFSR785 work at the 40m currently?
It does appear to work here. However, I've since supplanted TFSR785 and SPSR785 with SRmeasure, which has some simpler command line options for directly downloading a manually configured measurement. I've also set up a git repository for the gpib scripts I've done at https://github.com/e-q/netgpibdata, which could be easier than grabbing the whole 40m directory.
The MC autolocker hasn't been so snappy recently, and has been especially fussy today. Previously, the mcup script was triggered immediately once the transmission was above a certain threshold. However, this could waste time if it was just an errant flash. Hence, I've added a 0.5 second delay and a second threshold check before mcup is triggered.
After breaking the lock 5ish times, it does seem to come back quicker.
Tonight, we transitioned CARM from ALS directly to REFL11 I at 25% Mich Offset.
We attempted the transition twice, the first time worked, but we lost lock ~5 seconds after full transition due to a sudden ~400Hz ringup (see attached lockloss plot). The second barfed halfway, I think because I forgot to remove the CARM B offset from the first time
The key to getting to zero CARM offset with CARM and DARM on ALS is ekeing out every bit of PRMI phase margin that you can. Neither MICH nor PRCL had their RG filters on and I tweaked the MICH LP to attenuate less and give back more phase (the HF still isn't the dominant RMS source.) PRCL had ~60 degrees phase margin at 100Hz UGF, MICH had ~50 deg at 47Hz UGF. The error signals were comparitively very noisy, but we only cared that they held on. Also important was approaching zero slooooooooowly, and having the CARM and DARM UGF servo excitations off, because they made everything go nuts. Diego stewarded the MICH and PRCL excitation amplitudes admirably.
Oddly, and worringly, the arm powers at zero CARM offset were only around 10-12. Our previous estimations already include the high Xarm loss, so I'm not sure what's going on with this. Maybe we need to measure our recycling gain?
I hooked up the SR785 by the LSC rack to the CM board after the first success. For the second trial, I also took TFs with respect to CM slow, but they looked nowhere near as clean as the normal REFL11 I channel; I didn't really check all the connections. I will be revisiting the whole AO situation soon.
In any case, I think we're getting close...
I have removed REFLDC and the SR560 offsetter from the CM board IN2. Now, analog AS55 I lives there, for our single arm testing. (Analog I has more of the single arm Y PDH signal in it). REFL11 has been reconnected to IN1.
With ITMX super misaligned, Diego and I locked the Y-arm with the AO path on AS55, ultimately at 4kHz bandwidth, but with plenty of gain margin. We didn't allocate the gains too intelligently, and had the CM board input gain slider maxed out, but plenty of headroom in the digital and AO sliders, making it inconvenient to up the UGF even more, to engage the super boosts. However, since this is just a test case to make sure we still can AO lock, I'm not too worried about this.
Since LSC FMs and such had changed around, old recipies didn't neccesarily work 1:1. Diego is writing a script for the current recipe, and will post an elog with the steps.
Gains and signs are able to be tracked by loop TFs, the real sticking point is a stable crossover. We used the 1.6k:80 hardware filter in the CM board to give the AO Path a 1/f shape in the crossover region, and undid it digitally in the CM_SLOW input FM. However, we do use a 300:80 in the MC2 sus FM to make the digital loop like 1/f^2 around the crossover, once a little bit of AO has come in to pull up the digital loop's phase. We used the CARM filter bank to do this, so I think we should be able to use a similar technique to do it in the PRFPMI case, as long as the coupled cavity pole is around ~100Hz.
Attached are a few OLTFs from the progression.
At the lunch meeting, we were thinking about the exess high frequency content of the MICH control signal, which leads to railing against the FM output limiter and lock loss. I propsed that POPDC sensor/ADC noise was the culprit.
In short, I was wrong. Just now, I locked the PRMI with a MICH offset as we normally do, and then froze the POPDC output. The MICH spectrum did not change in any noticible way.
However, increasing the analog ASDC whitening gain showed a direct improvement of the error signal noise floor, and thus a reduction in the control signal spectrum. I.e. we have been suffereing from ASDC ADC noise.
We had previously set the ASDC whitening gain so that half fringe of the PRMI would be well within the ADC range, but since we're actually only ever going to 25%, I feel fine upping this gain. Interestingly, when increasing the whitening gain by 12dB, the control signal spectrum has fallen by more like 20 dB in the 400Hz-1kHz region, which is great.
The only potential hurdle I can think of is that when we start reducing the MICH offset at zero CARM offset, we may approach ADC saturation in ASDC before we can hand off to RF signals, in which case we would have to dynamically lower the whitening gain, which introduces offsets, which could get hairy. But, since MICH railing has been directly seen to lead to lock-loss, I'd rather solve that problem first.
Just now we had another EPICS freeze. The network was still up; i.e. I could ssh to chiara and fb, who both seemed to be working fine.
I could ping c1lsc successfully, but ssh just hung. fb's dmesg had some daqd segfault messages, so I telnet'ed to daqd and shut it down. Soon after, EPICS came back, but this is not neccesarily because of the daqd restart...
Unfortunately, we only had one good CARM offset reduction to powers of about 25, but then my QPD loop blew it. We spent the vast majority of the night dealing with headaches and annoyances.
Things that were a pain:
I've remeasured the QPD ASC sensing coefficients, and figured out what I did weird with the actuation coefficients. I've rearranged the controller filters to be able to turn on the boost in a triggered way, and written Up/Down scripts that I've tested numerous times, and Jenne has used as well; they are exposed on the ASC screen.
All four loops (2 arms * pit,yaw), have their gains set for 8Hz UGF, and have 60 degrees of phase margin. The loop shape is the same as the previous ELOG post. Here is the current on/off performance. The PDH signals (not shown, but in attached xml) show no extra noise, and the low frequency RIN goes down a bit, whic is good. The oplevs error signals are a bit noisier, but I suppose that's unavoidable. The Y-arm performs a bit better than the X-arm.
The up/down scripts don't touch the filters' trigger settings at all, just handles switching the input and output and clearing history. FM1 contains the boost which is intended to have a longer trigger delay than the filters themselves.
For future reference, I've taken spectra of our various RFPDs while the PRMI was sideband locked on REFL33, using a 20dB RF coupler at the RF input of the demodulator boards. The 20dB coupling loss has been added back in on the plots. Data files are attached in a zip.
I also completely removed the cabling for REFLDC -> CM board, since it doesn't look like we plan on using it anytime in the immediate future.
After some discussion with Koji, I've asked Steve to order some SBP-30+ bandpass filters as a quick and cheap way to help out REFL33. (Also some SBP-60+ for 55MHz, since we only have 1*fmod and 2*fmod bandpasses here in the lab).
I have been able to recover the ability to sit at zero CARM offset while the PRMI is locked on RELF33 and CARM/DARM are on ALS, effectively indefinitely. However, I feel like the transmon QPDs are not behaving ideally, because the reported arm powers freqently go negative as the interferometer is "buzzing" through resonance, so I'm not sure how useful they'll be as normalizing signals for REFL11. I tried tweaking the DARM offset to help the buildup, since ALS is only roughly centered on zero for both CARM and DARM, but didn't have much luck.
Turning off the whitening on the QPD segments seems to make everything saturate, so some thinking with daytime brain is in order.
How I got there:
It turns out triggering is more important than the phase margin story I had been telling myself. Also, I lost a lot of time to needing demod angle change in REFL33. Maybe I somehow caused this when I was all up on the LSC rack today?
We have previously put TRX and TRY triggering elements into the PRCL and MICH rows, to guard against temporary POP22 dips, because if arm powers are greater than 1, power recylcing is happening, so we should keep the loops engaged. However, since TRX and TRY are going negative when we buzz back and forth through the resonsnace, the trigger row sums to a negative value, and the PRMI loops give up.
Instead, we can used the fortuitously unwhitened POPDC, which can serve the same function, and does not have the tendancy to go negative. Once I enabled this, I was able to just sit there as the IFO angrily buzzed at me.
Here are my PRMI settings
REFL33 - Rotation 140.2 Degrees, -89.794 measured diff
PRCL = 1 x REFL33 I; G = -0.03; Acquire FMs 4,5; Trigger FMs 2, 9; Limit: 15k ; Acutate 1 x PRM
MICH = 1 x REFL33 Q, G= 3.0, Acquire FMs 4,5,8; Trigger FM 2, 3; Limit: 30k; Actuate -0.2625 x PRM + 0.5 x BS
Triggers = 1 x POP22 I + 0.1 * POPDC, 50 up 5 down
Just for kicks, here's a video of the buzzing as experienced in the control room
I've fixed the gpib scripts for the SR785 and AG4395A to output data in the same format as expected by older scripts when called by them. In addition, there are now some easier modes of operation through the measurement scripts SRmeasure and AGmeasure. These are on the $PATH for the main control room machines, and live in scripts/general/netgpib
Case 1: I manually set up a measurement on the analyzer, and just want to download / plot the data.
Make sure you have a yellow prologix box plugged in, and can ping the address it is labeled with. (i.e. 'vanna'). Then, in the directory you want to save the data, run:
SRmeasure -i vanna -f mydata --getdata --plot
This saves mydata_(datetime).txt and mydata_(datetime).pdf in the current directory.
In all cases, AGmeasure has the identical syntax. If the GPIB address is something other than 10, specifiy it with -a, but this is rarely the case.
Case 2: I want to remotely specify a measurement
Rather than a series of command line arguments, which may get lost to the mists of time, I've set the scripts up to use parameter files that serve as arguments to the scripts.
Get the templates for spectrum and TF measurements in your current directory by running
Set the parameters with your text editor of choice, such as frequency span, filename output, whether to create a plot or not, then run the measurement:
Case 3: I want to compare my data with previous measurements
In the template parameter files, there is an option 'plotRefs', that will automatically plot the data from files whose filenames start with the same string as the current measurement.
If, in the "#" commented out header of the data file, there is a line that contains "memo:" or "timestamp:", it will include the text that follows in the plot legend.
There are also methods to remotely trigger an already configured measurement, or remotely reset an unresponsive instrument. Options can be perused by looking at the help in SRmeasure -h
I've tested, debugged, and used them for a bit, but wrinkles may remain. They've been svn40m committed, and I also set up a separate git repository for them at github.com/e-q/netgpibdata
I have re-enabled the second whitening stage switching on each quadrant of each end's QPD whitening board, to try and avoid saturations at full power. Looking at the spectra while single arm locked, I confirmed that the FM2 whitening switch works as expected. FM1 should be left on, as it is still hard-wired to whiten.
The oscillations in the Y QPD still exist. Jenne is updating the schematics on the DCC.
Went to zero CARM offset on ALS; transmission QPDs are still saturating :(
Maybe we need to switch off all whitening.
As Koji found one of the spare channels of the ALS/FOL RF amplifier box nonfunctional yesterday, I pulled it out to fix it. I found that one of the sma cables did not conduct.
It was replaced with a new cable from Koji. Also, I rearranged the ports to be consistent across the box, and re-labeled with the gains I observed.
It has been reinstalled, and the Y frequency counter that is using one of the channels shows a steady beat freq.
I cannot test the amplitude of the green X beat at this time, as Koji is on the PSL table with the PSL shutter closed, and is using the control room spectrum analyzer. However, the dataviewer trace for the fine_phase_out_Hz looks like free swinging cavity motion, so its probably ok.
I'm playing around with the lastest ALS fool feedforward on the Yarm, and I like what I'm seeing.
First, I verified that I could reproduce the TF shapes in ELOG 11041, which I was able to do with a gain of +9.3 and FMs 5 and 6 in the FF module.
Then, I locked the arm on ALS with full bandwidth, and on POY with the settings currently used the MC module, and took their spectra as references. (I put an excitation on the arm at 443Hz to line them up to the same arbitrary units.)
Then, with ALS at its usual 100Hz UGF and boosts on, the Fool path on, and the MC FM set to trigger on/off at 0.8/0.5, I slowly brought ALS towards zero offset and saw it pop right into resonance. I then manually triggered the PDH boosts.
Here are some spectra, showing that, with the Fool path on:
Once the PDH loop is running, the ALS loop can be switched out at the CARM FM output without much of an effect beyond a small kick.
However, looking at the loop shapes, maybe I got lucky here. I took the usual injection TFs at the MC FM, the CARM FM, and at ETMY, to get the overall OLG; all of them have >0.9 coherence pretty much everywhere except the first two points.
As desired, the PDH loop looks pretty normal.
I have no intuition about how the fooled CARM loop should look, since this is even more complicated than a two-loop system.
I don't currently know what is causing the odd feature in the overall at ~50Hz, and it spooked me out when I saw the multiple UGF crossings. The only thing I could think of happening there is the pole in the ALS phase tracker boost. I turned it off, and remeasured; the feature persists...
To wrap it up, here's something I think is pretty cool. Here's what happens when I give ETMY a 1000 count position step impulse (no ramp). [Here, CARM is on ALS with G=12, but only FM5 on]
Although the arm was controlled with IR before the kick, ALS maintained control. As soon as ALS brought the arm back towards resonance, the PDH loop picked it right back up.
Some random notes:
DTT data is attached, in case it's useful to anyone!
Koji raised a good question about the step response I wrote about. Namely, if the UGF of the ALS servo is around 100Hz, we would expect it to settle with a characteristic time on the order of tens of milliseconds, not seconds, as was seen in the plot I posted.
I claim that the reason for the slow response was the fact that the feedforward FM stayed on after the kick, despite the MC filter bank being triggered off. Since it has filters that look like a oscillator at 1Hz, the ringdown or exponential decay of this filter's output in response to the large impulsive output of the PDH control signal just before being triggered down would slowly push the ALS error signal around through the feedforward path.
Given this reasoning, this should be helped by adding output triggering to the FF filter that uses the MC trigger matrix row, as I wanted to do anyways. I have now added this into the LSC model (as well as DQ at 2kHz for the MC_CTRL_FF_OUT), and the impulse response is indeed much quicker.
In the following plot, I hit ETMY with a five sample, 5000 amplitude, impulse (rather than a step, as I did yesterday). The system comes back to PDH lock after ~40ms.
X end QPD has recieved 0.2+0.4 absorptive ND filters. Y end QPD got one at 0.6. This appears to have mitigated the saturations for now; the unwhitened signals no longer go negative. The digital gains have been reset.
At about 10AM, the C1LSC frontend stopped reporting any EPICS information. The arms were locked at the time, and remained so for some hours, until I noticed the totally whited-out MEDM screens. The machine would respond to pings, but did not respond to ssh, so we had to manually reboot.
Soon thereafter, we had a global 15min EPICS freeze, and have been in a weird state ever since. Epics has come back (and frozen again), but the fast frontends are still wonky, even when EPICS is not frozen. Intermittantly, the status blinkers and GPS time EPICS values will freeze for multiple seconds at a time, sporadically updating. Looking at a StripTool trace of an IOPs GPS time value shows a line with smooth portions for about 30 seconds, about 2 minutes apart. Between this is totally jagged step function behavior. C1LSC needed to be power cycled again; trying to restart the models is tough, because the EPICS slowdown makes it hard to hit the BURT button, as is needed for the model to start without crashing.
The DAQ network switch, and martian switch inside were power cycled, to little effect. I'm not sure how to diagnose network issues with the frontends. Using iperf, I am able to show hundreds of Mbit/s bandwidth betweem the control room machines and the frontends, but their EPICS is still totally wonky.
What can we do???