I was trying to debug why the NPRO PZT is all over the place, and it turns out that the new FSS SLOW script is not actually running.
The BLINKY is blinking, but the script is not running. I wasn't able to figure out how to kill the broken Docker thing, but if the code reports that its running but actually does not, we should probably just put back the old perl or python script that ran before. I don't know how to debug this current issue, but the IMC locks will be limited in length due to this servo being broken. Whoever knows about this, please stop that Docker PID and we can just run the old python script on megatron.
I also tried to post a trend plot, but the minute trends don't yet reach the current date (!!!). They seem to have stopped recording a few days ago, so I guess the Framebuilder still needs some help or its tough to figure out things like when exactly the new SLOW servo stopped working.
The problem with trends was due to the epics data collection process (standalone_edc) that runs on c1sus. When all the FEs were rebooted earlier this week, this process was started automatically, but for some reason it hasn’t been doing its thing and sending epics data to the framebuilder. I restarted it just now, and it’s working again. Until this problem is sorted out, we need to remember to check on this process after rebooting c1sus.
I stopped the Docker PID script and started the old python script on megatron. Instructions on how to do this are here.
On optimus I ran:
On megatron I ran:
However, the daemon service keeps failing and restarting. So currently the FSSSlow is not running. I do not know how to debug this script.
On a side note, I tested the docker service by restarting it, and it was working. From the logs, it seems like it got stuck because it could not find C1:IOO-MC_LOCK channel which occurs when c1psl epics servers fail or get stuck. The blinker on this script runs when the script is running but it does not stop if the script gets stuck somewhere. If someone decides to use this script in the future, they would need to correct error catching so that no reply from caget looks like an error and the script restarts rather than keep trying to get the channel value. Or the blinker implementation should change in the script so that it displays a stuck state.
Whoever knows about this, please stop that Docker PID and we can just run the old python script on megatron.
I've moved the FSS Slow PID script running to megatron through systemd daemons. The script is working as expected right now. I've updated megatron motd and the always running scripts page here.
FSS SLOWDC slider is at around 0.
Please someone relock this at ~-4.0 to exploit some last juice of the fruit.
See this entry for the details of the operating point.
The FSS Slow DC servo was turned off.
As MCL stabilizes the MC_F (Fast PZT), we no longer need to use the laser temp to do so.
In other word, if you like to turn off the MCL servo for some reason, we need to turn it on in order to keep the MC locked.
A couple of weeks ago, I was trying to modernize the python version of the FSS Slow temperature control loops, when I accidentally ended up deleting it . There was no svn backup. So the old Perl PID script has been running for the last few days.
Today, I checked out the latest version that Andrew and co. have running in the PSL lab. I had to make some important modifications for the script to work for the 40m setup.
python FSSSlow.py -i FSSSlowPy.ini
Then I stopped the Perl process on megatron by running
sudo initctl stop FSSslow
and started the Python process by running
sudo initctl start FSSslowPy
I have now committed the files FSSSlow.py and FSSSlowPy.ini to the 40m svn. Things seem to be stable for the last 20 mins or so, let's keep an eye on this though - although we had been running the Python PID loop for some months, this version is a slightly modified one.
The initctl stuff still isn't very robust - I think both the Autolocker and the FSS slow servos have to be manually restarted if megatron is shutdown/restarted for whatever reason. It doesn't seem to be a problem with the initctl routine itself - looking at the logs, I can see that init is trying to start both processes, but is failing to do so each time. To be investigated. The wiki procedure to restart this process is up to date.
GV Edit 0000 25 Aug 2017: I had to add a line to the script that checks MC transmission before enabling the PID loop. Change has been committed to svn. Now, when the MC loses lock or if the PSL shutter is kept closed for an extended period of time, the temperature loop doesn't rail.
[yinzi, craig, gautam]
Yinzi had translated the Perl PID script used to implement the discrete-time PID control, and had implemented it with Andrew at the PSL lab. Today afternoon we made some minor edits to make this suitable for the FSS Slow loop (essentially just putting the right channel names into her Python script). I then made an init file to run this script on megatron, and it looks to be working fine over the last half hour of observation or so. I am going to leave things in this state over the weekend to see how it performs.
We have been running with just the MC2 Transmission QPD for angular control of the IMC for a couple of months now because the WFS loops seemed to drag the alignment away from the optimum. We did the following to try and re-engage the WFS feedback:
GV addendum 23Nov2016: The WFS have been working well over the last few days - I've had to periodically (~ once in a day) run the WFS reflief script to keep the outputs to the suspension PIT and YAW DOFs below 50cts, but the WFS aren't dragging the alignment away as we had noticed before. The only thing I did differently is to follow Rana's suggestion and set the RF offsets with the MC unlocked as opposed to locked. I've added a line to the script to remind the user to do so... Also, note that EricQ has recently cleaned up the scripts directory to remove the numerous obsolete scripts in there...
I made a very slighly updated version of Yinzi's script that pulls the channel names and actuator hardstop limits from an external .ini config file. The idea was to avoid having as many versions of the script as there are implimentations of it. Seems like slighly better practice, but maybe I'm wrong. The config files are also easier to read. Its posted on the elog (PSL:1758) with lastest on the 40mSVN .../trunk/CTNLab/current/computing/scripts .
If you're working off her first implimentation 'RCAV_thermalPID.py' then there is an indent issue after the if statement on line 88: only line 89 should be indended. If you deactivate the debug flag then the script fails to read in PID factors and dies.
PSL NPRO PZT voltage showed large low frequency (hour timescale) excursions on the control room StripTool trace, leading me to suspect the slow servo wasn't working as expected. Yesterday evening, I keyed the unresponsive c1psl crate at ~9 PM PST, and had to run the burtrestore to get the PMC locking working. I must have pressed the wrong button on burtgooey or something, because all the FSS_SLOW channels were reset to 0. What's more, their values were not being saved by the hourly burt-snap script, so I don't have any lookback on what these values were. There isn't any detailed record on the elog about what the optimal values for these are, and the most recent reference I could find was Ki=0.1, Kp=Kd=0, which is what I've set it now to. The servo isn't running away, so I'm leaving things in this state, PID tuning can be done later.
I also added the FSS Slow servo channels to the burt snapshot requirement file at /cvs/cds/caltech/target/c1psl/autoBurt.req, and confirmed that the snapshots are getting the channels from now onwards.
While looking at the req file, I saw a bunch of *_MOPA* channels and also several other currently unused channels. Probably would benefit from going through these and commenting out all the legacy channels, to minimize disk space wastage (though we compress the snapshot files every few years anyways I guess).
Reminder that this (unrelated) issue still needs to be looked into... Note also that the new vacuum system does not have burt snapshot set up (i.e. it is still trying to get the old channels from the c1vac1 and c1vac2 databases, which while has significant overlap with the new system, should probably be setup correctly).
Given that op340m showed some undesired behavior, and that the FSS slow seems prone to railing lately, I've moved the FSS slow servo job over to megatron in the same way I did for the MC autolocker.
Namely, there is an upstart configuration (megatron:/etc/init/FSSslow.conf), that invokes the slow servo. Log file is in the same old place (/cvs/cds/caltech/logs/scripts), and the servo can be (re)started by running:
controls@megatron|~ > sudo initctl start FSSslow
Maybe this won't really change the behavior. We'll see
Today Q moved the FSS slow servo over to some init thing on megatron, and some time ago he did the same thing to the MC auto locker script. It isn't working though.
Even though megatron was rebooted, neither script started up automatically. As Diego mentioned in elog 10823, we ran sudo initctl start MCautolocker and sudo initctl start FSSslow, and the blinky lights for both of the scripts started. However, that seems to be the only thing that the scripts are doing. The MC auto locker is not detecting lockloses, and is not resetting things to allow the MC to relock. The MC is happy to lock if I do it by hand though. Similarly, the blinky light for the FSS is on, but the PSL temperature is moving a lot faster than normal. I expect that it will hit one of the rails in under an hour or so.
The MC autolocker and the FSS loop were both running earlier today, so maybe Q had some magic that he used when he started them up, that he didn't include in the elog instructions?
I ssh'd in, and was able to run each script manually successfully. I ran the initctl commands, and they started up fine too.
We've seen this kind of behavior before, generally after reboots; see ELOGS 10247 and 10572.
In the plot it is shown the behaviour of the PSL-FSS_SLOWDC signal during the last week; the blue rectangle marks an approximate estimate of the time when the scripts were moved to megatron. Apart from the bad things that happened on Friday during the big crash, and the work ongoing since yesterday, it seems that something is not working well. The scripts on megatron are actually running, but I'll try and have a look at it.
I reset the threshold to +6666 counts (the aligned MC transmission is ~16000 for the TEM00 mode) so that it only turns on when we're in a good locked state.
I've now also trended the MOPA output power for the last 200 days to check a possible correlation with the FSS reflected power. See attachment.
The trend shows that the laser power has decayed but it seems that the FSS reflected power has done it even faster: 30% drop in the FSS vs 7% for the MOPA in the last 60 days (attachment n.2).
Just before working on the FSS today, I noticed that the VCO RF output level was set incorrectly.
This should ALWAYS be set so as to give the maximum power in the first order diffracted sideband. One should set this by maximizing the out of lock FSS RFPD DC level to max.
The value was at 2.8 on the VCOMODLEVEL slider. In the attached plot (taken with the FSS input disabled) you can see that this puts us in the regime where the output power to the FSS is first order sensitive to the amplitude noise on the electrical signal to the AOM. This is an untenable situation.
For adjusting the power level to the FSS, we must always use the lamba/2 plate between the AOM and the RC steering mirrors. This dumps power out to the side via a PBS just before the periscope.
What is the Transfer Function of the suspension of the reference cavity? What were the design requirements? What is the Q and how well does the eddy current damping work? What did Wolfowitz know about the WMD and when? Who cooked the RTV in there and why didn't we use Viton??
To get to the bottom of these questions, today I shook the cavity and measured the response. To read out the pitch and yaw modes separately, I aligned the input beam to be misaligned to the cavity. If the beam is mis-aligned in yaw, for example, the transmitted light power becomes first order sensitive to the yaw motion of the cavity.
In the attached image (10 minute second-trend), you can see the second trends for the transmitted and relfected power. The first ringdown comes from the pitch or vertical mode. The second (shorter) one comes from the yaw misalignment and the yaw shake.
To achieve the up/down shake, I leaned onto the table and pumped it at its eigenfrequency. For the yaw shake, I put two fingers on the RC can's sweater and pushed with several pounds of force at the yaw eigenfrequency (2.6 Hz). For the vertical, I jumped up and down at half the vertical eigenfrequency (4 Hz).
I also made sure that the .SCAN field on these EPICS records were set to 9 so that there is no serious effect from a beating between the eigenfrequency and the EPICS sample rate.
f_vert = 4 Hz
tau_vert = 90 seconds
Q_vert = 1000 (yes, that number over there has 3 zeros)
f_hor = 2.6 Hz
tau_hor = 30 seconds
Q_hor = 250
This is an absurd and probably makes us very sensitive to seismic noise - let's make sure to open up the can and put some real rubber in there to damp it. My guess is that these high Q modes
are just the modes of the last-stage steel spring / pendulum.
I measured the open loop gain of the FSS (as usual, I have multiplied the whole OLG by 10dB to account for the forward loop gain in the box). I used a source level of -20 dBm and made sure this was not saturating by changing the level.
Its clear that the BW is limited by the resonance at ~1.7 MHz. Does anyone know what that is?
EO resonance in the RC path?
I measured the RF spectrum coming out the FSS RFPD to look for saturations - its close to the hairy edge. This is with the 8x power increase from my AOM drive increase. I will increase the FSS's modulation frequency which will lower the Q and gain of the PD to compensate somewhat. The lower Q will also gain us phase margin in the FSS loooop.
I put in a bi-directional 20 dB coupler (its only rated down to 30 MHz, but its only off by ~0.3 dB at 21 MHz) between the RFPD and the FSS box. I looked at the time series on the 300 MHz scope and measured the power spectrum.
The peak signal on the scope was 40 mV; that translates to 400 mV at the RFPD output. Depending on whether the series resistor in the box is 20 or 50 Ohms, it means the MAX4107 is close to saturating.
As you can see from the spectrum, its mostly likely to hit its slew rate limit (500 V/us) first. Actually its not going to hit the limit: but even getting within a factor of 10 is bad news in terms of distortion.
Besides the multiples of the modulation frequency, you can see that most of the RMS comes from the strange large peaks at 137.9 and 181.1 MHz. Anyone know what these are from?
On the middle plot above, I have enabled the 20 MHz BW limit so you can see how much the amplitude goes down when only the 21.5 MHz SB is included. You can also see from the leftmost plot that once in awhile there is some 400mV/10ns slewing. Its within a factor of 10 of the slew rate limit.
This is the error point spectrum - it is filled with huge multiples of ~75 kHz as Yoichi noticed a couple years ago.
I tried to use the netgpib.py package to read out the Agilent 4395, but the SVN had been corrupted by someone saving over the netgpib.py package. To get it to work on rosalba I reverted to the previous version, but whoever is busy hacking on netgpib.py needs to checkin the original package and work on some test code instead.
I also noticed that the default output format for the AG4395.py file is in units of Watts. This is kind of dumb - we need someone to develop this package a little as Yoichi did for the SRS785.
I made some measurements of the FSS box today, to have TFs for a loop model, but also to see what the difference between the different inputs was.
As a reminder, the FSS box takes the error signal from the MC servo, does some filtering, and sends out two outputs: one to the laser PZT via KojiBox and Thorlabs HV amplifier, and one to be summed with the PMC modulation signal to the PC. Rana found the schematic at D040105
The MC error signal currently enters via a port called "IN1", but there is also a "Test 1 in," which experiences different filtering. I measured the TFs from each of these inputs to both the FAST and PC outputs. There is also an IN2, that is added after the offset point, but was not able to make a good measurement, for reasons unknown. From these TFs, I inferred the difference between the PC and FAST path, as well as the difference between IN1 and Test 1 in.
Specifically, I plugged the cable that is usually connected to the MC servo output, labelled "TO FSS BOX", into the RF out of the AG4395. I then took a BNC cable from the FAST out, or PC out, and fed it into a mini circuits DC block (BLK-89-S+), and then into input A, after checking on a scope that the signal was roughly zeroed and not too huge. Unbeknownst to me at the time, the PC drive output can be pretty big, and could potentially fry the analyzer's input. Fortunately, I think I avoided this fate.
A ~1.3 MHz bump can be seen here, which would conspire with the bump in the demod board I measured yesterday, to steal even more phase around 1MHz. Maybe we can modify the FSS box to help our gain peaking situation out.
The data is attached.
As EQ pointed out recently, we're injecting into the FSS error point just after an RF pi filter, but before the VGA. We wondered what the weird filter impedance was doing to our signal if we inject after it. I used LISO to model this FSS common section and attach the plots.
The first plot shows the TF between the Test 1 input and the AD602 VGA input. This is NOT the input that we are actually using.
The second plot shows the TF between the IN1 port (which we are actually using) and the VGA input.
Neither of them shows the 1 MHz bump that we see in the measurements, so I suspect that the board has been modified...the hunt continues. We've got to pop the top of the TTFSS and take photos and measure from IN1 to VGA input.
** FSScomm.fil is now in the LISO SVN. The following command line will run it with two different cases and cat the PDF files into one. If you use an auto-refresh PDF viewer like Okular or Mac Preview, its a nicer display than the usual GNUplot window:
./mfil FSScomm.fil; sleep 1; pdftk FSScomm_run*.pdf cat output FSScomm.pdf
In 1X1, there is a box labelled "FSS REF" below a KEPCO HV supply. This box had a power cable that wasn't actually connected to any power. I removed said cable.
- connected the TTFSS cables (FSS fast goes directly to NPRO PZT for now)
- measured the reference cavity 21.5 MHz EOM drive to be 17.8 dBm
- turned on the HV for the FSS phase correcting EOM (aka PC) drive
- connected and turned on the reference cavity temperature stabilization
- connected the RefCav TRANS PD
- fine tuned the RefCav REFL PD angle
On Monday, I hooked up an AG4395 to the PMC error point (using the active probe). The idea was to take a spectrum of the PMC error point every time the FSS PC drive RMS channel indicated an excursion from the nominal value. An initial look at the results don't suggest that this technique is particularly informative. I'll have to think more about a workaround, but please share your ideas/thoughts if you have some.
Also, the feature in the spectrum at ~110 kHz makes me suspect some kind of loop instability. I'll measure the IMC loop OLG at the next opportunity.
I had set up the 4395 to do this automatically a few years ago, but it looked at the FSS/IMC instead. When the PCDRIVE goes high there is this excess around ~500 kHz in a broad hump.
But the IMC loop gain changes sometimes with alignment, so I don't know if its a loop instability or if its laser noise. However, I think we have observed PCDRIVE to go up without IMC power dropping so my guess is that it was true laser noise.
This works since the IMC is much more sensitive than PMC. Perhaps one way to diagnose would be to lock IMC at a low UGF without any boosts. Then the UGF would be far away from that noise making frequency. However, the PCDRIVE also wouldn't have much activity.
I'm not sure yet what this points to as the best gain settings. We can of course explore more of the space. I'm going to leave it at 13/23.5, which leaves the PC RMS at ~1.5 and the FAST Monitor at ~6.0.
I changed the value of the nominal FSS common gain in the PSL Settings screen (It's called the 'FSS global gain' there). To get to this screen: sitemap -> PSL_main -> PSL_settings. The MC autolocker reads these settings from the screen and uses those values. Now the FSS returns to this value of 13 that Jamie has chosen. For the past few days, it's been going back to the old value of 10.1 . The FAST gain was already set to this 23.5 value.
Since Rana's overhaul of the IMC, the FSS input offset had been sitting at zero.
Over the last day or so, I had noticed the MC refl wall striptool trace looking noisier, and earlier this evening, we were suffering from a fair amount of downtime due to IMC unlocks, and failure to autolock for times on the order of ten minutes.
While we had used ezcaservo for this in the past, I set the FSS offset manually tonight. Namely, I popped open a dataviewer trace of MC_F, and scanned the FSS offset to make MC_F go to zero. It required a good amount of offset, 4.66 V according to the FSS screen. I did this while the FSS slow servo was on, which held the FSS Fast output at zero.
That was four hours ago; MC_F is still centered on zero, and we have not had a single IMC unlock since then. The MC refl trace is thinner too, though this may be from nighttime seismic.
The RefCav is locked and aligned. I changed the fast gain sign by changing the jumper setting on the TTFSS board. The RefCav visibility is 70%. The FSS loop ugf is about 80 kHz (plot attached. there is 10 dB gain in the test point path. this is why the ugf is at 10 dB when measured using in1 and in2 spigots on the front of the board.) with FSS common gain max out at 30 dB. There is about 250 mW coming out of the laser and 1 mW going to RefCav out of the back of the PMC. So the ugf can be made higher at full power. I have not made any changes to account for the PMC pole (the FSS is after the PMC now). The FSS fast gain was also maxed out at 30 dB to account for the factor of 5 smaller PZT actuation coefficient - it used to be 16 dB according to the (previous) snap shot. The RefCav TRANS PD and camera are aligned. I tuned up the phase of the error signal by putting cables in the LO and PD paths. The maximum response of the mixer output to the fast actuator sweep of the fringe was with about 2 feet of extra cable in the PD leg.
I am leaving the FSS unlocked for the night in case it will start oscillating as the phase margin is not good at this ugf.
Brilliant! This is the VERY way how the things are to be conquered!
The RefCav is locked and aligned. I changed the fast gain sign by changing the jumper setting on the TTFSS board. The RefCav visibility is 70%. The FSS loop ugf is about 80 kHz (plot attached) with FSS common gain max out at 30 dB. There is about 50 mW coming out of the laser and a few mW going to RefCav out of the back of the PMC. So the ugf can be made higher at full power. I have not made any changes to account for the PMC pole (the FSS is after the PMC now). The FSS fast gain was also maxed out at 30 dB to account for the factor of 5 smaller PZT actuation coefficient - it used to be 16 dB according to the (previous) snap shot. The RefCav TRANS PD and camera are aligned. I tuned up the phase of the error signal by putting cables in the LO and PD paths. The maximum response of the mixer output to the fast actuator sweep of the fringe was with about 2 feet of extra cable in the PD leg.
I've been measuring a bunch of transfer functions of the FSS related stuffs.
There are a lot to be analyzed yet, but here I put one mystery I'm having now.
Maybe I'm missing something stupid, so your suggestions are welcome.
Here is a conceptual diagram of the FSS control board
RF PD -->--[Mixer]-----[Sum Amp]------>--[Common Gain]--->----[Fast Gain]----[Filter]--> NPRO PZT
^ | ^ | |
| V | V |
LO ---->------- TP1 IN TP2 -->---[Filter]--[High Volt. Amp.] --> Phase Corrector
What I did was first to measure a "normal" openloop transfer function of the FSS servo.
The FSS was operated in the normal gain settings, and a signal was injected from "IN" port.
The open loop gain was measured by TP1/TP2.
Now, I disconnected the BNC cable going to the phase corrector to disable the PC path and locked the ref. cav.
only using the PZT. This was done by reducing the "Common Gain" and "Fast Gain" by some 80dB.
Then I measured the open loop gain of this configuration. The UGF in this case was about 10kHz.
I also measured the gain difference between the "normal" and "PZT only" configurations by injecting
a signal from "IN" and measuring TP3/TP2 and TP4/TP3 with both configurations (The signal from the Mixer was
disconnected in this measurement).
The first attachment shows the normal open loop gain (purple) and the PZT only open loop gain scaled by the
gain difference (about 80dB). The scaled PZT open loop gain should represent the open loop gain of the PZT
path in the normal configuration. So I expected that, at low frequencies, the scaled PZT loop TF overlaps the normal
open loop TF.
However, it is actually much larger than the normal open loop gain.
When I scale the PZT only TF by -30dB, it looks like the attachment #2.
The PZT loop gain and the total open loop gain match nicely between 20kHz and 70kHz.
Closer look will show you that small structures (e.g. around 30kHz and 200kHz) of the two
TFs also overlap very well. I repeated measurements many times and those small structures are always there (the phase is
also consistently the same). So these are not random noise.
I don't know where this 30dB discrepancy comes from. Is it the PC path eating the PZT gain ?
I have measured many other TFs. I'm analyzing these.
Here is the TO DO list:
* Cavity response plot from AOM excitation measurements.
* Cavity optical gain plot.
* Reconstruct the open loop gain from the electric gain measurements and the optical gain above.
* Using a mixer and SR560(s), make a separate feedback circuit for the PZT lock. Then use the PC path
to measure the PC path response.
* See the response of the FSS board to large impulse/step inputs to find the cause of the PC path craziness.
f (MHz) | before filter (dBm) | after filter (dBm)
21.5 | -12.8 -13.1
43 -24 -46
64.5 -50 < -80
86 -64 < -80
The attached plot shows the beam scans of the beam leaking from the back mirror of the PMC to the BS cube that first turns the S-pol beam 90 deg to the AOM and then transmits the AOM double passed and polarization rotated P-pol beam to the reference cavity. The beam from the PMC is mode matched to the AOM using a single lens f=229 mm. The ABCD file is attached. The data were taken with VCO control voltage at 5 V. We then reduced the voltage to 4 V to reduce the astigmatism. Tara has the data for the beam scan in this configuration in his notebook.
The beam from AOM is mode matched to the reference cavity using a single lens f=286.5 mm. The ABCD file is attached.