FSS SLOW control did not drift during the lock at night with MCL path working and AC coupled.
The IR sensitive Olympus 570 camera gives us a really nice view of these IR beams. Its actually a lot better than what you can get with the analog IR viewers:
I measured the phase noise of the LO output of the FSS box from the DAQ. I'm attaching the results.
As we expected, the measurement is limited by the internal phase noise of the Marconi.
The measurement was done as shown in this diagram.
The differences between this setup and the one used previously is the lack of the 50 Ohm terminator in the mixer output and
that the SR560 readout with the G=100 should come before the first SR560 via T, so as not to be spoiled by the high noise of the G=1 SR560.
I removed the 50 Ohm in-line terminator when I did the measurement with the SR785. The for some reason I was getting more noise, so I removed it.
Now I put it back in and I did the measurement with the DAQ. I also moved the SR560 that amplifies the signal for the DAQ, Tee'ing it with the input of the in-loop SR560.
Now the setup looks like this:
And the phase noise that I measure is this:
Comparing it with the phase noise measured with the previous setup (see entry 3506), you can see that the noise effectively is reduced by about a factor of 2 above 10 Hz.
With the setup now working, we should now test the power filtering for the crystal and amplifier.
I have put in a new nominal value for the FSS fast gain: 21.5 dB.
There is an oscillation peak in the MC error point spectra around 41.5 kHz if the FSS gain is set too high. I used the 4395 to have a look at the MC error point, and saw that if I set the FSS fast gain any lower than about 18 dB, the peak wasn't getting any smaller than -41 dBm. If I set the fast gain any higher than about 26 dB the peak wouldn't get any larger than about -34 dBm.
However, if I set the gain to 19.5dB, the PC RMS drive is consistently above 2 V, which isn't so good. If I crank the gain up to 27 dB or more, the PC RMS will stay below 0.9 V, which is great.
As a compromise, I have decided on 21.5 dB as the new FSS fast gain. This puts the oscillation peak at about -39.5 dBm, and the PC RMS around 1.6 V.
I changed the nominal gain by ezcawrite C1:PSL-STAT_FSS_NOM_F_GAIN 21.5. This sets the nominal value so that the FSS screen's fast slider doesn't turn red at the new value. And, since the MC autolocker reads this epics channel and puts that into the gain during the mcup script, the MC autolocker now uses this new gain. For reference, it used to be set to 23.5 dB.
ezcawrite C1:PSL-STAT_FSS_NOM_F_GAIN 21.5
A few weeks ago, on Jul 24, Rana and I measured the phase noise of the FSS frequency box (aka the 'Kalmus Box'). See elog entry 3286.
That time, for some reason, we measured a phase noise higher than we expected; higher than that of the Marconi.
I repeated the measurement today using the SR785 spectrum analyzer. Here is the result:
(The measurement of July 24 on the plot was not corrected for the loop gain. The UGF was at about 30 Hz)
To make sure that my measurement procedure was correct, I also measured the combined phase noise of two Marconis. I then confirmed the consistency of that with what already measured by other people in the past (i.e. Rana elog entry 823 in the ATF elog).
This time the noise seemed reasonable; closer to the Marconi's phase noise, as we would expect. I don't know why it was so bad on July 24.
The shoulder in the Marconi-to-Marconi measurement between 80Hz and 800Hz is probably due to the phase noise of the other Marconi, the one used as LO.
I'm going to repeat the measurement connecting the setup to the DAQ, and locking the Marconi to the Rubidium standard.
Ultimately, the goal is to measure the phase noise of the new Sideband Frequency Generation Box of the 40m Upgrade.
I've taken the FSS frequency generation box out of the 1Y1 rack. It's sitting on one of the electronics benches. I'm measuring its phase noise.
Today we measured the phase noise of the oscillator used for the FSS.
The source is a Wenzel crystal at about 21.5MHz that Peter Kalmus built some time ago.
We basically used the same technique that Frank and Megan have been using lately to measure the Marconi's phase noise.
Today we just did a quick measurement but today next week we are going to repeat it more carefully.
Attached is a plot that shows the measurement calibrated for a UGF at about 60 Hz. The noise is compared to that specified by Wenzel for their crystal.
The noise is bigger than that of the MArconi alone locked to the Rubidium standard (see elog entry). We don't know the reason for sure yet.
We'll get back to this problem next week.
I reconnected the RF signal to the FSS and to the FSS' EOM so that we could lock the refcav again.
I then started a 3 sec. period trianglewave on the AOM drive amplitude to see if there is a direct coupling from RIN to Frequency. Ideally we will be able to measure this by looking at the RCTRANS and the FSS-FAST.
1) Check cable between RFPD and FSS box for quality. Replace with a good short cable.
2) Using a directional coupler, look at the RFPD output in lock on a scope with 50 Ohm term.
I suspect its a lot of harmonics because we're overmodulating to compensate for the bad
3) Purchase translation stages for the FSS mode matching lenses. Same model as the PMC lenses.
Fix the mode matching.
4) Get the shop to build us up some more bases for the RFPDs on the PSL such as we have for the LSC.
Right now they're on some cheesy Delrin pedestals. Too soft...
5) Dump the beam reflected off the FSS RFPD with a little piece of black glass or a razor dump.
Anodized aluminum is no good and wiggles too much.
I found that FSS SLOW servo is not engaged. Is this intentional test to keep the NPRO temp constant?
This is making the FSS Fast unhappy (~ -7.5V right now).
Yes, I had turned it off while looking for the PSL/X AUX beat, and forgot to turn it back on.
I will post an elog with more detail this evening, but I found a temperature which restored the X green beatnote at its nominal amplitude (-30dBm) with no mode hops within +-1 IR beat GHz, and offloaded the slow offset slider to the X-end laser crystal dial. I will look for the Y beatnote after dinner.
Currently the control room analyzer is hooked up to recieve the Y IR and green beats; no X signals.
I was trying to debug why the NPRO PZT is all over the place, and it turns out that the new FSS SLOW script is not actually running.
The BLINKY is blinking, but the script is not running. I wasn't able to figure out how to kill the broken Docker thing, but if the code reports that its running but actually does not, we should probably just put back the old perl or python script that ran before. I don't know how to debug this current issue, but the IMC locks will be limited in length due to this servo being broken. Whoever knows about this, please stop that Docker PID and we can just run the old python script on megatron.
I also tried to post a trend plot, but the minute trends don't yet reach the current date (!!!). They seem to have stopped recording a few days ago, so I guess the Framebuilder still needs some help or its tough to figure out things like when exactly the new SLOW servo stopped working.
The problem with trends was due to the epics data collection process (standalone_edc) that runs on c1sus. When all the FEs were rebooted earlier this week, this process was started automatically, but for some reason it hasn’t been doing its thing and sending epics data to the framebuilder. I restarted it just now, and it’s working again. Until this problem is sorted out, we need to remember to check on this process after rebooting c1sus.
I stopped the Docker PID script and started the old python script on megatron. Instructions on how to do this are here.
On optimus I ran:
On megatron I ran:
However, the daemon service keeps failing and restarting. So currently the FSSSlow is not running. I do not know how to debug this script.
On a side note, I tested the docker service by restarting it, and it was working. From the logs, it seems like it got stuck because it could not find C1:IOO-MC_LOCK channel which occurs when c1psl epics servers fail or get stuck. The blinker on this script runs when the script is running but it does not stop if the script gets stuck somewhere. If someone decides to use this script in the future, they would need to correct error catching so that no reply from caget looks like an error and the script restarts rather than keep trying to get the channel value. Or the blinker implementation should change in the script so that it displays a stuck state.
Whoever knows about this, please stop that Docker PID and we can just run the old python script on megatron.
I've moved the FSS Slow PID script running to megatron through systemd daemons. The script is working as expected right now. I've updated megatron motd and the always running scripts page here.
FSS SLOWDC slider is at around 0.
Please someone relock this at ~-4.0 to exploit some last juice of the fruit.
See this entry for the details of the operating point.
The FSS Slow DC servo was turned off.
As MCL stabilizes the MC_F (Fast PZT), we no longer need to use the laser temp to do so.
In other word, if you like to turn off the MCL servo for some reason, we need to turn it on in order to keep the MC locked.
A couple of weeks ago, I was trying to modernize the python version of the FSS Slow temperature control loops, when I accidentally ended up deleting it . There was no svn backup. So the old Perl PID script has been running for the last few days.
Today, I checked out the latest version that Andrew and co. have running in the PSL lab. I had to make some important modifications for the script to work for the 40m setup.
python FSSSlow.py -i FSSSlowPy.ini
Then I stopped the Perl process on megatron by running
sudo initctl stop FSSslow
and started the Python process by running
sudo initctl start FSSslowPy
I have now committed the files FSSSlow.py and FSSSlowPy.ini to the 40m svn. Things seem to be stable for the last 20 mins or so, let's keep an eye on this though - although we had been running the Python PID loop for some months, this version is a slightly modified one.
The initctl stuff still isn't very robust - I think both the Autolocker and the FSS slow servos have to be manually restarted if megatron is shutdown/restarted for whatever reason. It doesn't seem to be a problem with the initctl routine itself - looking at the logs, I can see that init is trying to start both processes, but is failing to do so each time. To be investigated. The wiki procedure to restart this process is up to date.
GV Edit 0000 25 Aug 2017: I had to add a line to the script that checks MC transmission before enabling the PID loop. Change has been committed to svn. Now, when the MC loses lock or if the PSL shutter is kept closed for an extended period of time, the temperature loop doesn't rail.
[yinzi, craig, gautam]
Yinzi had translated the Perl PID script used to implement the discrete-time PID control, and had implemented it with Andrew at the PSL lab. Today afternoon we made some minor edits to make this suitable for the FSS Slow loop (essentially just putting the right channel names into her Python script). I then made an init file to run this script on megatron, and it looks to be working fine over the last half hour of observation or so. I am going to leave things in this state over the weekend to see how it performs.
We have been running with just the MC2 Transmission QPD for angular control of the IMC for a couple of months now because the WFS loops seemed to drag the alignment away from the optimum. We did the following to try and re-engage the WFS feedback:
GV addendum 23Nov2016: The WFS have been working well over the last few days - I've had to periodically (~ once in a day) run the WFS reflief script to keep the outputs to the suspension PIT and YAW DOFs below 50cts, but the WFS aren't dragging the alignment away as we had noticed before. The only thing I did differently is to follow Rana's suggestion and set the RF offsets with the MC unlocked as opposed to locked. I've added a line to the script to remind the user to do so... Also, note that EricQ has recently cleaned up the scripts directory to remove the numerous obsolete scripts in there...
I made a very slighly updated version of Yinzi's script that pulls the channel names and actuator hardstop limits from an external .ini config file. The idea was to avoid having as many versions of the script as there are implimentations of it. Seems like slighly better practice, but maybe I'm wrong. The config files are also easier to read. Its posted on the elog (PSL:1758) with lastest on the 40mSVN .../trunk/CTNLab/current/computing/scripts .
If you're working off her first implimentation 'RCAV_thermalPID.py' then there is an indent issue after the if statement on line 88: only line 89 should be indended. If you deactivate the debug flag then the script fails to read in PID factors and dies.
PSL NPRO PZT voltage showed large low frequency (hour timescale) excursions on the control room StripTool trace, leading me to suspect the slow servo wasn't working as expected. Yesterday evening, I keyed the unresponsive c1psl crate at ~9 PM PST, and had to run the burtrestore to get the PMC locking working. I must have pressed the wrong button on burtgooey or something, because all the FSS_SLOW channels were reset to 0. What's more, their values were not being saved by the hourly burt-snap script, so I don't have any lookback on what these values were. There isn't any detailed record on the elog about what the optimal values for these are, and the most recent reference I could find was Ki=0.1, Kp=Kd=0, which is what I've set it now to. The servo isn't running away, so I'm leaving things in this state, PID tuning can be done later.
I also added the FSS Slow servo channels to the burt snapshot requirement file at /cvs/cds/caltech/target/c1psl/autoBurt.req, and confirmed that the snapshots are getting the channels from now onwards.
While looking at the req file, I saw a bunch of *_MOPA* channels and also several other currently unused channels. Probably would benefit from going through these and commenting out all the legacy channels, to minimize disk space wastage (though we compress the snapshot files every few years anyways I guess).
Reminder that this (unrelated) issue still needs to be looked into... Note also that the new vacuum system does not have burt snapshot set up (i.e. it is still trying to get the old channels from the c1vac1 and c1vac2 databases, which while has significant overlap with the new system, should probably be setup correctly).
Given that op340m showed some undesired behavior, and that the FSS slow seems prone to railing lately, I've moved the FSS slow servo job over to megatron in the same way I did for the MC autolocker.
Namely, there is an upstart configuration (megatron:/etc/init/FSSslow.conf), that invokes the slow servo. Log file is in the same old place (/cvs/cds/caltech/logs/scripts), and the servo can be (re)started by running:
controls@megatron|~ > sudo initctl start FSSslow
Maybe this won't really change the behavior. We'll see
Today Q moved the FSS slow servo over to some init thing on megatron, and some time ago he did the same thing to the MC auto locker script. It isn't working though.
Even though megatron was rebooted, neither script started up automatically. As Diego mentioned in elog 10823, we ran sudo initctl start MCautolocker and sudo initctl start FSSslow, and the blinky lights for both of the scripts started. However, that seems to be the only thing that the scripts are doing. The MC auto locker is not detecting lockloses, and is not resetting things to allow the MC to relock. The MC is happy to lock if I do it by hand though. Similarly, the blinky light for the FSS is on, but the PSL temperature is moving a lot faster than normal. I expect that it will hit one of the rails in under an hour or so.
The MC autolocker and the FSS loop were both running earlier today, so maybe Q had some magic that he used when he started them up, that he didn't include in the elog instructions?
I ssh'd in, and was able to run each script manually successfully. I ran the initctl commands, and they started up fine too.
We've seen this kind of behavior before, generally after reboots; see ELOGS 10247 and 10572.
In the plot it is shown the behaviour of the PSL-FSS_SLOWDC signal during the last week; the blue rectangle marks an approximate estimate of the time when the scripts were moved to megatron. Apart from the bad things that happened on Friday during the big crash, and the work ongoing since yesterday, it seems that something is not working well. The scripts on megatron are actually running, but I'll try and have a look at it.
I reset the threshold to +6666 counts (the aligned MC transmission is ~16000 for the TEM00 mode) so that it only turns on when we're in a good locked state.
I've now also trended the MOPA output power for the last 200 days to check a possible correlation with the FSS reflected power. See attachment.
The trend shows that the laser power has decayed but it seems that the FSS reflected power has done it even faster: 30% drop in the FSS vs 7% for the MOPA in the last 60 days (attachment n.2).
Just before working on the FSS today, I noticed that the VCO RF output level was set incorrectly.
This should ALWAYS be set so as to give the maximum power in the first order diffracted sideband. One should set this by maximizing the out of lock FSS RFPD DC level to max.
The value was at 2.8 on the VCOMODLEVEL slider. In the attached plot (taken with the FSS input disabled) you can see that this puts us in the regime where the output power to the FSS is first order sensitive to the amplitude noise on the electrical signal to the AOM. This is an untenable situation.
For adjusting the power level to the FSS, we must always use the lamba/2 plate between the AOM and the RC steering mirrors. This dumps power out to the side via a PBS just before the periscope.
What is the Transfer Function of the suspension of the reference cavity? What were the design requirements? What is the Q and how well does the eddy current damping work? What did Wolfowitz know about the WMD and when? Who cooked the RTV in there and why didn't we use Viton??
To get to the bottom of these questions, today I shook the cavity and measured the response. To read out the pitch and yaw modes separately, I aligned the input beam to be misaligned to the cavity. If the beam is mis-aligned in yaw, for example, the transmitted light power becomes first order sensitive to the yaw motion of the cavity.
In the attached image (10 minute second-trend), you can see the second trends for the transmitted and relfected power. The first ringdown comes from the pitch or vertical mode. The second (shorter) one comes from the yaw misalignment and the yaw shake.
To achieve the up/down shake, I leaned onto the table and pumped it at its eigenfrequency. For the yaw shake, I put two fingers on the RC can's sweater and pushed with several pounds of force at the yaw eigenfrequency (2.6 Hz). For the vertical, I jumped up and down at half the vertical eigenfrequency (4 Hz).
I also made sure that the .SCAN field on these EPICS records were set to 9 so that there is no serious effect from a beating between the eigenfrequency and the EPICS sample rate.
f_vert = 4 Hz
tau_vert = 90 seconds
Q_vert = 1000 (yes, that number over there has 3 zeros)
f_hor = 2.6 Hz
tau_hor = 30 seconds
Q_hor = 250
This is an absurd and probably makes us very sensitive to seismic noise - let's make sure to open up the can and put some real rubber in there to damp it. My guess is that these high Q modes
are just the modes of the last-stage steel spring / pendulum.
I measured the open loop gain of the FSS (as usual, I have multiplied the whole OLG by 10dB to account for the forward loop gain in the box). I used a source level of -20 dBm and made sure this was not saturating by changing the level.
Its clear that the BW is limited by the resonance at ~1.7 MHz. Does anyone know what that is?
EO resonance in the RC path?
I measured the RF spectrum coming out the FSS RFPD to look for saturations - its close to the hairy edge. This is with the 8x power increase from my AOM drive increase. I will increase the FSS's modulation frequency which will lower the Q and gain of the PD to compensate somewhat. The lower Q will also gain us phase margin in the FSS loooop.
I put in a bi-directional 20 dB coupler (its only rated down to 30 MHz, but its only off by ~0.3 dB at 21 MHz) between the RFPD and the FSS box. I looked at the time series on the 300 MHz scope and measured the power spectrum.
The peak signal on the scope was 40 mV; that translates to 400 mV at the RFPD output. Depending on whether the series resistor in the box is 20 or 50 Ohms, it means the MAX4107 is close to saturating.
As you can see from the spectrum, its mostly likely to hit its slew rate limit (500 V/us) first. Actually its not going to hit the limit: but even getting within a factor of 10 is bad news in terms of distortion.
Besides the multiples of the modulation frequency, you can see that most of the RMS comes from the strange large peaks at 137.9 and 181.1 MHz. Anyone know what these are from?
On the middle plot above, I have enabled the 20 MHz BW limit so you can see how much the amplitude goes down when only the 21.5 MHz SB is included. You can also see from the leftmost plot that once in awhile there is some 400mV/10ns slewing. Its within a factor of 10 of the slew rate limit.
This is the error point spectrum - it is filled with huge multiples of ~75 kHz as Yoichi noticed a couple years ago.
I tried to use the netgpib.py package to read out the Agilent 4395, but the SVN had been corrupted by someone saving over the netgpib.py package. To get it to work on rosalba I reverted to the previous version, but whoever is busy hacking on netgpib.py needs to checkin the original package and work on some test code instead.
I also noticed that the default output format for the AG4395.py file is in units of Watts. This is kind of dumb - we need someone to develop this package a little as Yoichi did for the SRS785.