In 1X1, there is a box labelled "FSS REF" below a KEPCO HV supply. This box had a power cable that wasn't actually connected to any power. I removed said cable.

- connected the TTFSS cables (FSS fast goes directly to NPRO PZT for now)

- measured the reference cavity 21.5 MHz EOM drive to be 17.8 dBm

-  turned on the HV for the FSS phase correcting EOM (aka PC) drive

- connected and turned on the reference cavity temperature stabilization

- connected the RefCav TRANS PD

- fine tuned the RefCav REFL PD angle

Summary:

On Monday, I hooked up an AG4395 to the PMC error point (using the active probe). The idea was to take a spectrum of the PMC error point every time the FSS PC drive RMS channel indicated an excursion from the nominal value. An initial look at the results don't suggest that this technique is particularly informative. I'll have to think more about a workaround, but please share your ideas/thoughts if you have some.

Also, the feature in the spectrum at ~110 kHz makes me suspect some kind of loop instability. I'll measure the IMC loop OLG at the next opportunity.

Details:

• The PMC servo bandwidth is ~2 kHz, so above this, the PMC error point should be a faithful monitor of the PSL frequency noise, provided the sensing noise is low enough.
• The PMC error point sensing noise is ~100nV/rtHz (I'm monitoring this straight after the Minicircuits mixer+bandpass filter that we are using as a demodulator). This corresponds to ~2 Hz/rtHz, using the ~10 MHz/V PDH discriminant calibration from January. Seems consistent with this elog.
• I was hoping to see if there was a particular frequency band in which the noise gets elevated, and if the crossover frequency is a few kHz and the IMC servo BW is ~110 kHz, I would have expected this to be in the 10-100 kHz region. Possibly my frequency resolution isn't good enough? But with the Agilent, doing a finer grid would mean a longer measurement time, in which case the IMC might lose lock before the measurement is done.
• But, as shown in Attachment #1, there isn't any clear evidence, from the ~20 excursions that were recorded last night. The color of the line is meant to be indicative of the average value of the PC drive RMS channel in the measurement time.
• A significant bottleneck in this whole process is that it takes ~1 minute to initiate the GPIB measurement, and download the data. The pseudo-code I used is:
  • While the IMC is locked, watch PCdrive RMS EPICS channel's "ALARM" state, which becomes non-zero when the PCdrive RMS exceeds 1 V (this is how it is defined in the EPICS db record right now).
  • Make sure this isn't a transient feature - I do this by waiting 5 seconds and checking that the ALARM flag is still flagged.
  • Initiate a AG4395 measurement over GPIB - I use the measurement span of 1 kHz - 1 MHz with a BW/span ratio of 0.1%, 5 averages.
  • Check that the IMC is still locked (if it got unlocked while the measurement was made, presumably the measurement is garbage).
• Is there a better monitor of the laser frequency noise? I can imagine using POX/POY which I think have a lower electronics noise floor but I'm not sure if that's true at 100 kHz and having the arms locked in addition to the IMC seems more complicated...
• Since we are planning a laser upgrade, is this worth spending more time on? I may leave the measurement running on pianosa in a tmux session while I'm not in the lab...

I had set up the 4395 to do this automatically a few years ago, but it looked at the FSS/IMC instead. When the PCDRIVE goes high there is this excess around ~500 kHz in a broad hump.

But the IMC loop gain changes sometimes with alignment, so I don't know if its a loop instability or if its laser noise. However, I think we have observed PCDRIVE to go up without IMC power dropping so my guess is that it was true laser noise.

This works since the IMC is much more sensitive than PMC. Perhaps one way to diagnose would be to lock IMC at a low UGF without any boosts. Then the UGF would be far away from that noise making frequency. However, the PCDRIVE also wouldn't have much activity.

 I changed the value of the nominal FSS common gain in the PSL Settings screen (It's called the 'FSS global gain' there).  To get to this screen:  sitemap -> PSL_main -> PSL_settings.  The MC autolocker reads these settings from the screen and uses those values.  Now the FSS returns to this value of 13 that Jamie has chosen.  For the past few days, it's been going back to the old value of 10.1 .  The FAST gain was already set to this 23.5 value.

Since Rana's overhaul of the IMC, the FSS input offset had been sitting at zero. 

Over the last day or so, I had noticed the MC refl wall striptool trace looking noisier, and earlier this evening, we were suffering from a fair amount of downtime due to IMC unlocks, and failure to autolock for times on the order of ten minutes. 

While we had used ezcaservo for this in the past, I set the FSS offset manually tonight. Namely, I popped open a dataviewer trace of MC_F, and scanned the FSS offset to make MC_F go to zero. It required a good amount of offset, 4.66 V according to the FSS screen. I did this while the FSS slow servo was on, which held the FSS Fast output at zero. 

That was four hours ago; MC_F is still centered on zero, and we have not had a single IMC unlock since then. The MC refl trace is thinner too, though this may be from nighttime seismic. 

The RefCav is locked and aligned. I changed the fast gain sign by changing the jumper setting on the TTFSS board. The RefCav visibility is 70%. The FSS loop ugf is about 80 kHz (plot attached. there is 10 dB gain in the test point path. this is why the ugf is at 10 dB when measured using in1 and in2 spigots on the front of the board.)  with FSS common gain max out at 30 dB. There is about 250 mW coming out of the laser and 1 mW going to RefCav out of the back of the PMC. So the ugf can be made higher at full power. I have not made any changes to account for the PMC pole (the FSS is after the PMC now). The FSS fast gain was also maxed out at 30 dB to account for the factor of 5 smaller PZT actuation coefficient - it used to be 16 dB according to the (previous) snap shot. The RefCav TRANS PD and camera are aligned. I tuned up the phase of the error signal by putting cables in the LO and PD paths. The maximum response of the mixer output to the fast actuator sweep of the fringe was with about 2 feet of extra cable in the PD leg.

I am leaving the FSS unlocked for the night in case it will start oscillating as the phase margin is not good at this ugf.

Brilliant! This is the VERY way how the things are to be conquered!

I've been measuring a bunch of transfer functions of the FSS related stuffs.
There are a lot to be analyzed yet, but here I put one mystery I'm having now.
Maybe I'm missing something stupid, so your suggestions are welcome.

Here is a conceptual diagram of the FSS control board

                                                          TP3             TP4
                                                           ^               ^
                                                           |               |
RF PD -->--[Mixer]-----[Sum Amp]------>--[Common Gain]--->----[Fast Gain]----[Filter]--> NPRO PZT
              ^     |      ^        |                  |     
              |     V      |        V                  |
LO ---->-------    TP1     IN      TP2                 -->---[Filter]--[High Volt. Amp.] --> Phase Corrector

What I did was first to measure a "normal" openloop transfer function of the FSS servo.
The FSS was operated in the normal gain settings, and a signal was injected from "IN" port.
The open loop gain was measured by TP1/TP2.
Now, I disconnected the BNC cable going to the phase corrector to disable the PC path and locked the ref. cav. 
only using the PZT. This was done by reducing the "Common Gain" and "Fast Gain" by some 80dB.
Then I measured the open loop gain of this configuration. The UGF in this case was about 10kHz.
I also measured the gain difference between the "normal" and "PZT only" configurations by injecting 
a signal from "IN" and measuring TP3/TP2 and TP4/TP3 with both configurations (The signal from the Mixer was
disconnected in this measurement). 

The first attachment shows the normal open loop gain (purple) and the PZT only open loop gain scaled by the 
gain difference (about 80dB). The scaled PZT open loop gain should represent the open loop gain of the PZT
path in the normal configuration. So I expected that, at low frequencies, the scaled PZT loop TF overlaps the normal
open loop TF.
However, it is actually much larger than the normal open loop gain.
When I scale the PZT only TF by -30dB, it looks like the attachment #2.
The PZT loop gain and the total open loop gain match nicely between 20kHz and 70kHz.
Closer look will show you that small structures (e.g. around 30kHz and 200kHz) of the two
TFs also overlap very well. I repeated measurements many times and those small structures are always there (the phase is
also consistently the same). So these are not random noise.

I don't know where this 30dB discrepancy comes from. Is it the PC path eating the PZT gain ?

I have measured many other TFs. I'm analyzing these.
Here is the TO DO list:

* Cavity response plot from AOM excitation measurements.
* Cavity optical gain plot.
* Reconstruct the open loop gain from the electric gain measurements and the optical gain above.
* Using a mixer and SR560(s), make a separate feedback circuit for the PZT lock. Then use the PC path
  to measure the PC path response.
* See the response of the FSS board to large impulse/step inputs to find the cause of the PC path craziness.
etc ...

   f (MHz) |  before filter (dBm) | after filter (dBm)
   ---------------------------------------------------
     21.5  |       -12.8               -13.1       
     43            -24                 -46
     64.5          -50               < -80
     86            -64               < -80

The attached plot shows the beam scans of the beam leaking from the back mirror of the PMC to the BS cube that first turns the S-pol beam 90 deg to the AOM and then transmits the AOM double passed and polarization rotated P-pol beam to the reference cavity. The beam from the PMC is mode matched to the AOM using a single lens f=229 mm. The ABCD file is attached. The data were taken with VCO control voltage at 5 V. We then reduced the voltage to 4 V to reduce the astigmatism. Tara has the data for the beam scan in this configuration in his notebook.

The beam from AOM is mode matched to the reference cavity using a single lens f=286.5 mm. The ABCD file is attached.

Koji, Yoichi

The MC has been unstable and unhappy for the last several hours.  When I looked, I saw that the FSS_FAST monitor has been hovering around 1 V, when it is supposed to be closer to 5ish. 

I changed the C1:PSL-FSS_INOFFSET from -0.08 to -0.8537, and will see if the MC sticks around for longer this time around.

The fast feedback should be around zero now!

 Dang it, I completely forgot.  Well, anyhow, it pulled itself back down to less than 1V, and the MC stayed happy for several hours.  I'm not totally sure what changing the offset did, but the MC seems happy for right now.  I should take a quick look at the error point to make sure that I didn't mess up your tuning.

I stopped by the 40m for some reason and found that the MC trans was 7.5.
This was caused by an oscillation of FSS, which seemed to be started by itself.

The oscillation stopped by reducing the FSS total gain to +9dB (from +11dB).
This is not a permanent fix (i.e. autolocker will restore the gain).
If it seems necessary to reduce the FSS gain always, we change the MC autolocker script.

Today Rana pointed out that our FSS slow servo is malfunctioning. It has been for a while that our laser temperature control voltage drifted from 0 to 10.

I looked at FSSSlowServo script that runs at op340m and controls the servo. Script disables the servo when MC transmission is less then FSS_LOCKEDLEVEL. But his value was set to 0.2 probably till reference cavity time.

This means that slow servo was not disabled when MC was unlocked. I changed this value to 7000.

Also I increased integral gain from 0.0350 to 0.215 such that fast control is always in the range 4.5 - 5.5

I bet you thought that the NPRO slow actuator response could be well represented by a pole at ~0.1 Hz? Well, that's just what they want you to believe.

I attach the response measured in FSS-FAST (with no feedback to the SLOW actuator) when the SLOW is given a step. As you may remember from

kindergarten, the response of a single pole low pass should just be an exponential. Clearly, there's more here than 1 pole.

 I also inserted a factor of 0.01 in the FSSSlowServo code so I could make the gain sliders have reasonable values (they used to all be ~1e-3). The SVN and the MEDM snapshot are updated.

MCAutolocker/FSSSlow are not properly documented and not properly working.
Tidy up the script and documentation, or bring it back to megatron

I was aware that the FSSSlow was misbehaving since the shutdown upon the July power outage.
- FSS Slow servo did nothing even though the apparent settings in C1:PSL_SLOW screen looked fine and heart beat blinking
- Wanted to restart FSSSlow at megatron. Despite the login message showing how to do it, the system service does not exist anymore, because it was moved somewhere.
- Searched 40m wiki but found no info how to kill and restart it
- Found an elog. It was moved to docker on optimus ELOG 16480 . The restart procedure can only be found here. Please fix all the documentation inconsistency >> Anchal

According to this elog, the following commands need to be run for starting up MCAutolocker and FSSSlow on optimus:

cd /opt/rtcds/caltech/c1/Git/40m/scripts/MC
sudo docker-compose up -d

- Problem continues. Now FSSSlow is running but only when the IMC is locked. It does not stop even when the IMC lock is lost. How can we debug docker thing?
- This is minor but the MCAutolocker log (/opt/rtcds/caltech/c1/scripts/MC/logs/AutoLocker.log) is no longer updated even though MCAutolocker is running. Was it moved somewhere?

The FSSSlow script was not properly documented and it was not working, so I had to use one that I knew worked from CTN. This scripts lives in

/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDLocker.py

and uses a configuration file

/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDConfigFSS.yml

The script runs all the time inside docker container which keeps it running. The heart-beat blinker shows if the script is active or not, but it only starts working when C1:IOO-MC_LOCK is 1 and C1:PSL-FSS_SLOWLOOP is 1. The second channel is a button on C1PSL_SLOW screen to engage autolocker. It has to be turned on for FSS to work.

docker instructions:

The following message is displayed on login in optimus:

For checking log files of autolocker script, on optimus do:

For checking log files of FSS PID loop, on optimus do:

In the above commands, add < | tail -15> to just see the most recent 15 lines in the log file. change 15 to whatever number of lines you want to see from the end.

At any time, if you want to know how docker is running stuff, check out the /opt/rtcds/caltech/c1/Git/40m/scriptsdocker-compose.yml file for self-explanatory script usage.

I'll add some documentation on the wiki soon. That is indeed required and should have been done already.

Debugging scripts:

All scripts could be debugged by running them on rossa by directly using python command. You can stop the docker container on optimus using:

and then run the file on rossa to check it's behavior. After debugging and fixing any issues, please commit the file to gitlab repo and go back to optimus and restart the docker container:

I'll add this procedure to a wiki page as well.

Reverting back to systemd on megatron

The setup on megatron was not removed at all. All service files exist in same place and old scripts can be started in the old manner by doing following on megatron:

For FSSSlow:

For MC autolocker:

For diabling these services again, do:

Note that one should stop docker containers on optimus before starting these systemd services to avoid conflicting scripts running together.

I have added above instructions on megatron motd. So on loging into megatron, these updated instructions would come.

If someone wants to fix the old scripts and use systemd for managing those scripts, please do so but I won't be able to help in debugging those old scripts. The shell scripts are very complicated and beyond my knowledge and python scripts are lacking documentation.

I'm happy to help debug or extend the functionality of the new scripts that live in the git directory.

> it only starts working when C1:IOO-MC_LOCK is 1 and C1:PSL-FSS_SLOWLOOP is 1.

- OK. Your new MCAutolocker does not reflect the lock status to C1:IOO-MC_LOCK. This causes FSS Slow to go crazy when the IMC is not locked. Can you fix that?

- So C1PSL_SLOW.adl screen, which spawns by the "SLOW Servo" button on the FSS screen, has no effect on the FSS SLOW servo anymore. It is obsolete. At least the screen (or the link to the screen) should be removed. (Work on it once you are back.)

- Also, please make a wiki page and copy the description on the previous page.

Added C1:IOO-MC_LOCK to ALConfigMC.yml which solved the isse with FSS Slow. We should tune the FSS Slow Servo PID coefficients for a better performance.

the C1PSL_SLOW.adl screen is not obsolete. It can be used to change the PID coefficients, engage/disengage the PID loop, monitor the PID script blinker, and monitor FAST actuator value C1:PSL-FSS_FAST. the functionality of this screen has not changed from before.

I've also added a wiki page for scripts documentation.

can't we just go back to the old python script that was working for many years, and tested? I imagine that as soon as someone besides you tries to debug the docker setup, this is what will happen.

FSS Slow set point to be zero

op340m:FSS>cat FSSSlowServo
#!/usr/bin/perl -w

# PID Servo for PSL-FSS (Slow)
# Tobin Fricke 2007-01-09

use strict;
#use Scalar::Util qw(looks_like_number);

sub looks_like_number {
    return ($_[0] =~ /^-?\d+\.?\d*$/);  #FIXME
}

use EpicsTools;

# Parameters
my $process  = 'C1:PSL-FSS_FAST';
my $actuator = 'C1:PSL-FSS_SLOWDC';
#my $setpoint = 5.5;
my $setpoint = 0;
my $blinkystatus = 0;
 

op340m:scripts>/opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo > /cvs/cds/caltech/logs/scripts/FSSslow.cronlog &

It seems that FSS slow servo stopped working.

I found that megatron was restarted (by Rana, to finish an apt-get upgrade) on ~18:47 PDT today.

controls@megatron|~> last -5
controls pts/0        192.168.113.216  Mon May 15 19:15   still logged in   
controls pts/0        192.168.113.216  Mon May 15 19:14 - 19:15  (00:01)    
reboot   system boot  3.2.0-126-generi Mon May 15 18:50 - 19:19  (00:29)    
controls pts/0        192.168.113.200  Mon May 15 18:43 - down   (00:04)    
controls pts/0        192.168.113.200  Mon May 15 15:25 - 17:38  (02:12)

FSSslow / MCautolocker were restarted on megatron.

As part of investigation into this issue, Jonathan Hanks pointed out that the "minute trends" being recorded by our system were actually only being recorded every 120 seconds (a.k.a. 2 minutes). He had fixed the appropriate line in the parameter file, but had not restarted the FW processes. I had restarted it on Friday. (but failed to elog it !) 

To check if this made any difference, I pulled 1 hour of "minute trend" data for the PSL table temperature channel from ~1 hour ago, and compared the number of datapoints against a 1 hour minute trend time series from 2 May. I've put the display with # of datapoints (for an identical length of time) from before [left] and after [right] the restart next to the plots in Attachment #1. Seems like we are getting minute trends written every 60 seconds now, as it should be .

The unavailability of trends from nodus is a separate issue for which JH has suggested another fix, to be elogged separately.

I have to blame Jamie for putting extra 2 randomly.
Measured PRM-PR2 cavity finesse was actually 108 +/- 3 (even if you use digital system to get data).

Lorentzian fit:
  Lorentzian function is;

f(x;x0,gamma,A) = A * gamma**2/((x-x0)**2+gamma**2)

  where x0 is the location of the peak, gamma is HWHM, and A is the peak height.
  Lorentzian fitting function in my original code (/users/yuta/scripts/modescanresults/analyzemodescan.py) was

fitFunc = lambda p,x,m: (m-p[2])*p[0]**4/(4*(x-p[1])**2+p[0]**4)+p[2]

  In this function, p[0] is sqrt(FWHM), not sqrt(HWHM). I doubled gamma to make it FWHM and squared it because they should be positive.
  During Jamie's modification of my code, he doubled p[0]**2 to get FWHM, which is wrong (/users/jrollins/modescan/modescan.py).

  I should have commented that p[0] is sqrt(FWHM).

Redoing the analysis:
  1. I pulled 2 out, and modified Jamie's modescan.py so that you can name each peak with peakdistinguish=True option. I also modified fitpeak function so that it throws away "peaks" which don't look like a peak.

  2. If you run /users/yuta/PRCmodescan/run.py and name each peak, you will get peaks.csv which includes peak position, FWHM, and the type of the peak;

0.065017,0.001458,l
0.070446,0.001463,3
0.075940,0.001509,2
0.081552,0.001526,1
0.087273,0.001565,0
0.112027,0.001911,u
0.278660,0.002211,u
0.306486,0.001658,0
0.312480,0.001576,1
0.313626,2.507910,
0.318486,0.001626,2
0.319730,2.633097,
0.324801,0.001739,3
0.331848,0.001922,l
0.527509,0.001603,l
0.533231,0.001445,3
0.538648,0.001488,2
0.544081,0.001455,1
0.549517,0.001498,0
0.551725,2.422759,
0.570972,0.001346,u

  3. /users/yuta/PRCmodescan/calcmodescanresults.py reads peaks.csv and tells you the results;

Time between TEM00 and sideband  0.0239435  pm  0.00115999887452  sec
Calibration factor is  462.167602898  pm  22.3907907867  MHz/sec
FSR is  78.4797010471  MHz
FWHM is  0.729828720682  pm  0.0174145743828  MHz
TMS is  2.64718671684  pm  0.0538858477824  MHz
Finesse is  107.53166986  pm  2.5658325169
Cavity g-factor is  0.994390582331  pm  0.000228155661075
Cavity g-factor is  0.988812630228  pm  0.000453751681357   (Edited by YM; see elog #8056)
RoC of PR2 is  -187.384503001  pm  4.26100999578  m (assuming PRM RoC= 122.1  m)
RoC of PRM is  217.915890722  pm  5.65451518991  m (assuming PR2 RoC= -600  m)