Dang it, I completely forgot.  Well, anyhow, it pulled itself back down to less than 1V, and the MC stayed happy for several hours.  I'm not totally sure what changing the offset did, but the MC seems happy for right now.  I should take a quick look at the error point to make sure that I didn't mess up your tuning.

I stopped by the 40m for some reason and found that the MC trans was 7.5.
This was caused by an oscillation of FSS, which seemed to be started by itself.

The oscillation stopped by reducing the FSS total gain to +9dB (from +11dB).
This is not a permanent fix (i.e. autolocker will restore the gain).
If it seems necessary to reduce the FSS gain always, we change the MC autolocker script.

Today Rana pointed out that our FSS slow servo is malfunctioning. It has been for a while that our laser temperature control voltage drifted from 0 to 10.

I looked at FSSSlowServo script that runs at op340m and controls the servo. Script disables the servo when MC transmission is less then FSS_LOCKEDLEVEL. But his value was set to 0.2 probably till reference cavity time.

This means that slow servo was not disabled when MC was unlocked. I changed this value to 7000.

Also I increased integral gain from 0.0350 to 0.215 such that fast control is always in the range 4.5 - 5.5

I bet you thought that the NPRO slow actuator response could be well represented by a pole at ~0.1 Hz? Well, that's just what they want you to believe.

I attach the response measured in FSS-FAST (with no feedback to the SLOW actuator) when the SLOW is given a step. As you may remember from

kindergarten, the response of a single pole low pass should just be an exponential. Clearly, there's more here than 1 pole.

 I also inserted a factor of 0.01 in the FSSSlowServo code so I could make the gain sliders have reasonable values (they used to all be ~1e-3). The SVN and the MEDM snapshot are updated.

I came to the lab to see the recovery work from the power glitch this morning 8:15AM. All CDS seems up. The suspensions are somewhat aligned. Some of them were not damped. The oplevs were off. Radhika is working on the recovery of the FP arms.

I noticed that FSS Slow servo is not working. I always forget what is the right way to turn it on. Here is the summary:

How to turn on FSSSlow (2023 Sept version)

• Go to megatron
• sudo systemctl enable FSSSlow
• sudo systemctl start FSSSlow
• sudo systemctl status FSSSlow

  ● FSSSlow.service - Script to run the PID temperature control servo for the PSL
     Loaded: loaded (/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/FSSSlow.service; enabled; vendor pre
     Active: active (running) since Fri 2023-09-08 17:10:52 PDT; 1s ago
   Main PID: 2088 (python3)
      Tasks: 6 (limit: 4674)
     CGroup: /system.slice/FSSSlow.service
             └─2088 /usr/bin/python3 /opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDLocker.py PIDConf

  Sep 08 17:10:52 megatron systemd[1]: Started Script to run the PID temperature control servo for the

MCAutolocker/FSSSlow are not properly documented and not properly working.
Tidy up the script and documentation, or bring it back to megatron

I was aware that the FSSSlow was misbehaving since the shutdown upon the July power outage.
- FSS Slow servo did nothing even though the apparent settings in C1:PSL_SLOW screen looked fine and heart beat blinking
- Wanted to restart FSSSlow at megatron. Despite the login message showing how to do it, the system service does not exist anymore, because it was moved somewhere.
- Searched 40m wiki but found no info how to kill and restart it
- Found an elog. It was moved to docker on optimus ELOG 16480 . The restart procedure can only be found here. Please fix all the documentation inconsistency >> Anchal

According to this elog, the following commands need to be run for starting up MCAutolocker and FSSSlow on optimus:

cd /opt/rtcds/caltech/c1/Git/40m/scripts/MC
sudo docker-compose up -d

- Problem continues. Now FSSSlow is running but only when the IMC is locked. It does not stop even when the IMC lock is lost. How can we debug docker thing?
- This is minor but the MCAutolocker log (/opt/rtcds/caltech/c1/scripts/MC/logs/AutoLocker.log) is no longer updated even though MCAutolocker is running. Was it moved somewhere?

The FSSSlow script was not properly documented and it was not working, so I had to use one that I knew worked from CTN. This scripts lives in

/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDLocker.py

and uses a configuration file

/opt/rtcds/caltech/c1/Git/40m/scripts/PSL/FSS/PIDConfigFSS.yml

The script runs all the time inside docker container which keeps it running. The heart-beat blinker shows if the script is active or not, but it only starts working when C1:IOO-MC_LOCK is 1 and C1:PSL-FSS_SLOWLOOP is 1. The second channel is a button on C1PSL_SLOW screen to engage autolocker. It has to be turned on for FSS to work.

docker instructions:

The following message is displayed on login in optimus:

For checking log files of autolocker script, on optimus do:

For checking log files of FSS PID loop, on optimus do:

In the above commands, add < | tail -15> to just see the most recent 15 lines in the log file. change 15 to whatever number of lines you want to see from the end.

At any time, if you want to know how docker is running stuff, check out the /opt/rtcds/caltech/c1/Git/40m/scriptsdocker-compose.yml file for self-explanatory script usage.

I'll add some documentation on the wiki soon. That is indeed required and should have been done already.

Debugging scripts:

All scripts could be debugged by running them on rossa by directly using python command. You can stop the docker container on optimus using:

and then run the file on rossa to check it's behavior. After debugging and fixing any issues, please commit the file to gitlab repo and go back to optimus and restart the docker container:

I'll add this procedure to a wiki page as well.

Reverting back to systemd on megatron

The setup on megatron was not removed at all. All service files exist in same place and old scripts can be started in the old manner by doing following on megatron:

For FSSSlow:

For MC autolocker:

For diabling these services again, do:

Note that one should stop docker containers on optimus before starting these systemd services to avoid conflicting scripts running together.

I have added above instructions on megatron motd. So on loging into megatron, these updated instructions would come.

If someone wants to fix the old scripts and use systemd for managing those scripts, please do so but I won't be able to help in debugging those old scripts. The shell scripts are very complicated and beyond my knowledge and python scripts are lacking documentation.

I'm happy to help debug or extend the functionality of the new scripts that live in the git directory.

> it only starts working when C1:IOO-MC_LOCK is 1 and C1:PSL-FSS_SLOWLOOP is 1.

- OK. Your new MCAutolocker does not reflect the lock status to C1:IOO-MC_LOCK. This causes FSS Slow to go crazy when the IMC is not locked. Can you fix that?

- So C1PSL_SLOW.adl screen, which spawns by the "SLOW Servo" button on the FSS screen, has no effect on the FSS SLOW servo anymore. It is obsolete. At least the screen (or the link to the screen) should be removed. (Work on it once you are back.)

- Also, please make a wiki page and copy the description on the previous page.

Added C1:IOO-MC_LOCK to ALConfigMC.yml which solved the isse with FSS Slow. We should tune the FSS Slow Servo PID coefficients for a better performance.

the C1PSL_SLOW.adl screen is not obsolete. It can be used to change the PID coefficients, engage/disengage the PID loop, monitor the PID script blinker, and monitor FAST actuator value C1:PSL-FSS_FAST. the functionality of this screen has not changed from before.

I've also added a wiki page for scripts documentation.

can't we just go back to the old python script that was working for many years, and tested? I imagine that as soon as someone besides you tries to debug the docker setup, this is what will happen.

FSS Slow set point to be zero

op340m:FSS>cat FSSSlowServo
#!/usr/bin/perl -w

# PID Servo for PSL-FSS (Slow)
# Tobin Fricke 2007-01-09

use strict;
#use Scalar::Util qw(looks_like_number);

sub looks_like_number {
    return ($_[0] =~ /^-?\d+\.?\d*$/);  #FIXME
}

use EpicsTools;

# Parameters
my $process  = 'C1:PSL-FSS_FAST';
my $actuator = 'C1:PSL-FSS_SLOWDC';
#my $setpoint = 5.5;
my $setpoint = 0;
my $blinkystatus = 0;
 

op340m:scripts>/opt/rtcds/caltech/c1/scripts/PSL/FSS/FSSSlowServo > /cvs/cds/caltech/logs/scripts/FSSslow.cronlog &

It seems that FSS slow servo stopped working.

I found that megatron was restarted (by Rana, to finish an apt-get upgrade) on ~18:47 PDT today.

controls@megatron|~> last -5
controls pts/0        192.168.113.216  Mon May 15 19:15   still logged in   
controls pts/0        192.168.113.216  Mon May 15 19:14 - 19:15  (00:01)    
reboot   system boot  3.2.0-126-generi Mon May 15 18:50 - 19:19  (00:29)    
controls pts/0        192.168.113.200  Mon May 15 18:43 - down   (00:04)    
controls pts/0        192.168.113.200  Mon May 15 15:25 - 17:38  (02:12)

FSSslow / MCautolocker were restarted on megatron.

As part of investigation into this issue, Jonathan Hanks pointed out that the "minute trends" being recorded by our system were actually only being recorded every 120 seconds (a.k.a. 2 minutes). He had fixed the appropriate line in the parameter file, but had not restarted the FW processes. I had restarted it on Friday. (but failed to elog it !) 

To check if this made any difference, I pulled 1 hour of "minute trend" data for the PSL table temperature channel from ~1 hour ago, and compared the number of datapoints against a 1 hour minute trend time series from 2 May. I've put the display with # of datapoints (for an identical length of time) from before [left] and after [right] the restart next to the plots in Attachment #1. Seems like we are getting minute trends written every 60 seconds now, as it should be .

The unavailability of trends from nodus is a separate issue for which JH has suggested another fix, to be elogged separately.

I have to blame Jamie for putting extra 2 randomly.
Measured PRM-PR2 cavity finesse was actually 108 +/- 3 (even if you use digital system to get data).

Lorentzian fit:
  Lorentzian function is;

f(x;x0,gamma,A) = A * gamma**2/((x-x0)**2+gamma**2)

  where x0 is the location of the peak, gamma is HWHM, and A is the peak height.
  Lorentzian fitting function in my original code (/users/yuta/scripts/modescanresults/analyzemodescan.py) was

fitFunc = lambda p,x,m: (m-p[2])*p[0]**4/(4*(x-p[1])**2+p[0]**4)+p[2]

  In this function, p[0] is sqrt(FWHM), not sqrt(HWHM). I doubled gamma to make it FWHM and squared it because they should be positive.
  During Jamie's modification of my code, he doubled p[0]**2 to get FWHM, which is wrong (/users/jrollins/modescan/modescan.py).

  I should have commented that p[0] is sqrt(FWHM).

Redoing the analysis:
  1. I pulled 2 out, and modified Jamie's modescan.py so that you can name each peak with peakdistinguish=True option. I also modified fitpeak function so that it throws away "peaks" which don't look like a peak.

  2. If you run /users/yuta/PRCmodescan/run.py and name each peak, you will get peaks.csv which includes peak position, FWHM, and the type of the peak;

0.065017,0.001458,l
0.070446,0.001463,3
0.075940,0.001509,2
0.081552,0.001526,1
0.087273,0.001565,0
0.112027,0.001911,u
0.278660,0.002211,u
0.306486,0.001658,0
0.312480,0.001576,1
0.313626,2.507910,
0.318486,0.001626,2
0.319730,2.633097,
0.324801,0.001739,3
0.331848,0.001922,l
0.527509,0.001603,l
0.533231,0.001445,3
0.538648,0.001488,2
0.544081,0.001455,1
0.549517,0.001498,0
0.551725,2.422759,
0.570972,0.001346,u

  3. /users/yuta/PRCmodescan/calcmodescanresults.py reads peaks.csv and tells you the results;

Time between TEM00 and sideband  0.0239435  pm  0.00115999887452  sec
Calibration factor is  462.167602898  pm  22.3907907867  MHz/sec
FSR is  78.4797010471  MHz
FWHM is  0.729828720682  pm  0.0174145743828  MHz
TMS is  2.64718671684  pm  0.0538858477824  MHz
Finesse is  107.53166986  pm  2.5658325169
Cavity g-factor is  0.994390582331  pm  0.000228155661075
Cavity g-factor is  0.988812630228  pm  0.000453751681357   (Edited by YM; see elog #8056)
RoC of PR2 is  -187.384503001  pm  4.26100999578  m (assuming PRM RoC= 122.1  m)
RoC of PRM is  217.915890722  pm  5.65451518991  m (assuming PR2 RoC= -600  m)

I've spent an inordinate amount of time trying to figure out how to get pianosa to support dual monitors.  It apparently has some special nvidia graphics that are not well supported in Ubuntu (10.04 at least).  I've tried installing a newer kernel (3.0.0) and installing the special nvidia compile-from-source Ubuntu packages, but nothing is working.  And now unfortunately he's not giving me any video at all.  I'm sick of this today, so I'll try again tomorrow.

In the future we should avoid these bullshit nvidia cards like the plague.

I managed to get pianosa working again with just a single monitor.  I'm done trying to configure it for dual, though.

[Jamie, JC, Chris]

Today we made a failed attempt to take over the 40m hardware with the new front ends on the test stand.

As an initial test, we connected the new c1iscey to its I/O chassis using the OneStop fiber link. This went smoothly, so we tried to proceed with the rest of the system, which uncovered several problems. Consequently, we’ve reverted control back to the old front ends tonight, and will regroup and make another attempt tomorrow.

Status summary:

• c1iscey worked on the first try
• c1lsc worked, after we sorted out which of the two OneStop cables run to its rack we needed to use
• c1sus2 sort of worked (its models have been crashing sporadically)
• c1ioo had a busted OneStop cable, and worked after that was replaced
• c1sus refused to work with the fiber OneStop cables (we tried several, including the known working one from c1ioo), but we jury-rigged it to run over a copper cable, after nudging the teststand rack a bit closer to the chassis
• c1iscex refused to work with the fiber OneStop cables, and substituting copper was not an option, so we were stuck

There are various pathologies that we've seen with the OneStop interface cards in the I/O chassis. We don't seem to have the documentation for these cards, but our interpretive guesses are as follows:

• When working, it is supposed to illuminate all the green LEDs along the top of the card, and the four next to the connector. In this state, you can run lspci -vt on the host, and see the various PLX/Contec/etc devices that populate the chassis.
• When the cable is unplugged or bad, only four green LEDs illuminate on the card, and none by the connector. No devices from the chassis can be seen from the host.
• On c1iscex and c1sus, when a fiber link is plugged in, it turns on all the LEDs along the top of the card, but the four next to the connector remain dark. We’re not sure yet what this is trying to tell us, but lspci finds no devices from the chassis, same as if it is unplugged.
• Also, sometimes on c1iscex, no LEDs would illuminate at all (possibly the card was not seated properly).

Tomorrow, we plan to swap out the c1iscex I/O chassis for the one in the test stand, and see if that lets us get the full system up and running.

I assume it's because the burt restore didn't work for the SUS summary screen, but all of the values for the ifo suspensions (not the MCs...they're okay) are red. 

I am trying to run Rana's setSensors.py script, but am failing.  Any inspiration would be appreciated:

rosalba:SUS_SUMMARY>./setSensors.py 1001708529 500 .1 .25
['./setSensors.py', '1001708529', '500', '.1', '.25']
/cvs/cds/caltech/apps/linux64/python/lib64/python2.4/site-packages/nds/__init__.py:28: RuntimeWarning: No protocol specified, attempting protocol nds_v2
  super(daq, self).__init__(host, port)
Connecting NDS2 .... authenticate done
Traceback (most recent call last):
  File "./setSensors.py", line 81, in ?
    mean = acquire(x)
  File "./setSensors.py", line 73, in acquire
    daq.request_channel(chans[x])
Boost.Python.ArgumentError: Python argument types in
    daq.request_channel(daq, str)
did not match C++ signature:
    request_channel(_daq_t {lvalue}, daq_channel_t*)

I'm not exactly sure what the problem is.  Line 73, looks like it should have 2 arguments in the daq.request_channel, but even if I put in the "daq" variable (which is set a few lines above), I get the exact same error.  So...something else is wrong.  Ideas from someone who "speaks" python??

 My guess is that this has something to do with the NDS client version you're using.  Try running the script on a machine where pynds and nds-client are known to be compatible, like pianosa.

 Doesn't work on pianosa either. Has someone changed the python environment?

pianosa:SUS_SUMMARY 0> ./setSensors.py 1000123215 600 0.1 0.25
Traceback (most recent call last):
  File "./setSensors.py", line 2, in <module>
    import nds
ImportError: No module named nds

Chub mentioned to Paco and I that the Nitrogen seemed to be losing pressure quicker than usual. After looking over the previous 4 months on Dataviewer, the pressure has a pretty consistent slope. I have ran into the issue of the neck of the N2 tank slighty leaking, so this may have been the issue. For now, there does not seem to be any significant or obvious problem.

Zach and Koji

The old small MMT was removed and wrapped by Al foils.

The steering mirror IM2-IM4 were displaced and aligned.

The Faraday isolator block is moved and aligned.

The MC is realigned and resonatng TEM-00.

Now the MC has slightly miscentered beam on the mirrors owing to change of the stack leveling.
OSEMs are also in a strange state. We should check this later.

 Neither of these computers were listed in the Martian Host Table in the wiki, so I put them on there.  It's handy to keep this updated, so that we know what IP addresses are available.

(Kiwamu, Yuta)

Symptom:
 Farfalla(Acer Aspire one KAV60 netbook) couldn't run Firefox and returned the following error:
  Error: platform version '1.9.2.8' is not compatible with
  minVersion >= 1.9.2.9
  maxVersion <= 1.9.2.9

What we did:
1. Added the following line to ~/.cshrc.
  alias firefox "/usr/lib/firefox-3.6/firefox"
2. Made a cool launcher for firefox.

Result:
 Farfalla can fly into the web with Firefox now.

Notes:
 Even if it was then before, bash could run firefox. tsch couldn't.
 The command firefox was /cvs/cds/caltech/apps/linux/bin/firefox for tsch, because of the source /cvs/cds/caltech/cshrc.40m.
 I did this to zita which had the same symptom, too.

Next work:
 Farfalla wakes up on the wrong side of the bed. This has to be fixed.

A few years ago, Koji and I setup a delay line phase shifter, which can be used to impart a (switchable) delay to a signal path. Since we talked about reviving the fast (= high bandwidth) ALS control scheme at the meeting, I reminded myself of the infrastructure available.

• Schematic
• Comprehensive note on theory of operation / performance.
• Past elog threads - #11603 and #11604.
• Attachment #1 - my modification to the ALS screen to add a slider that controls the channel C1:LSC-BO_1_0_SET. The label is a bit misleading for now - elog11604 tells you the conversion between this slider value and the actual delay in nanoseconds, but I couldn't get a soft channel set up that correctly FLNKed to this record. In the process of trying to do so, I edited the C1_ISC-AUX_ALS.db file, and also restarted the modbus and latch processes on c1iscaux a few times.
• Attachment #2 - frequency dependent loss for some representative delays. At ~200 MHz, I find the measured loss to be > 8dB, which is ~2dB more than what the D. Sigg note tells me to expect. This is rather a lot of loss, but I guess it's okay. Measurement cable loss was calibrated out with the AG4395A.
• Attachment #3 - confirmation of constantness of delay as a function of frequency, for some representative delays. The "undelayed" setting corresponds to a fixed delay of ~4 nsec, which is consistent with what the D. Sigg note tells me to expect. Once again, I calibrated out the delay of the measurement setup using the AG4395A.

For a beat note in the regime 10-100 MHz, we should have plenty of range in this module to add a delay such that we zero one quadrature of the ALS DFD output (for a linear error signal). 

I then proceeded to connect the single-ended front panel BNC corresponding to the ALS_X_I DFD channel to the IN2 input of the CM board (this would be what we use for high bandwidth ALS feedback). The conventional ALS system uses the differential output from a rear-panel D-sub, so in principle, both systems could run in parallel. I confirmed that I could see a signal when the IN2 path on the CM board was engaged (monitored using ndscope at the CM_Slow output), and that this signal stabilized when the green laser was locked to the X-arm length, which itself was slaved to the PSL frequency using the usual POX locking scheme. I have not yet routed the LO leg of the ALS_X beat through the delay line phase shifter - see next elog for details.

Update about the ALS MEDM screen slider: the trick was to change the OMSL field of the C1:LSC-BO_1_0 channel to "closed_loop" instead of "supervisory". Once this is done, the DOL value of the same channel can be set to the soft channel C1:ALS-DelayCalc, which sets the 16 bit binary string that controls the delay. Because arbitrary delays are not possible, I think it's more natural for the user to interact with this 16-bit binary string rather than the actual delay itself. So the MEDM screen has been slightly modified from what is shown in Attachment #1.

In the process of setting up some cabling at 1Y2, I must've bumped a cable to the c1lsc expansion chassis. Anyways, the c1lsc models crashed. I ran the reboot script around 530pm PDT. Usual locking behavior was recovered after this. The work at 1Y2 was:

• Ran a cable from X Beat power splitter ("LO" leg of the analog delay line) to variable delay line. 
• Ran cable from delay line to demodulator's LO input.
• Set up the SR785 for some CM board TF measurements.

The IN2 to CM board was already connected to I single ended output of the ALS X demodulator. The ~100 Hz UGF digital locking using the CM_SLOW path is straightforward but I didn't have any success with the AO path tonight. I wonder how high BW this lock can be made without injecting a ton of noise into the IMC loop, given that the EX uPDH only has ~ 10 kHz UGF.

Attachment #1 shows the spectra of the ALS signal 

• The two "CM Slow" traces are the digitized "SLOW" output of the common mode board, whose IN2 is connected to the demodulated I output of the analog delay line.
• The delay in the LO line of the analog delay line is adjusted to zero the DC value of this signal to best effort.
• These spectra are measured with the arm cavities POX/POY locked, and the EX laser locked to the arm cavity using the end PDH box.
• I simultaneously monitor the output of the digital phase tracker servo, and scale the CM Slow signal such that the spectra line up. The scaling factor required was to multiply the CM_SLOW signal x10 (CM board IN2 gain was set to +6dB, to account for the x2 gain in going from single ended to differential inside the ALS demodulator box).
• One puzzline feature is why switching on the ADC whitening makes the ALS spectrum noisier (even though it clearly changes the digitization noise floor). There is a peak that appears at ~ 8 kHz with the whitening on, and it may also be downconverted noise from some peak at higher frequencies I guess (if the AA isn't sharp enough). 

Attachment #2 is an OLTF measurement.

• In the blue trace, the arm length is controlled by using the CM Slow signal as an error signal, applying feedback to IMC length via MC2.
• In the red trace, I turned the digital MC2 violin notches off, and added upped the IMC IN2 gain to -12 dB (AO gain slider = 0dB).
• This was as high as I could go before the PC drive RMS began to go crazy.
• But still, there isn't any significant phase advance.
• It is possible I need to tack on a low-pass filter to prevent noise injection at higher frequencies...

I've made a cascaded passive 2-pole pomona box for fast ALS use, using LISO to check that it'll give the right shape when hooked up to the CM board's input stage. 

First stage is a 133Ohm + 10uF cap for ~120Hz LP, second is 1.15kOhm + 47nF cap for ~3.8kHz LP. The DC gain is ~0.75, which is much better than what I was doing before. The second stage would normally make a 2.9kHz LPF on its own, but the loading of the input stage moves the corner up. 

It seems the 133 Ohm resistor is a reasonable load on the output AD829 of the ALS demod board (short-circuit output current of 32mA and a series output resistor of 499Ohm). To be able to use the digitized ALSX I and the lowpassed analog version simultaneously, I had to buffer the signal with a SR560 before the pomona box, otherwise the signals looked distorted. This isn't a good long-term solution. Maybe I can used the further-buffered differential output to drive the LPF+CM board. 

The LISO files used to model the filter and CM board input stage, and fit the pole frequencies are attached. 

I made some attempts to get the AO path going today, but I suspect this daytime noise is just too much; the PC drive seems too irritable

I've been using an SR560 to experiment with differnent pole frequencies, to try and cancel the mystery zero. It's after the ALS demod board, before the pomona LPF with a gain of five. 

A pole frequency of 3kHz seems to recover sensible loop shapes. I've been able to crossover the AO path to make a nice long phase bubble which isn't the prettiest, but seems workable.

Getting to this point is now almost entirely scripted and repeatable; one just has to make sure that the ALS beat has the correct sign and adjust the delay line length. Most frustratingly, due to the dependence of the ALS gain on beat frequency / magnitude / delay, which can all vary on the order of a few dB, the AO gain settings to get to the crossed over point are not always the same, so at the end it's a lot of small steps and frequent loop measurements. 

The FSS crossover and overall IMC loop gain have to be pretty actively managed too. It's all too easy to drive the pockel's cell crazy. And if it's going crazy on its own anyways, there's no hope in trying to pile ALS sensing noise on top of it... It would really help in this effort to fix the whole PC situation up. 

Unfortunately, lock is lost when increasing the overall gain on the common mode board even by 1dB. We've seen in the single arm tests, that the gain settings have an appreciable difference in offset between them. Maybe this step is more than what the loop can handle? Or maybe it's the voltage glitches... Maybe some gain reallocation can put me on a region of the slider that glitches less.

In terms of the mystery plant features, I figure I'd like to take the analog TF of AO control signal to, say, AS55, and see what may or may not be there. I just haven't done this tonight since it would involve recabling the analyzer, and I still need frequent loop measurements to get to the crossed over state. Having ITMY misaligned and using the digital AS55Q spectrum as an out of loop monitor has been very helpful. 

To get around the problems between the pomona LPF and low CM board input impedance, I've placed the LPF at the CM board fast output. This won't work as a permanent solution, since we only want to lowpass the ALS signal, but it should be fine for a single arm test. 

However, I kept getting blown out of lock when turning up the AO gain, but well before I really expect any real action from the fast path. Looking at the OLTF, I was seeing some large spike at ~36kHz nearing 0dB loop gain with unstable phase. This prompted me to look at the ALS error signal out to higher bandwidth with the SR785; before I only ever looked at it through the digital system. 

So, with the X arm locked via POX11 I, and ITMY misaligned to use AS55 as an out of loop sensor, I measured the spectrum of the I ouput of the ALS X demod board (which was set to be near a zero crossing via the delay line), and the Q Mon of the AS55 demod board. 

Both ALS and AS55 show a sharp line at around 36.5kHz, so something is really happening in the IFO at this frequency. Koji might have seen an indication of this back in March.

What's going on here? And what would be different about PRFPMI that wouldn't have made this a problem for locking?

 I looked at REFL11 and REFL55 during PRMI lock - the line is there.

In fact, it is even visible in REFL11 I from a single bounce off of the PRM (ITMs misaligned).

This led me to look at the IMC error point (via the OUT2 on the servo board, no compensation for the input gain). Also there!

One the Wiki (https://wiki-40m.ligo.caltech.edu/40mHomePage), we have a Mech Resonance page for mechanical frequencies and a PEM page where we want to list the sources of all of our environmental lines. So please put in an entry when you find out what's at this frequency. This reminds me that I need to upload my MC2 COMSOL eigenmode analysis.

Fast ALS control continues to elude me. 

I fixed my LPF to take the input impedance of the CM board input into account; this unfortunately results in about -12dB DC gain of the ALS signal due to voltage-divider-y things, but by my estimation, this still puts the DFD noise above the input-referred voltage noise of the input AD829 on the CM board, so it'll do for now. The 120Hz pole shows up as expected when comparing the usual digital channels and the CM_SLOW output, and is digitally compensated with a zero at 120Hz (with a digital pole at 5k so nothing blows up). 

However, there seems to be some zero in the analog path somewhere that spoils the loop shape for the AO path. Here's a measurement of the X arm OLG from 10-100kHz, when the digital control is happening with ~100Hz UGF via ALS X I -> CM IN2 -> CM_SLOW -> LSC_CARM -> ETMX, and there is some AO action via ALS X I -> CM IN2 -> IMC IN2

The peak is recognizable as the gain peaking in the IMC servo (and changes predictably with changes to the IMC crossover and loop gains), which is expected. However, one can see that the magnitude is roughly flat before the peak, and the phase is around 0. With the 1/f LPF, we should see some downward slope and phase starting around -90. 

Thus, there must be some zero in the fast or common path, maybe at a few kHz where the digital loop wouldn't really see its effect. I'm not sure what it could be at this point in time.

One thought I had is that I never really checked the TF of DFD response to frequency modulation of the RF beat. I used an SR785 to drive the external FM input of a Fluke 1061A synthesizer, and saw it to be totally flat from 1-100kHz with carriers from 30-100MHz, so that should be fine. (For a little while I was confused by what seemed to be some heavy high-passing going on, but it turns out that the Fluke just can't push much low frequency FM; the manual says -3dB at 20Hz.)

INCORRECT INFO IN THIS ELOG HAS BEEN REMOVED. SEE THIS ELOG FOR THE UPDATED INFO.

Summary:

With the help of a tester board, I verified the mapping between fast BIO DB37 pins, and pins on the IDC50 connectors that are to be broken out to the whitening boards. I will enlist Chub to implement this mapping in hardware later today.

Details:

1. The LSC PD demodulated signals are optionally whitened before acquisition by our RTCDS ADCs.
2. The switching of each channel's whitening (enable/disable) is done by a single bit from the fast (a.k.a. RTCDS) system's BIO cards.
3. The whitening boards live inside Eurocrates.
4. The aforementioned switching signal needs to be sent to the whitening boards via the backplane of the Eurocrate.
5. This requires some cross-connect based cable splicing between the BIO card outputs and the P2 connectors of the whitening boards in the Eurocrates.
6. This connection was accidentally destroyed during the war on cross-connects at 1Y2. I couldn't find a wiring diagram anywhere.
7. Today, with the help of a tester board, I verified the mapping by toggling the appropriate channels on the MEDM screen, and verifying the correct LEDs on the tester board were toggled.
8. Map will be posted here after the meeting... Also now on the wiki.

Update 2019 Sep 19 1730: The pin numbers of the IDC 50 connector are all off by 1. i.e. 3-->4 and so on. I will fix this shortly. The problem was because of me looking at the pinout for the wrong gender of IDC50 connectors.

[KA, GV]

Summary:

1. New cross connect system for splicing the fast BIO signals for whitening switching to the P2 connectors was installed and tested at 1Y2.
2. It passed a first round of tests. 😁 
3. As of now, I believe all the necessary electrical connections have been made at 1Y2/1Y3, and we are ready for testing the c1iscaux system.

Details:

1. We did some testing in the office area, and found several wiring mistakes. These were all rectified. Attachment #1 is an accurate reflection of the implemented wiring scheme (softcopy in the 40m google sheets area). Be aware that the IDC 50 pin connector pin-out is tricky, and you have to be aware of the difference between male/female connector when looking for this pin-out on the internet.
2. In order to facilitate further testing, we re-routed the ADC0 SCSI cable that was unplugged on the overhead cable tray, and plugged it back into the c1lsc expansion chassis. This action necessitated a reboot of the vertex FEs, but everything came back alright.
3. Did some general neatenign and strain relieving. Removed a few existing cross-connects to make space for our new terminal blocks.
4. Attachment #2 shows the layout of the terminal blocks. Note the unusual (vertical) order of the orange terminal blocks.
5. The final integrated CDS test done was the following:
   • Set whitening gain for channel under test to 45dB, so that the dark noise level is boosted to a measurable level such that a change can be seen with the whitening enabled/disabled.
   • Compare the ASD of the signal between 30-100 Hz with the whitening engaged/disengaged.
   • Example result shown in Attachment #3.I believe the whitening is 15:150 (z:p) 

Tomorrow:

1. Recover POX/POY locking,.
2. ...

[Paco, Anchal-remote-support, Yuta]

We added fast channels to BHD DC PDs.
C1:LSC-DCPD_(A|B)_IN1 are now available, but C1:HPC-DCPD_(A|B)_IN1 still gives us zero.

c1hpc situation -> not good
 - We can see the slow signal at C1:X07-MADC1_EPICS_CH16 (DC PD A) and CH17 (DC PD B)
 - C1:HPC-DCPD_(A|B)_IN1 is there, but zero.
 - We have modified c1hpc model to add DCPD_(A|B) filters in front of the input matrix (see Attachment #1).
 - After modifying the model, we run
ssh c1sus2
rtcds make c1hpc
rtcds install c1hpc
ssh fb1
sudo systemctl restart daqd_*

 - After this, we got 0x2000 error. So, we ran the following. This removed 0x2000 error, but DCPD signals are still zero. They are also not available in C1HPC-MONITOR_ADC1.adl screen (see Attachment #3).
ssh c1sus2
rtcds restart c1hpc

c1lsc situation -> good
 - We could see the slow signal at C1:X04-MADC1_EPICS_CH4 (DC PD A) and CH5 (DC PD B), and also C1:LSC-DCPD_(A|B)_NORM after making C1:LSC-DCPD_(A|B)_POW_NORM=1. The ADC channel and DCPD channel are exactly the same.
 - After confirming the above, we modified the c1lsc model to add DCPD_(A|B) filters in front of the input matrix (see Attachment #2).
 - After modifying the model, we run
ssh c1lsc
rtcds make c1lsc
rtcds install c1lsc
ssh fb1
sudo systemctl restart daqd_*

 - After this, we also got 0x2000 error. We also noticed that, for example, C1:X04-MADC0_EPICS_CH31 and C1:LSC-ASDC_INMON are different, which used to be the same (ASDC_INMON was largely attenuated).
 - In the end, we run the following to remove 0x2000 error, but it crashed c1lsc, as well as c1sus, c1ioo.
ssh c1lsc
rtcds restart c1lsc

 - So, we did rebootC1LSC.sh. This made c1lsc, c1ioo and c1sus as green as before, except for RFM issue in TRX/TRY, like we saw in June. We followed the steps in 40m/16887 to hard reboot c1iscex/c1iscey and ran rebootC1LSC.sh again. This made C1CDS_FE_STATUS.adl screen as green as before (see Attachment #3).

 - Fast channels C1:LSC-DCPD_(A|B)_IN1 are now available. They are also available in C1LSC-MONITOR_ADC1.adl screen (see Attachment #3).

Rana and I now seem to have the fast front end computers (c1lsc, c1sus, c1ioo, c1iscex and c1iscey) up and running!  Hooray!

It seemed that we needed to change the soft links back to hard links for rtcds and rtapps on the front end machines.  On c1ioo, we did:

cd /opt

sudo rm -rf rtcds

sudo rm -rf rtapps

sudo mkdir rtcds

sudo mkdir rtapps

sudo chown controls:1001 rtcds

sudo chown controls:1001 rtapps

mount /opt/rtcds

mount /opt/rtapps

At this time, the front end fstab had several other options in addition to "nolock" for both rtcds and rtapps.  They had rw,bg,user,nolock.  This state still had some permissions problems.  (Later, we have decided that perhaps our next step was unneccesary, since it still left us with (fewer) permissions problems. Taking out the rw,bg,user options from the front end fstab seems to have fixed all permissions issues, so maybe this next chmod step didn't need to be done.  But it was done, so I record it for completeness).

On chiara, we did:

cd /home/cds/rtcds

sudo chmod -R 777 *

Then on c1iscex, I didn't have to deal with the soft links, but I did need to mount the rtcds and rtapps directories so that I could see files in them.  I just did the last 2 operations from the c1ioo list above (mount /opt/rtcds and mount /opt/rtapps). 

Since we were still seeing some (fewer) permissions problems, we took out the extra options in the front ends' fstab that Rana had added.  Rebooting c1iscex after this, everything came back as expected.  Nice!

I think that, at this point, remotely rebooting (sudo shutdown -r now) the other front ends made everything come back nicely. Since we had gotten the fstab situation correct, we didn't have to by-hand mount any directories, and all of the models restarted on their own.  Finally!

For posterity, here are things that we'll want to remember:

Frame builder's fstab, in /etc/fstab (only the uncommented lines, since there are lots of comments):

/dev/sdb1               /               ext3            noatime         0 1
/swapfile               none            swap            sw              0 0
shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0
/dev/sda1               /frames         ext3    noatime         0 0
192.168.113.104:/home/cds/                      /cvs/cds        nfs     _netdev,auto,rw,bg,soft      0 0
192.168.113.104:/home/cds/rtcds                  /opt/rtcds     nfs     _netdev,auto,rw,bg,soft 0 0
192.168.113.104:/home/cds/rtapps                 /opt/rtapps    nfs     _netdev,auto,rw,bg,soft 0 0


Fast front end fstabs, which are on the framebuilder in /diskless/root/etc/fstab:

master:/diskless/root                   /               nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
master:/usr                             /usr            nfs     sync,hard,intr,ro,nolock,rsize=8192,wsize=8192    0 0
master:/home                            /home           nfs     sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
none                                    /proc           proc    defaults          0 0
none                                    /var/log        tmpfs   size=100m,rw    0 0
none                                    /var/lib/init.d tmpfs   size=100m,rw    0 0
none                                    /dev/pts        devpts  rw,nosuid,noexec,relatime,gid=5,mode=620        0 0
none                                    /sys            sysfs   defaults        0 0
master:/opt                             /opt            nfs     async,hard,intr,rw,nolock  0 0
192.168.113.104:/home/cds/rtcds         /opt/rtcds      nfs     nolock                     0 0
192.168.113.104:/home/cds/rtapps        /opt/rtapps     nfs     nolock                     0 0

We ran again this method but with the 'b' parameter as a matrix instead. This provides more gain on some off-diagonal terms than others. This gave us a better convergence with the code reaching to the tolerance level provided (0.01 distance of S matrix from identity) within 16 iterations (~17 mins).

Attachment 1 again shows how the off-diagonal terms go down and how the overall distance of sensing matrix from identity goes down. This is 'Cross coupling budget' of the coils as iterations move forward.

Jumping to near zero-crossing:

• Rana mentioned a ezlockin code which first makes 5 step changes in output matrix without using feedback and calculates the changes required to reach zero-crossing in the behavior of the off-diagonal terms during these steps.
• This is similar to what we did above by hand where we increased the value of b for slowly converging off-diagonal elements.
• We plan to implement this 'jump' to near zero-crossing method next. Aim is to get a coil balancing code that does the job in ~5 min.
• We have been throwing away imaginary part of sensing matrix so far. We wanted to get to some owrking solution before we try more complex stuff. We have to figure out global phases in each transfer function estimate to rotate the measured transfer function appropriately.

convergence is great.

Next we wanna get the F2A filters made since most of the IMC control happens at f < 3 Hz. Once you have the SUS state space model, you should be able to see how this can be done using only the free'swinging eigenfrequencies. Then you should get the closed loop model including the F2A filters and the damping filters to see what the closed loop behavior is like.