[chub, gautam]

Installation: The following equipment were installed in 1Y3, see Attachment #1:

1. Supermicro server, which is the new c1iscaux machine, with IP Address 192.168.113.83.
2. 6U Acromag chassis which contains all the ADCs, DACs and BIO units.
3. 2 Sorensen DC power supplies to provide +24 V DC and +15 V DC to the Acromags.
4. Fusable DIN rail power blocks were installed on the North side of the 1Y3 rack - I placed 2 banks of 5 connectors each for +15 V DC and +24 V DC.

Removal: The following equipment was removed from 1Y3:

1. VME crates that were the old c1iscaux and c1iscaux2 machines.
2. Spare VME crate that used to be c1susaux, which Chub and I brought over to 1Y3 in an attempt to revive the broken c1iscaux2.
3. Approximately 30 twisted ribbon cables that were going to the cross connects. For now, we have not done a full cleanup and they are just piled along the east arm (see Attachment #2), beware if you are walking there!

Software: 

1. I connected the c1iscaux machine to the martian network.
2. Then I edited the relevant files on chiara to free up the IP addresses previously used by c1iscaux (192.168.113.81) and c1iscaux2 (192.168.113.82), and re-assigned the IP address used for c1iscaux to be 192.168.113.83.
3. I also changed the hostname of the c1iscaux machine (it was temporarily called c1iscaux3 to allow bench testing).
4. I moved the old /cvs/cds/caltech/target/c1iscaux and /cvs/cds/caltech/target/c1iscaux2 directories to /cvs/cds/caltech/target/preAcromag_oldVME/c1iscaux and /cvs/cds/caltech/target/preAcromag_oldVME/c1iscaux2 respectively.
5. I moved the temporarily named /cvs/cds/caltech/target/c1iscaux3 directory, from which I was running all the tests, to /cvs/cds/caltech/target/c1iscaux.
6. I edited all references to c1iscaux3 in the systemd files so that we can run the approriate systemd services.

Next steps: 

1. We did not get around to running the DB37 cables between the Acromag chassis and the 1Y2 Eurocrates today - this operation itself took the whole day as we also needed to lay out some support struts etc on the rack to support the Sorensens and the Acromag chassis.
2. Once the Acromags are connected to the Eurocrates, we have to run in-situ tests to make sure the appropriate functionality has been restored.
3. We must have bumped something in the c1lsc expansion chassis - the CDS FE overview screen is reporting some errors (see Attachment #3). I will fix this.
4. General tidiness, strain-relief etc.

Work done today:

1. All ribbon cable connections to the backplane of the 1Y2 Eurocrates were removed. The cables themselves were cleared for more space to work with.
2. 20x 15ft DB37 Cables were run between 1Y2 and 1Y3 via overhead cable tray.
3. Backplane interface boards were installed for 1Y2 Eurocrate boards.
4. Connections were made between the Acromag chassis and the eurocrate electronics modules.

Testing of functionality:

1. Fast BIO switching was verified to work for the following photodiodes:
   • AS55, AS110, REFL11, REFL33, REFL55, REFL165, POX11, POY11, POP22, POP110.
   • No light was incident on the PDs.
   • Test was done by increasing the whitening gain to +45 dB, and then looking at the ASD of the electronics noise between 50 Hz and 500 Hz with the whitening enabled/disabled. We expect x10 difference between the two states. This was seen.
2. "DetMon" channels were verified to work - see Attachment #1
   • Y-axis units is volts
   • Test was done by toggling the output of the 11 MHz Marconi, and looking for a change.
   • As seen in the attachment, all 5 monitor channels show a change.
   • This needs to be calibrated into some sensible units - I don't know why the different modulation frequencies have such different readbacks from supposedly identical Demod Board monitor points.
   • Not sure if the ~10 V reported by the REFL165 monitor point is real or saturated.
   • These channels are installed to signal/help debug the infamous ERA-5 decay problem, but maybe already some are decayed?
3. QPD interface channels were verified to work - see Attachment #2. 
   • Test was done by shining a green laser pointer on QPD quadrants.

Much testing remains to be done, but I defer further testing till Monday - the main functionality to be verified in the short run is the whitening gain stepping. The strain-relief of cables and general cleanup will be undertaken by Chub. Current state of affairs is in Attachment #3, leaves much to be desired in terms of cleanliness.

I will also setup the autoburt for the new machine on Monday. We will also need to add some channels to C0EDCU.ini if we want to trend them over some years (e.g. RF signal powers for monitoring ERA-5 health).

* c1lsc FE was rebooted using the usual script, and everything seems to be healthy in CDS-land again, see Attachment #4.

Here is what is left to do:

1. Strain relief of all cabling. Chub will take care of this in the coming days. I have said he can connect and disconnect cables as he pleases, but after this work, we may require a hard reboot of the Acromag chassis before restoring functionality to the channels, as it is known that the Acromags can sometimes get "stuck" by a sudden connection of voltage.
2. Installation of DB15 cable to the P2 connector of the CM board and a DB9 cable to the ALS demod unit (LO and RF power monitors). These will arrive in the next couple of days and Chub will take care of the install.
3. Design, manufacture and install of a custom version of the backplane P1 adaptor board with only 1 D37 connector - for some of the PD DC signals, a custom adaptor board, part number D010005 for which I can't find any schematics is already installed on the P2 connector, and makes the DC monitor signals available to 4 LEMO connectors. These signals are then digitized by the fast CDS system, presumably for PDH signal normalization. The footprint of this P2--->LEMO adaptor is such that we cannot simply install our P1---> 2xDB37 adaptor boards, because of space constraints. Fortunately, there is a simple fix to reduce the footprint of the board: remove the bottom DB37 connector, which is unused in the c1iscaux system except for the CM board. I recommend getting ~10 pcs of such boards, as it is also useful in a few other places, where the power cabling to the eurocrates are a space constraint. See Attachment #1 for a picture explaining this situation. Anyone want to volunteer to take care of this?
4. In-situ testing. This is easiest done with some light available in the interferometer. Which in turn requires IMC to be locked. Which in turn requires satellite box fixing. Anyone want to volunteer to take care of this?
5. Modify C0EDCU.ini to trend the new slow channels we may want long-term monitoring of (e.g. LO power levels to the Demod boards). Anyone want to volunteer to take care of this?
6. Decide what to do about the CM latch logic. There are some contraints with the way the acromag register addressing works, that I've had to change the way the mbboDirect bits are controlled. Unfortunately, this seems to sometimes and unpredictably cause the bits to flip in a non-robust way, which is the whole point of having the latch in the first place. Either the latch logic needs to be improved, or we need to implement the latch logic in the fast CDS system, not the slow.

Today I set up the autoburt.req file for the c1iscaux channels, and confirmed that the snapshots are getting recorded. There were a lot of channels in the old autoburt.req file which I thought were un-necessary (and several which no longer exist), so now the only channels that are burt-ed are the whitening gains and states of the AA filters. If someone feels we need more channels to be snapshot recorded, you can add them to the file.

In the old target directory, there were also various versions of a "saverestore.req" file - why do we need this in addition to an autoburt? I guess it is possible they are used by the IFOconfigure scripts to setup some whitening gains etc...

Summary:

I did some more investigation of what the appropriate cavity geometry would be for the OMC. Unsurprisingly, depending on the incident mode content, the preferred operating point changes. So how do we choose what the "correct" model is? Is it accurate to model the output beam HOM content from NPROs (is this purely determined by the geometry of the lasing cavity?), which we can then propagate through the PMC, IMC, and CARM cavities? This modeling will be written up in the design document shortly.

*Colorbar label errata - instead of 1 W on BS, it should read 1 W on PRM. The heatmaps take a while to generate, so I'll fix that in a bit.

Update 230pm PDT: I realize there are some problems with these plots. The critically coupled f2 sideband getting transmitted through the T=10% SRM should have significantly more power than the transmission through a T=100ppm optic. For similar modulation depth (which we have), I think it is indeed true that there will be x1000 more f2 power than f1 power for both the IFO AS beam and the LO pickoff through the PRC. But if the LO is picked off elsewhere, we have to do the numbers again.

Details:

Attachment #1: Two candidate models. The first follows the power law assumption of G1201111, while in the second, I preserved the same scaling, but for the f1 sideband, I set the DC level by assuming a PRG of 45, modulation depth of 0.18, and 100 ppm pickoff from the PRC such that we get 50 mW of carrier light (to act as a local oscillator) for 10 W incident on the back of PRM. Is this a reasonable assumption?

Attachment #2: Heatmaps of the OMC transmission, assuming (i) 0 contrast defect light in the carrier TEM00 mode, (ii) PRG=45 and (iii) 1 W incident on the back of PRM. The color bar limits are preserved for both plots, so the "dark" areas of the plot, which indicate candidate operating points, are darker in the left-hand plot. Obviously, when there is more f1 power incident on the OMC, more of it is transmitted. But my point is that the "best operating point(s)" in both plots are different.

Why is this model refinement necessary? In the aLIGO OMC design, an assumption was made that the light level of the f1 sideband is 1/1000th that of the f2 sideband in the interferometer AS beam. This is justified as the RC lengths are chosen such that the f2 sideband is critically coupled to the AS port, but the f1 is not (it is not quite anti-resonant either). For the BHD application, this assumption is no longer true, as long as the LO beam is picked off after the RF sidebands are applied. There will be significant f1 content as well, and so the mode content of the f1 field is critical in determining the OMC filtering performance.

There were a bunch of useless / degenerate channels added - e.g. whitening gains which are alreay burt-snapshot. Maybe there are many more useless channels being trended, but no need to add more.

Copy-pasting wasn't done correctly - the first 4 added channels were duplicates. There are in fact 5 LO power mons, one for each of the frequencies 11, 33, 55, 110 and 165 MHz. 

I cleaned up. Basically only the detect-mon channels, and the ALS channels, are new in the setup now. I will review if any extra channels are required later. While checking that the daqd is happy, I noticed c1lsc FEs are in their stuck state, see Attachment #1. I guess a cable was bumped when the strain relief operation was underway. I'm not attempting a remote resuscitation.

Chub wanted to get the correct part number for the replacement UPS batteries which necessitated opening up the UPS. To be cautious, all the workstations were shutdown at ~9:30am while the unit is pulled out and inspected. While looking at the UPS, we found that the insulation on the main power cord is damaged at both ends. Chub will post photos.

However, despite these precautions, rossa reports some error on boot up (not the same xdisp junk that happened before). pianosa and donatella came back up just fine. It is remotely accessible (ssh-able) though so maybe we can recover it...

I installed 6 of these in 1Y2. Three were for PD INTF #1-3, and I used three more for the AS110, REFL11, and REFL33 Demod board FEs, where the strain-reflief of the DC power cables to the Eurocrate was becoming a problem. So now there are only 4 units available as spares.

Once the strain-relieving of the Dsub cabling to 1Y3 is done, we can move ahead with testing. I'd like to put this to bed this week if possible.

1. Jordan (new Engineer) and Chub neatened out the cabling at 1Y2/1Y3 today. After their work, I plugged in all the Dsubs to the rear Eurocrate DB37->DIN96 adaptors. Jordan nicely fixed up the labels on the cable with some extra sellotape for a more durable label.
2. As part of the war on cross-connects, Chub removed some cables that were piping BIO signals from the fast CDS system to the whitening boards.
   • There is a SCSI to DB37 custom ribbon cable going from the BIO card in the expansion chassis to a 1U chassis box at the very bottom of 1Y2.
   • This 1U box, with DCC number D080478 (but no schematic exists on the DCC or any of the usual secret hidey-holes) breaks out the 32 BIO channels to 16+16.
   • Each set of 16 channels was supposed to get broken out to 8+8 via some cross connects and then goto the whitening boards. This is the part that got distrubed.
   • Koji and I discussed options - if Chub cannot resotre this easily, we will make a D37--> 4*D15 breakout board, and pipe the signals via the backplane P2 connectors. This will mean ~10 more days before the LSC system can be tested.
   • Some cabling to the TT DACs and an ADC were also disturbed, but these are easily restored.
3. From the hardware standpoint, some cross-struts for strain relief on the back of 1Y2 need to be installed --> Chub.

Let's not worry about C1LSC until the c1iscaux upgrade is done.

For some reason, the daqd_fw service was dead on FB. This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 . Simply restarting the fw service does not work, it crashes again after ~20 seconds. The problem may have to do with the indeterminate state of the c1lsc expansion chassis. However, this is not something that can immediately be fixed, as Chub is still working on the wiring there. So in summary, no frame data will be available until we fix this problem (it is still unclear what exactly the problem is). Team WFS can still work by getting online data.

Why were the CDS overview DC indicators not red???

Unrelated to this work: I had to key the c1psl crate to get the IMC autolocker functioning again. However, I found that the key 🔑 turns continuously - as opposed to having two well defined states, ON and OFF. Be careful while handling this.

Came across this while looking up the BIO situation at 1Y2. For reference, the fix Koji mentions can be seen in the attached screenshot (one example, the other BIO cards also have a similar fix). The 16th bit of the BIO is grounded, and some bit-shifting magic is used to implement the desired output.

Summary:

1. Frames seem to be written again.Slowly but surely, we are converging to an operable state...
2. No frames are available for the period 23 Aug to 17 September 2019
3. Don't edit the C0EDCU.ini file unless you know what you're doing.
4. If you make some changes to the RT system/channel list or reboot FEs, please make sure all the dependent systems are back up and running. There shouldn't be a need to willy-nilly reboot things.
5. Tomorrow I will prepare the map of BIO channels for Chub to restore the whitening switching capability. Then we can try locking some cavities.

Details:

1. First, I checked to make sure the /frames partition wasn't full. It wasn't.
2. Next, I looked into the C0EDCU.ini file.
   • The last date for which frames are available, 23 Aug, coincided with the date when this file was modified.
   • It is a known problem that the daqd_fw service can crash if one of the channels in this file is reporting an unusually large number.
   • Several channels were added to this file - in the end, only 9 new ones were required, 5x "DetectMon" channels for each of the RF demodulation frequencies, and 4 for the new ALS LO and RF signal power monitor channels.
   • It is highly likely that one of the other channels was what caused the daqd_fw service to crash - though I can't say for sure, because I did not exhaustively search through the ~100 un-necessary channels that were in this file to see what values they were reporting.
3. For good measure, I ran the reboot script, and brought the c1lsc models back online.
   • I want to do the mapping of the BIO channels to the pin-out of the BIO adaptor unit, which requires c1lsc to run.
   • Reboot script ran smoothly.
4. Then I went into fb and restarted all the daqd services. This time, they all seem to run without crashing, at least in the ~10min window it took me to type out this elog.

Unrelated to this work: c1auxey was keyed.

INCORRECT INFO IN THIS ELOG HAS BEEN REMOVED. SEE THIS ELOG FOR THE UPDATED INFO.

Summary:

With the help of a tester board, I verified the mapping between fast BIO DB37 pins, and pins on the IDC50 connectors that are to be broken out to the whitening boards. I will enlist Chub to implement this mapping in hardware later today.

Details:

1. The LSC PD demodulated signals are optionally whitened before acquisition by our RTCDS ADCs.
2. The switching of each channel's whitening (enable/disable) is done by a single bit from the fast (a.k.a. RTCDS) system's BIO cards.
3. The whitening boards live inside Eurocrates.
4. The aforementioned switching signal needs to be sent to the whitening boards via the backplane of the Eurocrate.
5. This requires some cross-connect based cable splicing between the BIO card outputs and the P2 connectors of the whitening boards in the Eurocrates.
6. This connection was accidentally destroyed during the war on cross-connects at 1Y2. I couldn't find a wiring diagram anywhere.
7. Today, with the help of a tester board, I verified the mapping by toggling the appropriate channels on the MEDM screen, and verifying the correct LEDs on the tester board were toggled.
8. Map will be posted here after the meeting... Also now on the wiki.

Update 2019 Sep 19 1730: The pin numbers of the IDC 50 connector are all off by 1. i.e. 3-->4 and so on. I will fix this shortly. The problem was because of me looking at the pinout for the wrong gender of IDC50 connectors.

Summary:

The custom ribbon cables piping the coil driver board outputs to the eLIGO (?) TTs (a.k.a. TT1 and TT2) are damaged. They need to be re-made. I can't find any pin-mapping for them.

Details:

While waiting for the LSC photodiode whitening switching cross-connect work to be done, I thought I'd re-align the IFO a bit. However, I was unable to find any beam making it to the REFL/AS ports despite some TT steering. I remembered that Chub had undone the TT connections at 1Y2 as well, and thought I'd check the cabling to make sure all was in order. On going to the rack, however, I found that these connections were damaged at the coil-driver end (see Attachment #1), presumably during the cable extraction. These need to be re-made...😔 

While debugging this problem, c1lsc models crashed. I ran the reboot script this morning to bring the models back. There was a 0x4000 error on the DC indicators for the c1lsc models (mx_stream error which couldn't be fixed by restarting the mx service) the first time I ran the script so I did it again, now the indicator lights are in their nominal state.

False alarm - the mistake was mine. Looking at the schematic diagram, the AI/Dewhite board, D000316, accepts the inputs from the DAC on the P2 connector. While restoring the connections at 1Y2, I had plugged the outputs of the DAC interface board into the P1 connectors of the AI boards. Having rectified this problem, I am now able to move the beam on the AS camera in both PIT and YAW using TT1 or TT2. So to zero-th order, this subsystem appears to work. A more in-depth analysis of the angular stability of the TTs can only be done once we re-align the arms and lock some cavities.

[KA, GV]

Summary:

1. New cross connect system for splicing the fast BIO signals for whitening switching to the P2 connectors was installed and tested at 1Y2.
2. It passed a first round of tests. 😁 
3. As of now, I believe all the necessary electrical connections have been made at 1Y2/1Y3, and we are ready for testing the c1iscaux system.

Details:

1. We did some testing in the office area, and found several wiring mistakes. These were all rectified. Attachment #1 is an accurate reflection of the implemented wiring scheme (softcopy in the 40m google sheets area). Be aware that the IDC 50 pin connector pin-out is tricky, and you have to be aware of the difference between male/female connector when looking for this pin-out on the internet.
2. In order to facilitate further testing, we re-routed the ADC0 SCSI cable that was unplugged on the overhead cable tray, and plugged it back into the c1lsc expansion chassis. This action necessitated a reboot of the vertex FEs, but everything came back alright.
3. Did some general neatenign and strain relieving. Removed a few existing cross-connects to make space for our new terminal blocks.
4. Attachment #2 shows the layout of the terminal blocks. Note the unusual (vertical) order of the orange terminal blocks.
5. The final integrated CDS test done was the following:
   • Set whitening gain for channel under test to 45dB, so that the dark noise level is boosted to a measurable level such that a change can be seen with the whitening enabled/disabled.
   • Compare the ASD of the signal between 30-100 Hz with the whitening engaged/disengaged.
   • Example result shown in Attachment #3.I believe the whitening is 15:150 (z:p) 

Tomorrow:

1. Recover POX/POY locking,.
2. ...

While working on recovering interferometer alignment, I noticed that the ETMX Oplev SUM channel reported 0 counts. Attachment #1 shows the 200 day trend - despite the missing data, the accelerating downward decay is evident. I confirmed that there is no light coming out of the HeNe by walking down to EX. The label on the HeNe says it was installed in March 2017, so the lifetime was ~30 months. Seems a little short? I may replace this later today.

I was hoping that the dark / electronics noise level on the LSC photodiodes would be sufficient for me to test the whitening gain switching on the iLIGO Pentek whitening boards. However, this does not seem to be the case. I guess to be thorough, we have to do this kind of test. It's a bit annoying to have to undo and redo the SMA connections, but I can't think of any obvious easier way to test this functionality. More annoyingly, the sensing matrix infrastructure necessary to do the kind of test described in the linked elog is only available for some PDs. I don't really want to modify the c1cal model and go through another mass reboot cycle.

While I was at it, I was also thinking about the tests we want to do. Here is a quick first pass - if you can think of other tests we ought to do, please add them to the list!

1. Whitening gain switching on the D990694 boards.
   • Need to inject some signal to do this in a clean way. 
   • With some signal injected, we need to switch the whitening gain through the 15 available levels and confirm that we see a +3dB gain for each step.
   • An example script to do this operation and make a diagnostic plot is at /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py.
2. AA enable/disable on the D000076 boards. Do we really need this functionality? Can't we permanently enable the AA, as was done for WF2?
   • Need to measure the TF with an SR785 or drive a high-freq line and confirm that the aliased peak height is attenuated as expected in DTT.
3. LO Det Mon channel check
   • Zeroth level test can be done by turning Marconi OFF/ON, and confirming we see a change in the corresponding monitor channel, like I did here.
   • A more rigorous diagnostic would require these channels to be calibrated to dBm of LO power.
4. PD INTF board check
   • Zeroth level check can be done by shining light onto PDs one at a time and confirming that the correct channel shows a response.
   • A more rigorous diagnostic would require these channels to be calibrated to mW of optical power incident on the PDs.
5. QPD INTF board check
   • This is the IP-POS QPD readback.
   • Need to confirm the quadrant mapping, and that Pitch is really Pitch, Yaw is really Yaw.
   • A more rigorous diagnostic would require these channels to be calibrated to mm of position shift.
6. CM Board
   • Need to determine what tests need to be done.
   • I have not yet implemented the fix for the MBBO gain channels for all the gains - only REFL1_GAIN is set up correctly now. Need to look at the hardware for the correct addressing of bits.
7. ALS INTF board
   • This board isn't actually connected yet, pending strain relief of cabling at 1Y2.
   • The calibration of the board output volts to dBm is known, so we can easily check this functionality.

I tried to lock the Y arm cavity length to the PSL frequency using POY11_I as an error signal. Even though I think the cavity alignment is good (I see TRY flashes ~0.8), I am unable to achieve a lock. I checked the signal conditioning, and as far as I can tell, all the settings are correct, but there may be some settings that have not been re-assigned correct values. The other possibility is that something is not quite right with the new c1iscaux. The PDH error signal and arm cavity flashes all seem good though (see Attachment #1), so I'm not sure what obvious thing I'm missing.

To be continued...

I reset the normalization for both arms on Jul 9 2019.

Summary:

There is no visible PDH error signal on the POX11 channels. As a result, I am unable to lock the XARM length to the laser frequency. See Attachment #1 - the Y arm length is locked to the PSL frequency, and control is disabled for the XARM servo.

Details:

Now that several of the c1iscaux functionality tests have been completed, I wanted to push ahead with some locking. However, I was foiled at this early stage, for reasons as yet unknown. One possibility is that the

• I am able to see TRX cavity flashes >0.8, which suggest to me that the cavity is well aligned.
• Moreover, I am able to lock some (admittedly high TEM order) mode of the green laser, which further supports the above hypothesis.
• However, there are no visible PDH-like features in the POX11_I or POX11_Q channels.
• I checked that the cables from the output of the POX11 demod board are in fact going to the correct channels on WF1 (#5 and #6 respectively), and that the whitening gain for this channel is set to the nominal +30 dB.
• Next, I went to the POX table and looked for the POX IR beam. I couldn't see anything, but this beam is expected to be weak (of the order of 1 W * T_PRM * R_AR_ITM ~ 30 uW), which is probably not so easily visible.

Next steps:

• Look for the POX beam with an IR viewer.
• Confirm that everything is order on the LSC Demod board for POX 11 - maybe the LO isn't connected (somehow)?

To facilitate POX locking investigations, I replaced this HeNe today with one of the spares Chub/Steve had acquired some time ago. Details:

• Part number: Lumentum 22037130 (1103P)
• Serial number: PA00836
• Manufacture date: 01/2019
• Power output: ~2.64 mW (Measured with Ophir power meter in the 632nm setting)
• Power received on QPD: ~0.37 mW = ~18700 cts (Measured with Ophir power meter in the 632nm setting)

The RIN of the sum channel with the Oplev servo engaged, along with that for the other core FPMI optics, in shown in Attachment #1. The ETMX HeNe RIN is compatible with the other HeNes in the lab (the high-frequency behaviour of the BS Oplev is different from the other four because the QPD whitening electronics are different).

Not sure what to make of the ETMY RIN profile being so different from the others, seems like some kind of glitchy behaviour, I could see the mean level of the ASD moving up and down as I was taking the averages in DTT. Needs further investigation.

The old / broken HeNe is placed i(nside the packaging of the abovementioned replacement HeNe) on Steve's old desk for disposal in the proper way.

*It looked like Steve had hooked up a thermocouple to be able to monitor the temperature of the HeNe head. I removed this feature as I figured if we don't have this hooked up to the DAQ, it isn't a really useful diagnostic. If we want, we can restore this in a more useful way.

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I modified /cvs/cds/caltech/target/c1iscaux/latch.py and /cvs/cds/caltech/target/c1iscaux/C1_ISC-AUX_CM.db to set up the mbbo logic for the other three channels on the CM board, namely REFL2 Gain, AO Gain, and the Super boosts. The systemctl processes were restarted on c1iscaux. We are now ready to perform systematic checks on the CM board functionality.

Remarks:

The addressing of the Acromag BIO registers is done in a way that is kind of inconvenient to use the EPICS mbboDirect protocol

• The control word going to the Acromag is 16 bits in length
• However, only the 4 least significant bits actually correspond to physical channels - the remaining 12 bits are "unused".
• Because each Acromag BIO unit has 16 BIO channels, this means that they are grouped into four "banks" of 4 bits each.
• The mbboDirect EPICS/modbus protocol is used to control multiple physical BIO channels using a single input, which is exactly what we want for the gain sliders on the CM board. However, one caveat is that the bits need to be consecutive.
• This means that we have to break up the 6 bits used for the gain sliders (and in fact also the 2 bits used for the super boosts) into a least-significant-bits (LSB) group and a most-significant-bits (MSB) group.
• What's more annoying is that our physical wiring scheme means that we can't uniformly decide on how this division into LSBs and MSBs work for all the channels - e.g. for REFL1 Gain, the LSB is the 4 least significant bits, while the MSB is the 2 most significant ones, while for REFL2 Gain, the roles are reversed.
• In hindsight, the "clever" way to do the wiring assignment would have been to factor this in - but the problem is (sort of) easily fixed in software, and so I recommend we stick with the existing wiring scheme.

I tested the new latch.py script by toggling the various sliders (one at a time) between two values and monitoring the states of the various soft and "*_BITS" channels, see Attachment #1. The behavior seems consistent to me, but to be sure, we have to use Koji's LED tester board and confirm that the physical bits are being toggled correctly. The StripTool templates live in /cvs/cds/caltech/target/c1iscaux/CMdiag.

I confirmed that there is light incident on the POX photodiode. So the problem must lie downstream in the demod / whitening / AA electronics. With the PRM aligned (i.e. PRFPMI config with all DoFs uncontrolled), I could see the flashing beam on an IR card. I could also see the spikes in DC power incident on the photodiode using the "DC Monitor" port on the photodiode head and an oscilloscope.

Update 245 pm: I confirmed that I could see a 11 MHz sine wave by connecting the POX11 RFPD output cable at the 1Y2 end to an oscilloscope. The amplitude of this signal was also changing, corresponding to the cavity fringing in and out of resonance. I couldn't, however, see any signal on the RFPDmon port, or the I/Q demodulated output ports. So as of now, the culprit seems to be something on the Demod board. Further investigations underway...

Update 315pm: I did the following checks:

1. Checked the LO signal level into a 50ohm input scope - it was ~720 mVpp, which was compatible with the LO level into the POY Demod board, so the LO signal level couldn't be to blame.
2. Connected an RF funcgen to the PD input of the demod board. Drove it at 11.066210 MHz, 50 mVpp, and saw a signal 400 cts-pp in the CDS system - so the demod + digitizaiton electronics also seemed fine.
3. #2, coupled with the fact I could see no signal at the RF-mon port of the demod board (even though there was a signal visible at the cable coming to 1Y2) suggested that the cable routing the POX11 PD output from the Heliax-breakout in 1Y2 to the demod board was busted - indeed this was the case!
4. Koji replaced the cable without changing its length, and now the XARM locks readily 👏 . I ran ASS and got TRX ~ 0.95. See Attachment #1. 

I did the following:

1. symlinked /cvs/cds/rtcds to /opt/rtcds.
2. Added a line to /etc/systemd/system/modbusIOC.service that executes a burt-restore of the latest c1iscaux.snap file so that whitening gains etc are restored to their last saved value in the event of a service restart.

I made some model changes to c1lsc. To propagate the changes, I tried the usual rtcds make sequence. But I got an error about the model file not being in the path. This is down to my re-organization of the paths to cleanly get everything under git version control. So I had to run the following path modification. Where is this variable set and how can I add the new paths to it? The model compilation, installation and restart all went smooth after I made this change. 

For smooth reboot of the models, I used the reboot script. I had to restart the daqd processes on FB, but now all the CDS indicator lights are green.

Attachment #1 shows a first look at the IR ALS noise after my re-coupling of the IR light into the fiber at EY. 

Measurement configuration: 

• Each arm length was individually stabilized to the PSL frequency using POX/POY locking.
• The respective AUX laser frequencies were locked to the arm cavity length using the AUX PDH loops.
• GTRX ~0.3 (usually I can get ~0.5) and GTRY ~ 0.2 (the mode-matching to the arm cavities is pretty horrible as suggested by the multitude of bullseye modes seen when toggling the shutter).
• The control signal to the AUX PZT had the DC part offloaded by the slow temperature control servos to the AUX laser crystal temperature.

CDS model changes:

• The c1lsc model was modified to route the input signals to the Y phase tracker servo from ADC1_2 and ADC1_3 (originally, they were ADC0_20 and ADC0_21).
• This change was necessary because the DFD output is sent differentially to the ADC1 card in the c1lsc expansion chassis (bypassing the iLIGO whitening and AA electronics, for now just going through an aLIGO AA board with no whitening available yet).
• I chose to use the differential receiving (as opposed to using the front-panel single ended BNC connectors) as in principle, it is capable of delivering better noise performance.
• After making the model changes, I compiled and restarted the model. Apart from the missing path issue, the compile/restart went smoothly.

Next steps:

• Get the easy fixes done (better GTRX, GTRY).
• Test the noise with POX and POY as the OOL sensors, and the arms controlled using the ALS error signal - this is the relevant metric for how ALS will be used in locking.
• Noise budget. Need to double-check the DFD output calibration into Hz.
• For the general interferometer recovery, I think I will push ahead with trying to lock some other configurations like the PRMI (should be easy to recover), DRMI (potentially more difficult to find the right settings), and the FPMI (I'd like to use this config to get an estimate for how much contrast defect we have in the interferometer, but I think it'll be pretty challenging to lock in this configuration).

1. With TRX and TRY maximized using ASS, I centered the Oplev spots on the respective QPDs for the four test masses and the BS. I also centered the spot onto the IPPOS QPD by moving the available steering mirror.
2. At EX, I tweaked the input pointing of the green beam into the arm by manually twiddling with the PZT mirrors. I was able to get GTRX~0.4.
3. On the AS table - Koji and I found that there was a steering mirror placed in the AS beam path such that there was no light reaching the AS110 or AS55 PDs. Please - when you are done with your measurement, return the optical configuration to the state it was in before so that the usual locking activity isn't disturbed by a needless few hours troubleshooting electronics.

Once Koji is done with his checkout of the whitening electronics, I will try and lock the PRMI.

Summary:

The PRMI was locked with the carrier field resonant in the PRC 🙌. The lock is pretty stable (I only let it stay locked for ~10mins and then deliberately unlocked to see if I could readily re-lock, but it has stayed locked for the last ~20mins while I typed this up). See Attachment #1 for the DC power monitor StripTool for a short section of lock.

Details:

• This is the opposite of the config we'd want usually for locking the IFO, but it is a useful configuration for setting the alignment of the vertex optics, and also to train angular feedforward filters, so I decided to try it out.
• Some patient alignment work was required. I started with the single arm locks, maximized TRX/TRY with ASS, and then misaligned the ETMs and brought the PRM into alignment.
• The PRM Oplev spot was roughly centerd on its QPD once I judged I was getting decent PRMI cavity flashes on the POP camera. The PRMI Oplev servo needs some tuning, it is currently susceptible to oscillations in Pitch.
• The error signals used were: REFL11_I ---> PRCL and AS55_Q ---> MICH.
• The whitening gains were: REFL11 --> +18 dB, AS55 ---> +6 dB.
• Triggering was done using POPDC, this worked better for me than any of the RF signals (e.g. POP22/POP110). Trigger ON --> 200cts, Trigger OFF --> 100 cts.
• The DCPD whitening gains may not be set correctly - I think I remember POPDC being ~4000 cts in this configuration, but it may also be that we are not well centered on the POP photodiode.
• The dominant cause of the POP circulating power seems to be the usual angular instability ascribed to the TTs. The OAF model wasn't running tonight (and I didn't want to try starting it and have to do a full vertex FE reboot tonight) so I didn't get a chance to engage the angular FF.

Next (for LSC activities):

• PRMI locking with the sidebands resonant in the PRC.
• DRMI locking

I'm leaving the LSC mode off for tonight, but with the PRMI optics aligned and ETMs misaligned.

This morning, I restarted the c1oaf model on the c1lsc machine, so as to have the option of enabling some feedforward action. Unsurprisingly, the "DC" indicator is red, citing a "0x2bad". In the past, I've been able to correct this by simply restarting the model. But given the fragility of the c1lsc machine, I think I'll live with not having the OAF model signals in frames. Medium-term, I'd like to pare down the c1oaf model a bit - I think it has way too many options/matrices right now, and is an un-necessarily bloated and heavy model. Unless there are serious objections, I will do this work when I next feel like it.

The anaconda distribution used by the control room workstations is actually installed on the shared drive (/cvs/cds/ligo/apps/anaconda/) for consistency reasons. The version was 4.5.11. I ran the following commands to update it today. Now it is version 4.7.12.

The second command takes a while to resolve conflicts, so I've left it running inside a tmux session for now.

Recall that the bash alias for using the anaconda managed python is "apython". I recommend everyone set up a virtual environment when trying out new package installs, to avoid destroying the locking scripts.

I measured the OLTF of both the PRM Oplev loops. Nothing odd sticks out as odd to me in this measurement - there seems to be ~40 degrees of phase margin and >10 dB gain margin for both loops, see Attachment #1. I didn't measure down to the second UGF at ~0.2 Hz (the Oplev loops are AC coupled), so there could be something funky going on there. The problem still persists - if I misalign and realign the PRM using the ifoalign scripts, the automatic engagement of Oplev loops causes the loop to oscillate. Could be that the script doesn't wait for long enough for the alignment transient to die out.

Update 1230pm: Indeed, this was due to the integrator transient. It dies away after a couple of seconds.

Summary:

I was able to lock the FPMI. The lock was quite stable. However, the fluctuations in the ASDC power suggest that it will be difficult to make a DC measurement of the contrast defect in this configuration. This problem can be circumvented in part by some electronics tuning. However, the alignment jitter couples some HOM light which is an independent effect. Can this be a good testbed for the proposed AS WFS system? 

Details:

• First, the arm cavities were locked and TRX/TRY were maximized using ASS.
• Next, AS55_Q-->MICH_A (MICH-->BS) matrix element was set to 1 in the LSC input (output) matrix. The trigger was set to always on.
• AS55 digital demod phase was -37 degrees.
• I was then able to increase the gain on the MICH servo and turn on some integrators without any problem.
• Some guesswork had to be done to get the correct sign. Final servo gain used was -0.8. 

I didn't do any serious budgeting yet - need to think about / do some modeling on how this configuration can be made useful.

Today, I found out that this type of "0x2bad" DC error is connected to the 1e+20 cts output. The solution was to bite the bullet and stop/start the c1oaf model (at the risk of crashing the vertex FEs). Today, I was lucky and the model came back online with all CDS indicators green. At which point I was able to engage length feedforward to MC2 (with some admittedly old filter). Some subtraction is happening, see Attachment #1. This was just meant to test whether the signal routing is happening - the feedforward signal goes to the "ALTPOS" input of the suspension CDS block, which AFAIK does not have a corresponding MEDM EPICS indicator. So I couldn't figure out whether the feedforward control signal was in fact making it to the suspension. On the evidence of the suppression of MCL in the 1-3 Hz band, I would conclude that it is. Useful to be able to engage these FF filters for better lockability.

I propose the following re-organization of the PDFR measurement breadboard. We have all the parts on hand, just needs ~30mins of setup work and some characterization afterwards. The fiber beamsplitter will not be PM, but for this measurement, I don't think that matters (the patch fiber from the diode laser head isn't PM anyways). We have one spare 1 GHz BW NF1611 that is fiber coupled (used to live on the ITMY in-air table, and is (conveniently) labelled "REF DET", but I'm not sure what the function of this was). In any case, we have at least 1 free-space NF1611 photodiode available as well. I suggest confirming that the FC version works as expected by calibrating against the free space PD first.

Update 245pm: Implemented, see Attachment #2. Aaron is testing it now, and will post the characterization results.

Summary:

There is an imbalance between the POX and POY detector outputs reported in the CDS system. Possibilities are (i) the POX PD has a uncoated glass window whereas POY does not or (ii) there is some problem in the elctronics.

Details:

1. Nominally, we run the POX/POY locking with +18dB whitening gain on POY and +30 dB on POX. This is a factor of 4 difference.
2. The DC levels reported in C1:LSC-POXDC_OUT and C1:LSC-POYDC_OUT differ by a factor of 10 (24 cts for POY vs 2.4 cts for POX with 0dB whitening gain). These channels come from the P2 connector on the back of the PD Interface board into the fast CDS system.
3. The levels reported by the Acromag system (which come out of the P1 connector) are 60mV for POY  vs 15 mV for POX.
4. I confirmed that this imbalance is not due to clipping on the POX photodiode - I tweaked the steering mirror and observed the plateau (I did not, however, look at the beam on the PD active area with an IR viewew which would be a more conclusive test).
5. I measured the power incident on either PD (using Ophir power meter, filter OFF). They were both ~10uW, as expected since the beam extraction for POY and POX are identical - a single HR mirror and the vacuum viewport.

Update 820pm: 

1. I checked that there is no glass window on the PD.
2. It is hard to see the beam on a viewer - but with the PRM aligned, I think I convinced myself that the beam is pretty well centered on the PD. 

So increasingly, it looks like the electronics are the source of the problem.

I think the metric of interest here is the consistency of the AC transimpedance of the proposed new "Reference PD" (= fiber coupled NF1611) vs the old reference (free space NF1611), since everything will be calibrated against that.

Summary:

I managed to achieve a few transitions of control of the XARM length using the ALS error signal. The lock is sort of stable, but there are frequent "glitches" in the TRX level. Needs more noise hunting, but if the YARM ALS is also "good enough", I think we'd be well placed to try PRMI/DRMI locking with the arms held off resonance (while variable finesse remains an alternative).

Details:

Attachment #1: One example of a lock stretch. 

Attachment #2: ASD of the frequency noise witnessed by POX with the arm controlled by ALS. The observed RMS of ~30pm is ~3-4 times higher than the best performance I have seen, which makes me question if the calibration is off. To be checked...

[KA, GV]

This elog is meant to be a summary of some of the many subtleties on the CM board. The latest schematic of the version used at the 40m can be found at D1500308 .

Latch logic:

• There are several Binary Outputs and one Binary Input to the CM board.
• The outputs control ENABLE/DISABLE switches and gains of amplifier stages, while the input reports whenever the limiter has been reached.
• The variable gain feature is implemented by enabling/bypassing several cascaded fixed gain stages. So in order to change the gain of a single composite amplifier stage, multiple individual amplifier stages have to be switched.
• This is implemented by the user interacting with the hardware via a "control word", consisting of a number of bits depending on the number of cascaded stages that have to be switched. 
• This control word is sent to the device via modbus EPICS, which is an asynchronous communication protocol. Hence, it may be that the individual bits composing the control word get switched asynchronously. This would be disastrous, as there can be transient glitches in the gain of the stage being controlled. 
• To protect against such problems, there is a latch IC in the hardware between the Binary Inputs to the board (= Binary Outputs from Acromags), and the actual switches (= MAX333) that enable/bypass the cascaded gain stages. The latch IC used is a SN74ALS573. This device acts as a bus, which transmits/blocks changes for multiple bits (= our control word) from propagating, depending on the state of a single bit (= the LATCH ENABLE bit). Thus, by controlling a single bit, we can guarantee that multiple bits get switched synchronously. 
• In order to use this latch capability, we need some software logic that sets/disables the LATCH ENABLE bit. For our system, this logic is implemented in the form of a continuously running python 🐍 script, located at /cvs/cds/caltech/target/c1iscaux/latch.py. It is implemented as a systemctl service on the c1iscaux Supermicro. The logic implemented in this script is shown in Attachment #1. While the channels referred to in that attachment are for REFL1_GAIN, the same logic is implemented for REFL2_GAIN, AO_GAIN, and the SuperBoosts.
• Some FAQ:
  1. Q: Why do we need the soft channels C1:LSC-REFL1_SET_LSB and C1:LSC-REFL1_SET_MSB?
     A: These soft channels are what is physically linked to the Acromag Binary Outputs. In order for our latch logic to be effective, we need to detect when the user asks for a change, and then disable the LATCH ENABLE bit (which is on by default, see FAQ #3) before changing the physical acromag channels. The soft channels form the protective layer between the user and the hardware, allowing latch.py to function.
  2. Q: Why is there an "_MSB" and "_LSB" soft channel? 
     A: This has to do with the mbboDirect EPICS channel type, which is used to control the multiple bits in our control word using a single input (= an MEDM gain slider). The mbboDirect data-type requires the bits it controls to have consecutive hardware addresses. However, the Acromag hardware addressing scheme is not always compatible with this requirement (see pg 33 of the manual for why this is the case). Hence, we have to artifically break up the control word into two separate control words compatible with the Acromag addressing scheme. This functionality is implemented in latch.py.
  3. Q: Why is the default state of LATCH ENABLE set to ON? 
     A: This has to do with the fact that all Binary Inputs, not just the multi-bit ones, to the CM board are propagated to the control hardware via a latch IC. For the single-bit channels, there is no requirement that the switching be synchronous. Hence, rather than setting up ~10 more single-bit soft channels and detecting changes before propagating them, we decided to leave the LATCH ENABLE ON by default, and only disable it when changing the multi-bit gain channels. This is the same way the logic was implemented in the VME state code, and we think that there are no logic reasons why it would fail. But if someone comes up with something, we can change the logic.

Acromag BIO testing:

During my bench testing of the Acromag chassis, I had not yet figured out mbboDirect and the latch logic, so I did not fully verify the channel mapping (= wiring inside the Acromag box), and whether the sitching behavior was consistent with what we expect. Koji and I verified (using the LED tester breakout board) that all the channels have the expected behavior 👏. Note that this is only a certification at the front-panel DB37 connectors of the Acromag chassis  testing of the integrated electronics chain including the CM board is in progress...

Summary:

I improved the alignment of the green beam into the Y arm cavity.

• GTRY went from ~0.2 to ~0.25, see Attachment #1.
• This resulted in improvement of the Y arm ALS noise above 💯Hz by a factor of ~5, see Attachment #2.
• I tried controlling the two arm cavities in the CARM/DARM basis using ALS error signals - but didn't manage to successfully execute this transition today - this will be the commissioning goal for the upcoming week.

Details:

• I had to do the alignment by tweaking the steering mirrors at EY - the PZTs didn't give me anywhere near enough range.
• While I was at EY, I tried moving the two MM lenses mounted on translation stages to try and improve the mode-matching into the arm cavity - wasn't successful, still see a bunch of bullseye modes when I toggle the shutter.
• They EY green layout would benefit from a do-over (basically just copy the EX layout), but this isn't the priority right now, the ALS noise RMS is dominated by low frequency noise (as usual). 
• There is a ~5% leakage of the GTRX beam onto the GTRY photodiode.
• One thing to try would be to revive the MCL loop to reduce the <1 Hz laser frequency noise and see if that helps - basically testing this hypothesis.
• I had done some careful noise-budgeting of the EX green PDH system, the EY system would benefit from the same, but not critical.
• The improvement of the high-frequency noise is clear, and now we are consistent with the "known good reference" level from the time the DRFPMI locking was working back in early 2016.

Other changes made today:

1. /opt/rtcds/caltech/c1/scripts/general/videoscripts/videoswitch was modified to be python3 compatible - for some reason, there were many syntax errors being thrown (even though I was using python2.7) and I wasn't able to change the displays in the VEA using the MEDM screen, but now it works again 👍.
2. The LSC overview and several daughter MEDM screens were edited to remove references to channels that no longer exist. All screens I edited have a backup stored in the MEDM directory with today's date as a suffix.
3. Input pointing into the PMC was tweaked.
4. Noted that some pump is noisy at pumpspool - also noted that the annuli are no longer pumped. Some event seems to have triggered an interlock condition that closed off the annular volume from TP3, needs investigation...

Summary:

I managed to execute the first few transitions of locking the arm lengths to the laser frequency in the CARM/DARM basis using the IR ALS system 🎉 🎊 . The performance is not quite optimized yet, but at the very least, we are back where we were in the green days.

Details:

1. Locking laser frequency to Y arm cavity length using MC2 as a frequency actuator
   • This is the usual diagnostic done to check the single-arm ALS noise using POY as an out of loop sensor.
   • The procedure is now scripted - I had to guess the sign and optimize the gains a few times, but this works deterministically now. 
   • Script lives at /opt/rtcds/caltech/c1/scripts/YARM/Lock_ALS_YARM.py.
   • Attachment #1 shows the result. If we believe the POY sensor calibration, the RMS displacement noise is ~6 pm
2. Encouraged by the good performance of the Y arm, I decided to try the overall transition from the POX/POY basis to the CARM/DARM basis using ALS error signals.
   • The procedure starts with the arm cavities locked with POX/POY, and the respective green frequencies locked to the arm cavity length by the end PDH servos.
   • The DFD outputs serve as the ALS error signals - the PSL frequency is adjusted to the average value of DFD_X_OUT and DFD_Y_OUT.
   • I changed the LSC output matrix element for DARM-->ETMX from -1 to -5, to make it symmetric in actuation force w.r.t. ETMY (since the series resistane on ETMX is x5 that on ETMY).
   • After some guesswork, I fould the right signs for the gains. After enabling the boosts etc, I was able to keep both arms (approximately) on resonance for several minutes. See Attachment #2 for the time series of the transition process - the whole thing takes ~ 1 minute. 
   • A script to automate this procedure lives at /opt/rtcds/caltech/c1/scripts/ALS/Transition_IR_ALS.py.
   • The transition isn't entirely robust when executed by script - the main problem seems to be that in the few seconds between ramping off the IR servos and enabling the CARM/DARM integrators/boosts, the DARM error-point offset can become rather large. Consequently, when the integrator is engaged, ETMX/ETMY get a large kick that misalign the cavity substantially, degrade the green lock, and destroy the CARM lock as well. The problem doesn't seem to exist for the CARM loop. 
   • Anyways, I think this is easily fixed, just need to optimize sleep times and handoff gains etc a bit. For now, I just engage the DARM boosts by hand, putting in a DARM offset if necessary to avoid any kicking of the optic.
   • Attachment #3 shows the length noise witnessed by POX/POY when the arm cavities are under ALS control. If we believe the sensor calibration, the RMS displacement noise is ~15 (20) pm for the Y (X) arm.
   • This is rather larger than I was hoping would be the case, and the RMS is dominated by the <1 Hz "mystery noise".
   • Nevertheless, for a first pass, it's good to know that we can achieve this sort of ALS performance with the new IR ALS system.

Over the week, I'll try some noise budgeting, to improve the performance. The next step in the larger scheme of things is to see if we can lock the PRMI/DRMI with CARM detuned off resonance.

See trend. This is NOT symptomatic of some frozen slow machine - if I disable the WFS servo inputs, the lock holds just fine.

Turns out that the beam was almost completely missing the WFS2 QPD. WTF 😤. I re-aligned the beam using the steering mirror immediately before the WFS2 QPD, and re-set the dark offsets for good measure. Now the IMC remains stably locked. 

Please - after you work on the interferometer, return it to the state it was in. Locking is hard enough without me having to hunt down randomly misaligned/blocked beams or unplugged cables.

I took this opportunity to do some WFS offset updates.

• First I let the WFS servo settle to some operating point, and then offloaded the DC offsets to the IMC suspensions.
• Then I disabled the WFS servo.
• I hand-tweaked MC1 and MC3 PIT/YAW (while leaving MC2 untouched) to minimize IMC REFL (a more sensitive indicator of the optimal cavity alignment than the transmission).
• Once I felt the IMC REFL was minimized (~1-2% improvement), I set the RF offsets for the WFS while the IMC remained locked. I chose this way of setting the RF offsets as opposed to unlocking the cavity and having the high-power TEM00 mode incident on the WFS QPDs.
• Overnight, I'm going to run the MC2 spot position scanning code (in a tmux session on pianosa, started ~945pm) to see if we can find a place where the transmission is higher, looking at Kruthi's code now to see it makes sense...
• The convergence time of the MC2 spot position loop is pretty slow, so the scan is expected to take a while... Should be done by tomorrow morning though, and I expect no work with the IFO tonight.
• Does this loop have to be so slow? Why can't the gain be higher?

Yesterday, Koji and I noticed (from the wall StripTool traces) that the vertex seismometer RMS between 0.1-0.3 Hz in the X-direction increased abruptly around 6pm PDT. This morning, when I came in, I noticed that the level had settled back to the normal level. Trending the BLRMS channels over the last 24 hours, I  see that the 0.3-1 Hz band in the Z direction shows some anomalous behaviour almost in the exact same time-band. Hard to believe that any physical noise was so well aligned to the seismometer axes, I'm inclined to think this is indicative of some electronics issues with the Trillium interface unit, which has been known to be flaky in the past.

Summary:

There is ~ 7% variation in the power seen by the MC2 trans QPD, depending on the WFS offsets applied to the MC2 PIT/YAW loops. Some more interpretation is required however, before attributing this to spot-position-dependent loss variation inside the IMC cavity.

Analysis:

Attachment #1: This shows a scatter plot of the MC2 transmission and IMC REFL average values after the WFS loops have converged to the set offset positions. The size of the points are proportional to the normalized variance of the quantity. The purpose of this plot is to show that there is significant variation of the transmission, much more than the variance of an individual datapoint during the course of the averaging (again, the size of the circles is only meant to be indicative, the actual variance in counts is much smaller and wouldn't be visible on this plot scale). For a critically coupled cavity, I would have expected that the TRANS/REFL to be perfectly anti-correlated, but in fact, they are, if anything, correleated. So maybe the WFS loops aren't exactly converging to optimize the inoput pointing for a given offset? 

Attachment #2: Maps of the transmission/reflection as a function of the (YAW, PIT) offset applied. The radial coordinate does not yet mean anything physical - I have to figure out the calibration from offset counts to spot position motion on the optic in mm, to get an idea for how much we scanned the surface of the optic relative to the beam size. The gray circles indicate the datapoints, while the colormaps are scipy-based interpolation. 

Attachment #3: After talking with Koji, I explicitly show the correlation structure between the IMC REFL DCMON and MC2 TRANS. The shaded ellipses indicate the 1, 2 and 3-sigma bounds for the 2D dataset going radially outwards. The correlation coefficient for this dataset is 0.46, which implies moderate positive correlation. 🤔 

Scan algorithm:

The following was implemented in a python scipt:

1. Choose 2 independent random numbers from the uniform distribution in the interval [-0.5, 0.5] (in uncalibrated counts).
2. One of these numebrs is set as the error point offset for the QPD spot-centering PITCH WFS loop, while the other is the YAW offset.
3. Wait for 600 seconds - this long wait is required because the step-response time for these loops is long. 
4. If there is an MC unlock event - wait till the MC relocks, and then another 600 seconds, to give the WFS loops sufficient time to converge.
5. Once the WFS loops have converged, average a few data channels (MC TRANS, REFL, WFS loop error points etc) for 10 seconds, and write these to a file.

I am now setting the offsets to the WFS QPD loop to the place where there was maximum transmission, to see if this is repeatable. In fact it was. Looking at the QPD segment outputs, I noticed that the MC2 transmission spot was rather off-center on the photodiode. So I went to the MC2 in-air optical table and centered the beam till the output on the 4 segments were more balanced, see Attachment #4. Then I re-set the MC2 QPD offsets and re-enabled the WFS servos. The transmission is now a little lower at ~14,500 counts (but still higher than the ~14200 counts we had before), presumably because we have more of the brightest part of the beam falling on the gap between quadrants. For a more reliable measurement, we should use a single-element photodiode for the MC2 transmission.

In preparation for some locking work tonight, I did the following at the POP in air table with the PRMI locked on carrier:

1. Raised the POP camera by ~5mm. The POP spot is now well centered on the CCD view.
2. Tweaked alignment onto the PDA10CF photodiode that serves as (i) POP22, (ii) POP110, and (iii) POP DC. In lock the POPDC level went from ~800 cts to ~1200 cts.
3. Moved the QPD that witnesses part of the POP beam such that the spot was centered on the photodiode. This may be useful for collecting some FF data or if we want to try feedback to stabilize the PRMI.

TBC...

Looking at the old latch.st code, looks like this is just a heartbeat signal to indicate the code is alive. I'll implement this. Aesthetically, it'd be also nice to have the hex representation of the "*_SET" channels visible on the MEDM screen.

After making sure the beams were hitting the 3f photodiodes on the "AP" table, I was able to lock the PRMI with the sidebands resonant inside the RC using 3f error signals. This would be the config we run in when trying to lock some more complicated configuration, such as the PRFPMI (i.e. start with the arms controlled by ALS, held off resonance). Tonight, I will try this (even though obviously I am not ready for the CARM transition step). The 3f lock is pretty robust, I was able to stay locked for minutes at a time and re-acquisition was also pretty quick. See Attachment #1. Not sure how significant it is, but I set the offsets to the 3f paths by averaging the REFL33_I and REFL33_Q signals when the PRMI was locked with the 1f error signals.

As usual, there's a lot of angular motion of the POP spot on the CCD monitor, but the lock seems to be able to ride it out.

Lock-settings (I modified the .snap file accordingly):

REFL33_I --> PRCL, loop gain = -0.019, Trigger on POP22, ON @ 20cts, OFF@0.5cts.

REFL33_Q --> MICH, loop gain = +1.4, Trigger on POP22, ON @ 20cts, OFF@0.5 cts.

This problem has re-surfaced. Is this indicative of some problem with the on-board VGA? Even with 0dB of whitening gain, I see PDH horns that are 10,000 ADC counts in amplitude, whereas the nominal whitening gain for this channel is +18dB. I'll look at it in the daytime, not planning to use REFL55 for any locking tonight.

Summary:

1. ALS control of arms in the CARM/DARM basis seems pretty robust - I was able to hold lock for >40mins tonight. The scripted transition from POX/POY control to ALS control is pretty deterministic now.
2. The PRMI could be locked with the arms detuned from resonance by applying an offset to the CARM loop error point.
3. Much daytime work remains to be done before attempting any sort of reliable locking.

Hardware issues that need addressing:

1. Both EX and EY Trans QPDs need a look. I believe the one at EY is simply blocked (on account of the mode spectroscopy project), while the one at EX shows a weird discontinuity between the Thorlabs PD and the QPD. Could be just a gain/normalization issue I guess. See Attachment #1.
2. While the PRMI stayed locked, I don't think I was using anywhere close to optimal settings. Need to run some sensing lines, measure transfer functions etc, to make the PRMI + arms lock more robust. The PRMI always lost lock when I brought the CARM offset to 0. Could also benefit from some finesse modeling I guess. I could not get a reliable estimate of what the PRG is tonight, because the PRMI didn't stay locked as I approached 0 CARM offset.
3. REFL 55 whitening board needs a checkup.