[rika, aaron]

At Seiji and Gautam's suggestion, we added an additional RF photodiode (NewFocus 1611) to the system so we can calibrate our transfer functions. The configuration is now laser -> BS --> lenses -> QPD and BS --> lenses -> RFPD. We added lenses to get the beams focused on the RFPD and QPD heads, and are again set up for TF measurement.

We took the following data. These parameters were consistent across all measurements:

• 1kHz IF BW
• log sweep with 801 points
• 32 averages
• auto attenuation
• -10 dBm excitation amplitude
• 19.2 mA DC current to the laser
• The DC level of the reference PD is -, and with the beam blocked (dark current) it is

After taking the data for segment 1, I moved the beam to segment 2. The beam didn't fit on segment 2 without partially illuminating segment 1 (tested by maximizing the signal on segment 2, then blocking the beam. If the beam is entirely on one segment, only that segment should be effected; in this case, we found that segment 1's DC signal also changed when the beam was blocked). We readjusted the telescoping lenses to get the beam a bit smaller, and now the beam fits on segment 2. We know it is entirely on segment 2 because small beam movements do not change the signal on segment 2.

We are trying to take the remaining data, but AGmeasure keeps hanging while sending the data (after taking the measurement, over 10 min). We tried restarting the network analyzer to no avail. I was able to grab the data by cancelling the measurement and running

I've uploaded the spectrum for segment 1 in the meantime. Zero model is on the way.

When I finished up the measurements on WFS2, I removed the cables from the AP table and closed the cover.

EDIT: I forgot to switch the LEMO connector to measure the other segments, so we measured the RF signal from segment 1 even when the beam was on segments 2-4. We'll have to try again tomorrow.

[rika, aaron]

We are at it again. Rika is setting up the TF measurement, I'm looking into scripting the WFS sensing matrix measurement we made earlier in the week so we can return to it next week.

When we mesuring TF of SEG4, the beam leaking to SEG1 about 1%.

We finished mesurement SEG2-4 and get the figure by running PDH_calibrate.ipynb .

edit: We observed during segment 2 measurements that blocking the beam reduced the DC level of segment 1 by less than 1%, but still clearly observable. As you can see in the plots, something is suspicious about the normalization of these TFs. We took segment 1 data a few days before the other segments, so perhaps we weren't getting the full beam on the reference PD during the later measurements? When I make this measurement for WFS1, I will try to fix some of these problems by choosing different telescoping optics, and I will consider whether removing the QPD heads from their table will improve the measurement.

I'm scripting the WFS sensing matrix measurements. I haven't really scripted DTT before, so I'm trying to find documentation or existing scripts. I came across this elog where Gautam measured a sensing matrix during DRMI lock, and he pointed me to some .xml files used for these measurments.

The PCB board of the adapter for DIN 96pin to DSUB37 conversion (single DSUB version) was delivered yesterday and I quickly soldered the connectors.

They are ready for use and stored in a JLCPCB cardboard box on a pile of acromag stuff. (Note that the lacel is written on the box with Sharpie)

not need to use DTT. I'm attaching some half-finished notebooks that give the gist.

1. Download the data with NDS2
2. Downsample the data for ease of use.
3. save the data as hdf5 for easy loading later.
4. demodulate the data at the specified frequencies.

That's it! Now you have the complex, single frequency TFs. Next you invert the matrix.

I installed 6 of these in 1Y2. Three were for PD INTF #1-3, and I used three more for the AS110, REFL11, and REFL33 Demod board FEs, where the strain-reflief of the DC power cables to the Eurocrate was becoming a problem. So now there are only 4 units available as spares.

Once the strain-relieving of the Dsub cabling to 1Y3 is done, we can move ahead with testing. I'd like to put this to bed this week if possible.

[rika, aaron]

We aligned optics of WFS as it was. Now auto-locker is working to lock MC.

But it still doesn't lock. We notice that the c1lsc machine doesn't work. So we run rebootCILSC.sh.

Now we reset the hardware!

17:11

After reset, auto locking didn't work well. Gautum and Aaron reboot slow c1ioo. Then it works, and Gautam returned the MC to a good alignment.

We found the beam is not in the center of the QPD, we (turned off the MC autolocker and MC loop, then) realigned to make beam to get in to the QPD center. Afterwards we start auto locking.

With the WFS on, the maximum MC transmission we observe is 14,700 counts; after the transmission level stabilizes (MC_TRANS pit and yaw brought to 0), the MC transmission is only 14,200 counts. Perhaps the MC_TRANS QPD offsets need adjustment. We relieve the WFS servo of its DC offsets. This is the configuration we'll use for WFS loop measurements this week.

When I put away the lenses we had used for measuring the RF transfer functions of the QPD heads, I saw that I'd removed them from the cabinet containing green endtable optics, but hadn't noticed the sign forbidding their removal. I'll talk with Koji/Gautam about what happened and what should be done.

I wanted to make a zero model of this circuit to get a handle on the results. I couldn't import zero on pianosa, and I tried pip installing zero, but was denied due to not finding version 3.0.3 of matplotlib. I finally got it to install using

 Oddly, even though I can now import zero when I open a python3 session from the command line, when I open a jupyter notebook and switch to a python3 kernel, the zero module is still unavailable. I think I recall that conda manages the jupyter environment -- is pip managing an entirely separate environment (annoying)?

edit: Yeah, it was something like that. I reminded myself how this works with this article.

We noticed last week that the MC2 trans camera has pitch and yaw swapped; I rotated what I thought is the correct camera by 90 degrees clockwise (as viewed from above, like in the attachment), but I now have doubts. It's the camera on the right in the attachment.

The left one is analog and 90deg rotated.

See also: This issue tracker

1. Jordan (new Engineer) and Chub neatened out the cabling at 1Y2/1Y3 today. After their work, I plugged in all the Dsubs to the rear Eurocrate DB37->DIN96 adaptors. Jordan nicely fixed up the labels on the cable with some extra sellotape for a more durable label.
2. As part of the war on cross-connects, Chub removed some cables that were piping BIO signals from the fast CDS system to the whitening boards.
   • There is a SCSI to DB37 custom ribbon cable going from the BIO card in the expansion chassis to a 1U chassis box at the very bottom of 1Y2.
   • This 1U box, with DCC number D080478 (but no schematic exists on the DCC or any of the usual secret hidey-holes) breaks out the 32 BIO channels to 16+16.
   • Each set of 16 channels was supposed to get broken out to 8+8 via some cross connects and then goto the whitening boards. This is the part that got distrubed.
   • Koji and I discussed options - if Chub cannot resotre this easily, we will make a D37--> 4*D15 breakout board, and pipe the signals via the backplane P2 connectors. This will mean ~10 more days before the LSC system can be tested.
   • Some cabling to the TT DACs and an ADC were also disturbed, but these are easily restored.
3. From the hardware standpoint, some cross-struts for strain relief on the back of 1Y2 need to be installed --> Chub.

Let's not worry about C1LSC until the c1iscaux upgrade is done.

I'm using the notebooks from rana as a starting point, and making a script to measure and fill the WFS sensing matrix. It lives at /users/aaron/WFS/scripts/WFSsensingMatrix.ipynb for now. Here's what it does; what's been tested is in green, untested is goldenrod, uncoded is fire brick.

1. Sets up an nds connection, listening to the WFS channels and the MC#_PIT/YAW IN1 channels.
2. Loops over the excitation channels. For now, I'm assuming the user is injecting excitations one at a time in awggui; in principle, we could excite the various MC angular dof at several frequencies and take a single measurement, or use the natural frequencies of the suspensions.
   1. For each excitation, grab the data
   2. Filter the data. I'm using a 30 Hz to 40 Hz cheby filter
   3. Take an FFT, hold on to that for future reference
   4. Generate an LO at the excitation frequency, and demodulate the signals. Strong low pass.
   5. The single-frequency transfer function is now [WFS channel] / [excited MC channel]. Each iteration of this loop generates a column of the sensing matrix.
3. Invert the sensing matrix
4. Populate in the appropriate channels of the WFS_OUTMATRIX

Grabbing data with nds

To run these on pianosa, I ran (inside the jupyter notebook)

I'm getting an error when starting the nds2 connection

conn = nds2.connection('192.168.113.201', 31200)
Failed to establish a connection[INFO: Request SASL authentication protocol]+

 I didn't find anything on the elog about this error, but I'm looking at the nds user manual. The problem was, I didn't have a valid Kerberos ticket; I opened one on Pianosa with my albert.einstein (note all caps ligo.org).

 I'm now able to run the scripts Rana mentions, but I haven't been able to grab the channels I want (eg C1:SUS-MC1_ASCPIT_IN1_OUT); it says the channel isn't found. When I check how many of the Caltech channels are available (conn.count_channels('C1*')), there are none. I was connecting to nds.ligo.caltech.edu, but this must be the wrong server (it has all the channels for the sites). fb and fb1 (and the IP they point to, 192.168.113.201) cannot be connected to, giving the error 'Error occurred trying to write to socket.'

I recall that in the cryo lab, we need to use port 8088 to get data from cymac1, and indeed substituting 31200 -> 8088 lets me access the C1 channels (I can count the channels), but no matter what time I request, nds tells me there is no data available (gap). Gautam came by and diagnosed that the gaps I'm seeing in the frames' data are real, fb is down (see elog).

WFS Sensing Matrix Script

Saving extra channels

Continuing, I'm going to modify the script to grab live data. I'm using the iterate and next methods. I noticed that the MC2_TRANS pit/yaw channels are not saved to frames, even though WFS1/2 pit/yaw are. Since I expect I'll want to lookback at these channels, I followed the instructions for adding a daq channel, uncommenting the following line in /opt/rtcds/caltech/c1/chans/daq/C1IOO.ini:

I made a backup of the old version of this .ini file, which can be found in /users/aaron/backups/190917_C1IOO.ini. I did not remake the model, as I couldn't find the c1ioo model in /opt/rtcds/caltech/c1/userapps/trunk or from the matlab command prompt. I restarted the fb via telnet, but didn't restart the model or check the svn (got an error?). The _DQ channels are now reachable on dataviewer, so things seem to be working.

awgpy

I also tried importing cdsutils, so I can control awg in the same script that we read out the sensing matrix, but I'm getting the python3 error when I import cdsutils:

I tried pip upgrading cdsutils, but it's already up-to-date. I get the above error even if I switch to a python 2 kernel; cdsutils is installed in the python2.7 directory, so I don't know why pip is finding it when I'm running a python 3 kernel. I can move on from this for now, but it would be useful to be able to script the excitation along with the measurement.

Changes to the user environment

jupyter on donatella

Tangentially related, Rika wanted to be running some jupyter notebooks while working on donatella. I ran, on donatella:

 hm, that didn't work. Also jupyter is installed when you install conda, so I'm not sure how there is a version of conda but not of jupyter. I also see that pip and pip3 are not recognized commands on donatella.

scipy on pianosa

I noticed that some of the functions in the scipy signal processing toolbox were out of date on pianosa. The cheby and welch filters now accept additional kwargs (for eg, before you needed to give IIR filter methods a cutoff frequency normalized to the Nyquist rate, but now you can give it the frequencies and sampling rate separately).

I want to update this package, but I hesitate to break everyone's existing scripts.

[aaron, rika]

Once stop the auto-locker and realigned to make beam to get into QPD again.

After we lock MC, we took TFs from suspension MC1/2/3 PIT/YAW to WFS1/2 PIT/YAW. 

----- 

Diagnotics test tools

range: 7 Hz to 50 Hz

 avarage=61

Column 0: WFS2_PIT   1: WFS2_YAW   2:WFS1_PIT   3: WFS1_YAW   4: TRANCE_PIT   5:TRANCE_YAW 

-----

I'm wondering weather the MC1data I saved is correct, becouse I found the channel was changed when I exported MC2 data. So I took MC1 data again.

We got all data for TFs already.  Each data is devided to real part and imaginary part. Then we are arranging the datas to obtain TFs. 

TF of MC2 is attachiment 1. So tomorrow, I make other TF.

For some reason, the daqd_fw service was dead on FB. This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 . Simply restarting the fw service does not work, it crashes again after ~20 seconds. The problem may have to do with the indeterminate state of the c1lsc expansion chassis. However, this is not something that can immediately be fixed, as Chub is still working on the wiring there. So in summary, no frame data will be available until we fix this problem (it is still unclear what exactly the problem is). Team WFS can still work by getting online data.

Why were the CDS overview DC indicators not red???

Unrelated to this work: I had to key the c1psl crate to get the IMC autolocker functioning again. However, I found that the key 🔑 turns continuously - as opposed to having two well defined states, ON and OFF. Be careful while handling this.

Came across this while looking up the BIO situation at 1Y2. For reference, the fix Koji mentions can be seen in the attached screenshot (one example, the other BIO cards also have a similar fix). The 16th bit of the BIO is grounded, and some bit-shifting magic is used to implement the desired output.

Summary:

1. Frames seem to be written again.Slowly but surely, we are converging to an operable state...
2. No frames are available for the period 23 Aug to 17 September 2019
3. Don't edit the C0EDCU.ini file unless you know what you're doing.
4. If you make some changes to the RT system/channel list or reboot FEs, please make sure all the dependent systems are back up and running. There shouldn't be a need to willy-nilly reboot things.
5. Tomorrow I will prepare the map of BIO channels for Chub to restore the whitening switching capability. Then we can try locking some cavities.

Details:

1. First, I checked to make sure the /frames partition wasn't full. It wasn't.
2. Next, I looked into the C0EDCU.ini file.
   • The last date for which frames are available, 23 Aug, coincided with the date when this file was modified.
   • It is a known problem that the daqd_fw service can crash if one of the channels in this file is reporting an unusually large number.
   • Several channels were added to this file - in the end, only 9 new ones were required, 5x "DetectMon" channels for each of the RF demodulation frequencies, and 4 for the new ALS LO and RF signal power monitor channels.
   • It is highly likely that one of the other channels was what caused the daqd_fw service to crash - though I can't say for sure, because I did not exhaustively search through the ~100 un-necessary channels that were in this file to see what values they were reporting.
3. For good measure, I ran the reboot script, and brought the c1lsc models back online.
   • I want to do the mapping of the BIO channels to the pin-out of the BIO adaptor unit, which requires c1lsc to run.
   • Reboot script ran smoothly.
4. Then I went into fb and restarted all the daqd services. This time, they all seem to run without crashing, at least in the ~10min window it took me to type out this elog.

Unrelated to this work: c1auxey was keyed.

For the investigation of the latch logic issue for the CARM CM board, I have made the LED logic checkers with DB breakout boards. They require the pull up voltage supply of +15V because the acromag digital out is a open corrector (well... open "source") output.

The logic from Pin1 to Pin16 of DB37 can be monitored. The DB15 connector is only for monitoring the latch enable logic.

What Gautam and I found with the logic outputs was that the latch logic works fine but occasionally we found that the top 2 bits and the bottom 4bit were processed independently.

[Koji Gautam]

We continued to check the latch logic. Today we found that latch.py didn't catch the change of LSB but did for MSB. We determined that this happens when the slider value is chaged between the polling for LSB and MSB.
SInce these two should always be related to a single gain value, latch.py was modified so.

Now we don't observe any logic error for ~100 gain transisitions (see attached).

INCORRECT INFO IN THIS ELOG HAS BEEN REMOVED. SEE THIS ELOG FOR THE UPDATED INFO.

Summary:

With the help of a tester board, I verified the mapping between fast BIO DB37 pins, and pins on the IDC50 connectors that are to be broken out to the whitening boards. I will enlist Chub to implement this mapping in hardware later today.

Details:

1. The LSC PD demodulated signals are optionally whitened before acquisition by our RTCDS ADCs.
2. The switching of each channel's whitening (enable/disable) is done by a single bit from the fast (a.k.a. RTCDS) system's BIO cards.
3. The whitening boards live inside Eurocrates.
4. The aforementioned switching signal needs to be sent to the whitening boards via the backplane of the Eurocrate.
5. This requires some cross-connect based cable splicing between the BIO card outputs and the P2 connectors of the whitening boards in the Eurocrates.
6. This connection was accidentally destroyed during the war on cross-connects at 1Y2. I couldn't find a wiring diagram anywhere.
7. Today, with the help of a tester board, I verified the mapping by toggling the appropriate channels on the MEDM screen, and verifying the correct LEDs on the tester board were toggled.
8. Map will be posted here after the meeting... Also now on the wiki.

Update 2019 Sep 19 1730: The pin numbers of the IDC 50 connector are all off by 1. i.e. 3-->4 and so on. I will fix this shortly. The problem was because of me looking at the pinout for the wrong gender of IDC50 connectors.

[aaron, rika]

Gettng TFs

In the data we got yesterday, we can see some filter's effect. 

But it is not good coherence above 10Hz, so we mesured again. And this time we save the data as xml file.

And also we chaned the frequency regions broader to watch corner frequency of suspension.

-----

 Diagnotics test tools

 range: 0.1 Hz to 100 Hz

 points: 120 

 Amplitude: 1000

----

but at low frequency, the mode maching cavity was unloked cause of too much shaking.

So, we saw single frequency TF, and searched the good amplitude.

First, I tried to get TF @0.1~1 Hz .

-----

0.1 to 1 Hz

points: 61 (I think it's too much becous it takes about an hour)

amplitude: 5

-----

The TFs and coherence of MC1/PIT to each QPD is below. [above window: coherence, below: TF]

During the mesurement, something happened @0.2-0.3Hz so I stopped it.

We found the coherence of WFS1P and WFS2Y is not good, but others are good.

we guess that it could come from alignment which made Q chainging to small.

Finaly, I also got the  .xml data of MC1P 1 Hz to 10 Hz. In this time,

-----

1 to 10 Hz

points: 41 

amplitude: 90

-----

Making matrics

Now we took single frequency 6 TFs (MC1/2/3 PIT/YAW) @7Hz (Because this frequency has good coherence in all channel).

Aaron wrote the script using dtt to making matrics. 

Summary:

The custom ribbon cables piping the coil driver board outputs to the eLIGO (?) TTs (a.k.a. TT1 and TT2) are damaged. They need to be re-made. I can't find any pin-mapping for them.

Details:

While waiting for the LSC photodiode whitening switching cross-connect work to be done, I thought I'd re-align the IFO a bit. However, I was unable to find any beam making it to the REFL/AS ports despite some TT steering. I remembered that Chub had undone the TT connections at 1Y2 as well, and thought I'd check the cabling to make sure all was in order. On going to the rack, however, I found that these connections were damaged at the coil-driver end (see Attachment #1), presumably during the cable extraction. These need to be re-made...😔 

While debugging this problem, c1lsc models crashed. I ran the reboot script this morning to bring the models back. There was a 0x4000 error on the DC indicators for the c1lsc models (mx_stream error which couldn't be fixed by restarting the mx service) the first time I ran the script so I did it again, now the indicator lights are in their nominal state.

False alarm - the mistake was mine. Looking at the schematic diagram, the AI/Dewhite board, D000316, accepts the inputs from the DAC on the P2 connector. While restoring the connections at 1Y2, I had plugged the outputs of the DAC interface board into the P1 connectors of the AI boards. Having rectified this problem, I am now able to move the beam on the AS camera in both PIT and YAW using TT1 or TT2. So to zero-th order, this subsystem appears to work. A more in-depth analysis of the angular stability of the TTs can only be done once we re-align the arms and lock some cavities.

New DIMM cards have arrived. I stored them in the digital cabinet along y arm.

[KA, GV]

Summary:

1. New cross connect system for splicing the fast BIO signals for whitening switching to the P2 connectors was installed and tested at 1Y2.
2. It passed a first round of tests. 😁 
3. As of now, I believe all the necessary electrical connections have been made at 1Y2/1Y3, and we are ready for testing the c1iscaux system.

Details:

1. We did some testing in the office area, and found several wiring mistakes. These were all rectified. Attachment #1 is an accurate reflection of the implemented wiring scheme (softcopy in the 40m google sheets area). Be aware that the IDC 50 pin connector pin-out is tricky, and you have to be aware of the difference between male/female connector when looking for this pin-out on the internet.
2. In order to facilitate further testing, we re-routed the ADC0 SCSI cable that was unplugged on the overhead cable tray, and plugged it back into the c1lsc expansion chassis. This action necessitated a reboot of the vertex FEs, but everything came back alright.
3. Did some general neatenign and strain relieving. Removed a few existing cross-connects to make space for our new terminal blocks.
4. Attachment #2 shows the layout of the terminal blocks. Note the unusual (vertical) order of the orange terminal blocks.
5. The final integrated CDS test done was the following:
   • Set whitening gain for channel under test to 45dB, so that the dark noise level is boosted to a measurable level such that a change can be seen with the whitening enabled/disabled.
   • Compare the ASD of the signal between 30-100 Hz with the whitening engaged/disengaged.
   • Example result shown in Attachment #3.I believe the whitening is 15:150 (z:p) 

Tomorrow:

1. Recover POX/POY locking,.
2. ...

While working on recovering interferometer alignment, I noticed that the ETMX Oplev SUM channel reported 0 counts. Attachment #1 shows the 200 day trend - despite the missing data, the accelerating downward decay is evident. I confirmed that there is no light coming out of the HeNe by walking down to EX. The label on the HeNe says it was installed in March 2017, so the lifetime was ~30 months. Seems a little short? I may replace this later today.

I was hoping that the dark / electronics noise level on the LSC photodiodes would be sufficient for me to test the whitening gain switching on the iLIGO Pentek whitening boards. However, this does not seem to be the case. I guess to be thorough, we have to do this kind of test. It's a bit annoying to have to undo and redo the SMA connections, but I can't think of any obvious easier way to test this functionality. More annoyingly, the sensing matrix infrastructure necessary to do the kind of test described in the linked elog is only available for some PDs. I don't really want to modify the c1cal model and go through another mass reboot cycle.

While I was at it, I was also thinking about the tests we want to do. Here is a quick first pass - if you can think of other tests we ought to do, please add them to the list!

1. Whitening gain switching on the D990694 boards.
   • Need to inject some signal to do this in a clean way. 
   • With some signal injected, we need to switch the whitening gain through the 15 available levels and confirm that we see a +3dB gain for each step.
   • An example script to do this operation and make a diagnostic plot is at /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py.
2. AA enable/disable on the D000076 boards. Do we really need this functionality? Can't we permanently enable the AA, as was done for WF2?
   • Need to measure the TF with an SR785 or drive a high-freq line and confirm that the aliased peak height is attenuated as expected in DTT.
3. LO Det Mon channel check
   • Zeroth level test can be done by turning Marconi OFF/ON, and confirming we see a change in the corresponding monitor channel, like I did here.
   • A more rigorous diagnostic would require these channels to be calibrated to dBm of LO power.
4. PD INTF board check
   • Zeroth level check can be done by shining light onto PDs one at a time and confirming that the correct channel shows a response.
   • A more rigorous diagnostic would require these channels to be calibrated to mW of optical power incident on the PDs.
5. QPD INTF board check
   • This is the IP-POS QPD readback.
   • Need to confirm the quadrant mapping, and that Pitch is really Pitch, Yaw is really Yaw.
   • A more rigorous diagnostic would require these channels to be calibrated to mm of position shift.
6. CM Board
   • Need to determine what tests need to be done.
   • I have not yet implemented the fix for the MBBO gain channels for all the gains - only REFL1_GAIN is set up correctly now. Need to look at the hardware for the correct addressing of bits.
7. ALS INTF board
   • This board isn't actually connected yet, pending strain relief of cabling at 1Y2.
   • The calibration of the board output volts to dBm is known, so we can easily check this functionality.

I tried to lock the Y arm cavity length to the PSL frequency using POY11_I as an error signal. Even though I think the cavity alignment is good (I see TRY flashes ~0.8), I am unable to achieve a lock. I checked the signal conditioning, and as far as I can tell, all the settings are correct, but there may be some settings that have not been re-assigned correct values. The other possibility is that something is not quite right with the new c1iscaux. The PDH error signal and arm cavity flashes all seem good though (see Attachment #1), so I'm not sure what obvious thing I'm missing.

To be continued...

• I'd say permanently enable AA and AI. There's no reason to turn these off for usual channels. We can always undo one switch later if we want to use aliasing to sample a high frequency signal (ala SoCal).
• The PD output should ~20 nV/rHz into the mixer, so that's ~7 nV into the whitening filter. We need 60 dB to be above the ADC noise.
• I've forgotten what the current config is, but in iLIGO we hacked in a fixed whtiening on the Lt1128 input amp to the WF board so that the lock acquisition could be a little easier (better SNR). On Ch1, that's replacing R60 with a RC network. We want to make sure that the lock acq transients are not saturating the ADC, but can maybe put in a 40:200 stage.

== Test Status ==
[none] Whitening gain switching test
[none] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

- LO Det Mon channel check

The StripTool template for the test was made:
/cvs/cds/caltech/target/c1iscaux/testScripts/testDetectMons.str
Then, the RF output of the main Marconi was toggled a few times. -> Confirmed the channels are respopnding. (Attachment 1)

- IPPOS channel check

(0th order check) The StripTool template for the test was made:
/cvs/cds/caltech/target/c1iscaux/testScripts/testIPPOS.str
Then, the IPPOS QPD was shined with a phone LED. Initially I saw no response of the QPD. It turned out that the IPPOS IF module had no input cable connected. After the connection, all the 4 segments are responding to the phone LED and also the IFO beam.

(more careful check)
I decided to do more careful check of IPPOS. As there was a f~30mm lens on the oplev table, beam was focused such that only one element reacted to the incident beam. The beam power (a few mW) was too strong for a single QPD element, which saturates at ~6, an ND filter of OD0.6 was used to reduce the incident power.

Here are the results:
SEG1 (UPPER LEFT seen from the beam) | C1:ASC-IP_POS_QPD_Seg1_Mon 3.651+/-0.003 (N=10) | Incident Power 2.35+/-0.01 mW, QPD X_Calc (+) Y_Calc (+)

After the measurement, the lens and the filter were removed and the beam was adjusted to the center of the QPD.

Yesterday (Sep 25) evening: I had to reboot c1psl, c1iool0, and c1aux to recover nominal IMC locking

Today megatron had no response and I had to reboot it with the reset button. MCautolocker and FSSSlow were recovered and the IMC is locking as usual.

== Test Status ==

[done] Whitening gain switching test => Some issues found (POP110Q, Whitening3_8 not switching, ASDC overall behavior, REFL33Q needs recheck)
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

And, the Y-arm lock was recovered! After some alignment work, the Y-arm was locked. The whitening gain for POY11 was +18dB. The servo gain was 0.015 (nominal).
Once the transmission reached 0.8, I could use ASS to align the cavity and the TTs.
The transmission reached just 1.00 at the end. Was the transmission recently normalized? (See attachment 5)

- Whitening Filter Gain Switching Test

Each whitening filters were tested individually. +50mV DC signal was connected to the 8 inputs using an SMA octopus cable.
The existing script ( /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py ) did not work because cds.getdata failed to fetch all of the data requested. By giving some sleep before start downloading the data, the problem was avoided. Still some truncated data are seen in the result, but StripTools screenshots compliments the missing part.

Whitening Filters #2~4 were a little tricky because the code needed modification so that the spare channels can be tested.
The modified script is stored as /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain_190926.py 

Whitening #1: No issue found.

Whitening #2: No issue found. Some of the step plots showed truncation of the data at the end. But this is an artifact of cds.getdat. The striptool show nothing irregular.

Whitening #3: POP110Q and the spare channel (CH8) did not show the reaction. REFL33Q showed some systematic gain deviation. It could just be the offset problem, but needs to be rechecked.

Whitening #4: The DC channels were found to be OK  except for ASDC. ASDC shows earlier saturation. The input was lowered to 5mVDC to avoid saturation in the second trial. The circuit needs to be checked. The spare channels look noisy, but this is because there is no way to turn off the whitening filters for them.

- AA Filter Test

Injected 11kHz 1Vpp sine wave to the whitening filters. The whiter gains were kept at 0dB. If the AA is disabled, the alias of the 11kHz signal appears in the time series.
-> Whitening #1, #3 and #4: the enable/disable worked correctly.
-> Whitening #2 AA Bbypass no effect. this is an expected behavior.
 

I reset the normalization for both arms on Jul 9 2019.

Summary:

There is no visible PDH error signal on the POX11 channels. As a result, I am unable to lock the XARM length to the laser frequency. See Attachment #1 - the Y arm length is locked to the PSL frequency, and control is disabled for the XARM servo.

Details:

Now that several of the c1iscaux functionality tests have been completed, I wanted to push ahead with some locking. However, I was foiled at this early stage, for reasons as yet unknown. One possibility is that the

• I am able to see TRX cavity flashes >0.8, which suggest to me that the cavity is well aligned.
• Moreover, I am able to lock some (admittedly high TEM order) mode of the green laser, which further supports the above hypothesis.
• However, there are no visible PDH-like features in the POX11_I or POX11_Q channels.
• I checked that the cables from the output of the POX11 demod board are in fact going to the correct channels on WF1 (#5 and #6 respectively), and that the whitening gain for this channel is set to the nominal +30 dB.
• Next, I went to the POX table and looked for the POX IR beam. I couldn't see anything, but this beam is expected to be weak (of the order of 1 W * T_PRM * R_AR_ITM ~ 30 uW), which is probably not so easily visible.

Next steps:

• Look for the POX beam with an IR viewer.
• Confirm that everything is order on the LSC Demod board for POX 11 - maybe the LO isn't connected (somehow)?

To facilitate POX locking investigations, I replaced this HeNe today with one of the spares Chub/Steve had acquired some time ago. Details:

• Part number: Lumentum 22037130 (1103P)
• Serial number: PA00836
• Manufacture date: 01/2019
• Power output: ~2.64 mW (Measured with Ophir power meter in the 632nm setting)
• Power received on QPD: ~0.37 mW = ~18700 cts (Measured with Ophir power meter in the 632nm setting)

The RIN of the sum channel with the Oplev servo engaged, along with that for the other core FPMI optics, in shown in Attachment #1. The ETMX HeNe RIN is compatible with the other HeNes in the lab (the high-frequency behaviour of the BS Oplev is different from the other four because the QPD whitening electronics are different).

Not sure what to make of the ETMY RIN profile being so different from the others, seems like some kind of glitchy behaviour, I could see the mean level of the ASD moving up and down as I was taking the averages in DTT. Needs further investigation.

The old / broken HeNe is placed i(nside the packaging of the abovementioned replacement HeNe) on Steve's old desk for disposal in the proper way.

*It looked like Steve had hooked up a thermocouple to be able to monitor the temperature of the HeNe head. I removed this feature as I figured if we don't have this hooked up to the DAQ, it isn't a really useful diagnostic. If we want, we can restore this in a more useful way.

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I modified /cvs/cds/caltech/target/c1iscaux/latch.py and /cvs/cds/caltech/target/c1iscaux/C1_ISC-AUX_CM.db to set up the mbbo logic for the other three channels on the CM board, namely REFL2 Gain, AO Gain, and the Super boosts. The systemctl processes were restarted on c1iscaux. We are now ready to perform systematic checks on the CM board functionality.

Remarks:

The addressing of the Acromag BIO registers is done in a way that is kind of inconvenient to use the EPICS mbboDirect protocol

• The control word going to the Acromag is 16 bits in length
• However, only the 4 least significant bits actually correspond to physical channels - the remaining 12 bits are "unused".
• Because each Acromag BIO unit has 16 BIO channels, this means that they are grouped into four "banks" of 4 bits each.
• The mbboDirect EPICS/modbus protocol is used to control multiple physical BIO channels using a single input, which is exactly what we want for the gain sliders on the CM board. However, one caveat is that the bits need to be consecutive.
• This means that we have to break up the 6 bits used for the gain sliders (and in fact also the 2 bits used for the super boosts) into a least-significant-bits (LSB) group and a most-significant-bits (MSB) group.
• What's more annoying is that our physical wiring scheme means that we can't uniformly decide on how this division into LSBs and MSBs work for all the channels - e.g. for REFL1 Gain, the LSB is the 4 least significant bits, while the MSB is the 2 most significant ones, while for REFL2 Gain, the roles are reversed.
• In hindsight, the "clever" way to do the wiring assignment would have been to factor this in - but the problem is (sort of) easily fixed in software, and so I recommend we stick with the existing wiring scheme.

I tested the new latch.py script by toggling the various sliders (one at a time) between two values and monitoring the states of the various soft and "*_BITS" channels, see Attachment #1. The behavior seems consistent to me, but to be sure, we have to use Koji's LED tester board and confirm that the physical bits are being toggled correctly. The StripTool templates live in /cvs/cds/caltech/target/c1iscaux/CMdiag.

I booted Rossa in rescue mode; though I see no errors on bootup, I still see the same error ("a problem has occurred") after boot, and a prompt to logout. I powered rossa off/on (single short press of power button), no change.

Booting in debug mode, I see that the error occurs when mounting /cvs/cds, with the error

Which is odd, because when I boot in recovery mode, is mounts /cvs/cds successfully. 

I booted in emergency mode by adding to the boot command

systemd.unit=emergency.target

but didn't have the appropriate root password to troubleshoot further (the usual two didn't work).

I wanted to measure the RF transimpedance of the WFS heads, as outlined above.

Summary: Measurement is not done.

Details:

• closed the PSL shutter
• taped over the WFS 2 opening with frosted scotch tape
• illuminated the QPD with an incandescent flashlight.
  • All of the D batteries were close to dead, so it seemed dimmer than usual
• Observed the WFS2 segment 1 RF spectrum on the Agilent, but saw no difference between the spectrum with and without the flashlight. Must need a brighter light, and possibly also better alignment.
• Needed to skype someone and pass off the IFO to gautam, so I untaped the QPD, returned the appropriate LEMO connector, and opened the PSL shutter.

I confirmed that there is light incident on the POX photodiode. So the problem must lie downstream in the demod / whitening / AA electronics. With the PRM aligned (i.e. PRFPMI config with all DoFs uncontrolled), I could see the flashing beam on an IR card. I could also see the spikes in DC power incident on the photodiode using the "DC Monitor" port on the photodiode head and an oscilloscope.

Update 245 pm: I confirmed that I could see a 11 MHz sine wave by connecting the POX11 RFPD output cable at the 1Y2 end to an oscilloscope. The amplitude of this signal was also changing, corresponding to the cavity fringing in and out of resonance. I couldn't, however, see any signal on the RFPDmon port, or the I/Q demodulated output ports. So as of now, the culprit seems to be something on the Demod board. Further investigations underway...

Update 315pm: I did the following checks:

1. Checked the LO signal level into a 50ohm input scope - it was ~720 mVpp, which was compatible with the LO level into the POY Demod board, so the LO signal level couldn't be to blame.
2. Connected an RF funcgen to the PD input of the demod board. Drove it at 11.066210 MHz, 50 mVpp, and saw a signal 400 cts-pp in the CDS system - so the demod + digitizaiton electronics also seemed fine.
3. #2, coupled with the fact I could see no signal at the RF-mon port of the demod board (even though there was a signal visible at the cable coming to 1Y2) suggested that the cable routing the POX11 PD output from the Heliax-breakout in 1Y2 to the demod board was busted - indeed this was the case!
4. Koji replaced the cable without changing its length, and now the XARM locks readily 👏 . I ran ASS and got TRX ~ 0.95. See Attachment #1. 

I did the following:

1. symlinked /cvs/cds/rtcds to /opt/rtcds.
2. Added a line to /etc/systemd/system/modbusIOC.service that executes a burt-restore of the latest c1iscaux.snap file so that whitening gains etc are restored to their last saved value in the event of a service restart.

I made some model changes to c1lsc. To propagate the changes, I tried the usual rtcds make sequence. But I got an error about the model file not being in the path. This is down to my re-organization of the paths to cleanly get everything under git version control. So I had to run the following path modification. Where is this variable set and how can I add the new paths to it? The model compilation, installation and restart all went smooth after I made this change. 

For smooth reboot of the models, I used the reboot script. I had to restart the daqd processes on FB, but now all the CDS indicator lights are green.

Attachment #1 shows a first look at the IR ALS noise after my re-coupling of the IR light into the fiber at EY. 

Measurement configuration: 

• Each arm length was individually stabilized to the PSL frequency using POX/POY locking.
• The respective AUX laser frequencies were locked to the arm cavity length using the AUX PDH loops.
• GTRX ~0.3 (usually I can get ~0.5) and GTRY ~ 0.2 (the mode-matching to the arm cavities is pretty horrible as suggested by the multitude of bullseye modes seen when toggling the shutter).
• The control signal to the AUX PZT had the DC part offloaded by the slow temperature control servos to the AUX laser crystal temperature.

CDS model changes:

• The c1lsc model was modified to route the input signals to the Y phase tracker servo from ADC1_2 and ADC1_3 (originally, they were ADC0_20 and ADC0_21).
• This change was necessary because the DFD output is sent differentially to the ADC1 card in the c1lsc expansion chassis (bypassing the iLIGO whitening and AA electronics, for now just going through an aLIGO AA board with no whitening available yet).
• I chose to use the differential receiving (as opposed to using the front-panel single ended BNC connectors) as in principle, it is capable of delivering better noise performance.
• After making the model changes, I compiled and restarted the model. Apart from the missing path issue, the compile/restart went smoothly.

Next steps:

• Get the easy fixes done (better GTRX, GTRY).
• Test the noise with POX and POY as the OOL sensors, and the arms controlled using the ALS error signal - this is the relevant metric for how ALS will be used in locking.
• Noise budget. Need to double-check the DFD output calibration into Hz.
• For the general interferometer recovery, I think I will push ahead with trying to lock some other configurations like the PRMI (should be easy to recover), DRMI (potentially more difficult to find the right settings), and the FPMI (I'd like to use this config to get an estimate for how much contrast defect we have in the interferometer, but I think it'll be pretty challenging to lock in this configuration).

1. With TRX and TRY maximized using ASS, I centered the Oplev spots on the respective QPDs for the four test masses and the BS. I also centered the spot onto the IPPOS QPD by moving the available steering mirror.
2. At EX, I tweaked the input pointing of the green beam into the arm by manually twiddling with the PZT mirrors. I was able to get GTRX~0.4.
3. On the AS table - Koji and I found that there was a steering mirror placed in the AS beam path such that there was no light reaching the AS110 or AS55 PDs. Please - when you are done with your measurement, return the optical configuration to the state it was in before so that the usual locking activity isn't disturbed by a needless few hours troubleshooting electronics.

Once Koji is done with his checkout of the whitening electronics, I will try and lock the PRMI.

Summary:

The PRMI was locked with the carrier field resonant in the PRC 🙌. The lock is pretty stable (I only let it stay locked for ~10mins and then deliberately unlocked to see if I could readily re-lock, but it has stayed locked for the last ~20mins while I typed this up). See Attachment #1 for the DC power monitor StripTool for a short section of lock.

Details:

• This is the opposite of the config we'd want usually for locking the IFO, but it is a useful configuration for setting the alignment of the vertex optics, and also to train angular feedforward filters, so I decided to try it out.
• Some patient alignment work was required. I started with the single arm locks, maximized TRX/TRY with ASS, and then misaligned the ETMs and brought the PRM into alignment.
• The PRM Oplev spot was roughly centerd on its QPD once I judged I was getting decent PRMI cavity flashes on the POP camera. The PRMI Oplev servo needs some tuning, it is currently susceptible to oscillations in Pitch.
• The error signals used were: REFL11_I ---> PRCL and AS55_Q ---> MICH.
• The whitening gains were: REFL11 --> +18 dB, AS55 ---> +6 dB.
• Triggering was done using POPDC, this worked better for me than any of the RF signals (e.g. POP22/POP110). Trigger ON --> 200cts, Trigger OFF --> 100 cts.
• The DCPD whitening gains may not be set correctly - I think I remember POPDC being ~4000 cts in this configuration, but it may also be that we are not well centered on the POP photodiode.
• The dominant cause of the POP circulating power seems to be the usual angular instability ascribed to the TTs. The OAF model wasn't running tonight (and I didn't want to try starting it and have to do a full vertex FE reboot tonight) so I didn't get a chance to engage the angular FF.

Next (for LSC activities):

• PRMI locking with the sidebands resonant in the PRC.
• DRMI locking

I'm leaving the LSC mode off for tonight, but with the PRMI optics aligned and ETMs misaligned.

I worked on more troubleshooting of the whitening filters Tuesday afternoon

== Test Status ==

[done] Whitening gain switching test => Remaining issues ASDC overall behavior
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

Issue 1: POP110Q did not show any gain switching [Resolved]

A DB37 breakout board was connected to the acromag front panel. I found that Ch6 (POP110Q) did not show any differential DC output. I searched around the other pins and found that the corresponding signal showed up on PIn36  instead of Pin35. Opening the front panel revealed that the internal wiring was wrong (Attachment 1). The wire which should have gone to Pin 35 was connected to Pin 36. By correcting the wiring, POP110Q started to show identical behavior to POP110I. (Attachment 2)

Issue 2: LSC reboot [Resolved]

A rough activity around the acromag chassis crashed c1lsc realtime processes (as usual). I ran usual rebooting script /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh. This successfully restored the status of the vertex RT processes.

Issue 3: REFL33 different behavior between I and Q [Resolved]

REFL33I and Q consistently showed a difference (Attachment 3). The whitening board was pulled out and powered with an extension card. The raw outputs were checked with a function generator and an oscilloscope connected. The outputs for 33I and Q were identical (Attachment 4). So I concluded that the observed difference was an artifact of the checking script.

Issue 4: Whitening 3_8 did not switch at all [Resolved]

To switch the gain stages, each channel of the whitening board takes a DAC output from acromag and convert it into 4bit digital signals. For CH8 of the WF#3, this signal did not reach the instrumentation amplifier AD620. After tracing the signal on the electronics bench, it was found that the CH8 gain input to the DIN96 connector is not conducive to the input of the AD620. As there were no exposed pads between the DIN96 connector and the AD620 input (pin2), a wire was additionally soldered (forgot to take a photo). This solved the gain switching issue as the test result indicates (Attachment 5). The noisiness came from the whitening filter which can not be turned off right now. For this reason, the test of the whitening part is pending too.

The StripTool plot during the overall WF#3 test is shown in Attachment 6.

Issue 5: ASDC behavior [Unresolved]

First of all, at this test, I found that WF#4 was not responding to the gain change at all. This issue was restored by power cycling the acromag chassis (as usual).

The whitening filter #4 was pulled, and the behavior of CH5,6,7,8 (CH8=ASDC) was compared. It was found that the analog outputs were identical and the problem lies further downstream.

Issue 6: Illeagal REFL11 LO cable [Unresolved]

This is a newly found issue. The cable between the LO distributor and the REFL11 demodulator is not the legit solder soaked RG402 coax, but flexible coax (Attachment 7). This cable needs to be replaced in the end. But for today, it was not so that we can have a consistent configuratin as before.

Issue 7: Signature of a damaged POPDC cable [Resolved]

The cable for POPDC cale indicated some damage at the WF#4 side. It was not a complete damage, and therefore the solder coating was added (Attachment 8).

This morning, I restarted the c1oaf model on the c1lsc machine, so as to have the option of enabling some feedforward action. Unsurprisingly, the "DC" indicator is red, citing a "0x2bad". In the past, I've been able to correct this by simply restarting the model. But given the fragility of the c1lsc machine, I think I'll live with not having the OAF model signals in frames. Medium-term, I'd like to pare down the c1oaf model a bit - I think it has way too many options/matrices right now, and is an un-necessarily bloated and heavy model. Unless there are serious objections, I will do this work when I next feel like it.

The anaconda distribution used by the control room workstations is actually installed on the shared drive (/cvs/cds/ligo/apps/anaconda/) for consistency reasons. The version was 4.5.11. I ran the following commands to update it today. Now it is version 4.7.12.

The second command takes a while to resolve conflicts, so I've left it running inside a tmux session for now.

Recall that the bash alias for using the anaconda managed python is "apython". I recommend everyone set up a virtual environment when trying out new package installs, to avoid destroying the locking scripts.

I measured the OLTF of both the PRM Oplev loops. Nothing odd sticks out as odd to me in this measurement - there seems to be ~40 degrees of phase margin and >10 dB gain margin for both loops, see Attachment #1. I didn't measure down to the second UGF at ~0.2 Hz (the Oplev loops are AC coupled), so there could be something funky going on there. The problem still persists - if I misalign and realign the PRM using the ifoalign scripts, the automatic engagement of Oplev loops causes the loop to oscillate. Could be that the script doesn't wait for long enough for the alignment transient to die out.

Update 1230pm: Indeed, this was due to the integrator transient. It dies away after a couple of seconds.