Saw these slightly delayed.

Q1: Not sure--is it a safe operation for me to remove the DIMM on CPU0, replace CPU0 (with no DIMM), and boot up to try this?

Q2: Specifically, it's this DIMM. The CPU core is compatible with DDR2, clock rate up to 333 MHz (DDR2-667) and 1, 2, or 4 GB of memory.

One pair of DIMM cards from the Sunstone box had the same Sun part number as those in c1ioo, so I swapped them in and reinstalled c1ioo's CPU0. c1ioo now boots up an seems ready to go, I'm able to log on from nodus. I also reinstalled optimus' CPU0, and optimus boots up with no problems.

• old C1OMC RT
  • I found the old c1omc RT machine (a Sun Systems v40z) under y arm, but when I followed the instructions for replacing its CPU card, I found no CPU card installed.
  • I then found how to replace the memory module on the motherboard,
• Megatron
  • I also found that megatron will require a CPU filler board if we remove one of its DIMM (it cannot operate with empty CPU module slots)
• optimus
  • Rana says I can also consider using two of optimus' DIMM cards. Optimus appears to not be running any scripts currently, and I don't find any recent elog entries or wiki pages mentioning optimus with critical use.
  • I shutdown optimus (from the command line Mon Sep 9 13:17:58 2019).

While opening up optimus, I noticed a box labelled 'SUNSTONE' sitting below the rack--it contains two CPU modules a similar type as in c1ioo! I'm going to try swapping in the DIMM cards from this SUNSTONE box; I didn't find any elogs about sunstone--where are these modules from?

I reset c1lsc and c1sus, then ran rebootC1LSC.sh as before. All models started by the script are running with minimal red lights; c1oaf, c1cal, c1dnn, c1daf, and c1omc are not started by the script. I manually started these in the order c1cal->c1oaf->c1daf->c1dnn. Starting c1dnn crashed the other FE on c1ioo, so I reset all three FE again, and ran the script again (this time, including the startup for c1cal, c1oaf, and c1daf, but excluding c1dnn).

Except for c1dnn and c1omc, all models are started. The status lights are attached.

I picked up a unit of D1900068 SR785 accessory box from Dean's office at Downs. 

The PCB board of the adapter for DIN 96pin to DSUB37 conversion (single DSUB version) was delivered yesterday and I quickly soldered the connectors.

They are ready for use and stored in a JLCPCB cardboard box on a pile of acromag stuff. (Note that the lacel is written on the box with Sharpie)

I installed 6 of these in 1Y2. Three were for PD INTF #1-3, and I used three more for the AS110, REFL11, and REFL33 Demod board FEs, where the strain-reflief of the DC power cables to the Eurocrate was becoming a problem. So now there are only 4 units available as spares.

Once the strain-relieving of the Dsub cabling to 1Y3 is done, we can move ahead with testing. I'd like to put this to bed this week if possible.

1. Jordan (new Engineer) and Chub neatened out the cabling at 1Y2/1Y3 today. After their work, I plugged in all the Dsubs to the rear Eurocrate DB37->DIN96 adaptors. Jordan nicely fixed up the labels on the cable with some extra sellotape for a more durable label.
2. As part of the war on cross-connects, Chub removed some cables that were piping BIO signals from the fast CDS system to the whitening boards.
   • There is a SCSI to DB37 custom ribbon cable going from the BIO card in the expansion chassis to a 1U chassis box at the very bottom of 1Y2.
   • This 1U box, with DCC number D080478 (but no schematic exists on the DCC or any of the usual secret hidey-holes) breaks out the 32 BIO channels to 16+16.
   • Each set of 16 channels was supposed to get broken out to 8+8 via some cross connects and then goto the whitening boards. This is the part that got distrubed.
   • Koji and I discussed options - if Chub cannot resotre this easily, we will make a D37--> 4*D15 breakout board, and pipe the signals via the backplane P2 connectors. This will mean ~10 more days before the LSC system can be tested.
   • Some cabling to the TT DACs and an ADC were also disturbed, but these are easily restored.
3. From the hardware standpoint, some cross-struts for strain relief on the back of 1Y2 need to be installed --> Chub.

For some reason, the daqd_fw service was dead on FB. This meant that no frames were being written since Aug 23, which probably coincides with when the c1lsc frontend crashed. Sad 😢 😭 🙁 . Simply restarting the fw service does not work, it crashes again after ~20 seconds. The problem may have to do with the indeterminate state of the c1lsc expansion chassis. However, this is not something that can immediately be fixed, as Chub is still working on the wiring there. So in summary, no frame data will be available until we fix this problem (it is still unclear what exactly the problem is). Team WFS can still work by getting online data.

Why were the CDS overview DC indicators not red???

Unrelated to this work: I had to key the c1psl crate to get the IMC autolocker functioning again. However, I found that the key 🔑 turns continuously - as opposed to having two well defined states, ON and OFF. Be careful while handling this.

Came across this while looking up the BIO situation at 1Y2. For reference, the fix Koji mentions can be seen in the attached screenshot (one example, the other BIO cards also have a similar fix). The 16th bit of the BIO is grounded, and some bit-shifting magic is used to implement the desired output.

Summary:

1. Frames seem to be written again.Slowly but surely, we are converging to an operable state...
2. No frames are available for the period 23 Aug to 17 September 2019
3. Don't edit the C0EDCU.ini file unless you know what you're doing.
4. If you make some changes to the RT system/channel list or reboot FEs, please make sure all the dependent systems are back up and running. There shouldn't be a need to willy-nilly reboot things.
5. Tomorrow I will prepare the map of BIO channels for Chub to restore the whitening switching capability. Then we can try locking some cavities.

Details:

1. First, I checked to make sure the /frames partition wasn't full. It wasn't.
2. Next, I looked into the C0EDCU.ini file.
   • The last date for which frames are available, 23 Aug, coincided with the date when this file was modified.
   • It is a known problem that the daqd_fw service can crash if one of the channels in this file is reporting an unusually large number.
   • Several channels were added to this file - in the end, only 9 new ones were required, 5x "DetectMon" channels for each of the RF demodulation frequencies, and 4 for the new ALS LO and RF signal power monitor channels.
   • It is highly likely that one of the other channels was what caused the daqd_fw service to crash - though I can't say for sure, because I did not exhaustively search through the ~100 un-necessary channels that were in this file to see what values they were reporting.
3. For good measure, I ran the reboot script, and brought the c1lsc models back online.
   • I want to do the mapping of the BIO channels to the pin-out of the BIO adaptor unit, which requires c1lsc to run.
   • Reboot script ran smoothly.
4. Then I went into fb and restarted all the daqd services. This time, they all seem to run without crashing, at least in the ~10min window it took me to type out this elog.

Unrelated to this work: c1auxey was keyed.

For the investigation of the latch logic issue for the CARM CM board, I have made the LED logic checkers with DB breakout boards. They require the pull up voltage supply of +15V because the acromag digital out is a open corrector (well... open "source") output.

The logic from Pin1 to Pin16 of DB37 can be monitored. The DB15 connector is only for monitoring the latch enable logic.

What Gautam and I found with the logic outputs was that the latch logic works fine but occasionally we found that the top 2 bits and the bottom 4bit were processed independently.

[Koji Gautam]

We continued to check the latch logic. Today we found that latch.py didn't catch the change of LSB but did for MSB. We determined that this happens when the slider value is chaged between the polling for LSB and MSB.
SInce these two should always be related to a single gain value, latch.py was modified so.

Now we don't observe any logic error for ~100 gain transisitions (see attached).

INCORRECT INFO IN THIS ELOG HAS BEEN REMOVED. SEE THIS ELOG FOR THE UPDATED INFO.

Summary:

With the help of a tester board, I verified the mapping between fast BIO DB37 pins, and pins on the IDC50 connectors that are to be broken out to the whitening boards. I will enlist Chub to implement this mapping in hardware later today.

Details:

1. The LSC PD demodulated signals are optionally whitened before acquisition by our RTCDS ADCs.
2. The switching of each channel's whitening (enable/disable) is done by a single bit from the fast (a.k.a. RTCDS) system's BIO cards.
3. The whitening boards live inside Eurocrates.
4. The aforementioned switching signal needs to be sent to the whitening boards via the backplane of the Eurocrate.
5. This requires some cross-connect based cable splicing between the BIO card outputs and the P2 connectors of the whitening boards in the Eurocrates.
6. This connection was accidentally destroyed during the war on cross-connects at 1Y2. I couldn't find a wiring diagram anywhere.
7. Today, with the help of a tester board, I verified the mapping by toggling the appropriate channels on the MEDM screen, and verifying the correct LEDs on the tester board were toggled.
8. Map will be posted here after the meeting... Also now on the wiki.

Update 2019 Sep 19 1730: The pin numbers of the IDC 50 connector are all off by 1. i.e. 3-->4 and so on. I will fix this shortly. The problem was because of me looking at the pinout for the wrong gender of IDC50 connectors.

New DIMM cards have arrived. I stored them in the digital cabinet along y arm.

[KA, GV]

Summary:

1. New cross connect system for splicing the fast BIO signals for whitening switching to the P2 connectors was installed and tested at 1Y2.
2. It passed a first round of tests. 😁 
3. As of now, I believe all the necessary electrical connections have been made at 1Y2/1Y3, and we are ready for testing the c1iscaux system.

Details:

1. We did some testing in the office area, and found several wiring mistakes. These were all rectified. Attachment #1 is an accurate reflection of the implemented wiring scheme (softcopy in the 40m google sheets area). Be aware that the IDC 50 pin connector pin-out is tricky, and you have to be aware of the difference between male/female connector when looking for this pin-out on the internet.
2. In order to facilitate further testing, we re-routed the ADC0 SCSI cable that was unplugged on the overhead cable tray, and plugged it back into the c1lsc expansion chassis. This action necessitated a reboot of the vertex FEs, but everything came back alright.
3. Did some general neatenign and strain relieving. Removed a few existing cross-connects to make space for our new terminal blocks.
4. Attachment #2 shows the layout of the terminal blocks. Note the unusual (vertical) order of the orange terminal blocks.
5. The final integrated CDS test done was the following:
   • Set whitening gain for channel under test to 45dB, so that the dark noise level is boosted to a measurable level such that a change can be seen with the whitening enabled/disabled.
   • Compare the ASD of the signal between 30-100 Hz with the whitening engaged/disengaged.
   • Example result shown in Attachment #3.I believe the whitening is 15:150 (z:p) 

Tomorrow:

1. Recover POX/POY locking,.
2. ...

I was hoping that the dark / electronics noise level on the LSC photodiodes would be sufficient for me to test the whitening gain switching on the iLIGO Pentek whitening boards. However, this does not seem to be the case. I guess to be thorough, we have to do this kind of test. It's a bit annoying to have to undo and redo the SMA connections, but I can't think of any obvious easier way to test this functionality. More annoyingly, the sensing matrix infrastructure necessary to do the kind of test described in the linked elog is only available for some PDs. I don't really want to modify the c1cal model and go through another mass reboot cycle.

While I was at it, I was also thinking about the tests we want to do. Here is a quick first pass - if you can think of other tests we ought to do, please add them to the list!

1. Whitening gain switching on the D990694 boards.
   • Need to inject some signal to do this in a clean way. 
   • With some signal injected, we need to switch the whitening gain through the 15 available levels and confirm that we see a +3dB gain for each step.
   • An example script to do this operation and make a diagnostic plot is at /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py.
2. AA enable/disable on the D000076 boards. Do we really need this functionality? Can't we permanently enable the AA, as was done for WF2?
   • Need to measure the TF with an SR785 or drive a high-freq line and confirm that the aliased peak height is attenuated as expected in DTT.
3. LO Det Mon channel check
   • Zeroth level test can be done by turning Marconi OFF/ON, and confirming we see a change in the corresponding monitor channel, like I did here.
   • A more rigorous diagnostic would require these channels to be calibrated to dBm of LO power.
4. PD INTF board check
   • Zeroth level check can be done by shining light onto PDs one at a time and confirming that the correct channel shows a response.
   • A more rigorous diagnostic would require these channels to be calibrated to mW of optical power incident on the PDs.
5. QPD INTF board check
   • This is the IP-POS QPD readback.
   • Need to confirm the quadrant mapping, and that Pitch is really Pitch, Yaw is really Yaw.
   • A more rigorous diagnostic would require these channels to be calibrated to mm of position shift.
6. CM Board
   • Need to determine what tests need to be done.
   • I have not yet implemented the fix for the MBBO gain channels for all the gains - only REFL1_GAIN is set up correctly now. Need to look at the hardware for the correct addressing of bits.
7. ALS INTF board
   • This board isn't actually connected yet, pending strain relief of cabling at 1Y2.
   • The calibration of the board output volts to dBm is known, so we can easily check this functionality.

• I'd say permanently enable AA and AI. There's no reason to turn these off for usual channels. We can always undo one switch later if we want to use aliasing to sample a high frequency signal (ala SoCal).
• The PD output should ~20 nV/rHz into the mixer, so that's ~7 nV into the whitening filter. We need 60 dB to be above the ADC noise.
• I've forgotten what the current config is, but in iLIGO we hacked in a fixed whtiening on the Lt1128 input amp to the WF board so that the lock acquisition could be a little easier (better SNR). On Ch1, that's replacing R60 with a RC network. We want to make sure that the lock acq transients are not saturating the ADC, but can maybe put in a 40:200 stage.

== Test Status ==
[none] Whitening gain switching test
[none] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

- LO Det Mon channel check

The StripTool template for the test was made:
/cvs/cds/caltech/target/c1iscaux/testScripts/testDetectMons.str
Then, the RF output of the main Marconi was toggled a few times. -> Confirmed the channels are respopnding. (Attachment 1)

- IPPOS channel check

(0th order check) The StripTool template for the test was made:
/cvs/cds/caltech/target/c1iscaux/testScripts/testIPPOS.str
Then, the IPPOS QPD was shined with a phone LED. Initially I saw no response of the QPD. It turned out that the IPPOS IF module had no input cable connected. After the connection, all the 4 segments are responding to the phone LED and also the IFO beam.

(more careful check)
I decided to do more careful check of IPPOS. As there was a f~30mm lens on the oplev table, beam was focused such that only one element reacted to the incident beam. The beam power (a few mW) was too strong for a single QPD element, which saturates at ~6, an ND filter of OD0.6 was used to reduce the incident power.

Here are the results:
SEG1 (UPPER LEFT seen from the beam) | C1:ASC-IP_POS_QPD_Seg1_Mon 3.651+/-0.003 (N=10) | Incident Power 2.35+/-0.01 mW, QPD X_Calc (+) Y_Calc (+)

After the measurement, the lens and the filter were removed and the beam was adjusted to the center of the QPD.

Yesterday (Sep 25) evening: I had to reboot c1psl, c1iool0, and c1aux to recover nominal IMC locking

Today megatron had no response and I had to reboot it with the reset button. MCautolocker and FSSSlow were recovered and the IMC is locking as usual.

== Test Status ==

[done] Whitening gain switching test => Some issues found (POP110Q, Whitening3_8 not switching, ASDC overall behavior, REFL33Q needs recheck)
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

And, the Y-arm lock was recovered! After some alignment work, the Y-arm was locked. The whitening gain for POY11 was +18dB. The servo gain was 0.015 (nominal).
Once the transmission reached 0.8, I could use ASS to align the cavity and the TTs.
The transmission reached just 1.00 at the end. Was the transmission recently normalized? (See attachment 5)

- Whitening Filter Gain Switching Test

Each whitening filters were tested individually. +50mV DC signal was connected to the 8 inputs using an SMA octopus cable.
The existing script ( /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py ) did not work because cds.getdata failed to fetch all of the data requested. By giving some sleep before start downloading the data, the problem was avoided. Still some truncated data are seen in the result, but StripTools screenshots compliments the missing part.

Whitening Filters #2~4 were a little tricky because the code needed modification so that the spare channels can be tested.
The modified script is stored as /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain_190926.py 

Whitening #1: No issue found.

Whitening #2: No issue found. Some of the step plots showed truncation of the data at the end. But this is an artifact of cds.getdat. The striptool show nothing irregular.

Whitening #3: POP110Q and the spare channel (CH8) did not show the reaction. REFL33Q showed some systematic gain deviation. It could just be the offset problem, but needs to be rechecked.

Whitening #4: The DC channels were found to be OK  except for ASDC. ASDC shows earlier saturation. The input was lowered to 5mVDC to avoid saturation in the second trial. The circuit needs to be checked. The spare channels look noisy, but this is because there is no way to turn off the whitening filters for them.

- AA Filter Test

Injected 11kHz 1Vpp sine wave to the whitening filters. The whiter gains were kept at 0dB. If the AA is disabled, the alias of the 11kHz signal appears in the time series.
-> Whitening #1, #3 and #4: the enable/disable worked correctly.
-> Whitening #2 AA Bbypass no effect. this is an expected behavior.
 

I reset the normalization for both arms on Jul 9 2019.

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I modified /cvs/cds/caltech/target/c1iscaux/latch.py and /cvs/cds/caltech/target/c1iscaux/C1_ISC-AUX_CM.db to set up the mbbo logic for the other three channels on the CM board, namely REFL2 Gain, AO Gain, and the Super boosts. The systemctl processes were restarted on c1iscaux. We are now ready to perform systematic checks on the CM board functionality.

Remarks:

The addressing of the Acromag BIO registers is done in a way that is kind of inconvenient to use the EPICS mbboDirect protocol

• The control word going to the Acromag is 16 bits in length
• However, only the 4 least significant bits actually correspond to physical channels - the remaining 12 bits are "unused".
• Because each Acromag BIO unit has 16 BIO channels, this means that they are grouped into four "banks" of 4 bits each.
• The mbboDirect EPICS/modbus protocol is used to control multiple physical BIO channels using a single input, which is exactly what we want for the gain sliders on the CM board. However, one caveat is that the bits need to be consecutive.
• This means that we have to break up the 6 bits used for the gain sliders (and in fact also the 2 bits used for the super boosts) into a least-significant-bits (LSB) group and a most-significant-bits (MSB) group.
• What's more annoying is that our physical wiring scheme means that we can't uniformly decide on how this division into LSBs and MSBs work for all the channels - e.g. for REFL1 Gain, the LSB is the 4 least significant bits, while the MSB is the 2 most significant ones, while for REFL2 Gain, the roles are reversed.
• In hindsight, the "clever" way to do the wiring assignment would have been to factor this in - but the problem is (sort of) easily fixed in software, and so I recommend we stick with the existing wiring scheme.

I tested the new latch.py script by toggling the various sliders (one at a time) between two values and monitoring the states of the various soft and "*_BITS" channels, see Attachment #1. The behavior seems consistent to me, but to be sure, we have to use Koji's LED tester board and confirm that the physical bits are being toggled correctly. The StripTool templates live in /cvs/cds/caltech/target/c1iscaux/CMdiag.

I did the following:

1. symlinked /cvs/cds/rtcds to /opt/rtcds.
2. Added a line to /etc/systemd/system/modbusIOC.service that executes a burt-restore of the latest c1iscaux.snap file so that whitening gains etc are restored to their last saved value in the event of a service restart.

I made some model changes to c1lsc. To propagate the changes, I tried the usual rtcds make sequence. But I got an error about the model file not being in the path. This is down to my re-organization of the paths to cleanly get everything under git version control. So I had to run the following path modification. Where is this variable set and how can I add the new paths to it? The model compilation, installation and restart all went smooth after I made this change. 

For smooth reboot of the models, I used the reboot script. I had to restart the daqd processes on FB, but now all the CDS indicator lights are green.

I worked on more troubleshooting of the whitening filters Tuesday afternoon

== Test Status ==

[done] Whitening gain switching test => Remaining issues ASDC overall behavior
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

Issue 1: POP110Q did not show any gain switching [Resolved]

A DB37 breakout board was connected to the acromag front panel. I found that Ch6 (POP110Q) did not show any differential DC output. I searched around the other pins and found that the corresponding signal showed up on PIn36  instead of Pin35. Opening the front panel revealed that the internal wiring was wrong (Attachment 1). The wire which should have gone to Pin 35 was connected to Pin 36. By correcting the wiring, POP110Q started to show identical behavior to POP110I. (Attachment 2)

Issue 2: LSC reboot [Resolved]

A rough activity around the acromag chassis crashed c1lsc realtime processes (as usual). I ran usual rebooting script /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh. This successfully restored the status of the vertex RT processes.

Issue 3: REFL33 different behavior between I and Q [Resolved]

REFL33I and Q consistently showed a difference (Attachment 3). The whitening board was pulled out and powered with an extension card. The raw outputs were checked with a function generator and an oscilloscope connected. The outputs for 33I and Q were identical (Attachment 4). So I concluded that the observed difference was an artifact of the checking script.

Issue 4: Whitening 3_8 did not switch at all [Resolved]

To switch the gain stages, each channel of the whitening board takes a DAC output from acromag and convert it into 4bit digital signals. For CH8 of the WF#3, this signal did not reach the instrumentation amplifier AD620. After tracing the signal on the electronics bench, it was found that the CH8 gain input to the DIN96 connector is not conducive to the input of the AD620. As there were no exposed pads between the DIN96 connector and the AD620 input (pin2), a wire was additionally soldered (forgot to take a photo). This solved the gain switching issue as the test result indicates (Attachment 5). The noisiness came from the whitening filter which can not be turned off right now. For this reason, the test of the whitening part is pending too.

The StripTool plot during the overall WF#3 test is shown in Attachment 6.

Issue 5: ASDC behavior [Unresolved]

First of all, at this test, I found that WF#4 was not responding to the gain change at all. This issue was restored by power cycling the acromag chassis (as usual).

The whitening filter #4 was pulled, and the behavior of CH5,6,7,8 (CH8=ASDC) was compared. It was found that the analog outputs were identical and the problem lies further downstream.

Issue 6: Illeagal REFL11 LO cable [Unresolved]

This is a newly found issue. The cable between the LO distributor and the REFL11 demodulator is not the legit solder soaked RG402 coax, but flexible coax (Attachment 7). This cable needs to be replaced in the end. But for today, it was not so that we can have a consistent configuratin as before.

Issue 7: Signature of a damaged POPDC cable [Resolved]

The cable for POPDC cale indicated some damage at the WF#4 side. It was not a complete damage, and therefore the solder coating was added (Attachment 8).

This morning, I restarted the c1oaf model on the c1lsc machine, so as to have the option of enabling some feedforward action. Unsurprisingly, the "DC" indicator is red, citing a "0x2bad". In the past, I've been able to correct this by simply restarting the model. But given the fragility of the c1lsc machine, I think I'll live with not having the OAF model signals in frames. Medium-term, I'd like to pare down the c1oaf model a bit - I think it has way too many options/matrices right now, and is an un-necessarily bloated and heavy model. Unless there are serious objections, I will do this work when I next feel like it.

The anaconda distribution used by the control room workstations is actually installed on the shared drive (/cvs/cds/ligo/apps/anaconda/) for consistency reasons. The version was 4.5.11. I ran the following commands to update it today. Now it is version 4.7.12.

The second command takes a while to resolve conflicts, so I've left it running inside a tmux session for now.

Recall that the bash alias for using the anaconda managed python is "apython". I recommend everyone set up a virtual environment when trying out new package installs, to avoid destroying the locking scripts.

Today, I found out that this type of "0x2bad" DC error is connected to the 1e+20 cts output. The solution was to bite the bullet and stop/start the c1oaf model (at the risk of crashing the vertex FEs). Today, I was lucky and the model came back online with all CDS indicators green. At which point I was able to engage length feedforward to MC2 (with some admittedly old filter). Some subtraction is happening, see Attachment #1. This was just meant to test whether the signal routing is happening - the feedforward signal goes to the "ALTPOS" input of the suspension CDS block, which AFAIK does not have a corresponding MEDM EPICS indicator. So I couldn't figure out whether the feedforward control signal was in fact making it to the suspension. On the evidence of the suppression of MCL in the 1-3 Hz band, I would conclude that it is. Useful to be able to engage these FF filters for better lockability.

The AA filter for ASDC was fixed.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

The AA filter for the 4th section of the LSC analog electronics bank (D000076) was pulled out for the test. On the workbench, questionable CH8 was checked. It tuned out that the filter amplifier module for the 8th-order elliptic filter at 7.5kHz was not properly working and exhibited unusual attenuation. This filter module (Frequency Devices Inc D68L8E-7.50kHz) was desoldered and replaced with a module from a spare board. Note that Gautam and I had tried to use this spare board instead of the current one, but it didn't give us any signal for an unknown reason. Since the desoldering required a lot of force and had a risk of damaging the PCB, a socket was made from an IC socket (see Attached 1). This change made CH8 functioning equally to the other channels do.

I took this opportunity to ckech the performance of the AA filters. For each channel, the input signal was injected from J3 using a pomona clip. The output was taken from pin 1, 5, 9, ... of J2. This is the + side of the differential output. The - side just has the equivalent performance but the signal polarity. The digital signals for the AA bypass switches were not connected. Fortunately, this was just fine as it made the anti-aliasing filters engaged.

Attachment 2 shows the transfer functions of all the channels. All the channels showed an identical response (at least visually). The transfer function for CH1 was fitted by LISO. The ZPK values are listed here:

pole 5.2860544577k 503.1473053928m
pole 5.9752193716k 1.0543411596
pole 8.9271953580k 3.5384364788
pole 8.2181747850k 3.4220607928
pole 182.1403534923k 1.1187869426 # This has almost no effect
zero 13.5305051680k 423.6130434049M
zero 15.5318357741k 747.6895990654k
zero 23.1746351749k 1.5412966100M

factor 989.1003181564m
delay 24.4846075283n

Attachment 3 shows the ASD of the output voltage noise measurement. Note the input was shorted for this measurement. The nominal output voltage was found to be 0.1 uV/rtHz and the 1/f noise corner freq was about 100Hz. Only CH3 showed a deviation from the typical values. It looks like this is neither an artifact nor transient noise. Fortunately, nothing is connected to this channel right now.

[KA, GV]

This elog is meant to be a summary of some of the many subtleties on the CM board. The latest schematic of the version used at the 40m can be found at D1500308 .

Latch logic:

• There are several Binary Outputs and one Binary Input to the CM board.
• The outputs control ENABLE/DISABLE switches and gains of amplifier stages, while the input reports whenever the limiter has been reached.
• The variable gain feature is implemented by enabling/bypassing several cascaded fixed gain stages. So in order to change the gain of a single composite amplifier stage, multiple individual amplifier stages have to be switched.
• This is implemented by the user interacting with the hardware via a "control word", consisting of a number of bits depending on the number of cascaded stages that have to be switched. 
• This control word is sent to the device via modbus EPICS, which is an asynchronous communication protocol. Hence, it may be that the individual bits composing the control word get switched asynchronously. This would be disastrous, as there can be transient glitches in the gain of the stage being controlled. 
• To protect against such problems, there is a latch IC in the hardware between the Binary Inputs to the board (= Binary Outputs from Acromags), and the actual switches (= MAX333) that enable/bypass the cascaded gain stages. The latch IC used is a SN74ALS573. This device acts as a bus, which transmits/blocks changes for multiple bits (= our control word) from propagating, depending on the state of a single bit (= the LATCH ENABLE bit). Thus, by controlling a single bit, we can guarantee that multiple bits get switched synchronously. 
• In order to use this latch capability, we need some software logic that sets/disables the LATCH ENABLE bit. For our system, this logic is implemented in the form of a continuously running python 🐍 script, located at /cvs/cds/caltech/target/c1iscaux/latch.py. It is implemented as a systemctl service on the c1iscaux Supermicro. The logic implemented in this script is shown in Attachment #1. While the channels referred to in that attachment are for REFL1_GAIN, the same logic is implemented for REFL2_GAIN, AO_GAIN, and the SuperBoosts.
• Some FAQ:
  1. Q: Why do we need the soft channels C1:LSC-REFL1_SET_LSB and C1:LSC-REFL1_SET_MSB?
     A: These soft channels are what is physically linked to the Acromag Binary Outputs. In order for our latch logic to be effective, we need to detect when the user asks for a change, and then disable the LATCH ENABLE bit (which is on by default, see FAQ #3) before changing the physical acromag channels. The soft channels form the protective layer between the user and the hardware, allowing latch.py to function.
  2. Q: Why is there an "_MSB" and "_LSB" soft channel? 
     A: This has to do with the mbboDirect EPICS channel type, which is used to control the multiple bits in our control word using a single input (= an MEDM gain slider). The mbboDirect data-type requires the bits it controls to have consecutive hardware addresses. However, the Acromag hardware addressing scheme is not always compatible with this requirement (see pg 33 of the manual for why this is the case). Hence, we have to artifically break up the control word into two separate control words compatible with the Acromag addressing scheme. This functionality is implemented in latch.py.
  3. Q: Why is the default state of LATCH ENABLE set to ON? 
     A: This has to do with the fact that all Binary Inputs, not just the multi-bit ones, to the CM board are propagated to the control hardware via a latch IC. For the single-bit channels, there is no requirement that the switching be synchronous. Hence, rather than setting up ~10 more single-bit soft channels and detecting changes before propagating them, we decided to leave the LATCH ENABLE ON by default, and only disable it when changing the multi-bit gain channels. This is the same way the logic was implemented in the VME state code, and we think that there are no logic reasons why it would fail. But if someone comes up with something, we can change the logic.

Acromag BIO testing:

During my bench testing of the Acromag chassis, I had not yet figured out mbboDirect and the latch logic, so I did not fully verify the channel mapping (= wiring inside the Acromag box), and whether the sitching behavior was consistent with what we expect. Koji and I verified (using the LED tester breakout board) that all the channels have the expected behavior 👏. Note that this is only a certification at the front-panel DB37 connectors of the Acromag chassis  testing of the integrated electronics chain including the CM board is in progress...

[Gautam, Koji]

Input gain part of the CM servo board D1500308 was tested. A couple of problems were detected. One still remains.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[in progress] CM Board
[none] ALS I/F board

We started to test the CM Servo board from the input stages. Initially, DC offsets were provided to IN1 and IN2 to check the gain on the oscilloscope or a StripTool plot. However, the results were confusing, AC measurements with SR785 was carried out in the end. It turned out that both IN1 and IN2 had some issues. IN1 showed an increment of the gain by 2dB every two gain steps, having suggested that the 1dB gain stage had a problem. IN2 showed sudden drop of the signal at the gain +8~+15dB and +24~+31dB, having suggested that a particular 8dB stage had a problem. The board was exposed with the extender and started tracing the signals.

CH1: The digital signal to switch the 1dB stage reached Pin 1A of the DIN96 connector. However, the latch logic (U47 74ALS573) does not spit out the corresponding level for this bit. Note that the next bit was properly working. We concluded that this 74ALS573 had failed and need to be replaced. We have no spare of this wide SOIC-20 chip, but Downs seems to have some spares (see Todd's spare parts list). We will try to get the chip on Monday.

CH2: The stage only used between +8dB and +15dB and between +24dB and +31dB is the +8dB stage (U9 and U2A). I found that the amped output signal did not reach the FET switch U2A (MAX333A). Therefore it was concluded that the opamp U9 (AD829) has an issue. In fact, the amp itself was working, but the output pin was not properly soldered to the pad.  Resoldering this chip made the issue gone. Note that this particular channel has some OP27s soldered instead of AD829. Gautam mentioned that there was some action on the board a few years back to deal with the offset issue. Next time when the board is polled out, I'll take the photos of the board.

Using SR785, the swept sine measurements between 100 and 100kHz were taken for all the gain settings for each channel. Between -31dB and -11dB, the input signal amplitude of 300mV was used. Between -10dB and +10dB, it was reduced to 100mV. For the rest, the amplitude was 10mV. Note that the data for +11dB for CH1 and +2dB for CH2 are missing presumably due to a data transfer issue.

The results are shown in Attachments 1~4.

Attachments 1 and 3 show the gain at each slider value. The measured gain was represented by the average between 1kHz and 10kHz. The missing 1dB every two slide values are seen for CH1. The phase delay at 100kHz is show in the lower plot. There is some delay and delay variation seen but it is in fact less than 1deg at 10kHz (see later) so it's effectfor CM servo (IMC AO path) is minimum. The gain for CH2 tracks the slider value nicely. The phase delay is larger than that of CH1, as expected because of OP27.

Attachments 2 and 4 show the transfer functions. The slider value was subtracted from the measured gain magnitude to indicate the deviation between them. The missing 1dB is obviously visible for CH1 in addition to the overall gain offset of ~0.2dB. CH2 also shows the gain offset of 0.1dB~0.2dB. The phase delay comes into the play around 20kHz particularly at higher gains where the UGF of the AO path is.

gautam: Here is the elog thread for IN2 opAmps going AD829-->OP27. Also, I guess Attachment #1 and #3 x-axes should be "Gain [dB]" rather than "Frequency [Hz]".

Now with the CM board tested with the signal injected, it turned out that the latch logic was flipped. As the default state locked the digital levels, the buttons other than the mbbo channels were inactive.

By giving 0 to C1:LSC-CM_LATCH_ENABLE, the modification of the digital state is enabled. And with the value of 1, the digital bits on the board is locked.

In order to reflect this, latch.py was modified and now the controls are all activated.

[Koji]

The logic chips 74ALS573 were replaced. And now the gain sliders are working properly.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[done] CM Board
[none] ALS I/F board

Last week we found that the logic chip for the REFL1 gain switching was not transmitting the input logic. I went to Downs and obtained the chips. After some inspection some other latch chips were suspicious. Therefore U46, U47, and U48 (#1, #3, and #4 from the top) were replaced. After the replacement, the gain measurements were repeated. This time the test for the AO gain was also performed. Now all three slideres show the gain as expected except for the consistent -0.2dB deficit.

Note that the transfer functions for the REFL gains were measured with the input at IN1 or IN2 and the output at TESTA1. The TFs for the AO gain was measured with the excitation at EXC B, the input at TESTB2 and the output at the SERVO output. The gain and phase variantions for the AO gain at low frequency is the effect of AC coupling existing between the excitation and the servo output.

[Update on Oct 14, 2019]

The measured transfer functions show the phase delay determined by the opamps involved. The phase delay well below the pole frequencies can be represented well by a simple time delay (a phase delay linear to the frequency). Attachment 7 shows the time delay estimated by LISO for each gain setting of each gain stage. REFL2 has particularly large phase delay because of the use of OP27s. The delay is even larger when the gain is high presunmably because of the limited GBW.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[done] CM Board
[none] ALS I/F board

The photos of the latest board can be found as Attachments 3/4

With some input signals, the functionarities of the CM servo switches were tested.

• Latch logic works. But latch alive signal is missing.
• IN1 enable/disable, IN2 enable/disable are properly working
• OUT2 toggle switch for REFL1/REFL2 mon is wokring
• Boost / Super Boosts are working
• EXC A enable/disable, EXC B enable/disable switches are working
• Option 1 and Option 2 now isolate the input when either is enabled (as there is no option board)
• 79Hz-1.6kHz pole zero pair works fine
• OUT1 works fine
• Disable/Enable switch for the fast path works
• Polarity switch works
• AO Gain property changes the gain
• Limitter switch works (Attachments 4/5). The limitter clipps the output at 4~4.5V. The Limitter indicator also works.

After the tests the LSC cables were reconnected (Attachment 6)

The boost filters of the CM servo board were tested. Their ZPK models were made.

The transfer functions of the boost filters were measured with the SG output of a SR785 connected to IN1. The IN1 gain was set to be 0dB. The transfer function was taken between the IN1 input and the TEST1A output.
With no boost and normal boost, the input signal amplitude was fixed to 20mVpk. For the other boosts, however, I could expect large gain variation through a single sweep. Therefore automatic SG amplitude tracking was used. The target was to have the output to be 1V with maximum amplitude of 100mV.

Attachment 1 shows the measured transfer functions.

The pole and zero frequencies of the boosts were estimated using LISO. Here the TFs were normalized by the TF of 'no boost' to cancel the delay of the other stages including that of the monitor channel.

ZPK model of Normal Boost:

pole 44.0597566447
zero 4.3927650910k
factor 98.8275377818

ZPK model of Super Boost (State1):

pole 878.5368382789
zero 17.5107366335k
factor 20.0840668188
 

ZPK model of Super Boost (State2):

pole 714.8112014271
pole 1.0147609373k
zero 13.2470941080k
zero 22.2259701828k
factor 404.5411036031
 

ZPK model of Super Boost (State3):

pole 886.3650348470
pole 420.4089305781
pole 887.8490768202
zero 8.3635166134k
zero 15.7953592754k
zero 20.5144907279k
factor 8.2051379423k

Looking at the old latch.st code, looks like this is just a heartbeat signal to indicate the code is alive. I'll implement this. Aesthetically, it'd be also nice to have the hex representation of the "*_SET" channels visible on the MEDM screen.

CM Board Slow out (digital length control) path transfer function / pole-zero filter pair (79Hz/1.6kHz) transfer function

The excitation was given from EXC A. The denominator was TESTA2, and the numerator was OUT1.

Attachment 1 shows the measured transfer function with and without PZ filter off and on. The PZ filter provides ~26dB attenuation at  high frequency. The output stage has a single order 100kHz LPF and it is visible in the transfer function.

The transfer function without the PZ filter was modelled by LISO as the following PZK representation. There looked a small step in the TF which caused the additional PZ pair (66~67Hz) but has very minor effect in the mag and phase.

pole 66.2720207366
zero 67.2660731875
pole 93.3044858160k
factor -995.5583556921m

The transfer function of the PZ filter was separately analyzed. The TF with the switch ON was normalized by the one with the switch OFF. Thus it revealed the pure effect of the switch. The PZK model of the stage was estimated to be

pole 79.7312926438
zero 1.6395485993k
factor 996.2196584165m

For the CM board modeling purpose, the transfer function from TESTA2 to TESTB2 was needed. (Attachment 1)

The ZPK model of this part is

pole 76.2369881805
zero 77.4655685092
pole 7.0761486105M
factor -993.0593433578m

The output stage (and AO GAIN stage) of the MC board was modelled. The transfer function was measured with the injection from EXC B. The denominator was TESTB2, and the numerator was SERVO OUT.

This stage is AC coupled by 2x 1st order HPFs. Firstly, this transfer function was measured with AO GAIN set to be 0dB. (Attachment 1)
This TF was used to characterize the cutoffs of the HPF stages, represented as the following ZPK:

zero 1m
zero 1m
pole 6.0502599855
pole 6.0624642854
factor -26.2725046079n

Then the AO GAIN was already measured as seen in [ELOG 14948]. The AO gain TF was then modeled by LISO with the above HPF as the preset. This allows us to characterize the time delay of the AO GAIN part.

Input referred offsets on the IN1/IN2 were tested with different gain settings. The two inputs were plugged by the 50 ohm terminators. The output was monitored at OUT1 (SLOW Length Output). The fast path is AC coupled and has no sensitivity to the offset.

There is the EPICS monitor point for OUT1. With the multimeter it was confirmed that the EPICS monitor (C1:LSC-CM_REFL1_GAIN) has the right value except for the opposite sign because the output stage of OUT1 is inverting. The previous stages have no sign inversion. Therefore, the numbers below does not compensate the sign inversion.

Attachment 1 shows the output offset observed at C1:LSC-CM_REFL1_GAIN. There is some gain variation, but it is around the constant offset of ~26mV. This suggested that the most of the offset is not from the gain stages but from the later stages (like the boost stages). Note that the boost stages were turned off during the measurements.

Attachment 2 shows the input refered offset naively calculated from the above output offset. In dependent from which path was used, the offset with low gain was hugely enhanced.

Since the input referred offset without subtracting the static offset seemed useless, a constant offset of -26mV was subtracted from the calculation (Attachment 2). This shows that the input refered offset can go up to ~+/-20mV when the gain is up to -16dB. Above that, the offset is mV level.

I don't think this level of offset by whichever OP27 or AD829 becomes an issue when the input error signal is the order of a volt.
This suggests that it is more important to properly set the internal offset cancellation as well as to keep the gain setting to be high.
 

Updated Circuit Diagram and photos: https://dcc.ligo.org/D1500308-v2

- (1) and (6) of the diagram: TFs with various gain slider values for REFL1/REFL2/AO GAIN [ELOG 14948] (gain values and time delay modeling)
- Switching checks, latest photo of the board, Limiter check  [ELOG 14953]
- (2): Boost transfer functions [ELOG 14955]
- (3): Slow (aka Length) CM output path [ELOG 14965]
- (4): Pole-Zero filter TF [ELOG 14965]
- (5): TF from TESTA2 to TESTB2 [ELOG 14966]
- (6): AC coupling TF of the AO GAIN stage [ELOG 14967]
- (7): AC coupling TF of the IN2 stage on IMC servo board [ELOG 15044]

Slow path = (1)*(2 if necessary)*(3)*(4 if necessary)

Fast path = (1)*(2 if necessary)*(4 if necessary)*(5)*(6)

gautam 20191122: Adding the measured AC coupling of the IN2 input of the IMC servo board for completeness.

I wanted to restart the c1oaf model. As usual, the first time the model was restarted, it came back online with a 0x2bad error. This isn't even listed in the diagnostics manual as one of the recognized error states (unless there is a typo and they mean 0x2bad when they say 0xbad). The fix that has worked for me is to stop and start the model again, but of course, there is some chance of taking all the vertex FEs down in the process. No permutation of mxstream and daqd process restarts have cleared this error. We need some CDS/RCG support to look into this issue and fix it, it is not reasonable to go through reboots of all the vertex FEs every time we want to make a model change.

Jon and I were surveying the CDS situation so that he can prepare a report for discussion with Rolf/Rich about our upcoming BHD upgrade. In our poking around, we must have bumped something somewhere because the c1ioo machine went offline, and consequently, took all the vertex models out. I rebooted everything with the reboot script, everything seems to have come back smoothly. I took this opportunity to install some saturation counters for the arm servos, as we have for the CARM/DARM loops, because I want to use these for a watch script that catches when the ALS loses lock and shuts stuff off before kicking optics around needlessly. See Attachment #1 for my changes.

I have been experiencing frequent crashes of DTT on pianosa in the past few weeks. This is pretty annoying to deal with when trying to characterize the interferometer loops. I attach the error log dumped to console. The error has to do with some kind of memory corruption. Recall that we aren't using a GDS version that is packaged with the SL7 lscsoft packages, we are using a pretty ancient (2.15) version that is built from source. I have been unable to build a newer version from source (though I didn't spend much time trying). pianosa is the only usable workstation at the moment, but perhaps someone can make this work on donatella / rossa for general improvement in quality of life.

After the CDSs crashed we run the rebootC1LSC.sh script.

The script is a bit annoying in that it requires entering the CDSs' passwords multiple times over the time it runs which is long.

The resulting CDS screen is a bit different than what was reported before (attached). Also, not all watchdogs were restored.

We restore the remaining watchdogs and do XARM locking. Everything seems to be fine.

It was way more annoying without a script and took longer than the 4 minutes it does now.

You can fix the requirement to enter password by changing the sshd settings on the FEs like I did for pianosa.

After running the script, you should verify that there are no red flags in the output to console. Yesterday, some of the settings the script was supposed to reset weren't correctly reset, possibly due to python/EPICS problems on donatella, and this cost me an hour of searching last night because the locking wasn't working. Anyway, best practise is to not crash the FEs.

I tried starting the c1oaf model, but got a DQ error (I want the option of running feedforward during locking even if the filters aren't particularly well tuned yet). Note that this isn't "just a warning light" - some channels are initialized to +/- 1e20, so if you try turning some filters on, you will deliver a massive kick to the optics. Restarting it crashed c1lsc (this is not unexpected behavior - the only way to clear the DQ error is to restart the model, and empirically, the success rate is ~50%). The reboot script brought everything back online smoothly, and the second, time, c1oaf started without any issues.

While looking at the CDS overview screen, I noticed that the c1scy model was reporting frequent RFM errors for the C1:SCY-RFM_ETMY_LSC channel (but none of the others). On the sender model (c1rfm), no errors were being reported. The diag reset button / mxstream restart didn't really work either. See Attachment #1. Just restarting the c1scy model didn't fix the error - I had to reboot the machine and restart the models, and now no errors are being reported.

Attachment #2 shows the current nominal CDS status - the red light on c1lsc is due to some missing c1dnn channels (I'll remove these at the next c1lsc model change because I don't want to un-necessarily reboot the vertex FEs), and the c1omc model is obsolete I guess. c1daf isn't running right now but once I get the new fiber (ordered), I'm gonna restart this model as well.

P.S. The ALS temperature sliders are not SDF-ed. So when the model was restarted, I had to change the sliders back to their old values to get the beat back in the usable range.

Summary:

Every new year (on Dec 31 or Jan 1), all of the realtime models will report a "0x4000" error. This happens due to an offset to the GPStime driver not being updated. Here is how this can be fixed (slightly modified version of what was done at LASTI).

Steps to fix the DC errors:

1. ssh into FB machine. 
2. Edit the file /opt/rtcds/rtscore/release/src/include/drv/spectracomGPS.c:
   • Look for the code block with a text string that reads something like

     /* 2019 had 365 days and no leap seconds */
                  pHardware->gpsOffset += 31536000;

   • Copy and paste the above string for the appropriate number of years of offset you are adding, and edit the comment string appropriately!.
3. Navigate to /opt/rtcds/rtscore/release/src/drv/symmetricom. Run the following commands:

   sudo make
   sudo make install

4. Stop all the daqd processes and reload symmetricom:

   sudo systemctl daqd_* stop
   sudo modprobe -r symmetricom
   sudo modprobe symmetricom

5. Re-start the daqd processes:

   sudo service daqd_* start

Independent of this, there is a 1 second offset between the gpstimes reported by /proc/gps and gpstime. However, this doesn't seem to drift. We had effected a static offset to correct for this in the daqd config files, and it looks like these do not need to be updated on a yearly basis. All the daqd indicators are now green, see Attachment #1.

Seems that the GPS is out of sync on donatella. We could not get any data from diaggui...

$TARGET_DIR = /cvs/cds/caltech/target

• $TARGET_DIR/c1psl and $TARGET_DIR/c1iool0 moved to $TARGET_DIR/preAcromag_oldVME/
• $TARGET_DIR/c1psl1 moved to $TARGET_DIR/c1psl 
• $TARGET_DIR/c1psl/*.service and $TARGET_DIR/C1_PSL.cmd modified - i executed :%s/c1psl1/c1psl/g in vim.
• $TARGET_DIR/preAcromag_oldVME/c1psl/autoBurt.req and $TARGET_DIR/preAcromag_oldVME/c1iool0/autoBurt.req catenated into $TARGET_DIR/c1psl/autoBurt.req. The first snapshot at 16:19 has been verified.

It remains to (Jon is taking care of these)

• add a line to modbusIOC.service on the new c1psl machine that restores the latest burt snapshot on startup (this necessitated installation of a debian jessie libXp6 package on our debian buster machine because our shared EPICS is soooooooooooooo oooooooold)
• change the hostname from c1psl1 to c1psl
• update martian.hosts

Channel list with test status
== Test Status ==

[done] Lock PMC and IMC
[done] IMC Servo board test
[done] IMC LO Det Mon channel check
[0th order] WFS quadrant DC mon
[none] WFS I/F monitors
[0th order] WFS attenuators
[none] IOO QPD channels
[done] FSS readbacks 
[done] PMC readbacks

Some more detailed elogs about the individual tests will follow.

Basically, I have characterized the IMC Servo board in detail. The summary finding is that the IN2 (=AO gain) slider needs to be investigated. 

All other channels need to be verified in a more thorough fashion than my basic checks which were just to guarantee the core interferometer functionality, which is important to me.