40m QIL Cryo_Lab CTN SUS_Lab TCS_Lab OMC_Lab CRIME_Lab FEA ENG_Labs OptContFac Mariner WBEEShop
  40m Log, Page 272 of 337  Not logged in ELOG logo
ID Date Author Type Category Subjectup
  918   Thu Sep 4 00:38:14 2008 ranaUpdatePSLc1iovme power cycled
Entry 663 has a plot of this using the PSL/FSS/SLOWscan script. It shows that the SB's were ~8x smaller than the carrier.
P_carrier   J_0(Gamma)^2 
--------- = ------------
P_SB        J_1(Gamma)^2

Which I guess we have to solve numerically for large Gamma?
  919   Thu Sep 4 07:29:52 2008 YoichiUpdatePSLc1iovme power cycled

Entry 663 has a plot of this using the PSL/FSS/SLOWscan script. It shows that the SB's were ~8x smaller than the carrier.
P_carrier   J_0(Gamma)^2 
--------- = ------------
P_SB        J_1(Gamma)^2

Which I guess we have to solve numerically for large Gamma?

P_carrier/P_SB = 8 yields gamma=0.67.
  3159   Tue Jul 6 17:05:30 2010 Megan and JoeUpdateComputersc1iovme reboot

We rebooted c1iovme because the lines stopped responding to inputs on C1:I00-MC_DRUM1. This fixed the problem.

  3393   Tue Aug 10 16:55:38 2010 JennaUpdateElectronicsc1iovme restarted

 Alastair and I restarted the c1iovme around the time of my last elog entry (~3:20).

  14916   Mon Sep 30 15:51:59 2019 gautamUpdateCDSc1iscaux - some admin

I did the following:

  1. symlinked /cvs/cds/rtcds to /opt/rtcds.
  2. Added a line to /etc/systemd/system/modbusIOC.service that executes a burt-restore of the latest c1iscaux.snap file so that whitening gains etc are restored to their last saved value in the event of a service restart.
  14765   Tue Jul 16 16:00:01 2019 gautamUpdateCDSc1iscaux Supermicro setup

I worked on preparing for the c1iscaux upgrade a bit today.

  1. Attachment #1: This shows where the 120 GB solid-state hard-drive and the 2 RAM cards (2GB each) are installed.
    • I found that it required considerable application of force to get the RAM cards into their slots.
    • Note: the 4GB RAM is broken up into two separate physical cards, each 2GB. The labeling is a bit confusing, as each card suggests it is by itself 4GB.
  2. OS install for c1iscaux:
    • I followed Jon's instructions (and added some of mine to the wiki page to hopefully make this process even less thinking-intensive).
    • To be able to use the IP address, removed "bscteststand" from chiara martian.hosts and rev.113.168.192.in-addr.arpa as the last mention I could find of this machine was from 2009 (and I'm pretty sure it isn't an active unit anymore). I then restarted the bind9 process. 
    • The hostname for this machine is currently "c1iscaux3" for testing purposes, I will change it once we do the actual install.
    • There was an error in the installation instructions to allow incoming ssh connections - it is openssh-server that is required, not openssh-client. This has now been fixed on the wiki page instructions.
  3. Acromag static IP assignment:
    • Assigned 2 ADCs (XT1221), 5 DACs (XT1541) and 5 sinking BIO units (XT1111) static IP addresses (and labelled them for easy reference) using the windows laptop and the Acromag IP config utility.
    • I saw no reason not to use the 192.168.114.yyy scheme for the Acromag subnet on this machine, even though c1auxex and c1vac both have subnets with this addressing prefix. For reasons unknown to me, Jon opted to use 192.168.115.yyy for the c1susaux Acromag subnet.
  4. Followed the excellent step-by-step to install EPICS, Modbus and Asyn.
    • This took a while, ~1 hour, dominated by the building of EPICS. The other two took only a couple of minutes each.
    • The same combination suggested on Jon's wiki, of Modbus R2-11, EPICS base-7.0.1 and asyn4-33, are the most current at the time of installation.
    • Couple of typos that prevented straight up copy-pasting were fixed on the wiki.
  5. Playground for testing new database files:
    • made a directory /cvs/cds/caltech/target/c1iscaux3 and copied over the .db files from /cvs/cds/caltech/target/c1iscaux and /cvs/cds/caltech/target/c1iscaux2 over.
    • Johannes said he did not develop any code to automate the process of translating the old .db files into the new ones for the Acromag - I won't invest the time in developing any either as I think just manually editing the files will be faster. 
    • I think I will follow the c1susaux convention of grouping .db files by the physical electronics system where possible (e.g. REFL11 channels in one file, CM channels in one file etc), as I think this makes for easier debugging.
    • There is an old "PZT_AI.db" file which I think consists completely of obsolete channels.
  6. Next steps:
    • Wire up the crate [Chub]
    • Make the database files and modbus files for talking to the Acromags on the internal subnet [Gautam], check the .db files [Koji]
    • Wiring of whitening switching from P1 to P2 connector, Issue #1 in this elog (this will also requrie the installation of the DIN shrouds) [Koji]
    • Soldering of P2 interface boards [Gautam]
    • Bench testing [Gautam, Koji, Chub]
    • Installation and in-situ testing [Gautam, Koji, Chub]

All the required additional parts should be here by the end of the week - I'd like to aim for Wednesday 7/24 for the installation in 1Y3 and in-situ testing. While talking to Rana, I realized that we should also factor in the c1aux slow channels into this acromag crate - there is no need for a separate machine to handle the shutters and illuminators. But let's not worry about that for now, those channels can simply be added later.

Attachment 1: IMG_7769.JPG
  14850   Mon Aug 19 14:36:21 2019 gautamUpdateCDSc1iscaux remaining work

Here is what is left to do:

  1. Strain relief of all cabling. Chub will take care of this in the coming days. I have said he can connect and disconnect cables as he pleases, but after this work, we may require a hard reboot of the Acromag chassis before restoring functionality to the channels, as it is known that the Acromags can sometimes get "stuck" by a sudden connection of voltage.
  2. Installation of DB15 cable to the P2 connector of the CM board and a DB9 cable to the ALS demod unit (LO and RF power monitors). These will arrive in the next couple of days and Chub will take care of the install.
  3. Design, manufacture and install of a custom version of the backplane P1 adaptor board with only 1 D37 connector - for some of the PD DC signals, a custom adaptor board, part number D010005 for which I can't find any schematics is already installed on the P2 connector, and makes the DC monitor signals available to 4 LEMO connectors. These signals are then digitized by the fast CDS system, presumably for PDH signal normalization. The footprint of this P2--->LEMO adaptor is such that we cannot simply install our P1---> 2xDB37 adaptor boards, because of space constraints. Fortunately, there is a simple fix to reduce the footprint of the board: remove the bottom DB37 connector, which is unused in the c1iscaux system except for the CM board. I recommend getting ~10 pcs of such boards, as it is also useful in a few other places, where the power cabling to the eurocrates are a space constraint. See Attachment #1 for a picture explaining this situation. Anyone want to volunteer to take care of this?
  4. In-situ testing. This is easiest done with some light available in the interferometer. Which in turn requires IMC to be locked. Which in turn requires satellite box fixing. Anyone want to volunteer to take care of this?
  5. Modify C0EDCU.ini to trend the new slow channels we may want long-term monitoring of (e.g. LO power levels to the Demod boards). Anyone want to volunteer to take care of this?
  6. Decide what to do about the CM latch logic. There are some contraints with the way the acromag register addressing works, that I've had to change the way the mbboDirect bits are controlled. Unfortunately, this seems to sometimes and unpredictably cause the bits to flip in a non-robust way, which is the whole point of having the latch in the first place. Either the latch logic needs to be improved, or we need to implement the latch logic in the fast CDS system, not the slow.

Today I set up the autoburt.req file for the c1iscaux channels, and confirmed that the snapshots are getting recorded. There were a lot of channels in the old autoburt.req file which I thought were un-necessary (and several which no longer exist), so now the only channels that are burt-ed are the whitening gains and states of the AA filters. If someone feels we need more channels to be snapshot recorded, you can add them to the file.

In the old target directory, there were also various versions of a "saverestore.req" file - why do we need this in addition to an autoburt? I guess it is possible they are used by the IFOconfigure scripts to setup some whitening gains etc...

Attachment 1: caseForSmallerFootprint.pdf
  14855   Fri Aug 23 18:46:17 2019 JonUpdateCDSc1iscaux remaining work

I added the list of new c1iscaux channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini and restarted the framebuilder. Koji had thought some of these channels might have previously existed under slightly different names. However, after looking through C0EDCU.ini and the other _SLOW.ini files, I did not find any candidates for removal. As far as I can tell, all of these channels are being recorded for the first time.

  1. Modify C0EDCU.ini to trend the new slow channels we may want long-term monitoring of (e.g. LO power levels to the Demod boards). Anyone want to volunteer to take care of this?
  14857   Sun Aug 25 14:18:08 2019 gautamUpdateCDSc1iscaux remaining work

There were a bunch of useless / degenerate channels added - e.g. whitening gains which are alreay burt-snapshot. Maybe there are many more useless channels being trended, but no need to add more.

Copy-pasting wasn't done correctly - the first 4 added channels were duplicates. There are in fact 5 LO power mons, one for each of the frequencies 11, 33, 55, 110 and 165 MHz. 

I cleaned up. Basically only the detect-mon channels, and the ALS channels, are new in the setup now. I will review if any extra channels are required later. While checking that the daqd is happy, I noticed c1lsc FEs are in their stuck state, see Attachment #1. I guess a cable was bumped when the strain relief operation was underway. I'm not attempting a remote resuscitation.


I added the list of new c1iscaux channels to /opt/rtcds/caltech/c1/chans/daq/C0EDCU.ini and restarted the framebuilder. Koji had thought some of these channels might have previously existed under slightly different names. However, after looking through C0EDCU.ini and the other _SLOW.ini files, I did not find any candidates for removal. As far as I can tell, all of these channels are being recorded for the first time.

Attachment 1: Screen_Shot_2019-08-25_at_10.38.37_PM.png
  14903   Fri Sep 20 12:55:02 2019 gautamUpdateCDSc1iscaux testing

I was hoping that the dark / electronics noise level on the LSC photodiodes would be sufficient for me to test the whitening gain switching on the iLIGO Pentek whitening boards. However, this does not seem to be the case. I guess to be thorough, we have to do this kind of test. It's a bit annoying to have to undo and redo the SMA connections, but I can't think of any obvious easier way to test this functionality. More annoyingly, the sensing matrix infrastructure necessary to do the kind of test described in the linked elog is only available for some PDs. I don't really want to modify the c1cal model and go through another mass reboot cycle.

While I was at it, I was also thinking about the tests we want to do. Here is a quick first pass - if you can think of other tests we ought to do, please add them to the list!

  1. Whitening gain switching on the D990694 boards.
    • Need to inject some signal to do this in a clean way. 
    • With some signal injected, we need to switch the whitening gain through the 15 available levels and confirm that we see a +3dB gain for each step.
    • An example script to do this operation and make a diagnostic plot is at /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py.
  2. AA enable/disable on the D000076 boards. Do we really need this functionality? Can't we permanently enable the AA, as was done for WF2?
    • Need to measure the TF with an SR785 or drive a high-freq line and confirm that the aliased peak height is attenuated as expected in DTT.
  3. LO Det Mon channel check
    • Zeroth level test can be done by turning Marconi OFF/ON, and confirming we see a change in the corresponding monitor channel, like I did here.
    • A more rigorous diagnostic would require these channels to be calibrated to dBm of LO power.
  4. PD INTF board check
    • Zeroth level check can be done by shining light onto PDs one at a time and confirming that the correct channel shows a response.
    • A more rigorous diagnostic would require these channels to be calibrated to mW of optical power incident on the PDs.
  5. QPD INTF board check
    • This is the IP-POS QPD readback.
    • Need to confirm the quadrant mapping, and that Pitch is really Pitch, Yaw is really Yaw.
    • A more rigorous diagnostic would require these channels to be calibrated to mm of position shift.
  6. CM Board
    • Need to determine what tests need to be done.
    • I have not yet implemented the fix for the MBBO gain channels for all the gains - only REFL1_GAIN is set up correctly now. Need to look at the hardware for the correct addressing of bits.
  7. ALS INTF board
    • This board isn't actually connected yet, pending strain relief of cabling at 1Y2.
    • The calibration of the board output volts to dBm is known, so we can easily check this functionality.
  14905   Mon Sep 23 10:49:34 2019 ranaUpdateCDSc1iscaux testing
  • I'd say permanently enable AA and AI. There's no reason to turn these off for usual channels. We can always undo one switch later if we want to use aliasing to sample a high frequency signal (ala SoCal).
  • The PD output should ~20 nV/rHz into the mixer, so that's ~7 nV into the whitening filter. We need 60 dB to be above the ADC noise.
  • I've forgotten what the current config is, but in iLIGO we hacked in a fixed whtiening on the Lt1128 input amp to the WF board so that the lock acquisition could be a little easier (better SNR). On Ch1, that's replacing R60 with a RC network. We want to make sure that the lock acq transients are not saturating the ADC, but can maybe put in a 40:200 stage.
  14906   Wed Sep 25 20:10:13 2019 KojiUpdateCDSc1iscaux testing

== Test Status ==
Whitening gain switching test
[none] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

- LO Det Mon channel check

The StripTool template for the test was made:
Then, the RF output of the main Marconi was toggled a few times. -> Confirmed the channels are respopnding. (Attachment 1)

- IPPOS channel check

(0th order check) The StripTool template for the test was made:
Then, the IPPOS QPD was shined with a phone LED. Initially I saw no response of the QPD. It turned out that the IPPOS IF module had no input cable connected. After the connection, all the 4 segments are responding to the phone LED and also the IFO beam.

(more careful check)
I decided to do more careful check of IPPOS. As there was a f~30mm lens on the oplev table, beam was focused such that only one element reacted to the incident beam. The beam power (a few mW) was too strong for a single QPD element, which saturates at ~6, an ND filter of OD0.6 was used to reduce the incident power.

Here are the results:
SEG1 (UPPER LEFT seen from the beam) | C1:ASC-IP_POS_QPD_Seg1_Mon 3.651+/-0.003 (N=10) | Incident Power 2.35+/-0.01 mW, QPD X_Calc (+) Y_Calc (+)

Segment Arrangement
(Seen from the beam)
Epics Channel CH output

Incident Power

Polarity for the
X/Y_Calc channels
SEG1 UPPER LEFT C1:ASC-IP_POS_QPD_Seg1_Mon 3.651+/-0.003
2.35+/-0.01 X(+) / Y(+)
SEG2 LOWER LEFT C1:ASC-IP_POS_QPD_Seg2_Mon 3.607+/-0.002
2.35+/-0.01 X(+) / Y(-)
SEG3 LOWER RIGHT C1:ASC-IP_POS_QPD_Seg3_Mon 3.658+/-0.002
2.37+/-0.01 X(-) / Y(-)
SEG4 UPPER RIGHT C1:ASC-IP_POS_QPD_Seg4_Mon 3.529+/-0.004
2.30+/-0.01 X(-) / Y(+)

After the measurement, the lens and the filter were removed and the beam was adjusted to the center of the QPD.

Attachment 1: testDetectMons_190925.png
Attachment 2: testIPPOS_190925.png
  14908   Thu Sep 26 20:09:40 2019 KojiUpdateCDSc1iscaux testing

== Test Status ==

[done] Whitening gain switching test => Some issues found (POP110Q, Whitening3_8 not switching, ASDC overall behavior, REFL33Q needs recheck)
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

And, the Y-arm lock was recovered! After some alignment work, the Y-arm was locked. The whitening gain for POY11 was +18dB. The servo gain was 0.015 (nominal).
Once the transmission reached 0.8, I could use ASS to align the cavity and the TTs.
The transmission reached just 1.00 at the end. Was the transmission recently normalized? (See attachment 5)

- Whitening Filter Gain Switching Test

Each whitening filters were tested individually. +50mV DC signal was connected to the 8 inputs using an SMA octopus cable.
The existing script ( /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain.py ) did not work because cds.getdata failed to fetch all of the data requested. By giving some sleep before start downloading the data, the problem was avoided. Still some truncated data are seen in the result, but StripTools screenshots compliments the missing part.

Whitening Filters #2~4 were a little tricky because the code needed modification so that the spare channels can be tested.
The modified script is stored as /cvs/cds/caltech/target/c1iscaux/testScripts/testWhtGain_190926.py 

Whitening #1: No issue found.

Whitening #2: No issue found. Some of the step plots showed truncation of the data at the end. But this is an artifact of cds.getdat. The striptool show nothing irregular.

Whitening #3: POP110Q and the spare channel (CH8) did not show the reaction. REFL33Q showed some systematic gain deviation. It could just be the offset problem, but needs to be rechecked.

Whitening #4: The DC channels were found to be OK  except for ASDC. ASDC shows earlier saturation. The input was lowered to 5mVDC to avoid saturation in the second trial. The circuit needs to be checked. The spare channels look noisy, but this is because there is no way to turn off the whitening filters for them.

- AA Filter Test

Injected 11kHz 1Vpp sine wave to the whitening filters. The whiter gains were kept at 0dB. If the AA is disabled, the alias of the 11kHz signal appears in the time series.
-> Whitening #1, #3 and #4: the enable/disable worked correctly.
-> Whitening #2 AA
Bbypass no effect. this is an expected behavior.

Attachment 1: Wht1.pdf
Wht1.pdf Wht1.pdf Wht1.pdf Wht1.pdf Wht1.pdf
Attachment 2: Wht2.pdf
Wht2.pdf Wht2.pdf Wht2.pdf Wht2.pdf Wht2.pdf Wht2.pdf
Attachment 3: Wht3.pdf
Wht3.pdf Wht3.pdf Wht3.pdf Wht3.pdf Wht3.pdf Wht3.pdf Wht3.pdf
Attachment 4: Wht4.pdf
Wht4.pdf Wht4.pdf Wht4.pdf Wht4.pdf Wht4.pdf Wht4.pdf Wht4.pdf Wht4.pdf
Attachment 5: lock.png
  14909   Fri Sep 27 15:59:53 2019 gautamUpdateCDSc1iscaux testing

I reset the normalization for both arms on Jul 9 2019.


The transmission reached just 1.00 at the end. Was the transmission recently normalized? (See attachment 5)

  14921   Wed Oct 2 01:11:40 2019 KojiUpdateCDSc1iscaux testing

I worked on more troubleshooting of the whitening filters Tuesday afternoon

== Test Status ==

[done] Whitening gain switching test => Remaining issues ASDC overall behavior
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

Issue 1: POP110Q did not show any gain switching [Resolved]

A DB37 breakout board was connected to the acromag front panel. I found that Ch6 (POP110Q) did not show any differential DC output. I searched around the other pins and found that the corresponding signal showed up on PIn36  instead of Pin35. Opening the front panel revealed that the internal wiring was wrong (Attachment 1). The wire which should have gone to Pin 35 was connected to Pin 36. By correcting the wiring, POP110Q started to show identical behavior to POP110I. (Attachment 2)

Issue 2: LSC reboot [Resolved]

A rough activity around the acromag chassis crashed c1lsc realtime processes (as usual). I ran usual rebooting script /opt/rtcds/caltech/c1/scripts/cds/rebootC1LSC.sh. This successfully restored the status of the vertex RT processes.

Issue 3: REFL33 different behavior between I and Q [Resolved]

REFL33I and Q consistently showed a difference (Attachment 3). The whitening board was pulled out and powered with an extension card. The raw outputs were checked with a function generator and an oscilloscope connected. The outputs for 33I and Q were identical (Attachment 4). So I concluded that the observed difference was an artifact of the checking script.

Issue 4: Whitening 3_8 did not switch at all [Resolved]

To switch the gain stages, each channel of the whitening board takes a DAC output from acromag and convert it into 4bit digital signals. For CH8 of the WF#3, this signal did not reach the instrumentation amplifier AD620. After tracing the signal on the electronics bench, it was found that the CH8 gain input to the DIN96 connector is not conducive to the input of the AD620. As there were no exposed pads between the DIN96 connector and the AD620 input (pin2), a wire was additionally soldered (forgot to take a photo). This solved the gain switching issue as the test result indicates (Attachment 5). The noisiness came from the whitening filter which can not be turned off right now. For this reason, the test of the whitening part is pending too.

The StripTool plot during the overall WF#3 test is shown in Attachment 6.

Issue 5: ASDC behavior [Unresolved]

First of all, at this test, I found that WF#4 was not responding to the gain change at all. This issue was restored by power cycling the acromag chassis (as usual).

The whitening filter #4 was pulled, and the behavior of CH5,6,7,8 (CH8=ASDC) was compared. It was found that the analog outputs were identical and the problem lies further downstream.

Issue 6: Illeagal REFL11 LO cable [Unresolved]

This is a newly found issue. The cable between the LO distributor and the REFL11 demodulator is not the legit solder soaked RG402 coax, but flexible coax (Attachment 7). This cable needs to be replaced in the end. But for today, it was not so that we can have a consistent configuratin as before.

Issue 7: Signature of a damaged POPDC cable [Resolved]

The cable for POPDC cale indicated some damage at the WF#4 side. It was not a complete damage, and therefore the solder coating was added (Attachment 8).

Attachment 1: WF3_wiring.png
Attachment 2: POP110.pdf
Attachment 3: REFL33.pdf
Attachment 4: P_20191001_174548_vHDR_On.jpg
Attachment 5: Whitening3_8.pdf
Attachment 6: Screenshot_WF3_191001.png
Attachment 7: P_20191001_181052_vHDR_On.jpg
Attachment 8: POPDCcable.png
  14939   Fri Oct 4 01:57:09 2019 KojiUpdateCDSc1iscaux testing

The AA filter for ASDC was fixed.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[none] CM Board
[none] ALS I/F board

The AA filter for the 4th section of the LSC analog electronics bank (D000076) was pulled out for the test. On the workbench, questionable CH8 was checked. It tuned out that the filter amplifier module for the 8th-order elliptic filter at 7.5kHz was not properly working and exhibited unusual attenuation. This filter module (Frequency Devices Inc D68L8E-7.50kHz) was desoldered and replaced with a module from a spare board. Note that Gautam and I had tried to use this spare board instead of the current one, but it didn't give us any signal for an unknown reason. Since the desoldering required a lot of force and had a risk of damaging the PCB, a socket was made from an IC socket (see Attached 1). This change made CH8 functioning equally to the other channels do.

I took this opportunity to ckech the performance of the AA filters. For each channel, the input signal was injected from J3 using a pomona clip. The output was taken from pin 1, 5, 9, ... of J2. This is the + side of the differential output. The - side just has the equivalent performance but the signal polarity. The digital signals for the AA bypass switches were not connected. Fortunately, this was just fine as it made the anti-aliasing filters engaged.

Attachment 2 shows the transfer functions of all the channels. All the channels showed an identical response (at least visually). The transfer function for CH1 was fitted by LISO. The ZPK values are listed here:

pole 5.2860544577k 503.1473053928m
pole 5.9752193716k 1.0543411596
pole 8.9271953580k 3.5384364788
pole 8.2181747850k 3.4220607928
pole 182.1403534923k 1.1187869426 # This has almost no effect
zero 13.5305051680k 423.6130434049M
zero 15.5318357741k 747.6895990654k
zero 23.1746351749k 1.5412966100M

factor 989.1003181564m
delay 24.4846075283n

Attachment 3 shows the ASD of the output voltage noise measurement. Note the input was shorted for this measurement. The nominal output voltage was found to be 0.1 uV/rtHz and the 1/f noise corner freq was about 100Hz. Only CH3 showed a deviation from the typical values. It looks like this is neither an artifact nor transient noise. Fortunately, nothing is connected to this channel right now.

Attachment 1: P_20191003_172956_vHDR_On.jpg
Attachment 2: TF.pdf
Attachment 3: PSD.pdf
Attachment 4: 191003_AA_Filter.zip
  14942   Sat Oct 5 00:03:21 2019 KojiUpdateCDSc1iscaux testing

[Gautam, Koji]

Input gain part of the CM servo board D1500308 was tested. A couple of problems were detected. One still remains.

== Test Status ==

[done] Whitening gain switching test
[done] AA enable/disable switching
[0th order] LO Det Mon channel check
[none] PD I/F board check
[done] QPD I/F board check
[in progress] CM Board
[none] ALS I/F board

We started to test the CM Servo board from the input stages. Initially, DC offsets were provided to IN1 and IN2 to check the gain on the oscilloscope or a StripTool plot. However, the results were confusing, AC measurements with SR785 was carried out in the end. It turned out that both IN1 and IN2 had some issues. IN1 showed an increment of the gain by 2dB every two gain steps, having suggested that the 1dB gain stage had a problem. IN2 showed sudden drop of the signal at the gain +8~+15dB and +24~+31dB, having suggested that a particular 8dB stage had a problem. The board was exposed with the extender and started tracing the signals.

CH1: The digital signal to switch the 1dB stage reached Pin 1A of the DIN96 connector. However, the latch logic (U47 74ALS573) does not spit out the corresponding level for this bit. Note that the next bit was properly working. We concluded that this 74ALS573 had failed and need to be replaced. We have no spare of this wide SOIC-20 chip, but Downs seems to have some spares (see Todd's spare parts list). We will try to get the chip on Monday.

CH2: The stage only used between +8dB and +15dB and between +24dB and +31dB is the +8dB stage (U9 and U2A). I found that the amped output signal did not reach the FET switch U2A (MAX333A). Therefore it was concluded that the opamp U9 (AD829) has an issue. In fact, the amp itself was working, but the output pin was not properly soldered to the pad.  Resoldering this chip made the issue gone. Note that this particular channel has some OP27s soldered instead of AD829. Gautam mentioned that there was some action on the board a few years back to deal with the offset issue. Next time when the board is polled out, I'll take the photos of the board.

Using SR785, the swept sine measurements between 100 and 100kHz were taken for all the gain settings for each channel. Between -31dB and -11dB, the input signal amplitude of 300mV was used. Between -10dB and +10dB, it was reduced to 100mV. For the rest, the amplitude was 10mV. Note that the data for +11dB for CH1 and +2dB for CH2 are missing presumably due to a data transfer issue.

The results are shown in Attachments 1~4.

Attachments 1 and 3 show the gain at each slider value. The measured gain was represented by the average between 1kHz and 10kHz. The missing 1dB every two slide values are seen for CH1. The phase delay at 100kHz is show in the lower plot. There is some delay and delay variation seen but it is in fact less than 1deg at 10kHz (see later) so it's effectfor CM servo (IMC AO path) is minimum. The gain for CH2 tracks the slider value nicely. The phase delay is larger than that of CH1, as expected because of OP27.

Attachments 2 and 4 show the transfer functions. The slider value was subtracted from the measured gain magnitude to indicate the deviation between them. The missing 1dB is obviously visible for CH1 in addition to the overall gain offset of ~0.2dB. CH2 also shows the gain offset of 0.1dB~0.2dB. The phase delay comes into the play around 20kHz particularly at higher gains where the UGF of the AO path is.

gautam: Here is the elog thread for IN2 opAmps going AD829-->OP27. Also, I guess Attachment #1 and #3 x-axes should be "Gain [dB]" rather than "Frequency [Hz]".

Attachment 1: REFL1_GAIN1.pdf
Attachment 2: REFL1_GAIN2.pdf
Attachment 3: REFL2_GAIN1.pdf
Attachment 4: REFL2_GAIN2.pdf
  14956   Tue Oct 8 20:23:03 2019 gautamUpdateCDSc1iscaux testing

Looking at the old latch.st code, looks like this is just a heartbeat signal to indicate the code is alive. I'll implement this. Aesthetically, it'd be also nice to have the hex representation of the "*_SET" channels visible on the MEDM screen.



Latch logic works. But latch alive signal is missing.

  14912   Mon Sep 30 11:20:43 2019 gautamUpdateCDSc1iscaux testing - CM board code updated

DATED, SEE ELOG14941 for the most up-to-date info on latch.py.

I modified /cvs/cds/caltech/target/c1iscaux/latch.py and /cvs/cds/caltech/target/c1iscaux/C1_ISC-AUX_CM.db to set up the mbbo logic for the other three channels on the CM board, namely REFL2 Gain, AO Gain, and the Super boosts. The systemctl processes were restarted on c1iscaux. We are now ready to perform systematic checks on the CM board functionality.


The addressing of the Acromag BIO registers is done in a way that is kind of inconvenient to use the EPICS mbboDirect protocol

  • The control word going to the Acromag is 16 bits in length
  • However, only the 4 least significant bits actually correspond to physical channels - the remaining 12 bits are "unused".
  • Because each Acromag BIO unit has 16 BIO channels, this means that they are grouped into four "banks" of 4 bits each.
  • The mbboDirect EPICS/modbus protocol is used to control multiple physical BIO channels using a single input, which is exactly what we want for the gain sliders on the CM board. However, one caveat is that the bits need to be consecutive.
  • This means that we have to break up the 6 bits used for the gain sliders (and in fact also the 2 bits used for the super boosts) into a least-significant-bits (LSB) group and a most-significant-bits (MSB) group.
  • What's more annoying is that our physical wiring scheme means that we can't uniformly decide on how this division into LSBs and MSBs work for all the channels - e.g. for REFL1 Gain, the LSB is the 4 least significant bits, while the MSB is the 2 most significant ones, while for REFL2 Gain, the roles are reversed.
  • In hindsight, the "clever" way to do the wiring assignment would have been to factor this in - but the problem is (sort of) easily fixed in software, and so I recommend we stick with the existing wiring scheme.

I tested the new latch.py script by toggling the various sliders (one at a time) between two values and monitoring the states of the various soft and "*_BITS" channels, see Attachment #1. The behavior seems consistent to me, but to be sure, we have to use Koji's LED tester board and confirm that the physical bits are being toggled correctly. The StripTool templates live in /cvs/cds/caltech/target/c1iscaux/CMdiag.


I have not yet implemented the fix for the MBBO gain channels for all the gains - only REFL1_GAIN is set up correctly now. Need to look at the hardware for the correct addressing of bits

Attachment 1: CMsoftTest.png
  15423   Mon Jun 22 17:51:50 2020 gautamUpdateCDSc1iscaux was down

The machine needed a hard reboot as it was un-ssh-able. 

The exact time that the machine went down is unknown because the blinkys were not DQ-ed. I've now added these to the EDCU to make these channels actually useful, and we may look back on the reliability (or otherwise) of the Acromag system. To my memory, this is the ~5th time one of the new Acromag servers has needed a hard reboot. While this may be less frequent (?) than the VME machines, perhaps there is some other reason for these dropouts. Maybe something to do with the martian network?

Anyway the machine is back up and running now.

  4570   Tue Apr 26 22:56:01 2011 kiwamuUpdateLSCc1iscaux2 and c1iscaux restrated

While checking whitening filters on the LSC rack, I found some epics controls for the whitening looked not working.

So I powered two crates off : the top one and the bottom one on 1Y3 rack.

These crates contain c1iscaux and c1iscaux2. Then powered them on. But it didn't solve the issue.

  5992   Wed Nov 23 22:58:33 2011 KojiUpdateGeneralc1iscaux2 is back (re: recovery from the power shutdown)

Keying again did not help to solve the issue. I turned off the power at the back of the crate, and turn it on again.
Then the key worked again.

c1iscaux2 is burtrestored and running fine now.


 - One of the VME rack on 1X3 is not showing the +/-15V green LED lights.

   This is the one on very upper side of the rack, which contains the old c1lsc machine and c1iscaux2. If we are still using c1iscaux2, it needs to be fixed.


  6598   Thu May 3 17:15:38 2012 KojiUpdateLSCc1iscaux2 rebooted/burtrestored


We saw some white boxes on the LSC screens.
We found c1iscaux2 is not running.

Once the target was power-cycled, these epics channels are back.
Then c1iscaux2 were burtrestored using the snapshot at 5:07 on 4/16, a day before the power glitch.

  5979   Tue Nov 22 18:15:39 2011 jamieUpdateCDSc1iscex ADC found dead. Replaced, c1iscex working again

c1iscex has not been running for a couple of days (since the power shutdown at least). I was assuming that the problem was recurrence of the c1iscex IO chassis issue from a couple weeks ago (5854).  However, upon investigation I found that the timing signals were all fine.  Instead, the IOP was reporting that it was finding now ADC, even though there is one in the chassis.

Since I had a spare ADC that was going to be used for the CyMAC, I decided to try swapping it out to see if that helped.  Sure enough, the system came up fine with the new ADC.  The IOP (c1x01) and c1scx are now both running fine.

I assume the issue before might have been caused by a failing and flaky ADC, which has now failed.  We'll need to get a new ADC for me to give back to the CyMAC.

  8088   Fri Feb 15 15:21:07 2013 JamieUpdateComputersc1iscex IO-chassis dead

I appears that the c1iscex IO-chassis is either dead or in a very bad state.  The PCIe interface card in the IO-chassis is showing four red lights, where it's supposed to be showing a dozen or so green lights.  Obviously this is going to prevent anything from running.

We've had power issues with this chassis before, so possibly that's what we're running into now.  I'll pull the chassis and diagnose asap.


  8109   Tue Feb 19 15:10:02 2013 JamieUpdateCDSc1iscex alive again

c1iscex is back up.  It is communicating with it's IO chassis, and all of it's models (c1x01, c1scx, c1spx) are running again.

The problem was that the IO chassis had no connection to the computer.  The One Stop card in the IO chassis, which is the PCIe bridge from the front-end machine and the IO chassis, was showing four red lights instead of the dozen or so green lights that it usually shows.  Upon closer inspection, the card appeared to be complaining that it had no connection to the host card in the front-end machine.  Un-illuminated lights on the host card seemed to be pointing to the same thing.

There are two connector slots on the expansion card, presumably for a daisy chain situation.  Looking at other IO chassis in the lab I determined that the cable from the front-end machine was plugged into the wrong slot in the One Stop card.  wtf.

Did someone unplug the cable connecting c1iscex to it's IO chassis, and then replug it in in the wrong slot?  A human must have done this.

  4179   Thu Jan 20 18:20:55 2011 josephbUpdateCDSc1iscex computer and c1sus computer swapped

Since the 1U sized computers don't have enough slots to hold the host interface board, RFM card, and a dolphin card, we had to move the 2U computer from the end to middle to replace c1sus.

We're hoping this will reduce the time associated with reads off the RFM card compared to when its in the IO chassis.  Previous experience on c1ioo shows this change provides about a factor of 2 improvement, with 8 microseconds per read dropping to 4 microseconds per read, per this elog.

So the dolphin card was moved into the 2U chassis, as well as the RFM card.  I had to swap the PMC to PCI adapter on the RFM card since the one originally on it required an external power connection, which the computer doesn't provide.  So I swapped with one of the DAC cards in the c1sus IO chassis.

But then I forgot to hit submit on this elog entry..............

  4478   Thu Mar 31 19:58:11 2011 kiwamuUpdateCDSc1iscex crashed

After I did several things to add new DAQ channels on c1iscex it suddenly became out of network. Maybe crashed.

Then c1iscex didn't respond to a ping and all the epics values associated with c1iscex became not accessible.

I physically shut it down by pushing the reset button. Then it came back and is now running fine.


(how I broke it)

Since activateDAQ.py has screwed up the 'ini' files including C1SCX.ini, I was not able to add a channel to C1SCX.ini by the usual daqconfig GUI.

So I started editing it in a manual way with an editor and changed some sentences to that shown below


Then I rebooted fb to reflect the new DAQ channels.

After that I looked at the C1_FE_STATUS.adl screen and found some indicator lights were red.

So I pushed "Diag reset" button and "DAQ Reload" button on the C1SCX_GDS_TP.adl screen and then c1iscex died.

After the reboot the new DAQ channels looked acquired happily.

This is my second time to crash a front end machine (see this entry)

  8126   Thu Feb 21 12:56:38 2013 JenneUpdateCDSc1iscex dead again

c1iscex is dead again.  Red lights, no "breathing" on the FE status screen.

  6194   Thu Jan 12 23:19:56 2012 KojiUpdateSUSc1iscex is fine now

c1iscex is working as before and the optic is damped.

What I checked

1. I went to the X-end rack. I found the io-chassis was turned off.

2. I shutdown c1iscex, turned off, and turned on everything. Again, we did not have any signal from the ADC into c1scx model.

However, I found that c1x01 indicates healthy ADC signals.
This means that the connection between the IOP and the c1scx model was wrong ==> Simulated Plant

3. Burtrestored X'mas eve snapshot. This restored the gains and matrices as well as C1:SCX-SIM_SWITCH
which switches the input between the real ADCs and simulated plant.

4. The signals came back to c1scx.

  3965   Mon Nov 22 17:48:11 2010 josephbUpdateCDSc1iscex is not seeing its Binary Output card


c1iscex does not even see its 32 channel Binary output card.  This means we have no control over the state of the analog whitening and dewhitening filters.  The ADC, DAC, and the 1616 Binary Input/Output cards are recognized and working.

Things tried:

Tried recreating the IOP code from the known working c1x02 (from the c1sus front end), but that didn't help.

Checked seating of the card, but it seems correctly socketed and tightened down nicely with a screw.

Tomorrow will try moving cards around and see if there's an issue with the first slot, which the Binary Output card is in.

Current Status:

The ETMX is currently damping, including POS, PIT, YAW and SIDE degrees of freedom.  However, the gds screen is showing a 0x2bad status for the c1scx front end (the IOP seems fine with a 0x0 status).  So for the moment, I can't seem to bring up c1scx testpoints.  I was able to do so earlier when I was testing the status of the binary outputs, so during one of the rebuilds, something broke. I may have to undo the SVN update and/or a change made by Alex today to allow for longer filter bank names beyond 19 characters.

  3926   Mon Nov 15 16:26:46 2010 josephbUpdateCDSc1iscex is now running and the network hasn't died


c1iscex was spamming the network with error messages.


Updated the front end codes to current standards (they were on the order of months out of date).  After fixing them up and rebuilding the codes on c1iscex, it no longer had problems connecting to the frame builder.\


I can look at test points for ETMX.  It is not currently damping however.

To Do:

Move filters for ETMX into the correct files. 

Need to add a Binary output blue and gold box to the end rack, and plug it into the binary output card.  Confirm the binary output logic is correct for the OSEM whitening, coil dewhitening, and QPD whitening boards. 

Get ETMX damped.

Figure out what we're going to do with the aux crate which is currently running y-end code at the new x-end.  Koji suggested simply swapping auxilliary crates - this may be the easiest.  Other option would be to change the IP address, so that when it PXE boots it grabs the x-end code instead of the y-end code.

Current CDS status:

MC damp dataviewer diaggui AWG c1ioo c1sus c1iscex RFM Sim.Plant Frame builder TDS
  13242   Tue Aug 22 17:11:15 2017 gautamUpdateComputersc1iscex model restarts

[jamie, gautam]

We tried to implement the fix that Rolf suggested in order to solve (perhaps among other things) the inability of some utilities like dataviewer to open testpoints. The problem isn't wholly solved yet - we can access actual testpoint data (not just zeros, as was the case) using DTT, and if DTT is used to open a testpoint first, then dataviewer, but DV itself can't seem to open testpoints.

Here is what was done (Jamie will correct me if I am mistaken).

  1. Jamie checked out branch 3.4 of the RCG from the SVN.
  2. Jamie recompiled all the models on c1iscex against this version of RCG.
  3. I shutdown ETMX watchdog, then ran rtcds stop all on c1iscex to stop all the models, and then restarted them using rtcds start <model> in the order c1x01, c1scx and c1asx. 
  4. Models came back up cleanly. I then restarted the daqd_dc process on FB1. At this point all indicators on the CDS overview screen were green.
  5. Tried getting testpoint data with DTT and DV for ETMX Oplev Pitch and Yaw IN1 testpoints. Conclusion as above.

So while we are in a better state now, the problem isn't fully solved. 

Comment: seems like there is an in-built timeout for testpoints opened with DTT - if the measurement is inactive for some time (unsure how much exactly but something like 5mins), the testpoint is automatically closed.

  13135   Mon Jul 24 10:45:23 2017 gautamUpdateCDSc1iscex models died

This morning, all the c1iscex models were dead. Attachment #1 shows the state of the cds overview screen when I came in. The machine itself was ssh-able, so I just restarted all the models and they came back online without fuss.


All front ends and model are (mostly) running now

Attachment 1: c1iscexFailure.png
  13136   Mon Jul 24 10:59:08 2017 JamieUpdateCDSc1iscex models died

This morning, all the c1iscex models were dead. Attachment #1 shows the state of the cds overview screen when I came in. The machine itself was ssh-able, so I just restarted all the models and they came back online without fuss.

This was me.  I had rebooted that machine and hadn't restarted the models.  Sorry for the confusion.

  8128   Thu Feb 21 14:32:02 2013 JamieUpdateCDSc1iscex models restarted


c1iscex is dead again.  Red lights, no "breathing" on the FE status screen.

The c1iscex machine itself wasn't dead, the models were just not running.  Here are the last messages in dmesg:

[130432.926002] c1spx: ADC TIMEOUT 0 7060 20 7124
[130432.926002] c1scx: ADC TIMEOUT 0 7060 20 7124
[130433.941008] c1x01: timeout 0 1000000 
[130433.941008] c1x01: exiting from fe_code()

I'm guessing maybe the timing signal was lost, so the ADC stopped clocking.   Since the ADC clock is the everything clock, all the "fe" code (ie. models) aborted. Not sure what would have caused it.

I restarted all the models ("rtcds restart all") and everything came up fine. Obviously we should keep our eyes on things, and note if anything strange was happening if this happens again.

  5060   Fri Jul 29 12:39:26 2011 jamieUpdateCDSc1iscex mysteriously crashed

c1iscex was behaving very strangely this morning.  Steve earlier reported that he was having trouble pulling up some channels from the c1scx model.  I went to investigate and noticed that indeed some channels were not responding.

While I was in the middle of poking around, c1iscex stopped responding altogether, and became completely unresponsive.  I walked down there and did a hard reset.  Once it rebooted, and I did a burt restore from early this morning, everything appeared to be working again.

The fact that problems were showing up before the machine crashed worries me.  I'll try to investigate more this afternoon.

  9000   Mon Aug 12 21:27:03 2013 manasaUpdateCDSc1iscex needs help

I started to modify the c1asx model to reduce the RFM model from hitting its max time.
Instead of bringing in ASS, I have modified ASX to do everything and only the clock signals to ITMX pitch and yaw are now going through RFM. RFM is still hitting 62usec and I suppose that is because of the problems with c1iscex.

c1iscex not happy

Cause and symptoms

While restarting the models, c1iscex crashed a couple of times because of some errors and had to be powercycled. The models were modified and they seem to start ok.
But it looks like there is something wrong with c1iscex since the models were started. The GPS time is off and C1:DAQ-DC0_C1X01_CRC_SUM keeps building up even for c1x01 which was left untouched.

Trial treatments

1. Since c1x01 ans c1spx were not touched,c1scx and c1asx were killed and we tried to start the other models. This did not help.
2. Koji did a manual daqd restart which did not help either.

We are leaving c1iscex as is for the time being and calling Jamie for help.

P.S. While making the models, I had created IPCx_PCIE blocks in c1iscex which do not exist. I changed them to RFM and SHMEM blocks. This did not allow me to compile the model and was only spitting errors of IPCx mismatch. After some struggle and elog search I figured out from an old elog that eventhough the IPCx blocks are changed in the model, the old junk exists in the ipc file in chans directory. I deleted all junk channels related to the ASX model. The model compiled right away.

  9002   Tue Aug 13 07:40:53 2013 SteveUpdateCDSc1iscex needs help


 Sorrensen ps ouput of +15V at rack 1X9 was current limited to 10.3V @ 2A

Increased threshold to 2.1A  and the voltage is up to 14.7V

Attachment 1: c1iscexSick.png
  1126   Mon Nov 10 11:32:49 2008 robUpdateComputersc1iscex rebooted

it was running a few cycles late
  10642   Mon Oct 27 22:19:17 2014 JenneUpdateCDSc1iscex restarted

I'm not sure why, but c1iscex did not want to do an mxstream restart.  It would complain at me that  "* ERROR:  mx_stream is already stopping."
Koji suggested that I reboot the machine, so I did.  I turned off the ETMX watchdog, and then did a remote reboot.  Everything came back nicely, and the mx_stream process seems to be running.

  4020   Tue Dec 7 16:09:53 2010 josephbUpdateCDSc1iscex status

I swapped out the IO chassis which could only handle 3 PCIe cards with the another chassis which has space for 17, but which previously had timing issues.  A new cable going between the timing slave and the rear board seems to have fixed the timing issues. 

I'm hoping to get a replacement PCI extension board which can handle more than 3 cards this week from Rolf and then eventually put it in the Y-end rack.  I'm also still waiting for a repaired Host interface board to come in for that as well.

At this point, RFM is working to c1iscex, but I'm still debugging the binary outputs to the analog filters.  As of this time they are not working properly (turning the digital filters on and off seems to have no effect on the transfer function measured from an excitation in SUSPOS, all the way around to IN1 of the sensor inputs (but before measuring the digital fitlers).  Ideally I should see a difference when I switch the digital filters on and off (since the analog ones should also switch on and off), but I do not.

  7963   Wed Jan 30 13:50:27 2013 JenneUpdateComputersc1iscex still down

[Koji, Jenne]

We noticed that the iscex computer is still down, but the IOP is (was) running.  When we sat down to look at it, c1x01 was 'breathing', had a non-zero CPU_METER time, and the error was 0x4000, which I've never seen before.  The fb connection was still red though.  Also, it is claiming that its sync source is 1pps, not TDS like it usually is. 

Since things were different, Koji restarted the 2 other models running on iscex, with no resulting change.  We then did a 'rtcds restart all', and the IOP is no longer breathing, and the error message has changed to 0xbad.  The sync source is still 1pps.

Moral of the story:  c1iscex is still down, but temporarily showed signs of life that we wanted to record.

  7970   Thu Jan 31 10:23:39 2013 JamieUpdateComputersc1iscex still down


[Koji, Jenne]

We noticed that the iscex computer is still down, but the IOP is (was) running.  When we sat down to look at it, c1x01 was 'breathing', had a non-zero CPU_METER time, and the error was 0x4000, which I've never seen before.  The fb connection was still red though.  Also, it is claiming that its sync source is 1pps, not TDS like it usually is. 

Since things were different, Koji restarted the 2 other models running on iscex, with no resulting change.  We then did a 'rtcds restart all', and the IOP is no longer breathing, and the error message has changed to 0xbad.  The sync source is still 1pps.

Moral of the story:  c1iscex is still down, but temporarily showed signs of life that we wanted to record.

There's definitely a timing issue with this machine.  I looked at it a bit yesterday.  I'll try to get to it by the end of the week.

  9433   Mon Dec 2 16:04:47 2013 JamieUpdateCDSc1iscex timing problem mysteriously disappears??? (thanksgiving miracle???)


There is definitely a timing distribution malfunction at the c1iscex IO chassis.  There is no timing link between the "Master Timer Sequencer D050239" at the 1X6 and the c1iscex IO chassis.  Link lights at both ends are dead.  No timing, no running models.

It does not appear to be a problem with the Master Timer Sequencer.  I moved the c1iscey link to the J15 port on the sequencer and it worked fine.  This means its either a problem with the fiber or the timing card in the IO chassis.  The IO timing card is powered and does have what appear to be normal status lights on (except for the fiber link lights).  It's getting what I think is the nominal 4V power.  The connection to the IO chassis backplane board look ok.  So maybe it's just a dead fiber issue?

I do not know what could have been the problem with c1auxex, or if it's related to the fast timing issue.

I just got over here from Downs, where I managed to convince Todd to let me borrow one of their three remaining timing slave boards for c1iscex.  I walked down to the X end to replace the board only to discover that the link light on the existing timing board was back!  c1iscex was not responding, so I hard rebooted the machine, and everything came up rosy (all green!):


To repeat, I DID NOTHING.  The thing was working when I got here.  I have no idea when it came back, or how, but it's at least working for the moment.  I re-enabled the watchdog for ETMX SUS and it's now damped normally.

I'm going to hold on to the timing card for a couple of days, in case the failure comes back, but we'll need to return it to Downs soon, and probably think about getting some spare backups from Columbia.

  9434   Mon Dec 2 17:05:13 2013 JenneUpdateCDSc1iscex timing problem mysteriously disappears??? (thanksgiving miracle???)

Steve was trying to do something to it this morning, but I'm not exactly clear on what it was.  Maybe that helped?  Steve, can you tell us what you were trying to do this morning?

  9435   Tue Dec 3 07:42:23 2013 SteveUpdateCDSc1iscex timing problem mysteriously disappears??? (thanksgiving miracle???)


Steve was trying to do something to it this morning, but I'm not exactly clear on what it was.  Maybe that helped?  Steve, can you tell us what you were trying to do this morning?

 I was trying to repeat  elog 9007  I did only get to line 2 of the Solution by Koji when Ottavia shut down, where I was working. This was all what I did.

  13079   Sun Jun 25 22:30:57 2017 gautamUpdateGeneralc1iscex timing troubles

I saw that the CDS overview screen indicated problems with c1iscex (also ETMX was erratic). I took a closer look and thought it might be a timing issue - a walk to the X-end confirmed this, the 1pps status light on the timing slave card was no longer blinking. 

I tried all versions of power cycling and debugging this problem known to me, including those suggested in this thread and from a more recent time. I am leaving things as it for the night, will look into this more tomorrow. I've also shutdown the ETMX watchdog for the time being. Looks like this has been down since 24Jun 8am UTC.

Attachment 1: c1iscex_status.png
  13081   Mon Jun 26 22:01:08 2017 KojiUpdateGeneralc1iscex timing troubles

I tried a couple of things, but no fundamental improvement of the missing LED light on the timing board.

- The power supply cable to the timing board at c1iscex indicated +12.3V

- I swapped the timing fiber to the new one (orange) in the digital cabinet. It didn't help.

- I swapped the opto-electronic I/F for the timing fiber with the Y-end one. The X-end one worked at Y-end, and Y-end one didn't work at X-end.

- I suspected the timing board itself -> I brought a "spare" timing board from the digital cabinet and tried to swap the board. This didn't help.


Some ideas:

- Bring the X-end fiber to C1SUS or C1IOO to see if the fiber is OK or not.

- We checked the opto-electronic I/F is OK

- Try to swap the IO chassis with the Y-end one.

- If this helps, swap the timing board only to see this is the problem or not.

  13085   Wed Jun 28 20:15:46 2017 gautamUpdateGeneralc1iscex timing troubles

[Koji, gautam]

Here is a summary of what we did today to fix the timing issue on c1iscex. The power supply to the timing card in the X end expansion chassis was to blame.

  1. We prepared the Y-end expansion chassis for transport to the X end. To do so, we disconnected the following from the expansion chassis
    • Cables going to the ADC/DAC adaptor boards
    • Dolphin connector
    • BIO connector
    • RFM fiber
    • Timing fiber
  2. We then carried the expansion chassis to the X end electronics rack. There we repeated the above steps for the X-end expansion chassis
  3. We swapped the X and Y end expansion chassis in the X end electronics rack. Powering the unit, we immediately saw the green lights on the front of the timing card turn on, suggesting that the Y-end expansion chassis works fine at the X end as well (as it should). To further confirm that all was well, we were able to successfully start all the RT models on c1iscex without running into any timing issues.
  4. Next, we decided to verify if the spare timing card is functional. So we swapped out the timing card in the expansion chassis brought over to the X end from the Y end with the spare. In this test too, all worked as expected. So at this stage, we concluded that
    • There was nothing wrong with the fiber bringing the timing signal to the X end
    • The Y-end expansion chassis works fine
    • The spare timing card works fine.
  5. Then we decided to try the original X-end expansion chassis timing card in the Y-end expansion chassis. This test too was successful - so there was nothing wrong with any of the timing card!
  6. Next, we decided to power the X-end timing chassis with its original timing card, which was just verified to work fine. Surprisingly, the indicator lights on the timing card did not turn on.
  7. The timing card has 3 external connections
    • A 40 pin IDE connector
    • Power
    • Fiber carrying the timing signal
  8. We went back to the Y-end expansion chassis, and checked that the indicator lights on the timing card turned on even when the 40 pin IDE connector was left unconnected (so the timing card just gets power and the timing signal).
  9. We concluded that the power supply in the X end expansion chassis was to blame. Indeed, when Koji jiggled the connector around a little, the indicator lights came on!
  10. The connection was diagnosed to be somewhat flaky - it employs the screw-in variety of terminal blocks, and one of the connections was quite loose - Koji was able to pull the cable out of the slot applying a little pressure.
  11. I replaced the cabling (swapped the wires for thicker gauge, more flexible variety), and re-tightened the terminal block screws. The connection was reasonably secure even when I applied some force. A quick test verified that the timing card was functional when the unit was powered.
  12. We then replaced the X and Y-end expansion chassis (complete with their original timing cards, so the spare is back in the CDS cabinet), in the racks. The models started up again without complaint, and the CDS overview screen is now in a good state [Attachment #1]. The arms are locked and aligned for maximum transmission now.
  13. There was some additional difficulty in getting the 40-pin IDE connector in on the Y-end expansion chassis. Looked like we had bent some of the pins on the timing board while pulling this cable out. But Koji was able to fix this with a screw driver. Care should be taken when disconnecting this cable in the future!

There were a few more flaky things in the Expansion chassis - the IDE connectors don't have "keys" that fix the orientation they should go in, and the whole timing card assembly is kind of difficult and not exactly secure. But for now, things are back to normal it seems.

Wouldn't it be nice if this fix also eliminates the mystery ETMX glitching problem? After all, seems like this flaky power supply has been a problem for a number of years. Let's keep an eye out.

Attachment 1: CDS_status_28Jun2017.png
ELOG V3.1.3-