Do we care about the AC? I thought what we care is the DC.

[Suresh / Koji / Rana / Kiwamu]

Last night we had a discussion about what we do for the RFAM issue. Here is the plan.

(PLAN)

  1. Build and install an RFAM monitor (a.k.a StochMon ) with a combination of a power splitter, band-pass-filters and Wenzel RMS detectors.

       => Some ordering has started (#5682). The Wenzel RMS detectors are already in hands.

  2. Install a temperature sensor on the EOM. And if possible install it with a new EOM resonant box.

      => make a wheatstone bridge circuit, whose voltage is modulated with a local oscillator at 100 Hz or so.

  3. Install a broadband RFPD to monitor the RFAMs and connect it to the StochMon network.

      => Koji's broadband PD or a commercial RFPD (e.g. Newfocus 1811 or similar)

  4. Measure the response of the amount of the RFAM versus the temperature of the EO crystal.

      => to see whether if stabilizing the temperature stabilizes the RFAM or not.

  5.  Measure the long-term behavior of the RFAM.

      => to estimate the worst amount of the RFAM and the time scale of its variation

  6. Decide which physical quantity we will stabilize, the temperature or the amount of the RFAM.

  7. Implement a digital servo to stabilize the RFAMs by feeding signals back to a heater

     => we need to install a heater on the EOM.

  8. In parallel to those actions, figure out how much offsets each LSC error signal will have due to the current amount of the RFAMs.

    => Optickle simulations.

  9. Set some criteria on the allowed amount of the RFAMs

    => With some given offsets in the LSC error signal, we investigate what kind of (bad) effects we will have.

The following channels have been registered in c1iool0 database, and are now recorded by FB

C1:IOO-RFAMPD_11MHZ
C1:IOO-RFAMPD_29_5MHZ
C1:IOO-RFAMPD_55MHZ
C1:IOO-RFAMPD_DCMON
C1:IOO-EOM_TEMPMON
C1:IOO-EOM_HEATER_DRIVEMON

PROCEDURE

1) The EPICS database file has been edited to rename/add some channels

/cvs/cds/caltech/target/c1iool0/ioo.db

REMOVED
#grecord(ao,"C1:IOO-RFAMPD_VC")
#grecord(ai,"C1:IOO-RFAMPD_TEMP")
#grecord(ai,"C1:IOO-RFAMPD_DCMON")
#grecord(bo,"C1:IOO-RFAMPD_BIAS_ENABLE")
#grecord(bi,"C1:IOO-RFAMPD_BIAS_STATUS")
#grecord(calc, "C1:IOO-RFAMPD_33MHZ_CAL")
#grecord(calc, "C1:IOO-RFAMPD_133MHZ_CAL")
#grecord(calc, "C1:IOO-RFAMPD_166MHZ_CAL")
#grecord(calc, "C1:IOO-RFAMPD_199MHZ_CAL")


ADDED/EDITED
grecord(ai,"C1:IOO-RFAMPD_11MHZ")
        field(DTYP,"VMIVME-3113")                                              
        field(INP,"#C1 S25 @")
...
grecord(ai,"C1:IOO-RFAMPD_29_5MHZ")
        field(DTYP,"VMIVME-3113")                                              
        field(INP,"#C1 S26 @")
...
grecord(ai,"C1:IOO-RFAMPD_55MHZ")
        field(DTYP,"VMIVME-3113")                                              
        field(INP,"#C1 S27 @")
...
grecord(ai,"C1:IOO-RFAMPD_DCMON")
        field(DTYP,"VMIVME-3113")                                              
        field(INP,"#C1 S28 @")
...
grecord(ai,"C1:IOO-EOM_TEMPMON")                                                
        field(DTYP,"VMIVME-3113")                                               
        field(INP,"#C1 S29 @")
...
grecord(ai,"C1:IOO-EOM_HEATER_DRIVEMON")
        field(DTYP,"VMIVME-3113")                                              
        field(INP,"#C1 S30 @")

2) The channels have been added to the frame builder database

/cvs/cds/rtcds/caltech/c1/chans/daq/C0EDCU.ini

[C1:IOO-RFAMPD_11MHZ]
[C1:IOO-RFAMPD_29_5MHZ]
[C1:IOO-RFAMPD_55MHZ]
[C1:IOO-RFAMPD_DCMON]
[C1:IOO-EOM_TEMPMON]
[C1:IOO-EOM_HEATER_DRIVEMON]

Note that this C0EDCU.ini is the file that has been registered in

/cvs/cds/rtcds/caltech/c1/target/fb/master

3) burt restore request files were updated

RFAM related settings were removed as they don't exist anymore.

/cvs/cds/caltech/target/c1iool0/autoBurt.req
/cvs/cds/caltech/target/c1iool0/saverestore.req

4) c1iool0 were rebooted. Framebuilder restarted. c1iool0 were burtrestored.

EOM TEMPMON and HEATER DRIVEMON have been hooked up to the following channels.

C1:IOO-EOM_TEMPMON
C1:IOO-EOM_HEATER_DRIVEMON 

What a fragile circuit...

I found some of the resistors popped up from the board because of the tension by the Pomona grabbers.
I tried to fix it based on the schematic (photo) and the board photo.

Here is a drawing of where the monitors are coming from:

 Since we can't put current into the ADC, the heater drivemon is measuring the input of the OP27, which is related to the amount of current sent to the heater.

Jenne: The point you indicate for the heater monitor is a virtual ground--it will be driven to zero by the circuit if it's functioning properly; the readout should be done at the input pin (2, I think) to the BUF634.

Koji: This is odd, as I made a point of not attaching any clips directly to resistors for exactly this reason. I was also careful to trim resistor/capacitor leads so that they were not towering over the breadboard and prone to bending (with the exception of the gain-setting resistor of the AD620, which was changed at the last minute). At the end of the day, it is a breadboard circuit with Pomona "readout", so it's not going to be truly resilient until I put it on a protoboard. Another thing: I think the small Pomona clips are absolutely terrible, since they slip off with piconewtons of tension; I could not find any more regular clips, so I used them against my better judgment.

Those clips for the readouts were the ones who popped out.
When I have restored the connections, I checked the schematic and the heater drive mon is clipped on the output side of the OP27.

I added PCIE memory cache flushing to c1rfm model by changing 0 to 1 in /opt/rtcds/rtscore/release/src/fe/commData2.c on line 159, recompiled and restarted c1rfm.

Jamie, do not be mad at me, Alex told me do that!

However, this did not help, C1RFM did not start. I decided to restart all models on C1SUS machine in hope that C1RFM uses some other models and can't connect to them but this suspended C1SUS machine. After reboot encounted the same C1SUS -> FB communication error and fixed it in the same was as in the previous case of C1SUS reboot. This happens already the second time (out of 2) after C1SUS machine reboot.

I changed /opt/rtcds/rtscore/release/src/fe/commData2.c back, recompiled and restarted c1rfm. Now everything is back. C1RFM -> C1OAF is still bad.

 This happened because of the code bug -

After this bug was fixed and the code was recompiled, C1:OAF_MCL_IN is OK, no errors occur during the transmission C1:OAF-MCL_ERR=0.

So the problem was in the PCIE card that could not send such amount of data and the last channel (MCL is the last) was corrupted. Now, when Alex added cache flushing, the problem is fixed.

We should spend some more attention to such problems. This time 2046 out of 2048 points were lost per second. But what if 10-20 points are lost, we would not notice that in the dataviewer, but this will cause problems.

The problem here was that the RFM errors cropped up again - seems like it started ~4am today morning judging by TRX trends. Of course without the triggering signal the arm cavity couldn't lock. I rebooted everything (since just restarting the rfm senders/receivers did not do the trick), now arm locking works fine again. It's a bit disappointing that the Rogue Master setting did not eliminate this problem completely, but oh well...

It's kind of cool that in this trend view of the TRX signal, you can see the drift of the ETMX suspension. The days are getting hot again and the temp at EX can fluctuate by >12C between day and night (so the "air-conditioning" doesn't condition that much I guess 😂 ), and I think that's what drives the drift (idk what the transfer function to the inside of the vacuum chamber is but such a large swing isn't great in any case). Not plotted here but i hypothesize TRY levels will be more constant over the day (modulo TT drift which affects both arms).

The IMC suspension team should double check their filters are on again. I am not familiar with the settings and I don't think they've been added to the SDF.

The other day I felt hot at the X end. I wondered if the Xend A/C was off, but the switch right next to the SP table was ON (green light).
I could not confirm if the A/C was actually blowing or not.

Transmission of signals between RFM and OAF is bad again. Now we do not see any errors in IPC_ERR monitors so models think that they get all data but the data is wrong

This morning I found that all the front end computers down. A failure of the RFM network drove all the computers down.

I was about to restart them all, but it wasn't necessary. After I power cycled and restarted C1SOSVME all the other computers and RFM network came back to their green status on the MEDM screen. After that I just had to reset and then restart C1SUSVME1/2.

I worked on the mitigation of c1mcs time-over issue this afternoon.

The timing for the c1mcs is successfully reduced from >60us to 45us.

The previous models are svned in redoubt as follows:

MCS rev. 6696
RFM rev. 6697
IOO rev. 6698

What I changed was:

- Remove connection from ALS (on c1ioo) to MCS (on c1sus). This should be all done in LSC. (# of RFM IPC in MCS -1)

- MC2 trans QPD filters are moved from IOO to MCS to reduce the RFM channels in MCS.
  Previously the signals for the 4 segments are sent. Now the processed siganls (pit/yaw/sum) are sent. (# of RFM IPC in IOO -1, MCS -1)

- WFS MC3 feedback channels are moved from MCS to RFM to distribute the RFM channels (# of RFM IPC in MCS -2, in RFM +2)

model    prev. timing[us] current timing[us]  diff in time[us]  diff in ch#
c1mcs         >60                45                -15              -4
c1rfm         47                 53                + 6              +2       
c1ioo         47                 36                -11              -1

Revisions of the new models:
MCS rev. 6702
RFM rev. 6701
IOO rev. 6700

I took the RFM 5565 card dropped off by Jay and installed it into megatron.  It is not very secure, as it was too tall for the slot and could not be locked down.  I did not connect the RFM fibers at this point, so just the card is plugged in.

Unfortunately, on power up, and immediately after the splash screen I get "NMI EVENT!" and "System halted due to fatal NMI". 

The status light on the RFM light remains a steady red as well.  There is a distinct possibility the card is broken in some way.

The card is a VMIPMC-5565 (which is the same as the card used by the ETMY front end machine).  We should get Alex to come in and look at it on Monday, but we may need to get a replacement.

Last night, we installed the VMI 5565 RFM card into Megatron.  After turning off the watchdogs for the ETMY optic, we disconnected the RFM fiber, and connected it to megatron, then powered it up. 

We modified the RCG code to have 3 rfmio blocks, which were reading 0x11a1c0 (ascPit), 0x11a1c4 (ascYaw), and 0x11a1c8 (lscPos).  These were connected to the approriate filter module inputs, and we also added grounds to the front of the rfmio blocks (we looked at the ass code which was setup that way, so we just did the same thing).  When we started it however, it didn't read properly.  If we turned off the input and set and offset, it calculated the output of the filter module correctly, (i.e. just the offset value), but as soon as we turned on the input, it was set to 0, no matter the offset value, which indicated it was reading something correctly.

After this test, the RFM fibers were reconnected to c1iscey, we rebooted c1iscey, and we confirmed that the system was working properly again.  We turned the watch dogs back on for ETMY.

Alex came over with a short RFM cable this morning.  We used it to connect the rfm card in c1iscey to the rfm card megatron

Alex renamed startup.cmd in /cvs/cds/caltech/target/c1iscey/ to startup.cmd.sav, so it doesn't come up automatically.  At the end we moved it back.

Alex used the vxworks command d to look at memory locations on c1iscey.  Such as d 0xf0000000, which is the start of the rfm code location.  So to look at 0x11a1c8 (lscPos) in the rfm memory, he typed "d 0xf011a1c8".  After doing some poking around, we look at the raw tst front end code (in /home/controls/cds/advLigo/src/fe/tst), and realized it was trying to read doubles.  The old rts code uses floats, so the code was reading incorrectly.

As a quick fix, we changed the code to floats for that part.  They looked like:

etmy_lsc = filterModuleD(dsp_ptr,dspCoeff,ETMY_LSC,cdsPciModules.pci_rfm[0]? *(\
(double *)(((void *)cdsPciModules.pci_rfm[0]) + 0x11a1c8)) : 0.0,0);

And we simply changed the double to float in each case.  In addition we changed the RCG scripts locally as well (if we do a update at some point, it'll get overwritten).  The file we updated was /home/controls/cds/advLigo/src/epics/util/lib/RfmIO.pm

Line 57 and Line 84 were changed, with double replaced with float.

return "cdsPciModules.pci_rfm[0]? *((float *)(((void *)cdsPciModules.pci
_rfm[$card_num]) + $rfmAddressString)) : 0.0";

. "  *((float *)(((char *)cdsPciModules.pci_rfm[$card_nu
m]) + $rfmAddressString)) = $::fromExp[0];\n"

This fixed our ability to read the RFM card, which now can read the LSC POS channel, for example.

Unfortunately, when we were putting everything the way it was with RFM fibers and so forth, the c1iscey started to get garbage (all the RFM memory locations were reading ffff).  We eventually removed the VME board, removed the RFM card, looked at it, put the RFM card back in a different slot on the board, and returned c1iscey to the rack.  After this it started working properly.  Its possible in all the plugging and unplugging that the card somehow had become loose.

The next step is to add all the channels that need to be read into the .mdl file, as well as testing and adding the channel which need to be written.

Alex added a new module to the RCG, for generating RFMIO using floats.  This has been commited to CVS.

Problem:

Currently the c1scy, c1mcs, and c1rfm models are reporting an error with receiving some data sent over the GE Fanuc Reflected memory cards.

To be more exact, the C1:SUS-ETMY_ALS signal from the c1gcv FE code on the c1ioo computer going too the Y end is not being received.  However, the C1:SUS-ETMY_LSC signal is.  So the physical RFM card seems to be working.

Similarly, the TRY signal is being sent correctly from the Y end computer.  The X end is working fine and receiving both LSC and ALS signals.

The c1mcs and c1rfm models also receive data from the c1ioo computer and reporting receiving errors.

Theory:

Because the RFM cards are transmitting and receiving at least some channels, I'm guessing there was changes made to the C1.ipc file, which defines the memory locations of these various channels on the RFM network, and that when a model was rebuilt, a different one using the previous IPC file was not, and thus one of the computer is going to the wrong place to either read or write data.

Tomorrow, I'm planning on the  following:

1) Clean out the C1.ipc file (/opt/rtcds/caltech/c1/chans/ipc/)

2) Rebuild all models

3) Run activate_daq.py script

4) Restart models via script

If this doesn't clear up the problem, I'll continue  to bug hunt.

This problem resurfaced, which I noticed when I couldn't get the single arm locks going.

The fix was NOT restarting the c1rfm model, which just brought the misery of all vertex FEs crashing and the usual dance to get everything back.

Restarting the sender models (i.e. c1scx and c1scy) seems to have done the trick though.

Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.

I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).

In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission. 

Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.

I'm starting the model restarts from remote. Then later I'll show up in the lab to do more hard resets.
==> It seems that the RFM errors are gone. Here are the steps.

1. Shutdown all the watchdogs
2. login to c1iscex. Shutdown all the realtime models: rtcds kill --all
3. login to c1iscey. Shutdown all the realtime models: rtcds kill --all
4. run scripts/cds/rebootC1LSC.sh on pianosa
5. reboot c1iscex
6. reboot c1isxey
7. Wait until all the machines/models are up by the script
8. restart c1iscex models
9. restart c1iscey models
10. some IPC errors are still visible on the CDS status screen. Lauch c1daf and c1oaf

As I was working on the IFO re-alignment just now, the rfm errors popped up again. I don't see any useful diagnostics on the web interface.

Do we want to take this opportunity to configure jumpers and set up the rogue master as Rolf suggested? Of course there's no guarantee that will fix anything, and may possibly make it impossible to recover the current state...

Most of the RFM went red this morning. I took the nuclear option and it seemed to be recovered.

We have too much crap in the rfm model.  CPU time for the rfm model is regularly above 60us, and sometimes in the mid-70's (but sometimes jumps down briefly to ~47us, which is where I think it "used" to sit, but I don't remember when I last thought about that number)

This is potentially causing lots of asynchronous grief.

I noticed yesterday evening that I wasn't able to engage the single arm locking servos - turned out that they weren't getting triggered, which in turn pointed me to the fact that the arm transmssion channels seemed dead. Poking around a little, I found that there was a red light on the CDS overview screen for c1rfm.

• The error seems to be in the receiving model only, i.e. c1rfm, all the sending models (e.g. c1scx) don't report any errors, at least on the CDS overview screen.
• Judging by dataviewer trending of the c1rfm status word, seems like this happened on Sunday morning, around 11am.
• I tried restarting both sender and receiver models, but error persists.
• I got no useful information from the dmesg logs of either c1sus (which runs c1rfm), or c1iscex (which runs c1scx).
• There are no physical red lights in the expansion chassis that I could see - in the past, when we have had some timing errors, this would be a signature.

Not sure how to debug further...

* Fix seems to be to restart the sender RFM models (c1scx, c1scy, c1asx, c1asy).

I wanted to lock the single arm POX/POY config to do some tests on the BeatMouth. But I was unable to.

• I tracked the problem down to the fact that the TRX and TRY triggers weren't getting piped correctly to the LSC model
• In fact, all RFM channels from the end machines were showing error rates of 16384/sec (i.e. every sample).
• After watchdogging ETMX, I tried restarting just the c1scx model - this promptly took down the whole c1iscex machine.
• Then I tried the same with c1iscey - this time the models restarted successfully without the c1iscey machine crashing, but the RFM errors persisted for the c1scy channels.
• I walked down to EX and hard rebooted c1iscex.
• c1iscex came back online, and I ssh-ed in and did rtcds start --all.
• This brought all the models back online, and the RFM errors on both c1iscex and c1iscey channels vanished.

Not sure what to make of all this, but I can lock the arms now.

[Joe, Alex]

Problem Symptoms:

There were red lights on the status screen indicating RFM errors for the c1scy, c1mcs and c1rfm processes.

The c1iscey, c1sus machines were receiving data sent over the RFM network from the c1ioo computer with a bad time stamp, a few cycles too late.  The c1iscex computer was receiving data from c1ioo fine.

Problem:

The c1iscex RFM card had gotten into a bad state and was somehow slowing things down/corrupting data.  It didn't affect itself, but due to the loop topology was messing everyone else up.  Basically the only one who wasn't throwing an error was the culprit.

Solution:

Hard power cycling the c1iscex computer reset the RFM card and fixed the problem.

As noted by Steve, the RFM network was down this morning.  I noticed that c1susvme1 sync counter was pegged at 16384, so I decided to start with reboots in that viscinity.

After power cycling crates containing c1sosvme, c1susvme1, and c1susvme2 (since the reset buttons didn't work) only c1sosvme and c1susvme2 came back normally.  I hooked up a monitor and keyboard to c1susvme1, but saw nothing.  I power cycled the c1susvme crate again, and this time I watched it boot properly.  I'm not sure why it failed the first time.

The RFM network is now operating normally.  I have re-enabled the watchdogs again after having turned them off for the reboots.  Steve and I also re-enabled the ITMY coil drivers when I noticed them not damping once the watch dogs were re-enabled.  The manual switches had been set to disabled, so we re-enabled them.

I am not so happy with the control signals that are coming into the OAF via the RFM/Dolphin/shmem. 

The MCL/MCF signal travels via RFM from the IOO computer to the RFM model on the SUS computer, and then via dolphin to the OAF model on the LSC computer.

The MICH and PRCL signals travel via shmem from the LSC model to the OAF model, all on the LSC computer.  They don't go through the RFM model.

The seismometer channels travel via shmem between the PEM model on the SUS computer and the RFM model on the SUS computer, and then via dolphin between the SUS computer and the OAF model on the LSC computer.

Each pdf shows the power spectrum and a time series of the signals in their "original" model, and in the OAF model.  The seismometer is the only one that seems fine.  The time series match, except for a delay which is not surprising, since the signals have to travel.  The other signals seem pretty distorted.  What is going on??? Why can we trust some, but not all, of the signals that move between models and between computers???

 (This data was all taken while the MC was locked, but MICH and PRCL were not.  I don't think this should have any effect on the signal transfer though).

The MCL isn't soooo bad, so maybe we can keep moving forward with it, but I'm concerned that we're not really going to be successful OAF-ing the other degrees of freedom if the signals are so distorted.

Problem:

Each RFM memory location which needs to be read by a front end model slows the model significantly.

With no RFM memory locations to be read (replaced with grounds), the c1mcs model runs around 25 microseconds per cycle.

With 1 RFM memory location (MC_L), it runs around 29-33 microseconds.

With 3 RFM memory locations (MC_L, MC1_PIT, MC1_YAW), it runs around 45 microseconds.

With 7 RFM memory locations, the code generally doesn't run at all, going past the 62 microsecond maximum required to be able to keep up with the 16 kHz sample rate.

Last night Yuta somehow got it running with 7 RFM memory locations, but in that case, all the odd numbered RFM channels (1,3,5 as counted by the ipc file) did not work.  It was running at around 55 microseconds in that case.

The c1ioo code which is writing the data to the RFM card is experiencing no such slow down.

Current CDS status:

I suspect what happened here is that the IP didn't get updated when we went from the 131.215.113.xxx system to 192.168.113.xxx system. I fixed it now and can access the web interface. This system is now ready for remote debugging (from inside the martian network obviously). The IP is 192.168.113.90.

Managed to pull this operation off without crashing the RFM network, phew.

BTW, a windows laptop that used to be in the VEA (I last remember it being on the table near MC2 which was cleared sometime to hold the spare suspensions) is missing. Anyone know where this is ?

Kiwamu and I strung a temporary RFM fiber from the c1iscex machine (in the new 1X9 rack) to the c1sus machine (in the new 1X4 rack).  This was connected into the respective RFM cards.  Once we put the fiber in correctly, the status lights came on the RFM card, which is a good sign.  This did not go through the RFM bypass, and did not interfere with any other RFM connections.

We created a simple model to test the RFM card, which basically was 4 RFM memory locations passing back and forth between 2 filters on each machine.  These models were called c1rf0 (on c1sus) and c1rf1 (on c1iscex).  We added 4 entries to the /cvs/cds/caltech/chans/ipc/C1.ipc file corresponding to the 4 RFM memory locations, set their ipcType=RFM and set the ipcRate to 65536.  The ipcNum were set from 0 to 3. The models ran, however, the data we were trying to pass over the RFM card did not seem to be being passed.  Currently trying to contact Alex via e-mail to get debugging advice, and confirm the ipc file is setup correctly.

Since the RFM-Dolphin bridges for the ASX model was added to the c1rfm model, c1rfm kept timing-out from the single sample time of 60us.

The model had 19 dolphin accesses, 21 RFM accesses, and 9 shared memory (SHM) accesses.

At the beginning 2 RFM and 2 SHM accesses were moved to c1sus (i.e. they were mistakenly placed on c1rfm).
But this actually made the c1sus model timed out. So the model was reverted.

The current configuration is that the WFS related bridges were accommdated in the c1mcs model.
This made the timing of c1rfm ~40us. So it is safe now.
On the other hand, the c1mcs model has the time consumption of ~59us. This is marginal now.

We need to understand why any RFM access takes such huge delay.

The RFMnetwork is down.  MC2 sus damping restored.

 For the RF PD Frequency Response Measurement project, we get each PD signal from the "PD RF Mon" output of each demodulator board corresponding to our PD under test. Therefore we can't neglect the frequency response of various filters inside the demodulator board. I used our Agilent 4395 Network Analyzer to gather frequency response data for each demodulator board being considered for the RFPD frequency response project (AS55, REFL11, REFL33, REFL55, REFL165, POX11, POP22, POP110).

The NA swept over a frequency range of 1-500 MHz. Data was collected using NWAG4395A (from the netgpibdata directory). It should be noted that the command line options -a 16 -x 15 (averaging=16 and excitation amplitude=15 dBm[the max]), in addition to the usual command line options described in the help file, were used to minimize noise. 

The data is located in /users/alex.cole. The file names are in the format [PDNAME]DemodFilt_1000000.dat (e.g. REFL11DemodFilt_1000000.dat). Results for POP110 are shown below.

Summary:

While the vacuum system was knocked out, I measured the RF transimpedance (using the AM laser setup, didn't do the shot noise intercept current measurement for now) of all the RFPDs (except PMC REFL). At the very least, the following photodiodes are suspect:

1. WFS heads - expected transimpedance is 50 kohm unattenuated, and 5 kohm attenuated. I measure values that are x10 lower than this, and the segments are significantly imbalanced. Morover, the attenuators for some quadrants appear to do nothing. This could be a problem with the Acromag system I guess, but the measured transimpedance is nowhere close to the "expected" value. See Attachments #1 and #2. You can also see that the response at 55 MHz is significantly attenuated, so I'm guessing trying to measure the AS port ASC sensing response is going to be difficult.
   
   Note that I assumed a 1kohm DC transimpedance, which is what I expect from the schematic and also is consistent with the DC voltage I measured, knowing the approximate optical power incident on the photodiode.
2. POP 22/ POP 110 - this is a Thorlabs PDA10CF diode. It should have a flat gain profile out to ~100 MHz, but I measure some weird features. The other PDA10CF we use, at AS110, shows a more reasonable response. See Attachment #3. I don't know what kind of failure mode this is? Anyway I'll try testing another PDA10CF and if it looks more reasonable, I'll switch out this diode. FWIW, the measured AS110 gain is ~3kohms, whereas the datasheet tells us to expect 5 kohms.

For the remaining photodiodes, I measure a transimpedance that is within ~20% of what is on the wiki page. The notches may benefit from some retuning. While I have the data, I will fit this and post a more complete report on the wiki.

Update July 6 1145am: WFS response plots now have legends mapping quadrants, and I've also added the response of a spare PDA10CF (which is now the new POP22/POP110 photodiode).

A more comprehensive report has been uploaded here. I'll zip the data files and add them there too. In summary:

1. There are several problems with the WFS heads
   • Some attenuators don't seem to work. This could be a problem with the Acromag BIO, or with the relay on the head itself.
   • The measured transimpedance at 29.5 MHz is much lower than expected. We expect ~50 kohms with no attenuation, and 5 kohms without. I measure 100 ohm - 2 kohm with the attenuation disabled, and ~200 ohms with it enabled.
   • Quadrant #3 on both WFS heads behaves differently from the others. There is also evidence of a 200 MHz oscillation for quadrant 3.
   • For some reason, there is a relative minus sign between the TFs measured for the WFS and for the RFPDs. I don't understand where this is coming from - all the OpAmps in the LSC PDs and WFS heads are configured as non-inverting, so why should there be a minus sign? Is this indicative of the polarity of the LEMO output being somehow flipped?
2. POX 11 photodiode does not have a notch at 22 MHz.
3. AS55 resonance appears to have shifted closer to 60 MHz, would benefit from a retuning. But the notches appear fine.
4. PDA10CF photodiode used as the POP22/POP110 readback appears broken in some strange way. As shown in the linked document, a spare PDA10CF in the lab has a much more reasonable response, so I am going to switch out the POP22/POP110 diode with this spare.

I'll upload the data and analysis notebook + liso fit files to the wiki as well shortly. The data, a Jupyter notebook making the plots, and the LISO fit files have been uploaded here.

I didn't do it this time but it'd be nice to also do the noise measurement and get an estimate for the shot-noise intercept current.

For future reference, I've taken spectra of our various RFPDs while the PRMI was sideband locked on REFL33, using a 20dB RF coupler at the RF input of the demodulator boards. The 20dB coupling loss has been added back in on the plots. Data files are attached in a zip.

Exceptions: 

• The REFL165 trace was taken at the input of the amplifier that immediately preceeds the demod board. 
• The 'POPBB' trace was taken with the coupler at the input of the bias tee, that leads to an amplifier, then splitter, then the 110 and 22 demod boards. 

I also completely removed the cabling for REFLDC -> CM board, since it doesn't look like we plan on using it anytime in the immediate future. 

After some discussion with Koji, I've asked Steve to order some SBP-30+ bandpass filters as a quick and cheap way to help out REFL33. (Also some SBP-60+ for 55MHz, since we only have 1*fmod and 2*fmod bandpasses here in the lab). 

The nonlinearity in the LSC detection chain (cf T050268) comes from the photodetector and not the demod board. The demod board has low pass or band pass filters which Suresh installed a long time ago (we should check out what's in REFL33 demod board). 

Inside the photodetector the nonlinearity comes about because of photodiode bias modulation (aka the Grote effect) and slew rate limited distortion in the MAX4107 preamp.

In this file (under Tommy), we have a notebook which runs through a spectrum of frequencies and determines the gain response of the attached filter. Below we have the output of a high pass filter. We use IQ demodulation to change IQ componets to DC. Then using a butterworth filter, we read out the DC components and determine the gain's magnitude and phase. However, the phase seems very noisy. This is because the oscillators in the different tiles are independent and a random phase is introduced by changing the mixer frequency in individual tiles. To resolve this we need Multi Tile Synchronization or "MTS". 

Original Pynq Support Forum Query: https://discuss.pynq.io/t/rfsoc-2x2-phase-measurement/3892

We also have the code to fit a resposne function using IIRregular, but this is not as useful without proper phase data.