I realized that for more phase margin, rana added 6:3lead filter on WFS PIT loops. Since we have increased the UGF on YAW loops too, I turned these on the YAW loops as well. The loops remain stable unlike with the previous matrix. Attachment 1 is the repeat of teh emasurement done by rana earlier but with the new matrix and updated gains in PIT loops. The dark green traces are the references from last measurment with higher gain and HEPA off. The remainging colored traces were measured today.

80 degrees is too uncomfortable in the control room so I turned on the AC. The set point is 74F.

This past Friday I swapped out a burnt resistor on the spare STACIS unit I'm working with and powered it up. Here's the setup:

And here's what happened:

X an Y directions: When I switched from open to closed loop (making the internal geophones provide feedback), the STACIS started making a loud noise- it seemed like it was oscillating uncontrollably.

Z direction: The same thing happened in z until I added some weight to the top of the STACIS- then it quieted down, and seemed to work okay. The geophone signal dropped considerably compared to the open loop signal, which is expected if the feedback is working.

Then I tried driving the PZTs with a signal from the SR785 network analyzer. With an amplitude of tens of mV and frequencies from around .1 to 200 Hz, I could see the accelerometers I mounted on top of the STACIS definitely register motion, which means I was successfully driving the PZTs.

Below are transfer functions of the STACIS as I drove the PZTs from .1 to 100 Hz at 10 mV. The top graph is open loop, the second is closed loop. These were measured with the internal geophones.

In the bottom graph, "A" is closed loop and "B" is open loop, where the transfer functions were taken with the accelerometers instead of the geophones.

The PMC trans power was a little low (0.77V or so).  I tweaked up the input pointing and now we're getting about 0.875V transmitted.

I spent some time this afternoon reviving some of my CESAR/ESCOBAR shenanigans on the Y arm. I found it neccesary to adjust a few things.

• PMC realigned 
• ETMY oplev centered
• Y End green realigned
• PSL/Y Green beat realigned

Afterwards, ALSY noise levels were good. 

EDIT: After discussing with Koji and checking the existing M2ISS PDs I put the two C30642G back and took two C30665GH (active diameter: 3mm) diodes. Only one of this type remains in storage.

I removed two C30642G photodiodes from the stash for the new M2ISS hardware and updated the wiki page accordingly.

Both Guralps and the Ranger have been placed in our nice new insulated foam box, complete with packing peanuts, in the corner between the x and y arms. The Guralp breakout box has been reinstalled and everything is plugged in in prepartion for the huddle test. However, we're having some issues with ADC channels, which will be worked out tomorrow (hopefully) so that data can be collected over the weekend.

Currently, one Guralp is plugged into the three SEIS-MC1 channels. We made new channels for the second Guralp (GUR-EW, GUR-NS, and GUR-VERT), but had issues with those. So, EW and NS have been plugged into PEM_AUDIO-MIC1 and MIC2 for the time being.

 I found that several of the cables are unlabeled so I'm not sure what's plugged in. In the end, I found that the TEMP_2, _3, & _44 channels were working and so I plugged in anything that looked seismic into there.

TEMP_2 is now apparently the X channel of the 2nd Guralp. If someone can figure out which cable belongs to the Y channel, please plug it into TEMP_3 and then we can fix the channel names.

I also removed (gently) all of the accelerometers from MC2's chamber. This didn't break the lock, but I intentionally broke it to make sure it reacquired fine. It did and the MC TRANS QPD showed no significant shift afterwards.

In eLog 7148 (http://nodus.ligo.caltech.edu:8080/40m/7148), Koji pointed out that the op-amp and SR560 noise values (which I took from specs and then multiplied by the geophone calibration factor to get m/s/rtHz) were waaay too low. My error was an extra multiplication factor in the plotting script.

The other change was recalculating the ADC noise by splitting a signal into two ADC channels and subtracting the time series (then taking the PSD and converting to m/s/rtHz). It compares well to the value I got by terminating the ADC channels, which was the ADC noise line in my last eLog.

Both these changes are included in the below plot:

 [Jenne, Diego]

Today we began implementing the UGF Servos. Things we did:

• we updated the LSC model with both DARM and CARM servos, and moved them from after the control system to before it, at the level of the error signal;
• we updated the medm screens; new buttons are located in the main LSC screen;
• we started commissioning the DARM servo, at first using DARM for the lock of the single Y arm, then we moved on to the PRFPMI lock and the usual transition from ALS to Transmission;
• although we had several lock losses during the night, we managed to tweak the parameters of the DARM UGF servo (phases, excitation, gains), which now seems to work sufficiently fine;
• the filters added to the I and Q filter banks are a single lowpass in each, while the only filter in the main servio is a standard integrator;
• we don't have a step response at the moment, but we can say that the settling time of the servo is in the range of 10 seconds;
• we updated the ALSdown.py and ALSwatch.py scripts with a call to a new UGFdown.py script; this script, located in the scripts/PRFPMI folder, takes care of disabling the servos and putting the excitation to zero in case of a lock loss; re-enablement of such things must be done manually;

I added UGF servos for the DRMI DoFs, after creating a library block for the servos. I also deleted the FMs before the phase rotation, since we can just do it afterwards in other existing FMs. I've only added the MICH and PRCL buttons to the LSC screen because in the end, I feel like a dropdown is better, but I just wanted to get it running quickly tonight. The LSC model and the UGF block have been committed to the svn. 

We were able to use the PRCL UGF servo successfully, as Jenne was exploring MICH offset space. 

TL;DR: When the laser has good lock, the OLTF moves up and the UGF moves over!

-----------------------------------------------------------

Figured out with Paco yesterday that when the laser is locked but kind of weakly (mirrors on the optical table sliiightly out of alignment, for example), we would get a UGF around 5 kHz, but when we had a very strong lock (adjusting the mirrors until the spot was brightest) we would get a UGF around 13-17 kHz. Attached are some plots of us going back and forth (you can kind of tell from the coherence/error that the one with the lower UGF is more weakly locked, too). Error on the plots is propagated using the coherence data (see Bendat and Piersol, Random Data, Table 9.6 for the formula). 

-------------------------------------------------------------

Want to take data next week to quantitatively compare optical gain to UGF!

The UGF servos have been moved to the control point, are are once again totally linear!

The UGF servos were recommissioned today:

• suitable values of frequency, excitation, phases and gain were found;
• the phases were chosen in order to maximize the I signal and suppress the Q one;
• the servos seemed sufficiently stable when in a quiet state, but they didn't performed well in other cases;
• I also found out that DARM & CARM and MICH & PRCL are maybe too much coupled, but that could be actually due to the main loops rather than the UGF ones;
• however, after some weird rampings with no apparent reasons, and after some quite bad and glitchy step responses, I found out that the effect of the chosen phases vanished: the I and Q signals were of the same order of magnitude again, probably causing the bad performance;
• Jenne and I tried to increase che SINGAINs and COSGAINs (but keeping them equal to each other): this has the good effect of separating more the I and Q signals, but it's just a zoom effect: there still are mixing effects and, more important, some zero-crossings into negative values that cause the signal going into the servo to go crazy.

Our idea is that we need with some thinking about these servos and most of all try to figure out all this phase thing before we can start to trust the servos to be used for locking.

UGF Servos' commissioning still going on, updates of today:

• on Rana's suggestion, we don't use anymore the Q-signal rejection at the level of Phase 1 and Phase 2; instead, a proper complex division is made between those two signals (with a check in case of zero); then the resulting signal is demodulated with a new Phase 3, which is the one used to select the I signal while zeroing the Q one; changes to the model and the screens have been made;
• a new evaluation of all the parameters for the four servos has been made; aside for the new phase, and the zeroing of the other phases (because they are not used anymore for the selection), the parameters are not dissimilar from the prevoius ones
• the PRCL and MICH servos seem sufficiently stable;
• CARM and DARM are stable only for a short amount of time; what usually happens is that one of the two starts drifting in one random direction, and usually the other one follows shortly after; it is not clear if there is a relation or if they stop being stable after a similar amount of time; I still noticed a few lowest limits appearing in the input signal, which should be avoided; I'll check the model again tomorrow;
• the weird thing about CARM and DARM is that at the same time when one of them starts drifting, its I and Q signals begin to be comparable; when the servo is shut off, they resume their normal state;
• an increase in the excitation gain improves the separation of I and Q and also reduces their variations, but a high peak in the loop due to this might not be a good idea.

I found an error in the model of the UGF servos, I have now corrected it; for future reference, now the division between TEST2 and TEST1 is properly done with complex math: given

we have that TEST3:

TEST3 is the actual signal that is now phase rotated to select only the I signal while rejecting the Q one.

All the updates to the model, the screens and the script have been SVNed.

 I lost UL osem voltage this morning when I was checking the actual connection at rack ETMY

This after noon I disconnected  the 64 pins IDE connector from satelite amp at the rack, and the two 25 pins Dsubs at this juction board.

UL OSEM recovered after reconnecting these three connectors.

Atm3, bad connection.........noisy UL

It took too long to get this box ready for action. I implemented all of the changes that I made on the previous one (#1437). In addition, since this one is to be used for phase locking, I also made it have a ~flat transfer function. With the Boost ON, the TF magnitude will go up like 1/f below ~1 kHz.

The main trouble that I had was with the -12V regulator. The output noise level was ~500 nV/rHz, but there was a large oscillation at its output at ~65 kHz. This was showing up in the output noise spectrum of U1 (the first op-amp after the mixer). Since the PSRR of the OP27 is only ~40 dB at such a high frequency, it is not strange to see the power supply noise showing up (the input referred noise of the OP27 is 3.5 nV/rHz, so any PS noise above ~350 nV/rHz becomes relavent).

I was able to tame this by putting a 10 uF tantalum cap on the output of the regulator. However, when I replaced the regulator with a LM7912 from the blue box, it showed an output noise that went up like 1/f below 50 kHz !! I replaced it a couple more times with no benefit. It seems that something on the board must now be damaged. I checked another of the UPDH boxes, and it has the same high frequency oscillation but not so much excess voltage noise. I found that removing the protection diode on the output of the regulator decreased the noise by a factor of ~2. I also tried replacing all of the 1 uF caps that are around the regulator. No luck.

Both of the +12 V regulators seem fine: normal noise levels of ~200 nV/rHz and no oscillations.

Its clear that the regulator is not functioning well and my only guess is that its a layout issue on the board or else there's a busted component somewhere that I can't find. In any case, it seems to be functioning now and can be used for the phase locking and PZT response measurements.

For your reference: Voltage noise of LM7815/LM7915 (with no load)

 APC Smart-UPS (uninterruptible power supply) batteries RBC12 replaced at 1Y8 vacuum rack.

Their life span were 22 months.

[steve, gautam]

At Rolf/Rich Abbott's request, we performed a check of the UPS today.

Steve believed that the UPS was functioning as it should, and the recent accidental vent was because the UPS batteries were insufficiently charged when the test was performed. Today, we decided to try testing the UPS.

We first closed V1, VM1 and VA6 using the MEDM screen. We prepared to pull power on all these valves by loosening the power connections (but not detaching them). [During this process, I lost the screw holding the power cord fixed to the gate valve V1 - we are looking for a replacement right now but it seems to be an odd size. It is cable tied for now.]

The battery charge indicator LEDs on the UPS indicated that the batteries were fully charged.

Next, we hit the "Test" button on the UPS - it has to be held down for ~3 seconds for the test to be actually initiated, seems to be a safety feature of the UPS. Once the test is underway, the LED indicators on the UPS will indicate that the loading is on the UPS batteries. The test itself lasts for ~5seconds, after which the UPS automatically reverts to the nominal configuration of supplying power from the main line (no additional user input is required).

In this test, one of the five battery charge indicator LEDs went off (5 ON LEDs indicate full charge).

So on the basis of this test, it would seem that the UPS is functioning as expected. It remains to be investigated if the various hardware/software interlocks in place will initiate the right sequence of valve closures when required.

Steve, Rana, Ben, Jenne, Alberto, Rob

UPS in the vacuum rack failed this afternoon, cutting off power to the vacuum control system.  After plugging all the stuff that had been plugged into the UPS into the wall, everything came back up.  It appears that V1 closed appropriately, TP1 spun down gracefully on its own battery, and the pressure did not rise much above 3e-6 torr. 

The UPS fizzed and smelled burnt.  Rana will order a new, bigger, better, faster one.

The Maglev is running on single phase 220V and that voltage  was not interrupted. TP1 was running undisturbed with V1 and V4 closed.

It is independent of the UPS 120V.

As Rob noted last Friday, the UPS which powers the Vacuum rack failed. When we were trying to move the plugs around to debug it, it made a sizzling sound and a pop. Bad smells came out of it.

Ben came over this week and measured the quiescent power consumption. The low power draw level was 11.9 A and during the reboot its 12.2 A. He measured this by ??? (Rob inserts method here).

So what we want is a 120 V * 12.2 A  ~ 1.4 kVA UPS with ~30-50% margin. We look for this on the APC-UPS site:

On Monday, we will order the SUA2200 from APC. It should last for ~25 minutes during an outage. Its $1300. The next step down is $200 cheaper and gives 10 minutes less uptime.

Summary:

1. The (120V) UPS at the vacuum rack is faulty.
2. The drypump backing TP2 is faulty.
3. Current status of vacuum system: 
   • The old UPS is now powering the rack again. Sometime ago, I noticed the "replace battery" indicator light on this unit was on. But it is no longer on. So I judged this is the best course of action. At least this UPS hasn't randomly failed before...
   • main vol is being pumped by TP1, backed by TP3.
   • TP2 remains off.
   • The annular volumes are isolated for now while we figure out what's up with TP2.
   • The pressure went up to ~1 mtorr (c.f. ~600utorr that is the nominal value with the stuck RV2) during the whole episode but is coming back down now.
4. Steve seems to have taken the reliability of the vacuum system with him.

Details:

Around 7pm, the UPS at the vacuum rack seems to have failed. Don't ask me why I decided to check the vacuum screen 10 mins after the failure happened, but the point is, this was a silent failure so the protocols need to be looked into.

Going to the rack, I saw (unsurprisingly) that the 120V UPS was off. 

• Pushed the power on button - the LCD screen would briefly light up, say the line voltage was 120 V, and then turned itself off. Not great.
• I traced the power connection to the UPS itself to a power strip under the rack - then I moved the plug from one port to another. Now the UPS stays on. okay...
• but after ~3 mins while I'm hunting for a VGA cable, I hear an incessant beeping. The UPS display has the "Fault" indicator lit up. 
• I decided to shift everything back to the old UPS. After the change was made, I was able to boot up the c1vac machine again, and began the recovery process.
• When I tried to start TP2, the drypump was unusually noisy, and I noticed PTP2 bottomed out at ~500 torr (yes torr). So clearly something is not right here. This pump supposedly had its tip-seal replaced by Jordan just 3 months ago. This is not a normal lifetime for the tip seal - we need to investigate more in detail what's going on here...
• Decided that an acceptable config is to pump the main volume (so that we can continue working on other parts of the IFO). The annuli are all <10mtorr and holding, so that's just fine I think.

Questions:

1. Are the failures of TP2 drypump and UPS related? Or coincidence? Who is the chicken and who is the egg?
2. What's up with the short tip seal lifetime?
3. Why did all of this happen without any of our systems catching it and sending an alert??? I have left the UPS connected to the USB/ethernet interface in case anyone wants to remotely debug this.

For now, I think this is a safe state to leave the system in. Unless I hear otherwise, I will leave it so - I will be in the lab another hour tonight (~10pm).

Some photos and a screen-cap of the Vac medm screen attached.

I've investigated the vacuum controls failure that occurred last night. Here's what I believe happened.

From looking at the system logs, it's clear that there was a sudden loss of power to the control computer (c1vac). Also, the system was actually down for several hours. The syslog shows normal EPICS channel writes (pressure readback updates, etc., and many of them per minute) which suddenly stop at 4:12 pm. There are no error or shutdown messages in the syslog or in the interlock log. The next activity is the normal start-up messaging at 7:39 pm. So this is all consistent with the UPS suddenly failing.

According to the Tripp Lite manual, the FAULT icon indicates "the battery-supported outlets are overloaded." The failure of the TP2 dry pump appears to have caused this. After the dry pump failure, the rising pressure in the TP2 foreline caused TP2's current draw to increase way above its normal operating range. Attachment 1 shows anomalously high TP2 current and foreline pressure in the minutes just before the failure. The critical system-wide failure is that this overloaded the UPS before overloading TP2's internal protection circuitry, which would have shut down the pump, triggering interlocks and auto-notifications.

Preventing this in the future:

First, there are too many electronics on the 1 kVA UPS. The reason I asked us to buy a dual 208/120V UPS (which we did buy) is to relieve the smaller 120V UPS. I envision moving the turbo pumps, gauge controllers, etc. all to the 5 kVA unit and reserving the smaller 1 kVA unit for the c1vac computer and its peripherals. We now have the dual 208/120V UPS in hand. We should make it a priority to get that installed.

Second, there are 1 Hz "blinker" channels exposed for c1vac and all the slow controls machines, each reporting the machine's alive status. I don't think they're being monitored by any auto-notification program (running on a central machine), but they could be. Maybe there already exists code that could be co-opted for this purpose? There is an MEDM screen displaying the slow machine statuses at Sitemap > CDS > SLOW CONTROLS STATUS, pictured in Attachment 2. This is the only way I know to catch sudden failures of the control computer itself.

I don't buy this story - P2 only briefly burped around GPStime 1291608000 which is around 8pm local time, which is when I was recovering the system.

Today. Jordan talked to Jon Feicht - apparently there is some kind of valve in the TP2 forepump, which only opens ~15-20 seconds after turning the pump on. So the loud sound I was hearing yesterday was just some transient phenomenon. So today morning at ~9am, we turned on TP2. Once again, PTP2 pressure hovered around 500 torr for about 15-20 seconds. Then it started to drop, although both Jordan and I felt that the time it took for the pressure to drop in the range 5 mtorr - 1 mtorr was unusually long. Jordan suspects some "soft-start" feature of the Turbo Pumps, which maybe spins up the pump in a more controlled way than usual after an event like a power failure. Maybe that explains why the pressure dropped so slowly? One thing is for sure - the TP2 controller displayed "TOO HIGH LOAD" yesterday when I tried the first restart (before migrating everything to the older UPS unit). This is what led me to interpret the loud sound on startup of TP2 to indicate some issue with the forepump - as it turns out, this is just the internal valve not being opened.

Anyway, we left TP2 on for a few hours, pumping only on the little volume between it and V4, and PTP2 remained stable at 20 mtorr. So we judged it's okay to open V4. For today, we will leave the system with both TP2 and TP3 backing TP1. Given the lack of any real evidence of a failure from TP2, I have no reason to believe there is elevated risk.

As for prioritising UPS swap - my opinion is that it's better to just replace the batteries in the UPS that has worked for years. We can run a parallel reliability test of the new UPS and once it has demonstrated stability for some reasonable time (>4 months), we can do the swap.

I was able to clear the FAULT indicator on the new UPS by running a "self-test". pressing and holding the "mute" button on the front panel initiates this test according to the manual, and if all is well, it will clear the FAULT indicator, which it did. I'm still not trusting this unit and have left all units powered by the old UPS.

Update 1100 Dec 11: The config remained stable overnight so today I reverted to the nominal config of TP3 pumping the annuli and TP2 backing TP1 which pumps the main volume (through the partially open RV2).

Is that a fault code that you can decipher in the manual, or just a light telling you nothing but your UPS is dead?

I can't find anything in the manual that describes the nature of the FAULT message.  In fact, it's not mentioned at all.  If the unit detects a fault at its output, I would expect a bit more information.  This unit does a programmable level of input error protection, too, usually set at 100%.  Still, there is no indication in the manual whether an input issue would be described as a fault; that usually means a short or lifted ground at the output.

Now that the old APC Smart-UPS 2200 is no longer in use by the vacuum system, I looked into whether it can be repurposed for the framebuilder machine. Yes, it can. The max power consumption of the framebuilder (a SunFire X4600) is 1.137kW. With fresh batteries, I estimate this UPS can power the framebuilder for >10 min. and possibly as long as 30 min., depending on the exact load.

@Chub/Jordan, this UPS is ready to be moved to rack 1X6/1X7. It just has to be disconnected from the wall outlet. All of the equipment it was previously powering has been moved to the new UPS. I have ordered a replacement battery (APC #RBC43) which is scheduled to arrive 9/09-11.

I'm in the lab this morning to interface the two new UPS units with the digital controls system. Will be out by lunchtime. The disruptions to the vac system should be very brief this time.

I'm leaving the lab shortly. We're not ready to switch over the vac equipment to the new UPS units yet.

The 120V UPS is now running and interfaced to c1vac via a USB cable. The unofficial tripplite python package is able to detect and connect to the unit, but then read queries fail with "OS Error: No data received." The firmware has a different version number from what the developers say is known to be supported.

The 230V UPS is actually not correctly installed. For input power, it has a general type C14 connector which is currently plugged into a 120V power strip. However this unit has to be powered from a 230V outlet. We'll have to identify and buy the correct adapter cable.

With the 120V unit now connected, I can continue to work on interfacing it with python remotely. The next implementation I'm going to try is item #2 of this plan [ELOG 15446].

​I looked into how the new UPS devices suggested by Chub would communicate with the vac interlocks. There are several possible ways, listed in order of preference:

• Python interlock service directly queries the UPS via a USB link using the (unofficial) tripplite package. Direct communication would be ideal because it avoids introducing a dependency on third-party software outside the monitoring/control capability of the interlock manager. However the documentation warns this package does not work for all models...
• Configure Tripp Lite's proprietary software (PowerAlert Local) to send SYSLOG event messages (UDP packets) to a socket monitored by the Python interlock manager.
• Configure the proprietary software to execute a custom script upon an event occurring. The script would, e.g., set an EPICS flag channel which the interlock manager is continually monitoring.

I recommend we proceed with ordering the Tripp Lite 36HW20 for TP1 and Tripp Lite 1AYA6 for TP2 and TP3 (and other 120V electronics). As far as I can tell, the only difference between the two 120V options is that the 6FXN4 model is TAA-compliant.

Chub has placed the order for two new UPS units (115V for TP2/3 and a 220V version for TP1).

They will arrive within the next two weeks.

To use the Sensoray 2250 USB frame grabber:

Ensure you have the following packages installed: build-essential, libusb-dev

Download the Linux manual and linux SDK from the Sensoray website at:

http://www.sensoray.com/products/2250data.htm

Go to the Software and Manual tab near the bottom to find the links.  The software can also be found on the 40m computers at /cvs/cds/caltech/users/josephb/sensoray/

The files are Manual2250LinuxV120.pdf and s2250_v120.tar.gz

Run the following commands in the directory where you have the files.

tar -xvf s2250_v120.tar.gz

cd s2250_v120

make

cd ezloader

make

sudo make modules_install

cd ..

At this point plug in the 2250 frame grabber.

sudo modprobe s2250_ezloader

Now you can run the demo with

./sraydemo or ./sraydemo64

Options will show up on screen.  A simple set to start with is "encode 0", which sets the recording type, "recvid test.mpg", which starts the recording in file test.mpg, and "stop", which stops recording.  Note there is no on screen playback.  One needs an installed mpeg player to view the saved file, such as Totem (which can screen cap to .png format) or mplayer.

All these instructions are on the first few pages of the Manual2250LinuxV120 pdf.

I have moved the USB flash drives from the electronics bench back into the middle drawer of the cabinet next to the AC which is west of the fridge. Drawer re-enlabeled.

 After some system updates this evening, firefox can no longer handle the html input encoding for the elog.  I'm not sure what happened.  You can still use the "ELCode" or "plain" input encodings, but "HTML" won't work.  The problem seems to be firefox 17.  ottavia and rosalba were upgraded, while rossa and pianosa have not yet been.

I've installed chromium-browser (debranded chrome) on all the machines as a backup.  Hopefully the problem will clear itself up with the next update.  In the mean time I'll try to figure out what happened.

To use chromium: Appliations -> Internet -> Chromium

 I can't create a new page on the 40m wiki.  The page that I was trying to create is

http://blue.ligo-wa.caltech.edu:8000/40m/Stewart

I get this message when I try to save the new page:

Page could not get locked. Unexpected error (errno=13).

This address for wiki is obsolete. Recently it was switched to https://wiki-40m.ligo.caltech.edu/
Jamie is working on automatic redirection from the old wiki to the new place.

The new one uses albert.einstein authentication.

[ericq, Gautam]

We were not able to fix the excess frequency noise of the AUX X laser by the usual laser diode current song and dance. Unfortunately, this level of noise is much too high to have any realistic chance of locking.  

We're leaving things back in the IR beat -> phase tracker state with free running AUX lasers, on the off chance that there may be anything interesting to see in the overnight data. This may be limited by our lack of automatic beatnote frequency control. (Gautam will soon implement this via digital frequency counter). I've upped the FINE_PHASE_OUT_HZ_DQ frame rate to 16k from 2k, so we can see more of the spectrum.

For the Y beat, there is the additional weird phenomenon that the beat amplitude slowly oscillates to zero over ~10 minutes, and then back up to its maximum. This makes it hard for the phase tracker servo to stay stable... I don't have a good explanation for this. 

Here's how we should diagnose the EX laser:

1. Compare IR RIN of laser out to 100 kHz with that of another similar NPRO.
2. Look at time series of IR beat signal with a fast scope. Are there any high frequency glitches?
3. Disconnect all of the cables to the EX laser PZT and temperature control. Does the frequency noise change?
4. Change the temperature by +/- 1 deg to move away from mode hop regions. Remeasure RIN and frequency noise and plot.

EDIT: Sleepy Eric doesn't understand loops. The conditions for this observation included active oplev loops. Thus, obviously, looking at the in-loop signal after the ASC signl joins the oplev signal will produce this kind of behavior. 

After some talking with Rana, I set out on making an even better-er QPD loop. I made some progress on this, but a new mystery halted my progress. 

I sought to have a more physical undertanding of the plant TF I had measured. Earlier, I had assumed that the 4Hz plant features I had measured for the QPD loops were coming from the oplev-modified pendulum response, but this isn't actually consistent with the loop algebra of the oplev servos. I had seen this feature in both the oplev and qpd error signals when pushing an excitation from the ASC-XARM_PIT (and so forth) FMs. 

However, when exciting via the SUS-ETMX-OLPIT FMs (and so forth), this feature would not appear in either the QPD or oplev error signals. That's weird. The outputs of these two FMs should just be summed, right before the coil matrix. 

I started looking at the TF from ASC-YARM_PIT_OUT to SUS-ETMY_TO_COIL_1_2, which should be a purely digital signal routing of unity, and saw it exhibit the phase shape at 4Hz that I had seen in earlier measurements. Here it is:

I am very puzzled by all of this.  Needs more investigation.

The measured transimpedance of the latest POY11 PD matches my model very well up to 100 MHz. But at about ~216MHz I have a resonance that I can't really explain.

 The following is a simplified illustration of the resonant circuit:

Perhaps my model misses that resonance because it doesn't include stray capacitances.

While I was tinkering with it, i noticed a couple of things:

- the frequency of that  oscillation changes by grasping with finger the last inductor of the circuit (the 55n above); that is adding inductance

- the RF probe of the scope clearly shows me the oscillation only after the 0.1u series capacitor

- adding a small capacitor in parallel to the feedback resistor of the output amplifier increases the frequency of the oscilaltion

Where did you get the 55nH based notch from? I don't remember anything like that from the other LSC PD schematics. This is certainly a bad idea. You should remove it and put the notch back over by the other notch.