I don't think the proposed scheme for sensing and controlling the homodyne phase will work without some re-thinking of the scheme. I'll try and explain my thinking here and someone can correct me if I've made a fatal flaw in the reasoning somewhere.
Field spectrum cartoon:
Attachment #1 shows a cartoon of the various field components.
So is there a 90 degree relative shift between the signal quadrature in the simple Michelson vs the DRFPMI? But wait, there are more problems...
Closing a feedback loop using the 44 MHz signal:
We still need to sense the 44 MHz signal with a photodiode, acquire the signal into our CDS system, and close a feedback loop.
I don't have any bright ideas at the moment - anyone has any suggestions?🤔
I wanted to check what kind of signal the photodiode sees when only the LO field is incident on the photodiode. So with the IFO field blocked, I connected the PDA10CF to the Agilent analyzer in "Spectrum" mode, through a DC block. The result is shown in Attachment #2. To calculate the PM/AM ratio, I assumed a modulation depth of 0.2. The RIN was calculated by dividing the spectrum by the DC value of the PDA10CF output, which was ~1V DC. The frequencies are a little bit off from the true modulation frequencies because (i) I didn't sync the AG4395 to a Rb 10 MHz signal, and (ii) the span/BW ratio was set rather coarsely at 3kHz.
I would expect only 44 MHz and 66 MHz peaks, from the interference between the 11 MHz and 55 MHz sideband fields, all other field products are supposed to cancel out (or are in orthogonal quadratures). This is most definitely not what I see - is this level of RIN normal and consistent with past characterization? I've got no history in this particular measurement.
While I stopped by the lab this morning to pick up some things, I took the opportunity to continue the recovery.
At some point, we should run the suspension eigenmode routine (kick optics, let them ringdown, measure peak locations and Qs) to confirm that the remaining suspensions are okay, will also help in actuation re-allocation efforts on ETMY. But I didn't do this today.
Leaving the lab at 1150.
See Attachment #1.
The beam spot on ETMY looks weird (looks almost like a TEM10 mode), but the one on ITMY seems fine, see Attachment #2. Wonder what's going on there, maybe just a focusing problem?
We need a vent to fix the suspension, but until then what we can do is to redistribute the POS/PIT/YAW actuations to the three coils.
The HEPA filters on the PSL enclosure are no longer running. I tried cycling the power switch on the NW corner of the enclosure, and also turned the dial on the Variac down to zero and back up to maximum, no effect. Judging by the indicator LED on it, the power is making it to the Variac - bypassing the Variac and directly connecting the HEPA to the AC power seems to have no effect either.
I can't be sure, but I'm almost certain I heard them running at max speed an hour ago while I was walking around inside the VEA. Probably any damage that can happen has already been done, but I dialled down the Innolight injection current, and closed its shutter till the issue can be resolved.
I got some feedback from Koji who pointed out that the phase tracker is not required here. This situation is similar to the phase locking of two lasers together, which we frequently do, except in that case, we usually we offset the absolute frequencies of the two lasers by some RF frequency, and we demodulate the resulting RF beatnote to use as an error signal. We can usually acquire the lock by simply engaging an integrator (ignoring the fact that if we actuate on the laser PZT, which is a frequency actuator, just a proportional feedback will be sufficient because of the phase->frequency conversion), the idea being that the error signal is frequently going through a zero-crossing (around which the sinusoidal error signal is approximately linear) and we can just "catch" one of these zero crossings, provided we don't run of actuation range.
So the question here becomes, is the RF44 signal a suitable error signal such that we can close a feedback loop in a similar way? To try and get more insight, I tried to work out the situation analytically. I've attached my thinking as a PDF note. I get some pretty messy complicated expressions for the RF44 signal contributions, so it's likely I've made a mistake (though Mathematica did most of the heavy lifting), it'll benefit from a second set of eyes.
Anyways, I definitely think there is some additional complications than my simple field cartoon from the preceeding elog would imply - the relative phases of the sidebands seem to have an effect, and I still think the lack of the PRC/SRC make the situation different from what Hang/Teng et. al. outlined for the A+ homodyne phase control analysis. Before the HEPA failed, I had tried closing the feedback loop using one quadrature of the demodulated RF44 signal, but had no success with even a simple integrator as the loop (which the experience with the PLL locking says should be sufficient, and pretty easily closed once we see a sinusoidally oscillating demodulated error signal). But maybe I'm overlooking something basic conceptually?
The interlocks tripped at ~630am local time. Jordan reported that TP2 was supposedly running at 52 C (!).
V1 was already closed, but TP2 was still running. With him standing by the rack, I remotely exectued the following sequence:
Jordan confirmed (by hand) that TP2 was indeed hot and this is not just some serial readback issue. I'll do the forensics later.
I was in the lab from 1630-1830. I have located and visually inspected all the parts required for the magnet regluing / optic cleaning parts of the planned vent, except the fresh batches of scpectroscopic grade solvents. I was in the cleanroom part of the clean and bake lab from 1630-1700.
The HEPAs work again. After running the HEPAs for ~1 hour, I checked the particle count on the PSL table - the meter registered 0 for both 0.3 um and 0.5 um. So I decided to turn the NPRO back on, at ~1730 local time. The PMC and IMC were readily locke, so the basic interferometer functionality is returned, and we can now go ahead with (i) vent prep (ii) air BHD tests and (iii) IMC debuggin as was discussed on the call today. The earth is shaking, but nothing serious so far, I will resume alignment of the interferometer later in the evening when hopefully things have calmed down a bit more...
Note that the many other issues Koji noted in the preceeding elog (e.g. flaky wiring) have not been addressed.
Chub kindly provided us with an electronic anemometer. With the meter held directly against the HEPA filter inside the enclosure, we measured ~700 cfm of airflow on each of the two HEPAs, with the Variac set to 100% and the HEPAs themselves set to "High". With the Variac at 50%, the flow drops to ~160 cfm. At the nominal setting of 33%, the meter didn't register any flow. I don't know what the spec'd flow rate is for this combination of blower + filter, but Jordan says similar units in Downs register ~1500 cfm at the "High" setting. The two protable (similarly sized) HEPA units in the 40m, one at ITMY and one at ETMY, register ~900 cfm and ~1100 cfm respectively, when set to high. So we may want to revisit what the "nominal" HEPA setting should be, in case the filters have become clogged over time.
Some photos of the HEPA blowers with the pre-filters off and the capacitors switched out may be found here.
The optomechanics stock in the lab was in a sad state. We have obtained the following from Thorlabs in the last two months:
I have used some of these for the ari BHD setup. The unused items are stored in the shelves that house the optomechanics ~halfway down the east arm. I'm wondering what's a good setup to document the stock of this stuff so we can always have a healthy stock of optomechanics (at least the non speciality ones like posts, spacers etc). It sucks to realize at 0000hrs that you're missing a 3mm shim or 250mm converging lens or something.
Attachment #1 shows that the ITMX, ETMY and beamsplitter Oplev light levels have decayed significantly from their values when installed. In particular, the ETMY and ITMX sum channels are now only 50% of the values when a new HeNe was installed. ELOG search revealed that ITMY and ETMX HeNes were replaced with newly acquired units in March and September of last year respectively. The ITMX oplev was also replaced in March 2019, but the replacement was a unit that was being used to illuminate our tourist attraction glass fiber at EX.
We should replace these before any vent as they are a useful diagnostic for the DC alignement reference.
2x500 ml bottles of spectroscopic grade isopropanol were delivered. I marked them with today's date and placed them in the solvent cabinet. In the process, my shoulder bumped the laser interlock switch by the door to the VEA in the drill press area, which turned the PSL NPRO off. I turned it back on just now. The other NPROs are not connected to the interlock and so were unaffected.
Attachment #1 shows that the c1rfm model isn't able to receive any signals from the front end machines at EX and EY. Attachment #2 shows that the problem appears to have started at ~430am today morning - I certainly wasn't doing anything with the IFO at that time.
I don't know what kind of error this is - what does it mean that the receiving model shows errors but the sender shows no errors? It is not a new kind of error, and the solution in the past has been a series of model reboots, but it'd be nice if we could fix such issues because it eats up a lot of time to reboot all the vertex machines. There is no diagnostic information available in all the places I looked. I'll ask the CDS group for help, but I'm not sure if they'll have anything useful since this RFM technology has been retired at the sites (?).
In the meantime, arm cavity locking in the usual way isn't possible since we don't have the trigger signals from the arm cavity transmission.
Update 1500 4 Oct: soft reboots of models didn't do the trick so I had to resort to hard reboots of all FEs/expansion chassis. Now the signals seem to be okay.
After the earthquake on September 19 2020, it looks to me like the only lasting damage to suspensions in vacuum is the ETMY UR magnet being knocked off.
Suspension ringdown tests:
I did the usual suspension kicking/ringdown test:
Attachment #2 also includes info about the matrix diagonalization, and the condition numbers of the resulting matrices are as large as ~30 for some suspensions, but I think this isn't a new feature.
The simple interferometer, composed of a single bounce reflection from ITMY and the LO beam deilvered via fiber to the AS table, can be locked - i.e. the phase of the LO beam can be controlled such that the DC light level on the DCPDs after the two beams are interfered can be stabilized. This test allows us to confirm that various parts of the sensing and actuation chain (e.g. PI PZT for homodyne phase control, Trek amplifier etc etc) are working.
I will post more quantitative analysis tomorrow.
Attachment #2 shows the servo topology.
We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:
I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.
This looks cool, we should have something similar, can be really useful.
These boxes were moved from the 40m hallway to the inside of the VEA so that we have some space to walk around. You can find some pictures here.
Attachment #1: spectra of the phase noise between LO and IFO output fields sensed using the RF44 signal.
Closing a feedback loop:
We cleared up some space in the 1Y1 electronics rack to install the 3 new FE machines. I removed the current driver and laser from 1Y1, they are now stored in the E10 cabinet. I will upload some photos to gPhotos soon.
Per Koji's suggestion, I turned the PSL HEPA Variac to 50% just now, so that the power load through the burnt electrical cable is reduced by 75%.
Summary of discussion between Koji and gautam on 14 Oct:
I tried all of these last night / overnight, here are my findings.
Analog locking of the homodyne phase:
See Attachment #1.
Relative stability of two IFR2023s synchronized to the same FS725 Rb standard:
The electrical LO signal for demodulation of the 44 MHz photocurrent is provided by an IFR2023 signal generator. To maintain a fixed phase relation between this signal, and the phase modulation sidebands imprinted on the interferometer light via a separate IFO2023 signal generator, I synchronize both to the same Rb timing standard (a 10 MHz signal from the FS725 is sent to the rear panel frequency standard input on the IFR). We don't have a direct 44 MHz electrical signal available from the main IFO Marconi at the LSC rack (or anywhere else for that matter). So I decided to do this test at 55 MHz.
A look at the time domain signal:
With the Michelson locked on the dark fringe, the RF44 I and Q signals in the time domain are shown in Attachment #3 for a 1 minute stretch.
I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.
I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?
So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?
It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.
what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?
Apart from the questionable wiring on the Acromags, one other important difference is in the way the connections were made between the old VME crates to the Eurocrate backplanes, and how we do it now. The thick cables had their sheilds connected to the eurocrate ground (or at least, there was a dedicated ground lug on those cables which we screwed on to the ground terminals on the Eurocrate backplanes). However, in our current configuration, we interface the Acromag ADCs and DACs to the backplane via these adaptor boards. The shields of the DSUB cables are presumably NOT connected to the Eurocrate grounds. This should also be investigated as one potential cause of the grounding issue - while on some of the Eurocrate modules, the P1/P2 connectors may have either the "A" or "C" row of connectors shorted to ground, some may not, and the TTFSS may suffer from such an issue?
Note that we have this problem in all of the slow machines that were upgraded to Acromag (if this turns out to be the issue).
In fact, the problem was the grounding issue (presumably on the IOO racks).
I wanted to look into the ISS situation. Some weeks ago, I found the PD that was previously used as the in-loop photodiode. I wanted to use this and measure the open-loop RIN at a few places (to see if there's any variation and also to check its functionality). However, I didn't get very far tonight - for a start, the PD height is 3" (while our beam height is 4" everywhere outside the vacuum), and I needed to put together a circuit to supply the 5V bias and +/- 15 V since the transimpedance is done on the head. I was only able to do a low-level functionality test tonight, checked that the DC voltage output varied linearly with the incident power (calibrated against an NF1611 photodiode, data will be put up later). I didn't get to measuring any noise performance - is an incandescent light bulb still shot noise limited at ~10 Hz < f < 10kHz? Some notes:
Unconnected to this work - this problem reared its ugly head again (i noticed it yesterday morning already actually). I don't have the energy to embark on a fix tonight, Koji is going to be in the lab all day tomorrow and so he will fix it.
Ok I was using the PD in the black mount because Rana recommended it a few weeks ago.
Regarding the M2ISS, I acquired the hardware from QIL some months ago, including a circuit board, and 2 PDs. These had LEMO outputs though (not BNC), and the mounts are not 4". These photodiodes are what I'm using as the airBHD DCPDs right now, and some photos are here - are these the photodiodes you mentioned? Or are there yet more M2ISS photodiodes? I remember Johannes had some custom mounts extruded to make them 4" high, do you mean those? Can I retrieve them his Cryo setup?
BTW, my elog scraping shows only one spectra from Stefan in the ATF elog, and the performance there is more like 1e-7/rtHz @ 100 Hz, and that’s using a dedicated high BW servo circuit, not the SR560. Am I just missing the measurement of 2e-8/rtHz?
that little PD in the black mount was never very good. The AD829 is not a good opamp for transimpedance and especially not good for low frequencies. Stefan Ballmer and I were able to get 2e-8 out of these (@100 Hz) many years ago.
I wonder if we have some of Zach's M2ISS photodetectors around, perhaps in QIL or Cryo. I doubt that any of them are in use now. Those had good performance nad BNC output.
If everything else looks good, I'll start letting the dry N2 into the main volume after lunch.
Now the green transmissions are visible by the green PDs. Attachment 9 shows the trans and ref of each green beams with and without locking to TEM00. The questionable green TRY was ~0.3. If we compare this with the histrical data (Attachment 10), it is about 1/4 of the value in the past. It's not too crazy but still quite low.
BTW, nice video! @ Koji, How difficult was it to edit it into this form?
We are now ready to take the doors off. I've already done the basic prep work (loosened bolts, cleaned chamber, carts for tools, fresh ameristat on portable HEPAs etc).
[koji, rana, gautam]
This morning, we did the following;
The OSEMs remain in the EY vacuum chamber. The next set of steps are:
We will most likely work on this tomorrow. At ~1615, I briefly opened the PSL shutter and tweaked the IMC alignment. We will almost certainly change the pointing into the IMC when we remove the old OMC and rebalance that table, so care should be taken when working on that...
To be a bit more clear about what we are going to do in the OMC chamber, I marked-up some photos, see Attachments #1 and #2.
I anticipate that after this work, the only components on the table will be
Are we in agreement with this plan?
See #15656 for the updated photo
Good point - looking back, I also see that I already removed the mirror at the SW corner of the table in 2016. Revised photo in Attachment #1. There is an optic on the east edge of this table whose purpose I'm not sure of, but I'm pretty sure it isn't essential to the main functionality and so can be removed.
I believe the mirror next to IM1 is for the green beams to be delivered to the PSL table. I think we still want to keep it. Otherwise, the plan looks fine.
I got a call from Calum ~830am today saying some facilities people entered the lab, opened the south entrance door, and tripped the alarm in the process. I came to the lab shortly after and was able to reset the alarm by flipping the switch on the alarm box at the south end entrance to "Alarm OFF". Then, I double checked that the door is closed, and re-enabled the alarm. The particle count at the SP table is not unusually high and the lasers (Oplev HeNe and AUX X) were still on, so doesn't look like any lasting damage was done. The facilities people were apparently wearing laser safety goggles.
The IMC isn't resonant for a TEM00 mode at the time of writing - we are waiting for the stack to relax, at which point if the IMC isn't resonant for a TEM00 mode, we will tweak the input pointing into the IMC (we want to use the suspended cavity as the reference, since it is presumably more reliable than the table from which we removed ~50 kgs of weight and shifted the balance.
I am working on the setup of a CDS FE, so please do not attempt any remote login to the IPMI interface of c1bhd until I'm done.
So all the primary vent objectives have been achieved 🙌 . The light doors are on the chamber right now. I'm measuring the free-swinging spectra of ETMY overnight. Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020. Some photos here.
Attachment #1 shows the main result - there are 4 peaks. The frequencies are a little different from what I have on file for ETMY and the Qs are a factor of 3-4 lower (except SIDE) than what they are in vacuum, which is not unreasonable I hypothesize. The fits suggest that the peak shape isn't really Lorentzian, the true shape seems to have narrower tails than a Lorentzian, but around the actual peak, the fit is pretty good. More detailed diagnostic plots (e.g. coil-to-coil TFs) are in the compressed Attachment #2. The condition number of the matrix to diagonalize the sensing matrix (i.e. what we multiply the "naive" OSEM 2 Euler basis matrix by) is ~40, which is large, but I wouldn't read too much into it at this point.
I see no red flags here - the PIT peak is a little less prominent than the others, but looking back through the elog, this kind of variation in peak heights doesn't seem unreasonable to me. If anyone wants to look at the data, the suspension was kicked every ~1100seconds from 1288673974, 15 times.
I'm measuring the free-swinging spectra of ETMY overnight.
The PMC servo railed and so I re-locked it at ~half range. I've been noticing that the diurnal drift of the PZT control voltage has been larger than usual - not sure if it's entirely correlated with temperature on the PSL table. Anyway the cavity is locked again so all is good.
I was able to boot one of the 3 new Supermicro machines, which I christened c1bhd, in a diskless way (with the boot image hosted on fb, as is the case for all the other realtime FEs in the lab). This is just a first test, but it is reassuring that we can get this custom linux kernel to boot on the new hardware. Some errors about dolphin drivers are thrown at startup but this is to be expected since the server isn't connected to the dolphin network yet. We have the Dolphin adaptor card in hand, but since we have to get another PCIe card (supposedly from LLO according to the BHD spreadsheet), I defer installing this in the server chassis until we have all the necessary hardware on hand.
I also have to figure out the correct BIOS settings for this to really run effectively as a FE (we have to disable all the "un-necessary" system level services) - these machines have BIOS v3.2 as opposed to the older vintages for which there are instructions from K.T. et al.
There may yet be issues with drivers, but this is all the testing that can be done without getting an expansion chassis. After the vent and recovering the IFO, I may try experimenting with the c1ioo chassis, but I'd much prefer if we can do the testing offline on a subnet that doesn't mess with the regular IFO operation (until we need to test the IPC).
Basic IFO alignment checks were done.
Tomorrow, we should do some visual checks of the chambers / EQ stops on ETMY etc but I don't see any major problems at the moment...
Barring any catastrophic failures and provided all required personnel are available, we will do the final pre-close-up checks, put the heavy doors back on, and pump down starting 10 am Monday, 9 Nov 2020.
1100 - EY chamber inspected, no issues were found --> EY heavy door on
1200 - OMC chamber was inspected. OM6 was marginally tweaked to bring the beam down a little in pitch, and also a little northwards in Yaw. --> Heavy door on.
1230 - Pumpdown started. Initially, the annuli volume was pumped down. The procedure calls for doing this with the small turbopumps. However, V7 was left open, and hence, in the process, the TP1 foreline pressure (=P2) hit ~30 torr. This caused TP1 to shutdown. We were able to restart it without issue. This case was not caught by the interlock code, which was running at the time. It should be recitified.
1330 - OMC breadboard clean optics and DCPD hardware were wrapped up and packed into tupperware boxes and stored along the south arm. OMC cavity itself, the OMMT, and the breadboard the OMC was sitting on are wrapped in foil/Ameristat and stored in cabinet S13, lower 2 shelves.
1915 - P1a = 0.5 torr pressure reached. Switched over to pumping the main volume with TP1, backed by TP2 and TP3, which themselves are backed by their respective dry pumps and also the AUX drypump for some extra oomph. All cooling fans available in the area were turned on and directed at the turbo pumps. RV2 was used to throttle the flow suitably.
It was at this point that we hit a snag - RV2 has gotten stuck in a partially open position, see Attachment #1. We can see that the thread doesn't move in response to turning the rotary dial. Fortunately, the valve is partially open, so the main volume continues to be pumped - see Attachment #2 for the full history of today's pumping. We are leaving the main volume pumped in this configuration overnight (TP1 pumping main volume backed by TPs 2 and 3, which are in turn backed by their respective drypumps and also the AUX dry pump). I think there is little to no risk of any damage to the turbo pumps, the interlocks should catch any anomalies. The roughing pumps RP1 and RP3 were turned off and that line was disconnected and capped.
What are our options?
We need some vacuum experts to comment. Why did this happen? Is this an acceptable failure mode of the valve?
2230 - P1a = 0.025 torr. The pressure is coming down with log-linear scale. x0.1 per 2.5 hours or so.
I've uploaded some more photos here. I believe the problem is a worn out thread where the main rotary handle attaches to the shaft that operates the valve.
This morning, I changed the valve config such that TP2 backs TP1 and that combo continues to pump on the main volume through the partially open RV2. TP3 was reconfigured to pump the annuli - initially, I backed it with the AUX drypump but since the load has decreased now, I am turning the AUX drypump off. At some point, if we want to try it, we can try pumping the main volume via the RGA line using TP2/TP3 and see if that allows us to get to a lower pressure, but for now, I think this is a suitable configuration to continue the IFO work.
There was a suggestion at the meeting that the saturation of the main volume pressure at 1mtorr could be due to a leak - to test, I closed V1 for ~5 hours and saw the pressure increased by 1.5 mtorr, which is in line with our estimates from the past. So I think we can discount that possibility.
Looking back through the elog, 1mtorr is the pressure at which it is deemed safe to send the full power beam into the IMC. After replacing the HR mirror in the MCREFL path with a 10% reflective BS, I just cranked the power back up. IMC is locked. With the increased exposure on the MC2T camera, lots of new scattered light has become visible.
While proceeding with the interferometer recovery, I noticed that there appeared to be no light on WFS2. I confirmed on the AP table that the beam was indeed hitting the QPD, but the DC quadrants are all returning 0. Looking back, it appears that the failure happened on Monday 26 October at ~6pm local time. For now, I hand-aligned the IMC and centered the beams on the WFS1 and MC2T QPDs - MCT is ~15000 cts and MC REFL DC is ~0.1, all consistent with the best numbers I've been able to obtain in the past. I don't think the servo will work without 1 sensor without some retuning of the output matrix.
It would appear that both the DC and RF outputs of WFS2 are affected - I dithered the MC2 optic in pitch (with the WFS loop disabled) at 3.33 Hz, the transmission and WFS1 sensors see the dither but not WFS2. It could be that I'm just not well centerd on the PD, but by eye, I am, so it would appear that the problem is present in both the DC and RF signal paths. I am not going into the PD head debugging today.
I want to test out an AS port WFS now that I have all the parts in hand - I guess the Michelson / PRMI will suffice until I make the ALS noise good again, and anyways, there is much assembly work to be done. Overnight, I'm repeating the suspension eigenmode measurement.
The results from the ringdown are attached - in summary:
I had to go through five SR560s in the lab yesterday evening to find one that had the expected 4 nV/rtHz input noise and worked on battery power. To confirm that the batteries were charged, I left 4 of them plugged in overnight. Today, I confirmed that the little indicator light on the back is in "Maintain" and not "Charge". However, when I unplug the power cord, they immediately turn off.
One of the units has a large DC output offset voltage even when the input is terminated (though it is not present with the input itself set to "GND" rather than DC/AC). Do we want to send this in for repair? Can we replace the batteries ourselves?
I now think the excess noise in this circuit could be coming from the KEPCO switching power supply (in fact, the supplies are linear, and specd for a voltage ripple at the level of <0.002% of the output - this is pretty good I think, hard to find much better).
All component references are w.r.t. the schematic. For this test, I decided to stuff a fresh channel on the board, with new components, just to rule out some funky behavior of the channel I had already stuffed. I decoupled the HV amplifier stage and the Acromag DAC noise filtering stages by leaving R3 open. Then, I shorted the non-inverting input of the PA95 (i.e. TP3) to GND, with a jumper cable. Then I measured the noise at TP5, using the AC coupling pomona box (although in principle, there is no need for this as the DC voltage should be zero, but I opted to use it just in case). The characteristic bump in the spectra at ~100Hz-1kHz was still evident, see the bottom row of Attachment #1. The expected voltage noise in this configuration, according to my SPICE model, is ~10 nV/rtHz, see the analysis note.
As a second test, I decided to measure the voltage noise of the power supply - there isn't a convenient monitor point on the circuit to directly probe the +/- HV supply rails (I didn't want any exposed HV conductors on the PCB) - so I measured the voltage noise at the 3-pin connector supplying power to the 2U chassis (i.e. the circuit itself was disconnected for this measurement, I'm measuring the noise of the supply itself). The output is supposedly differential - so I used the SR785 input "Float" mode, and used the Pomona AC coupling box once again to block the large DC voltage and avoid damage to the SR785. The results are summarized in the top row of Attachment #1.
The shape of the spectra suggests to me that the power supply noise is polluting the output noise - Koji suggested measuring the coherence between the channels, I'll try and do this in a safe way but I'm hesitant to use hacky clips for the High Voltage. The PA95 datasheet says nothing about its PSRR, and seems like the Spice model doesn't include it either. It would seem that a PSRR of <60dB at 100 Hz would explain the excess noise seen in the output. Typically, for other Op-Amps, the PSRR falls off as 1/f. The CMRR (which is distinct from the PSRR) is spec'd at 98 dB at DC, and for other OpAmps, I've seen that the CMRR is typically higher than the PSRR. I'm trying to make a case here that it's not unreasonable if the PA95 has a PSRR <= 60dB @100 Hz.
So what are the possible coupling mechanisms and how can we mitigate it?
What do the analog electronics experts think? I may be completely off the rails and imagining things here.
Update 2130: I measured the coherence between the positive supply rail and the output, under the same conditions (i.e. HV stage isolated, input shorted to ground). See Attachment #2 - the coherence does mirror the "bump" seen in the output voltage noise - but the coherence is. only 0.1, even with 100 averages, suggesting the coupling is not directly linear - anyways, I think it's worth it to try adding some extra decoupling, I'm sourcing the HV 10uF capacitors now.
Shruti picked it up @4pm.
It is stored along with the cables that arrived a few weeks ago, awaiting the gauges which are now expected next week sometime.
Where do we want to install the interface and readout electronics for the AS port WFS? Options are:
There isn't much difference in terms of cable length that will be required - I believe the AS WFS is going to go on the AP table even in the new optical layout and not on the ITMY in-air oplev table?
The project requires a large number of new electronics modules. Here is a short update and some questions I had:
Approximately half of the assembly of the various electronics is now complete. The basic electrical testing of the interface chassis and demod chassis are also done (i.e. they get power, the LEDs light up, and are stable for a few minutes). Detailed noise and TF characterization will have to be done.