We crossed off another couple of bullets today.
It took me ~1 hour to realize that c1susaux requries the running of sudo /sbin/ifup eth0 to be run in order to see the martian network - why???
I don't understand the exact chain of causation, but during this work, the fast c1sus model crashed. I had to go through a few iterations of the scripted vertex machine rebooting, but things seem to be back in a normal state now, see Attachment #2. Should probably run the IFO test suite to make sure everything is a-okay, but for now, I am able to lock the IMC so I'm moving on.
The main task remaining here is to take new pictures of everything and upload to the wiki. Also, need to update the Sorensen labels to reflect their current values, some of them are outdated.
I looked at the PSL/IOO racks to check for which boards, if any, require an additional P2 interface, so that we can try and design a generic one for the IMC/CM boards and whatever else may require it. While searching the elog, I saw that Koji and Johannes had already done this, see Koji's elog in this thread. Some remarks:
Conclusion: Only the IMC Servo and CM boards need their P2 connectors connected to Acromag.It would be helpful to remove the TTFSS Interface board and figure out what exactly the pin-mapping for the backplane connectors are, but I didn't do this today because there is a "High Voltage" line going to the Interface Board and I'm not actually sure of the signal chain for the FSS servo.
I borrowed an old-looking Variac variable transformer from the power supplies cabinet along the y-arm. It is currently in the TCS lab.
Mon Oct 7 14:51:53 2019. I closed the PSL shutter to measure the WFS head responsivity.
I made a thru calibration as in this elog, treating laser, reference PD, and WFS RF output as a three-port device. The DC current supplied to the laser is 20.0 mA in all cases. The Agilent spectrum analyzer supplies a -10 dBm excitation to Jenne laser's AM port, and A/B is measured with 20dB attenuation on each input port. Results are in /users/aaron/WFS/data/191007/. The calibration had 100 averages, all other measurements 32 averages; other parameters found in the yml file, same folder as the data.
I normalized the result by the difference between the dark and bright DC levels of each segment.
Mon Oct 7 17:29:58 2019 opened PSL shutter.
I simulated this circuit with zero, but haven't gotten the results to match the measurements above.
It would be good if you and Shruti can look at how to change the parameters in Zero so as to do a fit to the measured data. Usually, in scipy.optimize we give it a function with some changeable params, so maybe there's a way to pass params to a zero object in that way. I think Ian and Anchal are doing something similar to their FSS Pockel's cell simulator.
In 1X1, there is a box labelled "FSS REF" below a KEPCO HV supply. This box had a power cable that wasn't actually connected to any power. I removed said cable.
[JV, JWR, YD, GV]
Jordan and I removed another 10 kg of cabling from 1X2. The c1iool0 crate now has all cabling to it disconnected - but it remains in the rack because I can't think of a good way to remove it without disturbing a bunch of cabling to the fast c1iool0 machine. We can remove it the next time the vertex FEs crash. Cross connects have NOT been removed - we will identify which cross connects are not connected to the fast system and trash those.
Do we want to preserve the ability to use the PZT driver in 1X2?
I used existing BNC cables running from the PSL table to the PSL rack and reassigned them to the PSL Shutter and PMC transmission PD channels.
The PSL shutter turned out to be a sinking channel. Jordan reconnected the PSL shutter wires to a sinking BIO Acromag. Channel list is updated.
Both channels have been tested to be working as expected.
gautam add on about EPICS:
P.S - there is a problem we noticed - if the modbus process is started with the local subnet not having a fixed IP address, then all the EPICS channels will not be responsive. The way to fix this is to run the following sequence of commands:
After discussing with Koji, I removed the PZT driver and associated AI card from the Eurocrate at 1X2. The corresponding backplane connectors were also removed from the cross connects. An additional cable going from the DAC to IDC adaptor on 1X2 was removed. Finally, some cables going to the backplane P1 and P2 connectors for slots in which there were no cards were removed.
Finally, there is the IMC WFS whitening boards. These were reconfigured in ~2016 by Koji to have (i) forever whitening, and (ii) fixed gain. So the signals from the P1 connector no longer have any influence on the operation of this board. So I removed these backplane cables as well.
Some pics attached. The only cross connect cabling remaining on the south side of 1X2 is going to the fast BIO adaptor box - I suspect these are the triggered fast whitening switching for the aforementioned WFS whitening board. If so, we could potentially remove those as well, and remove all the cross connects from 1X1 and 1X2.
Update 1720: indeed, as Attachment #2 shows, the RTCDS BIO channels were for the WFS whitening switching so I removed those cables as well. This means all the xconnects can be removed. Also, the DAC and BIO cards in c1ioo are unused.
Jon is going to write up the details of todays adventures. But the C1PSL Acromag chassis is sitting on the floor between the IMC beamtube and the 1X1 electronics rack, and is very much a trip hazard. Be careful if youre in that area.
We are going to replace the old Sun c1ioo with a modernized supermicro. At the opportunity, remove the DAC and BIO cards to use them with the new machines. BTW I also have ~4 32ch BIO cards in my office.
The C1PSL crate has now been installed in a more permanent way in the rack.
After this work, I disabled logging and restarted the modbus service (and copied the current version of the systemd service file to the target directory for backup). The PMC and IMC lock alright. The system is now ready to be tested in-situ. I will separately continue my IMC Servo board tests in the evening.
One thought about how to protect against this kind of silent failure - how about we always run the modbus service with logging enabled, and then send out a warning email and stop the service if the logfile size suddenly blows up (which is characteristic of when the communications process dies)? This should be done in addition to the ping-ing of the individual IPs.
Regarding the burt-restore step that the systemd service runs after starting up the IOC - this is not even that useful, at least in the way it is currently setup (restore the "latest" burt snapshot file). If the maintenance takes >1hour as it often does, the "latest" snapshot for the system under maintenance is just garbage. So either the burt-restore should be for a "known good time" (dangerous because this will require frequent updates of the systemd service every time we find a new safe state) or we should just do it manually (my preference). Then there is no need to install custom packages on the server machine. Anyway, for now, I have not commented this step out.
Jordan is going to take pictures of all the electronics racks and update the relevant wiki pages.
With the Acromag chassis now permanently installed, we tested the C1PSL channels going over the channel list one by one, excluding the IMC channels which Gautam is taking responsibility for (the servo board itself is also in question).
The strategy is to check the response of input channels to specific output channels for expected behaviour whenever is possible.
We marked on the channel list spreadsheet the status of the channels that were tested.
In more detail
PMC Servo Card
Unlocked the PMC by switching C1:PSL-PMC_SW1. Tweaked C1:PSL-PMC_RAMP and observed a change in C1:PSL-PMC_PZT.
We misaligned MC1 to get a measurable signal in WFS channels. NDScoped the corresponding C1:IOO-WFS*_SEG*_I&Q channels and observed a change in those channels in response to switching the attenuation on and off.
The signals were compared to previous values for consistency. Then they were unplugged from the Acromag chassis to confirm their values went to 0 and returned to the same values after being reconnected.
I am running some tests on the IMC servo board with an extender card so the IMC will not be locking for a couple of hours.
It seems like the AO path gain stages on the IMC Servo board work just fine. The weird results I reported earlier were likely a measurement error arising from the fact that I did not disconnect the LEMO IN2 cable while measuring using the BNC IN2 connector, which probably made some parasitic path to ground that was screwing the measurement up. Today, I re-did the measurement with the signal injected at the IN2 BNC, and the TF measured being the ratio of TP3 on the board to a split-off of the SR785 source (T-eed off). Attachments #1, #2 shows the result - the gain deficit from the "expected" value is now consistent with that seen on other sliders.
Note that the signal from the CM board in the LSC rack is sent single-ended over a 2-pin LEMO cable (whose return pin is shorted to ground). But it is received differentially on the IMC Servo board. I took this chance to look for evidence of extra power line noise due to potential ground loops by looking at the IMC error point with various auxiliary cables connected to the board - but got distracted by some excess noise (next elog).
To test a hypothesis, I have left the PSL shutter closed. I notice significant glitches in the dark electronics offsets on all the 11 MHz photodiode I/Q demodulated input channels, which appear coherent. These are non-negligible in magnitude - for now they are uncalibrated in cts, but for an estimate, the POX11 channel shows a shift of ~20 cts (~200uV at the input to the whitening board), while the PDH fringe is ~200 cts pk2pk. A first look is in Attachment #1. The fact that it's in all the 11 MHz channels makes me suspect something in the RF chain, maybe some amplifier? I'll open the shutter tomorrow.
A big factor in how much IFO locking activities can take place is how cooperative the IMC is.
Since the c1psl upgrade, the IMC duty cycle has definitely deteriorated. I took a measurement of the dark noise at the IMC error point with 1 Hz FFT binwidth, with all electrical connections to the IMC servo board except the Acromag and Eurocrate power disconnected. I was horrified at the prominence of 60 Hz harmonics - see Attachment #1. In the past, this kind of feature has been indicative of some error in the measurement technique - but I confirmed that the lines remain even if I unplug the GPIB box, and all combinations of floating/grounded inputs that I tried. We know for sure that there is some excess noise imprinted on the laser light post upgrade. While these lines almost certainly are not responsible for the PCdrive RMS going bonkers, surely this kind of electrical situation isn't good?
Attachment #2 shows the same information translated to frequency noise units, taking into account the complementary sensitivity function, L/(1+L) - the sum contribution of the 60 Hz peaks to the RMS is ~11.5% of the total over the entire band (c.f. 1.7 % that is expected if the noise at multiples of 60 Hz was approximately equal to the surrounding noise levels). Moreover, the measured RMS is 55 times higher than a LISO model.
How can this be fixed?
There were many locklosses from the point where the arm powers were somewhat stabilized. Attachments #1 and #2 show two individual locklosses. I think what is happening here is that the BS seismometer X channel is glitching, and creating a transient in the angular feedforward filter that blows the lock. The POP QPD based feedback loop cannot suppress this transient, apparently. For now, I get around this problem by boosting the POP QPD feedback loop a bit, and then turning the feedforward filters off. The fact that the other seismometer channels don't report any transient makes me think the problem is either with the seismometer itself, or the readout electronics. The seismometer masses were recently recentered, so I'm leaning towards the latter.
I didn't explicitly check the data, but I am reasonably certain the same effect is responsible for many PRMI locklosses even with the arms held off resonance (though the tolerance to excursions there is higher). Pity really, the feedforward filters were a big help in the lock acquisition...
While the vacuum system was knocked out, I measured the RF transimpedance (using the AM laser setup, didn't do the shot noise intercept current measurement for now) of all the RFPDs (except PMC REFL). At the very least, the following photodiodes are suspect:
For the remaining photodiodes, I measure a transimpedance that is within ~20% of what is on the wiki page. The notches may benefit from some retuning. While I have the data, I will fit this and post a more complete report on the wiki.
Update July 6 1145am: WFS response plots now have legends mapping quadrants, and I've also added the response of a spare PDA10CF (which is now the new POP22/POP110 photodiode).
A more comprehensive report has been uploaded here. I'll zip the data files and add them there too. In summary:
I'll upload the data and analysis notebook + liso fit files to the wiki as well shortly. The data, a Jupyter notebook making the plots, and the LISO fit files have been uploaded here.
I didn't do it this time but it'd be nice to also do the noise measurement and get an estimate for the shot-noise intercept current.
While I have the data, I will fit this and post a more complete report on the wiki.
This problem reared its ugly head again. I am inclined to believe the problem is electronic and not on the light, since the POY channels seem immune to this issue (see Attachment #1). I will investigate in the daytime tomorrow. Note that while the POX photodiode head has ~twice the transimpedance than POY (per measurement), the POY signal gets amplified by a ZHL-500-HLN amplifier before heading to the demod electronics (nominal gain is 19dB = x9). There is also some imbalance in the light level at the photodiodes I guess, because overall, the PDH fringe is ~twice as large for the Y arm as the X arm. Basically, the y-axes of the attached plot cannot be directly compared between POX and POY.
Mostly this is an annoyance - right now, the POX signal is only used for locking and dither aligning the X arm cavity, and so once that is done, the locking can proceed (as long as the other channels, e.g. REFL11, aren't glitching as well...)
in the lab, checkin on the WFS
Sun Jul 5 18:25:50 2020
I redid Gautam's measurements to get a baseline before changing the head, and my results are very different: To me it looks like the WFS2 quadrants are all OK.
I've left the setup as is in case either me or Gautam want to double check. If we're agreed on this response, I'll remove the notches and disable the RF attenuators.
Sun Jul 5 21:42:45 2020
After some hunting, I found this old SURF report with the WFS head measurements. The y-axes don't make much sense to me, and I can't find the actual data anywhere (her wiki page doesn't actually exist). So I think it's still unknown if these heads ever had the advertised transimpedance gain, or if the measured transimpedance of ~1kohm was what it always was.
Koji and I had a discussion last Friday about the suspension electronics. I think there are still a few open questions - see Attachment #1. We should probably make a decision on these soon.
Other useful links:
To facilitate this investigation, I've DQed the 4 face coil outputs for the two ETMs. EX is currently running with 5 times the series resistance of EY, so it'll be a nice consistency check. Compilation, installation etc went smooth. But when restarting the c1scx model, there was a weird issue - the foton file, C1SCX.txt, got completely wiped (all filter coefficients were empty, even though the filter module names themselves existed). I just copied the chiara backup version, restarted the model, and all was well again.
This corresponds to 8 additional channels, recorded at 16k as float 32 numbers, so in the worst case (neglecting any clever compression algorithms), we are using disk space at a rate of ~4 MB/s more. Seems okay, but anyway, I will remove these DQ channels in a few days, once we're happy we have enough info to inform the coil driver design.
spoke too soon - there was an RFM error for the TRX channel, and restarting that model on c1sus took down all the vertex FEs. Anyways, now, things are back to normal I think. The remaining red light in c1lsc is from the DNN model not running - I forgot to remove those channels, this would've been a good chance! Anyways, given that there is an MLTI in construction, I'm removing these channels from the c1lsc model, so the next time we restart, the changes will be propagated.
For whatever reason, my usual locking scripts aren't able to get me to the PRFPMI locked state - some EPICS channel value must not have been set correctly after the model reboot 😞. I'll debug in the coming days.
Fun times lie ahead for getting the new BHD FEs installed I guess 🤡 ....
Looking at signals to the ETMs from the current lock acquisition sequence, the RMS current to a single coil is approximately _____ (to be filled in later).
So we may need a version of the fast coil driver that supports a low noise mode (with large series resistance) and a high-range mode (with lower series resistance for lock acquisition).
Looking at the signals to the test mass coils, it seems borderline to me that we will be able to acquire lock and run in a low noise configuration with the same series resistor in the coil driver circuit. The way I see it, options are:
I only looked at the ETMs for this study. The assumption is that we will have no length actuation on the ITMs, only local damping and Oplev loops (and maybe some ASC actuation?), which can be sufficiently low-pass filtered such that even with coil de-whitening, we won't have any range issues.
Attachment #1 shows the time-domain traces of the coil driver signals as we transition from POX/POY lock to the ALS lock. There are some transients, but I think we will be able to hold the lock even with a 5 kohm resistor (~twice what is on ETMX right now). From just these numbers, it would seem we can even go up to 10 kohms right away and still be able to acquire lock, especially if we re-design the digital feedback loop to have better low-pass filtering of the high-frequency ALS noise, see the next attachment.
Attachment #2 shows the f-domain picture, once the arm lengths are fully under ALS control (~25 seconds onwards in Attachment #1). The RMS is dominated by high frequency ALS length loop noise, which we can possibly improve with better design of the digital control loop.
Finally, Attachment #3 shows the situation once DARM control has been transitioned over to AS55_Q. Note that the vertex DoFs are still under 3f control, so there is the possibility that we can make this even lower noise. However, one thing that is not factored in here is that we will have to de-whiten these signals to low-pass filter the DAC noise (unless there is some demonstrated clever technique with noise-mons or something to subtract the DAC noise digitally). Nevertheless, it seems like we can run safely with 5 kohms on each ETM coil and still only use ~2000 cts RMS, which is ~1/10th the DAC range (to allow for dealing with spurious transients etc).
See Attachments #1 and #2. We don't have any Q3000 QPDs in hand, at least not in the photodiode box stored in the clean optics cabinet at the south end. I also checked a cabinet along the east arm where we store some photodiodes - but didn't find any there either. The only QPDs we have in hand are the YAG-444-4AH, which I believe is what is used in the iLIGO WFS heads.
So how many do we want to get?
The "source" output of the SR785 has a DC offset of -6.66 V. I couldn't make this up.
Upshot is, this SR785 is basically not usable for TF measurements. I was using the unit to characterize the newly stuffed ISC whitening board. The initial set of measurements were sensible, and at some point, I started getting garbage data. Unclear what the cause of this is. AFAIK, we don't have any knob to tune the offset - adjusting the "offset" in the source menu, I can change the level of the offset, but only by ~1 V even if I apply an offset of 10 V. I also tried connecting the ground connection on the rear of the SR785 to the bench power supply ground, no change.
Do we have to send this in for repair?
Grrr. Let's repair the unit. Let's get a help from Chub & Jordan.
Do you have a second unit in the lab to survive for a while?
When I tested Q3000 for aLIGO, the failure rate was pretty high. Let's get 10pcs.
A single channel of this board was stuffed (and other channels partially populated). The basic tests passed, and nothing exploded! Even though this is a laughably simple circuit, it's nice that it works.
HV power supplies:
A pair of unused KEPCO BHK300-130 switching power supplies that I found in the lab were used for this test. I pulled the programmable cards out at the rear, and shorted the positive output of one unit to the negative of the other (with both shorted to the supply grounds as well), thereby creating a bipolar supply from these unipolar models. For the purposes of this test, I set the voltage and current limits to 100V DC, 10mA respectively. I didn't ramp up the supply voltage to the rated 300 V maximum. The setup is shown in Attachment #1.
Need to think more about how to better characterize this noise. An estimate of the required actuation can be found here.
A more careful analysis has revealed some stability problems. I see oscillations at frequencies ranging from ~600kHz to ~1.5 MHz, depending on the voltage output requested, of ~2 V pp at the high-voltage output in a variety of different conditions (see details). My best guess for why this is happening is insufficient phase margin in the open-loop gain of the PA95 high voltage amplification stage, which causes oscillations to show up in the closed loop. I think we can fix the problem by using a larger compensation capacitor, but if anyone has a better suggestion, I'm happy to consider it.
The changes I wanted to make to the measurement posted earlier in this thread were: (i) to measure the noise with a load resistor of 20 ohms (~OSEM coil resistance) connected, instead of the unloaded config previously used, and (ii) measure the voltage noise on the circuit side (= TP5 on the schematic) with some high voltage output being requested. The point was to simulate conditions closer to what this board will eventually be used in, when it has to meet the requirement of <1pA/rtHz current noise at 100 Hz. The voltage divider formed by the 25 kohm series resistor and the 20 ohm OSEM coil simulated resistance makes it hopeless to measure this level of voltage noise using the SR785. On the other hand, the high voltage would destroy the SR785 (rated for 30 V max input). So I made a little Pomona box to alllow me to do this measurement, see Attachment #1. Its transfer function was measured, and I confirmed that the DC high voltage was indeed blocked (using a Fluke DMM) and that the output of this box never exceeded ~1V, as dictated by the pair of diodes - all seemed okay .
Next, I wanted to measure the voltage noise with ~10mA current flowing through the output path - I don't expect to require more than this amount of current for our test masses. However, I noticed some strange features in the spectrum (viewed continuously on the SR785 using exponential averaging setting). Closer investigation using an oscilloscope revealed:
Some literature review suggested that the capacitor in the feedback path, C4 on the schematic, could be causing problems. Specifically, I think that having that capacitor in the feeddback path necessitates the use of a larger compensation capacitor than the nominal 33pF value (which itself is higher than the 4.7pF recommended on the datasheet, based on experience of the ESD driver circuit which this is based on, oscillations were seen there too but the topology is a bit different). As a first test of this idea, I removed the feedback capacitor, C4 - this seemed to do the trick, the oscillations vanished and I was able to drive the output between the high voltage supply rails. However, we cannot operate in this configuration because we need to roll off the noise gain for the input voltage noise of the PA95 (~6 nV/rtHz at 100 Hz will become ~200 nV/rtHz, which I confirmed using the SR785). Using a passive RC filter at the output of the PA95 (a.k.a. a "snubber" network) is not an option because we need to sum in the fast actuation path voltage at the output of the 25 kohm resistor.
Some modeling confirms this hypothesis, see Attachment #2. The quantity plotted is the open-loop gain of the PA95 portion of the circuit. If the phase is 0 degrees, then the system goes unstable.
So my plan is to get some 470pF capacitors and test this idea out, unless anyone has better suggestions? I guess usually the OpAmps are compensated to be unconditionally stable, but in this case maybe the power op-amp is more volatile?
Attachment #1 is a summary of the current to each coil on the suspensions. The situation is actually a little worse than I remembered - several coils are currently drawing in excess of 10mA. However, most of this is due to a YAW correction, which can be fixed somewhat more easily than a PIT correction. So I think the circuit with a gain of 31 for an input range of +/-10 V, which gives us the ability to drive ~12mA per coil through a 25kohm series resistor, will still provide sufficient actuation range. As far as the HV supplies go, we will want something that can do +/- 350 V. Then the current to the coils will at most be ~50 mA per optic. The feedback path will require roughly the same current. The quiescent draw of each PA95 is ~10mA. So per SOS suspension, we will need ~150mA.
If it turns out that we need to get more current through the 25kohm series resistance, we may have to raise the voltage gain of the circuit. Reducing the series resistance isn't a good option as the whole point of the circuit is to be limited by the Johnson noise of the series resistance. Looking at these numbers, the only suspension on which we would be able to plug in a HV coil driver as is (without a vent to correct for YAW misalignment) is ITMY.
Update 2 Sep 2020 2100: I confirmed today that the number reported in the EPICS channel, and the voltage across the series resistor, do indeed match up. The test was done on the MC3 coil driver as it was exposed and I didn't need to disable any suspensions. I used a Fluke DMM to measure the voltage across the resistor. So there is no sneaky factor of 2 as far as the Acromag DACs are concerned (unlike the General Standards DAC).
I unboxed the Trek amplifier today, and performed some basic tests of the functionality. It seems to work as advertised. However, we may have not specified the correct specifications - the model seems to be configured to drive a bipolar output of +/- 125 V DC, whereas for PZT driving applications, we would typically want a unipolar drive signal. From reading the manual, it appears to me that we cannot configure the unit to output 0-250V DC, which is what we'd want for general PZT driving applications. I will contact them to find out more.
The tests were done using the handheld precision voltage source for now. I drove the input between 0 to +5 V and saw an output voltage (at DC) of 0-250 V. This is consistent with the voltage gain being 50V/V as is stated in the manual, but how am I able to get 250 V DC output even though the bipolar configuration is supposed to be +/- 125 V? On the negative side, I am able to see 50V/V gain from 0 to -1 V DC. At which point making the input voltage more negative does nothing to the output. The unit is supposed to accept a bipolar input of +/- 10 V DC or AC, so I'm pretty sure I'm not doing anything crazy here...
Okay based on the markings on the rear panel, the unit is in fact configured for unipolar output. What this means is we will have to map the +/- 10 V DC output from the DAC to 0-5 V DC. Probably, I will stick to 0-2.5 V DC for a start, to not exceed 125 V DC to the PI PZT. I'm not sure what the damage spec is for that. The Noliac PZT I think can do 250 V DC no problem. Good thing I have the inverting summing amplifier coming in tomorrow...
I set up to do the WFS head modifications today, but I was shot down in flames due to a missing AC/DC adapter.
The Prologix GPIB-ethernet dongle needs +8-13 V to run. Some riff raff has removed the adapter and I was thunderstruck to see that it had not been returned.
I did the usual hunt around the lab looking for something with the right specs and connector. I found one that could do +9V and had the right connector, but it didn't light up the adapter so I put it back in black SP table.
I'll order a couple of these (5 ordered for delivery on Wednesday) in case there's a hot demand for the jack / plug combo that this one has. The setup is in the walkway, but I returned the AS table to the usual state and made sure the IMC is locking well.
Clearly this "riff raff" is referring to me. It won't help today I guess but there is one each on the carts holding the SR785 (currently both in the office/electronics bench area), and the only other unit available in the lab is connected to a Prologix box on the Marconi inside the PSL enclosure.
Teledyne AP1053 etc were transported from Rich's office to the 40m. The box is placed on the shelf at the entrance.
My record tells that there are 7 AP1053 in the box. I did not check the number this time.
Increasing the compensation capacitance (470 pF now instead of 33 pF) seems to have fixed the oscillation issues associated with this circuit. However, the measured noise is in excess of the model at almost any frequency of relevance. I believe the problem is due to the way the measurement is done, and that we should re-do the measurement once the unit is packaged in a shielded environment.
Attachment #1 shows (schematically) the measurement setup. Main differences from the way I did the last round of testing are:
Attachment #2 shows the measurement results:
I didn't capture the data, but viewing the high voltage output on an Oscilloscope threw up no red flags - the oscillations which were previously so evident were nowhere to be seen, so I think the capacitor switch did the trick as far as stability is concerned.
There is a large excess between measurement and model out to a few kHz, if this is really what ends up going to the suspension then this circuit is useless. However, I suspect at least part of the problem is due to close proximity to switching power supplies, judging by the comb of ~10 Hz spaced peaks. This is a frequent problem in coil driver noise measurements - previously, the culprit was a switching power supply to the Prologix GPIB box, but now a Linear AC-DC converter is used (besides, disconnecting it had no visible effect). The bench supplies providing power to the board, however, is a switching supply, maybe that is to blame? I think the KEPCO supplies providing +/-250 V are linear. I tried the usual voodoo of twisting the wires used to receive the signal, moving the SR785 away from the circuit board etc, but these measures had no visible effect either.
The real requirement of this circuit is that the current noise above 100 Hz be <1pA/rtHz. This measurement suggests a level that is 5x too high. But the problem is likely measurement related. I think we can only make a more informed conclusion after shielding the circuit better and conducting the test in a more electromagnetically quiet environment.
The unit was repaired and returned to the 40m. Now, with a DMM, I measure a DC offset value that is ~1% of the AC signal amplitude. I measured the TF of a simple 1/20 voltage divider and it looks fine. In FFT mode, the high frequency noise floor levels out around 5-7nV/rtHz when the input is terminated in 50 ohms.
I will upload the repair documents to the wiki.
These were delivered to the 40m today and are on Rana's desk
I'll order a couple of these (5 ordered for delivery on Wednesday) in case there's a hot demand for the jack / plug combo that this one has.
We received 20pcs of stuffed demodulator boards from Screaming Circuits today. Some caveats:
I removed 1 from the group to stuff some components that weren't sent to Screaming Circuits and test the functionality on the benchtop, the remaining have been stored in a plastic box for now as shown in Attachment #1. The box has been delivered to Chub who will stuff the remaining 19 boards once I've tested the one piece.
I'll bring a file binder "40m wiring diagram" to home at the next chance.
There is another one on the shelf in the control room.
(I thought I put it in my bag, but it looks like that I left it somewhere around the fax area)
I packaged the HV coil driver into a 2U chassis, hoping for better shielding from pickup. There is still considerable excess noise in measurement vs model around 100 Hz, see Attachment #1. The projected displacement noise from this noise contribution is shown in Attachment #2 - I've also plotted the contribution from the 4.5kohm (planned value for fast path series resistance) for comparison. Attachment #3 has some photos of the measurement setup so if someone sees some red flags, please let me know.
I've run out of ideas to try and make the measurement cleaner - the presence of the rather prominent power line harmonics suggests that this is still not perfect, but what more shielding can we implement? I have to make the measurement on the circuit side of the 25 kohm series resistor, so I am using some Pomona minigrabbers to clip onto the leg of the wirewound resistor (see photos in Attachment #3), so that's not great maybe, but what's the alternative?
So if this is truly the noise of the circuit, then while it's an improvement on the current situaiton, it's unsatisfying that such a simple circuit can't match the design expectations. But how do we want to proceed?
what is the noise level before the HV stage? i.e. how well is the acromag noise being filtered?
It's not so easy to directly measure this I think, because the filtering is rather aggressive. Attachment #1 shows the measured transfer function (dots) vs the model and Attachment #2 shows the noise. I think this checks out - but I can't definitively rule out some excess noise at 100 Hz from this stage. Because the gain of the HV stage is x31, we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess. The Acromag noise model in Attachment #2 is based on a measurement I describe here.
Andrew made a battery-powered 0.7 nVrtHz input-referred noise pre-amplifier for gain of 200. That might help you.
we'd need a preamp with better than 1nV/rtHz to directly measure the noise I guess.
RXA: 0.7 nV is OK if you're not interested in low noise measurements. Otherwise, we have the transformer coupled pre-amp from SRS which does 0.15 nV/rHz and the Rai Weiss FET amp which has 0.35 nV for high impedance sources.
I had to go through five SR560s in the lab yesterday evening to find one that had the expected 4 nV/rtHz input noise and worked on battery power. To confirm that the batteries were charged, I left 4 of them plugged in overnight. Today, I confirmed that the little indicator light on the back is in "Maintain" and not "Charge". However, when I unplug the power cord, they immediately turn off.
One of the units has a large DC output offset voltage even when the input is terminated (though it is not present with the input itself set to "GND" rather than DC/AC). Do we want to send this in for repair? Can we replace the batteries ourselves?
I now think the excess noise in this circuit could be coming from the KEPCO switching power supply (in fact, the supplies are linear, and specd for a voltage ripple at the level of <0.002% of the output - this is pretty good I think, hard to find much better).
All component references are w.r.t. the schematic. For this test, I decided to stuff a fresh channel on the board, with new components, just to rule out some funky behavior of the channel I had already stuffed. I decoupled the HV amplifier stage and the Acromag DAC noise filtering stages by leaving R3 open. Then, I shorted the non-inverting input of the PA95 (i.e. TP3) to GND, with a jumper cable. Then I measured the noise at TP5, using the AC coupling pomona box (although in principle, there is no need for this as the DC voltage should be zero, but I opted to use it just in case). The characteristic bump in the spectra at ~100Hz-1kHz was still evident, see the bottom row of Attachment #1. The expected voltage noise in this configuration, according to my SPICE model, is ~10 nV/rtHz, see the analysis note.
As a second test, I decided to measure the voltage noise of the power supply - there isn't a convenient monitor point on the circuit to directly probe the +/- HV supply rails (I didn't want any exposed HV conductors on the PCB) - so I measured the voltage noise at the 3-pin connector supplying power to the 2U chassis (i.e. the circuit itself was disconnected for this measurement, I'm measuring the noise of the supply itself). The output is supposedly differential - so I used the SR785 input "Float" mode, and used the Pomona AC coupling box once again to block the large DC voltage and avoid damage to the SR785. The results are summarized in the top row of Attachment #1.
The shape of the spectra suggests to me that the power supply noise is polluting the output noise - Koji suggested measuring the coherence between the channels, I'll try and do this in a safe way but I'm hesitant to use hacky clips for the High Voltage. The PA95 datasheet says nothing about its PSRR, and seems like the Spice model doesn't include it either. It would seem that a PSRR of <60dB at 100 Hz would explain the excess noise seen in the output. Typically, for other Op-Amps, the PSRR falls off as 1/f. The CMRR (which is distinct from the PSRR) is spec'd at 98 dB at DC, and for other OpAmps, I've seen that the CMRR is typically higher than the PSRR. I'm trying to make a case here that it's not unreasonable if the PA95 has a PSRR <= 60dB @100 Hz.
So what are the possible coupling mechanisms and how can we mitigate it?
What do the analog electronics experts think? I may be completely off the rails and imagining things here.
Update 2130: I measured the coherence between the positive supply rail and the output, under the same conditions (i.e. HV stage isolated, input shorted to ground). See Attachment #2 - the coherence does mirror the "bump" seen in the output voltage noise - but the coherence is. only 0.1, even with 100 averages, suggesting the coupling is not directly linear - anyways, I think it's worth it to try adding some extra decoupling, I'm sourcing the HV 10uF capacitors now.