MC1, MC2 and MC3 damping turned off to see glitching action at 9:57am
There was a pretty large glitch in MC1 about an hour ago. The misalignment was so large that the autolocker wasn't able to lock the IMC. I manually re-aligned MC1 using the bias sliders, and now IMC locks fine. Attached is a 90 second plot of 2K data from the OSEMs showing the glitch. Judging from the wall StripTool, the IMC was well behaved for ~4 hours before this glitch - there is no evidence of any sort of misalignment building up, judging from the WFS control signals.
I re-enabled the MC SUS damping and IMC locking for some IFO work just now.
The signal path for the ASDC signal is AS55 PD --> D990543 (interface board) --> D990694 (whitening board) --> D000076 (AA board) --> ADC Ch 31. Everything in this signal chain should be able to handle signals in the range +/- 10V, which should correspond to the full range of our +/-10V, 16bit ADCs. But the ASDC signal seems to saturate at ~2000 counts (i.e. turning up the analog whitening gain doesn't make the signal get any bigger than this). I investigated this a little more today.
So the problem lies somewhere downstream of the D990694. There are other anomalous behaviours of this channel - e.g. engaging the analog whitening filters changes the DC offset of the signal. I am going to pull out this board to check it out.
Why does this matter? I want to calibrate the ASDC level (and eventually the other PD DC signals as well) into Watts. This is useful for IFO diagnostics, noise budgeting the shot noise level etc.
According to the AS55 schematic, the DC transimpedance is 66.7 ohms. I claim that the DC power on the AS55 photodiode during a DRMI (no arms) lock is ~1mW. The C30642 photodiode (InGaAs) responsivity is ~0.8 A/W. So I'd expect ~50mV to be the signal level into the ADC (assuming gain of all the other electronics in the signal chain at the start of this elog is unity). This corresponds to ~163 counts (since the ADC conversion factor is 2^16 counts over 20volts). The DC signal level I observed is ~200 counts. So things seem roughly consistent.
*Note: Despite my above statement, I don't think it is true that the AS110 PD has more light on it - the BS splitting the light between
AS55 and AS110 PDs is a 50-50 BS, and using the crude method of putting an Ophir power meter in front of both PDs and
monitoring the power while the Michelson was swinging around freely showed roughly the same maximum value.
We've been talking about increasing the series resistance for the coil driver path for the test masses. One consequence of this will be that we have reduced actuation range.
This may not be a big deal since for almost all of the LSC loops, we currently operate with a limiter on the output of the control filter bank. The value of the limit varies, but to get an idea of what sort of "threshold" velocities we are looking at, I calculated this for our Finesse 400 arm cavities. The calculation is rather simplistic (see Attachment #1), but I think we can still draw some useful conclusions from it:
So, from this rough calculation, it seems like we would lose ~25% efficiency in locking the arm cavity if we up the series resistance from 400ohm to 1kohm. Doesn't seem like a big deal, becuase currently, the single arm locking
If you go through this thread of elogs, there are lots of pictures of the SOS assembly with the optic in it from the vent last year. I think there are many different perspectives, close ups of the standoffs, and of the OSEMs in their holders in that thread.
This elog has a measurement of the pendulum resonance frequencies with ruby standoffs - although the ruby standoff used was cylindrical, and the newer generation will be in the shape of a prism. There is also a link in there to a document that tells you how to calculate the suspension resonance frequencies using analytic equations.
I've been observing this for a few days: ETMX's DC alignment seems to drift by so much that the previously well aligned X arm cavity is now totally misaligned.
The wall StripTool trace shows that both the X and Y arms were locked with arm transmissions around 1 till c1psl conked out - so in the attached plot, around 1400 UTC, the arm cavity was well aligned. So the sudden jump in the OSEM sensor signals is the time at which LSC control to the ETM was triggered OFF. But as seen in the attached plot, after the lockloss, the Oplev signals seem to show that the mirror alignment drifted by >50urad. This level of drift isn't consistent with the OSEM sensor signals - of course, the Oplev calibration could be off, but the tension in values is almost an order of magnitude. The misalignment seems real - the other Oplev spots have stuck around near the (0,0) points where I recentered them last night, only ETMX seems to have undergone misalignment.
Need to think about what's happening here. Note that this kind of "drift" behaviour seems to be distinct from the infamous ETMX "glitching" problem that was supposed to have been fixed in the 2016 vent.
I should've put in the SUSPIT and SUSYAW channels in the previous screenshot. I re-aligned ETMX till I could see IR flashes in the arm, and also was able to lock the green beam on a TEM00 mode with reasonable transmission. As I suspected, this brought the Oplev spot back near the center of it's QPD. But the answer to the question "How much did I move the ETM by" still varies by ~1 order of magnitude, depending on if you believe the OSEM SUSPIT and SUSYAW signals, or the Oplev error signals - I don't know which, if any, of these, are calibrated.
Best to just calibrate the ETM OL in the usual way. I bet the OSEM outputs have a cal uncertainty of ~50% since the input matrix changes as a function of the DC alignment. Still, a 30 urad pitch mis-alignment gives a (30e-6 rad)(40 m) ~ 1 mm beam spot shift. This would be enough to flash other modes, but it would still be easy to lock on a TEM00 like this. I also doubt that the OL calibration is valid outside of some region near zero - can easily check by moving the ETM bias sliders.
What we still don't know is if this is due to Johannes/Aaron working at the ETMX rack (bumping some of the flaky coil cables and/or bumping the blue beams which support the stack). Adding or substracting weight from the stack supports will give us an ETM mis alignment.
I've often gotten confused by the labeling on the SUS MEDM screens about the coil "Vmon" fields - they're labelled as "30 Hz HPF", and indeed this is one of the many readbacks available on the coil driver board. But the actual EPICS channel that is being displayed in this field is from the "EPICS VMON" monitor point on the coil driver board. It has a gain of 1/2, so the actual voltage going to the coil is twice the channel value. Today, I fixed the SUS master screen to avoid this confusion - new labeling is shown in Attachment #1.
The ETMX Sorrenson power supply -15V was running at -13.9V
Seems like there was a 5.3 magnitude EQ ~10km from us (though I didn't feel it). All watchdogs were tripped so our mirrors definitely felt it. ITMX is stuck (but all the other optics are damping fine). I tried the usual jiggling of DC bias voltage but ITMX still seems stuck. Probably a good sign that the magnet hasn't come off, but not ideal that I can't shake it free..
edit: after a bit more vigorous shaking, ITMX was freed. I had to move the bias slider by +/-10,000 cts, whereas initially I was trying +/-2000 cts. There is a tendency for the optic to get stuck again once it has been freed (while the optic's free swinging motion damps out), so I had to keep an eye out and as soon as the optic was freed, I re-engaged the damping servos to damp out the optic motion quickly.
Satellite amplifiers labeled with date. Old labels left on.
The stack weight measurement is going on at EX. ETMX watchdog is shutdown. Area is off limits over the weekend until the test is finished.
Not related to this work, but the dog clamps used on the EX table have to be re-positioned such that the clamping force is better distributed. The 2" beam splitter mount used to pick off a portion of the EX NPRO beam to the fiber has to be rotated. Also, there was a M6.9 EQ in Hawaii while we were doing this work it seems..
We tried to estimate what the load cell measurement should yield. Here is the weight breakdown (fudge factor for Al table is to try and account for tapped holes):
I'm working near 1X5 and there is an SR785 adjacent to the electronics rack with some cabling running along the floor. I plan to continue in the evening so please leave the setup as is.
During the course of this work, I noticed the +15V Sorensen in 1X6 has 6.8 A of current draw, while Steve's February2018 label says the current draw is 8.6A. Is this just a typo?
Steve: It was most likely my mistake. Tag is corrected to 6.8A
I'm still in the process of electronics characterization, so the SR785 is still hooked up. MC3 coil driver signal is broken out to measure the output voltage going to the coil (via Gainx100 SR560 Preamp), but MC is locked.
The ITMX oplev still clipping
The ITMX oplev beam is clipping. It will be corected with locked arm
To obtain a colored version with good contrast of the grayscale image of scattering of light by dust particles on the surface of test mass, got using GigE camera. The original and colored images are attached here.
Seems like as a result of my recent poking around at 1X6, MC3 is more glitchy than usual (I've noticed that the IMC lock duty cycle seems degraded since Tuesday). I'll try the usual cable squishing voodo.
gautam 8.15pm: Glitches persisted despite my usual cable squishing. I've left PSL shutter closed and MC watchdog shutdown to see if the glitches persist. I'll restore the MC a little later in the eve.
Per discussion today eve, barring objections, I will do the following tomorrow morning:
I finished the re-soldering work today, and have measured the coil driver noise pre-Mods and post-Mods. Analysis tomorrow. I am holding off on re-installing the board tonight as it is likely we will have to tune all the loops to make them work with the reduced range. So ETMX will remain de-commissioned until tomorrow.
I decided to take a quick look at the data. Changes made to the ETMX coil driver board:
I also took the chance to check the integrity of the LM6321 ICs. In the past, a large DC offset on the output pin of these has been indicative of a faulty IC. But I checked all the ICs with a DMM, and saw no anomalies.
Measurement condition was that (i) the Fast input was terminated to ground via 50ohm, (ii) the Bias input was shorted to ground. SR785 was used with G=100 Busby preamp (in which Steve installed new batteries today, for someone had left it on for who knows how long) for making the measurement. The voltage measurement was made at the D-Sub connector on the front panel which would be connected to the Sat. Box, with the coil driver not connected to anything downstream.
Summary of results:
[Attachment #1] - Noise measurement out to 800 Hz. The noise only seems to agree with the LISO model above 300 Hz. Not sure if the low-frequency excess is real or a measurement artefact. Tomorrow, I plan to make an LPF pomona box to filter out the HF pickup and see if the low-frequency characteristics change at all. Need to think about what this corner freq. needs to be. In any case, such a device is probably required to do measurements inside the VEA.
[Attachment #2] - Noise measurement for full SR785 span. The 19.5 kHz harmonics are visible. I have a theory about the origin of these, need to do a couple of more tests to confirm and will make a separate log.
[Attachment #3] - zip of LISO file used for modeling coil driver. I don't have the ASCII art in this, so need to double check to make sure I haven't connected some wrong nodes, but I think it's correct.
Measurements seem to be consistent with LISO model predictions.
*Note: Curves labelled "LISO model ..." are really quad sum of liso pred + busby box noise.
My main finding tonight is: With the increased series resistance (400 ohm ---> 2.25 kohm), LISO modeling tells me that even though the series resistance (Johnson noise) used to dominate the voltage noise at the output to the OSEM, the voltage noise of the LT1125 in the bias path now dominates. Since we are planning to re-design the entire bias path anyways, I am not too worried about this for the moment.
I will upload more details + photos + data + schematic + LISO model breakdown tomorrow to a DCC page.
gautam noon 21 June 2018: I was looking at the wrong LISO breakdown curves. So the input stage Op27 voltage noise used to dominate. Now the Bias path LT1125 voltage noise dominates. None of the conclusions are affected... I've uploaded the corrected plots and LISO file here now.
Initial tests look promising. Local damping works and I even locked the X arm using POX, although I did it in a fake way by simply inserting a x5.625 (=2.25 kohm / 400 ohm) gain in the coil driver filter banks. I will now tune the individual loop gains to account for the reduced actuation range.
Now I have changed the loop gains for local damping loops, Oplev loops, and POX locking loop to account for the reduced actuation range. The dither alignment servo (X arm ASS) has not been re-commissioned yet...
We may lost the UL magnet or LED
I think if the magnet fell off, we would see high DC signal, and not 0 as we do now. I suspect satellite box or PD readout board/cabling. I am looking into this, tester box is connected to ITMY sat. box for now. I will restore the suspension later in the evening.
Suspension has now been restored. With combination of multimeter, octopus cable and tester box, the problem is consistent with being in the readout board in 1X5/1X6 or the cable routing the signals there from the sat. box.
For a series resistance of 4.5 kohm, we are suffering from the noise-gain amplified voltage noise of the Op27 (2*3.2nV/rtHz), and the Johnson noise of the two 1 kohm input and feedback resistances. As a result, the current noise is ~2.7 pA/rtHz, instead of the 1.9 pA/rtHz we expect from just the Johnson noise of the series resistance. For the present EX coil driver configuration of 2.25 kohm, the Op27 voltage noise is actually the dominant noise source. Since we are modeling small amounts (<1dB) of measurable squeezing, such factors are important I think.
[Attachment #1] --- Sketch of the fast signal path in the coil driver board, with resistors labelled as in the following LISO model plots. Note that as long as the resistance of the coil itself << the series resistance of the coil driver fast and slow paths, we can just add their individual current noise contributions, hence why I have chosen to model only this section of the overall network.
[Attachment #2] --- Noise breakdown per LISO model with top 5 noises for choice of Rseries = 2.25 kohm. The Johnson noise contributions of Rin and Rf exactly overlap, making the color of the resulting line a bit confusing, due to the unfortunate order of the matplotlib default color cycler. I don't want to make a custom plot, so I am leaving it like this.
[Attachment #3] --- Noise breakdown per LISO model with top 5 noises for choice of Rseries = 4.5 kohm. Same comments about color of trace representing Johnson noise of Rin and Rf.
Possible mitigation strategies:
I've chosen to ignore the noise contribution of the high current buffer IC that is inside the feedback loop. Actually, it may be interesting to compare the noise measurements (on the electronics bench) of the circuit as drawn in Attachment #1, without and with the high current buffer, to see if there is any difference.
This study also informs about what level of electronics noise is tolerable from the De-Whitening stage (aim for ~factor of 5 below the Rseries Johnson noise).
Finally, in doing this model, I understand that the observation the voltage noise of the coil driver apparently decreased after increasing the series resistance, as reported in my previous elog. This is due to the network formed by the fast and slow paths (during the measurement, the series resistance in the slow path makes a voltage divider to ground), and is consistent with LISO modeling. If we really want to measure the noise of the fast path alone, we will have to isolate it by removing the series resistance of the slow bias path.
Comment about LISO breakdown plots: for the OpAmp noises, the index "0" corresponds to the Voltage noise, "1" and "2" correspond to the current noise from the "+" and "-" inputs of the OpAmp respectively. In future plots, I'll re-parse these...
I wanted to investigate my coil driver noise measurement technique under more controlled circumstances, so I spent yesterday setting up various configurations on a breadboard in the control room. The overall topology was as sketched in Attachment #1 of the previous elog, except for #4 below. Summary of configurations tried (series resistance was 4.5k ohm in all cases):
Attachment #1: Picture of the breadboard setup.
Attachment #2: Noise measurements (input shorted to ground) with 1 Hz linewidth from DC to 4 kHz.
Attachment #3: Noise measurements for full SR785 span.
Attachment #4: Apparent coupling due to PSRR.
Attachment #5: Comparison of low frequency noise with and without the LM6321 part of the fast DAC path implemented.
All SR785 measurements were made with input range fixed at -42dBVpk, input AC coupled and "Floating", with a Hanning window.
For the upcoming vent, we'd like to rotate the SOS towers to correct for the large YAW bias voltages used for DC alignment of the ITMs and ETMs. We could then use a larger series resistance in the DC bias path, and hence, reduce the actuation noise on the TMs.
Today, I used the calibrated Oplev error signals to estimate what angular correction is needed. I disabled the Oplev loops, and drove a ~0.1 Hz sine wave to the EPICS channel for the DC yaw bias. Then I looked at the peak-to-peak Oplev error signal, which should be in urad, and calibrated the slider counts to urad of yaw alignment, since I know the pk-to-pk counts of the sine wave I was driving. With this calibration, I know how much DC Yaw actuation (in mrad) is being supplied by the DC bias. I also know the directions the ETMs need to be rotated, I want to double check the ITMs because of the multiple steering mirrors in vacuum for the Oplev path. I will post a marked up diagram later.
Steve is going to come up with a strategy to realize this rotation - we would like to rotate the tower through an axis passing through the CoM of the suspended optic in the vertical direction. I want to test out whatever approach we come up with on the spare cage before touching the actual towers.
Here are the numbers. I've not posted any error analysis, but the way I'm thinking about it, we'd do some in air locking so that we have the cavity axis as a reference and we'd use some fine alignment adjust (with the DC bias voltages at 0) until we are happy with the DC alignment. Then hopefully things change by so little during the pumpdown that we only need small corrections with the bias voltages.
Oplev error signal readback
This bad connection is coming back
PRM watchdog was tripped around 7:15am PT today morning. I restored it.
For the heater setup on EY table, I EQ-stopped ETMY. Only the face EQ stops (3 on HR face, 2 on AR face) were engaged. The EY Oplev HeNe was also shutdown during this procedure.
Yesterday I inspected this BS oplev viewport. The heavy connector tube was shorting to table so It was moved back towards the chamber. The connection is air tight with kapton tape temporarly.
The beam paths are well centered. The viewport is dusty on the inside.
The motivation was to improve the oplev noise.
When I came in this morning:
Checking status of slow machines, it looked like c1sus, c1aux, and c1iscaux needed reboots, which I did. Still PMC would not lock. So I did a burtrestore, and then PMC was locked. But there seemed to be waaaaay to much motion of MCREFL, so I checked the suspension. The shadow sensor EPICS channels are reporting ~10,000 cts, while they used to be ~1000cts. No unusual red flags on the CDS side. Everything looked nominal when I briefly came in at 6:30pm PT yesterday, not sure if anything was done with the IFO last night.
Pending further investigation, I'm leaving all watchdogs shutdown and the PSL shutter closed.
A quick look at the Sorensens in 1X6 revealed that the +/- 20V DC power supplies were current overloaded (see Attachment #1). So I set those two units to zero until we figure out what's going on. Possibly something is shorted inside the ITMX satellite box and a fuse is blown somewhere. I'll look into it more once Steve is back.
[koji, steve, gautam]
We debugged this in the following way:
So for now, the power cable to the box is disconnected on the back end. We have to pull it out and debug it at some point.
Apart from this, megatron was un-sshable so I had to hard reboot it, and restart the MCautolocker, FSSslowPy and nds2 processes on it. I also restarted the modbusIOC processes for the PSL channels on c1auxex (for which the physical Acromag units sit in 1X5 and hence were affected by our work), mainly so that the FSS_RMTEMP channel worked again. Now, IMC autolocker is working fine, arms are locked (we can recover TRX and TRY~1.0), and everything seems to be back to a nominal state. Phew.
The trillium interface box was removed from the rack.
The problem was the incorrect use of an under-spec TVS (Transient Voltage Suppression) diodes (~ semiconductor fuse) for the protection circuit.
The TVS diodes we had had the breakdown voltages lower than the supplied voltages of +/-20V. This over-voltage eventually caused the catastrophic breakdown of one of the diodes.
I don't find any particular reason to have these diodes during the laboratory use of the interface. Therefore, I've removed the TVS diodes and left them unreplaced. The circuit was tested on the bench and returned to the rack. All the cables are hooked up, and now the BRLMs look as usual.
- The board version was found to be D1000749-v2
- There was an obvious sign of burning or thermal history around the components D17 and D14. The solder of the D17 was so brittle that just a finger touch was enough to remove the component.
- These D components are TVS diodes (Transient Voltage Suppression Diodes) manufactured by Littelfuse Inc. It is sort of a surge/overvoltage protector to protect rest of the circuit to be exposed to excess voltage. The specified component for D17/D14 was 5.0SMMDJ20A with reverse standoff voltage (~operating voltage) of 20V and the breakdown voltage of 22.20V(min)~24.50V(max). However, the spec sheet told that the marking of the proper component must be "5BEW" rather than "DEM," which is visible on the component. Some search revealed that the used component was SMDJ15A, which has the breakdown voltage of 16.70V~18.50V. This spec is way too low compared to the supplied voltage of +/-20V.
The idea we are going with to push the coil driver noise contribution down is to simply increase the series resistance between the coil driver board output and the OSEM coil. But there are two paths, one for fast actuation and one that provides a DC current for global alignment. I think the simplest way to reduce the noise contribution of the latter, while preserving reasonable actuation range, is to implement a precision DC high-voltage source. A candidate that I pulled off an LT application note is shown in Attachment #1.
If all this seems reasonable, I'd like to prototype this circuit and test it with ETMX, which already has the high series resistance for the fast path. So I will ask Steve to order the OpAmp and transistors.
Bah! Too complex.
The wall StripTool indicated that the IMC wasn't too happy when I came in today. Specifically:
The last time this happened, it was due to the Sorensens not spitting out the correct voltages. This time, there were no indications on the Sorensens that anything was funky. So I just disabled the MCautolocker and figured I'd debug later in the evening.
However, around 5pm, the shadow sensor values looked nominal again, and when I re-enabled the local damping, the MC REFL spot suggested that the local damping was working just fine. I re-enabled the MCautolocker, MC re-locked almost immediately. To re-iterate, I did nothing to the electronics inside the VEA. Anyways, this enabled us to work on the X arm ASS (next elog).
Independent from the problems the vertex machine has been having (I think, unless it's something happening over the shared memory network), I noticed on Friday that the ETMX watchdog was tripped. Today, once again, the ETMX watchdog was tripped. There is no evidence of any abnormal seismic activity around that time, and anyways, none of the other watchdogs tripped. Attachment #1 shows that this happened ~838am PT today morning. Attachment #2 shows the 2k sensor data around the time of the trip. If the latter is to be believed, there was a big impulse in the UL shadow sensor signal which may have triggered the trip. I'll squish cables and see if that helps - Steve and I did work at the EX electronics rack (1X9) on Friday but this problem precedes our working there...
OK, how about this:
The question still remains of how to combine the fast and bias paths in this proposed scheme. I think the following approach works for prototyping at least:
In the longer term, perhaps the Satellite Box revamp can accommodate a bias voltage summation connector.
I have neglected many practical concerns. Some things that come to mind:
Today while Rich Abbott was here, Koji and I had a brief discussion with him about the HV amplifier idea for the coil driver bias path. He gave us some useful tips, perhaps most useful being a topology that he used and tested for an aLIGO ITM ESD driver which we can adapt to our application. It uses a PA95 high voltage amplifier which differs from the PA91 mainly in the output voltage range (up to 900V for the former, "only" 400V for the former. He agrees with the overall design idea of
He also gave some useful suggestions like
I am going to work on making a prototype version of this box for 5 channels that we can test with ETMX. I have been told that the coupling from side coil to longitudinal motion is of the order of 1/30, in which case maybe we only need 4 channels.
A brief follow-up on this since we discussed this at the meeting yesterday: the attached DV screenshot shows the full 2k data for a period of 2 seconds starting just before the watchdog tripped. It is clear that the timescale of the glitch in the UL channel is much faster (~50 ms) compared to the (presumably mechanical) timescale seen in the other channels of ~250 ms, with the step also being much smaller (a few counts as opposed to the few thousand counts seen in the UL channel, and I guess 1 OSEM count ~ 1 um). All this supports the hypothesis that the problem is electrical and not mechanical (i.e. I think we can rule out the Acromag sending a glitchy signal to the coil and kicking the optic). The watchdog itself gets tripped because the tripping condition is the RMS of the shadow sensor outputs, which presumably exceeds the set threshold when UL glitches by a few thousand counts.
Here is an other big one
I took another pass at this. Here is what I have now:
Attachment #1: Composite amplifier design to suppress voltage noise of PA91 at low frequencies.
Attachment #2: Transfer function from input to output.
Attachment #3: Top 5 voltage noise contributions for this topology.
Attachment #4: Current noises for this topology, comparison to current noise from fast path and slow DAC noise.
Attachment #5: LISO file for this topology.
Looks like this will do the job. I'm going to run this by Rich and get his input on whether this will work (this design has a few differences from Rich's design), and also on how to best protect from HV incidents.
I had a very fruitful discussion with Rich about this circuit today. He agreed with the overall architecture, but made the following suggestions (Attachment #1 shows the circuit with these suggestions incorporated):
If all this sounds okay, I'd like to start making the PCB layout (with 5 such channels) so we can get a couple of trial boards and try this out in a couple of weeks. Per the current threat matrix and noises calculated, coil driver noise is still projected to be the main technical noise contribution in the 40m PonderSqueeze NB (more on this in a separate elog).
Glitch, small amplitude, 350 counts & no trip.
The second big glich trips ETMX sus. There were small earth quakes around the glitches. It's damping recovered.
All suspension tripped. Their damping restored. The MC is locked.
ITMX-UL & side magnets are stuck.