Attached is captured beatnote frequency between Aug 27th 2019 to Today, Sep 5th 2019.

I have attached a few more zoomed-in plots as well. This code will serve us in the future as well. Only the green regions are where both cavities were locked and cavity heater PID was engaged. This data involves the experiment time of CTN:2406 and hence have the disturbances of that time as well.

Essentially, there is no fixed stability number one can really quote for this PID. It looks stable in regions, but at times it shoots up or down randomly. Maybe some of them are because of me doing something on the table, but some are late at night which can only be explained by the movement of ghosts.

fromFBread.py updated

fromFBread.py now has an optional flag -d for decimating the read data, so that smaller files are created. Example: -d 160 will decimate the data by a factor of 160 making a sampling rate of 0.1 Hz. It calculates mean of the data, for a block size of 160 (corresponding to 10s) and also calculates standard deviation in this block and adds that as additional columns in the read data. Hence the plots attached here have uncertainties as well.

Code and Data

I was looking at the error signals of PMC servo and found some systematic oscillations in NPMC error signal at around 500 Hz.

So, I connected back the beatnote to the Marconi PLL and used the error signal from there to measure beatnote - NPMC error signal cross-spectrum. Here, the channel marked Mixer Out on the Servo card is FP3Test port which is marked as 'Mon1' on the schematic (latest schematic). This channel is essentially the output of mixer buffered with gain 1 through an LT1128. Note that currently on North side, we are using FP2Test to send the PD signal in with an external mixer.

Something is either physically wrong near North PMC or the card might have developed some bug, which needs to be tested.

Code and Data

• Changed North FSS Fast gain back to 10 dB.
• Increased threshold for gain cycling to 0.1 on both FSS paths. This was probably the reason for increased BN noise yesterday.
• Mokuis again taking signal from a 20 dB coupler.
• Changed input range of mokuback to 1Vpp.
• Marconi and PLL autolocker are back on.

There's an error here. The factor of 2 should not be there. Correct equation would be:

as mentioned in the quoted post CTN:1874 as well.

I have added these parameters to this wiki page.

I opened the NPMC servo box and ran TF analysis up to all test points and compared with their zero model. Everything looks normal in these transfer functions.
So the oscillations might be happening due to closing of the loop. I'll take a measurement of total open-loop transfer function next with the PDs and PZT in the loop.

Updated schematics and Zero Model

Code and Data

It turns out that the noise is either injected through the EOMs or through some mechanical oscillation in the table or through RFPD SN020. The servo card was working normally. However, since I had opened up the box, I used this opportunity to increase the first stage gain on the FP2test input and also made it 50-ohm input impedance by adding a 51 Ohm resistor in parallel to R8. Now, beware that on the actual board, R5 and R8 labels are actually swapped. This helped keep the gain set by AD602 to 7 dB now, instead of 30 dB maximum. This way we are able to increase the gain more instead of getting saturated. The hunt for 500 Hz bump would move to finding on table sources now.

• Changed detector to NF1811
• Changes in NPMC servo card. See CTN:2421.
• Updated noise budget with new bulk loss angle estimates as presented in LIGO-G1901676 and LIGO-G1901765

As presented in LIGO-G1901676 and LIGO-G1901765, I updated the noise budget with new calculation using 8e-5 rad as the bulk loss angle value. Here is a comparison of how coating brownian noise estimate has been moving with these changes.

Checkout the latest noisebudget. 500 Hz peak is gone now. This could be because of any of the changes I made in past 2 days:

1) NPMC servo gain increased. I changed R8 from 9.09k to 51 Ohms. That increased the gain by a factor of 200 which is equivalent to 46 dB. The slider for AD602 gain decreased by 23 dB to remain below oscillations wiht a 1dB gain margin. So maybe this reduced the noise a lot more (even though it is still present)

2) I did replace a normal beam dump (91,45) on the reflection path from North PMC with a triangular black glass cavity and I oriented another one (94,30) just before NPMC blocking rejection from a PBS as it was oriented horribly wrong.

3) We also changed the detector to NF1811, but that hasn't resulted in changing this feature in the past.

Note that the expected noise has now decreased roughly by 3 times in the range 200 Hz to 500 Hz. See CTN: 2423

• I added C3:PSL-PRECAV_BEATNOTE_FREQ_SLOPE2 as a new channel. This holds the seconds derivative of precavtybeatnote frequency in Hz/s/s.
• The slope calculation code is separated now from sign update code.
• Slope is now calculated with moving averageof changing quantity over 10s. This gives much better estimates of both the slopes.
• Sign update is also more robust now. It checks if frequency counter is stuck for atleast 0.5s and last read BN frequency was less than 5 MHz.
• Sign update also has a dead time on its trigger of 1 minute to avoid changing sign multiple times in a single jump.
• So far, the sign update code is working eallygood. Hopefully, the cavity temperature control will not loose track now.

Present understanding

• CTN:2035 was the last measurement of cavity shield heater resistances. These were reported as 85.6 Ohm and 156.8 Ohm.
• So Andrew, undestandably, configured the EPICs channel such that to put in 1 W power on the south heater, more current was driven through it in comparison to North heater.
• This should have worked well, but over the past year atleast, I have been noticing that south heater is much more effective than the north one.

My hypothesis

• I remeasured the resistances and they came almost same: South 88.4 Ohm and North was 159.7 Ohm.
• It could have been the case that North side was some parasitic resistance (due to bad connection with the heater) in series, increasing its effective resistance to 159.7 Ohm.
• This would mean that the heating part of the resistance is still almost same as South, but because we sense it larger, we send less current on this coil.
• I assume Tara or whoever put on these heaters, they made sure the length of the wires were almost same, so it is a fairly good assumption that the heating part of resistance is same.
• This also matches with the observation. North side is less effective in heating, because we send less current to it.

Changes made today

• I changed the dB files for the EPICs channels, so that same 88.4 Ohm resistance is assumed for both North and South heaters.
• From the performance so far, and looking at the second derivative of the beatnote signal, I feel the heaters are more balanced now.
• This should help the PID as the actuation is not biased now.
• I have made the hard actuation limits same on both ends of PID at 1.1 W.
• We might need to retune the PID to get new PID constants.

Gain was way too low

• The header H1 was not shorted on the board.
• This reduced the gain of first stage to only 2, while it wsmeant to be 101!
• I just opened the box and put back in jumper on this header.
• The gain slider for AD602 is at -1 dB, 1 dB below the unstability point.
• Earlier the slider was maxed out to 30dB, so we didn't know if we are close to optimum supression.
• Changes have been updated in the schematic.

Over the weekend, I ran Relay Tuning method for the PID of beatnote frequency control. After CTN:2426 this needed to be done to fix the PID constants to appropriate value. The results of the tuning were:

Critical period Tc = 45.900000000000006
Critical gain Kc = 18.382414309527164
Suggested kp, ki, kd are 3.676482861905433, 0.16019533167343933, 56.25018778715313

RXA: Wah!  precision help here

The relay amplitude was set to 0.5 W and I could see very good sustained oscillations which the code used to get above calculated values.

Testing performance

I tested the performance of PID today. Attached is the convergence of beatnote frequency, which happened in about 20 minutes only to 40 kHz offset value. After that point, the proportional gain of the PID is so high, that the actuator response essentially copies the fluctuations in the beatnote frequency itself. So no more stabilization happens. The integral constant is very low (I think it is required for quick convergence with no overshoot), so to travel this 40 kHz distance, it will probably take hours. But that's fine with us as our photodetectors work well enough with this offset too.

If you see the second plot, the beatnote did not drift beyond +/- 2kHz for over 40 min. I want to see if tonights beatnote will get any better due to this good stabilization.

Code And Data

• Beatnote is stabalized to 27.3 MHz. Changing detector to SN101.
• Reduced South PMC Variable Gain to -2dB from -1dB as it was oscillating. That's probably why 16th september BN was bad.
• Changed bandwidth of Moku for frequency nosie measurement to 2.5 kHz.

Attached is a first attempt at tracing the rays on reflection from the wedged and tilted window together with the cavity mirror.

traceit

I used Sean Leavey's zero and created a ray tracing module for simple purposes which is fast and easy to use. Check out the examples to see the capabilities.

To use, git pull labutils to update and keep labutils /traceit in your python path.

More info about each ray can be seen by layout['R4'] kind of statements. Or just write layout.rays to see info about all rays. This includes their vector positions, the origin phenomenon, and ray etc.

I know a lot can be done to make this look better. But I'm not going to dive into developing this module right now. However, suggestions on how to make the ray trace diagram more useful are welcome so that I can make it more informational.

Seems like most of the reflections would be bunched together in two directions where we should put the beam dumps.

Today I ran the two codes with the same parameter values to check if the effective reflectivities calculated during the calculation of thermo-optic nose matches. They do match exactly actually. Attached is an over plot.

Coating effective coefficient of thermo-refractive effect comes out to be:

Old Code: 8.61(46)e-05 K**-1

New Code: (8.59+/-0.21) e-05 K**-1

So this discrepancy was not really there. I was just comparing apples with oranges earlier.

Also, I found that the effective reflectivities from the top surface of layers calculated in Evans et a. PRD 78, 102003 (2008) were different from reflectivities calculated by matrix method in Hong et al. PRD 87, 082001 (2013). It turned out that the sign before phase in Eq. B3 and B4 in Evans et a. PRD 78, 102003 (2008) were opposite to what comes from the matrix method. After discussions with Gabriele, I came to the conclusion that this sign only creates a difference of giving complex conjugates of effective reflectivity. But to have consistency, I have corrected the sign of phase in the new code. Attached is a comparison of old effective reflectivity and the new one. One can see that the imaginary values are opposite in sign but everything else matches perfectly. Also, this has no effect on the effective coating coefficient of thermo-refraction.

Gabriele suggested that I plot the estimated coating Brownian noise for different values of bulk and shear loss angles and compare with the old code. The expectation is that the estimated noise should vary smoothly and should agree with old result where bulk and shear loss angles are taken the same. In the first plot, I have plotted estimate coating Brownian noise for CTN experiment over the nominal range of bulk and shear loss angles. Since this range doesn't really have an overlap with equal bulk and shear loss angle case, I made another plot in which such a square region is chosen. We can see that the estimated coating Brownian noise is slightly higher (roughly 2.5% from CTN:2390 figure 2).

Let me know if someone can think of a better viewing angle or more insights to take out from this. Code is attached.

Better Plots attached.

2D Heat Maps

Edit: Added comparison plot as well.

but then we lost the comparison with the old calculation somehow...

With comparison with old estimates:

CTN_Latest_BN_Spec_With_OldEst.pdf

CTN_Daily_BN_Spec_With_OldEst.pdf

Usual noise budget links:

CTN_Latest_BN_Spec.pdf

CTN_Daily_BN_Spec.pdf

Variation of the estimate of coating Brownian noise as beam spot radius on the mirror changes. Plotted in displacement noise as the frequency conversion factor is dependent on cavity length and beam radius also depends on that.

I used the method described in Section V of https://arxiv.org/abs/1406.4467, to do a bayesian inference using mechanical ring-down measurements of coating loss angles by Steve Penn (recently reported in https://dcc.ligo.org/LIGO-G1901676) as a prior. I assumed all probability distributions to be gaussian.  We are clearly not close to measuring the Brownian noise as with present uncertainty limits reported by Penn et al, the prior distribution and the likelihood distribution do not overlap with each other at all, giving us no meaningful result. ​Note the different limits in x-axis on first and second plot.

This was a quick run of this though, just to set up the method. Ideally, I should use integrated noise in some frequency range around 300 Hz to use as the measurement. Also, Steve Penn actually did not report any uncertainty value with his measurements, so I assumed the uncertainty the same as his previous measurement. Looking forward to the more formal release of his data.

Liquid Instrument's Application Engineer at La Jolla told me that connection with moku might be mroe stable if it is directly connected to the computer through a USB cable. It still gets identified with Name, Serial Number or IP address, just the connection is mroerobust. So today, I have connected our moku with USB. I have seen in past couple of weeks that every few days the moku data transfer gets stuck or it fails to connect through LAN. So trying this out.

After the new PID parameters were tuned (CTN:2428), I waited for some time and the beatnote was stably locked to its setpoint of 27.34 MHz for over 2 weeks now. It is a good time to assess the beatnote frequency stabilization. Here I took data of 10 days and plotted it in three different timescales. The standard deviation plotter in light blue is calculated by standard deviation over 10 s of averaging of data. Green background means everything was locked at that time. Other than green would mean that either something was unlocked or there is a gap in the channel data (this case).

Conclusions:

• Over 10 days, the beatnote hardly left +/- 2 kHz zone from the setpoint. Even with one standard deviation far away, the beatnote does note leave +/- 2.5 kHz zone. We are using Moku at 2.5 KHz bandwidth right now.
• Over a day, here it was Sep 28th (Saturday), the beatnote is within +/- 2kHz even up to one standard deviation point. 
• Over 1 hour which was between 1 am to 2 am on Sep 28th, the beatnote was similarly calm.
• In the last plot, I have plotted drift over 60s in beatnote frequency which is our measurement duration. This drift doesn't even cross 1.5 kHz mark.

But

How good is good? We were so bad, I never did this calculation. Are we hitting boundaries of how good the thermal controls can anyway do? Is the remaining noise in beatnote spectrums just scatter noise or there is still room for improvement in beatnote stabilization. Food for thought.

Code and Data

I updated this calculation by adding curves for contribution through bulk loss angle and shear loss angle separately. Rana suspected that shear loss and bulk loss should behave differently with the change in beam radius on the mirrors. But apparently the Hong et al. calculations do not suggest that way. I checked this analytically too. The definitions of Eq. (96) of power spectral densities of coating thickness fluctuation of a particular layer due to Bulk or Shear loss angles have the same dependence on the effective area of the beam on the layer.

The only thing different between the final contribution from these different fluctuations is in the transfer functions mentioned in Table I from bulk and shear noise fields to layer thicknesses and surface height of coating-substrate interface. These are also plotted in a second curve to get an idea of these transfer functions.

Along with the effects of layer thickness and surface height changes to final mirror displacement (effective) via phase change of reflected light is given by parameters q^B and q^S as defined in Eq. (94). These are plotted in the third plot and show the real difference in contribution from bulk and shear. This is in stark contradiction with what Ian just told me. They believe that for a Gaussian pressure profile, the energy stored in shear strain is 3 times higher than that stored in bulk strain. For comparison with a figure of the paper (fig.7.), I plotted the square root of these transfer functions in the fourth plot. However, the paper plots these for Silica and Tantla, not AlGaAs.

My conclusion is that at least in Hong et al.'s treatment, the effect of beam area on the mirror is equal to both bulk and shear contributions (Eq. 96).

I repeated the measurement done by Tara and Evan to update residual NPRO noise in our noise budget.

Conclusions:

• Clearly, the North side's boost isn't working as was reported in CTN:2384
• South side looks as healthy as North in CTN:1504
• Tomorrow I'll update the noisebudget with new residual NPRO noise, for now, the second plot gives the prediction.
• It matches very well with our observed floor of noise in 100Hz to 1 kHz region in the beatnote spectrum.

Code and Data

I and Ian discussed what the transfer functions would look like. Then today, using some old calculations, I put up this notebook which does the calculations for us. The notebook has the calculations typed up in latex too.

This is the first attempt. We have to work on making the EOM path's transfer function closer to the expected model transfer function. And we should use a faster opamp as well, I think.

Edit Fri Oct 4 14:44:33 2019 anchal:

• Replaced op27 with AD829
• Added a buffer at the input of PZT to isolate PZT capacitance from the adder circuit.
• Updated and better circuit schematic coming up by Ian. Removed this image.

Schematic of Plant Model

The nodes at input and output of buffer in the PZT path are connected together. That is wrong. Also, If possible, you should name the elements same as the zero model in the notebook. Anyways, I think we are ready to solder a circuit board.

comment on goodness of Temperature controls:

Since the frequency is wandering by ~3 kHz at the hours time scales, we can estimate the differential cavity temperature to be:

delta T = (1/CTE) * df / (c / lambda) = (1 / 5e-7) * (3 kHz / 300 THz) = 20 micro-K

If someone can plot the NPRO SLOW signals at the similar time scales, we would know what the CMRR is. But I think 20 uK is probably just fine. 

Whoever commented last, suggested a good idea. So I've here plotted the NPRO slow control voltage signals, converted into the inferred temperature of the cavities (see CTN:2415). I'm not so sure which CMRR the anonymous commentator is talking about. More clarification on that would be helpful.

Conclusions:

• As the anonymous commentator said, the ideal temperature difference between the cavities would be around 20 uK, but that is clearly not the case as North Cavity varies more than 0.1 K while the South Cavity is relatively more stable.
• This could also indicate that some external low-frequency noise source drifts the North Lasers frequency by corrupting the FSS.
• The thermal dynamics of the two cavities should be similar unless same driven current causes starkly different emission of heat from the coil heaters (which are supposedly of same length).
• Some more help in understanding these results are required, or atleast a person who can bounce back ideas with me.
• So the can upgrade might be useful afterall. If anything, it will reduce the mysteries related to the cavities locked inside there for the last 5 years.

Code and Data

I have updated the plant model to contain the cavity pole also. Cavity pole is a pair of positive and negative real poles, so it is hard (or maybe impossible) to imitate it exactly with an electronic circuit. Or maybe, my analysis is wrong.

Nevertheless, I have for now made this circuit which has a second-order pole, so it correctly matches the magnitude of the model transfer function up to 1 MHz for both PZT and EOM paths. Note that the elliptical filter is not included in this as we can connect the circuit to Test port 1 which injects just before the filter in LIGO-D0901894. Also, for the gains in EOM path, I had to add some factors to make it the same as the model transfer function. All components are calculated for E12 series resistors and capacitors.

Attached is a pdf of the notebook which contains all the mathematics in latex and a zip file with all files to recreate and further work on this. Ian can use these as support to learn zero further.

On Oct 11th at 15:04:04, the south laser switched off on its own. I would like to know if anyone entered the lab around this time. Koji did mention that our Laser Safety sign outside was blinking, but I have no more information than that. Attached is the data of south PMC reflection DC, which is the first photodiode that measures the laser. It suddenly went to zero, indicating the laser was switched off and the locks did not drive it to this point. I'm also finding that the laser intensity is reducedasit used to saturate the South PMC reflection photodiode when unlocked but presently shows around 5V. I'm trying to put the experiment back to same parameters as before.

Code and Data

South PMC Error signal is showing huge weird capacitor charging type oscillations. Attached is an oscilloscope measurement of it. A more weird thing is that this peak is appearing at random frequencies, that is, if I take single sequence measurements of 1 ms length, I see these peaks occurring at a difference from 100 us to 500 us, randomly.

Changes:

• I had to re-adjust the power going to the PMC to get 9.6mW power falling on the cavities. But I think the knob was more or less in same place.
• I had to restart whole lab (all channels and scripts) as I suspected some channels are not responsive.
• I can't increase the fast gain on southside beyond -7 dB (while this was 14 dB before). so this SPMC oscillations are really hindering badly.
• I also suspect that the NPRO temperature control is not as stable as before on the south side, probably because it was switched off for more than 2 days.
• I'm hoping by tomorrow, this should get better. but still need to investigate the issue on south PMC.

Data

Following up after CTN:2452, the laser safety sign is not working. Hence the lab has been shut down now. All lasers are switched off with key turned to off position. I'll fix the laser safettysign before turning the experiment back one. Possible reasons might be an interlock glitch or bulb fuse.

I have taken the bulb from ATF lab inside sign for now. I'm ordering a new one to replace that soon.

With Laser actuation not connected, I see that the South PMC Servo board is acting up. Firstly, it is not responding to changes made on the ramp when engage is off. This shows that maybe the engage DAC channel is faulty and PMC lock is always on. Need to investigate more on this so for now I have disconnected the PMC PZT from the servo board so that nothing further happens. North side is completely happy and sound.

I have a wierd observation. The following two combination work:

• North PMC locked through South PMC Servo Card and then the frequency locked through North FSS Box.
• South PMC locked through North PMC Servo Card and then the frequency locked through South FSS Box.

But when I connect the South PMC to its own South PMC Servo Card, the PZT output voltage does not change with changes made to Ramp. The other side, works.

I checked the capacitance of the South PMC PZT and it came to about 395 nF which is same as specifications with the error bars. So the PZT isn't bad.

But if I disconnect the South PMC PZT from the South PMC Servo Card, the output voltage at the servo card changes as expected with ramp voltage.

This is very perplexing. I think I need second opinion here to do sanity checks otherwise I'll go mad in the basement.

Updated schematics for reference: South PMC Servo Card

Are you sure that all the cables involved are isolated and there is no polarity inversion? e.g. The unfunctional combination provides HV to GND directry at the cable, for example.

Yeah, the cables are isolated and no inversion could happen.

Even more bizzarething is that it works now!  I'm not halucinating here but the same thing was not working before.I even have lab notes from yesterday when it wasn't working.

This is pretty bad as I don't want to be unaware of something in the lab that caused this. The only other clue here in all this is that the laser intensity changed. We control the intensity of light going into the experiment at (24,110) through the half waveplate before a PBS. Rana told me that the polarization direction of laser coming from NPRO shouldn't change, but since the incident of last Friday, I have had to change the rotation of this half waveplate in both directions to ensure same amound of light reaches the cavity as on North. since no alignment was changed at the south PMC, there is no reason for the mode matching to change drastically there or during the day. but this is the only fishy clue I've got for now.

Presently

Both cavities are locked. with same amount of gains in the FSS and PMC loops in both paths as before. The can's temperature as reached to the setpoint and the beatnote frequency PID is working to take it to 27.34 MHz. I'll set trigger for tonight for beatnote frequency noise measurement if the frequency reaches in the range of the photodiode.We then will know what is the impact on the experiment noise.

• Follow discussion in thread  CTN:2451.
• Changed polling rate of beatnote frequency counter to 1 Hz and the script is updating the EPICS channel at the same rate now.
• Changed BNFreqSignUpdate and BNFreqSlopeCalc scripts accordingly.

Latest BN Spectrum: CTN_Latest_BN_Spec.pdf

Daily BN Spectrum: CTN_Daily_BN_Spec.pdf

• Changed bandwidth of Moku PLL 10 10 kHz until beatnote frequency control settles back normally.
• We are back to same position in noise but there are some extra sharp peaks which need to be looked at. But they are very small.

I have designed a passive circuit that seems to match the ideal transfer functions in shape. Scaling should just be a game of playing with the values of the resistors and capacitors. The phase still seems to be an issue. There is an unwanted phase shift from 0 -> -90.

The next step is trying to finalize the values for the resistors and caps. Possibly model in zero if I have time. Then build and test. Also fix the phase.

I ran the code for aLIGO coating structure and tried to reproduce fig5 and fig7 of Hong et al. paper. It turned out that the derivatives of the complex reflectivity were not matching with the paper. I rewrote the code, with a fresh mind without looking at previous code and voila, after increasing computation time slightly due to more brute force calculations, I was able to reproduce the figures. These figures are attached. Since I do not have access to the data of the figures in the paper (I tried to email the authors but got no replies), I could only try to plot it on the same scale and limits and check the values by eyes. The values seem to match. So I am more confident now to declare that this code completely follows the paper's calculations.

However, this does not change the coating Brownian noise. I have updated the noise budgets at the Daily and Latest plots.

The same thing happened again. This time, not just with the SPMC actuation voltage, but the South Laser slow voltage control was also unresponsive. However, I am not very sure about the latter. This was resolved once the restarted the whole lab. This narrows down the problem to following possibilities:

• One or more channels in an acromag card become unresponsive. It could be the 10.0.1.49 acromag used for South PMC Controls.
• Some device in TCS or QIL lab is interfering by using an IP address reserved for docker services on which the EPICS channels are hosted and python scripts are run.

These still don't explain CTN:2456. Again, since this is an irreproducible error, I will just have to wait for it to happen to gather more clues. Right now, everything is fine and beatnote is traveling towards set frequency.

With discussions with Anchal and some reading

It seems impossible to create a casual circuit with a zero phase shift. (See this for more) 

If we have a circuit with an impulse response h(t) and transfer function H(f)=F[h(t)] where H(-f)=H*(f). For the filter to cause no phase shift then ∠H(f)=0 for a complex exponential input for all f. It is also impossible to have a constant phase shift unless that phase shift is zero.

Therefore "filter does not change the phase at all, then H(f) is a real-valued function, and because of the conjugacy constraint, it is also an even function of f. But then its Fourier transform h(t) is an even function of time, and thus the filter cannot be causal (except in trivial cases): if its impulse response is nonzero for any particular t>0, then it is also nonzero for −t (where −t<0)" 

Because this can't be done casually, it should be done using a Field Programmable Gate Array. Unfortunately, I don't think we have access to one. I am reading up on the Moku FIR Filter builder to find out if we can program it to do what we want.

Today, this problem happened again (check history for details). I have done the following investigative steps:

• Situation: The South PMC PZT Ramp was changing by the EPICS voltage but the PZT displacement value was stuck. Also, the South Laser Slow voltage channel also goes back to some value even after changing it.
• I took out the South PMC box from the rack. During this:
  • I had to disconnect the 9V supply used by acromag DACs for setting gain values and switch values.
  • I also had to disconnect the ethernet cable too.
  • I also disconnected RFPD input to the box, HV input, and PMC PZT.
• The first thing I noticed is that the PMC PZT not following the ramp problem is fixed.
• However, the slow output of the laser was going back to a set value (-6.1046V) without any reason.
• I attached the SPMC RFPD RF (Downconverted) signal back to FPTest1 port the PMC just got locked! This happened even though the engage button was switched off.
  • And again, the PMC PZT ramp stopped causing any changes to the PZT displacement value. This is common when the PMC is locked in a normal manner.
• I detached the SPMC RFPD, the changes were happening normally. So the presence of a strong signal might be the issue.
• But, when I connected the RFPD back, the problem was not there anymore. At this point, I found the following reproducible issue:
  • ​Engage button is off.
  • PZT displacement is following the ramp normally.
  • Slowly change the ramp and at the sweet spot, the PMC gets locked. Remember the engage is still off.
  • Now, changing the ramp doesn't change the PZT displacement anymore.
  • Switching on the engage ON doesn't change anything.
  • Switching it back OFF unlocks the PMC and the PZT displacement starts responding to ramp voltage again.
• That's weird right. Since this was reproducible, I did this a few times and found that the problem doesn't necessarily happen at the locking point. It can happen anywhere. And in that case, the 5th step of switching on the engage does show a difference in locked mode. And again, switching it back OFF resolves the issue.
• At this point, I checked back the slow voltage and it had changed to -6.1086. Now apparently this was the new point it liked to stay on regardless of any changes made to the EPICS channel controlling it.
• I checked if the signal for 'PMC Engage' to the U5A AD602. This signal works as expected. The gain setting signal also works as expected. The Ramp setting signal also works as expected. So there is no evidence that the acromag card or the EPICS channel is causing this issue (at least in the PMC problem).
• It could be that the AD602 switches ON itself due to some reason and doesn't switch off until it explicitly sees a low-high change on its pin 4.
• Also, it could be that the U9 PA85 is the one causing trouble. But I doubt that as the behavior matches more with engaging the lock upstream at U5A AD602.
• I restarted the lab by restarting the channels and scripts (this is done in a preset protocol by restartAll command).
• This resolved the issue with the slow voltage of the laser. But the PMC problem persists. In fact, it is not locking now.

I'll take another inspection with a fresh mind next time. This problem needs to be resolved as we can't leave some unexplained phenomenon to keep happening in the lab.

I reduced the power falling on the PMC to ensure the high signal level isn't causing this problem. It was not. The problem still persisted.

Then, I did this reproducible step (quoted below)again, but this time I had a small 10 mV signal from SR785 going into FP2TEST and I was taking transfer function to TP2. If the U5A AD602 is switched off by the Blanking pin, the transfer function should remain null. This gave me a way of checking if the AD602 is wrongly getting switched on on its own.

• To start, I kept Engage OFF. This gave a voltage of 4.41 V at pin 4 of U5A AD602. So it should be shut off.
• The PZT voltage was about 97 V at this point.
• The transfer function was flat to about -80 dB from 10Hz to 100Hz.
• Then I started scanning the RAMP voltage of PZT. As the PZT voltage reached near 70V value, the PMC got locked on its own.
• The transfer function value jumped suddenly to between -20 dB to 0 dB (an increase by about 70 dB).
• The gain on AD602 was set to 0dB. So if it got on its own, the transfer function was expected to be 0dB.
• The gate voltage at pin 4 of U5A AD602 was still 4.41 V. So ideally, it should still be off.
• I have attached the data I captured. During swept sine, the ramp was being increased and we see a jump clearly when the PMC got locked on its own.

This is good evidence in my opinion that the AD602 at U5A is faulty. I need comments on this conclusion. If I don't hear otherwise by tomorrow noon, I'll start working on replacing it.

Data

Edit Thu Oct 31 10:26:50 2019

Issue fixed. See CTN:2469.