In the last couple days, as the IMC ringdowns have been going on, we have noticed that the MC is behaving bad. Misaligning, drifting, etc.
Gautam told me a horror story about him, Koji, and melted wires inside the sat boxes.
I said, "Its getting too hot in there. So let's take the lids off!"
So then we:
After some minutes, we saw no drifting. So maybe my theory of "hot heatsink partially shorting a coil current to GND through partially melted ribbon cable" makes sense? IF this seems better after a month, lets de-lid all the optics.
Let's look at some longer trends and be very careful next to MC2 for the next 3 days! I have put a dangerous mousetrap there to catch anyone who walks near the vacuum chamber.
gautam: the grounding situation per my assessment is that the shield of all the IDC cables are connected to a common metal strip at 1X5 - but in my survey, I didn't see any grounding of this strip to a common ground.
The goal tonight was to go through the locking scripts to see if I could recover the state from November 2019, when I could have the arm lengths controlled by ALS, and sit at zero CARM offset with the PRMI remaining locked and the arm powers fluctuating between 0-300. The IR-->ALS transitions went smoothly tonight, and the PRMI locking was also fairly robust when the CARM offset was large, but was less good when reduced to 0. Nevertheless, it is good to know that the system can be restored to the state it was late last year. Next step is to figure out how to keep the PRMI locked and get the AO path engaged, this was what I was struggling with before the new year.
To fix the apparent slowness of execution of the caput commands on megatron, I changed the "ewrite" macro in the mcup and mcdown scripts to use ezcawrite instead of caput. The old lines are simply commented out, and can be reverted to at any point if we so desire. After these changes, we saw that both scripts complete execution much faster.
I resume my IMC ringdown activities now that the IMC is aligned again.
To avoid any accidental misalignments Gautam turned off all the inputs to the WFS servo.
I set up a PD and a lens as in attachment 1 (following Gautam's setup).
I connect the REFL, TRANS and INPut PDs to the oscilloscope.
I connect a Siglent function generator to the AOM driver. I try to shut off the light to the IMC using 1V DC waveform and pressing the output button manually. However, it produced heavily distorted step function in the PMC trans PD.
I use a square wave with a frequency of 20mHz instead with an amplitude of 0.5V offset of 0.25V and dutycycle of 1% so there will be minimal wasted time in the off state. I get nice ringdowns (attachment 2) - forgot to take pictures. The autolocker slightly misaligns the M2 every time it is acting, so I manually align it everytime the IMC gets unlocked.
Data analysis will come later.
I remove the PD and lens and reenable the WFS servo inputs. The IMC locks easily. The WFS outputs are very different than 0 now though.
Over the past few days, I have been trying to make measurements of the phase modulation transfer function by modulating the X end laser PZT via PLL.
The setup was modified every time during the experiment in the same manner as mentioned in elog 15148.
I could not make the PLL lock for long enough to take a proper TF measurement, resulting in TFs that look like Attachment 1. The next step would be to use the method of a delay line frequency discriminator instead of the PLL.
Comments about locking with LB1005 PI controller:
After some discussion with Gautam, I decided to build more spare channels into the new c1psl machine. This is anticipation of adding new laser and ISS channels in the near future, to avoid having to disconnect the installed chassis and pull it out of the rack. The spare channels will be wired to DB37M feedthroughs on the front side of the chassis, with enough wire length to be able to pull the breakout boards out of the front to reconfigure their wiring as needed (e.g., split off channels onto a separate connector).
To have enough overhead, this will require installing 1 additional ADC unit (XT1221) and 1 additional DAC (XT1541). We have enough spare BIO channels among the existing units (both sinking and sourcing). This will give us:
The updated c1psl chassis wiring assignments are attached. It adds 4 new DB37M connectors for the spare channels (highlighted in yellow) and fixes one typo Jordan found while wiring today. The most current spreadsheet is available here.
We found that the caput commands were taking much longer to execute on megatron than on pianosa (for example). Suspecting that this had something to do with the fact that megatron was using EPICS binaries from the shared NFS drive which were compiled for a much older OS, I installed the latest stable release of EPICS on megatron. The new caput commands execute much faster. I also added the local EPICS directory to the head of the $PATH variable used by the MC autolocker and FSS Slow scripts, so that they use the new caput command. But mcup is still slow - maybe my new path definition isn't picked up and it is still using the NFS binaries? To be looked into...
There were a bunch of medm processes stalled on megatron (connected with screenshot taking). To see if they were interfering with the other scripts, I killed all of the medm processes, and commented out the line in the crontab that runs the screenshots every 10 mins. Let's see if this improves stability.
With all of the shaking (man-made and divine), it was a hard to debug this problem. Summary of fixes:
At least the DC indicators are telling me that the IMC locking is back to a somewhat stable state. I have not yet checked the frequency noise / RIN.
To conclude my PMC noise investigations: Attachment #1 shows the PMC noise inferred from the calibrations earlier in this thread and the fitted OLTF for the PMC loop. Attachment #2 compares the frequency noise (inferred from the error point of the PMC servo) when the IMC is locked / unlocked. I don't know what to make of the fact that the PMC suggests improvement from ~20 Hz onwards already - does this mean that the NPRO noise model is wrong by 1 order of magnitude at 30 Hz?
While I initially thought the 1/f^2 rise below ~100 Hz is attributable to the IMC cavity length fluctuations, I found that this profile is present even in the measurement with the PSL shutter closed. I am not embarking on a detailed PMC noise budgeting project for now. Note however that we are not shot noise limited anywhere in this measurement band.
It's fine to block the WFS while doing ringdowns but please return the config to normal so I don't have to spend time every night recovering the interferometer before doing the locking. As I mention in that post, it is possible to do this in a non-invasive way without having to run any extra cables / permanently block any beams. If there is some issue with the data quality, then we can consider a new setup. But I see no reason to re-invent the wheel.
The IMC was also massively misaligned. I had to re-align both MC1 and MC2 to recover the lock. I took this opportunity to reset the WFS offsets. Please do not disturb the alignment of the existing optical layout unless you verify that everything is working as it should be after your changes.
And for whatever reason, ITMX was misaligned. If you do something with the interferometer, no matter how minor it seems, please leave a note on the ELOG. It will save many painful debugging hours.
As I fix these, the seismic activity has gone up . I'll wait around for an hour, but not an encouraging restart to the locking 😢
Zeroth order IMC ringdown
Following Gautam's IMC ringdown setup, I took the the REFL PD form the PMC ringdown experiment and installed it in the IMC REFL path blocking WFS2 (Attachment 1).
Zeroth order IMC ringdown setup
Following Gautam's IMC ringdown setup, I took the REFL PD form the PMC ringdown experiment and installed it in the IMC REFL path blocking WFS2 (Attachment 1).
I also ran a BNC cable from the transmission PD that Gautam installed on the IMC table to the vertex where the signals are measured on the scope.
I offloaded the WFS servo output values onto the MC alignment (using the WFS servo relief script) so that its dc values would be correct when the servo is off.
Unfortunately, it seems like the script severely misaligned the MC mirrors at some point when the MC got unlocked. We should fix the script such that it stops when the offloading is complete.
We got the MC realigned but left it in a state where it is not locking easily.
I've also been noticing that the IMC Autolocker scripts are running rather sluggishly on Megatron recently. Some evidence - on Feb 11 2019, the time between the mcup script starting and finishing is ~10 seconds (I don't post the raw log output here to keep the elog short). However, post upgrade, the mean time is more like ~45-50 seconds. Rana mentioned he didn't install any of the modern LIGO software tools post upgrade, so maybe we are using some ancient EPICS binaries. I suspect the cron job for the burt snapshot is also just timing out due to the high latency in channel access. Rana is doing the software install on the new rossa, and once he verifies things are working, we will try implementing the same solution on megatron. The machine is an old Sun Microsystems one, but the system diagnostics don't signal any CPU timeouts or memory overflows, so I'm thinking the problem is software related...
The burt snapshotting is still not so reliable - for whatever reason, the number of snapshot files that actually get written looks random. For example, the 14:19 backup today got all the snaps, but 15:19 did not. There are no obvious red flags in either the cron job logs or the autoburt log files. I also don't see any clues when I run the script in a shell. It'll be good if someone can take a look at this.
The 24 V Sorenson (2nd from bottom) in the small rack west of 1x2 was repurposed to 12V 600 mA, and was run to a terminal block on the north side of 1X1. Cables were routed underneath 1X1 and 1X2 to the terminal blocks. 12V was then routed to the PSL table and banana clip terminals were added.
In preparation for resuming IFO locking activities, I measured the ALS noise with the arm lengths locked to IR, AUX laser frequencies locked to the arm lengths. Looks promising (y-axis units are Hz/rtHz).
To avoid driving the PA85 without the HV rails connected, I removed R23. This was re-installed after my characterization.
Since we do the demodulation of the PMC PDH signal off this servo board, the I/F mixer output is connected to the "FP1test" front panel LEMO input.
Using some Pomona mini-grabbers, I measured the electronic TFs between various points on the circuit. There were no unexpected features, the TFs all have the expected shape as per the annotations on the DCC schematic. I did not measure down to 0.1 Hz to confirm the low frequency pole implemented by U6, and I also didn't measure the RF low pass filter at the input stage (expected corner frequency is 1 MHz).
After replacing the IC, I measured the transfer function between TP1 and TP2 for various values of the control voltage applied to pin 4A on the P1 connector, varying between +/- 5 V DC.
PZT Capacitance measurement
I confirmed that the PZT capacitance is 225 nF. The measurement was made using an LCR meter connected to the BNC cable delivering the HV to the PZT, at the 1X1 rack end.
After re-soldering R23, I put the board back into its Eurocrate, and was able to lock the PMC. For subsequent measurements, the PSL shutter was closed.
Looks like a M=4.6 earthquate in Barstow,CA tripped all the suspensions. ITMX got stuck. I restored the local damping on all the suspensions just now, and freed ITMX. Looks like all the suspensions damp okay, so I think we didn't suffer any lasting damage. IMC was re-aligned and is now locked.
Zero order beam PMC ringdown
On Wednesday I installed 3 PDs (see attached photos) measuring:
1. The input light to the PMC. Flip-mirror was installed (sorry Shruti) on the beam path to the fiber coupler.
2. Reflected light from the PMC.
3. PMC transmitted light.
I connected the three PDs to the oscilloscope and the AOM driver to a function generator. I drive the AOM with a square wave going from 1V to 0V.
I slowly increased the square wave frequency. The PMC servo doesn't seem to care. I reach 100KHz - it seems excessive but still works. In any case, I get the same results doing a single shut-down from a DC level.
I download the traces. I normalize the traces but I don't rescale them (Attachment 4) so that the small extinction can be investigated.
I notice now that the PDs show the same extinction. It probably means I should have taken dark currents data for the PDs.
Also, I forgot to take the reflected data when the PMC is out of resonance with the laser which could have helped us determine the PMC transmission.
Again, the shutdown is not as sharp as I want. There is a noticeable smoothening in the transition around t = 0 which makes the fit to an exponential difficult. I suspect that the function generator is the limiting device now. I hooked up the function generator to the oscilloscope which showed similar distortion (didn't save the trace)
I try to fit the transmission PD trace to a double exponential and to Zucker model (Attachment 5).
The two exponentials model, being much less restrictive, gives a better fit but the best fit gives two identical time constants of 92ns.
The Zucker model gives a time constant of 88ns. Both of these results are consistent with more or less with the linewidth measurement but this measurement is still ridden with systematics which hopefully will become minimized IMC ringdowns.
Jordan will write up the detailed elog but in summary,
I will install these at the next opportunity, so that we can get rid of the many attenuators in this path (the main difficulty will be sourcing the required +12V DC for operation, we only have +15V available near the PSL table).
The PMC servo was re-installed at ~345pm. HV supplies were re-energized to their nominal values. I will update the results of the investigation shortly. The new nominal PMC servo gain is +9dB.
While I have the board out, I'll try and do a thorough investigation of TFs and noise of the various stages. There is no light into the IFO until this is done.
I pulled the board out at 345pm after dialling down all the HV supplies in 1X1. I will reinstall it after running some tests.
I've started a spreadsheet for the BHD optics specifications and populated it with my best initial guesses. There are a few open questions we still need to resolve, mostly related to mode-matching:
The spreadsheet is editable by anyone. If you can contribute any information, please do!
To facilitate wiring the c1psl chassis and scripting loopback tests, I've compiled a distilled spreadsheet with the Acromag-to-breakout board wiring, broken down by connector. This information is extractable from the master spreadsheet, but not easily. There were also a few apparent typos which are fixed here.
The wiring assignments at the time of writing are attached below. Here is the link to the latest spreadsheet.
- No more SR 560, upgraded to LB1005 P-I controller. Because: Elog 14687. Schematic of new setup shown in Attachment 1.
- For this, the Marconi was moved to the other (east) side of the PSL table and a power supply was also placed in the enclosure.
I think that the RF power at the mixer in this new configuration is 0 dBm (since the spectrum analyzer read ~ -20 dBm)
- Turned up the HEPA to 100%, closed the PSL shutter, misaligned the ITMX, connected the LB1005 to the PZT. [The PZT has been reconnected to the X arm PDH servo, HEPA back to 20-30%]
- Tried to look for the PSL+X beat, but it was not there. Gautam identified the flipmount in the path which sorted it out (eventually), but there was no elog about it.
- After much trial, the loop seemed to lock with PI corner 1 kHz, gain ~2.9 (as read on knob), LFGL set to 90 dB. The beat note looked quite stable on the oscilloscope, but the error signal had an rms of ~100 mV (Rana pointed out that it could be the laser noise) and the lock lasted for ~1 min each time.
The parameters were similar to that in elog 14687. Why do we require such a high PI corner frequency and LFGL?
The RF transimpedance of the PMC PDH RFPD was measured, and found to be 1.03 kV/A.
With the new fiber coupled PDFR system, it was very easy to measure the response of this PD in-situ 🎉 . The usual transfer function measurement scheme was used, with the AG4395 RF out modulating the pump current of the diode laser, and the measured transfer function being the ratio of the response of the test PD to the reference PD.
I assume that the amount of light incident on the reference NF1611 photodiode and the test photodiode were equal - I don't know what the DC transimpedance of the PMC REFL photodiode is (can't find a schematic), but the DC voltage at the DC monitor point was 16.4 mV (c.f. -2.04 V for the NF1611). The assumption shouldn't be too crazy because assuming the reference PD has an RF transimpedance of 700 V/A (flat in the frequency range scanned), we get a reasonable shape for the PMC REFL photodiode's transimpedance.
The fitted parameters are overlaid in Attachment #1. The 2f notch is slightly mistuned it would appear, the ratio of transimpedance at f1/2*f1 is only ~10. The source files have been uploaded to the wiki.
Knowing this, the measured PDH discriminant of 0.064 GV/m is quite reasonable:
So why is this value so different from what Koji measured in 2015? Because the monitor point is different. I am monitoring the discriminant immediately after the mixer, whereas Koji was using the front panel monitor. The latter already amplifies the signal by a factor of x101 (see U2 in schematic).
I still haven't found anything that is obviously wrong in this system (apart from the slight nonlinearity in the VGA stage gain steps), which would explain why the PMC servo gain has to be lower now than 2018 in order to realize the same loop UGF.
So the next step is to characterize the RF transimpedance of the PMC RFPD.
doesn't seem so anomolous to me; we're getting ~25 dB of gain range and the ideal range would be 40 dB. My guess is that even thought this is not perfect, the real problem is elsewhere.
The AD602 chip which implements the overall servo gain for the PMC seems to be damaged. We should switch this out at the next opportunity.
I will pull the board and effect the change later today.
The mixer + LPF combo used to demodulate the PMC PDH error signal seems to work as advertised.
Measurement setup --- Attachment #1. The IF signal was monitored using the scope in High-Z mode.
Results --- Attachment #2.
upgrade was done
cronjob testing wasn't one by one 😢
burt snapshots were gone
i brought them back home 🏠
Megatron is now running Ubuntu 18.04 LTS.
wiped and install Debian 10 on rossa today
still to be done: config it as CDS workstation
please don't try to "fix" it in the meantime
Gautam and I debugged a communications problem with TP3 that was causing its python service to fail. We traced the problem back to the querying of the pump controller for its operational parameters (speed, voltage, temp). Some small percentage of the time (~5%, indeterministically), the pump controller is returning an invalid response which causes the service to shut itself down and signal a NO COMM error.
As a temporary fix, I wrapped the failing query in an exception handler to continue past this particular error. However, we suspect the microprocessor in the TP3 controller may be beginning to fail. There is a spare controller sitting right next to it in the vacuum rack. We will ask Chub to install the replacement in the near future.
gautam: this pump is responsible for pumping the annular volume under normal operations. while this problem is being resolved, the annular volume is valved off (as it has been since July 2019 anyway which is when this problem first manifested).
I estimate the PMC servo modulation depth to be approximately 50 mrad. This is only 15% lower than what was measured in Jan 2018, and cannot explain the ~x50 reduction of optical gain measured earlier in this thread. Later in the day, I also confirmed that the LO input to the ZAD-6 mixer is +7 dBm. So the crystal is not to blame.
Assuming a finesse of 700 for the PMC, we expect an optical gain of 2*Pin*J0(50e-3)*J1(50e-3)/fp ~ 1.2e-7 W/Hz (=0.089 GW/m). I can't find a measurement of the PMC RFPD transimpedance to map this onto a V/Hz value.
I looked into this a little more today.
Currently, the iris is set up such that the stronger beam makes it to the PMC RFPD, while the weaker one is blocked by the iris. As usual, this isn't a new issue - was noted last in 2014, but who knows whether the new window was intalled...
Today I noticed that the beam reflected from the PMC into the RFPD has a ghost (attachment) due to reflection from the back of the high transmission beam splitter that stirs the beam into the RFPD.
The PDH discriminant of the PMC servo was measured to be ~0.064 GV/m. This is ~50 times lower than what is reported here. Perhaps this is a signature of the infamous ERA decay, needs more investigation.
The light level hasn't changed by a factor of 50, leading me to suspect the modulation depth. Recall that the demodulation of the PMC is now done off the servo board using a minicircuits mixer (hence, the "C1:PSL-PMC_LODET" channel isn't a reliable readback of the LO signal strength over time). Although there is a C1:PSL-PMC_MODET channel which looks like it comes from the crystal reference card, and so should still work - this, however, shows no degradation over 1 year.
Somebody had removed the BLP-1.9 that I installed at the I/F output of the mixer to remove the sum frequency component in the demodulated signal, I reinstalled this. I find that there are oscillations in the error signal if the PMC servo gain is increased above 14.5 on the MEDM slider.
For a few days, I've noticed that the PSL overview StripTool panel shows PMC transmission and FSS RMTEMP channels with variation that is too large to be believable. Looking at these signals on an oscilloscope, there was no such fuzziness in the waveform. I ruled out flaky connections, and while these are the only two channels currently being acquired by the temporary Acromag setup underneath the PSL enclosure, the Acromags themselves are not to blame, because once I connected a function generator to the Acromag instead of the PMC transmission photodiode, both channels are well behaved. So the problem seems to be with the PMC transmission photodiode, perhaps a grouding issue? Someone please fix this.
Note that for all the alignment work, only the two steering mirrors immediately upstream of the PMC cavity were touched.
I prepare for the ringdown measurement of the PMC according to Gautam's previous experiments.
1. I assembled the needed PDs and power supplies, lenses, beamsplitters and optomechanics needed for the measurement.
2. I surveyed the laser power with an Ophir power meter in the different parts of the experiment. All the measurements were done with the AOM driver excited with 1V DC.
For the PMC reflection, we chose to split off the beam that goes into the reflection camera. The power in that beam is ~ 0.11mW when the PMC is locked and 2.1mW otherwise.
For the PMC transmission, we chose to split the beam that is transmitted through the second steering mirror after the PMC. The power in that beam is 2mW.
For the peak off before the PMC, we chose to split the beam that goes into the fiber coupler. That path contains also the other AOM diffraction orders: 2.26mW in the 0th order beam, 6.5mW in the 1st order beam, 0.14mW in the 2nd order beam.
3. I placed a 10% beam splitter in the peak-off path such that 90% still goes into the fiber coupler (Attachment 1). I place a lens and PDA255 to measure the peak-off (Attachment 2).
It's getting late, I'll continue with the PD placements on Tuesday.
The two beams are focused into the RFPD.
In the past, the ghost beam was probably blocked by the BS mirror mount.
I put an iris to block the ghost beam.
The measurements are consistent with the specifications, and there is no evidence of compression at any of the power levels we expect to supply to this box (<0dBm).
These "gain blocks" were acquired for the purpose of amplifying the IR ALS beat signals before transmission to the LSC rack for demodulation. The existing ZHL-3A amplifiers have a little too much gain, since our revamp to use IR light to generate the ALS beat.
Attachment #4: Setups used to measure transfer functions and noise.
For the transfer function measurement, I chose to send the output of the amplifier to a coupler, and measured the coupled port (output port of the coupler was terminated with 50 ohms). This was to avoid saturating the input of the AG4395. The "THRU" calibration feature of the AG4395 was used to remove the effect of cabling, coupler etc, so that the measurement is a true reflection of the transfer function of OUT/IN of this box. Yet, there are some periodic ripples present in the measured gain, though the size of these ripples is smaller than the spec-ed gain flatness of <0.6dB.
For the noise measurement, the plots I've presented in Attachment #3 are scaled by a factor of sqrt(2) since the noise of the ZFL-500-HLN+ and the ZHL-1010+ are nearly identical according to the specification. Note that the output noise measured was divided by the (measured) gain of the ZFL-500-HLN+ and the ZFL-1010+ to get the input referred noise. The trace labelled "Measurement noise floor" was measured with the input to the ZFL-500-HLN+ terminated with 50ohms, while for the other two traces, the inputs of the ZHL-1010+ were terminated with 50ohms.
Raw data in Attachment #5.
With Gautam's help today the PLL managed to be be locked for a few brief moments. Turns out the signal power of the beat was an issue.
What was changed prior to/ during the experiment:
1. The PSL shutter was closed so not light goes into the input mode cleaner.
2. HEPA turned up (will be turned back down to ~30%)
3. AOM driver offset voltage decreased from 1V to ~100 mV (this will be reverted to 1V by the end of today). This increases the beat signal by deflecting the zeroth order beam to create the beat.
4. Output of servo SR 560 sent to the PZT of the X NPRO laser (the cable was disconnected from the pomona box at the X end)
5. The SR560, mixer, LPF and cables required for connections were moved into the PSL enclosure.
6. The error and control signals were hooked up to the oscilloscope where the beat outputs were visible (the setup has been reverted back to the original).
Elog 14687 has a detailed description of the conditions that provide a stable lock. I was told that the PI controller (LB1005) may be a better servo than the SR560, but today it was not used.
1) Parameters during the more successful attempts:
LPF: 5 MHz, Mixer: ZP-3+
Gain set at SR560: varied, but generally 200
Filter at SR560: 1 Hz low pass (single pole? at least by the label)
2) The LO had to be very close (<2 MHz) to the beat frequency in order to achieve a lock for ~30s
For a Unity Gain Frequency (UGF) of 1 kHz, assumed PZT response of 1 MHz/V, Mixer response of 25 mV/ rad, the required gain of the amplifier is
G ~ 0.8
- Measured the mixer response
- PSL laser temperature was adjusted so that beat frequency was roughly 25 MHz and the amplitude was found to be roughly -30dBm.
- At the RF port instead of the beat signal, a signal of 25 MHz + few kHz at -30 dBm was inputted. The LO was a 25 MHz signal was sent from the Marconi at 7 dBm.
- The mixer output was measured, with setup as in Attachment 1 Figure (A), on an oscilloscope. The slope near the small angle region of the sine curve would be the gain (in V/rad) and was found to be: rad
- Since from the above calculations it seemed like an amplifer gain of 1 should work for the PLL, I rearranged the set up as in Figure (B) of Attachment 1 to actuate the X end NPRO PZT, I adjusted the PSL temperature (slow control) to try and match the frequency to 25 MHz, but couldn't lock the loop. I was monitoring the error signal after amplification (50 ohm output of the SR 560) which showed oscillations when the beat frequency was near 25 MHz and nothing significant otherwise.
- I used a 20 dB attenuator at the amplifier output and saw the beat note oscillate for longer, but maybe because it was a 50 ohm component in a high impedance channel it did not work either (?). I tried other attenuator combinations with no better luck.
- Is there a better location to add the attenuator? Should I pursue amplifying the beat signal instead?
- Also, it seemed like the beat note drift was higher than earlier. Could it be because the PMC was unlocke
Per Yehonathan's request, I removed one PDA10CF from a pickoff of REFL on the AS table (it was being used for the mode spectroscopy project). I placed a razor beam dump where the PD used to be, so that when the PRM is aligned, this pickoff is dumped. This is so that team ringdowns can use a fast PD.
You're right. We had the right idea before but we got confused about this issue. I changed all the XT1121s to XT1111 and vice versa. We already know which channels are sourcing and which not. Updated the wiring spreadsheet. The chassis seems to work. It's time to pass it over to Chub.
I don't think this is an accurate statement. XT1111 modules have sinking digital outputs, while XT1121 modules have sourcing digital outputs. Depending on the requirement, the appropriate units should be used. I believe the XT1111 is the appropriate choice for most of our circuits.
For digital outputs, one should XT1121. XT1111 should be used for digital inputs.
For the ringdowns, I suggest you replicate the setup I had - infrastructurally, this was quite robust, and the main problem I had was that I couldn't extinguish the beam completely. Now that we have the 1st order beam, it should be easy.
I have completed the new EPICS channel database for the c1psl and c1ioo channels (now combined into the new c1psl Acromag machine). I've tested a small subset of channels on the electronics bench to confirm that the addressing and analog channel calibrations are correct in a general sense. At this point, we are handing the chassis off to Chub to complete the wiring of the Acromag terminals to Dsub feedthroughs. At the 40m meeting today, we identified Feb. 17-22 as a potential window for installation in the interferometer (Gautam is out of town then). Below are some implementaton details for future reference.
For analog input (ai) channels, the Acromag outputs raw values ranging from +/-30,000 counts, but the EPICS IOC interprets the data type as ranging from +/-2^15 = 32,768. Similarly, for analog output (ao) channels, the Acromag expects a drive signal in the range +/-30,000 counts. To achieve proper scaling, Johannes had previously changed the EGUF and EGUL fields from +/-10 V to +/-10.923 V. However, changing the engineering fields makes it much harder for a human to read off the real physical I/O range of the channel.
A better way to achieve the correct scaling is to simply set the field ASLO=1.09225 (65,536 / 60,001) in addition to the normal EGUF and EGUL field values (+/-10 V). Setting this field forces a rescaling of the number of raw counts that works as so (assuming a 16-bit bipolar ADC or DAC, as are the Acromags):
OVAL = (RVAL * ASLO + AOFF + 2^15) * (EGUF - EGUL) / 2^16 + EGUL
In the above mapping, OVAL is the value of the channel in engineering units (e.g., V) and RVAL is its raw value in counts. It is not the case that either the ASLO/AOFF or EGUF/EGUL fields are used, but not both. The ASLO/AOFF parameters are always applied (but their default values are ASLO=1 and AOFF=0, so have no effect unless changed). The EGUF and EGUL parameters are then additionally applied if the field LINR="LINEAR" is set.
This conversion allows the engineering fields to remain unchanged from the real physical range. The ASLO value is the same for both analog input and output channels. I have implemented this on all the new c1psl and c1ioo channels and confirmed it to work using a calibrated input voltage source.
Single arm locking using POX and POY has been restored. After running the dither alignment servos, the TRX/TRY levels are ~0.7. This is consistent with the IMC transmission being ~11000 counts with the AOM 1st order diffracted beam (c.f. 15000 counts with the undiffracted beam).
Tomorrow, I'll check the single-arm locking and the ALS system.
Every new year (on Dec 31 or Jan 1), all of the realtime models will report a "0x4000" error. This happens due to an offset to the GPStime driver not being updated. Here is how this can be fixed (slightly modified version of what was done at LASTI).
Steps to fix the DC errors:
/* 2019 had 365 days and no leap seconds */
pHardware->gpsOffset += 31536000;
/* 2019 had 365 days and no leap seconds */
pHardware->gpsOffset += 31536000;
sudo make install
sudo systemctl daqd_* stop
sudo modprobe -r symmetricom
sudo modprobe symmetricom
sudo service daqd_* start
Independent of this, there is a 1 second offset between the gpstimes reported by /proc/gps and gpstime. However, this doesn't seem to drift. We had effected a static offset to correct for this in the daqd config files, and it looks like these do not need to be updated on a yearly basis. All the daqd indicators are now green, see Attachment #1.
There was no light entering the IFO. I worked on a few things to bring the interferometer to a somewhat usable state. The goal is to get back to PRFPMI locking ASAP.
Problem: All fast models report a "0x4000" DC error. See Attachment #1.
Solution: I think this is a "known" issue that happened last new year too. The fix was to add a hard-coded 1 second offset to the daqd config files. However, incrementing/decreasing this offset by +/- 1 second did not fix the errors for me today. I'll reach out to JH for more troubleshooting tips.
Update 15 Jan 2020 830am: The problem is now fixed. See here.
Problem: c1susaux and c1auxey were unresponsive.
Solution: Keyed c1auxey. Rebooted c1susaux and as usual, manually started the eth0/eth1 subnets. The Acromag crate did not have to be power-cycled. ITMY got stuck in this process - I released it using the usual bias jiggling. Why did c1susaux fail? When did it fail? Was there some un-elogged cable jiggling in that part of the lab?
Problem: IMC autolocker and FSS slow processes aren't running on megatron after the upgrade.
Solution: Since no one bothered to do this, I setup systemd infrastructure for doing this on megatron. To run these, you do:
and to check their status, use:
The systemd setup is currently done in a naive way (using the bash executable to run a series of commands rather than using the systemd infrastructure itself to setup variables etc) but it works. I confirmed that the autolocker can re-acquire IMC lock, and that the FSS loop only runs when the IMC is locked. I also removed the obsolete messages printed to megatron's console (by editing /etc/motd) on ssh-login, advising the usage of initctl - the updated message reflects the above instructions.
In order to do the IMC locking, I changed the DC voltage to the AOM to +1V DC (it was +0.8 V DC). In this setting, the IMC refl level is ~3.6 V DC. When using the undiffracted AOM beam, we had more like +5.6 V DC (so now we have ~65% of the nominal level) from the IMC REFL PD when the IMC was unlocked. IIRC, the diffraction efficiency of the AOM should be somewhat better, at ~85%. Needs investigation, or better yet, let's just go back to the old configuration of using the undiffracted beam.
There was also an UN-ELOGGED change of the nominal value of the PMC servo gain to 12.8, and no transfer function measurement. There needs to be a proper characterization of this loop done to decide what the new nominal value should be.
I'm going to leave the PSL shutter open and let the IMC stay locked for stability investigations. Tomorrow, I'll check the single-arm locking and the ALS system.