This required multiple hard reboots, but seems like all the RT models are back for now. The only indicator I can't explain is the red DC field on c1oaf. Also, the SUS model seems to be overclocking more frequently than usual, though I can't be sure. The "timing" field of this model's state word is RED, while the other models all seem fine. Not sure what could be going on.
Will debug further tomorrow, when I probably will have to do all this again as I'll need to recompile c1lsc for the ALS electronics test with the new ADC card from the differential AA board.
As I had found before, restarting the c1oaf model fixed the DC error. There is however still a pesky red indicator light on the "ADC0" in c1oaf. Trying to open up the ADC MEDM screen to investigate this further leads to the blank screen on the bottom right of Attachment #1. Probably has something to do with the fact that the model has an ADC block (because every model needs one?) but no signals are actually being piped to the model directly from the ADC.
Another observation, though I don't have any hypothesis as to why this was happening: on the c1sus machine, the c1sus model would frequently overclock, and then eventually, crash. I observed this behaviour at least 3 times between last night and now. The other models seemed fine though, in fact, IMC stayed locked. Why should this have been the case? It remains to be seen if this was somehow connected to the red DC indicator on c1oaf, though why should this be the case? Isn't the DC just concerned with writing data to frames? Any sort of IPC should be independent? Attachment #2 shows that there's been a definite increase in the maximum time on c1sus clock-cycle since yesterday (it's a 10 day minute trend plot of the model clock cycle timing and also the maximum time). Why? Koji and I did switch off all the Sorensens at the LSC rack for about 30mins, but why should this affect anything at 1X6? There are no red lights in either the c1lsc or c1sus expansion chassis. Curiously, the PRM also seems to be glitchy - as I'm sitting in the control room, I see a spot flashing across vertically on the REFL CRT monitor sporadically. Note that nominally, with PRM misaligned, the REFL CRT should be dark. dmesg on c1sus doesn't shed any light on the issue.
Seems like some high level voodoo .
Yesterday, we installed some new DIN rail connectors at the LSC rack to provide 3 new outputs each for +24V DC and -24V DC. The main motivation was to facilitate the installation and powering of the differential receiving AA board. The regulators used inside the 1U chassis actually claims a dropout voltage of 0.5V and outputs 14V nominally, so a +/-15V DC supply would've perhaps been sufficient, but we decided to leave a bit more margin, and unfortunately, there are no +/-18V DC KEPCO linear power supplies to the LSC rack. Procedure:
The c1lsc frontend models crashed for some reason during this procedure. Now the c1sus frontend model is also behaving weirdly. It is unclear to me if/how this work would have led to these problems, but the temporal correlation (but not causation?) is undeniable.
I was forced into a simultaneous power-cycle rebooting of the three vertex FEs just now. I took the opportunity to completely disconnect the c1sus expansion chassis from all power and then restart it.
Everything is back up right now, and the weird timing issues I noticed in the sus model seem to be gone now (I'll need a longer baseline to be sure and I'll post a trend of the CPU timing tomorrow). It's disconcerting that apparently the only way to get everything back up and running is the nuclear option of power-cycling all FE related electronics. I was considering borrowing an ADC adapter card from the Y end and measuring the calibrated IR ALS noise with the digital system, but if I'm going to have to go through this whole dance each time I do a model recompile on c1lsc (which I'm going to have to in order to get the extra ADC recognized), I'm wondering if it's just better to wait till we get the new adapter cards we ordered. I think I'm going to work on tuning the input coupling into the fiber at EX in the next couple of days instead.
I made a LISO fit of the measured TF of the daughter board, so that I can digitally invert the daughter board whitening. Results attached. (Inverse) Filters have been uploaded to the ALS X Foton filter banks.
Then I looked at the spectrum, see Attachment #1. Disappointingly, it looks like the arm PDH servo is dominating the noise, and NOT unsuppressed EX laser frequency noise,. Not sure why this is so, and I'm feeling too tired to debug this tonight. But encouragingly, the performance of the new ALS signal chain looks very promising. Once I tune up the X arm loop, I'm confident that the ALS noise will be at least as good as the reference trace.
I am leaving c1iscey shutdown until this is fixed. So ETMY is not available for the moment.
Random factoid: Trying to print a DTT trace with LaTeX in the label text on pianosa causes the DTT window to completely crash - so if you dont save the .xml file, you lose your measurement.
I was going to head out but then it occurred to me that I could do another simple test, which is to try and lock the X arm on ALS error signal (i.e. actuate on MC length to keep the beat between EX laser and PSL fixed, while the EX frequency is following the Xarm length). Comparing the in loop (i.e. ALS) error signal with the out-of-loop sensor (i.e. POX), it seems like POX is noisy. The curves were lined up by eye, by scaling the blue curve to match the red at the ~16Hz peaks. This supports my hypothesis in the previous elog. On the downside, could be anything. Electronics in the POX chain? The demod unit itself? Will look into it more tomorrow..
As an aside, controlling the arm with ALS error signal worked quite well, and the lock was maintained for ~1 hour.
Bulb went out ~10am today. Looks like the lifetime of this bulb was <100 days.
Steve: bulb is arriving next week
Bulb is replaced.
we tested my noisy POX hypothesis tonight. By locking the single arm with POX, the arm length is forced to follow PSL frequency, which is itself slaved to IMC length. From Attachment #1, there is no coherence between the arm control signal and MC_F. This suggests to me that the excess noise I am seeing in the arm control signal above 30 Hz is not originating from the PSL. It also seems unlikely that at >30Hz, anything mechanical is to blame. So I am sticking with the hypothesis that something is wonky with POX. For reference, a known "normal" arm control signal spectrum looks like the red curve in this elog.
Kevin suggested I shouldn't be so lazy and test the POY spectrum as well. So we moved the timing card back to c1iscey, went through the usual dance of vertex machine reboots, and then got both single arm locks going. Attached spectrum shows that both POX and POY are noisy. I'm not sure what has changed that could cause this effect. The fact that both POX and POY appear uniformly bad, but that there is no coherence with MC_F, suggests to me that perhaps this has something to do with the work I did with Koji w.r.t. the power situation at the LSC rack. But we just checked that
Another observation we made: note the huge bump around 70Hz in both arm control signals. We don't know what the cause of this is. But we occassionally noticed harmonics of this (i.e. 140, 210 Hz etc) appear in the control signal spectra, and they would grow with time - eventually, the X arm would lose lock (though the Y arm stayed locked).
I'm short on ideas for now so we will continue debugging tomorrow.
Unrelated to this work: Kevin reminded me that the high-pitched whine from the CRT TVs in the control room (which is apparently due to the flyback transformer) is DEAFENING. It's curious that the "chirp" to the eventual 15kHz whine is in opposite directions for the QUAD CRTs and the single display ones. Should be a Ph6 experiment maybe.
Update 2:30pm Mar 13: The furthest back I seem to be able to go in time with Frames is ~Jan 20 2018. Looking for a time when the arms were locked from back then, it seems like whatever is responsible for a noisy POX and POY was already a problem back in January. See Attachment #2. So it appears that the recent work at 1Y2 is not to blame...
The working hypothesis, since the excess noise in single arm locks is coherent between both arms, the excess sensing noise is frequency noise in the IMC locking loop (sensing because it doesn't show up in MC_F). I've started investigating the IMC sensing chain, starting with the power levels of the RF modulation source. Recall that we had changed the way the 29.5MHz signal was sent to the EOM and demod electronics in 2017. With the handheld RF power meter, I measured 13.2dBm coming out of the RF distribution box (this is routed straight from the Wenzel oscillator). This is amplified to 26dBm by an RF amplifier (ZHL-2-S) and sent to the EOM, with a coupled 16dBm part sent to a splitter that supplies the LO signal to the demod board and also the WFS boards. Lydia made a summary of expected RF power levels here, and I too seem to have labelled the "nominal" LO level to the MC_REFL demod board as +5dBm. But I measured 2.7dBm with the RF power meter. But looking closely at the schematic of the splitting circuitry, I think for a (measured) 16.7dBm input to it, we should in fact expect around 3dBm of output signal. So I don't know why I labelled the "nominal" signal level as 5dBm.
Bottom line: we are driving a level 17 mixer with more like +14dBm (a number inferred from this marked up schematic) of LO, which while isn't great, is unlikely to explain the excess noise I think (the conversion loss just degrades by ~1dB). So I will proceed to check further downstream in the signal chain.
It is not clear to me why installing an attenuator to prevent amplifier saturation has necessitated a 10dB increase in the IN1 gain and 3dB increase in the VCO gain. Initially, I was trying to compensate for the gain by increasing the FSS "Common Gain" but in that setting, I found an OLTF measurement impossible. The moment I enabled the excitation input to the CM board, the lock was blown, even with excitation amplitudes as small as -60dBm (from the Agilent network analyzer).
This may also be a good opportunity to test out one of the aLIGO style FET mixer demod boards (recall we have 2 spare from the 4 that were inside the ALS demod box). I'm going to ask Steve to package these into a 1U chassis so that I can try that setup out sometime. From a noise point of view, the aLIGO boards have the advantage of having a x100 preamp stage straight after the mixer+LPF. We may need to replace the lowpass filter though, I'm not sure if the one installed is 1.9MHz or 5MHz.
I've left an SR785 and AG4395 near 1X2 in anticipation of continuing this work tomorrow.
Unrelated to this work - seems like the WFS DC and RF offsets had not been set in a while so I reset these yesterday. The frequent model restarts in recent times may mean that we have to reset these to avoid using dated offset values.
Re-measured the demod board noises after replacing the suspect ERA-5SMs, with LO driven by a marconi at the "nominal" level of 2.5dBm, and RF input terminated. Attachment #1 is the input referred voltage noise spectra. I used the FET low noise pre-amp box for this purpose. I cannot explain the shape of the spectra above 1kHz. I tried doing the measurement on a minicircuits mixer (non-surface mount) and found the shape to be flat throughout the SR785 span. Unclear what else could be going on in the demod board though, all the other components on it are passive (except the ERA-5SMs which were replaced). I considered adopting a PMC style demod setup where we do the demod using some separate Minicircuits Mixer+LowPass filter combo. But the RF flashes for the IMC monitored at the RFmon port are ~0.2Vpp, and so the RF input to the mixer is expected to be ~2Vpp. The minicircuits mixer selection guide recommends choosing a diode mixer with LO level at least 10dBm above the expected RF input signal level, and we don't have any standalone mixers that are >Level 7. I've asked Steve to package the aLIGO demod board in the meantime, but even that might not be a plug and play replacement as the IF preamp stage has ~120degrees phase lag at 1MHz, which is significantly higher than the existing board which just has a SCLF5 low pass filter after the mixer and hence has <45degrees phase lag at 1MHz.
according to the temp sensor readout, which was ~-3.35V which corresponds to ~335K, the temperature of the can is now 60 deg C. This is a bit warm for my liking so i'm turning the heater current down to 0 now by writing 0 to C1:PEM-SEIS_EX_TEMP_CTRL
This elog by koji inspired me to consider power supply as a possible issue.
The demod board receives +/-24V DC (which is regulated down to +/-15V DC by 7815/7915), and also +15V DC via the backplane. The ERA-5SM receives DC power from the latter (unregulated) +15V DC. I can't think of why this is the case except perhaps the regulators can't source the current the amp wants? In any case, it doesn't look feasible to change this by cutting any traces on the PCB to me. While I had the board out, I decided to replace the JMS-1H mixers in a last ditch effort to improve the demod board noise. Unfortunately I'm having trouble de-soldering these MCL components from the board. So for now, I'm leaving the demod board out, IMC unlocked. Work will continue tomorrow.
After some persistence, I managed to get the mixers off.
Unfortunately, the coherent noise between the arms persists so the sensing noise injection must be happening elsewhere. IMC seems to lock fine though so I'm leving the autolocker on
As discussed at the meeting, I decided to calibrate the MC error point into physical units of Hz/rtHz (a.k.a. the PDH discriminant). This is to facilitate the debugging of the hypothesized excess IMC sensing noise. I did this as follows.
Using this, I can now make up a noise budget of sorts for the IMC sensing.
gautam 20180327 4.30pm: I re-checked the PDH error signal calibration using the oscilloscope method. Attachment #3 shows the PDH I and Q error signals and also the output of the RF monitor port, during a TEM00 flash. This attachment should be compared to Attachment #2 of elog 12822, and the answer lines up quite well. From my Finesse model of the IMC, I calculated that the x-axis of the PDH horn-to-horn is ~12.3kHz. Comparing to the top row of Attachment #3, I get a PDH error signal calibration of ~12.4kHz/Vrms, which lines up well with the number quoted above. So I trust my calibration, and hence, the y-axis of my noise budgets in reply to this elog.
I did a preliminary noise budget of the transmitted frequency noise of the IMC. Attachment #1 shows the NB. I'm going to use this opportunity to revisit my IMC modeling. Some notes:
Conclusion: From this study, assuming my PDH discriminant calibration was correct, looks like IMC demod / POX11 demod electronics noises are not to blame (this surprises me since there were apparently so many things wrong on the demod board, and yet that wasn't the worst thing in the IMC chain it would seem ). The POX11 photodiode "dark" noise is also not the problem I think, given the grey curve. Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
I've added two curves to the NB. Both are measured (with FET preamp) at the output of the demod board, with the LO driven at the nominal level by the Wenzel RF source pickoff (as it would be when the IMC is locked) and the RF input connected to the IMC REFL PD. For one curve, I simply closed the PSL shutter, while for the other, I left the PSL shutter open, but macroscopically misaligned MC2 so that there was no IMC cavity. So barring RFAM, there should be no PDH signal on the REFL PD, but I wanted to have light on there. I'm not sure if I understand the difference between these two curves though, need to think on it. Perhaps the IMC REFL PD's optical/electrical response needs to be characterized?
Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
we measured the RIN of the MC2 transmission using the PDA255 I had put on the MC2 trans table sometime ago for ringdowns. Attached are (i) spectra for the RIN, (ii) spectra for the classical rad. pressure noise assuming 500W circulating power and (iii) a tarball of data and code used to generate these plots.
We took a full span measurement (to make sure there aren't any funky high-freq features) and a measurement from DC-800 Hz (where we are looking for excess noise). The DC level of light on the photodiode was 2.76V (measured using o'scope)
I'll add this to the noise budget later. But the measured RIN seems consistent with a 2013 measurement at 100Hz (though the 2013 measurement is using DTT and so doesn't have high frequency information).
Todd informed me that the ADC Timing adaptor boards we had ordered arrived today. I had to solder on the components and connectors as per the schematic, though the main labor was in soldering the high density connectors. I then proceeded to shut down all models on c1lsc (and then the FE itself). Then classic problem of all vertex machines crashing when unloading models on c1lsc happened (actually Koji noticed that this was happening even on c1ioo). Anyways this was nothing new so I decided to push ahead.
I had to get a cable from Downs that connects the actual GS ADC card to this adaptor board. I powered off the expansion chassis, installed the adaptor board, connected it to the ADC card and restarted the expansion chassis and also the FE. I also reconnected the SCSI cable from the AA board to the adaptor card. It was a bit of a struggle to get all the models back up and running again, but everything eventually came back(after a few rounds of hard rebooting). I then edited the c1x04 and c1lsc simulink models to reflect the new path for the X arm ALS error signals. Seems to work alright.
At some point in the afternoon, I noticed a burning smell concentrated near the PSL table. Koji traced the smell down to the c1lsc expansion chassis. We immediately powered the chassis off. But Steve later informed me that he had already noticed an odd burning smell in the morning, before I had done any work at the LSC rack. Looking at the newly installed adaptor card, there wasn't any visual evidence of burning. So I decided to push ahead and try and reboot all models. Everything came back up normally eventually, see Attachment #1. Particle count in the lab seems a little higher than usual (actually, according to my midnight measurement, they are ~factor of 10 lower than Steve's 8am measurements), but Steve didn't seem to think we should read too much into this. Let's monitor the situation over the coming days, Steve should comment on the large variance seen in the particle counter output which seems to span 2 orders of magnitude depending on the time of the day the measurement is made... Also note that there is a BIO card in the C1LSC expansion chassis that is powered by a lab power supply unit. It draws 0 current, even though the label on it says otherwise. I a not sure if the observed current draw is in line with expectations.
The spare (unstuffed) adaptor cards we ordered, along with the necessary hardware to stuff them, are in the Digital FE hardware cabinet along the east arm.
Steve: particle count in the 40m is following outside count, wind direction, weather condition .....etc. The lab particle count is NOT logged ! This is bad practice.
While Kevin and Arijit were doing their MC_REFL PD noise measurements (which they will elog about separately shortly), I noticed a feature around 600kHz that reminded me of the NPRO noise eater feature. This is supposed to suppress the relaxation oscillation induced peak in the RIN of the PSL. Surprisingly, the noise eater switch on the NPRO front panel was set to "OFF". Is this the normal operating state? I thought we want the noise eater to be "ON"? Have to measure the RIN on the PSL table itself with one of the many available pick off PDs. In any case, as Attachment #1 showed, turning the noise eater back on did not improve the excess IMC frequency noise.
I've removed the MC REFL PD unit from the AS table for investigation. So MC won't lock.
PSL shutter was closed and location of PD was marked with sharpie (placing guides to indicate position wasn't convenient). I also kapton taped the PD to minimize dust settling on the PD while I have it out in the electronics area. Johannes has the camera, and my cellphone image probably isn't really high-res enough for diagnostics but I'm posting it here anyways for what it's worth. More importantly - the board is a D980454 revision B judging by the board, but there is no schematic for this revision on the DCC. The closest I can find is a D980454 Rev D. But I can already see several differences in the component layout (though not all of them may be important). Making a marked up schematic is going to be a pain . I'm also not sure what the specific make of the PD installed is.
The lid of the RF cage wasn't on.
More to follow tomorrow, PD is on the electronics workmench for now...
gautam 28 March 2018: Schematic has been found from secret Dale stash (which exists in addition to the secret Jay stash). It has also been added to the 40m electronics tree.
I re-installed the MC REFL photodiode. Centered beam on the PD by adjusting steering mirror to maximize the DC signal level (on o'scope) at the DC monitoring port. Curiously, the DC level on the scope (high-Z) was ~2.66V DC, whereas the MEDM screen reports ~twice that value, at ~5.44 "V". We may want to fix this "calibration" (or better yet, use physical units like mW). Noise-eater On/Off comparison of MC error signals to follow.
It's been a while - but today, all slow machines (with the exception of c1auxex) were un-telnetable. c1psl, c1iool0, c1susaux, c1iscaux1, c1iscaux2, c1aux and c1auxey were rebooted. Usual satellite box unplugging was done to avoid ITMX getting stuck.
I'm probably doing something stupid - but I've not been able to figure this out. In the MC1 and MC3 coil driver filter banks, we have a digital "28HzELP" filter module in FM9. Attachment #1 shows the MC1 filterbanks. In the shown configuration, I would expect the only difference between the "IN1" and "OUT" testpoints to be the transfer function of said ELP filter, after all, it is just a bunch of multiplications by filter coefficients. But yesterday looking at some DTT traces, their shapes looked suspicious. So today, I did the analysis entirely offline (motivation being to rule out DTT weirdness) using scipy's welch. Attachment #2 shows the ASDs of the IN1 and OUT testpoint data (collected for 30s, fft length is set to 2 seconds, and hanning window from scipy is used). I've also plotted the "expected" spectral shape, by loading the sos coefficients from the filter file and using scipy to compute the transfer function.
Clearly, there is a discrepancy for f>20Hz. Why?
Code used to generate this plot (and also a datafile to facilitate offline plotting) is attached in the tarball Attachment #3. Note that I am using a function from my Noise Budget repo to read in the Foton filter file...
*ChrisW suggested ruling out spectral leakage. I re-ran the script with (i) 180 seconds of data (ii) fft length of 15 seconds and (iii) blackman-harris window instead of Hanning. Attachment #4 shows similar discrepancy between expectation and measurement...
Seems like there was a 5.3 magnitude EQ ~10km from us (though I didn't feel it). All watchdogs were tripped so our mirrors definitely felt it. ITMX is stuck (but all the other optics are damping fine). I tried the usual jiggling of DC bias voltage but ITMX still seems stuck. Probably a good sign that the magnet hasn't come off, but not ideal that I can't shake it free..
edit: after a bit more vigorous shaking, ITMX was freed. I had to move the bias slider by +/-10,000 cts, whereas initially I was trying +/-2000 cts. There is a tendency for the optic to get stuck again once it has been freed (while the optic's free swinging motion damps out), so I had to keep an eye out and as soon as the optic was freed, I re-engaged the damping servos to damp out the optic motion quickly.
I found all the EPICS channels for the model c1ioo on the FE c1ioo to be blank just now. The realtime model itself seemed to be running fine, judging by the IMC alignment (as the WFS loops seemed to still be running okay). I couldn't find any useful info in demsg but I don't know what I'm looking for. So my guess is that somehow the EPICS process for that model died. Unclear why.
Kira informed me that she was having trouble accessing past data for her PID tuning tests. Looking at the last day of data, it looks like there are frequent EPICS data dropouts, each up to a few hours. Observations (see Attachment #1 for evidence):
It is difficult to diagnose how long this has been going on for, as once you start pulling longer stretches of data on dataviewer, any "data freezes" are washed out in the extent of data plotted.
While Kevin is working on the MC2 electronics chain - we disconnected the output to the optic (DB15 connector on coil driver board). I decided to look at the 'free' freeswinging MC2 OSEM shadow sensor data. Attachment #1 suggests that the suspension eigenmodes are showing up in the shadow sensors, but the 0.8Hz peak seems rather small, especially compared to the result presented in this elog.
Maybe I'll kick all 3 MC optics tonight and let them ringdown overnight, may not be a bad idea to checkup on the health of the MC suspensions/satellite boxes... [MC suspensions were kicked @1207113574]. PSL shutter will remain closed overnight...
Why is MC2 LR so different from the others???
I am working on IMC electronics. IMC is misaligned until further notice.
activities done today - details/plots/data/evidence tomorrow.
Today, I decided to check the power coupled into the PSL fiber for the BeatMouth. Surprisingly, it was only 200uW, while I had ~3.15mW going into it in January. Presumably some alignment drifting happened. So I re-aligned the beam into the fiber using the steering mirror immediately before the fiber coupler. I managed to get ~2.9mW in without much effort, and I figured this is sufficient for a first pass, so I didn't try too much more. I then tried making an ALS beat spectrum measurement (arm locked to IMC length using POX, green following the arm using end PDH servo). Surprisingly, the noise performannce was almost as good as the reference! See Attachment #1, in which the red curve is an IR beat (while all others are green beats). The Y arm green beat performance isn't stellar, but one problem at a time. Moreover, the kind of coherence structure between the arm error signal and the ALS beat signal that I reported here was totally absent today.
Upon further investigation, I found that the noise level was actually breathing quite significantly on timescales of minutes. While I was able to successfully keep the TEM00 mode of the PSL beam resonant inside the arm cavity by using the ALS beat frequency as an error signal and MC2 as a frequency actuator, the DC stability was very poor and TRX was wandering around by 50%. So my new hypothesis is that the excess ALS noise is because of one or more of
While I did some work in trying to align the PSL IR pickoff into the fiber along the fast (P-pol) axis, I haven't done anything for the X end pickoff beam. So perhaps the fluctuations in the EX IR power is causing beatnote amplitude fluctuations. In the delay line + phase tracker frequency discriminator, I think RF beatnote amplitude fluctuations can couple into phase noise linearly. For such an apparently important noise source, I can't remeber ever including it in any of the ALS noise budgets.
Before Ph237 today, I decided to use my polarization monitoring setup and check the "RIN" of power in the two polarizations coming out of the fiber on the PSL table. For this purpose, I decided to hijack the Acromag channels used for the PSL diagnostics connector Attachment #2 shows that there is fluctuations at the level of ~10% in the p-polarization. This is the "desired" polarization in that I aligned the PSL beam into the fiber to maximize the power in this polarization. So assuming the power fluctuations in the PSL beam are negligible, this translates to sqrt(10) ~3% fluctuation in the RF beat amplitude. This is at best a conservative estimate, as in reality, there is probably more AM because of the non PM fibers inside the beatmouth.
All of this still doesn't explain the coherence between the measured ALS noise and the arm error signal at 100s of Hz (which presumably can only happen via frequency noise in the PSL).
Another "mystery" - yesterday, while I was working on recovering the Y arm green beat signal on the PSL table, I eventually got a beat signal that was ~20mVpp into 50ohms, which is approximately the same as what I measured when the Y arm ALS performance was "nominal", more than a year ago. But while viewing the Y arm beats (green and IR) simultaneously on an o'scope, I wasn't able to keep both signals synchronised while triggering on one (even though the IR beat frequency was half the green beat frequency). This means there is a huge amount of relative phase noise between the green and IR beats. What (if anything) does this mean? The differential noise between these two beats should be (i) phase noise at the fiber coupler/ inside the fiber itself, and (ii) scatter noise in the green light transmitted through the cavity. Is it "expected" that the relative phase noise between these two signals is so large that we can't view both of them on a common trigger signal on an o'scope? Also - the green mode-matching into the Y arm is abysmal.
Anyways - I'm going to try and tweak the PER and mode-matching into the X end fiber a little and monitor the polarization stability (nothing too invasive for now, eventually, I want to install the new fiber couplers I acquired but for now I'll only change alignment into and rotation of the fiber coupler on the EX table). It would also be interesting to compare my "optimized" PSL drift to the unoptimized EX power drift. So the PSL diagnostic channels will not show any actual PSL diagnostic information until I plug it back in. But I suspect that the EPICS record names and physical channel wiring are wrong anyways - I hooked up my two photodiode signals into what I would believe is the "Diode 1 Power" and "Laser crystal temperature" monitors (as per the schematic), but the signals actually show up for me in "Diode 2 Power" (p-pol) and "Didoe 1 Temperature" (s-pol).
Annoyingly, there is no wiring diagram - on my todo list i guess...
@Steve - could you please take a photo of the EX table and update the wiki? I think the photo we have is a bit dated, the fiber coupler and transmon PDs aren't in it...
Attachment #1 shows the drift of the polarization content of the light from EX entering the BeatMouth. Seems rather large (~10%). I'm going to tweak the X end fiber coupling setup a bit to see if this can be improved. This performance is also a good benchmark to compare the PSL IR light polarization drift. I am going to ask Steve to order Thorlabs K6XS, which has a locking screw for the rotational DoF. With this feature, and by installing some HWPs at the input coupling point, we can ensure that we are coupling light into one of the special axes in a much more deterministic way.
THIS CALCULATION IS WRONG FOR THE OVERCOUPLED CAV.
Mode-matching efficiency of EX green light into the arm cavity is ~70*%, as measured using the visibility.
I wanted to get an estimate for the mode-matching of the EX green beam into the arm cavity. I did the following:
This amount of mode-matching is rather disappointing - using a la mode, the calculated mode-matching efficiency is nearly 100%, but 70% is a far cry from this. The fact that I can't improve this number by either tweaking the steering or by moving the MM lenses around suggests that the estimate of the target arm mode is probably incorrect (the non-gaussianity of the input beam itself is not quantified yet, but I don't believe this input beam can account for 30% mismatch). For the Y-arm, the green REFL DC level is actually higher when locked than when ITMY is misaligned. WTF?? Only explanation I can think of is that the PD is saturated when green is unlocked - but why does the ADC saturate at ~3000cts and not 32000?
This data is almost certainly bogus as the AA box at 1X9 is powered by +/-5VDC and not +/-15VDC. I didn't check but I believe the situation is the same at the Y-end.
3000 cts is ~1V into the ADC. I am going to change the supply voltage to this box (which also reads in ETMX OSEMS) to +/-15V so that we can use the full range of the ADC.
gautam Apr 26 630pm: I re-did the measurement by directly monitoring the REFLDC on a scope, and the situation is not much better. I calculate a MM of 70% into the arm. This is sensitive to the lens positions - while I was working on the EX fiber coupling, I had bumped the lens mounted on a translational stage on the EX table lightly, and I had to move that lens around today in order to recover the GTRX level of 0.5 that I am used to seeing (with arm aligned to maximize IR transmission). So there is definitely room for optimization here.
I swapped the EX fiber for the PSL fiber in the polarization monitoring setup. There is a lot more power in this fiber, and one of the PDs was saturated. I should really have put a PBS to cut the power, but I opted for putting an absorptive ND1.0 filter on the PD instead for this test. I want to monitor the stability in this beam and compare it to the EX beam's polarization wandering.
It looks like the drift in polarization content in the PSL pickoff is actually much higher than that in the EX pickoff. Note that to prevent the P-pol diode from saturating, I put an ND filter in front of the PD, so the Y axis actually has to be multiplied by 10 to compare power in S and P polarizations. If this drift is because of the input (linear) polarization being poorly matched to one of the fiber's special axes, then we can improve the situation relatively easily. But if the polarization drift is happening as a result of time-varying stress (due to temp. fluctuations, acoustics etc) on the (PM) fiber from the PSL fiber coupler to the BeatMouth, then I think this is a much more challenging problem to solve.
I'll attempt to quantify the contribution (in Hz/rtHz) of beat amplitude RIN to phase tracker output noise, which will tell us how much of a problem this really is and in which frequency bands. In particular, I'm interested in seeing if the excess noise around 100 Hz is because of beat amplitude fluctuations. But on the evidence thus far, I've seen the beat amplitude drift by ~15 dB (over long timescales) on the control room network analyzer, and this drift seems to be dominated by PSL light amplitude fluctuations.
A follow-up on the discussion from today's lunch meeting - Rana pointed out that rotation of the fiber in the mount by ~5degrees cannot account for such large power fluctuations. Here is a 3 day trend from my polarization monitoring setup. Assuming the output fiber coupler rotates in its mount by 5 degrees, and assuming the input light is aligned to one of the fiber's special axes, then we expect <1% fluctuation in the power. But the attached trend shows much more drastic variations, more like 25% in the p-polarization (which is what I assume we use for the beat, since the majority of light is in this polarization, both for PSL and EX). I want to say that the periodicity in the power fluctuations is ~12hours, and so this fluctuation is somehow being modulated by the lab temperature, but unfortunately, we don't have the PSL enclosure temperature logged in order to compare coherence.
Steve: your plots look like temperature driven
The "beat length" of the fiber is quoted as <=2.7mm. This means that a linearly polarized beam that is not oriented along one of the special axes of the fiber will be rotated through 180 degrees over 2.7mm of propagation through the fiber. I can't find a number for the coefficient of thermal expansion of the fiber, but if temperature driven fluctuations are changing the length of the fiber by 300um, it would account for ~12% power fluctuation between the two polarizations in the monitoring setup, which is in the ballpark we are seeing...
All slow machines (except c1auxex) were dead today, so I had to key them all. While I was at it, I also decided to update MC autolocker screen. Kira pointed out that I needed to change the EPCIS input type (in the RTCDS model) to a "binary input", as opposed to an "analog input", which I did. Model recompilation and restart went smooth. I had to go into the epics record manually to change the two choices to "ENABLE" and "DISABLE" as opposed to the default "ON" and "OFF". Anyways, long story short, MC autolocker controls are a bit more intuitive now I think.
I'm working on fixing these (and the associated MEDM scripts) up so that we can get some reliable readback on how well centered the spots are on the MC mirrors. Seems like a bunch of MEDM display paths were broken, and it looks like the optimal demod phases (to put most of the output in I quadrature) are not what the current iteration of the scripts set them to be. It may well be that the gains that convert demodulated counts to mm of spot offset are also not correct anymore. I ran the script ~4times in ~1 hour time span, and got wildly different answers for the spot centering each time, so I wouldn't trust any of those numbers atm...
As you can see in Attachment #1, I stepped the demod phase of one of the servos from -180 to 180 degrees in 5degree increments. The previously used value of 57degrees is actually close to the worst possible point (if you want the signal in the I quadrature, which is what the scripts assume).
I used Attachment #2 to change up the demod phases to maximize the I signal. I chose the demod phase such that it preserved the sign of the demodulated signal (relative to the old demod phases). I also made some StripTool templates for these, and they are in the MC directory. Doing the spot centering measurement with the updated demod phases yields the following output from the script:
spot positions in mm (MC1,2,3 pit MC1,2,3 yaw):
[0.72506586251144134, 7.1956908534034403, 0.54754354165324848, -0.94113388241389784, -3.5325883025464959, -2.4934027726657142]
Seems totally unbelievable still that we are so far off center on MC1 yaw. Perhaps the gains and calibration to convert from counts to mm of spot offset need to be rechecked.
As shown in the Attachments, it seems like IMC DAC and coil driver noise is the dominant noise source above 30Hz. If we assume the region around the bounce peak is real motion of the stack (to be confirmed with accelerometer data soon), this NB is starting to add up. Much checking to be done, and I'd also like to get a cleaner measurement of coil driver and DAC noise for all 3 optics, as there seems to be a factor of ~5 disagreement between the MC3 coil driver noise measurement and my previous foray into this subject. The measurement needs to be refined a little, but I think the conclusion holds.
Since I sunk some time into it already, the motivation behind this work is just to try and make the IMC noise budget add up. It is not directly related to lowering the IR ALS noise, but if it is true that we are dominated by coil driver noise, we may want to consider modifying the MC coil driver electronics along with the ITM and ETMs.
For the arm cavity ringdowns, I guess we don't need AS55/AS110 (although I think the camera will still be useful for alignment). But for something like RC Gouy phase characterization, I'd imagine we need the AS detectors to lock various cavities. So I think we should go for a solution that doesn't disturb the AS PD beams.
It's hard to tell from the plot in the manual (pg 52) what exactly the relaxation oscillation frequency is, but I think it's closer to 600 kHz (is this characteristic of NdYAG NPROs)?? Is the high RIN on the light straight out of the NPRO?
We could use a simple mirror if we don't need AS110 and AS55, we could use a polarizing BS and work with s polarization, or we find a Faraday Isolator.
There is a strong oscillation around 210 kHz. The oscillation frequency decreases when the output power is turned down, the noise eater has no effect.
The most up to date pictures of the AP table and ETMX table that Steve took have been uploaded to the relevant page on the wiki. It seems like the wiki doesn't display previews of jpg images - by using png, I was able to get the thumbnail of the attachment to show up. It would be nice to add beam paths to these two images. The older versions of these photos were moved to the archive section on the same page.
Today, I repeated the coil driver noise measurement. Now, the coil driver noise curve in the noise budget plot (Attachment #1) is the actual measurement of all 12 coils (made with G=100 SR560). I am also attaching the raw voltage noise measurement (input terminated in 50ohms, Attachment #2). Note that POX11 spectrum has now been re-measured with analog de-whitening engaged on all 3 optics such that the DAC noise contribution should be negligible compared to coil driver noise in this configuration. The rows in Attachment #2 correspond to 800 Hz span (top) and full span (bottom) on the FFT analyzer.
The main difference between this measurement, and the one I did middle of last year (which agreed with the expectation from LISO modeling quite well) is that this measurement was done in-situ inside the eurocrate box while last year, I did everything on the electronics benches. So I claim that the whole mess of harmonics seen in the raw measurements are because of some electronics pickup near 1X6. But even disregarding the peaky features, the floor of ~30nV/rtHz is ~6x than what one would expect from LISO modeling (~5nV/rtHz). I confirmed by looking that the series resistance for all 3 MC optics is 430ohms. I also did the measurement with the nominal bias voltages applied to the four channels (these come in via the slow ADCs). But these paths are low-passed by an 8th order low pass with corner @ 1Hz, so at 100 Hz, even 1uV/rtHz should be totally insignificant. I suppose I could measure (with EPICS sine waves) this low-pass filtering, but it's hard to imagine this being the problem. At the very least, I think we should get rid of the x3 gain on the MC2 coil driver de-whitening board (and also on MC1 and MC3 if they also have the x3 factor).
I think the dominant cause for the fact that we were seeing huge swing in the power coupled into the fiber was that the beam being sent in was in fact not linearly polarized, but elliptically polarized. I've rectified this with the help of a PBS. Fiber has been plugged into my polarization monitoring setup. Let's monitor for some long stretch and see if the situation has improved.
Aidan called saying nodus was down at ~345pm. I was able to access it at ~330pm. I couldn't ssh in from my machine or the control room ones. So I went to 1X7 and plugged in a monitor to nodus. It was totally unresponsive. Since the machine wasn't responding to ping either, I decided to hard-reboot it. Machine seemed to come back up smoothly. I had trouble getting the elog started - it wasn't clear to me that the web ports were closed by default, so even though the startELOGD.sh script was running fine, the 8080 port wasn't open to the outside world. Anyways, once I figured this out, I was able to start the elog. DokuWiki also seems to be up and running now...
I found megatron in a similar state to that which nodus was in yesterday. Clued by the fact that MCautolocker wasn't executing the mc scripts (as was evident from looking at the wall StripTool trace), I tried ssh-ing into megatron, but was unable to (despite it being responsive to ping requests). So I went into the VEA and plugged in a monitor to megatron - saw nothing on it. With no soft reboot options available, I power cycled the machine via the front panel button. It came back up smoothly. I manually restarted the autolocker, FSSslow and EX thermal control processes (the former two with initctl, while the latter runs in a tmux session). Everything seems alright for now. Not sure how long megatron has been dead for.
Seems like there is still a bit of variation in the power in the two polarizations, though it is much smaller now, at the ~5% level (see Attachment #1). Since the pattern is repeating itself over the day timescale, I think this effect is not because of rotation of the output coupler in the mount, but is in fact a temperature driven waveplate effect because of imperfect alignment at the input coupler (which itself is locked down). I'm going to rotate the input coupler by 5 degrees (old = 110 degrees, new=115degrees) to see if the situation improves...
gautam Apr 24 2pm: Steve suggested confirming the correlation by hooking up the PSL table temperature sensor. This used to be logged but since the c1psl ADC card failure, has not been recorded. Assuming the sensor and preamp still work fine, we can use the PSL diagnostic Acromag (whose channels I have hijacked to monitor polarization stability already) to at least temporarily monitor the temperature inside the PSL enclosure. I am in need of a DB25 breakout board for this purpose which I am missing right now, as soon as I obtain one, I'll hook this up...