For no apparent reason, the MC1 glitches are back. Nothing has been touched near the PD whitening chassis today, and the trend suggests the glitching started about 3 hours ago.. I had disabled the MC1 watchdog for a while to avoid the damping loop kicking the suspension around when these glitches occur, but have re-enabled it now. IMC is holding lock for some minutes... I was hoping to do another round of ringdowns tonight, but if this persists, its going to be difficult...
Over the weekend, I worked a bit on getting these ringdowns going. I will post a more detailed elog tomorrow but here is a quick summary of the changes I made hardware-wise in case anyone sees something unfamiliar in the lab...
I spent a while in preparation for these trials (details tomorrow) like optimizing AOM alignment/diffracted power ratio, checking AOM and PMC switching times etc, but once the hardware is laid out, it is easy to do a bunch of ringdowns in quick succession with an ethernet scope. Tonight I did about 12 ringdowns - but stupidly, for the first 10, I was only saving 1 channel from the oscilloscope instead of the 3 we want to apply the MIT method.
Here is a representative plot of the ringdown - at the moment, I don't have an explanation for the funky oscillations in the reflected PD signal, need to think on this.. More details + analysis to follow...
Dec 5 2016, 130pm:
Actually the plot I meant to put up is this one, which has the time window acquired slightly longer. The feature I am referring to is the 100kHz oscillation in the REFL signal. Any ideas as to what could be causing this?
ELOG of the work on Thursday
Gautam suggested looking at the preamplifier noise by shorting the input to the first stage. I thought it was a great idea.
To my surprise, the noise of the 2nd stage was really high compared to the model. I proceeded to investigate what was wrong.
It turned out that the resistors used in this sallen-key LPF were thick film resistors. I swapped them with thin film resistors and this gave the huge improvement of the preamplifier noise in the low frequency band.
Attachment 1 shows the summary of the results. Previously the input referred noise of the preamp was the curve in red. We the resistors replaced, it became the curve in magenta, which is pretty close to the expected noise level by LISO model above 3Hz (dashed curves). Unfortunately, the output of the unit with the demodulator connected showed no improvement (blue vs green), because the output is still limited by the demodulator noise. There were harmonic noise peaks at n x 10Hz before the resistor replacement. I wonder if this modification also removed the harmonic noise seen in the CDS signals. I will check this next week.
Attachment 2 shows the current schematic diagram of the demodulator board. The Q of the sallen key filter was adjusted by the gain to have 0.7 (butter worth). We can adjust the Q by the ratio of the capacitance. We can short 3.83K and remove 6.65K next to it. And use 22nF and 47nF for the capacitors at the positive input and the feedback, respectively. This reduces the number of the resistors.
ELOG of the Wednesday work.
It turned out that the IMC WFS demod boards have the PCB board that has a different pattern for each of 8ch.
In addition, AD831 has a quite narrow leg pitch with legs that are not easily accessible.
Because of these, we (Koji and Rana) decided to leave the demodulator chip untouched.
I have plugged in the board with the WFS2-Q1 channel modified in order to check the significance of the modification.
WFS performance before the modification
Attachment 1 shows the PSD of WFS2-I1_OUT calibrated to be referred to the demodulator output. (i.e. Measured PSDs (cnt/rtHz) were divided by 8.9*2^16/20)
There are three curves: One is the output with the MC locked (WFS servos not engaged). The second is the PSD with the PSL beam blocked (i.e. dark noise). The third is the electronics noise with the RF input terminated and the nominal LO supplied.
This tells us that the measured PSD was dominated by the demodulator noise in the dark condition. And the WFS signal was also dominated by the demod noise below 0.1Hz and above 20Hz. There are annoying features at 0.7, 1.4, 2.1, ... Hz. They basically impose these noise peaks on the stabilized mirror motion.
WFS performance after the modification
Attachment 2 shows the PSD of WFS2-Q1_OUT calibrated to be referred to the demodulator output. (i.e. Measured PSDs (cnt/rtHz) were divided by 21.4*2^16/20)
There are three same curves as the other plot. In addition to these, the PSD of WFS2-I1_OUT with the MC locked is also shown as a red curve for comparison.
This figure tells us that the measured PSD below 20Hz was dominated by the demodulator noise in the dark condition. And the WFS signal is no longer dominated by the electronics noise. However, there still are the peaks at the harmonics of 0.7, 1.4, 2.1, ... Hz. I need further inspection of the FWS demod and whtening boards to track down the cause of these peaks.
I've pulled out the 24V fuse block which supplies power to the AOM RF driver. The way things are set up on the PSL table, this same voltage source powers the RF amplifiers which amplify the green beatnote signals before sending them to the LSC rack. So I turned off the green beat PDs before pulling out the fuse. I then disconnected the input to the RF driver (it was plugged into a DS345 function generator on the PSL table) and terminated it with a 50 ohm terminator. I want to figure out a smart way of triggering the AOM drive and recording a ringdown on the scope, after which I will re-connect the RF driver to the DS345. The RF driver, as well as the green beat amplifiers and green beat PDs, remain unpowered for now...
The most recent power outage took out our projector and mixer. The projector was sent for repair while we ordered a new mixer. Both arrived today. Steve is working on re-installing the projector right now, and I installed the mixer which was verified to be working with our DAFI system (although the 60Hz issue still remains to be sorted out). The current channel configuration is:
Ch1: 3.5mm stereo output from pianosa
Ch2: DAFI (L)
Ch3: DAFI (R)
I've set some random gains for now, but we will have audio again when locking
I noticed 2 periods of frequent IMC locklosses on the StripTool trace, and so checked the MC1 PD readout channels to see if there were any coincident glitches. Turns out there wasnt BUT - the LR and UR signals had changed significantly over the last couple of days, which is when I've been working at 1X5. The fast LR readback was actually showing ~0, but the slow monitor channel had been steady so I suspected some cabling shenanigans.
Turns out, the problem was that the LEMO connector on the front of the MC1 whitening board had gotten jiggled ever so slightly - I re-jiggled it till the LR fast channel registered similar number of counts to the other channels. All looks good for now. For good measure, I checked the 3 day trend for the fast PD readback for all 8 SOS optics (40 channels in all, I didn't look at the ETMs as their whitening boards are at the ends), and everything looks okay... This while situation seems very precarious to me, perhaps we should have a more robust signal routing from the OSEMs to the DAQ that is more immune to cable touching etc...
The south end door leaky weather seals replaced.
The aim is here to get some overpressure inside / outside so the lab partical count would not depend on outside condition.
We want to measure the IMC round-trip loss using the Isogai et. al. ringdown technique. I spent some time looking at the various bits and pieces needed to make this measurement today, this elog is meant to be a summary of my thoughts.
Does this sound like a sensible plan? Or do I need to do any further checks?
As we suspected, the binary breakout board (D080478, no drawing available) is simply a bunch of tracks printed on the PCB to route the DB37 connector pins to two IDE50 connectors. There was no visible damage to any of the tracks (some photos uploaded to the 40m picasa). Further, I checked the continuity between pins that should be connected using a DMM.
I got a slightly better understanding of how the binary output signal chain is - the relevant pages are 44 and 48 in the CONTEC manual. The diagram on pg44 maps the pins on the DB37 connector, while the diagram on pg 48 maps how the switching actually occurs. The "load" in our case is the 4.99kohm resistor on the PD whitening board D000210. Following the logic in the diagram on pg48 is easy - setting a "high" bit in the software should pull the load resistor to 0V while setting a "low" bit keeps the load at 15V (so effectively the whole setup of CONTEC card + breakout board + pull-up resistor can be viewed as a simple NOT gate, with the software bit as the input, and the output connected to the "IN" pin of the MAX333).
Since I was satisfied with the physical condition of the BO breakout board, I re-installed the box on 1X5. Then, with the help of a breakout board, I diagnosed the situation further - I monitored the voltage to the pins on the backplane connector to the whitening boards while switching the MEDM switches to toggle the whitening state. For all channels except ITMY UL, the behaviour was as expected, in line with the preceeding paragraph - the voltage swings between ~0V and ~15V. As mentioned in my post yesterday, the ITMY UL channel remains dodgy, with voltages of 12.84V (bit=1) and 10.79V (bit=0). So unless I am missing something, this must point to a faulty CONTEC card? We do have spares, do we want to replace this? It also looks like this problem has been present since at least 2011...
In any case, why should this lead to ITMY UL glitching? According to the MAX333 datasheet, the switch wants "low"<0.8V and "high">2.4V - so even if the CONTEC card is malfunctioning and the output is toggling between these two states, the condition should be that the whitening stage is always bypassed for this channel. The bypassed route works just fine, I measured the transfer function and it is unity as expected.
So what could possibly be leading to the glitches? I doubt that replacing the BO card will solve this problem. One possibility that came up in today's meeting is that perhaps the +24V to the Sat. Box. (which is used to derive the OSEM LED drive current) may be glitching - of course we have no monitor for this, but given that all the Sat. Amp. Adaptor boards are on 1X5 near the Acromag, perhaps Lydia and Johannes can recommission the PSL diagnostic Aromag to a power supply monitoring Acromag?
What do these glitches look like anyway? Here is a few second snapshot from one of the many MC1 excursions from yesterday - the original glitch itself is very fast, and then that gives an impulse to the damping loop which eventually damps away.
And here is one from when there was a glitch when the tester box was plugged in to the ITMY signal chain (so we can rule out anything in the vacuum, and also the satellite box itself as the glitches seem to remain even when boxes are shuffled around, and don't migrate with the box). So even though the real glitch happens in the UL channel (note the y axes are very different for the channels), the UR, LR and LL channels also "feel" it. recall that this is with the tester box (so no damping loops involved), and the fact that the side channel is more immune to it than the others is hard to explain. Could this just be electrical cross-coupling?
Still beats me what in the signal chain could cause this problem.
Some good news - Koji was running some tests on the modified WFS demod board and locked the IMC for this. We noticed that MC1 seemed well behaved for extended periods of time unlike last night. I realigned the PMC and IMC, and we have been having lock streches of a few hours as we usually have. I looked at the MC1 OSEM PD readbacks during the couple of lock losses in the last few hours, and didn't notice anything dramatic . So if things remain in this state, at least we can do other stuff with the IFO... I have plugged in the ITMY sat. box again, but have left the watchdog disabled, lets see what the glitching situation is overnight... The original ITMY sat. box has been plugged into the ETMY DAQ signal chain with a tester box. The 3 day trend supports the hypothesis the sat. box is not to blame. So I am plugging the ETMY suspension back in as well...
To diagnose the glitches in OSEM readouts, we have removed one of the PCIE BO D37 to IDE50 adaptor boxes from 1X5. All the watchdogs were turned off, and the power to the unit was cut before the cables on the front panel were removed. I am working on the diagnosis, I will update more later in the evening. Note that according to the c1sus model, the box we removed supplies backplane logic inputs that control whitening for ITMX, ITMY, BS and PRM (in case anyone is wondering/needs to restore damping to any of these optics). The whitening settings for the IMC mirrors resides on the other unit in 1X5, and should not be affected.
I was talking with Larry yesterday, and he suggested the rack-mounted supermicro machines SYS-5017A-EP (~$400) or SYS-5018A-FTN4 (~$600) that he uses for moving data around in LIGO. They have ≥2 gigabit ethernet ports and can thus function as modbus gateways, conveniently placed in the rack close to the slow DAQ/DIO chassis and running some local ubuntu or other distro (I think Aidan uses CentOS in the PSL lab). These only have atom processors, which would be sufficient for the slow machine replacement, but there are many more powerful models with sometimes subtle differences. If we motion towards a more complete GigECam coverage in the lab it could be better to kill two birds with one stone and get something a little faster that can do the video capture/processing, since these machines will be distributed more or less strategically around the lab. Just a thought, as I have currently no clear idea what resources are required for this or how much we're throwing at this GigECam upgrade.
I've attached a schematic for how we will connect the Acromag mosules to the slow channel I/O curently going to c1auxex. The following changes are made:
The new SOS sus wire finally is stored in a nitrogen filled dessicator. This was recommended by Ca. Fine Wire to minimize the aging - oxidation.
The dessicator was pumped down with " aux-drypump " to 1 Torr and than filled up with N2 to 760 Torr. This was repeated 2x and the dessicator was sealed off.
Detailed story below...
Part 1: Satellite box swap
Yesterday, I switched the ITMY and ETMY satellite boxes, to see if the problems we have been seeing with ITMY UL move with the box to ETMY. It did not, while ITMY UL remained glitchy (based on data from approximately 10pm PDT on 28Nov - 10am PDT 29 Nov). Along with the tabletop diagnosis I did with the tester box, I concluded that the satellite box is not to blame.
Part 2: Tracing the signal chain (actually this was part 3 chronologically but this is how it should have been done...)
So if the problem isn't with the OSEMs themselves or the satellite box, what is wrong? I attempted to trace the signal chain from the satellite box into our CDS system as best as I could. The suspension wiring diagram on our wiki page is (I think) a past incarnation. Of course putting together a new diagram was a monumental task I wasn't prepared to undertake tonight, but in the long run this may be helpful. I will put up a diagram of the part I did trace out tomorrow, but the relevant links for this discussion are as follows (? indicates I am unsure):
I have linked to the DCC page for the various parts where available. Unfortunately I can't locate (on new DCC or old or elog or wiki) drawings for D010069 (Satellite Amplifier Adapter Board), D080281 ("anti-aliasing interface)" or D080478 (which is the binary output breakout box). I have emailed Ben Abbott who may have access to some other archive - the diagrams would be useful as it is looking likely that the problem may lie with the binary output.
So presumably the first piece of electronics after the Satellite box is the PD whitening board. After placing tags on the 3 LEMOs and 1 DB15 cable plugged into this board, I pulled out the ITMY board to do some tabletop diagnosis in the afternoon around 2pm 29Nov.
Part 3: PD whitening board debugging
This particular board has been reported as problematic in the recent past. I started by inserting a tester board into the slot occupied by this board - the LEDs on the tester board suggested that power-supply from the backplane connectors were alright, confirmed with a DMM.
Looking at the board itself, C4 and C6 are tantalum capacitors, and I have faced problems with this type of capacitor in the past. In fact, on the corresponding MC3 board (which is the only one visible, I didn't want to pull out boards unnecessarily) have been replaced with electrolytic capacitors, which are presumably more reliable. In any case, these capacitors do not seem to be at any fault, the board receives +/-15 V as advertised.
The whitening switching is handled by the MAX333 - this is what I looked at next. This IC is essentially a quad SPDT switch, and a binary input supplied via the backplane connector serves to route the PD input either through a whitening filter, or bypass it via a unity gain buffer. The logic levels that effect the switching are +15V and 0V (and not the conventional 5V and 0V), but according to the MAX333 datasheet, this is fine. I looked at the supply voltage to all ICs on the board, DC levels seemed fine (as measured with a DMM) and I also looked at it on an oscilloscope, no glitches were seen in ~30sec viewing stretch. I did notice something peculiar in that with no input supplied to the MAX333 IC (i.e. the logic level should be 15V), the NO and NC terminals appear shorted when checked with a DMM. Zach has noticed something similar in the past, but Koji pointed out that the DMM can be fooled into thinking there is a short. Anyway, the real test was to pull the logic input of the MAX333 to 0, and look at the output, this is what I did next.
The schematic says the whitening filter has poles at 30,100Hz and a zero at 3 Hz. So I supplied as "PD input" a 12Hz 1Vpp sinewave - there should be a gain of ~x4 when this signal passes through the path with the whitening filter. I then applied a low frequency (0.1Hz) square wave (0-5V) to the "bypass" input, and looked at the output, and indeed saw the signal amplitude change by ~4x when the input to the switch was pulled low. This behaviour was confirmed on all five channels, there was no problem. I took transfer functions for all 5 channels (both at the "monitor" point on the backplane connector and on the front panel LEMOs), and they came out as expected (plot to be uploaded soon).
Next, I took the board back to the eurocrate. I first put in a tester box into the slot and measured the voltage levels on the backplane pins that are meant to trigger bypassing of the whitening stage, all the pins were at 0V. I am not sure if this is what is expected, I will have to look inside D080478 as there is no drawing for it. Note that these levels are set using a Contec binary output card. Then I attached the PD whitening board to the tester board, and measured the voltages at the "Input" pins of all the 5 SPDT switches used under 2 conditions - with the appropriate bit sent out via the Contec card set to 0 or 1 (using the button on the suspension MEDM screens). I confirmed using the BIO medm screen that the bit is indeed changing on the software side, but until I look at D080478, I am not sure how to verify the right voltage is being sent out, except to check at the pins on the MAX333. For this test, the UL channel was indeed anomalous - while the other 4 channels yielded 0V (whitening ON, bit=1) and 15V (whitening OFF, bit=0), the corresponding values for the UL channel were 12V and 10V.
I didn't really get any further than this tonight. But this still leaves unanswered questions - if the measured values are faithful, then the UL channel always bypasses the whitening stage. Can this explain the glitchy behaviour?
Part 4: MC1 troubles
At approximately 8pm, the IMC started losing lock far too often - see the attached StripTool trace. There was a good ~2hour stretch before that when I realigned the IMC, and it held lock, but something changed abruptly around 8pm. Looking at the IMC mirror OSEM PD signals, all 5 MC1 channels are glitching frequently. Indeed, almost every IMC lockloss in the attached StripTool is because of the MC1 PD readouts glitching, and subsequently, the damping loops applying a macroscopic drive to the optic which the FSS can't keep up with. Why has this surfaced now? The IMC satellite boxes were not touched anytime recently as far as I am aware. The MC1 PD whitening board sits in the same eurocrate I pulled the ITMY board out of, but squishing cables/pushing board in did not do anything to alleviate the situation. Moreover, MC2 and MC3 look fine, even though their PD whitening boards also sit in the same eurocrate. Because I was out of ideas, I (soft) restarted c1sus and all the models (the thinking being if something was wrong with the Contec boards, a restart may fix it), but there was no improvement. The last longish lock stretch was with the MC1 watchdog turned off, but as soon as I turned it back on the IMC lost lock shortly after.
I am leaving the autolocker off for the night, hopefully there is an easy fix for all of this...
more U4 gain, lesssss U5 gain
We found that someone had violated all rules of computer security decency and was storing our nodus password as a plain text file in their bash_profile.
After the flogging we have changed the pwd and put the new one in the usual secret place.
Summary: The demodulator input noise level was improved by a factor of more than 2. This was not as much as we expected from the preamp noise improvement, but is something. If this looks OK, I will implement this modification to all the 16 channels.
The modification shown in Attachment 1 has actually been applied to a channel.
Attachment 2 shows the measured and expected output signal transfer of the demodulator. The actual behavior of the demodulator is as expected, and we still keep the over all LPF feature of 3rd order with fc=~1kHz.
Attachment 3 shows the improvement of the noise level with the signal reffered to the demodulator input. The improvement by a factor >2 was observed all over the frequency range. However, this noise level could not be explained by the preamp noise level. Actually this noise below 1kHz is present at the output of the demodulator. (Surprisingly, or as usual, the noise level of the previous preamp configuration was just right at the noise level of the demodulator below 100Hz.) The removal of the offset trimmer circuit contributed to the noise improvement below 0.3Hz.
400 days plot. Satelite amp ITMY has been swapped with ETMY
Unlabeled sat.amps are labeled. This plot only makes sense if you know the Cuh-Razy sat amp locations.
I left the tester box plugged in from Thursday night to Sunday afternoon, and in this period, the glitches still appeared in (and only in) the UL channel.
So yesterday evening, I pulled the Sat. Box. out and checked the DC voltages at various points in the circuit using a DMM, including the output of the high current buffer that supplies the drive current to the shadow sensor LEDs. When we had similar behaviour in the PRM box, this kind of analysis immediately identified the faulty component as the high current buffer IC (LM6321M) in the bad channel, but everything seems in order for the ITMY box.
I then checked the Satellite Amplifier Termination Board, which basically just adds 100ohm series resistors to the output of the PD readout, and all the resistors seem fine, the piece of insulating material affixed to the bottom of this board is also intact. I then used the SR785 in AC coupled mode to look at the high frequency spectrum at the same points I checked the DC voltages with the DMM (namely the drive voltage to the LEDs, and the PD readout voltages on the PCB as well as on the pins of the connector on the outside of the box after the termination board (leading to the DAQ), and nothing sticks out here in the UL channel either. Of course it could be that the glitches are intermittent, and during my tests they just weren't there...
I am hesitant to start pulling out ICs and replacing them without any obvious signs of failure from them, but I am out of debugging ideas...
One possibility is that the problem lies upstream of the Sat. Box - perhaps the UL channel in the Suspension PD Whitening and Interface Board is faulty. To test, I have now hooked up ITMY Sat. Box. + tester box to the signal chain of ETMY. If I can get the other tester box back from Ben, I will plug in the ETMY sat. box. + tester to the ITMY signal chain. This should tell us something...
The vacuum envelope ~1 C warmer today than 7 days ago. Bad peaks are coming down as normal.
1. The response of the IMC WFS board was measured. The LO signal with 0.3Vpp@29.5MHz on 50Ohm was supplied from DS345. I've confirmed that this signal is enough to trigger the comparator chip right next to the LO input. The RF signal with 0.1Vpp on the 50Ohm input impedance was provided from another DS345 to CH1 with a frequency offset of 20Hz~10kHz. Two DS345s were synced by the 10MHz RFreference at the rear of the units. The resulting low frequency signal from the 1st AF stage (AD797) and the 2nd AF stage (OP284) were checked.
Attachment 1 shows the measured and modelled response of the demodulator with various frequency offsets. The value shows the signal transfer (i.e. the output amplitude normalized by the input amplitude) from the input to the outputs of the 1st and 2nd stages. According to the datasheet, the demodulator chip provides a single pole cutoff of 340kHz with the 33nF caps between AP/AN and VP. The first stage is a broadband amplifier, but there is a passive LPF (fc=~1kHz). The second stage also provides the 2nd order LPF at fc~1kHz too. The measurement and the model show good agreement.
2. The output noise levels of the 1st and 2nd stages were meausred and compared with the noise model by LISO.
Attachment 2 shows the input referred noise of the demodulator circuit. The output noise is basically limited by the noise of the first stage. The noise of the 2nd stage make the significant contribution only above the cut off freq of the circuit (~1kHz). And the model supports this fact. The 6.65kOhm of the passive filter and the input current noise of AD797 cause the large (>30nV/rtHz) noise contribution below 100Hz. This completely spoils the low noiseness (~1nV/rtHz) of AD797. At lower frequency like 0.1Hz other component comes up above the modelled noise level.
3. Rana and I had a discussion about the modification of the circuit. Attachment 4 shows the possible improvement of the demod circuit and the 1st stage preamplifier. The demodulator chip can have a cut off by the attached capacitor. We will replace the 33nF caps with 1uF and the cut off will be pushed down to ~10kHz. Then the passive LPF will be removed. We don't need "rodeo horse" AD797 for this circuit, but op27 is just fine instead. The gain of the 1st stage can be increased from 9 to 21. This should give us >x10 improvement of the noise contribution from the demodualtor (Attachment 3). We also can replace some of the important resistors with the thin film low noise resistors.
We removed one set of the MC WFS demod board and whitening board from the IOO rack for the investigation.
The MC WFS servo loops are disabled with the EPICS screens.
Let us know when you need the MC WFS boards to be returned to the rack.
This is to investigate the signal chain and fix some issues. We ramped down the -100 V supply for the WFS QPD bias (why is it so big?), but everything else is still on. Koji is doing demod board. Rana will upload a marked up WFS whitening board schematic soon.
Previous elog entries on this:
I've noticed that the glitchy behaviour in ITMY UL shadow sensor readback is back - as mentioned above, I looked at the Sat. Box and could not find anything wrong with it, perhaps I'll plug the tester box in over the Thanksgiving weekend and see if the glitches persist...
In the Generic Pentek interface board, which is used to take in the analog 2-pin LEMO cable from the MC Servo board, there is some analog whitening before the signal is sent into the ADC.
There are jumpers in there to set whether it is 0, 1, or 2 stages of 150:15 (z:p) whitening.
TP3 drypump replaced at 655 mTorr, no load, tp3 0.3A
This seal lasted only for 33 days at 123,840 hrs
The replacement is performing well: TP3 foreline pressure is 55 mTorr, no load, tp3 0.15A at 15 min [ 13.1 mTorr at d5 ]
Valve configuration: Vacuum Normal, ITcc 8.5E-6 Torr
Dry pump of TP3 replaced after 9.5 months of operation.[ 45 mTorr d3 ]
The annulosses are pumped.
Valve configuration: vac normal, IFO pressure 4.5E-5 Torr [1.6E-5 Torr d3 ] on new ITcc gauge, RGA is not installed yet.
Note how fast the pressure is dropping when the vent is short.
IFO pressure 1.7E-4 Torr on new not logged cold cathode gauge. P1 <7E-4 Torr
Valve configuration: vac.normal with anunulossess closed off.
TP3 was turned off with a failing drypump. It will be replaced tomorrow.
All time stamps are blank on the MEDM screens.
I wanted to get a clearer idea of the FSS servo and the various boxes in the signal chain and so Lydia and I poked around the IOO rack and the PSL table - I will post a diagram here tomorrow.
We then wanted to characterize the existing loop. It occurred to me later in the evening to measure the plant itself to verify the model shape used to construct the invP filter in the feedback path. I made the measurement with a unity gain control path, and I think there may be an extra zero @10Hz in the model.
Earlier in the evening, we measured the OLG of the MCL loop using the usual IN1/IN2 prescription, in which above 10Hz, the measurement and FOTON disagree, which is not surprising given Attachment #1.
I didn't play around with the loop shape too much tonight, but we did perform some trials using the existing loop, taking into account some things I realized since my previous attempts. The summary of the performanceof the existing loop is:
All of the above is summarized in the below plots - this behaviour is (not surprisingly) in line with what Den observed back when he put these in.
The eventual goal here is to figure out if we can get an adaptive feedback loop working in this path, which can take into account prevailing environmental conditions and optimally shape the servo to make the arms follow the laser frequency more closely at low frequencies (i.e. minimize the effect of the noise injected by IMC length fluctuations at low frequency). But first we need to make a robust 'static' feedback path that doesn't inject control noise at higher frequencies, I need to think a little more about this and work out the loop algebra to figure out how to best do this...
Current Acromag chassis status:
I found out that Acromag offers DIN rail mounting kits for the open boards, so we can actually fit both XT series and ES/EN series in the same boxes, depending on the signal needs. The primary design driver will be the ES footprint, but if we find we don't need that many channels in some of the units, it's interchangable. For the wiring to the front panel - for which we will have a standard front panel express design, but may order modified ones for the custom needs of the 40m, I will contract the same company that Rich used for the wiring in his DIO box (Panel mount connectors terminating in loose wires/pre-routed plugs for Acromag units). We will either run a single DIN rail along the length of the chassis, or have two in parallel across.
Lydia and I took close looks at the breakout arrangements on the rack sides, and determined that because of the many cross-connects between non-DAQ ports it is not possible to redo and debug this in a reasonable amount of time without essentially shutting down the interferometer. So instead, we will connect the chassis directly to the slots that were previously leading to the slow machines. They come in two different flavors: The ADC modules have 64 pins, while the DIO and DAC ones have 50. There are a couple things we can do:
Based on Rich's design I will get started on a parts list and wiring diagrams to send out to the cable company.
The Rga was turned on yesterday.
The RGA is removed for repaire. It's volume at atmophere and sealed.. P4 reading of 38 Torr is not correct.
We connected and powered up the Acromag chassis today. It lives in 1X4 and is powered by the Sorensen +20V power supply in 1X5 via the fuse rail on the side of 1X4. For this we had to branch off the 20V path to the dewhitening and anti-image filter crate of the c1:susaux driven SOS optics. After confirming that none of the daughter modules in the crate draw from the 20V line, we added a wire leading to a new fuse we added for this unit and ran a power cable from there.
The diagnostic connector of the PSL laser is now connected to the unit and a tmux session was created on megatron that interfaces with the chassis and broadcasts the EPICS channels. We need to watch out in the coming days for epics freezes/outages, as in the past these seemed to occur around the same times we were toying with the Acromags.
We set up the chassis in 1X7 today. Steve is ordering a longer 25 pin cable to reach. Until then the PSL diagnostic channels will not be usable.
Following up on the discussion from last week's Wednesday meeting, two points were raised:
With regards to the coating on the AR side, I've put in R<300ppm@1064nm and R<1000ppm@532nm on the AR side. On the HR side, we have T>97% @ 532nm (copied from the current PR3/SR3 spec), and T<50ppm @1064nm. What are the ghost beams we need to be worried about?
So in conclusion, with the specs as they are now, I don't think the ALS noise performance is adversely affected. I have updated the spec to have the following numbers now.
HR side: T < 50ppm @1064nm, T>99.9% @532nm
AR side: R < 100ppm @1064nm and @532nm
As for the POP question, if we want to extract a stronger POP beam, we will have to relax the requirement on the transmission @1064nm on the HR side. But recall that the approach we are now considering is to replace only PR3, and flip PR2 back the right way around. Currently, POP is extracted at PR2, so if we want to stick with the idea of getting a new PR3 and extracting a stronger POP beam, there needs to be a major optical layout reshuffle in the BS/PRM chamber. Koji suggested that in the interest of keeping things moving along, we don't worry about POP for the time being...
Alternatively, if it turns out that the vendor can meet the specs for our second requirement (which requires 1.5% of lambda @632nm measurement precision to meet the 10+/-5km RoC tolerance on PR3), then we can ast for T<1000ppm @1064nm for the HR coating on PR2, and keep the coating specs on PR3 as above.
Attached is a pdf with the specs updated to reflect all the above considerations...
Over the weekend, I was successful in locking the DRMI with the arms held on ALS. The locks were fairly robust, lasting order of minutes, and was able to reacquire by itself when it lost the lock in <1min. I had to tweak the demod phases and loop gains further compared to the 1f lock with no arms, but eventually I was able to run a sensing matrix measurement as well. A summary of the steps I had to follow:
I've updated the appropriate fields in the restore script. Now that the DRMI locking is somewhat stable again, I think the next step towards the full lock would be to zero the CARM offset and turning on the AO path.
On the downside, I noticed yesterday that ITMY UL shadow sensor readback was glitching again - for the locking yesterday, I simply held the output of that channel to the input matrix, which worked fine. I had already done some debugging on the Sat. Box with the help of the tester box, but unlike the PRM sat. box, I did not find anything obviously wrong with the ITMY one... I also ran into a CDS issue when I tried to run the script that sets the phase tracker UGF - the script reported that the channels it was supposed to read (the I and Q outputs of the ALS signal, e.g. C1:ALS-BEATX_FINE_I_OUT) did not exist. The same channels worked on dataviever though, so I am not sure what the problem was. Some time later, the script worked fine too. Something to look out for in the future I guess..
Last jump at rack Y2.
I made a very slighly updated version of Yinzi's script that pulls the channel names and actuator hardstop limits from an external .ini config file. The idea was to avoid having as many versions of the script as there are implimentations of it. Seems like slighly better practice, but maybe I'm wrong. The config files are also easier to read. Its posted on the elog (PSL:1758) with lastest on the 40mSVN .../trunk/CTNLab/current/computing/scripts .
If you're working off her first implimentation 'RCAV_thermalPID.py' then there is an indent issue after the if statement on line 88: only line 89 should be indended. If you deactivate the debug flag then the script fails to read in PID factors and dies.
[yinzi, craig, gautam]
Yinzi had translated the Perl PID script used to implement the discrete-time PID control, and had implemented it with Andrew at the PSL lab. Today afternoon we made some minor edits to make this suitable for the FSS Slow loop (essentially just putting the right channel names into her Python script). I then made an init file to run this script on megatron, and it looks to be working fine over the last half hour of observation or so. I am going to leave things in this state over the weekend to see how it performs.
We have been running with just the MC2 Transmission QPD for angular control of the IMC for a couple of months now because the WFS loops seemed to drag the alignment away from the optimum. We did the following to try and re-engage the WFS feedback:
GV addendum 23Nov2016: The WFS have been working well over the last few days - I've had to periodically (~ once in a day) run the WFS reflief script to keep the outputs to the suspension PIT and YAW DOFs below 50cts, but the WFS aren't dragging the alignment away as we had noticed before. The only thing I did differently is to follow Rana's suggestion and set the RF offsets with the MC unlocked as opposed to locked. I've added a line to the script to remind the user to do so... Also, note that EricQ has recently cleaned up the scripts directory to remove the numerous obsolete scripts in there...
Vivitek D952HD sn2160130 was send out for warranty repair. It's hard to believe that it has a 5 year warranty...... RMA - WR16004483.....expected to be back by Friday, Dec 2
The projector failed just now with a pretty loud 'pop' sound - I've never been present when the lamp goes out, so I don't know if this is usual. I have left the power cable unplugged for now...
Replacement is ordered Nov 4
I had Rich show me his approach to a chassis for the Acromag modules. The document tree for his design can be found on the DCC. Note that he's using the high densitymodel ES series, which is available as a bare board variant with pluggable screw terminals:
He can fit up to 4 of these in a 2U chassis and has outsourced the wiring from front panel Dsubs to the board connectors to an external company. At the 40m (and in West Bridge) we currently only have the rail mounted XT series
At first glance the specs are very similar. Both A/D and D/A flavors have 16-bit precision in both cases. The high density ES series with Rich's layout can achieve 128 A/D per 2U, 64 D/A per 2U, or 384 DIO per 2U. Into a 4U chassis of the type we have currently we can fit ~32 XT modules (assuming two rows), which results in very similar numbers, except for the DAC, of which we could fit more.
XT1221-000 (8 diff. channel 16-bit ADC) $495.00 $61.88/ch
XT1541-000 (8 channel 16-bit DAC and 4 discrete I/O ) $525.00 $65.63/ch
XT1120-000 (16 channel DIO) $320.00 $20.00/ch
ES2162-0010 (32 diff. channel 16-bit ADC) $2050.00 $64.06/ch
ES2172-0010 (16 channel 16-bit DAC) $1400.00 $87.50/ch
ES2113-0010 (96 channel DIO) $1100.00 $11.46/ch
It's cheaper to stick with the current XT models, but they need the bulkier 4U chassis. The good news is that actually all these models have 16 bit precision, which wasn't clear to me before. Lydia and I will work out what connectors we want on the boxes, and how many modules/channels we need where. Rich also got me in touch with Keith Thorne, who handles the analog I/O Acromag at LLO, and I will ask him for advice. From his documents on the DCC it seems that he is using yet another series: EN. The 968EN-4008 for example is a rail-mounted ADC with pluggable connections, but looses quite clearly in price per channel.
For a generic multipurpose DAQ interface box the ES series is the best approach in my opinion, because it offers a more compact design. We could for example fit 1 ADC, 2 DAC, 1 DIO in a 2U chassis for 32/32/96 channels. The combined price tag for this scenario would be ~$6k.
I don't like AS110 or AS55. Neither of them are designed for DC and so the DC readout chain is hokey. How about use an actual transimpedance PD with a 100-1000 Ohm resistor and a 3 mm diode? This would eliminate the alignment sensitivity and the drifts due to electronics and room lights.
This looks much better. I'm planning to take more data with the AS110 PD rather than AS55 when I get the chance, increase the averaging time, and also sigma filter the datapoints. That should get us to a good spot and cut down the uncertainty even further.
As a starting point, I was looking at some of the old elogs and tried turning on the MCL feedback path with the existing control filters today. I tried various combinations of MCL Feedback and FF on and off, and looked at the MCL error signal (which I believe comes from the analog MC servo board?) spectrum for each case. We had used this earlier this year when EricQ and I were debugging the EX laser frequency noise to stabilize the low frequency excursions of the PSL frequency. The low frequency suppression can be seen in Attachment #1, there looks to be some excess MCL noise around 16Hz when the servo is turned on. But the MC transmission (and hence the arm transmission) decays and gets noisier when the MCL feedback path is turned on (see Attached StripTool screenshots).
Indeed. I suggest discussing with Joe B. I believe we should use a dedicated cam network to get the camera signals from the ends and corner all into one machine. Do not use the main CDS FE network for this since it might produce weird collissions. How about make a diagram, post it to elog, and send link to Joe?
It may be a good idea to leave the gigecam interfacing up to a dedicated machine.
I am currently looking at the acoustic noise around both arms to see if there are any frequencies from machinery around the lab that stand out and to see what we can remove/change.
After much trial and error with whitening gains, demod phases and overall loop gains, I was finally able to lock the DRMI on both 1f and 3f signals! I went through things in the following order tonight:
I have noted all the settings I used tonight, I will post them tomorrow. I was planning to try a DRFPMI lock if I was successful with the DRMI earlier tonight, but I'm calling it a night for now. But I think the DRMI locking is now back to a reliable level, and we can push ahead with the full IFO lock...
It remains to update the auto-configure scripts to restore the optimized settings from tonight, I am leaving this to tomorrow as well...
Updated 16 Nov 2016 1130am
Settings used were as follows:
I had a mistake in my script that reported the wrong error after averaging several datapoints, and because I hadn't looked at the individual numbers I didn't catch it so far. Thanks to Gautam it is no more.
The updated numbers are (with fresh, more trustworthy data):
XARM: 21 +/ 35 ppm
YARM: 69 +/- 45 ppm
I powered up the existing ace100gm GigE cam with the PoE injector and tried to interface with it as described in elog 4163. After a few initial problems with IP assignment and interfacing I connected it to one of the gigabit hubs and installed the most recent pre-compiled software suite on /opt/pylon5 on optimus, after which I was able to find it with the configuration software. I named it "c1gige_bas100-1" and gave it the static IP address 192.168.113.151.
Afterwards the image acquisition worked without problems.
It may be a good idea to leave the gigecam interfacing up to a dedicated machine. I was thinking I could use Mafalda for this, and also for developing the code for framegrabbing and imager settings, but found that it was dead, burnt at the stake so to say. I guess it wasn't running anything critical, since it wasn't even connected to the network and smelled like burnt electronics. I'll get a replacement desktop for it.
PRM and SRM sat. boxes have been switched for some time now - but the PRM sat. box has one channel with a different transimpedance gain, and the damping loops for the PRM and SRM were not systematically adjusted to take this into account (I just tweaked the gain for the PRM and SRM side damping loops till the optic damped). Since both sat. boxes are nominally functioning now, I saw no reason to maintain this switched configuration so I swapped the boxes back, and restored the damping settings to their values from March 29 2016, well before either of this summer's vents. In addition, I want to collect some data to analyze the sat. box noise performance so I am leaving the SRM sat. box connected to the DAQ, but with the tester box connected to where the vacuum feedthroughs would normally go (so SRM has no actuation right now). I will collect a few hours of data and revert later tonight for locking activities....
I did apt-get update and then apt-get upgrade on optimus. All systems are nominal.
Looking back at elog 12528, the uncertainty in the armloss number from the individual quantities in the equation for can be written as:
Making some generous assumption about the individual uncertainties and filling in typical values we get in our measurements, results in the following uncertainty budget:
In my recent round of measurements I had a 2.5% uncertainty in the ASDC reading, which completely dominates the armloss assessment.
The most recent numbers are 57 ppm for the YARM and 21 ppm for the XARM, but both with an uncertainty of near 150 ppm, so while these numbers fit well with Gautam's estimate of the average armloss via PRG, it's not really a confirmation.
I set the whitening gain in ASDC to 24 dB and ran LSC offsets, and now I'm getting a relative uncertainty in measured reflected power of .22%, which would be sufficient for ~25ppm accuracy according to the above formula. I'm going to start a series of measurements tonight when I leave, should be done in ~2 hours (10 pm) the latest.
If anybody wants to do some night work: I misaligned ITMY by a lot to get its reflection off ASDC. Approximate values are saved as a restore point. Also the whitening gain on ASDC will have to be rolled back (was at 0dB) and LSC offsets adjusted.