After labeling all cables, I pulled out one of the D990694s in the LSC rack (the one used for the ALS X signals, it is Rev-B1, S/N 118 according to the sticker on it).
Took some photos before cutting anything. Attachments #1-3 are my cutting plans (shown for 1 channel, plan is to do it for both ALS channels coming into this board). #1 & #2 are meant to show the physical locations of the cuts, and #3 is the corresponding location on the schematic. These are the most convenient locations I could identify on the board for this operation.
I don't know what the purpose of resistors R196, R197, R198 are. I'm assuming it has something to do with the way the ADG333ABR switches. The aLIGO board uses a different switch (MAX4659EUA+), and doesn't have an analogous resistor (though from what I can tell, it too is a CMOS SPDT switch just like the ADG333ABR, just has a lower ON resistance of 25ohm vs 45ohm for the ADG333ABR).
As for the actual resistance to be used: Let's say we don't have signals > 5V coming into this board. Then using 301ohms (as in the aLIGO boards) in series means the peak current draw will be <20mA, which sounds like a reasonable number to me. Larger series resistance is better, but I guess then the contribution of the current noise of the OpAmp keeps increasing.
This is proving much more challenging than I thought - while Cut #1 was easy to identify and execute, my initial plan for Cut #2 seems to not have isolated the input of the second opamp (as judged by DMM continuity). Koji pointed out that this is actually not a robust test, as the switches are in an undefined state while I am doing these tests with the board unpowered. It seems rather complicated to do a test with the board powered out here in the office area though - and I'd rather not desolder the 16 and 20 pin ICs to get a better look at the tracks. This PCB seems to be multilayered, and I don't have a good idea for what the hidden tracks may be. Does anyone know of a secret place where there is a schematic for the PCB layout of this board? The DCC page only has the electrical schematic drawings, and I can't find anything useful on the elog/wiki/old ilog on a keyword search for this DCC document number. The track layout also is not identical for all channels. So I'm holding off on exploratory cuts.
*I've asked Ben Abbott/Mike Pedraza about this and they are having a look in Dale Ouimette's old drives to see if they can dig up the Altium/Protel files.
I quickly put together some code that calculates the THD from CDS data and generates a plot (see e.g. Attachment #1).
I conducted a trial on the Y arm ALS channel whitening board (while the X arm counterpart is still undergoing surgery). With the whitening gain set to 0dB, and a 1Vpp input signal (so nothing should be saturated), I measure a THD of ~0.08% according to the above formula. Seems rather high - the LT1125 datasheet tells us to expect <0.001% THD+N at ~100Hz for a closed loop gain of ~10. I can only assume that the digitization process somehow introduces more THD? Of course the FoM we care about is what happens to this number as we increase the gain.
I'm going to work on putting together some code that gives me a quick readback on the measured THD, and then do the test for real with different amplitude input signal and whitening gain settings.
The main motivation for this work is that I want +15VDC power available on the PSL table to hookup the Teledyne box that Koji made a week ago and do some noise measurements on my revised IR ALS signal chain. But I think this is a good opportunity to effect a number of changes I've been wanting to do for a while.
Tomorrow, Steve and I will do the following:
So in summary, we will need, at 1X1, (at least, including 1 spare for future work):
We completed this work today. Need to clean up a little (i.e. coil excess cable lengths, remove unused cables etc), which we will do tomorrow. All connections have been made at the DIN rail end, but the fuses have not been inserted yet, so there is no voltage reaching the PSL table on any of the newly laid out cables. We also need to establish two +15VDC connections at the DIN rail side. I may establish this later in the evening, as the main point of this work was to get the Teledyne signal path operational. Setting up these DIN connectors is actually a huge pain, we tried to setup a few extra ports for the voltages we used today so that in future, life is easier for whoever wants to pipe DC power to the PSL table. The rule is, however, to re-establish the same number of open ports for each voltage as was available when you started.
For the ZHL-3A, Teledyne, and AOM driver cables, we used 18AWG, 2 conductor, twisted wire, while for the PSL fan we used 20AWG. For the FSS box, we decided to use the 3 conductor 24AWG twisted wire. I believe that these wire gauge choices are appropriate given the expected current in each of these paths.
Pictures + further details tomorrow.
gautam @ 1030pm: there was some mistake with the +15V wiring we did in the evening (the PSL fan and Teledyne cables were plugged into the wrong DIN terminal blocks). I fixed this, and also routed +15VDC to the newly installed set of terminal blocks for this purpose (since we had run out of +15VDC ports at 1X1). After checking voltages at both 1X1 and on the PSL table, I hooked up
to their newly laid out power supplies. IMC locks so looks like the FSS box is doing fine . So we can recover one bench power supply from under the PSL table on the east side. I didn't hook up the AOM driver just now because of some accessibility issues, and I'd also like to do an ALS beat spectrum measurement if possible.
I have been puzzled as to why the duty cycle of the EX green locks are much less than that of the EY NPRO. If anything, the PDH loop has higher bandwidth and comparable stability margins at the X end than at the Y end. I hypothesize that this is because the EX laser (Innolight 1W Mephisto) has actuation PZT coefficient 1MHz/V, while the EY laser (Lightwave 125/126) has 5MHz/V. I figure the EX laser is sometimes just not able to keep up with the DC Xarm cavity length drift. To test this hypothesis, I disabled the LSC locking for the Xarm, and enabled the SLOW (temperature of NPRO crystal) control on the EX laser. The logic is that this provides relief for the PZT path and prevents the PDH servo from saturating and losing lock. Already, the green lock has held longer than at any point tonight (>60mins). I'm going to leave it in this state overnight and see how long the lock holds. The slow servo path has a limiter set to 100 counts so should be fine to leave it on. The next test will be to repeat this test with LSC mode ON, as I guess this will enhance the DC arm cavity length drift (it will be forced to follow MCL).
Why do I care about this at all? If at some point we want to do arm feedforward, I thought the green PDH error signal is a great target signal for the Wiener filter calculations. So I'd like to keep the green locked to the arm for extended periods of time. Arm feedforward should help in lock acquisiton if we have reduced actuation range due to increased series resistances in the coil drivers.
As an aside - I noticed that the SLOW path has no digital low pass filter - I think I remember someone saying that since the NPRO controller itself has an in-built low pass filter, a digital one isn't necessary. But as this elog points out, the situation may not be so straightforward. For now, I just put in some arbitrary low pass filter with corner at 5Hz. Seems like a nice simple problem for optimal loop shaping...
gautam noon CNY2018: Looks like the green has been stably locked for over 8 hours (see Attachment #1), and the slow servo doesn't look to have railed. Note that 100 cts ~=30mV. For an actuation coefficient of 1GHz/V, this is ~30MHz, which is well above the PZT range of 10V-->10MHz (whereas the EY laser, by virtue of its higher actuation coefficient, has 5 times this range, i.e. 50MHz). Supports my hypothesis.
Having implemented the changes to the audio amplifier stage, I re-installed this unit at the LSC rack, and did some testing. The motivation was to determine the shape of the ALS error signal spectrum, so that I can design a whitening preamp accordingly. Attachment #1 is the measurement I've been after. The measurement was taken with EX NPRO PDH locked to the arm via green, and Xarm locked to MC via POX. Slow temperature relief servo for EX NPRO was ON. Here are the details:
Conclusion: In the current configuration, with x10 gain on the demodulated signals, we barely have SNR of 10 at ~500Hz. I think the generic whitening scheme of 2 zeros @15Hz, 2poles@150Hz will work just fine. The point is to integrate this whitening with the preamp stage, so we can just go straight into an AA board and then the ADC (sending this signal into D990694 and doing the whitening there won't help with the SNR). Next task is to construct a test daughter board that can do this...
c1mcs had died for some reason. Looking at dmesg, I see:
None of the other EPICS processes died. Not sure what to make of this. I was at the PSL table working, and had closed the PSL shutter to avoid MC autolocker trying to keep the MC locked while I was mucking about, but this shouldn't have had any effect on an EPICS process?
Anyway, I just logged into c1sus, stopped and restarted the model. IMC locks fine now.
After discussing with Koji, I decided to try and align the input beam polarization at the PSL fiber coupler to one of the special axes of the PM fiber. The motivation is to try and narrow down the source of the large RF beatnote amplitude drift I noticed and reported last night.
The setup for doing so is shown in Attachment #1 - essentially, I setup one of the newly purchased couplers in a mount, set up a PBS, and placed two photodiodes at the S and P ports of the PBS. The idea is to rotate the input coupler in its mount, thereby maximizing the PER (monitored on two Thorlabs PDA520s - I didn't check the gain balance of them).
I spent ~30mins doing some preliminary trials just now, and, I was able to achieve a PER of ~1/20. But I think much better numbers were reported in this SURF project (although I'm not entirely sure I understand that measurement). I will spend a little more time tweaking the alignment. The procedure is tricky as at some point, simply rotating the mount reduces the mode-matching efficiency into the fiber so much that it is not possible to get a meaningful PER measurement from the photodiodes. I'm adjouring for now, more to follow...
Current configuration of PSL free-space to fiber coupling is:
I had noticed that the RF beat amplitude was fluctuating by up to 20dBm as viewed on the control room analyzer. As detailed in my earlier elog, I suspected this to be because of random polarization drift between the PSL and EX fields incident on the Fiber coupled PDs. Since I am confident the problem is optical (as opposed to something funny in the electronics), we'd like to be able to isolate which of the many fiber segments is dominating the contribution to this random polarization drift.
Some useful references:
Procedure and details:
I wanted to lock the single arm POX/POY config to do some tests on the BeatMouth. But I was unable to.
Not sure what to make of all this, but I can lock the arms now.
Attachment #1 shows the ALS noise measurement today. Main differences from the spectrum posted last week is that
For comparison, I have plotted alongside today's measurement (left column) the measurement from last week (right column).
I made a voltage divider using a 20.47kohm and 1.07kohm (both values measured with a DMM). The whole thing is packaged inside a Pomona box I found lying around on the Electronics bench. I have hooked it up to the ALSY_I channel and will leave it so overnight. The INMON of this channel isn't DQed, but for this test, the 16Hz EPICS data will suffice. I've locked the EX laser to the arm, enabled slow temperature servo to allow overnight lock (hopefully) and disabled LSC mode (as locking the arm to the MC tends to break the green lock)
To convert the INMON counts to RF power, I will use (based on my earlier calibration of this monitor channel, see DCC document for the demod chassis).
1AM update: Attachment #1 shows that the RF amplitude has been relatively stable (less than 10% of nominal value variation) over the course of the last hour or so. Even though there is some low frequency drift over timescales of ~20mins, no evidence of the wild ~20dB amplitude changes I saw last week. The signs are encouraging...
overnight update: See Attachment #2 - looking at the past 11 hours of second trend data during which the arm stayed locked, there actually seems to have been more significant variation in the beatnote amplitude. Swings of up to 6dBm are seen on a ~20min timescale, while there is also some longer term drift over 12 hours by a couple of dBm. There is probably a systematic error in the Y-axis, as I measured the RF power at the input of the power splitter at the LSC rack to be ~3dBm, so I expect something closer to 0dBm to be the LO input power which is what I am monitoring. So further debugging is required - I think I'll start by aligning the X fiber coupled beam to one of the fiber's special axes.
To make this setup more permanent, I modified the c1lsc model to pipe the LO power monitor signals from the Demod chassis to unused channels ADC_0_25 (X channel LO) and ADC_0_26 (Y channel LO) in the c1lsc model. I also added a couple of CDS filter blocks inside the "ALS" namespace block in c1lsc so as to allow for calibration from counts to dBm. I didn't add any DQ channels for now as I think the slow EPICS records will be sufficient for diagnostics. It is kind of overkill to use the fast channels for DC voltage monitoring, but until we have acromag channels readily accessible at 1Y2, this will do.
Modified model compiled and installed successfully, though I have yet to restart it given that I'll likely have to do a major reboot of all vertex FEs
I thought a little bit about the design of the preamp we want for the demodulated ALS signals today. The requirements are:
Attachment #3 shows a design I think will work (for now it's a whiteboard sketch, I''ll make this a computer graphic tomorrow). I have basically retained the differential sending and receiving capabilities of the existing Audio I/F amplifier, but have incorporated some whitening gain with a pole at ~150Hz and zero at ~15Hz. I've preserved the DC gain of 10, which seems to have worked well in my tests in the last week or so. Attachments #1 and #2 show the liso modelled characteristics. Liso does not support input-referred noise measurements for differential voltage inputs, so I had to calculate that curve manually - I suspect there is some subtlety I am missing, as if I plot the input referred noise out to higher frequencies, it blows up quite dramatically.
Next step is to actually make a prototype of this. I am wondering if we need a second stage of whitening, as in the current config, we only get 20dB gain at 150Hz relative to DC. Yesterday's beat spectrum measurement shows that we can expect the frequency noise of the ALS signal at ~100Hz to be at the level of ~1uV/rtHz, but this is is around the ADC noise level? If so, 20dB of whitening gain may be sufficient?
Still have to make preamp prototype daughter board with the right whitening shape... This test suggests to me that I should also make the output differential sending...
*Side note: I was wondering why we need the differential receiving stage, followed by a difference amplifier, and then a differential sending stage. After discussing with Koji, we think this is to suppress any common-mode noise from the mixer outputs.
Using one of the prototype PCB boards given to me by Johannes, I put together v1 of this board and tested it.
Attachment #1 - Schematic with stages grouped by function and labelled.
Attachment #2 - Measured vs modelled Transfer function.
Attachment #3 - Measured vs modelled noise. Measurement shown only between positive output and ground, the other port is basically the same. I will update this attachment to reflect the expected signal level in comparison to the noise, but suffice it to say that given the measured input referred noise, we will have plenty of SNR between 0.1Hz and 10kHz. The single stage of whitening should also be sufficient to amplify the signal above ADC noise in the same frequency band
Attachment #4 - Positive output as viewed on a fast (300 MHz) scope using a Tektronix x1 voltage probe.
Attachment #5 - Daughter board noise with measured ALS noise overlaid (the gain of x10 on the existing audio pre-amp has been divided out).
Given the overall good agreement between model and measurement, I am going to test this with the actual RF beat. For this test, we will need a differential receiving AA board to interface the output of the daughter board with the ADC input.
Next step is to actually make a prototype of this.
I thought a little bit about the next steps in testing the daughter board. The idea is to install this into the existing 1U chassis and tap the differential output from the FET Mixers as inputs to the daughter board. Looking at the D0902745 schematic, I think the best way to do this is to simply remove L3, L4, C10, C11, C15 and C16. I will then use the pads for L3 and L4 to pipe the differential output of the FET mixer to the differential input of the daughter board.
The daughter board takes care of whitening the ALS signal.
Then we need to pipe the differential output of the daughter board into the differential input of a differential receiving AA board. Koji and Johannes surveyed the available stockpile from the WB workshop. The best option seems to be to use the available v5 of D070081 and install 4 of them into a 1U chassis unit (also available from WB EE shop). The v5s can be upgraded to v6 by replacing the set of input and output buffer OpAmps with AD8622, as per the revision history notes. Koji ordered 100pcs of these today.
The input to the proposed 1U chassis housing these 8 AA boards (each with 8 channels) is a DB9 connector. The aLIGO demod board chassis that we use to demodulate the ALS signals has a nice DB25 output connector that supplies all the differential I and Q demodulated signals. But since we will install a daughter board, we will hae to hack together some connector solution anyways. I propose using a DB9 connector to pipe the outputs of the daughter board to the inputs of the AA board. Space is tight in the LSC rack, but I think we have space for a 1U chassis (see Attachment #3).
Finally - how to interface the AA board with the ADC? Koji and I discussed options, and seems like the least painful way will be to install a new ADC in the c1lsc expansion chassis in 1Y3. I checked the computer hardware cabinet and there seems to be 1 spare general standards 16bit ADC in there (see Attachment #1). Its health/providence is unknown. But Koji and I will test it after the meeting tomorrow. I also have another ADC card that Jamie and I removed from c1ioo sometime ago. I have labelled it as "GPIO0 LED RED", though I don't remember exactly what the problem was and can't find any elog about it. Incidentally, there are also 2 spare DAC cards available in the cabinet, although their health/rpovidence too is unknown. There are sufficient free slots in the c1lsc expansion chassis (see Attachment #2 though we will need a LIGO ADC adaptor card). Then we can just change the input ADC channels for the ALS signals in the c1lsc model.
In the short term, while the hardware for this plan is being put together, I can test the uncalibrated noise performance of the demod + daughter board combo (uncalibrated because I will make a measurement of voltage noise with an SR785 as opposed to frequency noise). A second daughter board will also need to be assembled - I'm just going to do it on another prototyping board as figuring out how to use Altium will probably take me longer. There is also the matter of fine tuning the polarization axes alignment of the input to the EX fiber coupler.
we did a bunch of tests to figure out the feasibility of the plan I outlined last night. Bottom line is: we appear to have a working 64 channel ADC (but with differential receiving that means 32 channels). But we need an aLIGO ADC adaptor card (I'm not sure of the DCC number but I think it is D0902006). See attached screenshot where we managed to add an ADC block to the IOP model on c1lsc, and it recognizes the additional ADC. The firmware on the (newly installed) working card is much newer than that on the existing card inside the expansion chassis (see Attachment #1).
Note that we have left the working ADC card inside the c1lsc expansion chassis. Plan is to give Rolf the faulty ADC card and at the same time ask him for a working adapter board.
Unrelated to this work: we have also scavenged 4 pcs of v2 of the differential receiving AA board from WB EE shop, along with a 1U chassis for the same. These are under my desk at the 40m for the moment. We will need to re-stuff these with appropriate OpAmps (and also maybe change some Rs and Cs) to make this board the same as v6, which is the version currently in use.
Since we decided to use the Acromag for readback of the temperature sensor for Kira's seismometer temperature control, I enabled logging of the channel Johannes had reserved for this purpose last week. Kira has made the physical connection of a temperature sensor to the BNC input for this channel - it reads back -2.92 V right now, which is around what I remember it being when Kira was doing her benchtop tests. I edited C0EDCU.ini to enable logging of this channel at 16 Hz. Presumably, a study of the ADC noise of the Acromag at low frequencies has to be made to ensure appropriate whitening (if any) can be added. Channel name is C1:PEM-SEIS_EX_TEMP_MON. Similarly, there is C1:PEM_SEIS_EX_TEMP_CTRL which is meant to be the control channel for the servoing. Calibration of the temperature sensor readback into temperature units remains. It also remains to be verified if we can have these slow EPICS channels integrated with a fast control model, or if the PID temperature control will be purely custom-script based as we have for the FSS slow loop.
I removed the fast channels I had setup temporarily in c1als. Recompilation and restart of the model went smoothly.
I spent today making another daughter board (so that we can use the new scheme for I and Q for one arm), testing it (i.e. measuring noise and TF and comparing to LISO model), and arranging all of this inside the 1U demod chassis. To accommodate everything inside, I decided to remove the 2 unused demod units from inside the box. I then drilled a few holes, installed the daughter boards on some standoffs, removed the capacitors and inductors as I outlined yesterday, and routed input and output signals to/from the daughter board. The outputs are routed to a D-sub on the rear panel. More details + better photo + results of testing the combined demod+daughter board signal chain tomorrow...
In Steve's absence, I've tried to keep an eye on the health of the vacuum system. From Attachment #1, the pressure of the main volume seems stable, no red flags there. I also don't here any anomalously loud sounds near the vacuum pumps. I've changed the N2 cylinders that keep V1 open twice, on Wednesday and Sunday of last week. So in summary, the vacuum system looks fine based on all the metrics I know of.
For testing the new IR ALS noise, we had decided that we would like to use the differential output of the demodulated ALS beat signal, as opposed to a single-ended output, as measurements suggested the former to be a lower noise configuration than the latter. For this purpose, Koji and I acquired a couple of old AA boards from the WB electronics shop. These are however, rev2 of the board, whereas the latest version is v6. The main difference between v2 and v6 is that (i) the THS4131 instrumentation amplifier has the Vocm pin grounded in v6 but is floating in v2 and (ii) the buffer opamps are AD8622 in v6 but are AD8672 in v2. But in fact, the boards we have are stuffed with AD8682.
I talked to Rich on Friday, and he seemed to think the AD8672 didn't have any issues noise-wise, the main reason they changed it was because its power consumption was high, and was causing overheating when several of these 1U chassis were packed closely together in an electronics rack. But the AD8682, which is what we have, has comparable power consumption to the AD8622. It is however a JFET opamp, and the voltage noise is a bit higher than the AD8622.
I am sure there is a way to LISO model a differential output opamp like the THS4131, but I thought I'd simulate the noise in LTSPICE instead. But I couldn't get that to work. So instead, I just measured the transfer function and noise of a single channel, for which Koji had expertly hacked together a custom shorting of the THS4131 Vocm pin to ground. Attachments #1 and #2 show the measurement. All looks good. Note that the phase is 180 at DC because I had hooked up the input signal opposite to what it should have been. The voltage noise of the differential outputs (each measured w.r.t. ground, with both inputs shorted to ground by a short patch cable) at 10 Hz is <100nV/rtHz, and the ADC noise is expected to be ~1uV/rtHz, so I think this is fine.
Conclusion: I think for the ALS test, we can just use the AA board in this config without worrying too much about replacing the buffer stage opamps, even though we've ordered 100pcs of AD8622.
Addendum 7 Mar 2018 11am: As per this document, the output noise of the AA board should be <75nV/rtHz from 10 Hz-50 kHz. So maybe the AD8682 noise is a little high after all. I've gotten the LTSpice model working now, will post the comparison of modelled output noise for various combinations here shortly.
I did a quick test of the noise of the new ALS electronics with the X arm ALS. Attachment #1 shows the results - but something looks off in the measurement, especially the "LO driven, RF terminated" trace. I will have to defer further testing to tomorrow. Of course the real test is to digitize these signals and look at the spectrum of the phase tracker output, but I wanted a voltage noise comparison first. Also, note that I have NOT undone the whitening TFs of (z,p) = (15,150) on these traces. I wonder if these noisy signals (particularly the 10Hz multiple harmonics) are an artefact of measurement, or if something is wonky in the daughter board circuits themselves. I am measuring these with the help of a DB9 breakout board and some pomona minigrabbers. Reagrdless, the sort of ripple seen in the olive green trace for the I channel wasn't present when I did the same test with RF signal generators out on the electronics workbench, so I am inclined to think that this isn't a problem with the circuit. I'm measuring with the SR785 with the "A" input setting, but with the ground set to "Float". I need to look into what the difference is between this mode, and the "A-B" mode. At first glance, both seem to be equivalent differential measurements, but I wonder if there is some subtlety w.r.t. pickup noise.
Perhaps I can repeat the test at the output of the AA board. I looked into whether there is a spare +/- 24V DC power supply available at the LSC rack, to power the 1U AA chassis, but didn't see anything there.
Here are the plots. Comments:
I like LTspice for such modeling - the GUI is nice to have (though I personally think that typing out a nodal file a la LISO is faster), and compared to LISO, I think that the LTspice infrastructure is a bit more versatile in terms of effects that can be modeled. We can also easily download SPICE models for OpAmps from manufacturers and simply add them to the library, rather than manually type out parameters in opamp.lib for LISO. But the version available for Mac is somewhat pared down in terms of the UI, so I had to struggle a bit to find the correct syntax for the various simulation commands. The format of the exported data is also not as amenable to python plotting as LISO output files, but i'm nitpicking...
I've gotten the LTSpice model working now, will post the comparison of modelled output noise for various combinations here shortly.
I am almost ready for a digital test of the new ALS electronics. Today, Koji and I spent some time tapping new +/-24VDC DIN terminal blocks at the LSC rack to facilitate the installation of the 1U differential receiving AA chassis (separate elog entry). The missing piece of the puzzle now is the timing adapter card. I opted against trying a test tonight as I am having some trouble bringing c1lsc back online.
Incidentally, a repeat of the voltage noise measurement of the X arm ALS beat looked much cleaner today, see Attachment #1 - I don't have a good hypothesis as to why sometimes the signal has several harmonics at 10Hz multiples, and sometimes it looks just as expected. The problem may be more systematically debuggable once the signals are being digitally acquired.
This required multiple hard reboots, but seems like all the RT models are back for now. The only indicator I can't explain is the red DC field on c1oaf. Also, the SUS model seems to be overclocking more frequently than usual, though I can't be sure. The "timing" field of this model's state word is RED, while the other models all seem fine. Not sure what could be going on.
Will debug further tomorrow, when I probably will have to do all this again as I'll need to recompile c1lsc for the ALS electronics test with the new ADC card from the differential AA board.
As I had found before, restarting the c1oaf model fixed the DC error. There is however still a pesky red indicator light on the "ADC0" in c1oaf. Trying to open up the ADC MEDM screen to investigate this further leads to the blank screen on the bottom right of Attachment #1. Probably has something to do with the fact that the model has an ADC block (because every model needs one?) but no signals are actually being piped to the model directly from the ADC.
Another observation, though I don't have any hypothesis as to why this was happening: on the c1sus machine, the c1sus model would frequently overclock, and then eventually, crash. I observed this behaviour at least 3 times between last night and now. The other models seemed fine though, in fact, IMC stayed locked. Why should this have been the case? It remains to be seen if this was somehow connected to the red DC indicator on c1oaf, though why should this be the case? Isn't the DC just concerned with writing data to frames? Any sort of IPC should be independent? Attachment #2 shows that there's been a definite increase in the maximum time on c1sus clock-cycle since yesterday (it's a 10 day minute trend plot of the model clock cycle timing and also the maximum time). Why? Koji and I did switch off all the Sorensens at the LSC rack for about 30mins, but why should this affect anything at 1X6? There are no red lights in either the c1lsc or c1sus expansion chassis. Curiously, the PRM also seems to be glitchy - as I'm sitting in the control room, I see a spot flashing across vertically on the REFL CRT monitor sporadically. Note that nominally, with PRM misaligned, the REFL CRT should be dark. dmesg on c1sus doesn't shed any light on the issue.
Seems like some high level voodoo .
Yesterday, we installed some new DIN rail connectors at the LSC rack to provide 3 new outputs each for +24V DC and -24V DC. The main motivation was to facilitate the installation and powering of the differential receiving AA board. The regulators used inside the 1U chassis actually claims a dropout voltage of 0.5V and outputs 14V nominally, so a +/-15V DC supply would've perhaps been sufficient, but we decided to leave a bit more margin, and unfortunately, there are no +/-18V DC KEPCO linear power supplies to the LSC rack. Procedure:
The c1lsc frontend models crashed for some reason during this procedure. Now the c1sus frontend model is also behaving weirdly. It is unclear to me if/how this work would have led to these problems, but the temporal correlation (but not causation?) is undeniable.
I was forced into a simultaneous power-cycle rebooting of the three vertex FEs just now. I took the opportunity to completely disconnect the c1sus expansion chassis from all power and then restart it.
Everything is back up right now, and the weird timing issues I noticed in the sus model seem to be gone now (I'll need a longer baseline to be sure and I'll post a trend of the CPU timing tomorrow). It's disconcerting that apparently the only way to get everything back up and running is the nuclear option of power-cycling all FE related electronics. I was considering borrowing an ADC adapter card from the Y end and measuring the calibrated IR ALS noise with the digital system, but if I'm going to have to go through this whole dance each time I do a model recompile on c1lsc (which I'm going to have to in order to get the extra ADC recognized), I'm wondering if it's just better to wait till we get the new adapter cards we ordered. I think I'm going to work on tuning the input coupling into the fiber at EX in the next couple of days instead.
I made a LISO fit of the measured TF of the daughter board, so that I can digitally invert the daughter board whitening. Results attached. (Inverse) Filters have been uploaded to the ALS X Foton filter banks.
Then I looked at the spectrum, see Attachment #1. Disappointingly, it looks like the arm PDH servo is dominating the noise, and NOT unsuppressed EX laser frequency noise,. Not sure why this is so, and I'm feeling too tired to debug this tonight. But encouragingly, the performance of the new ALS signal chain looks very promising. Once I tune up the X arm loop, I'm confident that the ALS noise will be at least as good as the reference trace.
I am leaving c1iscey shutdown until this is fixed. So ETMY is not available for the moment.
Random factoid: Trying to print a DTT trace with LaTeX in the label text on pianosa causes the DTT window to completely crash - so if you dont save the .xml file, you lose your measurement.
I was going to head out but then it occurred to me that I could do another simple test, which is to try and lock the X arm on ALS error signal (i.e. actuate on MC length to keep the beat between EX laser and PSL fixed, while the EX frequency is following the Xarm length). Comparing the in loop (i.e. ALS) error signal with the out-of-loop sensor (i.e. POX), it seems like POX is noisy. The curves were lined up by eye, by scaling the blue curve to match the red at the ~16Hz peaks. This supports my hypothesis in the previous elog. On the downside, could be anything. Electronics in the POX chain? The demod unit itself? Will look into it more tomorrow..
As an aside, controlling the arm with ALS error signal worked quite well, and the lock was maintained for ~1 hour.
Bulb went out ~10am today. Looks like the lifetime of this bulb was <100 days.
Steve: bulb is arriving next week
Bulb is replaced.
we tested my noisy POX hypothesis tonight. By locking the single arm with POX, the arm length is forced to follow PSL frequency, which is itself slaved to IMC length. From Attachment #1, there is no coherence between the arm control signal and MC_F. This suggests to me that the excess noise I am seeing in the arm control signal above 30 Hz is not originating from the PSL. It also seems unlikely that at >30Hz, anything mechanical is to blame. So I am sticking with the hypothesis that something is wonky with POX. For reference, a known "normal" arm control signal spectrum looks like the red curve in this elog.
Kevin suggested I shouldn't be so lazy and test the POY spectrum as well. So we moved the timing card back to c1iscey, went through the usual dance of vertex machine reboots, and then got both single arm locks going. Attached spectrum shows that both POX and POY are noisy. I'm not sure what has changed that could cause this effect. The fact that both POX and POY appear uniformly bad, but that there is no coherence with MC_F, suggests to me that perhaps this has something to do with the work I did with Koji w.r.t. the power situation at the LSC rack. But we just checked that
Another observation we made: note the huge bump around 70Hz in both arm control signals. We don't know what the cause of this is. But we occassionally noticed harmonics of this (i.e. 140, 210 Hz etc) appear in the control signal spectra, and they would grow with time - eventually, the X arm would lose lock (though the Y arm stayed locked).
I'm short on ideas for now so we will continue debugging tomorrow.
Unrelated to this work: Kevin reminded me that the high-pitched whine from the CRT TVs in the control room (which is apparently due to the flyback transformer) is DEAFENING. It's curious that the "chirp" to the eventual 15kHz whine is in opposite directions for the QUAD CRTs and the single display ones. Should be a Ph6 experiment maybe.
Update 2:30pm Mar 13: The furthest back I seem to be able to go in time with Frames is ~Jan 20 2018. Looking for a time when the arms were locked from back then, it seems like whatever is responsible for a noisy POX and POY was already a problem back in January. See Attachment #2. So it appears that the recent work at 1Y2 is not to blame...
The working hypothesis, since the excess noise in single arm locks is coherent between both arms, the excess sensing noise is frequency noise in the IMC locking loop (sensing because it doesn't show up in MC_F). I've started investigating the IMC sensing chain, starting with the power levels of the RF modulation source. Recall that we had changed the way the 29.5MHz signal was sent to the EOM and demod electronics in 2017. With the handheld RF power meter, I measured 13.2dBm coming out of the RF distribution box (this is routed straight from the Wenzel oscillator). This is amplified to 26dBm by an RF amplifier (ZHL-2-S) and sent to the EOM, with a coupled 16dBm part sent to a splitter that supplies the LO signal to the demod board and also the WFS boards. Lydia made a summary of expected RF power levels here, and I too seem to have labelled the "nominal" LO level to the MC_REFL demod board as +5dBm. But I measured 2.7dBm with the RF power meter. But looking closely at the schematic of the splitting circuitry, I think for a (measured) 16.7dBm input to it, we should in fact expect around 3dBm of output signal. So I don't know why I labelled the "nominal" signal level as 5dBm.
Bottom line: we are driving a level 17 mixer with more like +14dBm (a number inferred from this marked up schematic) of LO, which while isn't great, is unlikely to explain the excess noise I think (the conversion loss just degrades by ~1dB). So I will proceed to check further downstream in the signal chain.
It is not clear to me why installing an attenuator to prevent amplifier saturation has necessitated a 10dB increase in the IN1 gain and 3dB increase in the VCO gain. Initially, I was trying to compensate for the gain by increasing the FSS "Common Gain" but in that setting, I found an OLTF measurement impossible. The moment I enabled the excitation input to the CM board, the lock was blown, even with excitation amplitudes as small as -60dBm (from the Agilent network analyzer).
This may also be a good opportunity to test out one of the aLIGO style FET mixer demod boards (recall we have 2 spare from the 4 that were inside the ALS demod box). I'm going to ask Steve to package these into a 1U chassis so that I can try that setup out sometime. From a noise point of view, the aLIGO boards have the advantage of having a x100 preamp stage straight after the mixer+LPF. We may need to replace the lowpass filter though, I'm not sure if the one installed is 1.9MHz or 5MHz.
I've left an SR785 and AG4395 near 1X2 in anticipation of continuing this work tomorrow.
Unrelated to this work - seems like the WFS DC and RF offsets had not been set in a while so I reset these yesterday. The frequent model restarts in recent times may mean that we have to reset these to avoid using dated offset values.
Re-measured the demod board noises after replacing the suspect ERA-5SMs, with LO driven by a marconi at the "nominal" level of 2.5dBm, and RF input terminated. Attachment #1 is the input referred voltage noise spectra. I used the FET low noise pre-amp box for this purpose. I cannot explain the shape of the spectra above 1kHz. I tried doing the measurement on a minicircuits mixer (non-surface mount) and found the shape to be flat throughout the SR785 span. Unclear what else could be going on in the demod board though, all the other components on it are passive (except the ERA-5SMs which were replaced). I considered adopting a PMC style demod setup where we do the demod using some separate Minicircuits Mixer+LowPass filter combo. But the RF flashes for the IMC monitored at the RFmon port are ~0.2Vpp, and so the RF input to the mixer is expected to be ~2Vpp. The minicircuits mixer selection guide recommends choosing a diode mixer with LO level at least 10dBm above the expected RF input signal level, and we don't have any standalone mixers that are >Level 7. I've asked Steve to package the aLIGO demod board in the meantime, but even that might not be a plug and play replacement as the IF preamp stage has ~120degrees phase lag at 1MHz, which is significantly higher than the existing board which just has a SCLF5 low pass filter after the mixer and hence has <45degrees phase lag at 1MHz.
according to the temp sensor readout, which was ~-3.35V which corresponds to ~335K, the temperature of the can is now 60 deg C. This is a bit warm for my liking so i'm turning the heater current down to 0 now by writing 0 to C1:PEM-SEIS_EX_TEMP_CTRL
This elog by koji inspired me to consider power supply as a possible issue.
The demod board receives +/-24V DC (which is regulated down to +/-15V DC by 7815/7915), and also +15V DC via the backplane. The ERA-5SM receives DC power from the latter (unregulated) +15V DC. I can't think of why this is the case except perhaps the regulators can't source the current the amp wants? In any case, it doesn't look feasible to change this by cutting any traces on the PCB to me. While I had the board out, I decided to replace the JMS-1H mixers in a last ditch effort to improve the demod board noise. Unfortunately I'm having trouble de-soldering these MCL components from the board. So for now, I'm leaving the demod board out, IMC unlocked. Work will continue tomorrow.
After some persistence, I managed to get the mixers off.
Unfortunately, the coherent noise between the arms persists so the sensing noise injection must be happening elsewhere. IMC seems to lock fine though so I'm leving the autolocker on
As discussed at the meeting, I decided to calibrate the MC error point into physical units of Hz/rtHz (a.k.a. the PDH discriminant). This is to facilitate the debugging of the hypothesized excess IMC sensing noise. I did this as follows.
Using this, I can now make up a noise budget of sorts for the IMC sensing.
gautam 20180327 4.30pm: I re-checked the PDH error signal calibration using the oscilloscope method. Attachment #3 shows the PDH I and Q error signals and also the output of the RF monitor port, during a TEM00 flash. This attachment should be compared to Attachment #2 of elog 12822, and the answer lines up quite well. From my Finesse model of the IMC, I calculated that the x-axis of the PDH horn-to-horn is ~12.3kHz. Comparing to the top row of Attachment #3, I get a PDH error signal calibration of ~12.4kHz/Vrms, which lines up well with the number quoted above. So I trust my calibration, and hence, the y-axis of my noise budgets in reply to this elog.
I did a preliminary noise budget of the transmitted frequency noise of the IMC. Attachment #1 shows the NB. I'm going to use this opportunity to revisit my IMC modeling. Some notes:
Conclusion: From this study, assuming my PDH discriminant calibration was correct, looks like IMC demod / POX11 demod electronics noises are not to blame (this surprises me since there were apparently so many things wrong on the demod board, and yet that wasn't the worst thing in the IMC chain it would seem ). The POX11 photodiode "dark" noise is also not the problem I think, given the grey curve. Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
I've added two curves to the NB. Both are measured (with FET preamp) at the output of the demod board, with the LO driven at the nominal level by the Wenzel RF source pickoff (as it would be when the IMC is locked) and the RF input connected to the IMC REFL PD. For one curve, I simply closed the PSL shutter, while for the other, I left the PSL shutter open, but macroscopically misaligned MC2 so that there was no IMC cavity. So barring RFAM, there should be no PDH signal on the REFL PD, but I wanted to have light on there. I'm not sure if I understand the difference between these two curves though, need to think on it. Perhaps the IMC REFL PD's optical/electrical response needs to be characterized?
Next curve to go on here is the demod board noise with the PSL shutter closed but the IMC REFL PD connected to the RF input (or maybe even better, have light on the PD, but macroscopically misalign MC2 so there is no 29.5MHz PDH signal), just to make sure there isn't anything funky going on there...
we measured the RIN of the MC2 transmission using the PDA255 I had put on the MC2 trans table sometime ago for ringdowns. Attached are (i) spectra for the RIN, (ii) spectra for the classical rad. pressure noise assuming 500W circulating power and (iii) a tarball of data and code used to generate these plots.
We took a full span measurement (to make sure there aren't any funky high-freq features) and a measurement from DC-800 Hz (where we are looking for excess noise). The DC level of light on the photodiode was 2.76V (measured using o'scope)
I'll add this to the noise budget later. But the measured RIN seems consistent with a 2013 measurement at 100Hz (though the 2013 measurement is using DTT and so doesn't have high frequency information).
Todd informed me that the ADC Timing adaptor boards we had ordered arrived today. I had to solder on the components and connectors as per the schematic, though the main labor was in soldering the high density connectors. I then proceeded to shut down all models on c1lsc (and then the FE itself). Then classic problem of all vertex machines crashing when unloading models on c1lsc happened (actually Koji noticed that this was happening even on c1ioo). Anyways this was nothing new so I decided to push ahead.
I had to get a cable from Downs that connects the actual GS ADC card to this adaptor board. I powered off the expansion chassis, installed the adaptor board, connected it to the ADC card and restarted the expansion chassis and also the FE. I also reconnected the SCSI cable from the AA board to the adaptor card. It was a bit of a struggle to get all the models back up and running again, but everything eventually came back(after a few rounds of hard rebooting). I then edited the c1x04 and c1lsc simulink models to reflect the new path for the X arm ALS error signals. Seems to work alright.
At some point in the afternoon, I noticed a burning smell concentrated near the PSL table. Koji traced the smell down to the c1lsc expansion chassis. We immediately powered the chassis off. But Steve later informed me that he had already noticed an odd burning smell in the morning, before I had done any work at the LSC rack. Looking at the newly installed adaptor card, there wasn't any visual evidence of burning. So I decided to push ahead and try and reboot all models. Everything came back up normally eventually, see Attachment #1. Particle count in the lab seems a little higher than usual (actually, according to my midnight measurement, they are ~factor of 10 lower than Steve's 8am measurements), but Steve didn't seem to think we should read too much into this. Let's monitor the situation over the coming days, Steve should comment on the large variance seen in the particle counter output which seems to span 2 orders of magnitude depending on the time of the day the measurement is made... Also note that there is a BIO card in the C1LSC expansion chassis that is powered by a lab power supply unit. It draws 0 current, even though the label on it says otherwise. I a not sure if the observed current draw is in line with expectations.
The spare (unstuffed) adaptor cards we ordered, along with the necessary hardware to stuff them, are in the Digital FE hardware cabinet along the east arm.
Steve: particle count in the 40m is following outside count, wind direction, weather condition .....etc. The lab particle count is NOT logged ! This is bad practice.
While Kevin and Arijit were doing their MC_REFL PD noise measurements (which they will elog about separately shortly), I noticed a feature around 600kHz that reminded me of the NPRO noise eater feature. This is supposed to suppress the relaxation oscillation induced peak in the RIN of the PSL. Surprisingly, the noise eater switch on the NPRO front panel was set to "OFF". Is this the normal operating state? I thought we want the noise eater to be "ON"? Have to measure the RIN on the PSL table itself with one of the many available pick off PDs. In any case, as Attachment #1 showed, turning the noise eater back on did not improve the excess IMC frequency noise.
I've removed the MC REFL PD unit from the AS table for investigation. So MC won't lock.
PSL shutter was closed and location of PD was marked with sharpie (placing guides to indicate position wasn't convenient). I also kapton taped the PD to minimize dust settling on the PD while I have it out in the electronics area. Johannes has the camera, and my cellphone image probably isn't really high-res enough for diagnostics but I'm posting it here anyways for what it's worth. More importantly - the board is a D980454 revision B judging by the board, but there is no schematic for this revision on the DCC. The closest I can find is a D980454 Rev D. But I can already see several differences in the component layout (though not all of them may be important). Making a marked up schematic is going to be a pain . I'm also not sure what the specific make of the PD installed is.
The lid of the RF cage wasn't on.
More to follow tomorrow, PD is on the electronics workmench for now...
gautam 28 March 2018: Schematic has been found from secret Dale stash (which exists in addition to the secret Jay stash). It has also been added to the 40m electronics tree.
I re-installed the MC REFL photodiode. Centered beam on the PD by adjusting steering mirror to maximize the DC signal level (on o'scope) at the DC monitoring port. Curiously, the DC level on the scope (high-Z) was ~2.66V DC, whereas the MEDM screen reports ~twice that value, at ~5.44 "V". We may want to fix this "calibration" (or better yet, use physical units like mW). Noise-eater On/Off comparison of MC error signals to follow.
It's been a while - but today, all slow machines (with the exception of c1auxex) were un-telnetable. c1psl, c1iool0, c1susaux, c1iscaux1, c1iscaux2, c1aux and c1auxey were rebooted. Usual satellite box unplugging was done to avoid ITMX getting stuck.